Add vector-sets module

The vector-sets module is a part of Redis Core and is available by default, just like any other data type in Redis. As a result, when building Redis from the source, the vector-sets module is also compiled as part of the Redis binary and loaded at server start-up. This new data type added as a preview currently doesn't support all the capabilities in Redis like: 32-bit OS C99 Short-read that might end with memory leak AOF rewirte defrag
2025-04-02 16:59:16 +03:00 · 2025-04-02 16:59:16 +03:00 · 41b1b5df18
parent 78e0d87177
commit 41b1b5df18
23 changed files with 539 additions and 1756 deletions
--- a/.github/workflows/daily.yml
+++ b/.github/workflows/daily.yml
@ -632,7 +632,7 @@ jobs:
          repository: ${{ env.GITHUB_REPOSITORY }}
          ref: ${{ env.GITHUB_HEAD_REF }}
      - name: make
-        run: make SANITIZER=undefined REDIS_CFLAGS='-DREDIS_TEST -Werror' LUA_DEBUG=yes # we (ab)use this flow to also check Lua C API violations
+        run: make SANITIZER=undefined REDIS_CFLAGS='-DREDIS_TEST -Werror' SKIP_VEC_SETS=yes LUA_DEBUG=yes # we (ab)use this flow to also check Lua C API violations
      - name: testprep
        run: |
          sudo apt-get update
--- a/modules/common.mk
+++ b/modules/common.mk
@ -25,6 +25,7 @@ all: $(TARGET_MODULE)

 $(TARGET_MODULE): get_source
 	$(MAKE) -C $(SRC_DIR)
+	cp ${TARGET_MODULE} ./

 get_source: $(SRC_DIR)/.prepared

@ -35,8 +36,9 @@ $(SRC_DIR)/.prepared:

 clean:
 	-$(MAKE) -C $(SRC_DIR) clean
+	rm ./*.so

-distclean:
+distclean: clean
 	-$(MAKE) -C $(SRC_DIR) distclean

 pristine:
--- a/modules/vector-sets/LICENSE
+++ b/modules/vector-sets/LICENSE
@ -1,2 +0,0 @@
-This code is Copyright (c) 2024-Present, Redis Ltd.
-All Rights Reserved.
--- a/modules/vector-sets/Makefile
+++ b/modules/vector-sets/Makefile
@ -53,7 +53,7 @@ all: vset.so
 .c.xo:
 	$(CC) -I. $(CFLAGS) $(SHOBJ_CFLAGS) -fPIC -c $< -o $@

-vset.xo: redismodule.h expr.c
+vset.xo: ../../src/redismodule.h expr.c

 vset.so: vset.xo hnsw.xo cJSON.xo
 	$(CC) -o $@ $^ $(SHOBJ_LDFLAGS) $(LIBS) $(SAN) -lc
--- a/modules/vector-sets/README.md
+++ b/modules/vector-sets/README.md
@ -156,7 +156,7 @@ Because vector sets perform insertion time normalization and optional
 quantization, the returned vector could be approximated. `VEMB` will take
 care to de-quantized and de-normalize the vector before returning it.

-It is possible to ask VEMB to return raw data, that is, the interal representation used by the vector: fp32, int8, or a bitmap for binary quantization. This behavior is triggered by the `RAW` option of of VEMB:
+It is possible to ask VEMB to return raw data, that is, the internal representation used by the vector: fp32, int8, or a bitmap for binary quantization. This behavior is triggered by the `RAW` option of of VEMB:

    VEMB word_embedding apple RAW

@ -530,10 +530,10 @@ Notably, this pattern can be implemented in a way that avoids paying the sum of

 Vector Sets, or better, HNSWs, the underlying data structure used by Vector Sets, combined with the features provided by the Vector Sets themselves (quantization, random projection, filtering, ...) form an implementation that has a non-trivial space of parameters that can be tuned. Despite to the complexity of the implementation and of vector similarity problems, here there is a list of simple ideas that can drive the user to pick the best settings:

-* 8 bit quantization (the default) is almost always a win. It reduces the memory usage of vectors by a factor of 4, yet the performance penality in terms of recall is minimal. It also reduces insertion and search time by around 2 times or more.
+* 8 bit quantization (the default) is almost always a win. It reduces the memory usage of vectors by a factor of 4, yet the performance penalty in terms of recall is minimal. It also reduces insertion and search time by around 2 times or more.
 * Binary quantization is much more extreme: it makes vector sets a lot faster, but increases the recall error in a sensible way, for instance from 95% to 80% if all the parameters remain the same. Yet, the speedup is really big, and the memory usage of vectors, compaerd to full precision vectors, 32 times smaller.
 * Vectors memory usage are not the only responsible for Vector Set high memory usage per entry: nodes contain, on average `M*2 + M*0.33` pointers, where M is by default 16 (but can be tuned in `VADD`, see the `M` option). Also each node has the string item and the optional JSON attributes: those should be as small as possible in order to avoid contributing more to the memory usage.
-* The `M` parameter should be incresed to 32 or more only when a near perfect recall is really needed.
+* The `M` parameter should be increased to 32 or more only when a near perfect recall is really needed.
 * It is possible to gain space (less memory usage) sacrificing time (more CPU time) by using a low `M` (the default of 16, for instance) and a high `EF` (the effort parameter of `VSIM`) in order to scan the graph more deeply.
 * When memory usage is seriosu concern, and there is the suspect the vectors we are storing don't contain as much information - at least for our use case - to justify the number of components they feature, random projection (the `REDUCE` option of `VADD`) could be tested to see if dimensionality reduction is possible with acceptable precision loss.

--- a/modules/vector-sets/examples/cli-tool/cli.py
+++ b/modules/vector-sets/examples/cli-tool/cli.py
@ -1,3 +1,11 @@
+#
+# Copyright (c) 2009-Present, Redis Ltd.
+# All rights reserved.
+#
+# Licensed under your choice of the Redis Source Available License 2.0
+# (RSALv2) or the Server Side Public License v1 (SSPLv1).
+#
+
 #!/usr/bin/env python3
 import redis
 import requests
--- a/modules/vector-sets/examples/glove-100/insert.py
+++ b/modules/vector-sets/examples/glove-100/insert.py
@ -1,3 +1,11 @@
+#
+# Copyright (c) 2009-Present, Redis Ltd.
+# All rights reserved.
+#
+# Licensed under your choice of the Redis Source Available License 2.0
+# (RSALv2) or the Server Side Public License v1 (SSPLv1).
+#
+
 import h5py
 import redis
 from tqdm import tqdm
--- a/modules/vector-sets/examples/glove-100/recall.py
+++ b/modules/vector-sets/examples/glove-100/recall.py
@ -1,3 +1,11 @@
+#
+# Copyright (c) 2009-Present, Redis Ltd.
+# All rights reserved.
+#
+# Licensed under your choice of the Redis Source Available License 2.0
+# (RSALv2) or the Server Side Public License v1 (SSPLv1).
+#
+
 import h5py
 import redis
 import numpy as np
--- a/modules/vector-sets/examples/movies/insert.py
+++ b/modules/vector-sets/examples/movies/insert.py
@ -1,3 +1,11 @@
+#
+# Copyright (c) 2009-Present, Redis Ltd.
+# All rights reserved.
+#
+# Licensed under your choice of the Redis Source Available License 2.0
+# (RSALv2) or the Server Side Public License v1 (SSPLv1).
+#
+
 import csv
 import requests
 import redis
--- a/modules/vector-sets/expr.c
+++ b/modules/vector-sets/expr.c
@ -3,7 +3,11 @@
 * general code to be used when we want to tell if a given object (with fields)
 * passes or fails a given test for scalars, strings, ...
 *
- * Copyright(C) 2024-Present, Redis Ltd. All Rights Reserved.
+ * Copyright (c) 2009-Present, Redis Ltd.
+ * All rights reserved.
+ *
+ * Licensed under your choice of the Redis Source Available License 2.0
+ * (RSALv2) or the Server Side Public License v1 (SSPLv1).
 * Originally authored by: Salvatore Sanfilippo.
 */

@ -83,7 +87,7 @@ typedef struct exprstate {
    char *expr;             /* Expression string to compile. Note that
                             * expression token strings point directly to this
                             * string. */
-    char *p;                // Currnet position inside 'expr', while parsing.
+    char *p;                // Current position inside 'expr', while parsing.

    // Virtual machine state.
    exprstack values_stack;
@ -685,7 +689,7 @@ double exprTokenToNum(exprtoken *t) {
    }
 }

-/* Conver obejct to true/false (0 or 1) */
+/* Convert object to true/false (0 or 1) */
 double exprTokenToBool(exprtoken *t) {
    if (t->token_type == EXPR_TOKEN_NUM) {
        return t->num != 0;
--- a/modules/vector-sets/hnsw.c
+++ b/modules/vector-sets/hnsw.c
@ -13,7 +13,7 @@
 *    be not close enough to replace old links in candidate.
 *
 * 2. We normalize on-insert, making cosine similarity and dot product the
- *    same. This means we can't use euclidian distance or alike here.
+ *    same. This means we can't use euclidean distance or alike here.
 *    Together with quantization, this provides an important speedup that
 *    makes HNSW more practical.
 *
@ -25,7 +25,11 @@
 *    bidirectional), and reliking the nodes orphaned of one link among
 *    them.
 *
- * Copyright(C) 2024-Present, Redis Ltd. All Rights Reserved.
+ * Copyright (c) 2009-Present, Redis Ltd.
+ * All rights reserved.
+ *
+ * Licensed under your choice of the Redis Source Available License 2.0
+ * (RSALv2) or the Server Side Public License v1 (SSPLv1).
 * Originally authored by: Salvatore Sanfilippo.
 */

@ -109,7 +113,7 @@ typedef struct {
 } pqueue;

 /* The HNSW algorithms access the pqueue conceptually from nearest (index 0)
- * to farest (larger indexes) node, so the following macros are used to
+ * to farthest (larger indexes) node, so the following macros are used to
 * access the pqueue in this fashion, even if the internal order is
 * actually reversed. */
 #define pq_get_node(q,i) ((q)->items[(q)->count-(i+1)].node)
@ -209,7 +213,7 @@ float vectors_distance_float(const float *x, const float *y, uint32_t dim) {
    }

    /* Handle the remaining elements. These are a minority in the case
-     * of a smal vector, don't optimze this part. */
+     * of a small vector, don't optimize this part. */
    for (; i < dim; i++) dot0 += x[i] * y[i];

    /* The following line may be counter intuitive. The dot product of
@ -897,7 +901,7 @@ void hnsw_update_worst_neighbor_on_remove(HNSW *index, hnswNode *node, uint32_t
    }
 }

-/* We have a list of candidate nodes to link to the new node, when iserting
+/* We have a list of candidate nodes to link to the new node, when inserting
 * one. This function selects which nodes to link and performs the linking.
 *
 * Parameters:
@ -906,14 +910,14 @@ void hnsw_update_worst_neighbor_on_remove(HNSW *index, hnswNode *node, uint32_t
 *   new node 'new_node'.
 * - 'required_links' is as many links we would like our new_node to get
 *   at the specified layer.
- * - 'aggressive' changes the startegy used to find good neighbors as follows:
+ * - 'aggressive' changes the strategy used to find good neighbors as follows:
 *
 * This function is called with aggressive=0 for all the layers, including
 * layer 0. When called like that, it will use the diversity of links and
 * quality of links checks before linking our new node with some candidate.
 *
 * However if the insert function finds that at layer 0, with aggressive=0,
- * few connections were made, it calls this function again with agressiveness
+ * few connections were made, it calls this function again with aggressiveness
 * levels greater up to 2.
 *
 * At aggressive=1, the diversity checks are disabled, and the candidate
@ -925,7 +929,7 @@ void hnsw_update_worst_neighbor_on_remove(HNSW *index, hnswNode *node, uint32_t
 * a connection (to make space for our new node link). In this case:
 *
 * 1. If such "dropped" node would remain with too little links, we try with
- *    some different neighbor instead, however as the 'aggressive' paramter
+ *    some different neighbor instead, however as the 'aggressive' parameter
 *    has incremental values (0, 1, 2) we are more and more willing to leave
 *    the dropped node with fever connections.
 * 2. If aggressive=2, we will scan the candidate neighbor node links to
@ -1047,7 +1051,7 @@ void select_neighbors(HNSW *index, pqueue *candidates, hnswNode *new_node,
        {
            /* Let's see if we can find at least a candidate link that
             * would remain with a few connections. Track the one
-             * that is the farest away (worst distance) from our candidate
+             * that is the farthest away (worst distance) from our candidate
             * neighbor (in order to remove the less interesting link). */
            worst_node = NULL;
            uint32_t worst_idx = 0;
@ -1192,7 +1196,7 @@ void hnsw_reconnect_nodes(HNSW *index, hnswNode **nodes, int count, uint32_t lay
    /* Step 1: Build the distance matrix between all nodes.
     * Since distance(i,j) = distance(j,i), we only compute the upper triangle
     * and mirror it to the lower triangle. */
-    float *distances = hmalloc(count * count * sizeof(float));
+    float *distances = hmalloc((unsigned long) count * count * sizeof(float));
    if (!distances) return;

    for (int i = 0; i < count; i++) {
@ -1206,7 +1210,7 @@ void hnsw_reconnect_nodes(HNSW *index, hnswNode **nodes, int count, uint32_t lay

    /* Step 2: Calculate row averages (will be used in scoring):
     * please note that we just calculate row averages and not
-     * colums averages since the matrix is symmetrical, so those
+     * columns averages since the matrix is symmetrical, so those
     * are the same: check the image in the top comment if you have any
     * doubt about this. */
    float *row_avgs = hmalloc(count * sizeof(float));
@ -1231,7 +1235,7 @@ void hnsw_reconnect_nodes(HNSW *index, hnswNode **nodes, int count, uint32_t lay
     * good is a given i,j nodes connection, with how badly connecting
     * i,j will affect the remaining quality of connections left to
     * pair the other nodes. */
-    float *scores = hmalloc(count * count * sizeof(float));
+    float *scores = hmalloc((unsigned long) count * count * sizeof(float));
    if (!scores) {
        hfree(distances);
        hfree(row_avgs);
@ -1397,7 +1401,7 @@ void hnsw_reconnect_nodes(HNSW *index, hnswNode **nodes, int count, uint32_t lay

        // If still no connection, search the broader graph.
        if (nodes[i]->layers[layer].num_links != wanted_links) {
-            debugmsg("No force linking possible with local candidats\n");
+            debugmsg("No force linking possible with local candidates\n");
            pq_free(candidates);

            // Find entry point for target layer by descending through levels.
@ -1414,7 +1418,7 @@ void hnsw_reconnect_nodes(HNSW *index, hnswNode **nodes, int count, uint32_t lay

            if (curr_ep) {
                /* Search this layer for candidates.
-                 * Use the defalt EF_C in this case, since it's not an
+                 * Use the default EF_C in this case, since it's not an
                 * "insert" operation, and we don't know the user
                 * specified "EF". */
                candidates = search_layer(index, nodes[i], curr_ep, HNSW_EF_C, layer, 0);
@ -1576,7 +1580,7 @@ int hnsw_delete_node(HNSW *index, hnswNode *node, void(*free_value)(void*value))
 }

 /* ============================ Threaded API ================================
- * Concurent readers should use the following API to get a slot assigned
+ * Concurrent readers should use the following API to get a slot assigned
 * (and a lock, too), do their read-only call, and unlock the slot.
 *
 * There is a reason why read operations don't implement opaque transparent
@ -2348,7 +2352,7 @@ hnswNode *hnsw_cursor_next(hnswCursor *cursor) {
 }

 /* Called by hnsw_unlink_node() if there is at least an active cursor.
- * Will scan the cursors to see if any cursor is going to yeld this
+ * Will scan the cursors to see if any cursor is going to yield this
 * one, and in this case, updates the current element to the next. */
 void hnsw_cursor_element_deleted(HNSW *index, hnswNode *deleted) {
    hnswCursor *x = index->cursors;
@ -2539,7 +2543,7 @@ int hnsw_validate_graph(HNSW *index, uint64_t *connected_nodes, int *reciprocal_
 *
 * This is just a debugging function that reports stuff in the standard
 * output, part of the implementation because this kind of functions
- * provide some visiblity on what happens inside the HNSW.
+ * provide some visibility on what happens inside the HNSW.
 */
 void hnsw_test_graph_recall(HNSW *index, int test_ef, int verbose) {
    // Stats
--- a/modules/vector-sets/hnsw.h
+++ b/modules/vector-sets/hnsw.h
@ -2,7 +2,12 @@
 * HNSW (Hierarchical Navigable Small World) Implementation
 * Based on the paper by Yu. A. Malkov, D. A. Yashunin
 *
- * Copyright(C) 2024-Pesent Redis Ltd. All Rights Reserved.
+ * Copyright (c) 2009-Present, Redis Ltd.
+ * All rights reserved.
+ *
+ * Licensed under your choice of the Redis Source Available License 2.0
+ * (RSALv2) or the Server Side Public License v1 (SSPLv1).
+ * Originally authored by: Salvatore Sanfilippo.
 */

 #ifndef HNSW_H
--- a/modules/vector-sets/redismodule.h
+++ b/modules/vector-sets/redismodule.h
--- a/modules/vector-sets/test.py
+++ b/modules/vector-sets/test.py
@ -2,8 +2,13 @@
 #
 # Vector set tests.
 # A Redis instance should be running in the default port.
-# Copyright(C) 2024-2025 Salvatore Sanfilippo.
-# All Rights Reserved.
+#
+# Copyright (c) 2009-Present, Redis Ltd.
+# All rights reserved.
+#
+# Licensed under your choice of the Redis Source Available License 2.0
+# (RSALv2) or the Server Side Public License v1 (SSPLv1).
+#

 #!/usr/bin/env python3
 import redis
--- a/modules/vector-sets/vset.c
+++ b/modules/vector-sets/vset.c
@ -1,7 +1,11 @@
 /* Redis implementation for vector sets. The data structure itself
 * is implemented in hnsw.c.
 *
- * Copyright(C) 2024-Present, Redis Ltd. All Rights Reserved.
+ * Copyright (c) 2009-Present, Redis Ltd.
+ * All rights reserved.
+ *
+ * Licensed under your choice of the Redis Source Available License 2.0
+ * (RSALv2) or the Server Side Public License v1 (SSPLv1).
 * Originally authored by: Salvatore Sanfilippo.
 *
 * ======================== Understand threading model =========================
@ -35,7 +39,7 @@
 *    the lock and immediately releases it, with the effect of waiting all the
 *    background threads still running from ending their execution.
 *
- *    Note that no ther thread can be spawned, since we only call
+ *    Note that no thread can be spawned, since we only call
 *    vectorSetWaitAllBackgroundClients() from the main Redis thread, that
 *    is also the only thread spawning other threads.
 *
@ -66,7 +70,7 @@
 *    time in vectorSetWaitAllBackgroundClients(). This prevents removal
 *    of objects that are about to be taken by threads.
 *
- *    Note that other competing soltuions could be used to fix the problem
+ *    Note that other competing solutions could be used to fix the problem
 *    but have their set of issues, however they are worth documenting here
 *    and evaluating in the future:
 *
@ -100,7 +104,7 @@
 #define _USE_MATH_DEFINES
 #define _POSIX_C_SOURCE 200809L

-#include "redismodule.h"
+#include "../../src/redismodule.h"
 #include <stdio.h>
 #include <stdlib.h>
 #include <ctype.h>
@ -178,7 +182,7 @@ static inline uint32_t bit_count(uint32_t n) {
 * Note that compared to other approaches (random gaussian weights), what
 * we have here is deterministic, it means that our replicas will have
 * the same set of weights. Also this approach seems to work much better
- * in pratice, and the distances between elements are better guaranteed.
+ * in practice, and the distances between elements are better guaranteed.
 *
 * Note that we still save the projection matrix in the RDB file, because
 * in the future we may change the weights generation, and we want everything
@ -315,7 +319,7 @@ int vectorSetInsert(struct vsetObject *o, float *vec, int8_t *qvec, float qrange
            RedisModule_DictReplace(o->dict,val,node);

            /* If attrib != NULL, the user wants that in case of an update we
-             * update the attribute as well (otherwise it reamins as it was).
+             * update the attribute as well (otherwise it remains as it was).
             * Note that the order of operations is conceinved so that it
             * works in case the old attrib and the new attrib pointer is the
             * same. */
@ -371,7 +375,7 @@ int vectorSetInsert(struct vsetObject *o, float *vec, int8_t *qvec, float qrange
 float *parseVector(RedisModuleString **argv, int argc, int start_idx,
                  size_t *dim, uint32_t *reduce_dim, int *consumed_args)
 {
-    int consumed = 0; // Argumnets consumed.
+    int consumed = 0; // Arguments consumed

    /* Check for REDUCE option first. */
    if (reduce_dim) *reduce_dim = 0;
@ -504,7 +508,7 @@ int VADD_CASReply(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {

        /* Also, if the element was already inserted, we just pretend
         * the other insert won. We don't even start a threaded VADD
-         * if this was an udpate, since the deletion of the element itself
+         * if this was an update, since the deletion of the element itself
         * in order to perform the update would invalidate the CAS state. */
        if (vset && RedisModule_DictGet(vset->dict,val,NULL) != NULL)
            vset = NULL;
@ -540,7 +544,7 @@ int VADD_CASReply(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
        }
        RedisModule_DictSet(vset->dict,val,newnode);
        val = NULL; // Don't free it later.
-        attrib = NULL; // Dont' free it later.
+        attrib = NULL; // Don't free it later.

        RedisModule_ReplicateVerbatim(ctx);
    }
@ -648,7 +652,7 @@ int VADD_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
        cas = 0; /* Do synchronous insert at creation, otherwise the
                  * key would be left empty until the threaded part
                  * does not return. It's also pointless to try try
-                  * doing threaded first elemetn insertion. */
+                  * doing threaded first element insertion. */
        vset = createVectorSetObject(reduce_dim ? reduce_dim : dim, quant_type, hnsw_create_M);
        if (vset == NULL) {
            // We can't fail for OOM in Redis, but the mutex initialization
@ -729,7 +733,7 @@ int VADD_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    }

    /* For existing keys don't do CAS updates. For how things work now, the
-     * CAS state would be invalidated by the detetion before adding back. */
+     * CAS state would be invalidated by the deletion before adding back. */
    if (cas && RedisModule_DictGet(vset->dict,val,NULL) != NULL)
        cas = 0;

@ -1072,7 +1076,7 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    if (filter_ef == 0) filter_ef = count * 100; // Max filter visited nodes.

    /* Disable threaded for MULTI/EXEC and Lua, or if explicitly
-     * requsted by the user via the NOTHREAD option. */
+     * requested by the user via the NOTHREAD option. */
    if (no_thread || (RedisModule_GetContextFlags(ctx) &
                      (REDISMODULE_CTX_FLAGS_LUA|
                       REDISMODULE_CTX_FLAGS_MULTI)))
@ -1910,6 +1914,9 @@ int ONLOAD(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    if (RedisModule_Init(ctx,"vectorset",1,REDISMODULE_APIVER_1)
        == REDISMODULE_ERR) return REDISMODULE_ERR;

+    /* TODO: Added to pass CI, need to make changes in order to support these options */
+    RedisModule_SetModuleOptions(ctx, REDISMODULE_OPTIONS_HANDLE_IO_ERRORS|REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD);
+
    RedisModuleTypeMethods tm = {
        .version = REDISMODULE_TYPE_METHOD_VERSION,
        .rdb_load = VectorSetRdbLoad,
@ -1972,3 +1979,7 @@ int ONLOAD(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {

    return REDISMODULE_OK;
 }
+
+int VectorSets_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    return RedisModule_OnLoad(ctx, argv, argc);
+}
--- a/modules/vector-sets/w2v.c
+++ b/modules/vector-sets/w2v.c
@ -2,7 +2,11 @@
 * HNSW (Hierarchical Navigable Small World) Implementation
 * Based on the paper by Yu. A. Malkov, D. A. Yashunin
 *
- * Copyright(C) 2024-Present, Redis Ltd. All Rights Reserved.
+ * Copyright (c) 2009-Present, Redis Ltd.
+ * All rights reserved.
+ *
+ * Licensed under your choice of the Redis Source Available License 2.0
+ * (RSALv2) or the Server Side Public License v1 (SSPLv1).
 * Originally authored by: Salvatore Sanfilippo
 */

--- a/redis-full.conf
+++ b/redis-full.conf
@ -0,0 +1,376 @@
+include redis.conf
+
+loadmodule ./modules/redisbloom/redisbloom.so
+loadmodule ./modules/redisearch/redisearch.so
+loadmodule ./modules/redisjson/rejson.so
+loadmodule ./modules/redistimeseries/redistimeseries.so
+
+############################## QUERY ENGINE CONFIG ############################
+
+# Keep numeric ranges in numeric tree parent nodes of leafs for `x` generations.
+# numeric, valid range: [0, 2], default: 0
+#
+# search-_numeric-ranges-parents 0
+
+# The number of iterations to run while performing background indexing
+# before we call usleep(1) (sleep for 1 micro-second) and make sure that we
+# allow redis to process other commands.
+# numeric, valid range: [1, UINT32_MAX], default: 100
+#
+# search-bg-index-sleep-gap 100
+
+# The default dialect used in search queries.
+# numeric, valid range: [1, 4], default: 1
+#
+# search-default-dialect 1
+
+# the fork gc will only start to clean when the number of not cleaned document
+# will exceed this threshold.
+# numeric, valid range: [1, LLONG_MAX], default: 100
+#
+# search-fork-gc-clean-threshold 100
+
+# interval (in seconds) in which to retry running the forkgc after failure.
+# numeric, valid range: [1, LLONG_MAX], default: 5
+#
+# search-fork-gc-retry-interval 5
+
+# interval (in seconds) in which to run the fork gc (relevant only when fork
+# gc is used).
+# numeric, valid range: [1, LLONG_MAX], default: 30
+#
+# search-fork-gc-run-interval 30
+
+# the amount of seconds for the fork GC to sleep before exiting.
+# numeric, valid range: [0, LLONG_MAX], default: 0
+#
+# search-fork-gc-sleep-before-exit 0
+
+# Scan this many documents at a time during every GC iteration.
+# numeric, valid range: [1, LLONG_MAX], default: 100
+#
+# search-gc-scan-size 100
+
+# Max number of cursors for a given index that can be opened inside of a shard.
+# numeric, valid range: [0, LLONG_MAX], default: 128
+#
+# search-index-cursor-limit 128
+
+# Maximum number of results from ft.aggregate command.
+# numeric, valid range: [0, (1ULL << 31)], default: 1ULL << 31
+#
+# search-max-aggregate-results 2147483648
+
+# Maximum prefix expansions to be used in a query.
+# numeric, valid range: [1, LLONG_MAX], default: 200
+#
+# search-max-prefix-expansions 200
+
+# Maximum runtime document table size (for this process).
+# numeric, valid range: [1, 100000000], default: 1000000
+#
+# search-max-doctablesize 1000000
+
+# max idle time allowed to be set for cursor, setting it high might cause
+# high memory consumption.
+# numeric, valid range: [1, LLONG_MAX], default: 300000
+#
+# search-cursor-max-idle 300000
+
+# Maximum number of results from ft.search command.
+# numeric, valid range: [0, 1ULL << 31], default: 1000000
+#
+# search-max-search-results 1000000
+
+# Number of worker threads to use for background tasks when the server is
+# in an operation event.
+# numeric, valid range: [1, 16], default: 4
+#
+# search-min-operation-workers 4
+
+# Minimum length of term to be considered for phonetic matching.
+# numeric, valid range: [1, LLONG_MAX], default: 3
+#
+# search-min-phonetic-term-len 3
+
+# the minimum prefix for expansions (`*`).
+# numeric, valid range: [1, LLONG_MAX], default: 2
+#
+# search-min-prefix 2
+
+# the minimum word length to stem.
+# numeric, valid range: [2, UINT32_MAX], default: 4
+#
+# search-min-stem-len 4
+
+# Delta used to increase positional offsets between array
+# slots for multi text values.
+# Can control the level of separation between phrases in different
+# array slots (related to the SLOP parameter of ft.search command)"
+# numeric, valid range: [1, UINT32_MAX], default: 100
+#
+# search-multi-text-slop 100
+
+# Used for setting the buffer limit threshold for vector similarity tiered
+# HNSW index, so that if we are using WORKERS for indexing, and the
+# number of vectors waiting in the buffer to be indexed exceeds this limit,
+# we insert new vectors directly into HNSW.
+# numeric, valid range: [0, LLONG_MAX], default: 1024
+#
+# search-tiered-hnsw-buffer-limit 1024
+
+# Query timeout.
+# numeric, valid range: [1, LLONG_MAX], default: 500
+#
+# search-timeout 500
+
+# minimum number of iterators in a union from which the iterator will
+# will switch to heap-based implementation.
+# numeric, valid range: [1, LLONG_MAX], default: 20
+# switch to heap based implementation.
+#
+# search-union-iterator-heap 20
+
+# The maximum memory resize for vector similarity indexes (in bytes).
+# numeric, valid range: [0, UINT32_MAX], default: 0
+#
+# search-vss-max-resize 0
+
+# Number of worker threads to use for query processing and background tasks.
+# numeric, valid range: [0, 16], default: 0
+# This configuration also affects the number of connections per shard.
+#
+# search-workers 0
+
+# The number of high priority tasks to be executed at any given time by the
+# worker thread pool, before executing low priority tasks. After this number
+# of high priority tasks are being executed, the worker thread pool will
+# execute high and low priority tasks alternately.
+# numeric, valid range: [0, LLONG_MAX], default: 1
+#
+# search-workers-priority-bias-threshold 1
+
+# Load extension scoring/expansion module. Immutable.
+# string, default: ""
+#
+# search-ext-load ""
+
+# Path to Chinese dictionary configuration file (for Chinese tokenization). Immutable.
+# string, default: ""
+#
+# search-friso-ini ""
+
+# Action to perform when search timeout is exceeded (choose RETURN or FAIL).
+# enum, valid values: ["return", "fail"], default: "fail"
+#
+# search-on-timeout fail
+
+# Determine whether some index resources are free on a second thread.
+# bool, default: yes
+#
+# search-_free-resource-on-thread yes
+
+# Enable legacy compression of double to float.
+# bool, default: no
+#
+# search-_numeric-compress no
+
+# Disable print of time for ft.profile. For testing only.
+# bool, default: yes
+#
+# search-_print-profile-clock yes
+
+# Intersection iterator orders the children iterators by their relative estimated
+# number of results in ascending order, so that if we see first iterators with
+# a lower count of results we will skip a larger number of results, which
+# translates into faster iteration. If this flag is set, we use this
+# optimization in a way where union iterators are being factorize by the number
+# of their own children, so that we sort by the number of children times the
+# overall estimated number of results instead.
+# bool, default: no
+#
+# search-_prioritize-intersect-union-children no
+
+# Set to run without memory pools.
+# bool, default: no
+#
+# search-no-mem-pools no
+
+# Disable garbage collection (for this process).
+# bool, default: no
+#
+# search-no-gc no
+
+# Enable commands filter which optimize indexing on partial hash updates.
+# bool, default: no
+#
+# search-partial-indexed-docs no
+
+# Disable compression for DocID inverted index. Boost CPU performance.
+# bool, default: no
+#
+# search-raw-docid-encoding no
+
+# Number of search threads in the coordinator thread pool.
+# numeric, valid range: [1, LLONG_MAX], default: 20
+#
+# search-threads 20
+
+# Timeout for topology validation (in milliseconds). After this timeout,
+# any pending requests will be processed, even if the topology is not fully connected.
+# numeric, valid range: [0, LLONG_MAX], default: 30000
+#
+# search-topology-validation-timeout 30000
+
+
+############################## TIME SERIES CONFIG #############################
+
+# The maximal number of per-shard threads for cross-key queries when using cluster mode
+# (TS.MRANGE, TS.MREVRANGE, TS.MGET, and TS.QUERYINDEX).
+# Note: increasing this value may either increase or decrease the performance.
+# integer, valid range: [1..16], default: 3
+# This is a load-time configuration parameter.
+#
+# ts-num-threads 3
+
+
+# Default compaction rules for newly created key with TS.ADD, TS.INCRBY, and TS.DECRBY.
+# Has no effect on keys created with TS.CREATE.
+# This default value is applied to each new time series upon its creation.
+# string, see documentation for rules format, default: no compaction rules
+#
+# ts-compaction-policy ""
+
+# Default chunk encoding for automatically-created compacted time series.
+# This default value is applied to each new compacted time series automatically
+# created when ts-compaction-policy is specified.
+# valid values: COMPRESSED, UNCOMPRESSED, default: COMPRESSED
+#
+# ts-encoding COMPRESSED
+
+
+# Default retention period, in milliseconds. 0 means no expiration.
+# This default value is applied to each new time series upon its creation.
+# If ts-compaction-policy is specified - it is overridden for created
+# compactions as specified in ts-compaction-policy.
+# integer, valid range: [0 .. LLONG_MAX], default: 0
+#
+# ts-retention-policy 0
+
+# Default policy for handling insertion (TS.ADD and TS.MADD) of multiple
+# samples with identical timestamps.
+# This default value is applied to each new time series upon its creation.
+# string, valid values: BLOCK, FIRST, LAST, MIN, MAX, SUM, default: BLOCK
+#
+# ts-duplicate-policy BLOCK
+
+# Default initial allocation size, in bytes, for the data part of each new chunk
+# This default value is applied to each new time series upon its creation.
+# integer, valid range: [48 .. 1048576]; must be a multiple of 8, default: 4096
+#
+# ts-chunk-size-bytes 4096
+
+# Default values for newly created time series.
+# Many sensors report data periodically. Often, the difference between the measured
+# value and the previous measured value is negligible and related to random noise
+# or to measurement accuracy limitations. In such situations it may be preferable
+# not to add the new measurement to the time series.
+# A new sample is considered a duplicate and is ignored if the following conditions are met:
+# - The time series is not a compaction;
+# - The time series' DUPLICATE_POLICY IS LAST;
+# - The sample is added in-order (timestamp >= max_timestamp);
+# - The difference of the current timestamp from the previous timestamp
+#   (timestamp - max_timestamp) is less than or equal to ts-ignore-max-time-diff
+# - The absolute value difference of the current value from the value at the previous maximum timestamp
+#   (abs(value - value_at_max_timestamp) is less than or equal to ts-ignore-max-val-diff.
+# where max_timestamp is the timestamp of the sample with the largest timestamp in the time series,
+# and value_at_max_timestamp is the value at max_timestamp.
+# ts-ignore-max-time-diff: integer, valid range: [0 .. LLONG_MAX], default: 0
+# ts-ignore-max-val-diff: double, Valid range: [0 .. DBL_MAX], default: 0
+#
+# ts-ignore-max-time-diff 0
+# ts-ignore-max-val-diff 0
+
+
+########################### BLOOM FILTERS CONFIG ##############################
+
+# Defaults values for new Bloom filters created with BF.ADD, BF.MADD, BF.INSERT, and BF.RESERVE
+# These defaults are applied to each new Bloom filter upon its creation.
+
+# Error ratio
+# The desired probability for false positives.
+# For a false positive rate of 0.1% (1 in 1000) - the value should be 0.001.
+# double, Valid range: (0 .. 1), value greater than 0.25 is treated as 0.25, default: 0.01
+#
+# bf-error-rate 0.01
+
+# Initial capacity
+# The number of entries intended to be added to the filter.
+# integer, valid range: [1 .. 1GB], default: 100
+#
+# bf-initial-size 100
+
+# Expansion factor
+# When capacity is reached, an additional sub-filter is created.
+# The size of the new sub-filter is the size of the last sub-filter multiplied
+# by expansion.
+# integer, [0 .. 32768]. 0 is equivalent to NONSCALING. default: 2
+#
+# bf-expansion-factor 2
+
+
+########################### CUCKOO FILTERS CONFIG #############################
+
+# Defaults values for new Cuckoo filters created with
+# CF.ADD, CF.ADDNX, CF.INSERT, CF.INSERTNX, and CF.RESERVE
+# These defaults are applied to each new Cuckoo filter upon its creation.
+
+# Initial capacity
+# A filter will likely not fill up to 100% of its capacity.
+# Make sure to reserve extra capacity if you want to avoid expansions.
+# value is rounded to the next 2^n integer.
+# integer, valid range: [2*cf-bucket-size .. 1GB], default: 1024
+#
+# cf-initial-size 1024
+
+# Number of items in each bucket
+# The minimal false positive rate is 2/255 ~ 0.78% when bucket size of 1 is used.
+# Larger buckets increase the error rate linearly, but improve the fill rate.
+# integer, valid range: [1 .. 255], default: 2
+#
+# cf-bucket-size 2
+
+# Maximum iterations
+# Number of attempts to swap items between buckets before declaring filter
+# as full and creating an additional filter.
+# A lower value improves performance. A higher value improves fill rate.
+# integer, Valid range: [1 .. 65535], default: 20
+#
+# cf-max-iterations 20
+
+# Expansion factor
+# When a new filter is created, its size is the size of the current filter
+# multiplied by this factor.
+# integer, Valid range: [0 .. 32768], 0 is equivalent to NONSCALING, default: 1
+#
+# cf-expansion-factor 1
+
+# Maximum expansions
+# integer, Valid range: [1 .. 65536], default: 32
+#
+# cf-max-expansions 32
+
+
+################################## SECURITY ###################################
+#
+# The following is a list of command categories and their meanings:
+#
+# * search - Query engine related.
+# * json - Data type: JSON related.
+# * timeseries -  Data type: time series related.
+# * bloom - Data type:  Bloom filter related.
+# * cuckoo - Data type: cuckoo filter related.
+# * topk -  Data type: top-k related.
+# * cms - Data type: count-min sketch related.
+# * tdigest -  Data type: t-digest related.
+
--- a/src/Makefile
+++ b/src/Makefile
@ -52,6 +52,7 @@ endif
 WARN=-Wall -W -Wno-missing-field-initializers -Werror=deprecated-declarations -Wstrict-prototypes
 OPT=$(OPTIMIZATION)

+SKIP_VEC_SETS?=no
 # Detect if the compiler supports C11 _Atomic.
 # NUMBER_SIGN_CHAR is a workaround to support both GNU Make 4.3 and older versions.
 NUMBER_SIGN_CHAR := \#
@ -61,6 +62,7 @@ C11_ATOMIC := $(shell sh -c 'echo "$(NUMBER_SIGN_CHAR)include <stdatomic.h>" > f
 ifeq ($(C11_ATOMIC),yes)
 	STD+=-std=gnu11
 else
+	SKIP_VEC_SETS=yes
 	STD+=-std=c99
 endif

@ -352,6 +354,13 @@ else
 	GEN_COMMANDS_FLAGS=
 endif

+
+ifneq ($(SKIP_VEC_SETS),yes)
+	vpath %.c ../modules/vector-sets
+	REDIS_VEC_SETS_OBJ=hnsw.o cJSON.o vset.o
+	CFLAGS+=-DINCLUDE_VEC_SETS=1
+endif
+
 REDIS_SERVER_NAME=redis-server$(PROG_SUFFIX)
 REDIS_SENTINEL_NAME=redis-sentinel$(PROG_SUFFIX)
 REDIS_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o ebuckets.o eventnotifier.o iothread.o mstr.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
@ -361,7 +370,7 @@ REDIS_BENCHMARK_NAME=redis-benchmark$(PROG_SUFFIX)
 REDIS_BENCHMARK_OBJ=ae.o anet.o redis-benchmark.o adlist.o dict.o zmalloc.o redisassert.o release.o crcspeed.o crccombine.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o
 REDIS_CHECK_RDB_NAME=redis-check-rdb$(PROG_SUFFIX)
 REDIS_CHECK_AOF_NAME=redis-check-aof$(PROG_SUFFIX)
-ALL_SOURCES=$(sort $(patsubst %.o,%.c,$(REDIS_SERVER_OBJ) $(REDIS_CLI_OBJ) $(REDIS_BENCHMARK_OBJ)))
+ALL_SOURCES=$(sort $(patsubst %.o,%.c,$(REDIS_SERVER_OBJ) $(REDIS_VEC_SETS_OBJ) $(REDIS_CLI_OBJ) $(REDIS_BENCHMARK_OBJ)))

 all: $(REDIS_SERVER_NAME) $(REDIS_SENTINEL_NAME) $(REDIS_CLI_NAME) $(REDIS_BENCHMARK_NAME) $(REDIS_CHECK_RDB_NAME) $(REDIS_CHECK_AOF_NAME) $(TLS_MODULE)
 	@echo ""
@ -408,7 +417,7 @@ ifneq ($(strip $(PREV_FINAL_LDFLAGS)), $(strip $(FINAL_LDFLAGS)))
 endif

 # redis-server
-$(REDIS_SERVER_NAME): $(REDIS_SERVER_OBJ)
+$(REDIS_SERVER_NAME): $(REDIS_SERVER_OBJ) $(REDIS_VEC_SETS_OBJ)
 	$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a ../deps/fast_float/libfast_float.a $(FINAL_LIBS)

 # redis-sentinel
@ -435,7 +444,7 @@ $(REDIS_CLI_NAME): $(REDIS_CLI_OBJ)
 $(REDIS_BENCHMARK_NAME): $(REDIS_BENCHMARK_OBJ)
 	$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/hdr_histogram/libhdrhistogram.a $(FINAL_LIBS) $(TLS_CLIENT_LIBS)

-DEP = $(REDIS_SERVER_OBJ:%.o=%.d) $(REDIS_CLI_OBJ:%.o=%.d) $(REDIS_BENCHMARK_OBJ:%.o=%.d)
+DEP = $(REDIS_SERVER_OBJ:%.o=%.d) $(REDIS_VEC_SETS_OBJ:%.o=%.d) $(REDIS_CLI_OBJ:%.o=%.d) $(REDIS_BENCHMARK_OBJ:%.o=%.d)
 -include $(DEP)

 # Because the jemalloc.h header is generated as a part of the jemalloc build,
@ -502,7 +511,7 @@ bench: $(REDIS_BENCHMARK_NAME)
 	@echo ""
 	@echo "WARNING: if it fails under Linux you probably need to install libc6-dev-i386"
 	@echo ""
-	$(MAKE) CFLAGS="-m32" LDFLAGS="-m32"
+	$(MAKE) CFLAGS="-m32" LDFLAGS="-m32" SKIP_VEC_SETS="yes"

 gcov:
 	$(MAKE) REDIS_CFLAGS="-fprofile-arcs -ftest-coverage -DCOVERAGE_TEST" REDIS_LDFLAGS="-fprofile-arcs -ftest-coverage"
--- a/src/config.c
+++ b/src/config.c
@ -1578,6 +1578,9 @@ void rewriteConfigLoadmoduleOption(struct rewriteConfigState *state) {
    dictEntry *de;
    while ((de = dictNext(di)) != NULL) {
        struct RedisModule *module = dictGetVal(de);
+        /* Internal modules doesn't have path and are not part of the configuration file */
+        if (sdslen(module->loadmod->path) == 0) continue;
+
        line = sdsnew("loadmodule ");
        line = sdscatsds(line, module->loadmod->path);
        for (int i = 0; i < module->loadmod->argc; i++) {
--- a/src/module.c
+++ b/src/module.c
@ -12249,6 +12249,15 @@ void moduleRemoveCateogires(RedisModule *module) {
    }
 }

+int VectorSets_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc);
+/* Load internal data types that bundled as modules */
+void moduleLoadInternalModules(void) {
+#ifdef INCLUDE_VEC_SETS
+    int retval = moduleOnLoad((int (*)(void *, void **, int)) VectorSets_OnLoad, NULL, NULL, NULL, 0, 0);
+    serverAssert(retval == C_OK);
+#endif
+}
+
 /* Load all the modules in the server.loadmodule_queue list, which is
 * populated by `loadmodule` directives in the configuration file.
 * We can't load modules directly when processing the configuration file
@ -12448,7 +12457,7 @@ void moduleUnregisterCleanup(RedisModule *module) {
    moduleUnregisterAuthCBs(module);
 }

-/* Load a module and initialize it. On success C_OK is returned, otherwise
+/* Load a module by path and initialize it. On success C_OK is returned, otherwise
 * C_ERR is returned. */
 int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loadex) {
    int (*onload)(void *, void **, int);
@ -12476,6 +12485,13 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
            "symbol. Module not loaded.",path);
        return C_ERR;
    }
+
+    return moduleOnLoad(onload, path, handle, module_argv, module_argc, is_loadex);
+}
+
+/* Load a module by its 'onload' callback and initialize it. On success C_OK is returned, otherwise
+ * C_ERR is returned. */
+int moduleOnLoad(int (*onload)(void *, void **, int), const char *path, void *handle, void **module_argv, int module_argc, int is_loadex) {
    RedisModuleCtx ctx;
    moduleCreateContext(&ctx, NULL, REDISMODULE_CTX_TEMP_CLIENT); /* We pass NULL since we don't have a module yet. */
    if (onload((void*)&ctx,module_argv,module_argc) == REDISMODULE_ERR) {
@ -12487,7 +12503,7 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
            moduleFreeModuleStructure(ctx.module);
        }
        moduleFreeContext(&ctx);
-        dlclose(handle);
+        if (handle) dlclose(handle);
        return C_ERR;
    }

@ -12504,12 +12520,12 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
        incrRefCount(ctx.module->loadmod->argv[i]);
    }

-    /* If module commands have ACL categories, recompute command bits 
+    /* If module commands have ACL categories, recompute command bits
     * for all existing users once the modules has been registered. */
    if (ctx.module->num_commands_with_acl_categories) {
        ACLRecomputeCommandBitsFromCommandRulesAllUsers();
    }
-    serverLog(LL_NOTICE,"Module '%s' loaded from %s",ctx.module->name,path);
+    if (path) serverLog(LL_NOTICE,"Module '%s' loaded from %s",ctx.module->name,path);
    ctx.module->onload = 0;

    int post_load_err = 0;
@ -12550,6 +12566,9 @@ int moduleUnload(sds name, const char **errmsg, int forced_unload) {
    if (module == NULL) {
        *errmsg = "no such module with that name";
        return C_ERR;
+    } else if (sdslen(module->loadmod->path) == 0) {
+        *errmsg = "the module can't be unloaded";
+        return C_ERR;
    } else if (listLength(module->types) && !forced_unload) {
        *errmsg = "the module exports one or more module-side data "
                  "types, can't unload";
--- a/src/server.c
+++ b/src/server.c
@ -7490,6 +7490,7 @@ int main(int argc, char **argv) {
    }
    if (!server.sentinel_mode) {
        moduleInitModulesSystemLast();
+        moduleLoadInternalModules();
        moduleLoadFromQueue();
    }
    ACLLoadUsersAtStartup();
--- a/src/server.h
+++ b/src/server.h
@ -2662,8 +2662,10 @@ void populateCommandLegacyRangeSpec(struct redisCommand *c);
 void moduleInitModulesSystem(void);
 void moduleInitModulesSystemLast(void);
 void modulesCron(void);
+int moduleOnLoad(int (*onload)(void *, void **, int), const char *path, void *handle, void **module_argv, int module_argc, int is_loadex);
 int moduleLoad(const char *path, void **argv, int argc, int is_loadex);
 int moduleUnload(sds name, const char **errmsg, int forced_unload);
+void moduleLoadInternalModules(void);
 void moduleLoadFromQueue(void);
 int moduleGetCommandKeysViaAPI(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result);
 int moduleGetCommandChannelsViaAPI(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result);
--- a/utils/req-res-log-validator.py
+++ b/utils/req-res-log-validator.py
@ -58,6 +58,18 @@ IGNORED_COMMANDS = {
    # Commands to which we decided not write a reply schema
    "pfdebug",
    "lolwut",
+    # TODO: for vector-sets module
+    "VADD",
+    "VCARD",
+    "VDIM",
+    "VEMB",
+    "VGETATTR",
+    "VINFO",
+    "VLINKS",
+    "VRANDMEMBER",
+    "VREM",
+    "VSETATTR",
+    "VSIM",
 }

 class Request(object):