263 Commits

Author	SHA1	Message	Date
Yuan Wang	33391a7b61	Support delay trimming slots after finishing migrating slots (#14567) ... This PR introduces a mechanism that allows a module to temporarily disable trimming after an ASM migration operation so it can safely finish ongoing asynchronous jobs that depend on keys in migrating (and about to be trimmed) slots. 1. **ClusterDisableTrim/ClusterEnableTrim** We introduce `ClusterDisableTrim/ClusterEnableTrim` Module APIs to allow module to disable/enable slot migration ``` /* Disable automatic slot trimming. */ int RM_ClusterDisableTrim(RedisModuleCtx *ctx) /* Enable automatic slot trimming */ int RM_ClusterEnableTrim(RedisModuleCtx *ctx) ``` **Please notice**: Redis will not start any subsequent import or migrate ASM operations while slot trimming is disabled, so modules must re-enable trimming immediately after completing their pending work. The only valid and meaningful time for a module to disable trimming appears to be after the MIGRATE_COMPLETED event. 2. **REDISMODULE_OPEN_KEY_ACCESS_TRIMMED** Added REDISMODULE_OPEN_KEY_ACCESS_TRIMMED to RM_OpenKey() so that module can operate with these keys in the unowned slots after trim is paused. And now we don't delete the key if it is in trim job when we access it. And `expireIfNeeded` returns `KEY_VALID` if `EXPIRE_ALLOW_ACCESS_TRIMMED` is set, otherwise, returns `KEY_TRIMMED` without deleting key. 3. **REDISMODULE_CTX_FLAGS_TRIM_IN_PROGRESS** We also extend RM_GetContextFlags() to include a flag REDISMODULE_CTX_FLAGS_TRIM_IN_PROGRESS indicating whether a trimming job is pending (due to trim pause) or in progress. Modules could periodically poll this flag to synchronize their internal state, e.g., if a trim job was delayed or if the module incorrectly assumed trimming was still active. Bugfix: RM_SetClusterFlags could not clear a flag after enabling it first. --------- Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>	2025-12-16 16:30:56 +08:00
Mincho Paskalev	837b14c89a	Fix ASan Daily (#14527) ... After https://github.com/redis/redis/pull/14226 module tests started running with ASan enabled. `auth.c` blocks the user on auth and spawns a thread that sleeps for 0.5s before unblocking the client and returning. A tcl tests unloads the module which may happen just after the spawned thread unblocks the client. In that case if the unloading finishes fast enough the spawned thread may try to execute code from the module's dynamic library that is already unloaded resulting in sefault. Fix: just wait on the thread during module's OnUnload method.	2025-11-19 10:56:18 +02:00
Mincho Paskalev	91b5808fd6	Fix numeric config boundry check (#14286) ... Revert a breaking change introduced in #14051 described in this comment https://github.com/redis/redis/pull/14051#discussion_r2281765769 The non-negative check inside `checkNumericBoundaries` was ignoring that passing a big unsigned long long value (> 2^63-1) will be passed as negative value and will never reach the lower/upper boundary check. The check is removed, reverting the breaking change. This allows for RedisModule_ConfigSetNumeric to pass big unsigned number, albeit via `long long` parameter. Added comments about this behaviour. Added tests for https://github.com/redis/redis/pull/14051#discussion_r2281765769	2025-10-28 13:40:44 +02:00
GuyAv46	340615255b	Add API RedisModule_ClusterKeySlotC (#14442) ... Enhancing #13069, by adding a more direct API for getting the slot of a string Introduces new module API: ```c /* Like `RM_ClusterKeySlot`, but gets a char pointer and a length. * Returns the cluster slot of a key, similar to the `CLUSTER KEYSLOT` command. * This function works even if cluster mode is not enabled. */ unsigned int RM_ClusterKeySlotC(const char *keystr, size_t keylen) ```	2025-10-22 18:28:30 +03:00
Ozan Tezcan	2bc4e0299d	Add Atomic Slot Migration (ASM) support (#14414) ... ## <a name="overview"></a> Overview This PR is a joint effort with @ShooterIT . I’m just opening it on behalf of both of us. This PR introduces Atomic Slot Migration (ASM) for Redis Cluster — a new mechanism for safely and efficiently migrating hash slots between nodes. Redis Cluster distributes data across nodes using 16384 hash slots, each owned by a specific node. Sometimes slots need to be moved — for example, to rebalance after adding or removing nodes, or to mitigate a hot shard that’s overloaded. Before ASM, slot migration was non-atomic and client-dependent, relying on CLUSTER SETSLOT, GETKEYSINSLOT, MIGRATE commands, and client-side handling of ASK/ASKING replies. This process was complex, error-prone, slow and could leave clusters in inconsistent states after failures. Clients had to implement redirect logic, multi-key commands could fail mid-migration, and errors often resulted in orphaned keys or required manual cleanup. Several related discussions can be found in the issue list, some examples: https://github.com/redis/redis/issues/14300 , https://github.com/redis/redis/issues/4937 , https://github.com/redis/redis/issues/10370 , https://github.com/redis/redis/issues/4333 , https://github.com/redis/redis/issues/13122, https://github.com/redis/redis/issues/11312 Atomic Slot Migration (ASM) makes slot rebalancing safe, transparent, and reliable, addressing many of the limitations of the legacy migration method. Instead of moving keys one by one, ASM replicates the entire slot’s data plus live updates to the target node, then performs a single atomic handoff. Clients keep working without handling ASK/ASKING replies, multi-key operations remain consistent, failures don’t leave partial states, and replicas stay in sync. The migration process also completes significantly faster. Operators gain new commands (CLUSTER MIGRATION IMPORT, STATUS, CANCEL) for monitoring and control, while modules can hook into migration events for deeper integration. ### The problems of legacy method in detail Operators and developers ran into multiple issues with the legacy method, some of these issues in detail: 1. **Redirects and Client Complexity:** While a slot was being migrated, some keys were already moved while others were not. Clients had to handle `-ASK` and `-ASKING` responses, reissuing requests to the target node. Not all client libraries implemented this correctly, leading to failed commands or subtle bugs. Even when implemented, it increased latency and broke naive pipelines. 2. **Multi-Key Operations Became Unreliable:** Commands like `MGET key1 key2` could fail with `TRYAGAIN` if part of the slot was already migrated. This made application logic unpredictable during resharding. 3. **Risk of failure:** Keys were moved one-by-one (with MIGRATE command). If the source crashed, or the destination ran out of memory, the system could be left in an inconsistent state: some keys moved, others lost, slots partially migrated. Manual intervention was often needed, sometimes resulting in data loss. 4. **Replica and Failover Issues:** Replicas weren’t aware of migrations in progress. If a failover occurred mid-migration, manual intervention was required to clean up or resume the process safely. 5. **Operational Overhead:** Operators had to coordinate multiple commands (CLUSTER SETSLOT, MIGRATE, GETKEYSINSLOT, etc.) with little visibility into progress or errors, making rebalancing slow and error-prone. 6. **Poor performance:** Key-by-key migration was inherently slow and inefficient for large slot ranges. 7. **Large keys:** Large keys could fail to migrate or cause latency spikes on the destination node. ### How Atomic Slot Migration Fixes This Atomic Slot Migration (ASM) eliminates all of these issues by: 1. **Clients:** Clients no longer need to handle ASK/ASKING; the migration is fully transparent. 2. **Atomic ownership transfer:** The entire slot’s data (snapshot + live updates) is replicated and handed off in a single atomic step. 3. **Performance**: ASM completes migrations significantly faster by streaming slot data in parallel (snapshot + incremental updates) and eliminating key-by-key operations. 4. **Consistency guarantees:** Multi-key operations and pipelines continue to work reliably throughout migration. 5. **Resilience:** Failures no longer leave orphaned keys or partial states; migration tasks can be retried or safely cancelled. 6. **Replica awareness:** Replicas remain consistent during migration, and failovers will no longer leave partially imported keys. 7. **Operator visibility:** New CLUSTER MIGRATION subcommands (IMPORT, STATUS, CANCEL) provide clear observability and management for operators. ### ASM Diagram and Migration Steps ``` ┌─────────────┐ ┌────────────┐ ┌───────────┐ ┌───────────┐ ┌───────┐ │ │ │Destination │ │Destination│ │ Source │ │Source │ │ Operator │ │ master │ │ replica │ │ master │ │ Fork │ │ │ │ │ │ │ │ │ │ │ └──────┬──────┘ └─────┬──────┘ └─────┬─────┘ └─────┬─────┘ └───┬───┘ │ │ │ │ │ │ │ │ │ │ │CLUSTER MIGRATION IMPORT │ │ │ │ │ <start-slot> <end-slot>..│ │ │ │ ├───────────────────────────►│ │ │ │ │ │ │ │ │ │ Reply with <task-id> │ │ │ │ │◄───────────────────────────┤ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ CLUSTER SYNCSLOTS│SYNC │ │ │ CLUSTER MIGRATION STATUS │ <task-id> <start-slot> <end-slot>.│ │ Monitor │ ID <task-id> ├────────────────────────────────────►│ │ task ┌─►├───────────────────────────►│ │ │ │ state │ │ │ │ │ │ till │ │ Reply status │ Negotiation with multiple channels │ │ completed └─ │◄───────────────────────────┤ (i.e rdbchannel repl) │ │ │ │◄───────────────────────────────────►│ │ │ │ │ │ Fork │ │ │ │ ├──────────►│ ─┐ │ │ │ │ │ │ Slot snapshot as RESTORE commands │ │ │ │◄────────────────────────────────────────────────┤ │ │ Propagate │ │ │ │ ┌─────────────┐ ├─────────────────►│ │ │ │ │ │ │ │ │ │ │ Snapshot │ Client │ │ │ │ │ │ delivery │ │ │ Replication stream for slot range │ │ │ duration └──────┬──────┘ │◄────────────────────────────────────┤ │ │ │ │ Propagate │ │ │ │ │ ├─────────────────►│ │ │ │ │ │ │ │ │ │ │ SET key value1 │ │ │ │ │ ├─────────────────────────────────────────────────────────────────►│ │ │ │ +OK │ │ │ │ ─┘ │◄─────────────────────────────────────────────────────────────────┤ │ │ │ │ │ │ │ │ Drain repl stream │ ──┐ │ │ │◄────────────────────────────────────┤ │ │ │ SET key value2 │ │ │ │ │ ├─────────────────────────────────────────────────────────────────►│ │Write │ │ │ │ │ │pause │ │ │ │ │ │ │ │ │ Publish new config via cluster bus │ │ │ │ +MOVED ├────────────────────────────────────►│ ──┘ │ │◄─────────────────────────────────────────────────────────────────┤ ──┐ │ │ │ │ │ │ │ │ │ │ │ │Trim │ │ │ │ │ ──┘ │ │ SET key value2 │ │ │ │ ├───────────────────────────►│ │ │ │ │ +OK │ │ │ │ │◄───────────────────────────┤ │ │ │ │ │ │ │ │ │ │ │ │ │ ``` ### New commands introduced There are two new commands: 1. A command to start, monitor and cancel the migration operation: `CLUSTER MIGRATION <arg>` 2. An internal command to manage slot transfer between source and destination: `CLUSTER SYNCSLOTS <arg>` For more details, please refer to the [New Commands](#new-commands) section. Internal command messaging is mostly omitted in the diagram for simplicity. ### Steps 1. Slot migration begins when the operator sends `CLUSTER MIGRATION IMPORT <start-slot> <end-slot> ...` to the destination master. The process is initiated from the destination node, similar to REPLICAOF. This approach allows us to reuse the same logic and share code with the new replication mechanism (see https://github.com/redis/redis/pull/13732). The command can include multiple slot ranges. The destination node creates one migration task per source node, regardless of how many slot ranges are specified. Upon successfully creating the task, the destination node replies IMPORT command with the assigned task ID. The operator can then monitor progress using `CLUSTER MIGRATION STATUS ID <task-id>` . When the task’s state field changes to `completed`, the migration has finished successfully. Please see [New Commands](#new-commands) section for the output sample. 2. After creating the migration task, the destination node will request replication of slots by using the internal command `CLUSTER SYNCSLOTS`. 3. Once the source node accepts the request, the destination node establishes another separate connection(similar to rdbchannel replication) so snapshot data and incremental changes can be transmitted in parallel. 4. Source node forks, starts delivering snapshot content (as per-key RESTORE commands) from one connection and incremental changes from the other connection. The destination master starts applying commands from the snapshot connection and accumulates incremental changes. Applied commands are also propagated to the destination replicas via replication backlog. Note: Only commands of related slots are delivered to the destination node. This is done by writing them to the migration client’s output buffer, which serves as the replication stream for the migration operation. 5. Once the source node finishes delivering the snapshot and determines that the destination node has caught up (remaining repl stream to consume went under a configured limit), it pauses write traffic for the entire server. After pausing the writes, the source node forwards any remaining write commands to the destination node. 6. Once the destination consumes all the writes, it bumps up cluster config epoch and changes the configuration. New config is published via cluster bus. 7. When the source node receives the new configuration, it can redirect clients and it begins trimming the migrated slots, while also resuming write traffic on the server. ### Internal slots synchronization state machine  1. The destination node performs authentication using the cluster secret introduced in #13763 , and transmits its node ID information. 2. The destination node sends `CLUSTER SYNCSLOTS SYNC <task-id> <start-slot> <end-slot>` to initiate a slot synchronization request and establish the main channel. The source node responds with `+RDBCHANNELSYNCSLOTS`, indicating that the destination node should establish an RDB channel. 3. The destination node then sends `CLUSTER SYNCSLOTS RDBCHANNEL <task-id>` to establish the RDB channel, using the same task-id as in the previous step to associate the two connections as part of the same ASM task. The source node replies with `+SLOTSSNAPSHOT`, and `fork` a child process to transfer slot snapshot. 4. The destination node applies the slot snapshot data received over the RDB channel, while proxying the command stream to replicas. At the same time, the main channel continues to read and buffer incremental commands in memory. 5. Once the source node finishes sending the slot snapshot, it notifies the destination node using the `CLUSTER SYNCSLOTS SNAPSHOT-EOF` command. The destination node then starts streaming the buffered commands while continuing to read and buffer incremental commands sent from the source. 6. The destination node periodically sends `CLUSTER SYNCSLOTS ACK <offset>` to inform the source of the applied data offset. When the offset gap meets the threshold, the source node pauses write operations. After all buffered data has been drained, it sends `CLUSTER SYNCSLOTS STREAM-EOF` to the destination node to hand off slots. 7. Finally, the destination node takes over slot ownership, updates the slot configuration and bumps the epoch, then broadcasts the updates via cluster bus. Once the source node detects the updated slot configuration, the slot migration process is complete. ### Error handling - If the connection between the source and destination is lost (due to disconnection, output buffer overflow, OOM, or timeout), the destination node automatically restarts the migration from the beginning. The destination node will retry the operation until it is explicitly cancelled using the CLUSTER MIGRATION CANCEL <task-id> command. - If a replica connection drops during migration, it can later resume with PSYNC, since the imported slot data is also written to the replication backlog. - During the write pause phase, the source node sets a timeout. If the destination node fails to drain remaining replication data and update the config during that time, the source node assumes the destination has failed and automatically resumes normal writes for the migrating slots. - On any error, the destination node triggers a trim operation to discard any partially imported slot data. - If node crashes during importing, unowned keys are deleted on start up. ### <a name="slot-snapshot-format-considerations"></a> Slot Snapshot Format Considerations When the source node forks to deliver slot content, in theory, there are several possible formats for transmitting the snapshot data: **Mini RDB**:A compact RDB file containing only the keys from the migrating slots. This format is efficient for transmission, but it cannot be easily forwarded to destination-side replicas. **AOF format**: The source node can generate commands in AOF form (e.g., SET x y, HSET h f v) and stream them. Individual commands are easily appended to the replication stream and propagated to replicas. Large keys can also be split into multiple commands (incrementally reconstructing the value), similar to the AOF rewrite process. **RESTORE commands**: Each key is serialized and sent as a `RESTORE` command. These can be appended directly to the destination’s replication stream, though very large keys may make serialization and transmission less efficient. We chose the `RESTORE` command as default approach for the following reasons: - It can be easily propagated to replicas. - It is more efficient than AOF for most cases, and some module keys do not support the AOF format. - For large **non-module** keys that are not string, ASM automatically switches to the AOF-based key encoding as an optimization when the key’s cardinality exceeds 512. This approach allows the key to be transferred in chunks rather than as a single large payload, reducing memory pressure and improving migration efficiency. In future versions, the RESTORE command may be enhanced to handle large keys more efficiently. Some details: - For RESTORE commands, normally by default Redis compresses keys. We disable compression while delivering RESTORE commands as compression comes with a performance hit. Without compression, replication is several times faster. - For string keys, we still prefer AOF format, e.g. SET commands as it is currently more efficient than RESTORE, especially for big keys. ### <a name="trimming-the-keys"></a> Trimming the keys When a migration completes successfully, the source node deletes the migrated keys from its local database. Since the migrated slots may contain a large number of keys, this trimming process must be efficient and non-blocking. In cluster mode, Redis maintains per-slot data structures for keys, expires, and subexpires. This organization makes it possible to efficiently detach all data associated with a given slot in a single step. During trimming, these slot-specific data structures are handed off to a background I/O (BIO) thread for asynchronous cleanup—similar to how FLUSHALL or FLUSHDB operate. This mechanism is referred to as background trimming, and it is the preferred and default method for ASM, ensuring that the main thread remains unblocked. However, unlike Redis itself, some modules may not maintain per-slot data structures and therefore cannot drop related slots data in a single operation. To support these cases, Redis introduces active trimming, where key deletion occurs in the main thread instead. This is not a blocking operation, trimming runs concurrently in the main thread, periodically removing keys during the cron loop. Each deletion triggers a keyspace notification so that modules can react to individual key removals. While active trim is less efficient, it ensures backward compatibility for modules during the transition period. Before starting the trim, Redis checks whether any module is subscribed to newly added `REDISMODULE_NOTIFY_KEY_TRIMMED` keyspace event. If such subscribers exist, active trimming is used; otherwise, background trimming is triggered. Going forward, modules are expected to adopt background trimming to take advantage of its performance and scalability benefits, and active trimming will be phased out once modules migrate to the new model. Redis also prefers active trimming if there is any client that is using client tracking feature (see [client-side caching](https://redis.io/docs/latest/develop/reference/client-side-caching/)). In the current client tracking protocol, when a database is flushed (e.g., via the FLUSHDB command), a null value is sent to tracking clients to indicate that they should invalidate all locally cached keys. However, there is currently no mechanism to signal that only specific slots have been flushed. Iterating over all keys in the slots to be trimmed would be a blocking operation. To avoid this, if there is any client that is using client tracking feature, Redis automatically switches to active trimming mode. In the future, the client tracking protocol can be extended to support slot-based invalidation, allowing background trimming to be used in these cases as well. Finally, trimming may also be triggered after a migration failure. In such cases, the operation ensures that any partially imported or inconsistent slot data is cleaned up, maintaining cluster consistency and preventing stale keys from remaining in the source or destination nodes. Note about active trim: Subsequent migrations can complete while a prior trim is still running. In that case, the new migration’s trim job is queued and will start automatically after the current trim finishes. This does not affect slot ownership or client traffic—it only serializes the background cleanup. ### <a name="replica-handling"></a> Replica handling - During importing, new keys are propagated to destination side replica. Replica will check slot ownership before replying commands like SCAN, KEYS, DBSIZE not to include these unowned keys in the reply. Also, when an import operation begins, the master now propagates an internal command through the replication stream, allowing replicas to recognize that an ASM operation is in progress. This is done by the internal `CLUSTER SYNCSLOTS CONF ASM-TASK` command in the replication stream. This enables replicas to trigger the relevant module events so that modules can adapt their behavior — for example, filtering out unowned keys from read-only requests during ASM operations. To be able to support full sync with RDB delivery scenarios, a new AUX field is also added to the RDB: `cluster-asm-task`. It's value is a string in the format of `task_id:source_node:dest_node:operation:state:slot_ranges`. - After a successful migration or on a failed import, master will trim the keys. In that case, master will propagate a new command to the replica: `TRIMSLOTS RANGES <numranges> <start-slot> <end-slot> ... ` . So, the replica will start trimming once this command is received. ### <a name="propagating-data-outside-the-keyspace"></a> Propagating data outside the keyspace When the destination node is newly added to the cluster, certain data outside the keyspace may need to be propagated first. A common example is functions. Previously, redis-cli handled this by transferring functions when a new node was added. With ASM, Redis now automatically dumps and sends functions to the destination node using `FUNCTION RESTORE ..REPLACE` command — done purely for convenience to simplify setup. Additionally, modules may also need to propagate their own data outside the keyspace. To support this, a new API has been introduced: `RM_ClusterPropagateForSlotMigration()`. See the [Module Support](#module-support) section for implementation details. ### Limitations 1. Single migration at a time: Only one ASM migration operation is allowed at a time. This limitation simplifies the current design but can be extended in the future. 2. Large key handling: For large keys, ASM switches to AOF encoding to deliver key data in chunks. This mechanism currently applies only to non-module keys. In the future, the RESTORE command may be extended to support chunked delivery, providing a unified solution for all key types. See [Slot Snapshot Format Considerations](#slot-snapshot-format-considerations) for details. 3. There are several cases that may cause an Atomic Slot Migration (ASM) to be aborted (can be retried afterwards): - FLUSHALL / FLUSHDB: These commands introduce complexity during ASM. For example, if executed on the migrating node, they must be propagated only for the migrating slots. However, when combined with active trimming, their execution may need to be deferred until it is safe to proceed, adding further complexity to the process. - FAILOVER: The replica cannot resume the migration process. Migration should start from the beginning. - Module propagates cross-slot command during ASM via RM_Replicate(): If this occurs on the migrating node, Redis cannot split the command to propagate only the relevant slots to the ASM destination. To keep the logic simple and consistent, ASM is cancelled in this case. Modules should avoid propagating cross-slot commands during migration. - CLIENT PAUSE: The import task cannot progress during a write pause, as doing so would violate the guarantee that no writes occur during migration. To keep things simple, the ASM task is aborted when CLIENT PAUSE is active. - Manual Slot Configuration Changes: If slot configuration is modified manually during ASM (for example, when legacy migration methods are mixed with ASM), the process is aborted. Note: This situation is highly unexpected — users should not combine ASM with legacy migration methods. 4. When active trimming is enabled, a node must not re-import the same slots while trimming for those slots is still in progress. Otherwise, it can’t distinguish newly imported keys from pre-existing ones, and the trim cron might delete the incoming keys by mistake. In this state, the node rejects IMPORT operation for those slots until trimming completes. If the master has finished trimming but a replica is still trimming, master may still start the import operation for those slots. So, the replica checks whether the master is sending commands for those slots; if so, it blocks the master’s client connection until trimming finishes. This is a corner case, but we believe the behavior is reasonable for now. In the worst case, the master may drop the replica (e.g., buffer overrun), triggering a new full sync. # API Changes ## <a name="new-commands"></a> New Commands ### Public commands 1. **Syntax:** `CLUSTER MIGRATION IMPORT <start-slot> <end-slot> [<start-slot> <end-slot>]...` **Args:** Slot ranges **Reply:** - String task ID - -ERR <message> on failure (e.g. invalid slot range) **Description:** Executes on the destination master. Accepts multiple slot ranges and triggers atomic migration for the specified ranges. Returns a task ID that can be used to monitor the status of the task. In CLUSTER MIGRATION STATUS output, “state” field will be `completed` on a successful operation. 2. **Syntax:** `CLUSTER MIGRATION CANCEL [ID <id> | ALL]` **Args:** Task ID or ALL **Reply:** Number of cancelled tasks **Description:** Cancels an ongoing migration task by its ID or cancels all tasks if ALL is specified. Note: Cancelling a task on the source node does not stop the migration on the destination node, which will continue retrying until it is also cancelled there. 3. **Syntax:** `CLUSTER MIGRATION STATUS [ID <id> | ALL]` **Args:** Task ID or ALL - **ID:** If provided, returns the status of the specified migration task. - **ALL:** Lists the status of all migration tasks. **Reply:** - A list of migration task details (both ongoing and completed ones). - Empty list if the given task ID does not exist. **Description:** Displays the status of all current and completed atomic slot migration tasks. If a specific task ID is provided, it returns detailed information for that task only. **Sample output:** ``` 127.0.0.1:5001> cluster migration status all 1) 1) "id" 2) "24cf41718b20f7f05901743dffc40bc9b15db339" 3) "slots" 4) "0-1000" 5) "source" 6) "1098d90d9ba2d1f12965442daf501ef0b6667bec" 7) "dest" 8) "b3b5b426e7ea6166d1548b2a26e1d5adeb1213ac" 9) "operation" 10) "migrate" 11) "state" 12) "completed" 13) "last_error" 14) "" 15) "retries" 16) "0" 17) "create_time" 18) "1759694528449" 19) "start_time" 20) "1759694528449" 21) "end_time" 22) "1759694528464" 23) "write_pause_ms" 24) "10" ``` ### Internal commands 1. **Syntax:** `CLUSTER SYNCSLOTS <arg> ...` **Args:** Internal messaging operations **Reply:** +OK or -ERR <message> on failure (e.g. invalid slot range) **Description:** Used for internal communication between source and destination nodes. e.g. handshaking, establishing multiple channels, triggering handoff. 2. **Syntax:** `TRIMSLOTS RANGES <numranges> <start-slot> <end-slot> ...` **Args:** Slot ranges to trim **Reply:** +OK **Description:** Master propagates it to replica so that replica can trim unowned keys after a successful migration or on a failed import. ## New configs - `cluster-slot-migration-max-archived-tasks`: To list in `CLUSTER MIGRATION STATUS ALL` output, Redis keeps last n migration tasks in memory. This config controls maximum number of archived ASM tasks. Default value: 32, used as a hidden config - `cluster-slot-migration-handoff-max-lag-bytes`: After the slot snapshot is completed, if the remaining replication stream size falls below this threshold, the source node pauses writes to hand off slot ownership. A higher value may trigger the handoff earlier but can lead to a longer write pause, since more data remains to be replicated. A lower value can result in a shorter write pause, but it may be harder to reach the threshold if there is a steady flow of incoming writes. Default value: 1MB - `cluster-slot-migration-write-pause-timeout`: The maximum duration (in milliseconds) that the source node pauses writes during ASM handoff. After pausing writes, if the destination node fails to take over the slots within this timeout (for example, due to a cluster configuration update failure), the source node assumes the migration has failed and resumes writes to prevent indefinite blocking. Default value: 10 seconds - `cluster-slot-migration-sync-buffer-drain-timeout`: Timeout in milliseconds for sync buffer to be drained during ASM. After the destination applies the accumulated buffer, the source continues sending commands for migrating slots. The destination keeps applying them, but if the gap remains above the acceptable limit (see `slot-migration-handoff-max-lag-bytes`), which may cause endless synchronization. A timeout check is required to handle this case. The timeout is calculated as **the maximum of two values**: - A configurable timeout (slot-migration-sync-buffer-drain-timeout) to avoid false positives. - A dynamic timeout based on the time that the destination took to apply the slot snapshot and the accumulated buffer during slot snapshot delivery. The destination should be able to drain the remaining sync buffer in less time than this. We multiply it by 2 to be more conservative. Default value: 60000 millliseconds, used as a hidden config ## New flag in CLIENT LIST - the client responsible for importing slots is marked with the `o` flag. - the client responsible for migrating slots is marked with the `g` flag. ## New INFO fields - `mem_cluster_slot_migration_output_buffer`: Memory usage of the migration client’s output buffer. Redis writes incoming changes to this buffer during the migration process. - `mem_cluster_slot_migration_input_buffer`: Memory usage of the accumulated replication stream buffer on the importing node. - `mem_cluster_slot_migration_input_buffer_peak`: Peak accumulated repl buffer size on the importing side ## New CLUSTER INFO fields - `cluster_slot_migration_active_tasks`: Number of in-progress ASM tasks. Currently, it will be 1 or 0. - `cluster_slot_migration_active_trim_running`: Number of active trim jobs in progress and scheduled - `cluster_slot_migration_active_trim_current_job_keys`: Number of keys scheduled for deletion in the current trim job. - `cluster_slot_migration_active_trim_current_job_trimmed`: Number of keys already deleted in the current trim job. - `cluster_slot_migration_stats_active_trim_started`: Total number of trim jobs that have started since the process began. - `cluster_slot_migration_stats_active_trim_completed`: Total number of trim jobs completed since the process began. - `cluster_slot_migration_stats_active_trim_cancelled`: Total number of trim jobs cancelled since the process began. ## Changes in RDB format A new aux field is added to RDB: `cluster-asm-task`. When an import operation begins, the master now propagates an internal command through the replication stream, allowing replicas to recognize that an ASM operation is in progress. This enables replicas to trigger the relevant module events so that modules can adapt their behavior — for example, filtering out unowned keys from read-only requests during ASM operations. To be able to support RDB delivery scenarios, a new field is added to the RDB. See [replica handling](#replica-handling) ## Bug fix - Fix memory leak when processing forgetting node type message - Fix data race of writing reply to replica client directly when enabling multi-threading We don't plan to back point them into old versions, since they are very rare cases. ## Keys visibility When performing atomic slot migration, during key importing on the destination node or key trimming on the source/destination, these keys will be filtered out in the following commands: - KEYS - SCAN - RANDOMKEY - CLUSTER GETKEYSINSLOT - DBSIZE - CLUSTER COUNTKEYSINSLOT The only command that will reflect the increasing number of keys is: - INFO KEYSPACE ## <a name="module-support"></a> Module Support **NOTE:** Please read [trimming](#trimming-the-keys) section and see how does ASM decide about trimming method when there are modules in use. ### New notification: ```c #define REDISMODULE_NOTIFY_KEY_TRIMMED (1<<17) ``` When a key is deleted by the active trim operation, this notification will be sent to subscribed modules. Also, ASM will automatically choose the trimming method depending on whether there are any subscribers to this new event. Please see the further details here: [trimming](#trimming-the-keys) ### New struct in the API: ```c typedef struct RedisModuleSlotRange { uint16_t start; uint16_t end; } RedisModuleSlotRange; typedef struct RedisModuleSlotRangeArray { int32_t num_ranges; RedisModuleSlotRange ranges[]; } RedisModuleSlotRangeArray; ``` ### New Events #### 1. REDISMODULE_EVENT_CLUSTER_SLOT_MIGRATION (RedisModuleEvent_ClusterSlotMigration) These events notify modules about different stages of Active Slot Migration (ASM) operations such as when import or migration starts, fails, or completes. Modules can use these notifications to track cluster slot movements or perform custom logic during ASM transitions. ```c #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_IMPORT_STARTED 0 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_IMPORT_FAILED 1 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_IMPORT_COMPLETED 2 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_STARTED 3 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_FAILED 4 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_COMPLETED 5 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_MODULE_PROPAGATE 6 ``` Parameter to these events: ```c typedef struct RedisModuleClusterSlotMigrationInfo { uint64_t version; /* Not used since this structure is never passed from the module to the core right now. Here for future compatibility. */ char source_node_id[REDISMODULE_NODE_ID_LEN + 1]; char destination_node_id[REDISMODULE_NODE_ID_LEN + 1]; const char *task_id; RedisModuleSlotRangeArray* slots; } RedisModuleClusterSlotMigrationInfoV1; #define RedisModuleClusterSlotMigrationInfo RedisModuleClusterSlotMigrationInfoV1 ``` #### 2. REDISMODULE_EVENT_CLUSTER_SLOT_MIGRATION_TRIM (RedisModuleEvent_ClusterSlotMigrationTrim) These events inform modules about the lifecycle of ASM key trimming operations. Modules can use them to detect when trimming starts, completes, or is performed asynchronously in the background. ```c #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_TRIM_STARTED 0 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_TRIM_COMPLETED 1 #define REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_TRIM_BACKGROUND 2 ``` Parameter to these events: ```c typedef struct RedisModuleClusterSlotMigrationTrimInfo { uint64_t version; /* Not used since this structure is never passed from the module to the core right now. Here for future compatibility. */ RedisModuleSlotRangeArray* slots; } RedisModuleClusterSlotMigrationTrimInfoV1; #define RedisModuleClusterSlotMigrationTrimInfo RedisModuleClusterSlotMigrationTrimInfoV1 ``` ### New functions ```c /* Returns 1 if keys in the specified slot can be accessed by this node, 0 otherwise. * * This function returns 1 in the following cases: * - The slot is owned by this node or by its master if this node is a replica * - The slot is being imported under the old slot migration approach (CLUSTER SETSLOT <slot> IMPORTING ..) * - Not in cluster mode (all slots are accessible) * * Returns 0 for: * - Invalid slot numbers (< 0 or >= 16384) * - Slots owned by other nodes */ int RM_ClusterCanAccessKeysInSlot(int slot); /* Propagate commands along with slot migration. * * This function allows modules to add commands that will be sent to the * destination node before the actual slot migration begins. It should only be * called during the REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_MODULE_PROPAGATE event. * * This function can be called multiple times within the same event to * replicate multiple commands. All commands will be sent before the * actual slot data migration begins. * * Note: This function is only available in the fork child process just before * slot snapshot delivery begins. * * On success REDISMODULE_OK is returned, otherwise * REDISMODULE_ERR is returned and errno is set to the following values: * * * EINVAL: function arguments or format specifiers are invalid. * * EBADF: not called in the correct context, e.g. not called in the REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_MODULE_PROPAGATE event. * * ENOENT: command does not exist. * * ENOTSUP: command is cross-slot. * * ERANGE: command contains keys that are not within the migrating slot range. */ int RM_ClusterPropagateForSlotMigration(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...); /* Returns the locally owned slot ranges for the node. * * An optional `ctx` can be provided to enable auto-memory management. * If cluster mode is disabled, the array will include all slots (0–16383). * If the node is a replica, the slot ranges of its master are returned. * * The returned array must be freed with RM_ClusterFreeSlotRanges(). */ RedisModuleSlotRangeArray *RM_ClusterGetLocalSlotRanges(RedisModuleCtx *ctx); /* Frees a slot range array returned by RM_ClusterGetLocalSlotRanges(). * Pass the `ctx` pointer only if the array was created with a context. */ void RM_ClusterFreeSlotRanges(RedisModuleCtx *ctx, RedisModuleSlotRangeArray *slots); ``` ## ASM API for alternative cluster implementations Following https://github.com/redis/redis/pull/12742, Redis cluster code was restructured to support alternative cluster implementations. Redis uses cluster_legacy.c implementation by default. This PR adds a generic ASM API so alternative implementations can initiate and coordinate Atomic Slot Migration (ASM) while Redis executes the data movement and emits state changes. Documentation rests in `cluster.h`: ```c There are two new functions: /* Called by cluster implementation to request an ASM operation. (cluster impl --> redis) */ int clusterAsmProcess(const char *task_id, int event, void *arg, char **err); /* Called when an ASM event occurs to notify the cluster implementation. (redis --> cluster impl) */ int clusterAsmOnEvent(const char *task_id, int event, void *arg); ``` ```c /* API for alternative cluster implementations to start and coordinate * Atomic Slot Migration (ASM). * * These two functions drive ASM for alternative cluster implementations. * - clusterAsmProcess(...) impl -> redis: initiates/advances/cancels ASM operations * - clusterAsmOnEvent(...) redis -> impl: notifies state changes * * Generic steps for an alternative implementation: * - On destination side, implementation calls clusterAsmProcess(ASM_EVENT_IMPORT_START) * to start an import operation. * - Redis calls clusterAsmOnEvent() when an ASM event occurs. * - On the source side, Redis will call clusterAsmOnEvent(ASM_EVENT_HANDOFF_PREP) * when slots are ready to be handed off and the write pause is needed. * - Implementation stops the traffic to the slots and calls clusterAsmProcess(ASM_EVENT_HANDOFF) * - On the destination side, Redis calls clusterAsmOnEvent(ASM_EVENT_TAKEOVER) * when destination node is ready to take over the slot, waiting for ownership change. * - Cluster implementation updates the config and calls clusterAsmProcess(ASM_EVENT_DONE) * to notify Redis that the slots ownership has changed. * * Sequence diagram for import: * - Note: shows only the events that cluster implementation needs to react. * * ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ * │ Destination │ │ Destination │ │ Source │ │ Source │ * │ Cluster impl │ │ Master │ │ Master │ │ Cluster impl │ * └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ * │ │ │ │ * │ ASM_EVENT_IMPORT_START │ │ │ * ├─────────────────────────────►│ │ │ * │ │ CLUSTER SYNCSLOTS <arg> │ │ * │ ├────────────────────────►│ │ * │ │ │ │ * │ │ SNAPSHOT(restore cmds) │ │ * │ │◄────────────────────────┤ │ * │ │ Repl stream │ │ * │ │◄────────────────────────┤ │ * │ │ │ ASM_EVENT_HANDOFF_PREP │ * │ │ ├────────────────────────────►│ * │ │ │ ASM_EVENT_HANDOFF │ * │ │ │◄────────────────────────────┤ * │ │ Drain repl stream │ │ * │ │◄────────────────────────┤ │ * │ ASM_EVENT_TAKEOVER │ │ │ * │◄─────────────────────────────┤ │ │ * │ │ │ │ * │ ASM_EVENT_DONE │ │ │ * ├─────────────────────────────►│ │ ASM_EVENT_DONE │ * │ │ │◄────────────────────────────┤ * │ │ │ │ */ #define ASM_EVENT_IMPORT_START 1 /* Start a new import operation (destination side) */ #define ASM_EVENT_CANCEL 2 /* Cancel an ongoing import/migrate operation (source and destination side) */ #define ASM_EVENT_HANDOFF_PREP 3 /* Slot is ready to be handed off to the destination shard (source side) */ #define ASM_EVENT_HANDOFF 4 /* Notify that the slot can be handed off (source side) */ #define ASM_EVENT_TAKEOVER 5 /* Ready to take over the slot, waiting for config change (destination side) */ #define ASM_EVENT_DONE 6 /* Notify that import/migrate is completed, config is updated (source and destination side) */ #define ASM_EVENT_IMPORT_PREP 7 /* Import is about to start, the implementation may reject by returning C_ERR */ #define ASM_EVENT_IMPORT_STARTED 8 /* Import started */ #define ASM_EVENT_IMPORT_FAILED 9 /* Import failed */ #define ASM_EVENT_IMPORT_COMPLETED 10 /* Import completed (config updated) */ #define ASM_EVENT_MIGRATE_PREP 11 /* Migrate is about to start, the implementation may reject by returning C_ERR */ #define ASM_EVENT_MIGRATE_STARTED 12 /* Migrate started */ #define ASM_EVENT_MIGRATE_FAILED 13 /* Migrate failed */ #define ASM_EVENT_MIGRATE_COMPLETED 14 /* Migrate completed (config updated) */ ``` ------ Co-authored-by: Yuan Wang <yuan.wang@redis.com> --------- Co-authored-by: Yuan Wang <yuan.wang@redis.com>	2025-10-22 15:56:20 +03:00
guybe7	ca9ede6968	Expose "touches-arbitrary-keys" flag to Redis modules (#14290) ... This commit adds support for the "touches-arbitrary-keys" command flag in Redis modules, allowing module commands to be properly marked when they modify keys not explicitly provided as arguments, to avoid wrapping replicated commands with MULTI/EXEC. Changes: - Added "touches-arbitrary-keys" flag parsing in commandFlagsFromString() - Updated module command documentation to describe the new flag - Added test implementation in zset module with zset.delall command to demonstrate and verify the flag functionality The zset.delall command serves as a test case that scans the keyspace and deletes all zset-type keys, properly using the new flag since it modifies keys not provided via argv. This commit adds a new `zset.delall` command to the zset test module that iterates through the keyspace and deletes all keys of type "zset". Key changes: - Added zset_delall() function that uses RedisModule_Scan to iterate through all keys in the keyspace - Added zset_delall_callback() that checks each key's type and deletes zset keys using RedisModule_Call with "DEL" command - Registered the new command with "write touches-arbitrary-keys" flags since it modifies arbitrary keys not provided via argv - Added support for "touches-arbitrary-keys" flag in module command parsing - Added comprehensive tests for the new functionality The command returns the number of deleted zset keys and properly handles replication by using the "s!" format specifier with RedisModule_Call to ensure DEL commands are replicated to slaves and AOF. Usage: ZSET.DELALL Returns: Integer count of deleted zset keys	2025-08-20 07:32:24 +08:00
Stav-Levi	82396716d0	Add API to allow Redis modules to unsubscribe from keyspace notifications ... This API complements module subscribe by enabling modules to unsubscribe from specific keyspace event notifications when they are no longer needed. This helps reduce performance overhead and unnecessary callback invocations. The function matches subscriptions based on event mask, callback pointer, and module identity. If a matching subscription is found, it is removed. Returns REDISMODULE_OK if a subscription was successfully removed, otherwise REDISMODULE_ERR.	2025-07-28 10:17:48 +03:00
debing.sun	457089b1fe	Create global data before test instead of module load for module defrag test (#13951) ... After #13816, we added defragmentation support for moduleDict, which significantly increased global data size. As a result, the defragmentation tests for non-global data were affected. Now, we move the creation of global data to before the global data test to avoid it interfering with other tests. Fixed the simple key test failure due to forgetting to reset stats.	2025-07-22 20:44:12 +08:00
Mincho Paskalev	15706f2e82	Module set/get config API (#14051) ... # Problem Some redis modules need to call `CONFIG GET/SET` commands. Server may be ran with `rename-command CONFIG ""`(or something similar) which leads to the module being unable to access the config. # Solution Added new API functions for use by modules ``` RedisModuleConfigIterator* RedisModule_GetConfigIterator(RedisModuleCtx *ctx, const char *pattern); void RedisModule_ReleaseConfigIterator(RedisModuleCtx *ctx, RedisModuleConfigIterator *iter); const char *RedisModule_ConfigIteratorNext(RedisModuleConfigIterator *iter); int RedisModule_GetConfigType(const char *name, RedisModuleConfigType *res); int RedisModule_GetBoolConfig(RedisModuleCtx *ctx, const char *name, int *res); int RedisModule_GetConfig(RedisModuleCtx *ctx, const char *name, RedisModuleString **res); int RedisModule_GetEnumConfig(RedisModuleCtx *ctx, const char *name, RedisModuleString **res); int RedisModule_GetNumericConfig(RedisModuleCtx *ctx, const char *name, long long *res); int RedisModule_SetBoolConfig(RedisModuleCtx *ctx, const char *name, int value, RedisModuleString **err); int RedisModule_SetConfig(RedisModuleCtx *ctx, const char *name, RedisModuleString *value, RedisModuleString **err); int RedisModule_SetEnumConfig(RedisModuleCtx *ctx, const char *name, RedisModuleString *value, RedisModuleString **err); int RedisModule_SetNumericConfig(RedisModuleCtx *ctx, const char *name, long long value, RedisModuleString **err); ``` ## Implementation The work is mostly done inside `config.c` as I didn't want to expose the config dict outside of it. That means each of these module functions has a corresponding method in `config.c` that actually does the job. F.e `RedisModule_SetEnumConfig` calls `moduleSetEnumConfig` which is implemented in `config.c` ## Notes Also, refactored `configSetCommand` and `restoreBackupConfig` functions for the following reasons: - code and logic is now way more clear in `configSetCommand`. Only caveat here is removal of an optimization that skipped running apply functions that already have ran in favour of code clarity. - Both functions needlessly separated logic for module configs and normal configs whereas no such separation is needed. This also had the side effect of removing some allocations. - `restoreBackupConfig` now has clearer interface and can be reused with ease. One of the places I reused it is for the individual `moduleSet*Config` functions, each of which needs the restoration functionality but for a single config only. ## Future Additionally, a couple considerations were made for potentially extending the API in the future - if need be an API for atomically setting multiple config values can be added - `RedisModule_SetConfigsTranscationStart/End` or similar that can be put around `RedisModule_Set*Config` calls. - if performance is an issue an API `RedisModule_GetConfigIteratorNextWithTypehint` or similar may be added in order not to incur the additional cost of calling `RedisModule_GetConfigType`. --------- Co-authored-by: Oran Agra <oran@redislabs.com>	2025-07-03 13:46:33 +03:00
debing.sun	fa040a72c0	Add XDELEX and XACKDEL commands for stream (#14130) ... ## Summary and detailed design for new stream command ## XDELEX ### Syntax ``` XDELEX key [KEEPREF | DELREF | ACKED] IDS numids id [id ...] ``` ### Description The `XDELEX` command extends the Redis Streams `XDEL` command, offering enhanced control over message entry deletion with respect to consumer groups. It accepts optional `DELREF` or `ACKED` parameters to modify its behavior: - **KEEPREF:** Deletes the specified entries from the stream, but preserves existing references to these entries in all consumer groups' PEL. This behavior is similar to XDEL. - **DELREF:** Deletes the specified entries from the stream and also removes all references to these entries from all consumer groups' pending entry lists, effectively cleaning up all traces of the messages. - **ACKED:** Only trims entries that were read and acknowledged by all consumer groups. **Note:** The `IDS` block can appear at any position in the command, consistent with other commands. ### Reply Array reply, for each `id`: - `-1`: No such `id` exists in the provided stream `key`. - `1`: Entry was deleted from the stream. - `2`: Entry was not deleted, but there are still dangling references. (ACKED option) ## XACKDEL ### Syntax ``` XACKDEL key group [KEEPREF | DELREF | ACKED] IDS numids id [id ...] ``` ### Description The `XACKDEL` command combines `XACK` and `XDEL` functionalities in Redis Streams. It acknowledges specified message IDs in the given consumer group and attempts to delete corresponding stream entries. It accepts optional `DELREF` or `ACKED` parameters: - **KEEPREF:** Acknowledges the messages in the specified consumer group and deletes the entries from the stream, but preserves existing references to these entries in all consumer groups' PEL. - **DELREF:** Acknowledges the messages in the specified consumer group, deletes the entries from the stream, and also removes all references to these entries from all consumer groups' pending entry lists, effectively cleaning up all traces of the messages. - **ACKED:** Acknowledges the messages in the specified consumer group and only trims entries that were read and acknowledged by all consumer groups. ### Reply Array reply, for each `id`: - `-1`: No such `id` exists in the provided stream `key`. - `1`: Entry was acknowledged and deleted from the stream. - `2`: Entry was acknowledged but not deleted, but there are still dangling references. (ACKED option) # Redis Streams Commands Extension ## XTRIM ### Syntax ``` XTRIM key <MAXLEN | MINID> [= | ~] threshold [LIMIT count] [KEEPREF | DELREF | ACKED] ``` ### Description The `XTRIM` command trims a stream by removing entries based on specified criteria, extended to include optional `DELREF` or `ACKED` parameters for consumer group handling: - **KEEPREF:** Trims the stream according to the specified strategy (MAXLEN or MINID) regardless of whether entries are referenced by any consumer groups, but preserves existing references to these entries in all consumer groups' PEL. - **DELREF:** Trims the stream according to the specified strategy and also removes all references to the trimmed entries from all consumer groups' PEL. - **ACKED:** Only trims entries that were read and acknowledged by all consumer groups. ### Reply No change. ## XADD ### Syntax ``` XADD key [NOMKSTREAM] [<MAXLEN | MINID> [= | ~] threshold [LIMIT count]] [KEEPREF | DELREF | ACKED] <* | id> field value [field value ...] ``` ### Description The `XADD` command appends a new entry to a stream and optionally trims it in the same operation, extended to include optional `DELREF` or `ACKED` parameters for trimming behavior: - **KEEPREF:** When trimming, removes entries from the stream according to the specified strategy (MAXLEN or MINID), regardless of whether they are referenced by any consumer groups, but preserves existing references to these entries in all consumer groups' PEL. - **DELREF:** When trimming, removes entries from the stream according to the specified strategy and also removes all references to these entries from all consumer groups' PEL. - **ACKED:** When trimming, only removes entries that were read and acknowledged by all consumer groups. Note that if the number of referenced entries is bigger than MAXLEN, we will still stop. ### Reply No change. ## Key implementation Since we currently have no simple way to track the association between an entry and consumer groups without iterating over all groups, we introduce two mechanisms to establish this link. This allows us to determine whether an entry has been seen by all consumer groups, and to identify which groups are referencing it. With this links, we can break the association when the entry is either acknowledged or deleted. 1) Added reference tracking between stream messages and consumer groups using `cgroups_ref` The cgroups_ref is implemented as a rax that maps stream message IDs to lists of consumer groups that reference those messages, and streamNACK stores the corresponding nodes of this list, so that the corresponding groups can be deleted during `ACK`. In this way, we can determine whether an entry has been seen but not ack. 2) Store a cache minimum last_id in the stream structure. The reason for doing this is that there is a situation where an entry has never been seen by the consume group. In this case, we think this entry has not been consumed either. If there is an "ACKED" option, we cannot directly delete this entry either. When a consumer group updates its last_id, we don’t immediately update the cached minimum last_id. Instead, we check whether the group’s previous last_id was equal to the current minimum, or whether the new last_id is smaller than the current minimum (when using `XGROUP SETID`). If either is true, we mark the cached minimum last_id as invalid, and defer the actual update until the next time it’s needed. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: moticless <moticless@github.com> Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com> Co-authored-by: Slavomir Kaslev <slavomir.kaslev@gmail.com> Co-authored-by: Yuan Wang <yuan.wang@redis.com>	2025-07-01 21:00:42 +08:00
Hüseyin Açacak	645858d518	Add size_t cast for RM_call() in module tests (#14061) ... This PR addresses a potential misalignment issue when using `va_args`. Without this fix, [argument](9a9aa921bc/src/module.c (L6249-L6264)) values may occasionally become incorrect due to stack alignment inconsistencies.	2025-05-23 10:10:11 +08:00
Mincho Paskalev	fdbf88032c	Add MSan and integrate it with CI (#13916) ... ## Description Memory sanitizer (MSAN) is used to detect use-of-uninitialized memory issues. While Address Sanitizer catches a wide range of memory safety issues, it doesn't specifically detect uninitialized memory usage. Therefore, Memory Sanitizer complements Address Sanitizer. This PR adds MSAN run to the daily build, with the possibility of incorporating it into the ci.yml workflow in the future if needed. Changes in source files fix false-positive issues and they should not introduce any runtime implications. Note: Valgrind performs similar checks to both ASAN and MSAN but sanitizers run significantly faster. ## Limitations - Memory sanitizer is only supported by Clang. - MSAN documentation states that all dependencies, including the standard library, must be compiled with MSAN. However, it also mentions there are interceptors for common libc functions, so compiling the standard library with the MSAN flag is not strictly necessary. Therefore, we are not compiling libc with MSAN. --------- Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>	2025-05-09 11:44:54 +03:00
Pieter Cailliau	d65102861f	Adding AGPLv3 as a license option to Redis! (#13997) ... Read more about [the new license option](http://redis.io/blog/agplv3/) and [the Redis 8 release](http://redis.io/blog/redis-8-ga/).	2025-05-01 14:04:22 +01:00
Yuan Wang	951ec79654	Cluster compatibility check (#13846) ... ### Background The program runs normally in standalone mode, but migrating to cluster mode may cause errors, this is because some cross slot commands can not run in cluster mode. We should provide an approach to detect this issue when running in standalone mode, and need to expose a metric which indicates the usage of no incompatible commands. ### Solution To avoid perf impact, we introduce a new config `cluster-compatibility-sample-ratio` which define the sampling ratio (0-100) for checking command compatibility in cluster mode. When a command is executed, it is sampled at the specified ratio to determine if it complies with Redis cluster constraints, such as cross-slot restrictions. A new metric is exposed: `cluster_incompatible_ops` in `info stats` output. The following operations will be considered incompatible operations. - cross-slot command If a command has multiple cross slot keys, it is incompatible - `swap, copy, move, select` command These commands involve multi databases in some cases, we don't allow multiple DB in cluster mode, so there are not compatible - Module command with `no-cluster` flag If a module command has `no-cluster` flag, we will encounter an error when loading module, leading to fail to load module if cluster is enabled, so this is incompatible. - Script/function with `no-cluster` flag Similar with module command, if we declare `no-cluster` in shebang of script/function, we also can not run it in cluster mode - `sort` command by/get pattern When `sort` command has `by/get` pattern option, we must ask that the pattern slot is equal with the slot of keys, otherwise it is incompatible in cluster mode. - The script/function command accesses the keys and declared keys have different slots For the script/function command, we not only check the slot of declared keys, but only check the slot the accessing keys, if they are different, we think it is incompatible. **Besides**, commands like `keys, scan, flushall, script/function flush`, that in standalone mode iterate over all data to perform the operation, are only valid for the server that executes the command in cluster mode and are not broadcasted. However, this does not lead to errors, so we do not consider them as incompatible commands. ### Performance impact test **cross slot test** Below are the test commands and results. When using MSET with 8 keys, performance drops by approximately 3%. **single key test** It may be due to the overhead of the sampling function, and single-key commands could cause a 1-2% performance drop.	2025-03-20 10:35:53 +08:00
debing.sun	f364dcca2d	Make RM_DefragRedisModuleDict API support incremental defragmentation for dict leaf (#13840) ... After https://github.com/redis/redis/pull/13816, we make a new API to defrag RedisModuleDict. Currently, we only support incremental defragmentation of the dictionary itself, but the defragmentation of values is still not incremental. If the values are very large, it could lead to significant blocking. Therefore, in this PR, we have added incremental defragmentation for the values. The main change is to the `RedisModuleDefragDictValueCallback`, we modified the return value of this callback. When the callback returns 1, we will save the `seekTo` as the key of the current unfinished node, and the next time we enter, we will continue defragmenting this node. When the return value is 0, we will proceed to the next node. ## Test Since each dictionary in the global dict originally contained only 10 strings, but now it has been changed to a nested dictionary, each dictionary now has 10 sub-dictionaries, with each sub-dictionary containing 10 strings, this has led to a corresponding reduction in the defragmentation time obtained from other tests. Therefore, the other tests have been modified to always wait for defragmentation to be turned off before the test begins, then start it after creating fragmentation, ensuring that they can always run for a full defragmentation cycle. --------- Co-authored-by: ephraimfeldblum <ephraim.feldblum@redis.com>	2025-03-04 17:19:41 +08:00
debing.sun	7939ba031d	Enable the callback to be NULL for RM_DefragRedisModuleDict() and reduce the system calls of RM_DefragShouldStop() (#13830) ... 1) Enable the callback to be NULL for RM_DefragRedisModuleDict() Because the dictionary may store only the key without the value. 2) Reduce the system calls of RM_DefragShouldStop() The API checks the following thresholds before performing a time check: over 512 defrag hits, or over 1024 defrag misses, and performs the time judgment if any of these thresholds are reached. 3) Added defragmentation statistics for dictionary items to cover the associated code for RM_DefragRedisModuleDict(). 4) Removed `module_ctx` from `defragModuleCtx` struct, which can be replaced by a temporary variable. --------- Co-authored-by: oranagra <oran@redislabs.com>	2025-02-26 20:04:29 +08:00
debing.sun	ee933d9e2b	Fixed passing incorrect endtime value for module context (#13822) ... 1) Fix a bug that passing an incorrect endtime to module. This bug was found by @ShooterIT. After #13814, all endtime will be monotonic time, and we should no longer convert it to ustime relative. Add assertions to prevent endtime from being much larger thatn the current time. 2) Fix a race in test `Reduce defrag CPU usage when module data can't be defragged` --------- Co-authored-by: ShooterIT <wangyuancode@163.com>	2025-02-23 12:58:48 +08:00
debing.sun	032357ec0f	Add RM_DefragRedisModuleDict module API (#13816) ... After #13815, we introduced incremental defragmentation for global data for module. Now we added a new module API `RM_DefragRedisModuleDict` to incremental defrag `RedisModuleDict`. This PR adds a new APIs and a new defrag callback: ```c RedisModuleDict *RM_DefragRedisModuleDict(RedisModuleDefragCtx *ctx, RedisModuleDict *dict, RedisModuleDefragDictValueCallback valueCB, RedisModuleString **seekTo); typedef void *(*RedisModuleDefragDictValueCallback)(RedisModuleDefragCtx *ctx, void *data, unsigned char *key, size_t keylen); ``` Usage: ```c RedisModuleString *seekTo = NULL; RedisModuleDict *dict = = RedisModule_CreateDict(ctx); ... populate the dict code ... /* Defragment a dictionary completely */ do { RedisModuleDict *new = RedisModule_DefragRedisModuleDict(ctx, dict, defragGlobalDictValueCB, &seekTo); if (new != NULL) { dict = new; } } while (seekTo); ``` --------- Co-authored-by: ShooterIT <wangyuancode@163.com> Co-authored-by: oranagra <oran@redislabs.com>	2025-02-20 21:09:29 +08:00
debing.sun	695126ccce	Add support for incremental defragmentation of global module data (#13815) ... ## Description Currently, when performing defragmentation on non-key data within the module, we cannot process the defragmentation incrementally. This limitation affects the efficiency and flexibility of defragmentation in certain scenarios. The primary goal of this PR is to introduce support for incremental defragmentation of global module data. ## Interface Change New module API `RegisterDefragFunc2` This is a more advanced version of `RM_RegisterDefragFunc`, in that it takes a new callbacks(`RegisterDefragFunc2`) that has a return value, and can use RM_DefragShouldStop in and indicate that it should be called again later, or is it done (returned 0). ## Note The `RegisterDefragFunc` API remains available. --------- Co-authored-by: ShooterIT <wangyuancode@163.com> Co-authored-by: oranagra <oran@redislabs.com>	2025-02-20 00:28:16 +08:00
kei-nan	d9134f8f95	Update tests/modules/moduleconfigs.c ... missing else clause Co-authored-by: debing.sun <debing.sun@redis.com>	2025-02-06 13:16:33 +02:00
jonathan keinan	b9361ad5fe	* only use new api if override-default was provided as an argument	2025-02-06 13:16:33 +02:00
jonathan keinan	f7353db7eb	* fix test ... * cleanup strval2 on if an error during the OnLoad was encountered.	2025-02-06 13:16:33 +02:00
jonathan keinan	294492dbf2	* fix tests ... * add some logging to test module	2025-02-06 13:16:33 +02:00
jonathan keinan	fd5c325886	* initial commit	2025-02-06 13:16:33 +02:00
Raz Monsonego	04589f90d7	Add internal connection and command mechanism (#13740) ... # PR: Add Mechanism for Internal Commands and Connections in Redis This PR introduces a mechanism to handle **internal commands and connections** in Redis. It includes enhancements for command registration, internal authentication, and observability. ## Key Features 1. **Internal Command Flag**: - Introduced a new **module command registration flag**: `internal`. - Commands marked with `internal` can only be executed by **internal connections**, AOF loading flows, and master-replica connections. - For any other connection, these commands will appear as non-existent. 2. **Support for internal authentication added to `AUTH`**: - Used by depicting the special username `internal connection` with the right internal password, i.e.,: `AUTH "internal connection" <internal_secret>`. - No user-defined ACL username can have this name, since spaces are not aloud in the ACL parser. - Allows connections to authenticate as **internal connections**. - Authenticated internal connections can execute internal commands successfully. 4. **Module API for Internal Secret**: - Added the `RedisModule_GetInternalSecret()` API, that exposes the internal secret that should be used as the password for the new `AUTH "internal connection" <password>` command. - This API enables the modules to authenticate against other shards as local connections. ## Notes on Behavior - **ACL validation**: - Commands dispatched by internal connections bypass ACL validation, to give the caller full access regardless of the user with which it is connected. - **Command Visibility**: - Internal commands **do not appear** in `COMMAND <subcommand>` and `MONITOR` for non-internal connections. - Internal commands **are logged** in the slow log, latency report and commands' statistics to maintain observability. - **`RM_Call()` Updates**: - **Non-internal connections**: - Cannot execute internal commands when the command is sent with the `C` flag (otherwise can). - Internal connections bypass ACL validations (i.e., run as the unrestricted user). - **Internal commands' success**: - Internal commands succeed upon being sent from either an internal connection (i.e., authenticated via the new `AUTH "internal connection" <internal_secret>` API), an AOF loading process, or from a master via the replication link. Any other connections that attempt to execute an internal command fail with the `unknown command` error message raised. - **`CLIENT LIST` flags**: - Added the `I` flag, to indicate that the connection is internal. - **Lua Scripts**: - Prevented internal commands from being executed via Lua scripts. --------- Co-authored-by: Meir Shpilraien <meir@redis.com>	2025-02-05 11:48:08 +02:00
Raz Monsonego	c688537d49	Add flag for ability of a module context to execute debug commands (#13774) ... This PR adds a flag to the `RM_GetContextFlags` module-API function that depicts whether the context may execute debug commands, according to redis's standards.	2025-02-03 09:52:41 +02:00
Moti Cohen	0dd057222b	Modules API: new HashFieldMinExpire(). Add flag REDISMODULE_HASH_EXPIRE_TIME to HashGet(). (#13676) ... This PR introduces API to query Expiration time of hash fields. # New `RedisModule_HashFieldMinExpire()` For a given hash, retrieves the minimum expiration time across all fields. If no fields have expiration or if the key is not a hash then return `REDISMODULE_NO_EXPIRE` (-1). ``` mstime_t RM_HashFieldMinExpire(RedisModuleKey *hash); ``` # Extension to `RedisModule_HashGet()` Adds a new flag, `REDISMODULE_HASH_EXPIRE_TIME`, to retrieve the expiration time of a specific hash field. If the field does not exist or has no expiration, returns `REDISMODULE_NO_EXPIRE`. It is fully backward-compatible (RM_HashGet retains its original behavior unless the new flag is used). Example: ``` mstime_t expiry1, expiry2; RedisModule_HashGet(mykey, REDISMODULE_HASH_EXPIRE_TIME, "field1", &expiry1, NULL); RedisModule_HashGet(mykey, REDISMODULE_HASH_EXPIRE_TIME, "field1", &expiry1, "field2", &expiry2, NULL); ```	2024-12-05 11:14:52 +02:00
Moti Cohen	06b144aa09	Modules API: Add RedisModule_ACLCheckKeyPrefixPermissions (#13666) ... This PR introduces a new API function to the Redis Module API: ``` int RedisModule_ACLCheckKeyPrefixPermissions(RedisModuleUser *user, RedisModuleString *prefix, int flags); ``` Purpose: The function checks if a given user has access permissions to any key that match a specific prefix. This validation is based on the user’s ACL permissions and the specified flags. Note, this prefix-based approach API may fail to detect prefixes that are individually uncovered but collectively covered by the patterns. For example the prefix `ID-*` is not fully included in pattern `ID-[0]*` and is not fully included in pattern `ID-[^0]*` but it is fully included in the set of patterns `{ID-[0]*, ID-[^0]*}`	2024-11-28 18:33:58 +02:00
Moti Cohen	155634502d	modules API: Support register unprefixed config parameters (#13656) ... PR #10285 introduced support for modules to register four types of configurations — Bool, Numeric, String, and Enum. Accessible through the Redis config file and the CONFIG command. With this PR, it will be possible to register configuration parameters without automatically prefixing the parameter names. This provides greater flexibility in configuration naming, enabling, for instance, both `bf-initial-size` or `initial-size` to be defined in the module without automatically prefixing with `<MODULE-NAME>.`. In addition it will also be possible to create a single additional alias via the same API. This brings us another step closer to integrate modules into redis core. **Example:** Register a configuration parameter `bf-initial-size` with an alias `initial-size` without the automatic module name prefix, set with new `REDISMODULE_CONFIG_UNPREFIXED` flag: ``` RedisModule_RegisterBoolConfig(ctx, "bf-initial-size|initial-size", default_val, optflags | REDISMODULE_CONFIG_UNPREFIXED, getfn, setfn, applyfn, privdata); ``` # API changes Related functions that now support unprefixed configuration flag (`REDISMODULE_CONFIG_UNPREFIXED`) along with optional alias: ``` RedisModule_RegisterBoolConfig RedisModule_RegisterEnumConfig RedisModule_RegisterNumericConfig RedisModule_RegisterStringConfig ``` # Implementation Details: `config.c`: On load server configuration, at function `loadServerConfigFromString()`, it collects all unknown configurations into `module_configs_queue` dictionary. These may include valid module configurations or invalid ones. They will be validated later by `loadModuleConfigs()` against the configurations declared by the loaded module(s). `Module.c:` The `ModuleConfig` structure has been modified to store now: (1) Full configuration name (2) Alias (3) Unprefixed flag status - ensuring that configurations retain their original registration format when triggered in notifications. Added error printout: This change introduces an error printout for unresolved configurations, detailing each unresolved parameter detected during startup. The last line in the output existed prior to this change and has been retained to systems relies on it: ``` 595011:M 18 Nov 2024 08:26:23.616 # Unresolved Configuration(s) Detected: 595011:M 18 Nov 2024 08:26:23.616 # >>> 'bf-initiel-size 8' 595011:M 18 Nov 2024 08:26:23.616 # >>> 'search-sizex 32' 595011:M 18 Nov 2024 08:26:23.616 # Module Configuration detected without loadmodule directive or no ApplyConfig call: aborting ``` # Backward Compatibility: Existing modules will function without modification, as the new functionality only applies if REDISMODULE_CONFIG_UNPREFIXED is explicitly set. # Module vs. Core API Conflict Behavior The new API allows to modules loading duplication of same configuration name or same configuration alias, just like redis core configuration allows (i.e. the users sets two configs with a different value, but these two configs are actually the same one). Unlike redis core, given a name and its alias, it doesn't allow have both configuration on load. To implement it, it is required to modify DS `module_configs_queue` to reflect the order of their loading and later on, during `loadModuleConfigs()`, resolve pairs of names and aliases and which one is the last one to apply. "Relaxing" this limitation can be deferred to a future update if necessary, but for now, we error in this case.	2024-11-21 09:55:02 +02:00
nafraf	5b84dc9678	Fix module loadex command crash due to invalid config (#13653) ... Fix to https://github.com/redis/redis/issues/13650 providing an invalid config to a module with datatype crashes when redis tries to unload the module due to the invalid config --------- Co-authored-by: debing.sun <debing.sun@redis.com>	2024-11-21 14:14:14 +08:00
Ozan Tezcan	54038811c0	Print command tokens on a crash when hide-user-data-from-log is enabled (#13639) ... If `hide-user-data-from-log` config is enabled, we don't print client argv in the crashlog to avoid leaking user info. Though, debugging a crash becomes harder as we don't see the command arguments causing the crash. With this PR, we'll be printing command tokens to the log. As we have command tokens defined in json schema for each command, using this data, we can find tokens in the client argv. e.g. `SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the log. Modules should introduce their command structure via `RM_SetCommandInfo()`. Then, on a crash we'll able to know module command tokens.	2024-11-11 09:34:18 +03:00
guybe7	ded8d993b7	Modules: defrag CB should take robj, not sds (#13627) ... Added a log of the keyname in the test modules to reproduce the problem (tests crash without the fix)	2024-10-30 17:32:51 +08:00
Moti Cohen	3a3cacfefa	Extend modules API to read also expired keys and subkeys (#13526) ... The PR extends `RedisModule_OpenKey`'s flags to include `REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired keys. It also allows to access expired subkeys. Currently relevant only for hash fields and has its impact on `RM_HashGet` and `RM_Scan`.	2024-09-19 20:47:00 +03:00
Meir Shpilraien (Spielrein)	d3d94ccf2e	Added new defrag API to allocate and free raw memory. (#13509) ... All the defrag allocations API expects to get a value and replace it, leaving the old value untouchable. In some cases a value might be shared between multiple keys, in such cases we can not simply replace it when the defrag callback is called. To allow support such use cases, the PR adds two new API's to the defrag API: 1. `RM_DefragAllocRaw` - allocate memory base on a given size. 2. `RM_DefragFreeRaw` - Free the given pointer. Those API's avoid using tcache so they operate just like `RM_DefragAlloc` but allows the user to split the allocation and the memory free operations into two stages and control when those happen. In addition the PR adds new API to allow the module to receive notifications when defrag start and end: `RM_RegisterDefragCallbacks` Those callbacks are the same as `RM_RegisterDefragFunc` but promised to be called and the start and the end of the defrag process.	2024-09-03 15:03:19 +03:00
debing.sun	52e12d8bac	Don't keep global replication buffer reference for replicas marked CLIENT_CLOSE_ASAP (#13363) ... In certain situations, we might generate a large number of propagates (e.g., multi/exec, Lua script, or a single command generating tons of propagations) within an event loop. During the process of propagating to a replica, if the replica is disconnected(marked as CLIENT_CLOSE_ASAP) due to exceeding the output buffer limit, we should remove its reference to the global replication buffer to avoid the global replication buffer being unable to be properly trimmed due to being referenced. --------- Co-authored-by: oranagra <oran@redislabs.com>	2024-06-26 08:26:23 +08:00
debing.sun	4581d43230	Fix daylight race condition and some thread leaks (#13191) ... fix some issues that come from sanitizer thread report. 1. when the main thread is updating daylight_active, other threads (bio, module thread) may be writing logs at the same time. ``` WARNING: ThreadSanitizer: data race (pid=661064) Read of size 4 at 0x55c9a4d11c70 by thread T2: #0 serverLogRaw /home/sundb/data/redis_fork/src/server.c:116 (redis-server+0x8d797) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #1 _serverLog.constprop.2 /home/sundb/data/redis_fork/src/server.c:146 (redis-server+0x2a3b14) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #2 bioProcessBackgroundJobs /home/sundb/data/redis_fork/src/bio.c:329 (redis-server+0x1c24ca) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) Previous write of size 4 at 0x55c9a4d11c70 by main thread (mutexes: write M0, write M1, write M2, write M3): #0 updateCachedTimeWithUs /home/sundb/data/redis_fork/src/server.c:1102 (redis-server+0x925e7) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #1 updateCachedTimeWithUs /home/sundb/data/redis_fork/src/server.c:1087 (redis-server+0x925e7) #2 updateCachedTime /home/sundb/data/redis_fork/src/server.c:1118 (redis-server+0x925e7) #3 afterSleep /home/sundb/data/redis_fork/src/server.c:1811 (redis-server+0x925e7) #4 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:389 (redis-server+0x85ae0) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #5 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:342 (redis-server+0x85ae0) #6 aeMain /home/sundb/data/redis_fork/src/ae.c:477 (redis-server+0x85ae0) #7 main /home/sundb/data/redis_fork/src/server.c:7211 (redis-server+0x7168c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) ``` 2. thread leaks in module tests ``` WARNING: ThreadSanitizer: thread leak (pid=668683) Thread T13 (tid=670041, finished) created by main thread at: #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x3d179) (BuildId: 28a9f70061dbb2dfa2cef661d3b23aff4ea13536) #1 HelloBlockNoTracking_RedisCommand /home/sundb/data/redis_fork/tests/modules/blockonbackground.c:200 (blockonbackground.so+0x97fd) (BuildId: 9cd187906c57e88cdf896d121d1d96448b37a136) #2 HelloBlockNoTracking_RedisCommand /home/sundb/data/redis_fork/tests/modules/blockonbackground.c:169 (blockonbackground.so+0x97fd) #3 call /home/sundb/data/redis_fork/src/server.c:3546 (redis-server+0x9b7fb) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #4 processCommand /home/sundb/data/redis_fork/src/server.c:4176 (redis-server+0xa091c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #5 processCommandAndResetClient /home/sundb/data/redis_fork/src/networking.c:2468 (redis-server+0xd2b8e) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #6 processInputBuffer /home/sundb/data/redis_fork/src/networking.c:2576 (redis-server+0xd2b8e) #7 readQueryFromClient /home/sundb/data/redis_fork/src/networking.c:2722 (redis-server+0xd358f) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #8 callHandler /home/sundb/data/redis_fork/src/connhelpers.h:58 (redis-server+0x288a7b) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #9 connSocketEventHandler /home/sundb/data/redis_fork/src/socket.c:277 (redis-server+0x288a7b) #10 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:417 (redis-server+0x85b45) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) #11 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:342 (redis-server+0x85b45) #12 aeMain /home/sundb/data/redis_fork/src/ae.c:477 (redis-server+0x85b45) #13 main /home/sundb/data/redis_fork/src/server.c:7211 (redis-server+0x7168c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0) ```	2024-04-04 13:49:51 +03:00
Pieter Cailliau	0b34396924	Change license from BSD-3 to dual RSALv2+SSPLv1 (#13157) ... [Read more about the license change here](https://redis.com/blog/redis-adopts-dual-source-available-licensing/) Live long and prosper 🖖	2024-03-20 22:38:24 +00:00
Viktor Söderqvist	9efc6ad6a6	Add API RedisModule_ClusterKeySlot and RedisModule_ClusterCanonicalKeyNameInSlot (#13069) ... Sometimes it's useful to compute a key's cluster slot in a module. This API function is just like the command CLUSTER KEYSLOT (but faster). A "reverse" API is also added: `RedisModule_ClusterCanonicalKeyNameInSlot`. Given a slot, it returns a short string that we can call a canonical key for the slot.	2024-03-12 09:26:12 -07:00
Binbin	74a6e48a3d	Fix module unblock crash due to no timeout_callback (#13017) ... The block timeout is passed in the test case, but we do not pass in the timeout_callback, and it will crash when unlocking. In this case, in moduleBlockedClientTimedOut we will check timeout_callback. There is the stack: ``` beforeSleep blockedBeforeSleep handleBlockedClientsTimeout checkBlockedClientTimeout unblockClientOnTimeout replyToBlockedClientTimedOut moduleBlockedClientTimedOut -- timeout_callback is NULL, invalidFunctionWasCalled bc->timeout_callback(&ctx,(void**)c->argv,c->argc); ```	2024-01-31 09:28:50 +02:00
Ozan Tezcan	c5273cae18	Add RM_TryCalloc() and RM_TryRealloc() (#12985) ... Modules may want to handle allocation failures gracefully. Adding RM_TryCalloc() and RM_TryRealloc() for it. RM_TryAlloc() was added before: https://github.com/redis/redis/pull/10541	2024-01-29 20:56:03 +02:00
debing.sun	d0640029dc	Fix race condition issues between the main thread and module threads (#12817) ... Fix #12785 and other race condition issues. See the following isolated comments. The following report was obtained using SANITIZER thread. ```sh make SANITIZER=thread ./runtest-moduleapi --config io-threads 4 --config io-threads-do-reads yes --accurate ``` 1. Fixed thread-safe issue in RM_UnblockClient() Related discussion: https://github.com/redis/redis/pull/12817#issuecomment-1831181220 * When blocking a client in a module using `RM_BlockClientOnKeys()` or `RM_BlockClientOnKeysWithFlags()` with a timeout_callback, calling RM_UnblockClient() in module threads can lead to race conditions in `updateStatsOnUnblock()`. - Introduced: Version: 6.2 PR: #7491 - Touch: `server.stat_numcommands`, `cmd->latency_histogram`, `server.slowlog`, and `server.latency_events` - Harm Level: High Potentially corrupts the memory data of `cmd->latency_histogram`, `server.slowlog`, and `server.latency_events` - Solution: Differentiate whether the call to moduleBlockedClientTimedOut() comes from the module or the main thread. Since we can't know if RM_UnblockClient() comes from module threads, we always assume it does and let `updateStatsOnUnblock()` asynchronously update the unblock status. * When error reply is called in timeout_callback(), ctx is not thread-safe, eventually lead to race conditions in `afterErrorReply`. - Introduced: Version: 6.2 PR: #8217 - Touch `server.stat_total_error_replies`, `server.errors`, - Harm Level: High Potentially corrupts the memory data of `server.errors` - Solution: Make the ctx in `timeout_callback()` with `REDISMODULE_CTX_THREAD_SAFE`, and asynchronously reply errors to the client. 2. Made RM_Reply*() family API thread-safe Related discussion: https://github.com/redis/redis/pull/12817#discussion_r1408707239 Call chain: `RM_Reply*()` -> `_addReplyToBufferOrList()` -> touch server.current_client - Introduced: Version: 7.2.0 PR: #12326 - Harm Level: None Since the module fake client won't have the `CLIENT_PUSHING` flag, even if we touch server.current_client, we can still exit after `c->flags & CLIENT_PUSHING`. - Solution Checking `c->flags & CLIENT_PUSHING` earlier. 3. Made freeClient() thread-safe Fix #12785 - Introduced: Version: 4.0 Commit: 3fcf959e60 - Harm Level: Moderate * Trigger assertion It happens when the module thread calls freeClient while the io-thread is in progress, which just triggers an assertion, and doesn't make any race condiaions. * Touch `server.current_client`, `server.stat_clients_type_memory`, and `clientMemUsageBucket->clients`. It happens between the main thread and the module threads, may cause data corruption. 1. Error reset `server.current_client` to NULL, but theoretically this won't happen, because the module has already reset `server.current_client` to old value before entering freeClient. 2. corrupts `clientMemUsageBucket->clients` in updateClientMemUsageAndBucket(). 3. Causes server.stat_clients_type_memory memory statistics to be inaccurate. - Solution: * No longer counts memory usage on fake clients, to avoid updating `server.stat_clients_type_memory` in freeClient. * No longer resetting `server.current_client` in unlinkClient, because the fake client won't be evicted or disconnected in the mid of the process. * Judgment assertion `io_threads_op == IO_THREADS_OP_IDLE` only if c is not a fake client. 4. Fixed free client args without GIL Related discussion: https://github.com/redis/redis/pull/12817#discussion_r1408706695 When freeing retained strings in the module thread (refcount decr), or using them in some way (refcount incr), we should do so while holding the GIL, otherwise, they might be simultaneously freed while the main thread is processing the unblock client state. - Introduced: Version: 6.2.0 PR: #8141 - Harm Level: Low Trigger assertion or double free or memory leak. - Solution: Documenting that module API users need to ensure any access to these retained strings is done with the GIL locked 5. Fix adding fake client to server.clients_pending_write It will incorrectly log the memory usage for the fake client. Related discussion: https://github.com/redis/redis/pull/12817#issuecomment-1851899163 - Introduced: Version: 4.0 Commit: 9b01b64430 - Harm Level: None Only result in NOP - Solution: * Don't add fake client into server.clients_pending_write * Add c->conn assertion for updateClientMemUsageAndBucket() and updateClientMemoryUsage() to avoid same issue in the future. So now it will be the responsibility of the caller of both of them to avoid passing in fake client. 6. Fix calling RM_BlockedClientMeasureTimeStart() and RM_BlockedClientMeasureTimeEnd() without GIL - Introduced: Version: 6.2 PR: #7491 - Harm Level: Low Causes inaccuracies in command latency histogram and slow logs, but does not corrupt memory. - Solution: Module API users, if know that non-thread-safe APIs will be used in multi-threading, need to take responsibility for protecting them with their own locks instead of the GIL, as using the GIL is too expensive. ### Other issue 1. RM_Yield is not thread-safe, fixed via #12905. ### Summarize 1. Fix thread-safe issues for `RM_UnblockClient()`, `freeClient()` and `RM_Yield`, potentially preventing memory corruption, data disorder, or assertion. 2. Updated docs and module test to clarify module API users' responsibility for locking non-thread-safe APIs in multi-threading, such as RM_BlockedClientMeasureTimeStart/End(), RM_FreeString(), RM_RetainString(), and RM_HoldString(). ### About backpot to 7.2 1. The implement of (1) is not too satisfying, would like to get more eyes. 2. (2), (3) can be safely for backport 3. (4), (6) just modifying the module tests and updating the documentation, no need for a backpot. 4. (5) is harmless, no need for a backpot. --------- Co-authored-by: Oran Agra <oran@redislabs.com>	2024-01-19 15:12:49 +02:00
Madelyn Olson	068051e378	Handle recursive serverAsserts and provide more information for recursive segfaults (#12857) ... This change is trying to make two failure modes a bit easier to deep dive: 1. If a serverPanic or serverAssert occurs during the info (or module) printing, it will recursively panic, which is a lot of fun as it will just keep recursively printing. It will eventually stack overflow, but will generate a lot of text in the process. 2. When a segfault happens during the segfault handler, no information is communicated other than it happened. This can be problematic because `info` may help diagnose the real issue, but without fixing the recursive crash it might be hard to get at that info.	2024-01-02 18:20:22 -08:00
Binbin	c85a9b7896	Fix delKeysInSlot server events are not executed inside an execution unit (#12745) ... This is a follow-up fix to #12733. We need to apply the same changes to delKeysInSlot. Refer to #12733 for more details. This PR contains some other minor cleanups / improvements to the test suite and docs. It uses the postnotifications test module in a cluster mode test which revealed a leak in the test module (fixed).	2023-12-11 20:15:19 +02:00
sundb	91309f7981	Fix compilation warning in KeySpace_ServerEventCallback and add CFLAGS=-Werror flag for module CI (#12786) ... Warning: ``` postnotifications.c:216:77: warning: format specifies type 'long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Wformat] RedisModule_Log(ctx, "warning", "Got an unexpected subevent '%ld'", subevent); ~~~ ^~~~~~~~ %llu ``` CI: https://github.com/redis/redis/actions/runs/6937308713/job/18871124342#step:6:115 ## Other Add `CFLAGS=-Werror` flag for module CI. --------- Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>	2023-11-30 17:41:00 +02:00
Binbin	d6f19539d2	Un-register notification and server event when RedisModule_OnLoad fails (#12809) ... When we register notification or server event in RedisModule_OnLoad, but RedisModule_OnLoad eventually fails, triggering notification or server event will cause the server to crash. If the loading fails on a later stage of moduleLoad, we do call moduleUnload which handles all un-registration, but when it fails on the RedisModule_OnLoad call, we only un-register several specific things and these were missing: - moduleUnsubscribeNotifications - moduleUnregisterFilters - moduleUnsubscribeAllServerEvents Refactored the code to reuse the code from moduleUnload. Fixes #12808.	2023-11-27 17:26:33 +02:00
Meir Shpilraien (Spielrein)	0ffb9d2ea9	Before evicted and before expired server events are not executed inside an execution unit. (#12733) ... Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`. This new event allows the module to read the key data just before it is removed from the database (either deleted, expired, evicted, or overwritten). When the key is removed from the database, either by active expire or eviction. The new event was not called as part of an execution unit. This can cause an issue if the module registers a post notification job inside the event. This job will not be executed atomically with the expiration/eviction operation and will not replicated inside a Multi/Exec. Moreover, the post notification job will be executed right after the event where it is still not safe to perform any write operation, this will violate the promise that post notification job will be called atomically with the operation that triggered it and **only when it is safe to write**. This PR fixes the issue by wrapping each expiration/eviction of a key with an execution unit. This makes sure the entire operation will run atomically and all the post notification jobs will be executed at the end where it is safe to write. Tests were modified to verify the fix.	2023-11-08 09:28:22 +02:00
Roshan Khatri	7519960527	Allows modules to declare new ACL categories. (#12486) ... This PR adds a new Module API int RM_AddACLCategory(RedisModuleCtx *ctx, const char *category_name) to add a new ACL command category. Here, we initialize the ACLCommandCategories array by allocating space for 64 categories and duplicate the 21 default categories from the predefined array 'ACLDefaultCommandCategories' into the ACLCommandCategories array while ACL initialization. Valid ACL category names can only contain alphanumeric characters, underscores, and dashes. The API when called, checks for the onload flag, category name validity, and for duplicate category name if present. If the conditions are satisfied, the API adds the new category to the trailing end of the ACLCommandCategories array and assigns the acl_categories flag bit according to the index at which the category is added. If any error is encountered the errno is set accordingly by the API. --------- Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>	2023-08-30 13:01:24 -07:00
Oran Agra	2b8cde71bb	Update supported version list. (#12488) ... Add 7.2, drop 6.0 as per https://redis.io/docs/about/releases/ Also replace a few concordances of the `’` char, with standard `'`	2023-08-16 08:36:40 +03:00
Meir Shpilraien (Spielrein)	153f8f082e	Fix use after free on blocking RM_Call. (#12342) ... blocking RM_Call was introduced on: #11568, It allows a module to perform blocking commands and get the reply asynchronously.If the command gets block, a special promise CallReply is returned that allow to set the unblock handler. The unblock handler will be called when the command invocation finish and it gets, as input, the command real reply. The issue was that the real CallReply was created using a stack allocated RedisModuleCtx which is no longer available after the unblock handler finishes. So if the module keeps the CallReply after the unblock handler finished, the CallReply holds a pointer to invalid memory and will try to access it when the CallReply will be released. The solution is to create the CallReply with a NULL context to make it totally detached and can be freed freely when the module wants. Test was added to cover this case, running the test with valgrind before the fix shows the use after free error. With the fix, there are no valgrind errors. unrelated: adding a missing `$rd close` in many tests in that file.	2023-06-25 14:12:27 +03:00
guybe7	3230199920	Modules: Unblock from within a timer coverage (#12337) ... Apart from adding the missing coverage, this PR also adds `blockedBeforeSleep` that gathers all block-related functions from `beforeSleep` The order inside `blockedBeforeSleep` is different: now `handleClientsBlockedOnKeys` (which may unblock clients) is called before `processUnblockedClients` (which handles unblocked clients). It makes sense to have this order. There are no visible effects of the wrong ordering, except some cleanups of the now-unblocked client would have happen in the next `beforeSleep` (will now happen in the current one) The reason we even got into it is because i triggers an assertion in logresreq.c (breaking the assumption that `unblockClient` is called **before** actually flushing the reply to the socket): `handleClientsBlockedOnKeys` is called, then it calls `moduleUnblockClientOnKey`, which calls `moduleUnblockClient`, which adds the client to `moduleUnblockedClients` back to `beforeSleep`, we call `handleClientsWithPendingWritesUsingThreads`, it writes the data of buf to the client, so `client->bufpos` became 0 On the next `beforeSleep`, we call `moduleHandleBlockedClients`, which calls `unblockClient`, which calls `reqresAppendResponse`, triggering the assert. (because the `bufpos` is 0) - see https://github.com/redis/redis/pull/12301#discussion_r1226386716	2023-06-22 23:15:16 +03:00