2021-06-09 20:13:24 +08:00
|
|
|
start_server {tags {"other"}} {
|
2011-07-11 18:56:00 +08:00
|
|
|
if {$::force_failure} {
|
|
|
|
# This is used just for test suite development purposes.
|
|
|
|
test {Failing test} {
|
|
|
|
format err
|
|
|
|
} {ok}
|
|
|
|
}
|
|
|
|
|
Add reply_schema to command json files (internal for now) (#10273)
Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
Since ironing the details of the reply schema of each and every command can take a long time, we
would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
### Background
In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
### Motivation
1. Documentation. This is the primary goal.
2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
testsuite, see the "Testing" section)
### Schema
The idea is to supply some sort of schema for the various replies of each command.
The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
and without the `FULL` modifier)
We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
Example for `BZPOPMIN`:
```
"reply_schema": {
"oneOf": [
{
"description": "Timeout reached and no elements were popped.",
"type": "null"
},
{
"description": "The keyname, popped member, and its score.",
"type": "array",
"minItems": 3,
"maxItems": 3,
"items": [
{
"description": "Keyname",
"type": "string"
},
{
"description": "Member",
"type": "string"
},
{
"description": "Score",
"type": "number"
}
]
}
]
}
```
#### Notes
1. It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
schema for documentation (and possibly to create a fuzzer that validates the replies)
3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
is with `WITHSCORES` and the other is without.
4. For documentation, there will be another optional field "notes" in which we will add a short description of
the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
array, for example)
Given the above:
1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
(given that "description" and "notes" are comprehensive enough)
2. We can generate a client in a strongly typed language (but the return type could be a conceptual
`union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
3. We can create a fuzzer for RESP3.
### Limitations (because we are using the standard json-schema)
The problem is that Redis' replies are more diverse than what the json format allows. This means that,
when we convert the reply to a json (in order to validate the schema against it), we lose information (see
the "Testing" section below).
The other option would have been to extend the standard json-schema (and json format) to include stuff
like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
seemed like too much work, so we decided to compromise.
Examples:
1. We cannot tell the difference between an "array" and a "set"
2. We cannot tell the difference between simple-string and bulk-string
3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
compares (member,score) tuples and not just the member name.
### Testing
This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
are indeed correct (i.e. describe the actual response of Redis).
To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
it executed and their replies.
For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
`--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
`--log-req-res --force-resp3`)
You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
`.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
These files are later on processed by `./utils/req-res-log-validator.py` which does:
1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
2. For each request-response pair, it validates the response against the request's reply_schema
(obtained from the extended COMMAND DOCS)
5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
the existing redis test suite, rather than attempt to write a fuzzer.
#### Notes about RESP2
1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
accept RESP3 as the future RESP)
2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
so that we can validate it, we will need to know how to convert the actual reply to the one expected.
- number and boolean are always strings in RESP2 so the conversion is easy
- objects (maps) are always a flat array in RESP2
- others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
handling (so the client will not be totally auto-generated)
Example for ZRANGE:
```
"reply_schema": {
"anyOf": [
{
"description": "A list of member elements",
"type": "array",
"uniqueItems": true,
"items": {
"type": "string"
}
},
{
"description": "Members and their scores. Returned in case `WITHSCORES` was used.",
"notes": "In RESP2 this is returned as a flat array",
"type": "array",
"uniqueItems": true,
"items": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": [
{
"description": "Member",
"type": "string"
},
{
"description": "Score",
"type": "number"
}
]
}
}
]
}
```
### Other changes
1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
regardless of the special log-req-res mode ("Pub/Sub PING" for example)
2. Update the history field of CLIENT LIST
3. Added basic tests for commands that were not covered at all by the testsuite
### TODO
- [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
- [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
- [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
- [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
of the tests - https://github.com/redis/redis/issues/11897
- [x] (probably a separate PR) add all missing schemas
- [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
- [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
fight with the tcl including mechanism a bit)
- [x] issue: module API - https://github.com/redis/redis/issues/11898
- [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Hanna Fadida <hanna.fadida@redislabs.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Shaya Potter <shaya@redislabs.com>
2023-03-11 16:14:16 +08:00
|
|
|
test {Coverage: HELP commands} {
|
|
|
|
assert_match "*OBJECT <subcommand> *" [r OBJECT HELP]
|
|
|
|
assert_match "*MEMORY <subcommand> *" [r MEMORY HELP]
|
|
|
|
assert_match "*PUBSUB <subcommand> *" [r PUBSUB HELP]
|
|
|
|
assert_match "*SLOWLOG <subcommand> *" [r SLOWLOG HELP]
|
|
|
|
assert_match "*CLIENT <subcommand> *" [r CLIENT HELP]
|
|
|
|
assert_match "*COMMAND <subcommand> *" [r COMMAND HELP]
|
|
|
|
assert_match "*CONFIG <subcommand> *" [r CONFIG HELP]
|
|
|
|
assert_match "*FUNCTION <subcommand> *" [r FUNCTION HELP]
|
|
|
|
assert_match "*MODULE <subcommand> *" [r MODULE HELP]
|
|
|
|
}
|
|
|
|
|
|
|
|
test {Coverage: MEMORY MALLOC-STATS} {
|
|
|
|
if {[string match {*jemalloc*} [s mem_allocator]]} {
|
|
|
|
assert_match "*jemalloc*" [r memory malloc-stats]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
test {Coverage: MEMORY PURGE} {
|
|
|
|
if {[string match {*jemalloc*} [s mem_allocator]]} {
|
|
|
|
assert_equal {OK} [r memory purge]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-05-14 23:31:11 +08:00
|
|
|
test {SAVE - make sure there are all the types as values} {
|
|
|
|
# Wait for a background saving in progress to terminate
|
|
|
|
waitForBgsave r
|
|
|
|
r lpush mysavelist hello
|
|
|
|
r lpush mysavelist world
|
|
|
|
r set myemptykey {}
|
|
|
|
r set mynormalkey {blablablba}
|
|
|
|
r zadd mytestzset 10 a
|
|
|
|
r zadd mytestzset 20 b
|
|
|
|
r zadd mytestzset 30 c
|
|
|
|
r save
|
2021-06-09 20:13:24 +08:00
|
|
|
} {OK} {needs:save}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
2011-07-11 05:25:48 +08:00
|
|
|
tags {slow} {
|
2011-07-11 18:15:35 +08:00
|
|
|
if {$::accurate} {set iterations 10000} else {set iterations 1000}
|
2010-06-03 06:16:10 +08:00
|
|
|
foreach fuzztype {binary alpha compr} {
|
|
|
|
test "FUZZ stresser with data model $fuzztype" {
|
|
|
|
set err 0
|
2011-07-11 18:15:35 +08:00
|
|
|
for {set i 0} {$i < $iterations} {incr i} {
|
2010-06-03 06:16:10 +08:00
|
|
|
set fuzz [randstring 0 512 $fuzztype]
|
|
|
|
r set foo $fuzz
|
|
|
|
set got [r get foo]
|
|
|
|
if {$got ne $fuzz} {
|
|
|
|
set err [list $fuzz $got]
|
|
|
|
break
|
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
}
|
2010-06-03 06:16:10 +08:00
|
|
|
set _ $err
|
|
|
|
} {0}
|
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
}
|
|
|
|
|
FLUSHDB and FLUSHALL add call forceCommandPropagation / FLUSHALL reset dirty counter to 0 if we enable save (#10691)
## FLUSHALL
We used to restore the dirty counter after `rdbSave` zeroed it if we enable save.
Otherwise FLUSHALL will not be replicated nor put into the AOF.
And then we do increment it again below.
Without that extra dirty++, when db was already empty, FLUSHALL
will not be replicated nor put into the AOF.
We now gonna replace all that dirty counter magic with a call
to forceCommandPropagation (REPL and AOF), instead of all the
messing around with the dirty counter.
Added tests to cover three part (dirty counter, REPL, AOF).
One benefit other than cleaner code is that the `rdb_changes_since_last_save` is correct in this case.
## FLUSHDB
FLUSHDB was not replicated nor put into the AOF when db was already empty.
Unlike DEL on a non-existing key, FLUSHDB always does something, and that's to call the module hook.
So basically FLUSHDB is never a NOP, and thus it should always be propagated.
Not doing that, could mean that if a module does something in that hook, and wants to
avoid issues of that hook being missing on the replica if the db is empty, it'll need to do complicated things.
So now FLUSHDB add call forceCommandPropagation, we will always propagate FLUSHDB.
Always propagating FLUSHDB seems like a safe approach that shouldn't have any drawbacks (other than looking odd)
This was mentioned in #8972
## Test section:
We actually found it while solving a race condition in the BGSAVE test (other.tcl).
It was found in extra_ci Daily Arm64 (test-libc-malloc).
```
[exception]: Executing test client: ERR Background save already in progress.
ERR Background save already in progress
```
It look like `r flushdb` trigger (schedule) a bgsave right after `waitForBgsave r` and before `r save`.
Changing flushdb to flushall, FLUSHALL will do a foreground save and then set the dirty counter to 0.
2022-05-11 16:21:16 +08:00
|
|
|
start_server {overrides {save ""} tags {external:skip}} {
|
|
|
|
test {FLUSHALL should not reset the dirty counter if we disable save} {
|
|
|
|
r set key value
|
|
|
|
r flushall
|
|
|
|
assert_morethan [s rdb_changes_since_last_save] 0
|
|
|
|
}
|
|
|
|
|
|
|
|
test {FLUSHALL should reset the dirty counter to 0 if we enable save} {
|
|
|
|
r config set save "3600 1 300 100 60 10000"
|
|
|
|
r set key value
|
|
|
|
r flushall
|
|
|
|
assert_equal [s rdb_changes_since_last_save] 0
|
|
|
|
}
|
2024-07-01 14:04:52 +08:00
|
|
|
|
|
|
|
test {FLUSHALL and bgsave} {
|
|
|
|
r config set save "3600 1 300 100 60 10000"
|
|
|
|
r set x y
|
|
|
|
r bgsave
|
|
|
|
r set x y
|
|
|
|
r multi
|
|
|
|
r debug sleep 1
|
|
|
|
# by the time we'll get to run flushall, the child will finish,
|
|
|
|
# but the parent will be unaware of it, and it could wrongly set the dirty counter.
|
|
|
|
r flushall
|
|
|
|
r exec
|
|
|
|
assert_equal [s rdb_changes_since_last_save] 0
|
|
|
|
}
|
FLUSHDB and FLUSHALL add call forceCommandPropagation / FLUSHALL reset dirty counter to 0 if we enable save (#10691)
## FLUSHALL
We used to restore the dirty counter after `rdbSave` zeroed it if we enable save.
Otherwise FLUSHALL will not be replicated nor put into the AOF.
And then we do increment it again below.
Without that extra dirty++, when db was already empty, FLUSHALL
will not be replicated nor put into the AOF.
We now gonna replace all that dirty counter magic with a call
to forceCommandPropagation (REPL and AOF), instead of all the
messing around with the dirty counter.
Added tests to cover three part (dirty counter, REPL, AOF).
One benefit other than cleaner code is that the `rdb_changes_since_last_save` is correct in this case.
## FLUSHDB
FLUSHDB was not replicated nor put into the AOF when db was already empty.
Unlike DEL on a non-existing key, FLUSHDB always does something, and that's to call the module hook.
So basically FLUSHDB is never a NOP, and thus it should always be propagated.
Not doing that, could mean that if a module does something in that hook, and wants to
avoid issues of that hook being missing on the replica if the db is empty, it'll need to do complicated things.
So now FLUSHDB add call forceCommandPropagation, we will always propagate FLUSHDB.
Always propagating FLUSHDB seems like a safe approach that shouldn't have any drawbacks (other than looking odd)
This was mentioned in #8972
## Test section:
We actually found it while solving a race condition in the BGSAVE test (other.tcl).
It was found in extra_ci Daily Arm64 (test-libc-malloc).
```
[exception]: Executing test client: ERR Background save already in progress.
ERR Background save already in progress
```
It look like `r flushdb` trigger (schedule) a bgsave right after `waitForBgsave r` and before `r save`.
Changing flushdb to flushall, FLUSHALL will do a foreground save and then set the dirty counter to 0.
2022-05-11 16:21:16 +08:00
|
|
|
}
|
|
|
|
|
2010-05-14 23:31:11 +08:00
|
|
|
test {BGSAVE} {
|
FLUSHDB and FLUSHALL add call forceCommandPropagation / FLUSHALL reset dirty counter to 0 if we enable save (#10691)
## FLUSHALL
We used to restore the dirty counter after `rdbSave` zeroed it if we enable save.
Otherwise FLUSHALL will not be replicated nor put into the AOF.
And then we do increment it again below.
Without that extra dirty++, when db was already empty, FLUSHALL
will not be replicated nor put into the AOF.
We now gonna replace all that dirty counter magic with a call
to forceCommandPropagation (REPL and AOF), instead of all the
messing around with the dirty counter.
Added tests to cover three part (dirty counter, REPL, AOF).
One benefit other than cleaner code is that the `rdb_changes_since_last_save` is correct in this case.
## FLUSHDB
FLUSHDB was not replicated nor put into the AOF when db was already empty.
Unlike DEL on a non-existing key, FLUSHDB always does something, and that's to call the module hook.
So basically FLUSHDB is never a NOP, and thus it should always be propagated.
Not doing that, could mean that if a module does something in that hook, and wants to
avoid issues of that hook being missing on the replica if the db is empty, it'll need to do complicated things.
So now FLUSHDB add call forceCommandPropagation, we will always propagate FLUSHDB.
Always propagating FLUSHDB seems like a safe approach that shouldn't have any drawbacks (other than looking odd)
This was mentioned in #8972
## Test section:
We actually found it while solving a race condition in the BGSAVE test (other.tcl).
It was found in extra_ci Daily Arm64 (test-libc-malloc).
```
[exception]: Executing test client: ERR Background save already in progress.
ERR Background save already in progress
```
It look like `r flushdb` trigger (schedule) a bgsave right after `waitForBgsave r` and before `r save`.
Changing flushdb to flushall, FLUSHALL will do a foreground save and then set the dirty counter to 0.
2022-05-11 16:21:16 +08:00
|
|
|
# Use FLUSHALL instead of FLUSHDB, FLUSHALL do a foreground save
|
|
|
|
# and reset the dirty counter to 0, so we won't trigger an unexpected bgsave.
|
|
|
|
r flushall
|
2010-05-14 23:31:11 +08:00
|
|
|
r save
|
|
|
|
r set x 10
|
|
|
|
r bgsave
|
|
|
|
waitForBgsave r
|
|
|
|
r debug reload
|
|
|
|
r get x
|
2021-12-19 23:41:51 +08:00
|
|
|
} {10} {needs:debug needs:save}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
|
|
|
test {SELECT an out of range DB} {
|
|
|
|
catch {r select 1000000} err
|
|
|
|
set _ $err
|
2021-06-09 20:13:24 +08:00
|
|
|
} {*index is out of range*} {cluster:skip}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
2011-07-11 05:25:48 +08:00
|
|
|
tags {consistency} {
|
2021-06-09 20:13:24 +08:00
|
|
|
proc check_consistency {dumpname code} {
|
|
|
|
set dump [csvdump r]
|
2021-12-19 23:41:51 +08:00
|
|
|
set sha1 [debug_digest]
|
2021-06-09 20:13:24 +08:00
|
|
|
|
|
|
|
uplevel 1 $code
|
|
|
|
|
2021-12-19 23:41:51 +08:00
|
|
|
set sha1_after [debug_digest]
|
2021-06-09 20:13:24 +08:00
|
|
|
if {$sha1 eq $sha1_after} {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
# Failed
|
|
|
|
set newdump [csvdump r]
|
|
|
|
puts "Consistency test failed!"
|
|
|
|
puts "You can inspect the two dumps in /tmp/${dumpname}*.txt"
|
|
|
|
|
|
|
|
set fd [open /tmp/${dumpname}1.txt w]
|
|
|
|
puts $fd $dump
|
|
|
|
close $fd
|
|
|
|
set fd [open /tmp/${dumpname}2.txt w]
|
|
|
|
puts $fd $newdump
|
|
|
|
close $fd
|
|
|
|
|
|
|
|
return 0
|
|
|
|
}
|
|
|
|
|
|
|
|
if {$::accurate} {set numops 10000} else {set numops 1000}
|
|
|
|
test {Check consistency of different data types after a reload} {
|
|
|
|
r flushdb
|
2024-05-29 19:47:48 +08:00
|
|
|
# TODO: integrate usehexpire following next commit that will support replication
|
|
|
|
createComplexDataset r $numops {usetag usehexpire}
|
2021-06-09 20:13:24 +08:00
|
|
|
if {$::ignoredigest} {
|
|
|
|
set _ 1
|
|
|
|
} else {
|
|
|
|
check_consistency {repldump} {
|
|
|
|
r debug reload
|
2010-07-27 20:42:11 +08:00
|
|
|
}
|
2021-06-09 20:13:24 +08:00
|
|
|
}
|
2021-12-19 23:41:51 +08:00
|
|
|
} {1} {needs:debug}
|
2021-06-09 20:13:24 +08:00
|
|
|
|
|
|
|
test {Same dataset digest if saving/reloading as AOF?} {
|
|
|
|
if {$::ignoredigest} {
|
|
|
|
set _ 1
|
|
|
|
} else {
|
|
|
|
check_consistency {aofdump} {
|
|
|
|
r config set aof-use-rdb-preamble no
|
|
|
|
r bgrewriteaof
|
|
|
|
waitForBgrewriteaof r
|
|
|
|
r debug loadaof
|
2010-07-27 20:42:11 +08:00
|
|
|
}
|
2021-06-09 20:13:24 +08:00
|
|
|
}
|
|
|
|
} {1} {needs:debug}
|
2010-05-14 23:31:11 +08:00
|
|
|
}
|
|
|
|
|
2011-11-11 22:11:50 +08:00
|
|
|
test {EXPIRES after a reload (snapshot + append only file rewrite)} {
|
2010-05-14 23:31:11 +08:00
|
|
|
r flushdb
|
|
|
|
r set x 10
|
|
|
|
r expire x 1000
|
2011-07-11 05:25:48 +08:00
|
|
|
r save
|
|
|
|
r debug reload
|
2010-05-14 23:31:11 +08:00
|
|
|
set ttl [r ttl x]
|
|
|
|
set e1 [expr {$ttl > 900 && $ttl <= 1000}]
|
2011-07-11 05:25:48 +08:00
|
|
|
r bgrewriteaof
|
|
|
|
waitForBgrewriteaof r
|
|
|
|
r debug loadaof
|
2010-05-14 23:31:11 +08:00
|
|
|
set ttl [r ttl x]
|
|
|
|
set e2 [expr {$ttl > 900 && $ttl <= 1000}]
|
|
|
|
list $e1 $e2
|
2021-06-09 20:13:24 +08:00
|
|
|
} {1 1} {needs:debug needs:save}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
2011-11-11 22:11:50 +08:00
|
|
|
test {EXPIRES after AOF reload (without rewrite)} {
|
|
|
|
r flushdb
|
|
|
|
r config set appendonly yes
|
2018-03-25 19:03:38 +08:00
|
|
|
r config set aof-use-rdb-preamble no
|
2011-11-11 22:11:50 +08:00
|
|
|
r set x somevalue
|
|
|
|
r expire x 1000
|
|
|
|
r setex y 2000 somevalue
|
|
|
|
r set z somevalue
|
|
|
|
r expireat z [expr {[clock seconds]+3000}]
|
|
|
|
|
|
|
|
# Milliseconds variants
|
|
|
|
r set px somevalue
|
|
|
|
r pexpire px 1000000
|
|
|
|
r psetex py 2000000 somevalue
|
|
|
|
r set pz somevalue
|
|
|
|
r pexpireat pz [expr {([clock seconds]+3000)*1000}]
|
|
|
|
|
|
|
|
# Reload and check
|
2011-12-21 16:24:14 +08:00
|
|
|
waitForBgrewriteaof r
|
2012-01-26 23:45:08 +08:00
|
|
|
# We need to wait two seconds to avoid false positives here, otherwise
|
|
|
|
# the DEBUG LOADAOF command may read a partial file.
|
|
|
|
# Another solution would be to set the fsync policy to no, since this
|
|
|
|
# prevents write() to be delayed by the completion of fsync().
|
|
|
|
after 2000
|
2011-11-11 22:11:50 +08:00
|
|
|
r debug loadaof
|
|
|
|
set ttl [r ttl x]
|
|
|
|
assert {$ttl > 900 && $ttl <= 1000}
|
|
|
|
set ttl [r ttl y]
|
|
|
|
assert {$ttl > 1900 && $ttl <= 2000}
|
|
|
|
set ttl [r ttl z]
|
|
|
|
assert {$ttl > 2900 && $ttl <= 3000}
|
|
|
|
set ttl [r ttl px]
|
|
|
|
assert {$ttl > 900 && $ttl <= 1000}
|
|
|
|
set ttl [r ttl py]
|
|
|
|
assert {$ttl > 1900 && $ttl <= 2000}
|
|
|
|
set ttl [r ttl pz]
|
|
|
|
assert {$ttl > 2900 && $ttl <= 3000}
|
|
|
|
r config set appendonly no
|
2021-06-09 20:13:24 +08:00
|
|
|
} {OK} {needs:debug}
|
2011-11-11 22:11:50 +08:00
|
|
|
|
2011-07-11 05:25:48 +08:00
|
|
|
tags {protocol} {
|
2011-01-10 02:42:56 +08:00
|
|
|
test {PIPELINING stresser (also a regression for the old epoll bug)} {
|
2019-09-12 15:56:54 +08:00
|
|
|
if {$::tls} {
|
2020-05-26 16:00:48 +08:00
|
|
|
set fd2 [::tls::socket [srv host] [srv port]]
|
2019-09-12 15:56:54 +08:00
|
|
|
} else {
|
2020-05-26 16:00:48 +08:00
|
|
|
set fd2 [socket [srv host] [srv port]]
|
2019-09-12 15:56:54 +08:00
|
|
|
}
|
2011-01-10 02:42:56 +08:00
|
|
|
fconfigure $fd2 -encoding binary -translation binary
|
2021-12-19 23:41:51 +08:00
|
|
|
if {!$::singledb} {
|
|
|
|
puts -nonewline $fd2 "SELECT 9\r\n"
|
|
|
|
flush $fd2
|
|
|
|
gets $fd2
|
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
2011-01-10 02:42:56 +08:00
|
|
|
for {set i 0} {$i < 100000} {incr i} {
|
|
|
|
set q {}
|
|
|
|
set val "0000${i}0000"
|
|
|
|
append q "SET key:$i $val\r\n"
|
|
|
|
puts -nonewline $fd2 $q
|
|
|
|
set q {}
|
|
|
|
append q "GET key:$i\r\n"
|
|
|
|
puts -nonewline $fd2 $q
|
|
|
|
}
|
|
|
|
flush $fd2
|
2010-05-14 23:31:11 +08:00
|
|
|
|
2011-01-10 02:42:56 +08:00
|
|
|
for {set i 0} {$i < 100000} {incr i} {
|
|
|
|
gets $fd2 line
|
|
|
|
gets $fd2 count
|
|
|
|
set count [string range $count 1 end]
|
|
|
|
set val [read $fd2 $count]
|
|
|
|
read $fd2 2
|
|
|
|
}
|
|
|
|
close $fd2
|
|
|
|
set _ 1
|
|
|
|
} {1}
|
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
|
|
|
|
test {APPEND basics} {
|
2016-04-25 21:49:57 +08:00
|
|
|
r del foo
|
2010-05-14 23:31:11 +08:00
|
|
|
list [r append foo bar] [r get foo] \
|
|
|
|
[r append foo 100] [r get foo]
|
|
|
|
} {3 bar 6 bar100}
|
|
|
|
|
|
|
|
test {APPEND basics, integer encoded values} {
|
|
|
|
set res {}
|
|
|
|
r del foo
|
|
|
|
r append foo 1
|
|
|
|
r append foo 2
|
|
|
|
lappend res [r get foo]
|
|
|
|
r set foo 1
|
|
|
|
r append foo 2
|
|
|
|
lappend res [r get foo]
|
|
|
|
} {12 12}
|
|
|
|
|
|
|
|
test {APPEND fuzzing} {
|
|
|
|
set err {}
|
|
|
|
foreach type {binary alpha compr} {
|
|
|
|
set buf {}
|
|
|
|
r del x
|
|
|
|
for {set i 0} {$i < 1000} {incr i} {
|
|
|
|
set bin [randstring 0 10 $type]
|
|
|
|
append buf $bin
|
|
|
|
r append x $bin
|
|
|
|
}
|
|
|
|
if {$buf != [r get x]} {
|
|
|
|
set err "Expected '$buf' found '[r get x]'"
|
|
|
|
break
|
|
|
|
}
|
|
|
|
}
|
|
|
|
set _ $err
|
|
|
|
} {}
|
|
|
|
|
|
|
|
# Leave the user with a clean DB before to exit
|
|
|
|
test {FLUSHDB} {
|
|
|
|
set aux {}
|
2021-06-09 20:13:24 +08:00
|
|
|
if {$::singledb} {
|
|
|
|
r flushdb
|
|
|
|
lappend aux 0 [r dbsize]
|
|
|
|
} else {
|
|
|
|
r select 9
|
|
|
|
r flushdb
|
|
|
|
lappend aux [r dbsize]
|
|
|
|
r select 10
|
|
|
|
r flushdb
|
|
|
|
lappend aux [r dbsize]
|
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
} {0 0}
|
|
|
|
|
|
|
|
test {Perform a final SAVE to leave a clean DB on disk} {
|
2011-07-11 21:44:38 +08:00
|
|
|
waitForBgsave r
|
2010-05-14 23:31:11 +08:00
|
|
|
r save
|
2021-06-09 20:13:24 +08:00
|
|
|
} {OK} {needs:save}
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
test {RESET clears client state} {
|
|
|
|
r client setname test-client
|
|
|
|
r client tracking on
|
|
|
|
|
|
|
|
assert_equal [r reset] "RESET"
|
|
|
|
set client [r client list]
|
|
|
|
assert_match {*name= *} $client
|
|
|
|
assert_match {*flags=N *} $client
|
2021-06-09 20:13:24 +08:00
|
|
|
} {} {needs:reset}
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
test {RESET clears MONITOR state} {
|
|
|
|
set rd [redis_deferring_client]
|
|
|
|
$rd monitor
|
|
|
|
assert_equal [$rd read] "OK"
|
|
|
|
|
|
|
|
$rd reset
|
|
|
|
assert_equal [$rd read] "RESET"
|
2021-11-15 17:07:43 +08:00
|
|
|
$rd close
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
assert_no_match {*flags=O*} [r client list]
|
2021-06-09 20:13:24 +08:00
|
|
|
} {} {needs:reset}
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
test {RESET clears and discards MULTI state} {
|
|
|
|
r multi
|
|
|
|
r set key-a a
|
|
|
|
|
|
|
|
r reset
|
|
|
|
catch {r exec} err
|
|
|
|
assert_match {*EXEC without MULTI*} $err
|
2021-06-09 20:13:24 +08:00
|
|
|
} {} {needs:reset}
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
test {RESET clears Pub/Sub state} {
|
|
|
|
r subscribe channel-1
|
|
|
|
r reset
|
|
|
|
|
|
|
|
# confirm we're not subscribed by executing another command
|
|
|
|
r set key val
|
2021-06-09 20:13:24 +08:00
|
|
|
} {OK} {needs:reset}
|
2020-11-05 16:51:26 +08:00
|
|
|
|
|
|
|
test {RESET clears authenticated state} {
|
|
|
|
r acl setuser user1 on >secret +@all
|
|
|
|
r auth user1 secret
|
|
|
|
assert_equal [r acl whoami] user1
|
|
|
|
|
|
|
|
r reset
|
|
|
|
|
|
|
|
assert_equal [r acl whoami] default
|
2021-06-09 20:13:24 +08:00
|
|
|
} {} {needs:reset}
|
2022-01-09 19:06:51 +08:00
|
|
|
|
|
|
|
test "Subcommand syntax error crash (issue #10070)" {
|
|
|
|
assert_error {*unknown command*} {r GET|}
|
|
|
|
assert_error {*unknown command*} {r GET|SET}
|
|
|
|
assert_error {*unknown command*} {r GET|SET|OTHER}
|
|
|
|
assert_error {*unknown command*} {r CONFIG|GET GET_XX}
|
2022-04-25 18:08:13 +08:00
|
|
|
assert_error {*unknown subcommand*} {r CONFIG GET_XX}
|
2022-01-09 19:06:51 +08:00
|
|
|
}
|
2010-05-14 23:31:11 +08:00
|
|
|
}
|
2020-11-03 23:16:11 +08:00
|
|
|
|
2021-06-09 20:13:24 +08:00
|
|
|
start_server {tags {"other external:skip"}} {
|
2022-01-09 19:06:51 +08:00
|
|
|
test {Don't rehash if redis has child process} {
|
2020-11-03 23:16:11 +08:00
|
|
|
r config set save ""
|
|
|
|
r config set rdb-key-save-delay 1000000
|
|
|
|
|
2024-02-07 15:19:18 +08:00
|
|
|
populate 4095 "" 1
|
2020-11-03 23:16:11 +08:00
|
|
|
r bgsave
|
2021-03-09 03:22:08 +08:00
|
|
|
wait_for_condition 10 100 {
|
|
|
|
[s rdb_bgsave_in_progress] eq 1
|
|
|
|
} else {
|
|
|
|
fail "bgsave did not start in time"
|
|
|
|
}
|
|
|
|
|
2020-11-03 23:16:11 +08:00
|
|
|
r mset k1 v1 k2 v2
|
|
|
|
# Hash table should not rehash
|
|
|
|
assert_no_match "*table size: 8192*" [r debug HTSTATS 9]
|
|
|
|
exec kill -9 [get_child_pid 0]
|
2022-03-07 19:44:07 +08:00
|
|
|
waitForBgsave r
|
2020-11-03 23:16:11 +08:00
|
|
|
|
|
|
|
# Hash table should rehash since there is no child process,
|
2024-02-07 15:19:18 +08:00
|
|
|
# size is power of two and over 4096, so it is 8192
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
wait_for_condition 50 100 {
|
|
|
|
[string match "*table size: 8192*" [r debug HTSTATS 9]]
|
|
|
|
} else {
|
|
|
|
fail "hash table did not rehash after child process killed"
|
|
|
|
}
|
2021-12-19 23:41:51 +08:00
|
|
|
} {} {needs:debug needs:local-process}
|
2020-11-03 23:16:11 +08:00
|
|
|
}
|
2021-01-29 00:17:39 +08:00
|
|
|
|
|
|
|
proc read_proc_title {pid} {
|
|
|
|
set fd [open "/proc/$pid/cmdline" "r"]
|
|
|
|
set cmdline [read $fd 1024]
|
|
|
|
close $fd
|
|
|
|
|
|
|
|
return $cmdline
|
|
|
|
}
|
|
|
|
|
2021-06-09 20:13:24 +08:00
|
|
|
start_server {tags {"other external:skip"}} {
|
2021-01-29 00:17:39 +08:00
|
|
|
test {Process title set as expected} {
|
|
|
|
# Test only on Linux where it's easy to get cmdline without relying on tools.
|
|
|
|
# Skip valgrind as it messes up the arguments.
|
|
|
|
set os [exec uname]
|
|
|
|
if {$os == "Linux" && !$::valgrind} {
|
|
|
|
# Set a custom template
|
|
|
|
r config set "proc-title-template" "TEST {title} {listen-addr} {port} {tls-port} {unixsocket} {config-file}"
|
|
|
|
set cmdline [read_proc_title [srv 0 pid]]
|
|
|
|
|
|
|
|
assert_equal "TEST" [lindex $cmdline 0]
|
|
|
|
assert_match "*/redis-server" [lindex $cmdline 1]
|
|
|
|
|
|
|
|
if {$::tls} {
|
Support TLS service when "tls-cluster" is not enabled and persist both plain and TLS port in nodes.conf (#12233)
Originally, when "tls-cluster" is enabled, `port` is set to TLS port. In order to support non-TLS clients, `pport` is used to propagate TCP port across cluster nodes. However when "tls-cluster" is disabled, `port` is set to TCP port, and `pport` is not used, which means the cluster cannot provide TLS service unless "tls-cluster" is on.
```
typedef struct {
// ...
uint16_t port; /* Latest known clients port (TLS or plain). */
uint16_t pport; /* Latest known clients plaintext port. Only used if the main clients port is for TLS. */
// ...
} clusterNode;
```
```
typedef struct {
// ...
uint16_t port; /* TCP base port number. */
uint16_t pport; /* Sender TCP plaintext port, if base port is TLS */
// ...
} clusterMsg;
```
This PR renames `port` and `pport` in `clusterNode` to `tcp_port` and `tls_port`, to record both ports no matter "tls-cluster" is enabled or disabled.
This allows to provide TLS service to clients when "tls-cluster" is disabled: when displaying cluster topology, or giving `MOVED` error, server can provide TLS or TCP port according to client's connection type, no matter what type of connection cluster bus is using.
For backwards compatibility, `port` and `pport` in `clusterMsg` are preserved, when "tls-cluster" is enabled, `port` is set to TLS port and `pport` is set to TCP port, when "tls-cluster" is disabled, `port` is set to TCP port and `pport` is set to TLS port (instead of 0).
Also, in the nodes.conf file, a new aux field displaying an extra port is added to complete the persisted info. We may have `tls_port=xxxxx` or `tcp_port=xxxxx` in the aux field, to complete the cluster topology, while the other port is stored in the normal `<ip>:<port>` field. The format is shown below.
```
<node-id> <ip>:<tcp_port>@<cport>,<hostname>,shard-id=...,tls-port=6379 myself,master - 0 0 0 connected 0-1000
```
Or we can switch the position of two ports, both can be correctly resolved.
```
<node-id> <ip>:<tls_port>@<cport>,<hostname>,shard-id=...,tcp-port=6379 myself,master - 0 0 0 connected 0-1000
```
2023-06-26 22:43:38 +08:00
|
|
|
set expect_port [srv 0 pport]
|
2021-01-29 00:17:39 +08:00
|
|
|
set expect_tls_port [srv 0 port]
|
Support TLS service when "tls-cluster" is not enabled and persist both plain and TLS port in nodes.conf (#12233)
Originally, when "tls-cluster" is enabled, `port` is set to TLS port. In order to support non-TLS clients, `pport` is used to propagate TCP port across cluster nodes. However when "tls-cluster" is disabled, `port` is set to TCP port, and `pport` is not used, which means the cluster cannot provide TLS service unless "tls-cluster" is on.
```
typedef struct {
// ...
uint16_t port; /* Latest known clients port (TLS or plain). */
uint16_t pport; /* Latest known clients plaintext port. Only used if the main clients port is for TLS. */
// ...
} clusterNode;
```
```
typedef struct {
// ...
uint16_t port; /* TCP base port number. */
uint16_t pport; /* Sender TCP plaintext port, if base port is TLS */
// ...
} clusterMsg;
```
This PR renames `port` and `pport` in `clusterNode` to `tcp_port` and `tls_port`, to record both ports no matter "tls-cluster" is enabled or disabled.
This allows to provide TLS service to clients when "tls-cluster" is disabled: when displaying cluster topology, or giving `MOVED` error, server can provide TLS or TCP port according to client's connection type, no matter what type of connection cluster bus is using.
For backwards compatibility, `port` and `pport` in `clusterMsg` are preserved, when "tls-cluster" is enabled, `port` is set to TLS port and `pport` is set to TCP port, when "tls-cluster" is disabled, `port` is set to TCP port and `pport` is set to TLS port (instead of 0).
Also, in the nodes.conf file, a new aux field displaying an extra port is added to complete the persisted info. We may have `tls_port=xxxxx` or `tcp_port=xxxxx` in the aux field, to complete the cluster topology, while the other port is stored in the normal `<ip>:<port>` field. The format is shown below.
```
<node-id> <ip>:<tcp_port>@<cport>,<hostname>,shard-id=...,tls-port=6379 myself,master - 0 0 0 connected 0-1000
```
Or we can switch the position of two ports, both can be correctly resolved.
```
<node-id> <ip>:<tls_port>@<cport>,<hostname>,shard-id=...,tcp-port=6379 myself,master - 0 0 0 connected 0-1000
```
2023-06-26 22:43:38 +08:00
|
|
|
set port [srv 0 pport]
|
2021-01-29 00:17:39 +08:00
|
|
|
} else {
|
|
|
|
set expect_port [srv 0 port]
|
|
|
|
set expect_tls_port 0
|
Support TLS service when "tls-cluster" is not enabled and persist both plain and TLS port in nodes.conf (#12233)
Originally, when "tls-cluster" is enabled, `port` is set to TLS port. In order to support non-TLS clients, `pport` is used to propagate TCP port across cluster nodes. However when "tls-cluster" is disabled, `port` is set to TCP port, and `pport` is not used, which means the cluster cannot provide TLS service unless "tls-cluster" is on.
```
typedef struct {
// ...
uint16_t port; /* Latest known clients port (TLS or plain). */
uint16_t pport; /* Latest known clients plaintext port. Only used if the main clients port is for TLS. */
// ...
} clusterNode;
```
```
typedef struct {
// ...
uint16_t port; /* TCP base port number. */
uint16_t pport; /* Sender TCP plaintext port, if base port is TLS */
// ...
} clusterMsg;
```
This PR renames `port` and `pport` in `clusterNode` to `tcp_port` and `tls_port`, to record both ports no matter "tls-cluster" is enabled or disabled.
This allows to provide TLS service to clients when "tls-cluster" is disabled: when displaying cluster topology, or giving `MOVED` error, server can provide TLS or TCP port according to client's connection type, no matter what type of connection cluster bus is using.
For backwards compatibility, `port` and `pport` in `clusterMsg` are preserved, when "tls-cluster" is enabled, `port` is set to TLS port and `pport` is set to TCP port, when "tls-cluster" is disabled, `port` is set to TCP port and `pport` is set to TLS port (instead of 0).
Also, in the nodes.conf file, a new aux field displaying an extra port is added to complete the persisted info. We may have `tls_port=xxxxx` or `tcp_port=xxxxx` in the aux field, to complete the cluster topology, while the other port is stored in the normal `<ip>:<port>` field. The format is shown below.
```
<node-id> <ip>:<tcp_port>@<cport>,<hostname>,shard-id=...,tls-port=6379 myself,master - 0 0 0 connected 0-1000
```
Or we can switch the position of two ports, both can be correctly resolved.
```
<node-id> <ip>:<tls_port>@<cport>,<hostname>,shard-id=...,tcp-port=6379 myself,master - 0 0 0 connected 0-1000
```
2023-06-26 22:43:38 +08:00
|
|
|
set port [srv 0 port]
|
2021-01-29 00:17:39 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
assert_equal "$::host:$port" [lindex $cmdline 2]
|
|
|
|
assert_equal $expect_port [lindex $cmdline 3]
|
|
|
|
assert_equal $expect_tls_port [lindex $cmdline 4]
|
|
|
|
assert_match "*/tests/tmp/server.*/socket" [lindex $cmdline 5]
|
|
|
|
assert_match "*/tests/tmp/redis.conf.*" [lindex $cmdline 6]
|
|
|
|
|
|
|
|
# Try setting a bad template
|
|
|
|
catch {r config set "proc-title-template" "{invalid-var}"} err
|
|
|
|
assert_match {*template format is invalid*} $err
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
start_cluster 1 0 {tags {"other external:skip cluster slow"}} {
|
2024-01-30 20:32:38 +08:00
|
|
|
r config set dynamic-hz no hz 500
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
test "Redis can trigger resizing" {
|
|
|
|
r flushall
|
|
|
|
# hashslot(foo) is 12182
|
|
|
|
for {set j 1} {$j <= 128} {incr j} {
|
|
|
|
r set "{foo}$j" a
|
|
|
|
}
|
|
|
|
assert_match "*table size: 128*" [r debug HTSTATS 0]
|
|
|
|
|
2024-02-08 22:39:58 +08:00
|
|
|
# disable resizing, the reason for not using slow bgsave is because
|
|
|
|
# it will hit the dict_force_resize_ratio.
|
|
|
|
r debug dict-resizing 0
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
|
Change the threshold of dict expand, shrink and rehash (#12948)
Before this change (most recently modified in
https://github.com/redis/redis/pull/12850#discussion_r1421406393), The
trigger for normal expand threshold was 100% utilization and the trigger
for normal shrink threshold was 10% (HASHTABLE_MIN_FILL).
While during fork (DICT_RESIZE_AVOID), when we want to avoid rehash, the
trigger thresholds were multiplied by 5 (`dict_force_resize_ratio`),
meaning 500% for expand and 2% (100/10/5) for shrink.
However, in `dictRehash` (the incremental rehashing), the rehashing
threshold for shrinking during fork (DICT_RESIZE_AVOID) was 20% by
mistake.
This meant that if a shrinking is triggered when `dict_can_resize` is
`DICT_RESIZE_ENABLE` which the threshold is 10%, the rehashing can
continue when `dict_can_resize` is `DICT_RESIZE_AVOID`.
This would cause unwanted CopyOnWrite damage.
It'll make sense to change the thresholds of the rehash trigger and the
thresholds of the incremental rehashing the same, however, in one we
compare the size of the hash table to the number of records, and in the
other we compare the size of ht[0] to the size of ht[1], so the formula
is not exactly the same.
to make things easier we change all the thresholds to powers of 2, so
the normal shrinking threshold is changed from 100/10 (i.e. 10%) to
100/8 (i.e. 12.5%), and we change the threshold during forks from 5 to
4, i.e. from 500% to 400% for expand, and from 2% (100/10/5) to 3.125%
(100/8/4)
2024-01-19 23:00:43 +08:00
|
|
|
# delete data to have lot's (96%) of empty buckets
|
|
|
|
for {set j 1} {$j <= 123} {incr j} {
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
r del "{foo}$j"
|
|
|
|
}
|
|
|
|
assert_match "*table size: 128*" [r debug HTSTATS 0]
|
|
|
|
|
|
|
|
# enable resizing
|
2024-02-08 22:39:58 +08:00
|
|
|
r debug dict-resizing 1
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
|
2024-01-30 20:32:38 +08:00
|
|
|
# waiting for serverCron to resize the tables
|
|
|
|
wait_for_condition 1000 10 {
|
|
|
|
[string match {*table size: 8*} [r debug HTSTATS 0]]
|
|
|
|
} else {
|
|
|
|
puts [r debug HTSTATS 0]
|
|
|
|
fail "hash tables weren't resize."
|
|
|
|
}
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
} {} {needs:debug}
|
|
|
|
|
|
|
|
test "Redis can rewind and trigger smaller slot resizing" {
|
2024-01-17 14:46:09 +08:00
|
|
|
# hashslot(foo) is 12182
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
# hashslot(alice) is 749, smaller than hashslot(foo),
|
|
|
|
# attempt to trigger a resize on it, see details in #12802.
|
|
|
|
for {set j 1} {$j <= 128} {incr j} {
|
|
|
|
r set "{alice}$j" a
|
|
|
|
}
|
2024-01-17 14:46:09 +08:00
|
|
|
|
2024-02-08 22:39:58 +08:00
|
|
|
# disable resizing, the reason for not using slow bgsave is because
|
|
|
|
# it will hit the dict_force_resize_ratio.
|
|
|
|
r debug dict-resizing 0
|
2024-01-17 14:46:09 +08:00
|
|
|
|
Change the threshold of dict expand, shrink and rehash (#12948)
Before this change (most recently modified in
https://github.com/redis/redis/pull/12850#discussion_r1421406393), The
trigger for normal expand threshold was 100% utilization and the trigger
for normal shrink threshold was 10% (HASHTABLE_MIN_FILL).
While during fork (DICT_RESIZE_AVOID), when we want to avoid rehash, the
trigger thresholds were multiplied by 5 (`dict_force_resize_ratio`),
meaning 500% for expand and 2% (100/10/5) for shrink.
However, in `dictRehash` (the incremental rehashing), the rehashing
threshold for shrinking during fork (DICT_RESIZE_AVOID) was 20% by
mistake.
This meant that if a shrinking is triggered when `dict_can_resize` is
`DICT_RESIZE_ENABLE` which the threshold is 10%, the rehashing can
continue when `dict_can_resize` is `DICT_RESIZE_AVOID`.
This would cause unwanted CopyOnWrite damage.
It'll make sense to change the thresholds of the rehash trigger and the
thresholds of the incremental rehashing the same, however, in one we
compare the size of the hash table to the number of records, and in the
other we compare the size of ht[0] to the size of ht[1], so the formula
is not exactly the same.
to make things easier we change all the thresholds to powers of 2, so
the normal shrinking threshold is changed from 100/10 (i.e. 10%) to
100/8 (i.e. 12.5%), and we change the threshold during forks from 5 to
4, i.e. from 500% to 400% for expand, and from 2% (100/10/5) to 3.125%
(100/8/4)
2024-01-19 23:00:43 +08:00
|
|
|
for {set j 1} {$j <= 123} {incr j} {
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
r del "{alice}$j"
|
|
|
|
}
|
|
|
|
|
2024-01-17 14:46:09 +08:00
|
|
|
# enable resizing
|
2024-02-08 22:39:58 +08:00
|
|
|
r debug dict-resizing 1
|
2024-01-17 14:46:09 +08:00
|
|
|
|
2024-01-30 20:32:38 +08:00
|
|
|
# waiting for serverCron to resize the tables
|
|
|
|
wait_for_condition 1000 10 {
|
|
|
|
[string match {*table size: 16*} [r debug HTSTATS 0]]
|
|
|
|
} else {
|
|
|
|
puts [r debug HTSTATS 0]
|
|
|
|
fail "hash tables weren't resize."
|
|
|
|
}
|
Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .
The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:
/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
if (dbit->next_slot == -1) return NULL;
dbit->slot = dbit->next_slot;
dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
return dbGetDictFromIterator(dbit);
}
When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.
BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
|
|
|
} {} {needs:debug}
|
|
|
|
}
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
|
|
|
|
start_server {tags {"other external:skip"}} {
|
|
|
|
test "Redis can resize empty dict" {
|
|
|
|
# Write and then delete 128 keys, creating an empty dict
|
|
|
|
r flushall
|
Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.
# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.
# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).
**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 19:07:26 +08:00
|
|
|
|
|
|
|
# Add one key to the db just to create the dict and get its initial size
|
|
|
|
r set x 1
|
|
|
|
set initial_size [dict get [r memory stats] db.9 overhead.hashtable.main]
|
|
|
|
|
|
|
|
# Now add 128 keys and then delete them
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
for {set j 1} {$j <= 128} {incr j} {
|
|
|
|
r set $j{b} a
|
|
|
|
}
|
Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.
# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.
# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).
**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 19:07:26 +08:00
|
|
|
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
for {set j 1} {$j <= 128} {incr j} {
|
|
|
|
r del $j{b}
|
|
|
|
}
|
Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.
# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.
# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).
**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 19:07:26 +08:00
|
|
|
|
|
|
|
# dict must have expanded. Verify it eventually shrinks back to its initial size.
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
wait_for_condition 100 50 {
|
Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.
# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.
# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).
**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 19:07:26 +08:00
|
|
|
[dict get [r memory stats] db.9 overhead.hashtable.main] == $initial_size
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
} else {
|
Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.
# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.
# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).
**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 19:07:26 +08:00
|
|
|
fail "dict did not resize in time to its initial size"
|
|
|
|
}
|
Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.
What this PR does:
1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-30 03:02:07 +08:00
|
|
|
}
|
|
|
|
}
|
Cluster compatibility check (#13846)
### Background
The program runs normally in standalone mode, but migrating to cluster
mode may cause errors, this is because some cross slot commands can not
run in cluster mode. We should provide an approach to detect this issue
when running in standalone mode, and need to expose a metric which
indicates the usage of no incompatible commands.
### Solution
To avoid perf impact, we introduce a new config
`cluster-compatibility-sample-ratio` which define the sampling ratio
(0-100) for checking command compatibility in cluster mode. When a
command is executed, it is sampled at the specified ratio to determine
if it complies with Redis cluster constraints, such as cross-slot
restrictions.
A new metric is exposed: `cluster_incompatible_ops` in `info stats`
output.
The following operations will be considered incompatible operations.
- cross-slot command
If a command has multiple cross slot keys, it is incompatible
- `swap, copy, move, select` command
These commands involve multi databases in some cases, we don't allow
multiple DB in cluster mode, so there are not compatible
- Module command with `no-cluster` flag
If a module command has `no-cluster` flag, we will encounter an error
when loading module, leading to fail to load module if cluster is
enabled, so this is incompatible.
- Script/function with `no-cluster` flag
Similar with module command, if we declare `no-cluster` in shebang of
script/function, we also can not run it in cluster mode
- `sort` command by/get pattern
When `sort` command has `by/get` pattern option, we must ask that the
pattern slot is equal with the slot of keys, otherwise it is
incompatible in cluster mode.
- The script/function command accesses the keys and declared keys have
different slots
For the script/function command, we not only check the slot of declared
keys, but only check the slot the accessing keys, if they are different,
we think it is incompatible.
**Besides**, commands like `keys, scan, flushall, script/function
flush`, that in standalone mode iterate over all data to perform the
operation, are only valid for the server that executes the command in
cluster mode and are not broadcasted. However, this does not lead to
errors, so we do not consider them as incompatible commands.
### Performance impact test
**cross slot test**
Below are the test commands and results. When using MSET with 8 keys,
performance drops by approximately 3%.
**single key test**
It may be due to the overhead of the sampling function, and single-key
commands could cause a 1-2% performance drop.
2025-03-20 10:35:53 +08:00
|
|
|
|
|
|
|
start_server {tags {"other external:skip"} overrides {cluster-compatibility-sample-ratio 100}} {
|
|
|
|
test {Cross DB command is incompatible with cluster mode} {
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# SELECT with 0 is compatible command in cluster mode
|
|
|
|
assert_equal {OK} [r select 0]
|
|
|
|
assert_equal $incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# SELECT with nonzero is incompatible command in cluster mode
|
|
|
|
assert_equal {OK} [r select 1]
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# SWAPDB is incompatible command in cluster mode
|
|
|
|
assert_equal {OK} [r swapdb 0 1]
|
|
|
|
assert_equal [expr $incompatible_ops + 2] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
|
|
|
|
# If destination db in COPY command is equal to source db, it is compatible
|
|
|
|
# with cluster mode, otherwise it is incompatible.
|
|
|
|
r select 0
|
|
|
|
r set key1 value1
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
assert_equal {1} [r copy key1 key2{key1}] ;# destination db is equal to source db
|
|
|
|
assert_equal $incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
assert_equal {1} [r copy key2{key1} key1 db 1] ;# destination db is not equal to source db
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# If destination db in MOVE command is not equal to source db, it is incompatible
|
|
|
|
# with cluster mode.
|
|
|
|
r set key3 value3
|
|
|
|
assert_equal {1} [r move key3 1]
|
|
|
|
assert_equal [expr $incompatible_ops + 2] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Function no-cluster flag is incompatible with cluster mode} {
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# no-cluster flag is incompatible with cluster mode
|
|
|
|
r function load {#!lua name=test
|
|
|
|
redis.register_function{function_name='f1', callback=function() return 'hello' end, flags={'no-cluster'}}
|
|
|
|
}
|
|
|
|
r fcall f1 0
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# It is compatible without no-cluster flag, should not increase the cluster_incompatible_ops
|
|
|
|
r function load {#!lua name=test2
|
|
|
|
redis.register_function{function_name='f2', callback=function() return 'hello' end}
|
|
|
|
}
|
|
|
|
r fcall f2 0
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Script no-cluster flag is incompatible with cluster mode} {
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# no-cluster flag is incompatible with cluster mode
|
|
|
|
r eval {#!lua flags=no-cluster
|
|
|
|
return 1
|
|
|
|
} 0
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# It is compatible without no-cluster flag, should not increase the cluster_incompatible_ops
|
|
|
|
r eval {#!lua
|
|
|
|
return 1
|
|
|
|
} 0
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {SORT command incompatible operations with cluster mode} {
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# If the BY pattern slot is not equal with the slot of keys, we consider
|
|
|
|
# an incompatible behavior, otherwise it is compatible, should not increase
|
|
|
|
# the cluster_incompatible_ops
|
|
|
|
r lpush mylist 1 2 3
|
|
|
|
for {set i 1} {$i < 4} {incr i} {
|
|
|
|
r set weight_$i [expr 4 - $i]
|
|
|
|
}
|
|
|
|
assert_equal {3 2 1} [r sort mylist BY weight_*]
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
# weight{mylist}_* and mylist have the same slot
|
|
|
|
for {set i 1} {$i < 4} {incr i} {
|
|
|
|
r set weight{mylist}_$i [expr 4 - $i]
|
|
|
|
}
|
|
|
|
assert_equal {3 2 1} [r sort mylist BY weight{mylist}_*]
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# If the GET pattern slot is not equal with the slot of keys, we consider
|
|
|
|
# an incompatible behavior, otherwise it is compatible, should not increase
|
|
|
|
# the cluster_incompatible_ops
|
|
|
|
for {set i 1} {$i < 4} {incr i} {
|
|
|
|
r set object_$i o_$i
|
|
|
|
}
|
|
|
|
assert_equal {o_3 o_2 o_1} [r sort mylist BY weight{mylist}_* GET object_*]
|
|
|
|
assert_equal [expr $incompatible_ops + 2] [s cluster_incompatible_ops]
|
|
|
|
# object{mylist}_*, weight{mylist}_* and mylist have the same slot
|
|
|
|
for {set i 1} {$i < 4} {incr i} {
|
|
|
|
r set object{mylist}_$i o_$i
|
|
|
|
}
|
|
|
|
assert_equal {o_3 o_2 o_1} [r sort mylist BY weight{mylist}_* GET object{mylist}_*]
|
|
|
|
assert_equal [expr $incompatible_ops + 2] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Normal cross slot commands are incompatible with cluster mode} {
|
|
|
|
# Normal cross slot command
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r mset foo bar bar foo
|
|
|
|
r del foo bar
|
|
|
|
assert_equal [expr $incompatible_ops + 2] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Transaction is incompatible with cluster mode} {
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# Incomplete transaction
|
|
|
|
catch {r EXEC}
|
|
|
|
r multi
|
|
|
|
r exec
|
|
|
|
assert_equal $incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# Transaction, SET and DEL have keys with different slots
|
|
|
|
r multi
|
|
|
|
r set foo bar
|
|
|
|
r del bar
|
|
|
|
r exec
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Lua scripts are incompatible with cluster mode} {
|
|
|
|
# Lua script, declared keys have different slots, it is not a compatible operation
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r eval {#!lua
|
|
|
|
redis.call('mset', KEYS[1], 0, KEYS[2], 0)
|
|
|
|
} 2 foo bar
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# Lua script, no declared keys, but accessing keys have different slots,
|
|
|
|
# it is not a compatible operation
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r eval {#!lua
|
|
|
|
redis.call('mset', 'foo', 0, 'bar', 0)
|
|
|
|
} 0
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# Lua script, declared keys have the same slot, but accessing keys
|
|
|
|
# have different slots in one command, even with flag 'allow-cross-slot-keys',
|
|
|
|
# it still is not a compatible operation
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r eval {#!lua flags=allow-cross-slot-keys
|
|
|
|
redis.call('mset', 'foo', 0, 'bar', 0)
|
|
|
|
redis.call('mset', KEYS[1], 0, KEYS[2], 0)
|
|
|
|
} 2 foo bar{foo}
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# Lua script, declared keys have the same slot, but accessing keys have different slots
|
|
|
|
# in multiple commands, and with flag 'allow-cross-slot-keys', it is a compatible operation
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r eval {#!lua flags=allow-cross-slot-keys
|
|
|
|
redis.call('set', 'foo', 0)
|
|
|
|
redis.call('set', 'bar', 0)
|
|
|
|
redis.call('mset', KEYS[1], 0, KEYS[2], 0)
|
|
|
|
} 2 foo bar{foo}
|
|
|
|
assert_equal $incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {Shard subscribe commands are incompatible with cluster mode} {
|
|
|
|
set rd1 [redis_deferring_client]
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
assert_equal {1 2} [ssubscribe $rd1 {foo bar}]
|
|
|
|
assert_equal [expr $incompatible_ops + 1] [s cluster_incompatible_ops]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
|
|
|
|
test {cluster-compatibility-sample-ratio configuration can work} {
|
|
|
|
# Disable cluster compatibility sampling, no increase in cluster_incompatible_ops
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r config set cluster-compatibility-sample-ratio 0
|
|
|
|
for {set i 0} {$i < 100} {incr i} {
|
|
|
|
r mset foo bar$i bar foo$i
|
|
|
|
}
|
|
|
|
# Enable cluster compatibility sampling again to show the metric
|
|
|
|
r config set cluster-compatibility-sample-ratio 1
|
|
|
|
assert_equal $incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# 100% sample ratio, all operations should increase cluster_incompatible_ops
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r config set cluster-compatibility-sample-ratio 100
|
|
|
|
for {set i 0} {$i < 100} {incr i} {
|
|
|
|
r mset foo bar$i bar foo$i
|
|
|
|
}
|
|
|
|
assert_equal [expr $incompatible_ops + 100] [s cluster_incompatible_ops]
|
|
|
|
|
|
|
|
# 30% sample ratio, cluster_incompatible_ops should increase between 20% and 40%
|
|
|
|
set incompatible_ops [s cluster_incompatible_ops]
|
|
|
|
r config set cluster-compatibility-sample-ratio 30
|
|
|
|
for {set i 0} {$i < 1000} {incr i} {
|
|
|
|
r mset foo bar$i bar foo$i
|
|
|
|
}
|
|
|
|
assert_range [s cluster_incompatible_ops] [expr $incompatible_ops + 200] [expr $incompatible_ops + 400]
|
|
|
|
} {} {cluster:skip}
|
|
|
|
}
|