# Description
Add optimistic locking for string objects via compare-and-set and
compare-and-delete mechanism.
## What's changed
Introduction of new DIGEST command for string objects calculated via
XXH3 hash.
Extend SET command with new parameters supporting optimistic locking.
The new value is set only if checks against a given (old) value or a
given string digest pass.
Introduction of new DELEX command to support conditionally deleting a
key. Conditions are also checks against string value or string digest.
## Motivation
For developers who need to to implement a compare-and-set and
compare-and-delete single-key optimistic concurrency control this PR
provides single-command based implementation.
Compare-and-set and compare-and-delete are mostly used for [Optimistic
concurrency
control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control):
a client (1) fetches the value, keeps the old value (or its digest, for
a large string) in memory, (2) manipulates a local copy of the value,
(3) applies the local changes to the server, but only if the server’s
value hasn’t been changed (still equal to the old value).
Note that compare-and-set [can also be
implemented](https://redis.io/docs/latest/develop/using-commands/transactions/#optimistic-locking-using-check-and-set)
with WATCH … MULTI … EXEC and Lua scripts. The new SET optional
arguments and the DELEX command do not enable new functionality,
however, they are much simpler and faster to use for the very common use
case of single-key optimistic concurrency control.
## Related issues and PRs
https://github.com/redis/redis/issues/12485https://github.com/redis/redis/pull/8361https://github.com/redis/redis/pull/4258
## Description of the new commands
### DIGEST
```
DIGEST key
```
Get the hash digest of the value stored in key, as an hex string.
Reply:
- Null if key does not exist
- error if key exists but holds a value which is not a string
- (bulk string) the XXH3 digest of the value stored in key, as an hex
string
### SET
```
SET key value [NX | XX | IFEQ match-value | IFNE match-value | IFDEQ match-digest | IFDNE match-digest] [GET] [EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds | KEEPTTL]
```
`IFEQ match-value` - Set the key’s value and expiration only if its
current value is equal to match-value. If key doesn’t exist - it won’t
be created.
`IFNE match-value` - Set the key’s value and expiration only if its
current value is not equal to match-value. If key doesn’t exist - it
will be created.
`IFDEQ match-digest` - Set the key’s value and expiration only if the
digest of its current value is equal to match-digest. If key doesn’t
exist - it won’t be created.
`IFDNE match-digest` - Set the key’s value and expiration only if the
digest of its current value is not equal to match-digest. If key doesn’t
exist - it will be created.
Reply update:
- If GET was not specified:
- Nil reply if either
- the key doesn’t exist and XX/IFEQ/IFDEQ was specified. The key was not
created.
- the key exists, and NX was specified or a specified
IFEQ/IFNE/IFDEQ/IFDNE condition is false. The key was not set.
- Simple string reply: OK: The key was set.
- If GET was specified, any of the following:
- Nil reply: The key didn't exist before this command (whether the key
was created or not).
- Bulk string reply: The previous value of the key (whether the key was
set or not).
### DELEX
```
DELEX key [IFEQ match-value | IFNE match-value | IFDEQ match-digest | IFDNE match-digest]
```
Conditionally removes the specified key. A key is ignored if it does not
exist.
`IFEQ match-value` - Delete the key only if its value is equal to
match-value
`IFNE match-value` - Delete the key only if its value is not equal to
match-value
`IFDEQ match-digest` - Delete the key only if the digest of its value is
equal to match-digest
`IFDNE match-digest` - Delete the key only if the digest of its value is
not equal to match-digest
Reply:
- error if key exists but holds a value that is not a string and
IFEQ/IFNE/IFDEQ/IFDNE is specified.
- (integer) 0 if not deleted (the key does not exist or a specified
IFEQ/IFNE/IFDEQ/IFDNE condition is false), or 1 if deleted.
### Notes
Added copy of xxhash repo to deps -
[version](c961fbe61a)
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Yuan Wang <wangyuancode@163.com>
CI / build-libc-malloc (push) Waiting to runDetails
CI / build-centos-jemalloc (push) Waiting to runDetails
CI / build-old-chain-jemalloc (push) Waiting to runDetails
Codecov / code-coverage (push) Waiting to runDetails
External Server Tests / test-external-standalone (push) Waiting to runDetails
External Server Tests / test-external-cluster (push) Waiting to runDetails
External Server Tests / test-external-nodebug (push) Waiting to runDetails
Spellcheck / Spellcheck (push) Waiting to runDetails
PR https://github.com/redis/redis/pull/13916 introduced a regression -
by overriding the `CFLAGS` and `LDFLAGS` variables for all of the
dependencies hiredis and fast_float lost some of their compiler/linker
flags.
This PR makes it so we can pass additional CFLAGS/LDFLAGS to hiredis,
without overriding them as it contains a bit more complex Makefile. As
for fast_float - passing CFLAGS/LDFLAGS from outside now doesn't break
the expected behavior.
The build step in the CI was changed so that the MacOS is now build with
TLS to catch such errors in the future.
## Description
Memory sanitizer (MSAN) is used to detect use-of-uninitialized memory
issues. While Address Sanitizer catches a wide range of memory safety
issues, it doesn't specifically detect uninitialized memory usage.
Therefore, Memory Sanitizer complements Address Sanitizer. This PR adds
MSAN run to the daily build, with the possibility of incorporating it
into the ci.yml workflow in the future if needed.
Changes in source files fix false-positive issues and they should not
introduce any runtime implications.
Note: Valgrind performs similar checks to both ASAN and MSAN but
sanitizers run significantly faster.
## Limitations
- Memory sanitizer is only supported by Clang.
- MSAN documentation states that all dependencies, including the
standard library, must be compiled with MSAN. However, it also mentions
there are interceptors for common libc functions, so compiling the
standard library with the MSAN flag is not strictly necessary.
Therefore, we are not compiling libc with MSAN.
---------
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Fixes#8825
We're using the fast_float library[1] in our (compiled-in)
floating-point fast_float_strtod implementation for faster and more
portable parsing of 64 decimal strings.
The single file fast_float.h is an amalgamation of the entire library,
which can be (re)generated with the amalgamate.py script (from the
fast_float repository) via the command:
```
python3 ./script/amalgamate.py --license=MIT > $REDIS_SRC/deps/fast_float/fast_float.h
```
[1]: https://github.com/fastfloat/fast_float
The used commit from fast_float library was the one from
https://github.com/fastfloat/fast_float/releases/tag/v3.10.1
---------
Co-authored-by: fcostaoliveira <filipe@redis.com>
* Fix integer overflows due to using wrong integer size.
* Add assertions / panic when overflow still happens.
* Deletion of dead code to avoid need to maintain it
* Some changes are not because of bugs, but rather paranoia.
* Improve cmsgpack and cjson test coverage.
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Apparently for large size classes Jemalloc allocate some extra
memory (can be up to 25% overhead for allocations of 16kb).
see https://github.com/jemalloc/jemalloc/issues/1098#issuecomment-1589870476
p.s. from Redis's perspective that looks like external fragmentation,
(i.e. allocated bytes will be low, and active pages bytes will be large)
which can cause active-defrag to eat CPU cycles in vain.
Some details about this mechanism we disable:
---------------------------------------------------------------
Disabling this mechanism only affects large allocations (above 16kb)
Not only that it isn't expected to cause any performance regressions,
it's actually recommended, unless you have a specific workload pattern
and hardware that benefit from this feature -- by default it's enabled and
adds address randomization to all large buffers, by over allocating 1 page
per large size class, and offsetting into that page to make the starting
address of the user buffer randomized. Workloads such as scientific
computation often handle multiple big matrixes at the same time, and the
randomization makes sure that the cacheline level accesses don't suffer
bad conflicts (when they all start from page-aligned addresses).
However the downsize is also quite noticeable, like you observed that extra
page per large size can cause memory overhead, plus the extra TLB entry.
The other factor is, hardware in the last few years started doing the
randomization at the hardware level, i.e. the address to cacheline mapping isn't
a direct mapping anymore. So there's debate to disable the randomization by default,
but we are still hesitant because when it matters, it could matter a lot, and having
it enabled by default limits that worst case behavior, even though it means the
majority of workloads suffers a regression.
So in short, it's safe and offers better performance in most cases.
Previously, jemalloc was explicitly configured to build in `gnu99` mode. As a result, `<stdatomic.h>` was presumed to be unavailable and never used.
This commit removes explicit build flags configuration and lets `autoconf` determine the supported build flags. In addition, we also no longer build C++ jemalloc code.
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
All commands / use cases that heavily rely on double to a string representation conversion,
(e.g. meaning take a double-precision floating-point number like 1.5 and return a string like "1.5" ),
could benefit from a performance boost by swapping snprintf(buf,len,"%.17g",value) by the
equivalent [fpconv_dtoa](https://github.com/night-shift/fpconv) or any other algorithm that ensures
100% coverage of conversion.
This is a well-studied topic and Projects like MongoDB. RedPanda, PyTorch leverage libraries
( fmtlib ) that use the optimized double to string conversion underneath.
The positive impact can be substantial. This PR uses the grisu2 approach ( grisu explained on
https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf section 5 ).
test suite changes:
Despite being compatible, in some cases it produces a different result from printf, and some tests
had to be adjusted.
one case is that `%.17g` (which means %e or %f which ever is shorter), chose to use `5000000000`
instead of 5e+9, which sounds like a bug?
In other cases, we changed TCL to compare numbers instead of strings to ignore minor rounding
issues (`expr 0.8 == 0.79999999999999999`)
* Support BUILD_TLS=module to be loaded as a module via config file or
command line. e.g. redis-server --loadmodule redis-tls.so
* Updates to redismodule.h to allow it to be used side by side with
server.h by defining REDISMODULE_CORE_MODULE
* Changes to server.h, redismodule.h and module.c to avoid repeated
type declarations (gcc 4.8 doesn't like these)
* Add a mechanism for non-ABI neutral modules (ones who include
server.h) to refuse loading if they detect not being built together with
redis (release.c)
* Fix wrong signature of RedisModuleDefragFunc, this could break
compilation of a module, but not the ABI
* Move initialization of listeners in server.c to be after loading
the modules
* Config TLS after initialization of listeners
* Init cluster after initialization of listeners
* Add TLS module to CI
* Fix a test suite race conditions:
Now that the listeners are initialized later, it's not sufficient to
wait for the PID message in the log, we need to wait for the "Server
Initialized" message.
* Fix issues with moduleconfigs test as a result from start_server
waiting for "Server Initialized"
* Fix issues with modules/infra test as a result of an additional module
present
Notes about Sentinel:
Sentinel can't really rely on the tls module, since it uses hiredis to
initiate connections and depends on OpenSSL (won't be able to use any
other connection modules for that), so it was decided that when TLS is
built as a module, sentinel does not support TLS at all.
This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
Example code of config in redis-tls.so(may be use in the future):
RedisModuleString *tls_cfg = NULL;
void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
UNUSED(for_crash_report);
RedisModule_InfoAddSection(ctx, "");
RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
}
int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
if (argc != 2) return RedisModule_WrongArity(ctx);
return RedisModule_ReplyWithString(ctx, argv[1]);
}
RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
REDISMODULE_NOT_USED(name);
REDISMODULE_NOT_USED(privdata);
return tls_cfg;
}
int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
REDISMODULE_NOT_USED(name);
REDISMODULE_NOT_USED(err);
REDISMODULE_NOT_USED(privdata);
if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
RedisModule_RetainString(NULL, new);
tls_cfg = new;
return REDISMODULE_OK;
}
int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
{
....
if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
if (tls_cfg) {
RedisModule_FreeString(ctx, tls_cfg);
tls_cfg = NULL;
}
return REDISMODULE_ERR;
}
...
}
Co-authored-by: zhenwei pi <pizhenwei@bytedance.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
msgpack lib missed using lua_checkstack and so on rare
cases overflow the stack by at most 2 elements. This is a
violation of the Lua C API. Notice that Lua allocates
additional 5 more elements on top of lua->stack_last
so Redis does not access an invalid memory. But it is an
API violation and we should avoid it.
This PR also added a new Lua compilation option. The new
option can be enable using environment variable called
LUA_DEBUG. If set to `yes` (by default `no`), Lua will be
compiled without optimizations and with debug symbols (`-O0 -g`).
In addition, in this new mode, Lua will be compiled with the
`-DLUA_USE_APICHECK` flag that enables extended Lua C API
validations.
In addition, set LUA_DEBUG=yes on daily valgrind flow so we
will be able to catch Lua C API violations in the future.
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default.
All it does is avoid updating unchanged objects (shouldn't have any impact on our build)
- Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead
code in redis anyway).
- Remove unused variable `c` in `f_parser()`
- Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()`
Co-authored-by: Oran Agra <oran@redislabs.com>
A first step to enable a consistent full percentile analysis on query latency so that we can fully understand the performance and stability characteristics of the redis-server system we are measuring. It also improves the instantaneous reported metrics, and the csv output format.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
They were under /deps since they originate from a different source tree,
however at this point they are very modified and we took ownership of
both the files making changes, fixing bugs, so there is no upgrade path
from the original code tree.
Given that, better to move the code under /src with proper dependencies
and with a more simpler editing experience.
This change is documented in deps/README.md but was lost in one way or
the other, neutralizing the benefits of 24 bytes size classes (and
others).
Close#3208.
Current todo:
- replace functions in zset.{c,h} with a new unified Redis
zset access API.
Once we get the zset interface fixed, we can squash
relevant commits in this branch and have one nice commit
to merge into unstable.
This commit adds:
- Geo commands
- Tests; runnable with: ./runtest --single unit/geo
- Geo helpers in deps/geohash-int/
- src/geo.{c,h} and src/geojson.{c,h} implementing geo commands
- Updated build configurations to get everything working
- TEMPORARY: src/zset.{c,h} implementing zset score and zset
range reading without writing to client output buffers.
- Modified linkage of one t_zset.c function for use in zset.c
Conflicts:
src/Makefile
src/redis.c
The new cjson has some improvements over our current version including
increased platform compatability, a new resource limit to restrict
decode depth, and better invalid number handling.
One minor change was required to deps/Makefile because this version
of cjson doesn't export itself globally, so we added a quick little
define of -DENABLE_CJSON_GLOBAL.
cjson now has an optional higher performing float parsing interface,
but we are not including it (g_fmt.c, dtoa.c) because it requires
endianness declaration during compile time.
This commit is exactly lua_cjson.c from 2.1.0 with one minor
change of altering the two Lua includes for local search
instead of system-wide importing.
It failed because of the way jemalloc was compiled (without passing the
right flags to make, but just to configure). Now the same set of flags
are also passed to the make command, fixing the issue.
This fixes issue #744
Remove unused variables. Instead of overriding non-standard variables
such as ARCH and PROF, use standard variables CFLAGS and LDFLAGS to
override Makefile settings. Move dependencies generated by `make dep` to
a separate file.
This change moves the build instructions for dependencies to a separate
Makefile in deps/. The ARCH environment variable is stored in a
.make-arch file in the same directory as the Makefile. The contents of
this file is read and compared to the current ARCH, and, on a mismatch
triggers rebuilding the entire source tree.
When file .make-arch exists and matches with ARCH from the environment,
the dependencies are assumed to already be built.
The new "clean" target only cleans the Redis source tree, not its
dependencies. To clear the dependencies as well, the "distclean" target
can be used.
Mincho Paskalev
Filipe Oliveira (Personal)
Oran Agra
Florian Weimer
Meir Shpilraien (Spielrein)
Yossi Gottlieb
Yoav Steinberg
YoongHM
antirez
Matt Stancliff
Daniel Price
Johan Bergström
Pieter Noordhuis