root
/
redis
mirror of https://github.com/redis/redis.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/unstable' into HEAD
CI / test-ubuntu-latest (push) Has been cancelled Details
CI / test-sanitizer-address (push) Has been cancelled Details
CI / build-debian-old (push) Has been cancelled Details
CI / build-macos-latest (push) Has been cancelled Details
CI / build-32bit (push) Has been cancelled Details
CI / build-libc-malloc (push) Has been cancelled Details
CI / build-centos-jemalloc (push) Has been cancelled Details
CI / build-old-chain-jemalloc (push) Has been cancelled Details
Codecov / code-coverage (push) Has been cancelled Details
External Server Tests / test-external-standalone (push) Has been cancelled Details
External Server Tests / test-external-cluster (push) Has been cancelled Details
External Server Tests / test-external-nodebug (push) Has been cancelled Details
Spellcheck / Spellcheck (push) Has been cancelled Details

This commit is contained in:
YaacovHazan 2025-05-06 21:40:34 +03:00
parent e91a340e24 a3f1d09a7d
commit 7bc6ff3442
14 changed files with 1005 additions and 3642 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
LICENSE.txt
View File

@ -745,17 +745,15 @@ exclusive jurisdiction for all purposes relating to this Agreement.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
@ -764,44 +762,34 @@ them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
@ -810,7 +798,7 @@ modification follow.
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
@ -1287,35 +1275,45 @@ to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
@ -1357,3 +1355,45 @@ Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

2
modules/redisbloom/Makefile
View File

@ -1,5 +1,5 @@
SRC_DIR = src
MODULE_VERSION = v7.99.90
MODULE_VERSION = v8.0.1
MODULE_REPO = https://github.com/redisbloom/redisbloom
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/redisbloom.so

2
modules/redisearch/Makefile
View File

@ -1,5 +1,5 @@
SRC_DIR = src
MODULE_VERSION = v8.0.0
MODULE_VERSION = v8.0.1
MODULE_REPO = https://github.com/redisearch/redisearch
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/search-community/redisearch.so

2
modules/redisjson/Makefile
View File

@ -1,5 +1,5 @@
SRC_DIR = src
MODULE_VERSION = v7.99.90
MODULE_VERSION = v8.0.1
MODULE_REPO = https://github.com/redisjson/redisjson
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/rejson.so

2
modules/redistimeseries/Makefile
View File

@ -1,5 +1,5 @@
SRC_DIR = src
MODULE_VERSION = v7.99.91
MODULE_VERSION = v8.0.1
MODULE_REPO = https://github.com/redistimeseries/redistimeseries
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/redistimeseries.so

5
modules/vector-sets/Makefile
View File

@ -55,7 +55,7 @@ all: vset.so
vset.xo: ../../src/redismodule.h expr.c
vset.so: vset.xo hnsw.xo cJSON.xo
vset.so: vset.xo hnsw.xo
$(CC) -o $@ $^ $(SHOBJ_LDFLAGS) $(LIBS) $(SAN) -lc
# Example sources / objects
@ -76,6 +76,9 @@ $(TARGET): $(OBJS)
%.o: %.c
$(CC) $(CFLAGS) -c $< -o $@
expr-test: expr.c fastjson.c fastjson_test.c
$(CC) $(CFLAGS) expr.c -o expr-test -DTEST_MAIN -lm
# Clean rule
clean:
rm -f $(TARGET) $(OBJS) *.xo *.so

3164
modules/vector-sets/cJSON.c
View File

File diff suppressed because it is too large Load Diff

306
modules/vector-sets/cJSON.h
View File

@ -1,306 +0,0 @@
/*
Copyright (c) 2009-2017 Dave Gamble and cJSON contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/
#ifndef cJSON__h
#define cJSON__h
#ifdef __cplusplus
extern "C"
{
#endif
#if !defined(__WINDOWS__) && (defined(WIN32) || defined(WIN64) || defined(_MSC_VER) || defined(_WIN32))
#define __WINDOWS__
#endif
#ifdef __WINDOWS__
/* When compiling for windows, we specify a specific calling convention to avoid issues where we are being called from a project with a different default calling convention. For windows you have 3 define options:
CJSON_HIDE_SYMBOLS - Define this in the case where you don't want to ever dllexport symbols
CJSON_EXPORT_SYMBOLS - Define this on library build when you want to dllexport symbols (default)
CJSON_IMPORT_SYMBOLS - Define this if you want to dllimport symbol
For *nix builds that support visibility attribute, you can define similar behavior by
setting default visibility to hidden by adding
-fvisibility=hidden (for gcc)
or
-xldscope=hidden (for sun cc)
to CFLAGS
then using the CJSON_API_VISIBILITY flag to "export" the same symbols the way CJSON_EXPORT_SYMBOLS does
*/
#define CJSON_CDECL __cdecl
#define CJSON_STDCALL __stdcall
/* export symbols by default, this is necessary for copy pasting the C and header file */
#if !defined(CJSON_HIDE_SYMBOLS) && !defined(CJSON_IMPORT_SYMBOLS) && !defined(CJSON_EXPORT_SYMBOLS)
#define CJSON_EXPORT_SYMBOLS
#endif
#if defined(CJSON_HIDE_SYMBOLS)
#define CJSON_PUBLIC(type) type CJSON_STDCALL
#elif defined(CJSON_EXPORT_SYMBOLS)
#define CJSON_PUBLIC(type) __declspec(dllexport) type CJSON_STDCALL
#elif defined(CJSON_IMPORT_SYMBOLS)
#define CJSON_PUBLIC(type) __declspec(dllimport) type CJSON_STDCALL
#endif
#else /* !__WINDOWS__ */
#define CJSON_CDECL
#define CJSON_STDCALL
#if (defined(__GNUC__) || defined(__SUNPRO_CC) || defined (__SUNPRO_C)) && defined(CJSON_API_VISIBILITY)
#define CJSON_PUBLIC(type) __attribute__((visibility("default"))) type
#else
#define CJSON_PUBLIC(type) type
#endif
#endif
/* project version */
#define CJSON_VERSION_MAJOR 1
#define CJSON_VERSION_MINOR 7
#define CJSON_VERSION_PATCH 18
#include <stddef.h>
/* cJSON Types: */
#define cJSON_Invalid (0)
#define cJSON_False (1 << 0)
#define cJSON_True (1 << 1)
#define cJSON_NULL (1 << 2)
#define cJSON_Number (1 << 3)
#define cJSON_String (1 << 4)
#define cJSON_Array (1 << 5)
#define cJSON_Object (1 << 6)
#define cJSON_Raw (1 << 7) /* raw json */
#define cJSON_IsReference 256
#define cJSON_StringIsConst 512
/* The cJSON structure: */
typedef struct cJSON
{
/* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */
struct cJSON *next;
struct cJSON *prev;
/* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */
struct cJSON *child;
/* The type of the item, as above. */
int type;
/* The item's string, if type==cJSON_String and type == cJSON_Raw */
char *valuestring;
/* writing to valueint is DEPRECATED, use cJSON_SetNumberValue instead */
int valueint;
/* The item's number, if type==cJSON_Number */
double valuedouble;
/* The item's name string, if this item is the child of, or is in the list of subitems of an object. */
char *string;
} cJSON;
typedef struct cJSON_Hooks
{
/* malloc/free are CDECL on Windows regardless of the default calling convention of the compiler, so ensure the hooks allow passing those functions directly. */
void *(CJSON_CDECL *malloc_fn)(size_t sz);
void (CJSON_CDECL *free_fn)(void *ptr);
} cJSON_Hooks;
typedef int cJSON_bool;
/* Limits how deeply nested arrays/objects can be before cJSON rejects to parse them.
* This is to prevent stack overflows. */
#ifndef CJSON_NESTING_LIMIT
#define CJSON_NESTING_LIMIT 1000
#endif
/* Limits the length of circular references can be before cJSON rejects to parse them.
* This is to prevent stack overflows. */
#ifndef CJSON_CIRCULAR_LIMIT
#define CJSON_CIRCULAR_LIMIT 10000
#endif
/* returns the version of cJSON as a string */
CJSON_PUBLIC(const char*) cJSON_Version(void);
/* Supply malloc, realloc and free functions to cJSON */
CJSON_PUBLIC(void) cJSON_InitHooks(cJSON_Hooks* hooks);
/* Memory Management: the caller is always responsible to free the results from all variants of cJSON_Parse (with cJSON_Delete) and cJSON_Print (with stdlib free, cJSON_Hooks.free_fn, or cJSON_free as appropriate). The exception is cJSON_PrintPreallocated, where the caller has full responsibility of the buffer. */
/* Supply a block of JSON, and this returns a cJSON object you can interrogate. */
CJSON_PUBLIC(cJSON *) cJSON_Parse(const char *value);
CJSON_PUBLIC(cJSON *) cJSON_ParseWithLength(const char *value, size_t buffer_length);
/* ParseWithOpts allows you to require (and check) that the JSON is null terminated, and to retrieve the pointer to the final byte parsed. */
/* If you supply a ptr in return_parse_end and parsing fails, then return_parse_end will contain a pointer to the error so will match cJSON_GetErrorPtr(). */
CJSON_PUBLIC(cJSON *) cJSON_ParseWithOpts(const char *value, const char **return_parse_end, cJSON_bool require_null_terminated);
CJSON_PUBLIC(cJSON *) cJSON_ParseWithLengthOpts(const char *value, size_t buffer_length, const char **return_parse_end, cJSON_bool require_null_terminated);
/* Render a cJSON entity to text for transfer/storage. */
CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item);
/* Render a cJSON entity to text for transfer/storage without any formatting. */
CJSON_PUBLIC(char *) cJSON_PrintUnformatted(const cJSON *item);
/* Render a cJSON entity to text using a buffered strategy. prebuffer is a guess at the final size. guessing well reduces reallocation. fmt=0 gives unformatted, =1 gives formatted */
CJSON_PUBLIC(char *) cJSON_PrintBuffered(const cJSON *item, int prebuffer, cJSON_bool fmt);
/* Render a cJSON entity to text using a buffer already allocated in memory with given length. Returns 1 on success and 0 on failure. */
/* NOTE: cJSON is not always 100% accurate in estimating how much memory it will use, so to be safe allocate 5 bytes more than you actually need */
CJSON_PUBLIC(cJSON_bool) cJSON_PrintPreallocated(cJSON *item, char *buffer, const int length, const cJSON_bool format);
/* Delete a cJSON entity and all subentities. */
CJSON_PUBLIC(void) cJSON_Delete(cJSON *item);
/* Returns the number of items in an array (or object). */
CJSON_PUBLIC(int) cJSON_GetArraySize(const cJSON *array);
/* Retrieve item number "index" from array "array". Returns NULL if unsuccessful. */
CJSON_PUBLIC(cJSON *) cJSON_GetArrayItem(const cJSON *array, int index);
/* Get item "string" from object. Case insensitive. */
CJSON_PUBLIC(cJSON *) cJSON_GetObjectItem(const cJSON * const object, const char * const string);
CJSON_PUBLIC(cJSON *) cJSON_GetObjectItemCaseSensitive(const cJSON * const object, const char * const string);
CJSON_PUBLIC(cJSON_bool) cJSON_HasObjectItem(const cJSON *object, const char *string);
/* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */
CJSON_PUBLIC(const char *) cJSON_GetErrorPtr(void);
/* Check item type and return its value */
CJSON_PUBLIC(char *) cJSON_GetStringValue(const cJSON * const item);
CJSON_PUBLIC(double) cJSON_GetNumberValue(const cJSON * const item);
/* These functions check the type of an item */
CJSON_PUBLIC(cJSON_bool) cJSON_IsInvalid(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsFalse(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsTrue(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsBool(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsNull(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsNumber(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsString(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsArray(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsObject(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsRaw(const cJSON * const item);
/* These calls create a cJSON item of the appropriate type. */
CJSON_PUBLIC(cJSON *) cJSON_CreateNull(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateTrue(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateFalse(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateBool(cJSON_bool boolean);
CJSON_PUBLIC(cJSON *) cJSON_CreateNumber(double num);
CJSON_PUBLIC(cJSON *) cJSON_CreateString(const char *string);
/* raw json */
CJSON_PUBLIC(cJSON *) cJSON_CreateRaw(const char *raw);
CJSON_PUBLIC(cJSON *) cJSON_CreateArray(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateObject(void);
/* Create a string where valuestring references a string so
* it will not be freed by cJSON_Delete */
CJSON_PUBLIC(cJSON *) cJSON_CreateStringReference(const char *string);
/* Create an object/array that only references it's elements so
* they will not be freed by cJSON_Delete */
CJSON_PUBLIC(cJSON *) cJSON_CreateObjectReference(const cJSON *child);
CJSON_PUBLIC(cJSON *) cJSON_CreateArrayReference(const cJSON *child);
/* These utilities create an Array of count items.
* The parameter count cannot be greater than the number of elements in the number array, otherwise array access will be out of bounds.*/
CJSON_PUBLIC(cJSON *) cJSON_CreateIntArray(const int *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateFloatArray(const float *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateDoubleArray(const double *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateStringArray(const char *const *strings, int count);
/* Append item to the specified array/object. */
CJSON_PUBLIC(cJSON_bool) cJSON_AddItemToArray(cJSON *array, cJSON *item);
CJSON_PUBLIC(cJSON_bool) cJSON_AddItemToObject(cJSON *object, const char *string, cJSON *item);
/* Use this when string is definitely const (i.e. a literal, or as good as), and will definitely survive the cJSON object.
* WARNING: When this function was used, make sure to always check that (item->type & cJSON_StringIsConst) is zero before
* writing to `item->string` */
CJSON_PUBLIC(cJSON_bool) cJSON_AddItemToObjectCS(cJSON *object, const char *string, cJSON *item);
/* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */
CJSON_PUBLIC(cJSON_bool) cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item);
CJSON_PUBLIC(cJSON_bool) cJSON_AddItemReferenceToObject(cJSON *object, const char *string, cJSON *item);
/* Remove/Detach items from Arrays/Objects. */
CJSON_PUBLIC(cJSON *) cJSON_DetachItemViaPointer(cJSON *parent, cJSON * const item);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromArray(cJSON *array, int which);
CJSON_PUBLIC(void) cJSON_DeleteItemFromArray(cJSON *array, int which);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObject(cJSON *object, const char *string);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObjectCaseSensitive(cJSON *object, const char *string);
CJSON_PUBLIC(void) cJSON_DeleteItemFromObject(cJSON *object, const char *string);
CJSON_PUBLIC(void) cJSON_DeleteItemFromObjectCaseSensitive(cJSON *object, const char *string);
/* Update array items. */
CJSON_PUBLIC(cJSON_bool) cJSON_InsertItemInArray(cJSON *array, int which, cJSON *newitem); /* Shifts pre-existing items to the right. */
CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemViaPointer(cJSON * const parent, cJSON * const item, cJSON * replacement);
CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemInArray(cJSON *array, int which, cJSON *newitem);
CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem);
CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemInObjectCaseSensitive(cJSON *object,const char *string,cJSON *newitem);
/* Duplicate a cJSON item */
CJSON_PUBLIC(cJSON *) cJSON_Duplicate(const cJSON *item, cJSON_bool recurse);
/* Duplicate will create a new, identical cJSON item to the one you pass, in new memory that will
* need to be released. With recurse!=0, it will duplicate any children connected to the item.
* The item->next and ->prev pointers are always zero on return from Duplicate. */
/* Recursively compare two cJSON items for equality. If either a or b is NULL or invalid, they will be considered unequal.
* case_sensitive determines if object keys are treated case sensitive (1) or case insensitive (0) */
CJSON_PUBLIC(cJSON_bool) cJSON_Compare(const cJSON * const a, const cJSON * const b, const cJSON_bool case_sensitive);
/* Minify a strings, remove blank characters(such as ' ', '\t', '\r', '\n') from strings.
* The input pointer json cannot point to a read-only address area, such as a string constant,
* but should point to a readable and writable address area. */
CJSON_PUBLIC(void) cJSON_Minify(char *json);
/* Helper functions for creating and adding items to an object at the same time.
* They return the added item or NULL on failure. */
CJSON_PUBLIC(cJSON*) cJSON_AddNullToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddTrueToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddFalseToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddBoolToObject(cJSON * const object, const char * const name, const cJSON_bool boolean);
CJSON_PUBLIC(cJSON*) cJSON_AddNumberToObject(cJSON * const object, const char * const name, const double number);
CJSON_PUBLIC(cJSON*) cJSON_AddStringToObject(cJSON * const object, const char * const name, const char * const string);
CJSON_PUBLIC(cJSON*) cJSON_AddRawToObject(cJSON * const object, const char * const name, const char * const raw);
CJSON_PUBLIC(cJSON*) cJSON_AddObjectToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddArrayToObject(cJSON * const object, const char * const name);
/* When assigning an integer value, it needs to be propagated to valuedouble too. */
#define cJSON_SetIntValue(object, number) ((object) ? (object)->valueint = (object)->valuedouble = (number) : (number))
/* helper for the cJSON_SetNumberValue macro */
CJSON_PUBLIC(double) cJSON_SetNumberHelper(cJSON *object, double number);
#define cJSON_SetNumberValue(object, number) ((object != NULL) ? cJSON_SetNumberHelper(object, (double)number) : (number))
/* Change the valuestring of a cJSON_String object, only takes effect when type of object is cJSON_String */
CJSON_PUBLIC(char*) cJSON_SetValuestring(cJSON *object, const char *valuestring);
/* If the object is not a boolean type this does nothing and returns cJSON_Invalid else it returns the new type*/
#define cJSON_SetBoolValue(object, boolValue) ( \
(object != NULL && ((object)->type & (cJSON_False|cJSON_True))) ? \
(object)->type=((object)->type &(~(cJSON_False|cJSON_True)))|((boolValue)?cJSON_True:cJSON_False) : \
cJSON_Invalid\
)
/* Macro for iterating over an array or object */
#define cJSON_ArrayForEach(element, array) for(element = (array != NULL) ? (array)->child : NULL; element != NULL; element = element->next)
/* malloc/free objects using the malloc/free functions that have been set with cJSON_InitHooks */
CJSON_PUBLIC(void *) cJSON_malloc(size_t size);
CJSON_PUBLIC(void) cJSON_free(void *object);
#ifdef __cplusplus
}
#endif
#endif

160
modules/vector-sets/expr.c
View File

@ -12,26 +12,32 @@
* Originally authored by: Salvatore Sanfilippo.
*/
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <math.h>
#include <string.h>
#include "cJSON.h"
#ifdef TEST_MAIN
#define RedisModule_Alloc malloc
#define RedisModule_Realloc realloc
#define RedisModule_Free free
#define RedisModule_Strdup strdup
#define RedisModule_Assert assert
#define _DEFAULT_SOURCE
#define _USE_MATH_DEFINES
#define _POSIX_C_SOURCE 200809L
#include <assert.h>
#include <math.h>
#endif
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <math.h>
#include <string.h>
#define EXPR_TOKEN_EOF 0
#define EXPR_TOKEN_NUM 1
#define EXPR_TOKEN_STR 2
#define EXPR_TOKEN_TUPLE 3
#define EXPR_TOKEN_SELECTOR 4
#define EXPR_TOKEN_OP 5
#define EXPR_TOKEN_NULL 6
#define EXPR_OP_OPAREN 0 /* ( */
#define EXPR_OP_CPAREN 1 /* ) */
@ -150,12 +156,7 @@ exprtoken *exprNewToken(int type) {
void exprTokenRelease(exprtoken *t) {
if (t == NULL) return;
if (t->refcount <= 0) {
printf("exprTokenRelease() against a token with refcount %d!\n"
"Aborting program execution\n",
t->refcount);
exit(1);
}
RedisModule_Assert(t->refcount > 0); // Catch double free & more.
t->refcount--;
if (t->refcount > 0) return;
@ -241,9 +242,10 @@ void exprConsumeSpaces(exprstate *es) {
while(es->p[0] && isspace(es->p[0])) es->p++;
}
/* Parse an operator, trying to match the longer match in the
* operators table. */
exprtoken *exprParseOperator(exprstate *es) {
/* Parse an operator or a literal (just "null" currently).
* When parsing operators, the function will try to match the longest match
* in the operators table. */
exprtoken *exprParseOperatorOrLiteral(exprstate *es) {
exprtoken *t = exprNewToken(EXPR_TOKEN_OP);
char *start = es->p;
@ -258,6 +260,12 @@ exprtoken *exprParseOperator(exprstate *es) {
int bestlen = 0;
int j;
// Check if it's a literal.
if (matchlen == 4 && !memcmp("null",start,4)) {
t->token_type = EXPR_TOKEN_NULL;
return t;
}
// Find the longest matching operator.
for (j = 0; ExprOptable[j].opname != NULL; j++) {
if (ExprOptable[j].oplen > matchlen) continue;
@ -302,7 +310,7 @@ exprtoken *exprParseSelector(exprstate *es) {
exprtoken *exprParseNumber(exprstate *es) {
exprtoken *t = exprNewToken(EXPR_TOKEN_NUM);
char num[64];
char num[256];
int idx = 0;
while(isdigit(es->p[0]) || es->p[0] == '.' || es->p[0] == 'e' ||
es->p[0] == 'E' || (idx == 0 && es->p[0] == '-'))
@ -468,10 +476,10 @@ int exprTokenize(exprstate *es, int *errpos) {
current = exprParseString(es);
} else if (*es->p == '.' && is_selector_char(es->p[1])) {
current = exprParseSelector(es);
} else if (isalpha(*es->p) || strchr(EXPR_OP_SPECIALCHARS, *es->p)) {
current = exprParseOperator(es);
} else if (*es->p == '[') {
current = exprParseTuple(es);
} else if (isalpha(*es->p) || strchr(EXPR_OP_SPECIALCHARS, *es->p)) {
current = exprParseOperatorOrLiteral(es);
}
if (current == NULL) {
@ -676,7 +684,7 @@ exprstate *exprCompile(char *expr, int *errpos) {
/* Convert a token to its numeric value. For strings we attempt to parse them
* as numbers, returning 0 if conversion fails. */
double exprTokenToNum(exprtoken *t) {
char buf[128];
char buf[256];
if (t->token_type == EXPR_TOKEN_NUM) {
return t->num;
} else if (t->token_type == EXPR_TOKEN_STR && t->str.len < sizeof(buf)) {
@ -696,6 +704,8 @@ double exprTokenToBool(exprtoken *t) {
return t->num != 0;
} else if (t->token_type == EXPR_TOKEN_STR && t->str.len == 0) {
return 0; // Empty string are false, like in Javascript.
} else if (t->token_type == EXPR_TOKEN_NULL) {
return 0; // Null is surely more false than true...
} else {
return 1; // Every non numerical type is true.
}
@ -714,77 +724,23 @@ int exprTokensEqual(exprtoken *a, exprtoken *b) {
return a->num == b->num;
}
/* If one of the two is null, the expression is true only if
* both are null. */
if (a->token_type == EXPR_TOKEN_NULL || b->token_type == EXPR_TOKEN_NULL) {
return a->token_type == b->token_type;
}
// Mixed types - convert to numbers and compare.
return exprTokenToNum(a) == exprTokenToNum(b);
}
/* Convert a json object to an expression token. There is only
* limited support for JSON arrays: they must be composed of
* just numbers and strings. Returns NULL if the JSON object
* cannot be converted. */
exprtoken *exprJsonToToken(cJSON *js) {
if (cJSON_IsNumber(js)) {
exprtoken *obj = exprNewToken(EXPR_TOKEN_NUM);
obj->num = cJSON_GetNumberValue(js);
return obj;
} else if (cJSON_IsString(js)) {
exprtoken *obj = exprNewToken(EXPR_TOKEN_STR);
char *strval = cJSON_GetStringValue(js);
obj->str.heapstr = RedisModule_Strdup(strval);
obj->str.start = obj->str.heapstr;
obj->str.len = strlen(obj->str.heapstr);
return obj;
} else if (cJSON_IsBool(js)) {
exprtoken *obj = exprNewToken(EXPR_TOKEN_NUM);
obj->num = cJSON_IsTrue(js);
return obj;
} else if (cJSON_IsArray(js)) {
// First, scan the array to ensure it only
// contains strings and numbers. Otherwise the
// expression will evaluate to false.
int array_size = cJSON_GetArraySize(js);
for (int j = 0; j < array_size; j++) {
cJSON *item = cJSON_GetArrayItem(js, j);
if (!cJSON_IsNumber(item) && !cJSON_IsString(item)) return NULL;
}
// Create a tuple token for the array.
exprtoken *obj = exprNewToken(EXPR_TOKEN_TUPLE);
obj->tuple.len = array_size;
obj->tuple.ele = NULL;
if (obj->tuple.len == 0) return obj; // No elements, already ok.
obj->tuple.ele =
RedisModule_Alloc(sizeof(exprtoken*) * obj->tuple.len);
// Convert each array element to a token.
for (size_t j = 0; j < obj->tuple.len; j++) {
cJSON *item = cJSON_GetArrayItem(js, j);
if (cJSON_IsNumber(item)) {
exprtoken *eleToken = exprNewToken(EXPR_TOKEN_NUM);
eleToken->num = cJSON_GetNumberValue(item);
obj->tuple.ele[j] = eleToken;
} else if (cJSON_IsString(item)) {
exprtoken *eleToken = exprNewToken(EXPR_TOKEN_STR);
char *strval = cJSON_GetStringValue(item);
eleToken->str.heapstr = RedisModule_Strdup(strval);
eleToken->str.start = eleToken->str.heapstr;
eleToken->str.len = strlen(eleToken->str.heapstr);
obj->tuple.ele[j] = eleToken;
}
}
return obj;
}
return NULL; // No conversion possible for this type.
}
#include "fastjson.c" // JSON parser implementation used by exprRun().
/* Execute the compiled expression program. Returns 1 if the final stack value
* evaluates to true, 0 otherwise. Also returns 0 if any selector callback
* fails. */
int exprRun(exprstate *es, char *json, size_t json_len) {
exprStackReset(&es->values_stack);
cJSON *parsed_json = NULL;
// Execute each instruction in the program.
for (int i = 0; i < es->program.numitems; i++) {
@ -792,35 +748,15 @@ int exprRun(exprstate *es, char *json, size_t json_len) {
// Handle selectors by calling the callback.
if (t->token_type == EXPR_TOKEN_SELECTOR) {
if (json != NULL) {
cJSON *attrib = NULL;
if (parsed_json == NULL) {
parsed_json = cJSON_ParseWithLength(json,json_len);
// Will be left to NULL if the above fails.
}
if (parsed_json) {
char item_name[128];
if (t->str.len > 0 && t->str.len < sizeof(item_name)) {
memcpy(item_name,t->str.start,t->str.len);
item_name[t->str.len] = 0;
attrib = cJSON_GetObjectItem(parsed_json,item_name);
}
/* Fill the token according to the JSON type stored
* at the attribute. */
if (attrib) {
exprtoken *obj = exprJsonToToken(attrib);
if (obj) {
exprStackPush(&es->values_stack, obj);
continue;
}
}
}
}
exprtoken *obj = NULL;
if (t->str.len > 0)
obj = jsonExtractField(json,json_len,t->str.start,t->str.len);
// Selector not found or JSON object not convertible to
// expression tokens. Evaluate the expression to false.
if (parsed_json) cJSON_Delete(parsed_json);
return 0;
if (obj == NULL) return 0;
exprStackPush(&es->values_stack, obj);
continue;
}
// Push non-operator values directly onto the stack.
@ -918,8 +854,6 @@ int exprRun(exprstate *es, char *json, size_t json_len) {
exprStackPush(&es->values_stack, result);
}
if (parsed_json) cJSON_Delete(parsed_json);
// Get final result from stack.
exprtoken *final = exprStackPop(&es->values_stack);
if (final == NULL) return 0;
@ -933,6 +867,8 @@ int exprRun(exprstate *es, char *json, size_t json_len) {
/* ============================ Simple test main ============================ */
#ifdef TEST_MAIN
#include "fastjson_test.c"
void exprPrintToken(exprtoken *t) {
switch(t->token_type) {
case EXPR_TOKEN_EOF:
@ -972,6 +908,12 @@ void exprPrintStack(exprstack *stack, const char *name) {
}
int main(int argc, char **argv) {
/* Check for JSON parser test mode. */
if (argc >= 2 && strcmp(argv[1], "--test-json-parser") == 0) {
run_fastjson_test();
return 0;
}
char *testexpr = "(5+2)*3 and .year > 1980 and 'foo' == 'foo'";
char *testjson = "{\"year\": 1984, \"name\": \"The Matrix\"}";
if (argc >= 2) testexpr = argv[1];

441
modules/vector-sets/fastjson.c Normal file
View File

@ -0,0 +1,441 @@
/* Ultra‑lightweight top‑level JSON field extractor.
* Return the element directly as an expr.c token.
* This code is directly included inside expr.c.
*
* Copyright (c) 2025-Present, Redis Ltd.
* All rights reserved.
*
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*
* Originally authored by: Salvatore Sanfilippo.
*
* ------------------------------------------------------------------
*
* DESIGN GOALS:
*
* 1. Zero heap allocations while seeking the requested key.
* 2. A single parse (and therefore a single allocation, if needed)
* when the key finally matches.
* 3. Same subset‑of‑JSON coverage needed by expr.c:
* - Strings (escapes: \" \\ \n \r \t).
* - Numbers (double).
* - Booleans.
* - Null.
* - Flat arrays of the above primitives.
*
* Any other value (nested object, unicode escape, etc.) returns NULL.
* Should be very easy to extend it in case in the future we want
* more for the FILTER option of VSIM.
* 4. No global state, so this file can be #included directly in expr.c.
*
* The only API expr.c uses directly is:
*
* exprtoken *jsonExtractField(const char *json, size_t json_len,
* const char *field, size_t field_len);
* ------------------------------------------------------------------ */
#include <ctype.h>
#include <string.h>
// Forward declarations.
static int jsonSkipValue(const char **p, const char *end);
static exprtoken *jsonParseValueToken(const char **p, const char *end);
/* Similar to ctype.h isdigit() but covers the whole JSON number charset,
* including exp form. */
static int jsonIsNumberChar(int c) {
return isdigit(c) || c=='-' || c=='+' || c=='.' || c=='e' || c=='E';
}
/* ========================== Fast skipping of JSON =========================
* The helpers here are designed to skip values without performing any
* allocation. This way, for the use case of this JSON parser, we are able
* to easily (and with good speed) skip fields and values we are not
* interested in. Then, later in the code, when we find the field we want
* to obtain, we finally call the functions that turn a given JSON value
* associated to a field into our of our expressions token.
* ========================================================================== */
/* Advance *p consuming all the spaces. */
static inline void jsonSkipWhiteSpaces(const char **p, const char *end) {
while (*p < end && isspace((unsigned char)**p)) (*p)++;
}
/* Advance *p past a JSON string. Returns 1 on success, 0 on error. */
static int jsonSkipString(const char **p, const char *end) {
if (*p >= end || **p != '"') return 0;
(*p)++; /* Skip opening quote. */
while (*p < end) {
if (**p == '\\') {
(*p) += 2;
continue;
}
if (**p == '"') {
(*p)++; /* Skip closing quote. */
return 1;
}
(*p)++;
}
return 0; /* unterminated */
}
/* Skip an array or object generically using depth counter.
* Opener and closer tells the function how the aggregated
* data type starts/stops, basically [] or {}. */
static int jsonSkipBracketed(const char **p, const char *end,
char opener, char closer) {
int depth = 1;
(*p)++; /* Skip opener. */
/* Loop until we reach the end of the input or find the matching
* closer (depth becomes 0). */
while (*p < end && depth > 0) {
char c = **p;
if (c == '"') {
// Found a string, delegate skipping to jsonSkipString().
if (!jsonSkipString(p, end)) {
return 0; // String skipping failed (e.g., unterminated)
}
/* jsonSkipString() advances *p past the closing quote.
* Continue the loop to process the character *after* the string. */
continue;
}
/* If it's not a string, check if it affects the depth for the
* specific brackets we are currently tracking. */
if (c == opener) {
depth++;
} else if (c == closer) {
depth--;
}
/* Always advance the pointer for any non-string character.
* This handles commas, colons, whitespace, numbers, literals,
* and even nested brackets of a *different* type than the
* one we are currently skipping (e.g. skipping a { inside []). */
(*p)++;
}
/* Return 1 (true) if we successfully found the matching closer,
* otherwise there is a parse error and we return 0. */
return depth == 0;
}
/* Skip a single JSON literal (true, null, ...) starting at *p.
* Returns 1 on success, 0 on failure. */
static int jsonSkipLiteral(const char **p, const char *end, const char *lit) {
size_t l = strlen(lit);
if (*p + l > end) return 0;
if (strncmp(*p, lit, l) == 0) { *p += l; return 1; }
return 0;
}
/* Skip number, don't check that number format is correct, just consume
* number-alike characters.
*
* Note: More robust number skipping might check validity,
* but for skipping, just consuming plausible characters is enough. */
static int jsonSkipNumber(const char **p, const char *end) {
const char *num_start = *p;
while (*p < end && jsonIsNumberChar(**p)) (*p)++;
return *p > num_start; // Any progress made? Otherwise no number found.
}
/* Skip any JSON value. 1 = success, 0 = error. */
static int jsonSkipValue(const char **p, const char *end) {
jsonSkipWhiteSpaces(p, end);
if (*p >= end) return 0;
switch (**p) {
case '"': return jsonSkipString(p, end);
case '{': return jsonSkipBracketed(p, end, '{', '}');
case '[': return jsonSkipBracketed(p, end, '[', ']');
case 't': return jsonSkipLiteral(p, end, "true");
case 'f': return jsonSkipLiteral(p, end, "false");
case 'n': return jsonSkipLiteral(p, end, "null");
default: return jsonSkipNumber(p, end);
}
}
/* =========================== JSON to exprtoken ============================
* The functions below convert a given json value to the equivalent
* expression token structure.
* ========================================================================== */
static exprtoken *jsonParseStringToken(const char **p, const char *end) {
if (*p >= end || **p != '"') return NULL;
const char *start = ++(*p);
int esc = 0; size_t len = 0; int has_esc = 0;
const char *q = *p;
while (q < end) {
if (esc) { esc = 0; q++; len++; has_esc = 1; continue; }
if (*q == '\\') { esc = 1; q++; continue; }
if (*q == '"') break;
q++; len++;
}
if (q >= end || *q != '"') return NULL; // Unterminated string
exprtoken *t = exprNewToken(EXPR_TOKEN_STR);
if (!has_esc) {
// No escapes, we can point directly into the original JSON string.
t->str.start = (char*)start; t->str.len = len; t->str.heapstr = NULL;
} else {
// Escapes present, need to allocate and copy/process escapes.
char *dst = RedisModule_Alloc(len + 1);
t->str.start = t->str.heapstr = dst; t->str.len = len;
const char *r = start; esc = 0;
while (r < q) {
if (esc) {
switch (*r) {
// Supported escapes from Goal 3.
case 'n': *dst='\n'; break;
case 'r': *dst='\r'; break;
case 't': *dst='\t'; break;
case '\\': *dst='\\'; break;
case '"': *dst='\"'; break;
// Escapes (like \uXXXX, \b, \f) are not supported for now,
// we just copy them verbatim.
default: *dst=*r; break;
}
dst++; esc = 0; r++; continue;
}
if (*r == '\\') { esc = 1; r++; continue; }
*dst++ = *r++;
}
*dst = '\0'; // Null-terminate the allocated string.
}
*p = q + 1; // Advance the main pointer past the closing quote.
return t;
}
static exprtoken *jsonParseNumberToken(const char **p, const char *end) {
// Use a buffer to extract the number literal for parsing with strtod().
char buf[256]; int idx = 0;
const char *start = *p; // For strtod partial failures check.
// Copy potential number characters to buffer.
while (*p < end && idx < (int)sizeof(buf)-1 && jsonIsNumberChar(**p)) {
buf[idx++] = **p;
(*p)++;
}
buf[idx]='\0'; // Null-terminate buffer.
if (idx==0) return NULL; // No number characters found.
char *ep; // End pointer for strtod validation.
double v = strtod(buf, &ep);
/* Check if strtod() consumed the entire buffer content.
* If not, the number format was invalid. */
if (*ep!='\0') {
// strtod() failed; rewind p to the start and return NULL
*p = start;
return NULL;
}
// If strtod() succeeded, create and return the token..
exprtoken *t = exprNewToken(EXPR_TOKEN_NUM);
t->num = v;
return t;
}
static exprtoken *jsonParseLiteralToken(const char **p, const char *end, const char *lit, int type, double num) {
size_t l = strlen(lit);
// Ensure we don't read past 'end'.
if ((*p + l) > end) return NULL;
if (strncmp(*p, lit, l) != 0) return NULL; // Literal doesn't match.
// Check that the character *after* the literal is a valid JSON delimiter
// (whitespace, comma, closing bracket/brace, or end of input)
// This prevents matching "trueblabla" as "true".
if ((*p + l) < end) {
char next_char = *(*p + l);
if (!isspace((unsigned char)next_char) && next_char!=',' &&
next_char!=']' && next_char!='}') {
return NULL; // Invalid character following literal.
}
}
// Literal matched and is correctly terminated.
*p += l;
exprtoken *t = exprNewToken(type);
t->num = num;
return t;
}
static exprtoken *jsonParseArrayToken(const char **p, const char *end) {
if (*p >= end || **p != '[') return NULL;
(*p)++; // Skip '['.
jsonSkipWhiteSpaces(p,end);
exprtoken *t = exprNewToken(EXPR_TOKEN_TUPLE);
t->tuple.len = 0; t->tuple.ele = NULL; size_t alloc = 0;
// Handle empty array [].
if (*p < end && **p == ']') {
(*p)++; // Skip ']'.
return t;
}
// Parse array elements.
while (1) {
exprtoken *ele = jsonParseValueToken(p,end);
if (!ele) {
exprTokenRelease(t); // Clean up partially built array token.
return NULL;
}
// Grow allocated space for elements if needed.
if (t->tuple.len == alloc) {
size_t newsize = alloc ? alloc * 2 : 4;
// Check for potential overflow if newsize becomes huge.
if (newsize < alloc) {
exprTokenRelease(ele);
exprTokenRelease(t);
return NULL;
}
exprtoken **newele = RedisModule_Realloc(t->tuple.ele,
sizeof(exprtoken*)*newsize);
t->tuple.ele = newele;
alloc = newsize;
}
t->tuple.ele[t->tuple.len++] = ele; // Add element.
jsonSkipWhiteSpaces(p,end);
if (*p>=end) {
// Unterminated array. Note that this check is crucial because
// previous value parsed may seek 'p' to 'end'.
exprTokenRelease(t);
return NULL;
}
// Check for comma (more elements) or closing bracket.
if (**p == ',') {
(*p)++; // Skip ','
jsonSkipWhiteSpaces(p,end); // Skip whitespace before next element
continue; // Parse next element
} else if (**p == ']') {
(*p)++; // Skip ']'
return t; // End of array
} else {
// Unexpected character (not ',' or ']')
exprTokenRelease(t);
return NULL;
}
}
}
/* Turn a JSON value into an expr token. */
static exprtoken *jsonParseValueToken(const char **p, const char *end) {
jsonSkipWhiteSpaces(p,end);
if (*p >= end) return NULL;
switch (**p) {
case '"': return jsonParseStringToken(p,end);
case '[': return jsonParseArrayToken(p,end);
case '{': return NULL; // No nested elements support for now.
case 't': return jsonParseLiteralToken(p,end,"true",EXPR_TOKEN_NUM,1);
case 'f': return jsonParseLiteralToken(p,end,"false",EXPR_TOKEN_NUM,0);
case 'n': return jsonParseLiteralToken(p,end,"null",EXPR_TOKEN_NULL,0);
default:
// Check if it starts like a number.
if (isdigit((unsigned char)**p) || **p=='-' || **p=='+') {
return jsonParseNumberToken(p,end);
}
// Anything else is an unsupported type or malformed JSON.
return NULL;
}
}
/* ============================== Fast key seeking ========================== */
/* Finds the start of the value for a given field key within a JSON object.
* Returns pointer to the first char of the value, or NULL if not found/error.
* This function does not perform any allocation and is optimized to seek
* the specified *toplevel* filed as fast as possible. */
static const char *jsonSeekField(const char *json, const char *end,
const char *field, size_t flen) {
const char *p = json;
jsonSkipWhiteSpaces(&p,end);
if (p >= end || *p != '{') return NULL; // Must start with '{'.
p++; // skip '{'.
while (1) {
jsonSkipWhiteSpaces(&p,end);
if (p >= end) return NULL; // Reached end within object.
if (*p == '}') return NULL; // End of object, field not found.
// Expecting a key (string).
if (*p != '"') return NULL; // Key must be a string.
// --- Key Matching using jsonSkipString ---
const char *key_start = p + 1; // Start of key content.
const char *key_end_p = p; // Will later contain the end.
// Use jsonSkipString() to find the end.
if (!jsonSkipString(&key_end_p, end)) {
// Unterminated / invalid key string.
return NULL;
}
// Calculate the length of the key's content.
size_t klen = (key_end_p - 1) - key_start;
/* Perform the comparison using the raw key content.
* WARNING: This uses memcmp(), so we don't handle escaped chars
* within the key matching against unescaped chars in 'field'. */
int match = klen == flen && !memcmp(key_start, field, flen);
// Update the main pointer 'p' to be after the key string.
p = key_end_p;
// Now we expect to find a ":" followed by a value.
jsonSkipWhiteSpaces(&p,end);
if (p>=end || *p!=':') return NULL; // Expect ':' after key
p++; // Skip ':'.
// Seek value.
jsonSkipWhiteSpaces(&p,end);
if (p>=end) return NULL; // Expect value after ':'
if (match) {
// Found the matching key, p now points to the start of the value.
return p;
} else {
// Key didn't match, skip the corresponding value.
if (!jsonSkipValue(&p,end)) return NULL; // Syntax error.
}
// Look for comma or a closing brace.
jsonSkipWhiteSpaces(&p,end);
if (p>=end) return NULL; // Reached end after value.
if (*p == ',') {
p++; // Skip comma, continue loop to find next key.
continue;
} else if (*p == '}') {
return NULL; // Reached end of object, field not found.
}
return NULL; // Malformed JSON (unexpected char after value).
}
}
/* This is the only real API that this file conceptually exports (it is
* inlined, actually). */
exprtoken *jsonExtractField(const char *json, size_t json_len,
const char *field, size_t field_len)
{
const char *end = json + json_len;
const char *valptr = jsonSeekField(json,end,field,field_len);
if (!valptr) return NULL;
/* Key found, valptr points to the start of the value.
* Convert it into an expression token object. */
return jsonParseValueToken(&valptr,end);
}

406
modules/vector-sets/fastjson_test.c Normal file
View File

@ -0,0 +1,406 @@
/* fastjson_test.c - Stress test for fastjson.c
*
* This performs boundary and corruption tests to ensure
* the JSON parser handles edge cases without accessing
* memory outside the bounds of the input.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <signal.h>
#include <time.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <errno.h>
#include <setjmp.h>
/* Page size constant - typically 4096 or 16k bytes (Apple Silicon).
* We use 16k so that it will work on both, but not with Linux huge pages. */
#define PAGE_SIZE 4096*4
#define MAX_JSON_SIZE (PAGE_SIZE - 128) /* Keep some margin */
#define MAX_FIELD_SIZE 64
#define NUM_TEST_ITERATIONS 100000
#define NUM_CORRUPTION_TESTS 10000
#define NUM_BOUNDARY_TESTS 10000
/* Test state tracking */
static char *safe_page = NULL; /* Start of readable/writable page */
static char *unsafe_page = NULL; /* Start of inaccessible guard page */
static int boundary_violation = 0; /* Flag for boundary violations */
static jmp_buf jmpbuf; /* For signal handling */
static int tests_passed = 0;
static int tests_failed = 0;
static int corruptions_passed = 0;
static int boundary_tests_passed = 0;
/* Test metadata for tracking */
typedef struct {
char *json;
size_t json_len;
char field[MAX_FIELD_SIZE];
size_t field_len;
int expected_result;
} test_case_t;
/* Forward declarations for test JSON generation */
char *generate_random_json(size_t *len, char *field, size_t *field_len, int *has_field);
void corrupt_json(char *json, size_t len);
void setup_test_memory(void);
void cleanup_test_memory(void);
void run_normal_tests(void);
void run_corruption_tests(void);
void run_boundary_tests(void);
void print_test_summary(void);
/* Signal handler for segmentation violations */
static void sigsegv_handler(int sig) {
boundary_violation = 1;
printf("Boundary violation detected! Caught signal %d\n", sig);
longjmp(jmpbuf, 1);
}
/* Wrapper for jsonExtractField to check for boundary violations */
exprtoken *safe_extract_field(const char *json, size_t json_len,
const char *field, size_t field_len) {
boundary_violation = 0;
if (setjmp(jmpbuf) == 0) {
return jsonExtractField(json, json_len, field, field_len);
} else {
return NULL; /* Return NULL if boundary violation occurred */
}
}
/* Setup two adjacent memory pages - one readable/writable, one inaccessible */
void setup_test_memory(void) {
/* Request a page of memory, with specific alignment. We rely on the
* fact that hopefully the page after that will cause a segfault if
* accessed. */
void *region = mmap(NULL, PAGE_SIZE,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS,
-1, 0);
if (region == MAP_FAILED) {
perror("mmap failed");
exit(EXIT_FAILURE);
}
safe_page = (char*)region;
unsafe_page = safe_page + PAGE_SIZE;
// Uncomment to make sure it crashes :D
// printf("%d\n", unsafe_page[5]);
/* Set up signal handlers for memory access violations */
struct sigaction sa;
sa.sa_handler = sigsegv_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGSEGV, &sa, NULL);
sigaction(SIGBUS, &sa, NULL);
}
void cleanup_test_memory(void) {
if (safe_page != NULL) {
munmap(safe_page, PAGE_SIZE);
safe_page = NULL;
unsafe_page = NULL;
}
}
/* Generate random strings with proper escaping for JSON */
void generate_random_string(char *buffer, size_t max_len) {
static const char charset[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
size_t len = 1 + rand() % (max_len - 2); /* Ensure at least 1 char */
for (size_t i = 0; i < len; i++) {
buffer[i] = charset[rand() % (sizeof(charset) - 1)];
}
buffer[len] = '\0';
}
/* Generate random numbers as strings */
void generate_random_number(char *buffer, size_t max_len) {
double num = (double)rand() / RAND_MAX * 1000.0;
/* Occasionally make it negative or add decimal places */
if (rand() % 5 == 0) num = -num;
if (rand() % 3 != 0) num += (double)(rand() % 100) / 100.0;
snprintf(buffer, max_len, "%.6g", num);
}
/* Generate a random field name */
void generate_random_field(char *field, size_t *field_len) {
generate_random_string(field, MAX_FIELD_SIZE / 2);
*field_len = strlen(field);
}
/* Generate a random JSON object with fields */
char *generate_random_json(size_t *len, char *field, size_t *field_len, int *has_field) {
char *json = malloc(MAX_JSON_SIZE);
if (json == NULL) {
perror("malloc");
exit(EXIT_FAILURE);
}
char buffer[MAX_JSON_SIZE / 4]; /* Buffer for generating values */
int pos = 0;
int num_fields = 1 + rand() % 10; /* Random number of fields */
int target_field_index = rand() % num_fields; /* Which field to return */
/* Start the JSON object */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "{");
/* Generate random field/value pairs */
for (int i = 0; i < num_fields; i++) {
/* Add a comma if not the first field */
if (i > 0) {
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, ", ");
}
/* Generate a field name */
if (i == target_field_index) {
/* This is our target field - save it for the caller */
generate_random_field(field, field_len);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "\"%s\": ", field);
*has_field = 1;
/* Sometimes change the last char so that it will not match. */
if (rand() % 2) {
*has_field = 0;
field[*field_len-1] = '!';
}
} else {
generate_random_string(buffer, MAX_FIELD_SIZE / 4);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "\"%s\": ", buffer);
}
/* Generate a random value type */
int value_type = rand() % 5;
switch (value_type) {
case 0: /* String */
generate_random_string(buffer, MAX_JSON_SIZE / 8);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "\"%s\"", buffer);
break;
case 1: /* Number */
generate_random_number(buffer, MAX_JSON_SIZE / 8);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "%s", buffer);
break;
case 2: /* Boolean: true */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "true");
break;
case 3: /* Boolean: false */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "false");
break;
case 4: /* Null */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "null");
break;
case 5: /* Array (simple) */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "[");
int array_items = 1 + rand() % 5;
for (int j = 0; j < array_items; j++) {
if (j > 0) pos += snprintf(json + pos, MAX_JSON_SIZE - pos, ", ");
/* Array items - either number or string */
if (rand() % 2) {
generate_random_number(buffer, MAX_JSON_SIZE / 16);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "%s", buffer);
} else {
generate_random_string(buffer, MAX_JSON_SIZE / 16);
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "\"%s\"", buffer);
}
}
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "]");
break;
}
}
/* Close the JSON object */
pos += snprintf(json + pos, MAX_JSON_SIZE - pos, "}");
*len = pos;
return json;
}
/* Corrupt JSON by replacing random characters */
void corrupt_json(char *json, size_t len) {
if (len < 2) return; /* Too short to corrupt safely */
/* Corrupt 1-3 characters */
int num_corruptions = 1 + rand() % 3;
for (int i = 0; i < num_corruptions; i++) {
size_t pos = rand() % len;
char corruption = " \t\n{}[]\":,0123456789abcdefXYZ"[rand() % 30];
json[pos] = corruption;
}
}
/* Run standard parser tests with generated valid JSON */
void run_normal_tests(void) {
printf("Running normal JSON extraction tests...\n");
for (int i = 0; i < NUM_TEST_ITERATIONS; i++) {
char field[MAX_FIELD_SIZE] = {0};
size_t field_len = 0;
size_t json_len = 0;
int has_field = 0;
/* Generate random JSON */
char *json = generate_random_json(&json_len, field, &field_len, &has_field);
/* Use valid field to test parser */
exprtoken *token = safe_extract_field(json, json_len, field, field_len);
/* Check if we got a token as expected */
if (has_field && token != NULL) {
exprTokenRelease(token);
tests_passed++;
} else if (!has_field && token == NULL) {
tests_passed++;
} else {
tests_failed++;
}
/* Test with a non-existent field */
char nonexistent_field[MAX_FIELD_SIZE] = "nonexistent_field";
token = safe_extract_field(json, json_len, nonexistent_field, strlen(nonexistent_field));
if (token == NULL) {
tests_passed++;
} else {
exprTokenRelease(token);
tests_failed++;
}
free(json);
}
}
/* Run tests with corrupted JSON */
void run_corruption_tests(void) {
printf("Running JSON corruption tests...\n");
for (int i = 0; i < NUM_CORRUPTION_TESTS; i++) {
char field[MAX_FIELD_SIZE] = {0};
size_t field_len = 0;
size_t json_len = 0;
int has_field = 0;
/* Generate random JSON */
char *json = generate_random_json(&json_len, field, &field_len, &has_field);
/* Make a copy and corrupt it */
char *corrupted = malloc(json_len + 1);
if (!corrupted) {
perror("malloc");
free(json);
exit(EXIT_FAILURE);
}
memcpy(corrupted, json, json_len + 1);
corrupt_json(corrupted, json_len);
/* Test with corrupted JSON */
exprtoken *token = safe_extract_field(corrupted, json_len, field, field_len);
/* We're just testing that it doesn't crash or access invalid memory */
if (boundary_violation) {
printf("Boundary violation with corrupted JSON!\n");
tests_failed++;
} else {
if (token != NULL) {
exprTokenRelease(token);
}
corruptions_passed++;
}
free(corrupted);
free(json);
}
}
/* Run tests at memory boundaries */
void run_boundary_tests(void) {
printf("Running memory boundary tests...\n");
for (int i = 0; i < NUM_BOUNDARY_TESTS; i++) {
char field[MAX_FIELD_SIZE] = {0};
size_t field_len = 0;
size_t json_len = 0;
int has_field = 0;
/* Generate random JSON */
char *temp_json = generate_random_json(&json_len, field, &field_len, &has_field);
/* Truncate the JSON to a random length */
size_t truncated_len = 1 + rand() % json_len;
/* Place at the edge of the safe page */
size_t offset = PAGE_SIZE - truncated_len;
memcpy(safe_page + offset, temp_json, truncated_len);
/* Test parsing with non-existent field (forcing it to scan to end) */
char nonexistent_field[MAX_FIELD_SIZE] = "nonexistent_field";
exprtoken *token = safe_extract_field(safe_page + offset, truncated_len,
nonexistent_field, strlen(nonexistent_field));
/* We're just testing that it doesn't access memory beyond the boundary */
if (boundary_violation) {
printf("Boundary violation at edge of memory page!\n");
tests_failed++;
} else {
if (token != NULL) {
exprTokenRelease(token);
}
boundary_tests_passed++;
}
free(temp_json);
}
}
/* Print summary of test results */
void print_test_summary(void) {
printf("\n===== FASTJSON PARSER TEST SUMMARY =====\n");
printf("Normal tests passed: %d/%d\n", tests_passed, NUM_TEST_ITERATIONS * 2);
printf("Corruption tests passed: %d/%d\n", corruptions_passed, NUM_CORRUPTION_TESTS);
printf("Boundary tests passed: %d/%d\n", boundary_tests_passed, NUM_BOUNDARY_TESTS);
printf("Failed tests: %d\n", tests_failed);
if (tests_failed == 0) {
printf("\nALL TESTS PASSED! The JSON parser appears to be robust.\n");
} else {
printf("\nSome tests FAILED. The JSON parser may be vulnerable.\n");
}
}
/* Entry point for fastjson parser test */
void run_fastjson_test(void) {
printf("Starting fastjson parser stress test...\n");
/* Seed the random number generator */
srand(time(NULL));
/* Setup test memory environment */
setup_test_memory();
/* Run the various test phases */
run_normal_tests();
run_corruption_tests();
run_boundary_tests();
/* Print summary */
print_test_summary();
/* Cleanup */
cleanup_test_memory();
}

2
src/Makefile
View File

@ -320,7 +320,7 @@ endif
ifneq ($(SKIP_VEC_SETS),yes)
vpath %.c ../modules/vector-sets
REDIS_VEC_SETS_OBJ=hnsw.o cJSON.o vset.o
REDIS_VEC_SETS_OBJ=hnsw.o vset.o
FINAL_CFLAGS+=-DINCLUDE_VEC_SETS=1
endif

5
src/replication.c
View File

@ -1125,9 +1125,7 @@ void syncCommand(client *c) {
"Full sync will continue with dedicated rdb channel.",
replicationGetSlaveName(c));
/* Send +RDBCHANNELSYNC with client id. Rdbchannel of replica
* will call 'replconf set-main-ch-id <client-id>' so we can
* associate replica connections on master.*/
/* Send +RDBCHANNELSYNC with client id so we can associate replica connections on master.*/
len = snprintf(buf, sizeof(buf), "+RDBCHANNELSYNC %llu\r\n",
(unsigned long long) c->id);
if (connWrite(c->conn, buf, strlen(buf)) != len)
@ -2732,6 +2730,7 @@ int slaveTryPartialResynchronization(connection *conn, int read_reply) {
if (!client_id) {
serverLog(LL_WARNING,
"Master replied with wrong +RDBCHANNELSYNC syntax: %s", reply);
sdsfree(reply);
return PSYNC_NOT_SUPPORTED;
}
server.repl_main_ch_client_id = strtoll(client_id, NULL, 10);;

4
tests/unit/obuf-limits.tcl
View File

@ -37,7 +37,9 @@ start_server {tags {"obuf-limits external:skip logreqres:skip"}} {
set omem 0
while 1 {
r publish foo bar
# The larger content size ensures that client.buf gets filled more quickly,
# allowing us to correctly observe the gradual increase of `omem`
r publish foo [string repeat bar 50]
set clients [split [r client list] "\r\n"]
set c [split [lindex $clients 1] " "]
if {![regexp {omem=([0-9]+)} $c - omem]} break