root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 5 Packages Projects Releases Wiki Activity
rabbitmq-server/deps/rabbit/test/unicode_SUITE.erl

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

114 lines
4.4 KiB
Erlang
Raw Permalink Normal View History

Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
-module(unicode_SUITE).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-compile(export_all).
Test Unicode char in queue names
2022-12-16 19:47:14 +08:00
%% Unicode U+1F407
-define(UNICODE_STRING, "bunny🐇bunny").
Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
all() ->
[
{group, queues}
].
groups() ->
[
{queues, [], [
classic_queue_v2,
quorum_queue,
stream
]}
].
%% -------------------------------------------------------------------
%% Testsuite setup/teardown.
%% -------------------------------------------------------------------
init_per_suite(Config) ->
rabbit_ct_helpers:log_environment(),
rabbit_ct_helpers:run_setup_steps(Config).
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config).
init_per_group(Group, Config0) ->
PrivDir0 = ?config(priv_dir, Config0),
Test Unicode char in queue names
2022-12-16 19:47:14 +08:00
PrivDir = filename:join(PrivDir0, ?UNICODE_STRING),
Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
ok = file:make_dir(PrivDir),
Config = rabbit_ct_helpers:set_config(Config0, [{priv_dir, PrivDir},
{rmq_nodename_suffix, Group}]),
rabbit_ct_helpers:run_steps(Config,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()
).
end_per_group(_, Config) ->
rabbit_ct_helpers:run_steps(Config,
rabbit_ct_client_helpers:teardown_steps() ++
rabbit_ct_broker_helpers:teardown_steps()).
init_per_testcase(Testcase, Config) ->
rabbit_ct_helpers:testcase_started(Config, Testcase).
end_per_testcase(Testcase, Config) ->
rabbit_ct_helpers:testcase_finished(Config, Testcase).
classic_queue_v2(Config) ->
ok = queue(Config, ?FUNCTION_NAME, []).
quorum_queue(Config) ->
ok = queue(Config, ?FUNCTION_NAME, [{<<"x-queue-type">>, longstr, <<"quorum">>}]).
queue(Config, QName0, Args) ->
Test Unicode char in queue names
2022-12-16 19:47:14 +08:00
QName1 = rabbit_data_coercion:to_binary(QName0),
QName = <<QName1/binary, ?UNICODE_STRING/utf8>>,
Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
Server = rabbit_ct_broker_helpers:get_node_config(Config, 0, nodename),
Ch = rabbit_ct_client_helpers:open_channel(Config, Server),
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = true,
arguments = Args
}),
rabbit_ct_client_helpers:publish(Ch, QName, 1),
{#'basic.get_ok'{}, #amqp_msg{payload = <<"1">>}} =
amqp_channel:call(Ch, #'basic.get'{queue = QName, no_ack = false}),
{'queue.delete_ok', 0} = amqp_channel:call(Ch, #'queue.delete'{queue = QName}),
ok.
stream(Config) ->
Server = rabbit_ct_broker_helpers:get_node_config(Config, 0, nodename),
Test Unicode char in queue names
2022-12-16 19:47:14 +08:00
ConsumerTag = QName0 = atom_to_binary(?FUNCTION_NAME),
QName = <<QName0/binary, ?UNICODE_STRING/utf8>>,
Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
Ch = rabbit_ct_client_helpers:open_channel(Config, Server),
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = true,
arguments = [{<<"x-queue-type">>, longstr, <<"stream">>}]
}),
rabbit_ct_client_helpers:publish(Ch, QName, 1),
?assertMatch(#'basic.qos_ok'{},
amqp_channel:call(Ch, #'basic.qos'{global = false,
prefetch_count = 1})),
amqp_channel:subscribe(Ch,
#'basic.consume'{queue = QName,
no_ack = false,
consumer_tag = ConsumerTag,
arguments = [{<<"x-stream-offset">>, long, 0}]},
self()),
receive
#'basic.consume_ok'{consumer_tag = ConsumerTag} ->
ok
end,
DelTag = receive
{#'basic.deliver'{delivery_tag = DeliveryTag}, _} ->
DeliveryTag
Test: Increase receive timeout in all rabbit test suites
2024-12-12 22:51:59 +08:00
after 30_000 ->
Make classic queues v2 memory efficient Store directory names as binary instead of string. This commit saves >1GB of memory per 100,000 classic queues v2. With longish node names, the memory savings are even much higher. This commit is especially a prerequisite for scalalbe MQTT where every subscribing MQTT connection creates its own classic queue. So, with 3 million MQTT subscribers, this commit saves >30 GB of memory. This commits stores file names as binaries and converts back to file:filename() when passed to file API functions. This is to reduce risk of breaking behaviour for path names containing unicode chars on certain platforms. Alternatives to the implementation in this commit: 1. Store common directory list prefix only once (e.g. put it into persistent_term) and store per queue directory names in ETS. 2. Use file:filename_all() instead of file:filename() and pass binaries to the file module functions. However this might be brittle on some platforms since these binaries are interpreted as "raw filenames". Using raw filenames requires more changes to classic queues which we want to avoid to reduce risk. The downside of the implemenation in this commit is that the binary gets converted to a list sometimes. This happens whenever a file is flushed or a new file gets created for example. Following perf tests did not show any regression in performance: ``` java -jar target/perf-test.jar -s 10 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10000 -x 1 -y 0 -u q -f persistent -z 30 java -jar target/perf-test.jar -s 10 -x 100 -qp q%d -qpf 1 -qpt 100 -y 0 -f persistent -z 60 -c 1000 ``` Furthermore `rabbit_file` did not show up in the CPU flame graphs either.
2022-12-14 01:25:13 +08:00
ct:fail(timeout)
end,
ok = amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DelTag,
multiple = false}),
amqp_channel:call(Ch, #'basic.cancel'{consumer_tag = ConsumerTag}),
{'queue.delete_ok', 0} = amqp_channel:call(Ch, #'queue.delete'{queue = QName}),
ok.