root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 9 Packages Projects Releases Wiki Activity
rabbitmq-server/deps/rabbitmq_mqtt/test/mqtt_shared_SUITE.erl

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

2087 lines
88 KiB
Erlang
Raw Normal View History

Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%
(c) year bumps
2024-01-02 11:02:20 +08:00
%% Copyright (c) 2007-2025 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. All rights reserved.
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
%% This test suite contains test cases that are shared between (i.e. executed across):
%% 1. plugins rabbitmq_mqtt and rabbitmq_web_mqtt
%% 2. MQTT versions v3, v4 and v5
%%
%% In other words, this test suite should not contain any test case that is executed
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% only with a particular plugin or particular MQTT version.
mqtt tests: Move web_mqtt tests out of rabbitmq_mqtt The shared test suite was renamed only for clarity, but the Web-MQTT test suites were renamed out of necessity: since we are now adding the MQTT test directory to the code path we need test suites to have different names to avoid conflicts. We can't (easily) addpath only for this test suite either since CT hooks don't call functions in a predictable enough manner; it would always be hacky.
2024-09-11 17:44:50 +08:00
%%
%% When adding a test case here the same function must be defined in web_mqtt_shared_SUITE.
-module(mqtt_shared_SUITE).
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
-compile([export_all,
nowarn_export_all]).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
-include_lib("common_test/include/ct.hrl").
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
-include_lib("eunit/include/eunit.hrl").
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
-include_lib("amqp_client/include/amqp_client.hrl").
-include_lib("rabbitmq_ct_helpers/include/rabbit_assert.hrl").
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
-include_lib("rabbitmq_ct_helpers/include/rabbit_mgmt_test.hrl").
-import(rabbit_ct_broker_helpers,
[rabbitmqctl_list/3,
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
rabbitmqctl/3,
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
rpc/4,
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
rpc/5,
rpc_all/4,
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
get_node_config/3,
drain_node/2,
Reduce Khepri test flakes Test case rabbit_mqtt_qos0_queue_kill_node flaked because after an MQTT client subscribes on node 0, RabbitMQ returns success and replicated the new binding to node 0 and node 1, but not yet to node 2. Another MQTT client then publishes on node 2 without the binding being present yet on node 2, and the message therefore isn't routed. This commit attempts to eliminate this flake. It adds a function to rabbit_ct_broker_helpers which waits until a given node has caught up with the leader node. We can reuse that function in future to eliminate more test flakes.
2024-08-06 22:16:14 +08:00
revive_node/2,
await_metadata_store_consistent/2
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
]).
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
-import(rabbit_ct_helpers,
[eventually/3,
eventually/1]).
-import(util,
[all_connection_pids/1,
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
get_global_counters/1, get_global_counters/2,
get_global_counters/3, get_global_counters/4,
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
expect_publishes/3,
connect/2, connect/3, connect/4,
get_events/1, assert_event_type/2, assert_event_prop/2,
await_exit/1, await_exit/2,
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
publish_qos1_timeout/4,
non_clean_sess_opts/0]).
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
-import(rabbit_mgmt_test_util,
[http_get/2,
http_delete/3]).
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
%% defined in MQTT v5 (not in v4 or v3)
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
-define(RC_SERVER_SHUTTING_DOWN, 16#8B).
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
-define(RC_KEEP_ALIVE_TIMEOUT, 16#8D).
-define(RC_SESSION_TAKEN_OVER, 16#8E).
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
-define(TIMEOUT, 30_000).
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
all() ->
mqtt tests: Move web_mqtt tests out of rabbitmq_mqtt The shared test suite was renamed only for clarity, but the Web-MQTT test suites were renamed out of necessity: since we are now adding the MQTT test directory to the code path we need test suites to have different names to avoid conflicts. We can't (easily) addpath only for this test suite either since CT hooks don't call functions in a predictable enough manner; it would always be hacky.
2024-09-11 17:44:50 +08:00
[{group, mqtt}].
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
%% The code being tested under v3 and v4 is almost identical.
%% To save time in CI, we therefore run only a very small subset of tests in v3.
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
groups() ->
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
[
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
{mqtt, [],
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
[{cluster_size_1, [],
[{v3, [], cluster_size_1_tests_v3()},
{v4, [], cluster_size_1_tests()},
{v5, [], cluster_size_1_tests()}]},
{cluster_size_3, [],
[{v4, [], cluster_size_3_tests()},
{v5, [], cluster_size_3_tests()}]}
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
]}
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
].
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
cluster_size_1_tests_v3() ->
[global_counters,
events
].
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
cluster_size_1_tests() ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
[
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
global_counters %% must be the 1st test case
Emit histogram metric for received message sizes per protocol (#12342) * Add global histogram metrics for received message sizes per-protocol fixup: add new files to bazel fixup: expose message_size_bytes as prometheus classic histogram type `rabbit_msg_size_metrics` does not use `seshat` any more, but `counters` directly. fixup: add msg_size_metrics unit test * Improve message size histogram 1. Avoid unnecessary time series emitted for stream protocol The stream protocol cannot observe message sizes. This commit ensures that the following time series are omitted: ``` rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="64"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="256"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="1024"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="4096"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="16384"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="65536"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="262144"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="1048576"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="4194304"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="16777216"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="67108864"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="268435456"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="+Inf"} 0 rabbitmq_global_message_size_bytes_count{protocol="stream"} 0 rabbitmq_global_message_size_bytes_sum{protocol="stream"} 0 ``` This reduces the number of time series by 15. 2. Further reduce the number of time series by reducing the number of buckets. Instead of 13 bucktes, emit only 9 buckets. Buckets are not free, each is an extra time series stored. Prior to this commit: ``` curl -s -u guest:guest localhost:15692/metrics | ag message_size | wc -l 92 ``` After this commit: ``` curl -s -u guest:guest localhost:15692/metrics | ag message_size | wc -l 57 ``` 3. The emitted metric should be called `rabbitmq_message_size_bytes_bucket` instead of `rabbitmq_global_message_size_bytes_bucket`. The latter is poor naming. There is no need to use `global` in the metric name given that this metric doesn't exist in the old flawed aggregated metrics. 4. This commit simplies module `rabbit_global_counters`. 5. Avoid garbage collecting the 10-elements list of buckets per message being received. --------- Co-authored-by: Péter Gömöri <peter@84codes.com>
2024-09-25 00:08:24 +08:00
,message_size_metrics
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,block_only_publisher
,many_qos1_messages
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
,session_expiry
Simplify test cases 1. Only run the CLI tests on a single node cluster. The shared_SUITE is already very big. Testing the same CLI commands against node-0 on a 3-node cluster brings no benefit. 2. Move the two new CLI test cases in front of management_plugin_connection because they are similar in that all three tests close the MQTT connection. 3. There is no need to query the HTTP API for the two new CLI test cases. 4. There is no need to set keepalive in the two new CLI test cases.
2024-08-28 18:27:46 +08:00
,cli_close_all_connections
,cli_close_all_user_connections
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,management_plugin_connection
,management_plugin_enable
,disconnect
,pubsub_shared_connection
,pubsub_separate_connections
,will_with_disconnect
,will_without_disconnect
,decode_basic_properties
,quorum_queue_rejects
,events
,internal_event_handler
,non_clean_sess_reconnect_qos1
,non_clean_sess_reconnect_qos0
,non_clean_sess_reconnect_qos0_and_qos1
,non_clean_sess_empty_client_id
,subscribe_same_topic_same_qos
,subscribe_same_topic_different_qos
,subscribe_multiple
,large_message_mqtt_to_mqtt
,large_message_amqp_to_mqtt
,keepalive
,keepalive_turned_off
,block
,amqp_to_mqtt_qos0
Trailing whitespace
2025-06-04 22:48:47 +08:00
,clean_session_disconnect_client
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,clean_session_node_restart
,clean_session_node_kill
,rabbit_status_connection_count
,trace
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
,trace_large_message
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,max_packet_size_unauthenticated
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
,max_packet_size_authenticated
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,default_queue_type
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
,message_interceptors
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
,utf8
Make retained message stores compatible with pre 3.13 The format of #mqtt_msg{} changes from 3.12 to 3.13. In 3.13 the record contains 2 additional fields: * props * timestamp The old #mqtt_msg{} might still be stored by the retained message store in ets or dets. This commit converts such an old message format when read from the database. The alternative would have been to run a migration function over the whole table which is slightly more complex to implement. Instead of giving the new message format a different record name, e.g. #mqtt_msg_v2{}, this commit decides to re-use the same name such that the new code only handles the record name #mqtt_msg{}.
2023-06-07 19:20:28 +08:00
,retained_message_conversion
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
,bind_exchange_to_exchange
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
,bind_exchange_to_exchange_single_message
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
,notify_consumer_classic_queue_deleted
,notify_consumer_quorum_queue_deleted
,notify_consumer_qos0_queue_deleted
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
].
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
cluster_size_3_tests() ->
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
[
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
pubsub,
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
queue_down_qos1,
consuming_classic_queue_down,
Restore credit_flow between channel/MQTT connection -> CQ processes The credit_flow between publishing AMQP 0.9.1 channel (or MQTT connection) and (non-mirrored) classic queue processes was unintentionally removed in 4.0 together with anything else related to CQ mirroring. By default we restore the 3.x behaviour for non-mirored classic queues. It is possible to disable flow-control (the earlier 4.0.x behaviour) with the new env `classic_queue_flow_control`. In 3.x this was possible with the config `mirroring_flow_control`. (cherry picked from commit d65bd7d07a48b6122256e62349e8fe441e488f32)
2024-12-04 00:34:32 +08:00
flow_classic_queue,
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
flow_quorum_queue,
flow_stream,
rabbit_mqtt_qos0_queue,
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
rabbit_mqtt_qos0_queue_kill_node,
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
cli_list_queues,
delete_create_queue,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_reconnect,
Run test duplicate_client_id with Khepri The only reason to skip tests with Khepri is if they use classic mirror queues or use Mnesia directly.
2023-10-27 15:15:14 +08:00
session_takeover,
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
duplicate_client_id,
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
publish_to_all_queue_types_qos0,
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
publish_to_all_queue_types_qos1,
maintenance
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
].
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
suite() ->
Bump test cluster creation timeouts in MQTT tests. For example in the ff_SUITE wee see in Buildbuddy sporadic failures that the cluster cannot be created within 2 minutes: ``` *** CT Error Notification 2023-04-24 10:58:55.628 ***🔗 rabbit_ct_helpers:port_receive_loop failed on line 945 Reason: {timetrap_timeout,120000} ... === Ended at 2023-04-24 10:58:55 === Location: [{rabbit_ct_helpers,port_receive_loop,945}, {rabbit_ct_helpers,exec,920}, {rabbit_ct_broker_helpers,cluster_nodes1,858}, {rabbit_ct_broker_helpers,cluster_nodes1,840}, {rabbit_ct_helpers,run_steps,141}, {ff_SUITE,init_per_group,last_expr}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1379}, {test_server,run_test_case_eval,1223}] === Reason: timetrap timeout === *** init_per_group failed. Skipping all cases. ``` The default time limit for a test case is 30 minutes.
2023-04-24 20:38:29 +08:00
[{timetrap, {minutes, 10}}].
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
%% -------------------------------------------------------------------
%% Testsuite setup/teardown.
%% -------------------------------------------------------------------
init_per_suite(Config) ->
rabbit_ct_helpers:log_environment(),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
Config1 = rabbit_ct_helpers:merge_app_env(
Config, {rabbit, [
{quorum_tick_interval, 1000},
rabbitmq_ct_broker_helpers: Use node 2 as the cluster seed node [Why] When running mixed-version tests, nodes 1/3/5/... are using the primary umbrella, so usually the newest version. Nodes 2/4/6/... are using the secondary umbrella, thus the old version. When clustering, we used to use node 1 (running a new version) as the seed node, meaning other nodes would join it. This complicates things with feature flags because we have to make sure that we start node 1 with new stable feature flags disabled to allow old nodes to join. This is also a problem with Khepri machine versions because the cluster would start with the latest version, which old nodes might not have. [How] This patch changes the logic to use a node running the secondary umbrella as the seed node instead. If there is no node running it, we pick the first node as before. V2: Revert part of "rabbitmq_ct_helpers: Fix how we set `$RABBITMQ_FEATURE_FLAGS` in tests" (commit 57ed962ef69669cb72c5dd7be93cb7fbf3ca4c0c). These changes are no longer needed with the new logic. V3: The check that verifies that the correct metadata store is used has a special case for nodes that use the secondary umbrella: if Khepri is supposed to be used but it's not, the feature flag is enabled. The reason is that the `v4.0.x` branch doesn't know about the `rel` configuration of `forced_feature_flags_on_init`. The nodes will have ignored thies parameter and booted with the stable feature flags only. Many testsuites are adapted to the new clustering order. If they manage which node joins which node, either the order is changed in the testcases, or nodes are started with only required feature flags. For testsuites that rely on peer discovery where the order is unknown, nodes are started with only required feature flags.
2025-01-20 18:24:11 +08:00
{stream_tick_interval, 1000},
Enable FFs required by 4.0
2025-07-02 17:35:41 +08:00
{forced_feature_flags_on_init, [
delete_ra_cluster_mqtt_node,
mqtt_v5,
rabbit_mqtt_qos0_queue,
restart_streams,
stream_sac_coordinator_unblock_group,
stream_update_config_command,
stream_filtering,
message_containers,
quorum_queue_non_voters
]},
Mqtt tests: start just required dependencies MQTT tests depend on a few plugins, which are just used in 1 or 2 suites each. These have caused issues in CI, triggering a bug in rabbitmq_federation where the mirrored supervisor submits a transaction while the cluster is being shut down. The transaction hangs and the whole rabbitmq_mqtt job times out. This bug has been addressed, however it is best to start just the required plugins on each SUITE.
2025-06-05 21:02:58 +08:00
{start_rmq_with_plugins_disabled, true}
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
]}),
rabbit_ct_helpers:run_setup_steps(Config1).
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config).
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
init_per_group(mqtt, Config0) ->
rabbit_ct_helpers:set_config(Config0, {websocket, false});
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
init_per_group(Group, Config)
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
when Group =:= v3;
Group =:= v4;
Group =:= v5 ->
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
rabbit_ct_helpers:set_config(Config, {mqtt_version, Group});
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
Run test duplicate_client_id with Khepri The only reason to skip tests with Khepri is if they use classic mirror queues or use Mnesia directly.
2023-10-27 15:15:14 +08:00
init_per_group(Group, Config0) ->
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
Nodes = case Group of
cluster_size_1 -> 1;
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
cluster_size_3 -> 3
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
end,
Run test duplicate_client_id with Khepri The only reason to skip tests with Khepri is if they use classic mirror queues or use Mnesia directly.
2023-10-27 15:15:14 +08:00
Suffix = rabbit_ct_helpers:testcase_absname(Config0, "", "-"),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
Config = rabbit_ct_helpers:set_config(
Config0,
[{rmq_nodes_count, Nodes},
{rmq_nodename_suffix, Suffix}]),
Mqtt tests: start just required dependencies MQTT tests depend on a few plugins, which are just used in 1 or 2 suites each. These have caused issues in CI, triggering a bug in rabbitmq_federation where the mirrored supervisor submits a transaction while the cluster is being shut down. The transaction hangs and the whole rabbitmq_mqtt job times out. This bug has been addressed, however it is best to start just the required plugins on each SUITE.
2025-06-05 21:02:58 +08:00
Config1 = rabbit_ct_helpers:run_steps(
Config,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
util:enable_plugin(Config1, rabbitmq_mqtt),
Config1.
Revert "Format MQTT code with `erlfmt`"
2023-01-28 02:25:57 +08:00
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
end_per_group(G, Config)
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
when G =:= cluster_size_1;
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
G =:= cluster_size_3 ->
Do not run_teardown_steps twice
2023-06-21 23:19:38 +08:00
rabbit_ct_helpers:run_steps(
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
Config,
rabbit_ct_client_helpers:teardown_steps() ++
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
rabbit_ct_broker_helpers:teardown_steps());
end_per_group(_, Config) ->
Config.
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
init_per_testcase(T, Config)
when T =:= management_plugin_connection;
Simplify test cases 1. Only run the CLI tests on a single node cluster. The shared_SUITE is already very big. Testing the same CLI commands against node-0 on a 3-node cluster brings no benefit. 2. Move the two new CLI test cases in front of management_plugin_connection because they are similar in that all three tests close the MQTT connection. 3. There is no need to query the HTTP API for the two new CLI test cases. 4. There is no need to set keepalive in the two new CLI test cases.
2024-08-28 18:27:46 +08:00
T =:= management_plugin_enable ->
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
inets:start(),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
init_per_testcase0(T, Config);
Queues with plugins - core
2025-02-03 22:59:34 +08:00
init_per_testcase(T, Config)
when T =:= clean_session_disconnect_client;
Delete mqtt qos0 when connection closes
2025-06-04 16:49:19 +08:00
T =:= zero_session_expiry_interval_disconnect_client;
Queues with plugins - core
2025-02-03 22:59:34 +08:00
T =:= clean_session_node_restart;
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
T =:= clean_session_node_kill;
T =:= notify_consumer_qos0_queue_deleted ->
Queues with plugins - core
2025-02-03 22:59:34 +08:00
ok = rpc(Config, rabbit_registry, register, [queue, <<"qos0">>, rabbit_mqtt_qos0_queue]),
init_per_testcase0(T, Config);
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
init_per_testcase(Testcase, Config) ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
init_per_testcase0(Testcase, Config).
init_per_testcase0(Testcase, Config) ->
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
rabbit_ct_helpers:testcase_started(Config, Testcase).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
end_per_testcase(T, Config)
when T =:= management_plugin_connection;
Simplify test cases 1. Only run the CLI tests on a single node cluster. The shared_SUITE is already very big. Testing the same CLI commands against node-0 on a 3-node cluster brings no benefit. 2. Move the two new CLI test cases in front of management_plugin_connection because they are similar in that all three tests close the MQTT connection. 3. There is no need to query the HTTP API for the two new CLI test cases. 4. There is no need to set keepalive in the two new CLI test cases.
2024-08-28 18:27:46 +08:00
T =:= management_plugin_enable ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = inets:stop(),
end_per_testcase0(T, Config);
Queues with plugins - core
2025-02-03 22:59:34 +08:00
end_per_testcase(T, Config)
when T =:= clean_session_disconnect_client;
Delete mqtt qos0 when connection closes
2025-06-04 16:49:19 +08:00
T =:= zero_session_expiry_interval_disconnect_client;
Queues with plugins - core
2025-02-03 22:59:34 +08:00
T =:= clean_session_node_restart;
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
T =:= clean_session_node_kill;
T =:= notify_consumer_qos0_queue_deleted ->
Queues with plugins - core
2025-02-03 22:59:34 +08:00
ok = rpc(Config, rabbit_registry, unregister, [queue, <<"qos0">>]),
end_per_testcase0(T, Config);
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
end_per_testcase(Testcase, Config) ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
end_per_testcase0(Testcase, Config).
end_per_testcase0(Testcase, Config) ->
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
rabbit_ct_client_helpers:close_channels_and_connection(Config, 0),
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
%% Assert that every testcase cleaned up their MQTT sessions.
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
_ = rpc(Config, ?MODULE, delete_queues, []),
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
eventually(?_assertEqual([], rpc(Config, rabbit_amqqueue, list, []))),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
rabbit_ct_helpers:testcase_finished(Config, Testcase).
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
delete_queues() ->
[catch rabbit_amqqueue:delete(Q, false, false, <<"dummy">>)
|| Q <- rabbit_amqqueue:list()].
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
%% -------------------------------------------------------------------
%% Testsuite cases
%% -------------------------------------------------------------------
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
disconnect(Config) ->
C = connect(?FUNCTION_NAME, Config),
eventually(?_assertEqual(1, length(all_connection_pids(Config)))),
process_flag(trap_exit, true),
ok = emqtt:disconnect(C),
await_exit(C, normal),
eventually(?_assertEqual([], all_connection_pids(Config))),
ok.
pubsub_shared_connection(Config) ->
C = connect(?FUNCTION_NAME, Config),
Topic = <<"/topic/test-mqtt">>,
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
Payload = <<"a\x00a">>,
?assertMatch({ok, #{packet_id := _,
reason_code := 0,
reason_code_name := success
}},
emqtt:publish(C, Topic, Payload, [{qos, 1}])),
ok = expect_publishes(C, Topic, [Payload]),
ok = emqtt:disconnect(C).
pubsub_separate_connections(Config) ->
Pub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_publisher">>, Config),
Sub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_subscriber">>, Config),
Topic = <<"/topic/test-mqtt">>,
{ok, _, [1]} = emqtt:subscribe(Sub, Topic, qos1),
Payload = <<"a\x00a">>,
?assertMatch({ok, #{packet_id := _,
reason_code := 0,
reason_code_name := success
}},
emqtt:publish(Pub, Topic, Payload, [{qos, 1}])),
ok = expect_publishes(Sub, Topic, [Payload]),
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(Sub).
will_with_disconnect(Config) ->
LastWillTopic = <<"/topic/last-will">>,
LastWillMsg = <<"last will message">>,
Add missing exchange write access check when an MQTT will message is published.
2023-03-02 16:52:11 +08:00
Opts = [{will_topic, LastWillTopic},
{will_payload, LastWillMsg},
{will_qos, 1}],
Pub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_publisher">>, Config, Opts),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Sub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_subscriber">>, Config),
{ok, _, [1]} = emqtt:subscribe(Sub, LastWillTopic, qos1),
%% Client sends DISCONNECT packet. Therefore, will message should not be sent.
ok = emqtt:disconnect(Pub),
?assertEqual({publish_not_received, LastWillMsg},
expect_publishes(Sub, LastWillTopic, [LastWillMsg])),
ok = emqtt:disconnect(Sub).
will_without_disconnect(Config) ->
LastWillTopic = <<"/topic/last-will">>,
LastWillMsg = <<"last will message">>,
Add missing exchange write access check when an MQTT will message is published.
2023-03-02 16:52:11 +08:00
Opts = [{will_topic, LastWillTopic},
{will_payload, LastWillMsg},
{will_qos, 1}],
Pub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_publisher">>, Config, Opts),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Sub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_subscriber">>, Config),
{ok, _, [1]} = emqtt:subscribe(Sub, LastWillTopic, qos1),
%% Client does not send DISCONNECT packet. Therefore, will message should be sent.
unlink(Pub),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
erlang:exit(Pub, test_will),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
?assertEqual(ok, expect_publishes(Sub, LastWillTopic, [LastWillMsg])),
ok = emqtt:disconnect(Sub).
Fix Native MQTT crash if properties encoded Fixes https://github.com/rabbitmq/rabbitmq-server/discussions/8252 The MQTT connection must decode AMQP 0.9.1 properties as they are getting encoded for example in: https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_variable_queue.erl#L2219 and https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_quorum_queue.erl#L1680 Prior to this commit, the MQTT connection process could crash with: ``` [{rabbit_mqtt_processor,deliver_one_to_client, [{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>}, <0.4546.0>,undefined,false, {basic_message, {resource,<<"/">>,exchange,<<"amq.topic">>}, [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>], {content,60,none, <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110,2,0,0,0,0,100,103,141,186>>, rabbit_framing_amqp_0_9_1, [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]}, <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>, true}}, false, {state, {cfg,#Port<0.282>,mqtt311,true,undefined, {resource,<<"/">>,exchange,<<"amq.topic">>}, undefined,false,none,<0.680.0>,flow,none,10,<<"/">>, <<"mqtt-explorer-c5351d21">>,undefined, {192,168,10,131}, 1883, {192,168,10,130}, 53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>}, {rabbit_queue_type, #{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} => {ctx,rabbit_classic_queue, {rabbit_classic_queue,<0.4546.0>,#{}, #{<0.4546.0> => ok}}}}}, #{},#{},1, #{<<"#">> => 0,<<"$SYS/#">> => 0}, {auth_state, {user,<<"rabbit">>,[], [{rabbit_auth_backend_internal, #Fun<rabbit_auth_backend_internal.3.114557357>}]}, #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}}, registered,#{},0}], [{file,"rabbit_mqtt_processor.erl"},{line,1414}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, {rabbit_mqtt_processor,handle_queue_event,2, [{file,"rabbit_mqtt_processor.erl"},{line,1345}]}, {rabbit_mqtt_reader,handle_cast,2, [{file,"rabbit_mqtt_reader.erl"},{line,134}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ```
2023-05-20 01:02:57 +08:00
%% Test that an MQTT connection decodes the AMQP 0.9.1 'P_basic' properties.
%% see https://github.com/rabbitmq/rabbitmq-server/discussions/8252
decode_basic_properties(Config) ->
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
set_durable_queue_type(Config),
Fix Native MQTT crash if properties encoded Fixes https://github.com/rabbitmq/rabbitmq-server/discussions/8252 The MQTT connection must decode AMQP 0.9.1 properties as they are getting encoded for example in: https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_variable_queue.erl#L2219 and https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_quorum_queue.erl#L1680 Prior to this commit, the MQTT connection process could crash with: ``` [{rabbit_mqtt_processor,deliver_one_to_client, [{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>}, <0.4546.0>,undefined,false, {basic_message, {resource,<<"/">>,exchange,<<"amq.topic">>}, [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>], {content,60,none, <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110,2,0,0,0,0,100,103,141,186>>, rabbit_framing_amqp_0_9_1, [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]}, <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>, true}}, false, {state, {cfg,#Port<0.282>,mqtt311,true,undefined, {resource,<<"/">>,exchange,<<"amq.topic">>}, undefined,false,none,<0.680.0>,flow,none,10,<<"/">>, <<"mqtt-explorer-c5351d21">>,undefined, {192,168,10,131}, 1883, {192,168,10,130}, 53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>}, {rabbit_queue_type, #{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} => {ctx,rabbit_classic_queue, {rabbit_classic_queue,<0.4546.0>,#{}, #{<0.4546.0> => ok}}}}}, #{},#{},1, #{<<"#">> => 0,<<"$SYS/#">> => 0}, {auth_state, {user,<<"rabbit">>,[], [{rabbit_auth_backend_internal, #Fun<rabbit_auth_backend_internal.3.114557357>}]}, #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}}, registered,#{},0}], [{file,"rabbit_mqtt_processor.erl"},{line,1414}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, {rabbit_mqtt_processor,handle_queue_event,2, [{file,"rabbit_mqtt_processor.erl"},{line,1345}]}, {rabbit_mqtt_reader,handle_cast,2, [{file,"rabbit_mqtt_reader.erl"},{line,134}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ```
2023-05-20 01:02:57 +08:00
ClientId = Topic = Payload = atom_to_binary(?FUNCTION_NAME),
Fix test decode_basic_properties This commit fixes 2 separate issues: 1. No quorum queue got created in v5 because Session Expiry Interval was 0. 2. Fix a function_clause error. Pass the decoded properties further to other functions looking up headers.
2023-05-21 18:32:14 +08:00
C1 = connect(ClientId, Config, non_clean_sess_opts()),
Fix Native MQTT crash if properties encoded Fixes https://github.com/rabbitmq/rabbitmq-server/discussions/8252 The MQTT connection must decode AMQP 0.9.1 properties as they are getting encoded for example in: https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_variable_queue.erl#L2219 and https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_quorum_queue.erl#L1680 Prior to this commit, the MQTT connection process could crash with: ``` [{rabbit_mqtt_processor,deliver_one_to_client, [{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>}, <0.4546.0>,undefined,false, {basic_message, {resource,<<"/">>,exchange,<<"amq.topic">>}, [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>], {content,60,none, <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110,2,0,0,0,0,100,103,141,186>>, rabbit_framing_amqp_0_9_1, [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]}, <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>, true}}, false, {state, {cfg,#Port<0.282>,mqtt311,true,undefined, {resource,<<"/">>,exchange,<<"amq.topic">>}, undefined,false,none,<0.680.0>,flow,none,10,<<"/">>, <<"mqtt-explorer-c5351d21">>,undefined, {192,168,10,131}, 1883, {192,168,10,130}, 53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>}, {rabbit_queue_type, #{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} => {ctx,rabbit_classic_queue, {rabbit_classic_queue,<0.4546.0>,#{}, #{<0.4546.0> => ok}}}}}, #{},#{},1, #{<<"#">> => 0,<<"$SYS/#">> => 0}, {auth_state, {user,<<"rabbit">>,[], [{rabbit_auth_backend_internal, #Fun<rabbit_auth_backend_internal.3.114557357>}]}, #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}}, registered,#{},0}], [{file,"rabbit_mqtt_processor.erl"},{line,1414}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, {rabbit_mqtt_processor,handle_queue_event,2, [{file,"rabbit_mqtt_processor.erl"},{line,1345}]}, {rabbit_mqtt_reader,handle_cast,2, [{file,"rabbit_mqtt_reader.erl"},{line,134}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ```
2023-05-20 01:02:57 +08:00
{ok, _, [1]} = emqtt:subscribe(C1, Topic, qos1),
QuorumQueues = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_quorum_queue]),
?assertEqual(1, length(QuorumQueues)),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Fix Native MQTT crash if properties encoded Fixes https://github.com/rabbitmq/rabbitmq-server/discussions/8252 The MQTT connection must decode AMQP 0.9.1 properties as they are getting encoded for example in: https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_variable_queue.erl#L2219 and https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_quorum_queue.erl#L1680 Prior to this commit, the MQTT connection process could crash with: ``` [{rabbit_mqtt_processor,deliver_one_to_client, [{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>}, <0.4546.0>,undefined,false, {basic_message, {resource,<<"/">>,exchange,<<"amq.topic">>}, [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>], {content,60,none, <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110,2,0,0,0,0,100,103,141,186>>, rabbit_framing_amqp_0_9_1, [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]}, <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>, true}}, false, {state, {cfg,#Port<0.282>,mqtt311,true,undefined, {resource,<<"/">>,exchange,<<"amq.topic">>}, undefined,false,none,<0.680.0>,flow,none,10,<<"/">>, <<"mqtt-explorer-c5351d21">>,undefined, {192,168,10,131}, 1883, {192,168,10,130}, 53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>}, {rabbit_queue_type, #{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} => {ctx,rabbit_classic_queue, {rabbit_classic_queue,<0.4546.0>,#{}, #{<0.4546.0> => ok}}}}}, #{},#{},1, #{<<"#">> => 0,<<"$SYS/#">> => 0}, {auth_state, {user,<<"rabbit">>,[], [{rabbit_auth_backend_internal, #Fun<rabbit_auth_backend_internal.3.114557357>}]}, #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}}, registered,#{},0}], [{file,"rabbit_mqtt_processor.erl"},{line,1414}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, {rabbit_mqtt_processor,handle_queue_event,2, [{file,"rabbit_mqtt_processor.erl"},{line,1345}]}, {rabbit_mqtt_reader,handle_cast,2, [{file,"rabbit_mqtt_reader.erl"},{line,134}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ```
2023-05-20 01:02:57 +08:00
amqp_channel:call(Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = Topic},
#amqp_msg{payload = Payload}),
ok = expect_publishes(C1, Topic, [Payload]),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, true}]),
ok = emqtt:disconnect(C2),
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
unset_durable_queue_type(Config),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Fix Native MQTT crash if properties encoded Fixes https://github.com/rabbitmq/rabbitmq-server/discussions/8252 The MQTT connection must decode AMQP 0.9.1 properties as they are getting encoded for example in: https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_variable_queue.erl#L2219 and https://github.com/rabbitmq/rabbitmq-server/blob/712c2b9ec96dca4f07f14dfed9c6028ef0e748fd/deps/rabbit/src/rabbit_quorum_queue.erl#L1680 Prior to this commit, the MQTT connection process could crash with: ``` [{rabbit_mqtt_processor,deliver_one_to_client, [{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>}, <0.4546.0>,undefined,false, {basic_message, {resource,<<"/">>,exchange,<<"amq.topic">>}, [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>], {content,60,none, <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110,2,0,0,0,0,100,103,141,186>>, rabbit_framing_amqp_0_9_1, [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]}, <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>, true}}, false, {state, {cfg,#Port<0.282>,mqtt311,true,undefined, {resource,<<"/">>,exchange,<<"amq.topic">>}, undefined,false,none,<0.680.0>,flow,none,10,<<"/">>, <<"mqtt-explorer-c5351d21">>,undefined, {192,168,10,131}, 1883, {192,168,10,130}, 53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>}, {rabbit_queue_type, #{{resource,<<"/">>,queue, <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} => {ctx,rabbit_classic_queue, {rabbit_classic_queue,<0.4546.0>,#{}, #{<0.4546.0> => ok}}}}}, #{},#{},1, #{<<"#">> => 0,<<"$SYS/#">> => 0}, {auth_state, {user,<<"rabbit">>,[], [{rabbit_auth_backend_internal, #Fun<rabbit_auth_backend_internal.3.114557357>}]}, #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}}, registered,#{},0}], [{file,"rabbit_mqtt_processor.erl"},{line,1414}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, {rabbit_mqtt_processor,handle_queue_event,2, [{file,"rabbit_mqtt_processor.erl"},{line,1345}]}, {rabbit_mqtt_reader,handle_cast,2, [{file,"rabbit_mqtt_reader.erl"},{line,134}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ```
2023-05-20 01:02:57 +08:00
Fix mixed version Web MQTT system tests In mixed verion tests all non-required feature flags are disabled. Therefore, MQTT client ID tracking happens in Ra. The Ra client sends commands asynchronously when registering and deregistering the client ID. Also, add more tests.
2022-11-08 23:46:37 +08:00
quorum_queue_rejects(Config) ->
Add test AMQP 0.9.1 to MQTT with QoS 0
2023-01-05 23:29:42 +08:00
{_Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Fix mixed version Web MQTT system tests In mixed verion tests all non-required feature flags are disabled. Therefore, MQTT client ID tracking happens in Ra. The Ra client sends commands asynchronously when registering and deregistering the client ID. Also, add more tests.
2022-11-08 23:46:37 +08:00
Name = atom_to_binary(?FUNCTION_NAME),
ok = rabbit_ct_broker_helpers:set_policy(
Config, 0, <<"qq-policy">>, Name, <<"queues">>, [{<<"max-length">>, 1},
{<<"overflow">>, <<"reject-publish">>}]),
declare_queue(Ch, Name, [{<<"x-queue-type">>, longstr, <<"quorum">>}]),
bind(Ch, Name, Name),
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(Name, Config, [{retry_interval, 1}]),
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
{ok, #{reason_code_name := success}} = emqtt:publish(C, Name, <<"m1">>, qos1),
{ok, #{reason_code_name := success}} = emqtt:publish(C, Name, <<"m2">>, qos1),
%% The queue will reject m3.
V = ?config(mqtt_version, Config),
if V =:= v3 orelse V =:= v4 ->
%% v3 and v4 do not support NACKs. Therefore, the server should drop the message.
?assertEqual(puback_timeout, util:publish_qos1_timeout(C, Name, <<"m3">>, 700));
V =:= v5 ->
%% v5 supports NACKs. Therefore, the server should send us a NACK.
?assertMatch({ok, #{reason_code_name := implementation_specific_error}},
emqtt:publish(C, Name, <<"m3">>, qos1))
end,
Fix mixed version Web MQTT system tests In mixed verion tests all non-required feature flags are disabled. Therefore, MQTT client ID tracking happens in Ra. The Ra client sends commands asynchronously when registering and deregistering the client ID. Also, add more tests.
2022-11-08 23:46:37 +08:00
?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m1">>}},
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
amqp_channel:call(Ch, #'basic.get'{queue = Name})),
Fix mixed version Web MQTT system tests In mixed verion tests all non-required feature flags are disabled. Therefore, MQTT client ID tracking happens in Ra. The Ra client sends commands asynchronously when registering and deregistering the client ID. Also, add more tests.
2022-11-08 23:46:37 +08:00
?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m2">>}},
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
amqp_channel:call(Ch, #'basic.get'{queue = Name})),
if V =:= v3 orelse V =:= v4 ->
%% m3 is re-sent by emqtt since we didn't receive a PUBACK.
?awaitMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m3">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Name}),
2000, 200);
V =:= v5 ->
%% m3 should not be re-sent by emqtt since we received a PUBACK.
?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Name}))
end,
Fix mixed version Web MQTT system tests In mixed verion tests all non-required feature flags are disabled. Therefore, MQTT client ID tracking happens in Ra. The Ra client sends commands asynchronously when registering and deregistering the client ID. Also, add more tests.
2022-11-08 23:46:37 +08:00
ok = emqtt:disconnect(C),
delete_queue(Ch, Name),
ok = rabbit_ct_broker_helpers:clear_policy(Config, 0, <<"qq-policy">>).
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
publish_to_all_queue_types_qos0(Config) ->
publish_to_all_queue_types(Config, qos0).
publish_to_all_queue_types_qos1(Config) ->
publish_to_all_queue_types(Config, qos1).
publish_to_all_queue_types(Config, QoS) ->
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
CQ = <<"classic-queue">>,
QQ = <<"quorum-queue">>,
SQ = <<"stream-queue">>,
Topic = <<"mytopic">>,
declare_queue(Ch, CQ, []),
bind(Ch, CQ, Topic),
declare_queue(Ch, QQ, [{<<"x-queue-type">>, longstr, <<"quorum">>}]),
bind(Ch, QQ, Topic),
declare_queue(Ch, SQ, [{<<"x-queue-type">>, longstr, <<"stream">>}]),
bind(Ch, SQ, Topic),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
NumMsgs = 1000,
C = connect(?FUNCTION_NAME, Config, [{max_inflight, 200},
{retry_interval, 2}]),
Self = self(),
lists:foreach(
fun(N) ->
%% Publish async all messages at once to trigger flow control
ok = emqtt:publish_async(C, Topic, integer_to_binary(N), QoS,
{fun(N0, {ok, #{reason_code_name := success}}) ->
Self ! {self(), N0};
(N0, ok) ->
Self ! {self(), N0}
end, [N]})
end, lists:seq(1, NumMsgs)),
ok = await_confirms_ordered(C, 1, NumMsgs),
Handle duplicate packet IDs "If a Client re-sends a particular Control Packet, then it MUST use the same Packet Identifier in subsequent re-sends of that packet." A client can re-send a PUBLISH packet with the same packet ID. If the MQTT connection process already received the original packet and sent it to destination queues, it will ignore this re-send. The packet ID will be acked to the publisher once a confirmation from all target queues is received. There should be no risk of "stuck" messages within the MQTT connection process because quorum and stream queue clients re-send the message and classic queues will send a monitor DOWN message in case they are down.
2022-11-08 19:59:16 +08:00
eventually(?_assert(
begin
L = rabbitmqctl_list(Config, 0, ["list_queues", "messages", "--no-table-headers"]),
Remove classic mirror queues
2023-10-04 18:11:54 +08:00
length(L) =:= 3 andalso
Handle duplicate packet IDs "If a Client re-sends a particular Control Packet, then it MUST use the same Packet Identifier in subsequent re-sends of that packet." A client can re-send a PUBLISH packet with the same packet ID. If the MQTT connection process already received the original packet and sent it to destination queues, it will ignore this re-send. The packet ID will be acked to the publisher once a confirmation from all target queues is received. There should be no risk of "stuck" messages within the MQTT connection process because quorum and stream queue clients re-send the message and classic queues will send a monitor DOWN message in case they are down.
2022-11-08 19:59:16 +08:00
lists:all(fun([Bin]) ->
N = binary_to_integer(Bin),
case QoS of
qos0 ->
N =:= NumMsgs;
qos1 ->
%% Allow for some duplicates when client resends
%% a message that gets acked at roughly the same time.
N >= NumMsgs andalso
N < NumMsgs * 2
end
end, L)
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
end), 1000, 20),
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
Remove classic mirror queues
2023-10-04 18:11:54 +08:00
delete_queue(Ch, [CQ, QQ, SQ]),
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
ok = emqtt:disconnect(C),
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
?awaitMatch([],
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
all_connection_pids(Config), 10_000, 1000),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
publish_to_all_non_deprecated_queue_types_qos0(Config) ->
publish_to_all_non_deprecated_queue_types(Config, qos0).
publish_to_all_non_deprecated_queue_types_qos1(Config) ->
publish_to_all_non_deprecated_queue_types(Config, qos1).
publish_to_all_non_deprecated_queue_types(Config, QoS) ->
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
CQ = <<"classic-queue">>,
QQ = <<"quorum-queue">>,
SQ = <<"stream-queue">>,
Topic = <<"mytopic">>,
declare_queue(Ch, CQ, []),
bind(Ch, CQ, Topic),
declare_queue(Ch, QQ, [{<<"x-queue-type">>, longstr, <<"quorum">>}]),
bind(Ch, QQ, Topic),
declare_queue(Ch, SQ, [{<<"x-queue-type">>, longstr, <<"stream">>}]),
bind(Ch, SQ, Topic),
NumMsgs = 2000,
C = connect(?FUNCTION_NAME, Config, [{retry_interval, 2}]),
lists:foreach(fun(N) ->
case emqtt:publish(C, Topic, integer_to_binary(N), QoS) of
ok ->
ok;
{ok, _} ->
ok;
Other ->
ct:fail("Failed to publish: ~p", [Other])
end
end, lists:seq(1, NumMsgs)),
eventually(?_assert(
begin
L = rabbitmqctl_list(Config, 0, ["list_queues", "messages", "--no-table-headers"]),
length(L) =:= 3 andalso
lists:all(fun([Bin]) ->
N = binary_to_integer(Bin),
case QoS of
qos0 ->
N =:= NumMsgs;
qos1 ->
%% Allow for some duplicates when client resends
%% a message that gets acked at roughly the same time.
N >= NumMsgs andalso
N < NumMsgs * 2
end
end, L)
end), 2000, 10),
delete_queue(Ch, [CQ, QQ, SQ]),
ok = emqtt:disconnect(C),
?awaitMatch([],
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
all_connection_pids(Config), 10_000, 1000),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
Restore credit_flow between channel/MQTT connection -> CQ processes The credit_flow between publishing AMQP 0.9.1 channel (or MQTT connection) and (non-mirrored) classic queue processes was unintentionally removed in 4.0 together with anything else related to CQ mirroring. By default we restore the 3.x behaviour for non-mirored classic queues. It is possible to disable flow-control (the earlier 4.0.x behaviour) with the new env `classic_queue_flow_control`. In 3.x this was possible with the config `mirroring_flow_control`. (cherry picked from commit d65bd7d07a48b6122256e62349e8fe441e488f32)
2024-12-04 00:34:32 +08:00
%% This test case does not require multiple nodes
%% but it is grouped together with flow test cases for other queue types
%% (and historically used to use a mirrored classic queue on multiple nodes)
flow_classic_queue(Config) ->
%% New nodes lookup via persistent_term:get/1 (since 4.0.0)
%% Old nodes lookup via application:get_env/2. (that is taken care of by flow/3)
%% Therefore, we set both persistent_term and application.
Key = credit_flow_default_credit,
Val = {2, 1},
DefaultVal = rabbit_ct_broker_helpers:rpc(Config, persistent_term, get, [Key]),
Result = rpc_all(Config, persistent_term, put, [Key, Val]),
?assert(lists:all(fun(R) -> R =:= ok end, Result)),
flow(Config, {rabbit, Key, Val}, <<"classic">>),
?assertEqual(Result, rpc_all(Config, persistent_term, put, [Key, DefaultVal])),
ok.
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
flow_quorum_queue(Config) ->
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
flow(Config, {rabbit, quorum_commands_soft_limit, 1}, <<"quorum">>).
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
flow_stream(Config) ->
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
flow(Config, {rabbit, stream_messages_soft_limit, 1}, <<"stream">>).
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
flow(Config, {App, Par, Val}, QueueType)
when is_binary(QueueType) ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
{ok, DefaultVal} = rpc(Config, application, get_env, [App, Par]),
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
Result = rpc_all(Config, application, set_env, [App, Par, Val]),
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
?assert(lists:all(fun(R) -> R =:= ok end, Result)),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
QueueName = Topic = atom_to_binary(?FUNCTION_NAME),
declare_queue(Ch, QueueName, [{<<"x-queue-type">>, longstr, QueueType}]),
bind(Ch, QueueName, Topic),
NumMsgs = 1000,
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(?FUNCTION_NAME, Config, [{retry_interval, 600},
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
{max_inflight, NumMsgs}]),
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
TestPid = self(),
lists:foreach(
fun(N) ->
%% Publish async all messages at once to trigger flow control
ok = emqtt:publish_async(C, Topic, integer_to_binary(N), qos1,
{fun(N0, {ok, #{reason_code_name := success}}) ->
TestPid ! {self(), N0}
end, [N]})
end, lists:seq(1, NumMsgs)),
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
ok = await_confirms_ordered(C, 1, NumMsgs),
Add missing unblock stream queue action Test flow control for all queue types.
2022-11-14 23:57:39 +08:00
eventually(?_assertEqual(
[[integer_to_binary(NumMsgs)]],
rabbitmqctl_list(Config, 0, ["list_queues", "messages", "--no-table-headers"])
), 1000, 10),
delete_queue(Ch, QueueName),
ok = emqtt:disconnect(C),
?awaitMatch([],
all_connection_pids(Config), 10_000, 1000),
Put credit flow config into persistent term Put configuration credit_flow_default_credit into persistent term such that the tuple doesn't have to be copied on the hot path. Also, change persistent term keys from `{rabbit, AtomKey}` to `AtomKey` so that hashing becomes cheaper.
2024-05-31 20:15:41 +08:00
?assertEqual(Result,
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
rpc_all(Config, application, set_env, [App, Par, DefaultVal])),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
events(Config) ->
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
ok = rabbit_ct_broker_helpers:add_code_path_to_all_nodes(Config, event_recorder),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
Server = get_node_config(Config, 0, nodename),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
ok = gen_event:add_handler({rabbit_event, Server}, event_recorder, []),
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
ClientId = atom_to_binary(?FUNCTION_NAME),
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(ClientId, Config),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
[E0, E1] = get_events(Server),
assert_event_type(user_authentication_success, E0),
assert_event_prop([{name, <<"guest">>},
{connection_type, network}],
E0),
assert_event_type(connection_created, E1),
Output username in connection closed event
2022-11-15 23:25:59 +08:00
[ConnectionPid] = all_connection_pids(Config),
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
ProtoName = case ?config(websocket, Config) of
true -> 'Web MQTT';
false -> 'MQTT'
end,
ProtoVer = case ?config(mqtt_version, Config) of
v3 -> {3,1,0};
v4 -> {3,1,1};
v5 -> {5,0}
end,
ExpectedConnectionProps = [{protocol, {ProtoName, ProtoVer}},
{node, Server},
{vhost, <<"/">>},
{user, <<"guest">>},
{client_properties, [{client_id, longstr, ClientId}]},
{pid, ConnectionPid}],
assert_event_prop(ExpectedConnectionProps, E1),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
Qos = 0,
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
MqttTopic = <<"my/topic">>,
AmqpTopic = <<"my.topic">>,
Attempt to fix flake Attempt to fix the following flake: ``` === Ended at 2023-06-26 07:13:34 === Location: [{shared_SUITE,events,570}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: no match of right hand side value [] in function shared_SUITE:events/1 (shared_SUITE.erl, line 570) in call from test_server:ts_tc/3 (test_server.erl, line 1782) in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1291) in call from test_server:run_test_case_eval/9 (test_server.erl, line 1223) ``` of test ``` -group [web_mqtt,v3,cluster_size_1] -case events ``` The logs showed that deletion of the exclusive queue took place in the same millisecond as the server received the DISCONNECT: ``` 2023-06-26 07:13:32.838282+00:00 [debug] <0.2494.0> Received a CONNECT, client ID: events, username: undefined, clean start: true, protocol version: 3, keepalive: 60, property names: [] 2023-06-26 07:13:32.838436+00:00 [debug] <0.2494.0> MQTT connection 127.0.0.1:38808 -> 127.0.0.1:21007 picked vhost using plugin_configuration_or_default_vhost 2023-06-26 07:13:32.838523+00:00 [debug] <0.2494.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2023-06-26 07:13:32.838672+00:00 [info] <0.2494.0> Accepted Web MQTT connection 127.0.0.1:38808 -> 127.0.0.1:21007 for client ID events 2023-06-26 07:13:33.147196+00:00 [debug] <0.2494.0> Received a SUBSCRIBE with subscription(s) [{mqtt_subscription,<<"my/topic">>, 2023-06-26 07:13:33.147196+00:00 [debug] <0.2494.0> {mqtt_subscription_opts,0,false, 2023-06-26 07:13:33.147196+00:00 [debug] <0.2494.0> false,0,undefined}}] 2023-06-26 07:13:33.457541+00:00 [debug] <0.2494.0> Received an UNSUBSCRIBE for topic filter(s) [<<"my/topic">>] 2023-06-26 07:13:33.762171+00:00 [debug] <0.2494.0> Received a DISCONNECT with reason code 0 and properties #{} 2023-06-26 07:13:33.762350+00:00 [info] <0.2494.0> Web MQTT closing connection 127.0.0.1:38808 -> 127.0.0.1:21007 2023-06-26 07:13:33.762780+00:00 [debug] <0.2504.0> Deleting exclusive queue 'mqtt-subscription-eventsqos0' in vhost '/' because its declaring connection <0.2494.0> was closed ``` However, there could be some delay between disconnecting on the client WebMQTT side and the processor processing the DISCONNECT packet.
2023-06-26 15:33:02 +08:00
{ok, _, [Qos]} = emqtt:subscribe(C, MqttTopic, Qos),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
QueueNameBin = <<"mqtt-subscription-", ClientId/binary, "qos0">>,
QueueName = {resource, <<"/">>, queue, QueueNameBin},
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
[E2, E3] = get_events(Server),
Add feature flag rabbit_mqtt_qos0_queue Routing to a queue of type rabbit_mqtt_qos0_queue hosted on a remote node requires knowledge of that queue type on the local node.
2022-11-30 02:13:50 +08:00
assert_event_type(queue_created, E2),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
assert_event_prop([{name, QueueName},
{durable, true},
{auto_delete, false},
{exclusive, true},
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
{type, rabbit_mqtt_qos0_queue},
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
{arguments, []}],
E2),
assert_event_type(binding_created, E3),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ExpectedBindingArgs = case ?config(mqtt_version, Config) of
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
v5 -> [{mqtt_subscription_opts, Qos, false, false, 0, undefined},
{<<"x-binding-key">>, longstr, AmqpTopic}];
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
_ -> []
end,
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
assert_event_prop([{source_name, <<"amq.topic">>},
{source_kind, exchange},
{destination_name, QueueNameBin},
{destination_kind, queue},
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
{routing_key, AmqpTopic},
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
{arguments, ExpectedBindingArgs}],
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
E3),
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
{ok, _, _} = emqtt:unsubscribe(C, MqttTopic),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
[E4] = get_events(Server),
assert_event_type(binding_deleted, E4),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
ok = emqtt:disconnect(C),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
[E5, E6] = get_events(Server),
assert_event_type(connection_closed, E5),
?assertEqual(E1#event.props, E5#event.props,
Add MQTT client id to connection closed event As requested in https://github.com/rabbitmq/rabbitmq-server/discussions/6331#discussioncomment-5796154 include all infos that were emitted in the MQTT connection created event also in the MQTT connection closed event. This ensures infos such as MQTT client ID are part of the connection closed event. Therefore, it's easy for the user to correlate between the two event types. Note that the MQTT plugin emits connection created and connection closed events only if the CONNECT packet was successfully processed, i.e.authentication was successful. Remove the disconnected_at property because it was never used. rabbit_event already adds a timestamp to any event.
2023-05-04 16:51:30 +08:00
"connection_closed event props should match connection_created event props. "
"See https://github.com/rabbitmq/rabbitmq-server/discussions/6331"),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
assert_event_type(queue_deleted, E6),
assert_event_prop({name, QueueName}, E6),
Add rabbit_event tests for MQTT Add tests that MQTT plugin sends correct events to rabbit_event. Add event connection_closed.
2022-10-28 01:07:43 +08:00
ok = gen_event:delete_handler({rabbit_event, Server}, event_recorder, []).
Simplify counters by storing mqtt310 and mqtt311 atoms directly in the processor state.
2022-11-14 00:23:03 +08:00
internal_event_handler(Config) ->
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
Server = get_node_config(Config, 0, nodename),
Simplify counters by storing mqtt310 and mqtt311 atoms directly in the processor state.
2022-11-14 00:23:03 +08:00
ok = gen_event:call({rabbit_event, Server}, rabbit_mqtt_internal_event_handler, ignored_request, 1000).
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
global_counters(Config) ->
C = connect(?FUNCTION_NAME, Config),
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
Subs from 1 connection counts as 1 consumer in global counter - rename proc state isPublisher to has_published - create macro for v3 and v4 mqtt protocol name for global counters - sub groups in integration suite
2022-11-11 20:26:55 +08:00
Topic0 = <<"test-topic0">>,
Topic1 = <<"test-topic1">>,
Implement message consuming counters in mqtt
2022-11-21 23:25:34 +08:00
Topic2 = <<"test-topic2">>,
{ok, _, [0]} = emqtt:subscribe(C, Topic0, qos0),
Subs from 1 connection counts as 1 consumer in global counter - rename proc state isPublisher to has_published - create macro for v3 and v4 mqtt protocol name for global counters - sub groups in integration suite
2022-11-11 20:26:55 +08:00
{ok, _, [1]} = emqtt:subscribe(C, Topic1, qos1),
Implement message consuming counters in mqtt
2022-11-21 23:25:34 +08:00
{ok, _, [1]} = emqtt:subscribe(C, Topic2, qos1),
Fix global counters Prior to this commit messages_delivered for queue_type_qos0 is wrongfully incremented if clean session is false Also, delete duplicate code.
2022-11-24 21:42:45 +08:00
Implement message consuming counters in mqtt
2022-11-21 23:25:34 +08:00
ok = emqtt:publish(C, Topic0, <<"testm0">>, qos0),
ok = emqtt:publish(C, Topic1, <<"testm1">>, qos0),
{ok, _} = emqtt:publish(C, Topic2, <<"testm2">>, qos1),
Fix global counters Prior to this commit messages_delivered for queue_type_qos0 is wrongfully incremented if clean session is false Also, delete duplicate code.
2022-11-24 21:42:45 +08:00
ok = emqtt:publish(C, <<"no/queue/bound">>, <<"msg-dropped">>, qos0),
PUBACK with reason code "No matching subscribers" Support reason code "No matching subscribers" in PUBACK. This somewhat corresponds to the `mandatory` message property in AMQP 0.9.1.
2023-06-06 01:23:46 +08:00
{ok, Pub} = emqtt:publish(C, <<"no/queue/bound">>, <<"msg-returned">>, qos1),
case ?config(mqtt_version, Config) of
v3 -> ok;
v4 -> ok;
v5 -> ?assertMatch(#{reason_code_name := no_matching_subscribers}, Pub)
end,
Fix global counters Prior to this commit messages_delivered for queue_type_qos0 is wrongfully incremented if clean session is false Also, delete duplicate code.
2022-11-24 21:42:45 +08:00
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic0, [<<"testm0">>]),
ok = expect_publishes(C, Topic1, [<<"testm1">>]),
ok = expect_publishes(C, Topic2, [<<"testm2">>]),
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
ProtoVer = ?config(mqtt_version, Config),
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
?assertEqual(#{publishers => 1,
consumers => 1,
Fix global counters Prior to this commit messages_delivered for queue_type_qos0 is wrongfully incremented if clean session is false Also, delete duplicate code.
2022-11-24 21:42:45 +08:00
messages_confirmed_total => 2,
messages_received_confirm_total => 2,
messages_received_total => 5,
Implement message consuming counters in mqtt
2022-11-21 23:25:34 +08:00
messages_routed_total => 3,
Fix global counters Prior to this commit messages_delivered for queue_type_qos0 is wrongfully incremented if clean session is false Also, delete duplicate code.
2022-11-24 21:42:45 +08:00
messages_unroutable_dropped_total => 1,
messages_unroutable_returned_total => 1},
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
get_global_counters(Config, ProtoVer)),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
?assertEqual(#{messages_delivered_total => 2,
messages_acknowledged_total => 1,
messages_delivered_consume_auto_ack_total => 1,
messages_delivered_consume_manual_ack_total => 1,
messages_delivered_get_auto_ack_total => 0,
messages_delivered_get_manual_ack_total => 0,
messages_get_empty_total => 0,
messages_redelivered_total => 0},
get_global_counters(Config, ProtoVer, 0, [{queue_type, rabbit_classic_queue}])),
?assertEqual(#{messages_delivered_total => 1,
messages_acknowledged_total => 0,
messages_delivered_consume_auto_ack_total => 1,
messages_delivered_consume_manual_ack_total => 0,
messages_delivered_get_auto_ack_total => 0,
messages_delivered_get_manual_ack_total => 0,
messages_get_empty_total => 0,
messages_redelivered_total => 0},
get_global_counters(Config, ProtoVer, 0, [{queue_type, rabbit_mqtt_qos0_queue}])),
Implement message consuming counters in mqtt
2022-11-21 23:25:34 +08:00
Subs from 1 connection counts as 1 consumer in global counter - rename proc state isPublisher to has_published - create macro for v3 and v4 mqtt protocol name for global counters - sub groups in integration suite
2022-11-11 20:26:55 +08:00
{ok, _, _} = emqtt:unsubscribe(C, Topic1),
?assertEqual(1, maps:get(consumers, get_global_counters(Config, ProtoVer))),
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
Subs from 1 connection counts as 1 consumer in global counter - rename proc state isPublisher to has_published - create macro for v3 and v4 mqtt protocol name for global counters - sub groups in integration suite
2022-11-11 20:26:55 +08:00
ok = emqtt:disconnect(C),
Fix flaky test Very rarely, the assertion failed with ``` === Ended at 2023-02-13 13:25:52 === Location: [{shared_SUITE,global_counters,590}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {assertEqual, [{module,shared_SUITE}, {line,590}, {expression,"get_global_counters ( Config , ProtoVer )"}, {expected, #{consumers => 0,messages_confirmed_total => 2, messages_received_confirm_total => 2, messages_received_total => 5, messages_routed_total => 3, messages_unroutable_dropped_total => 1, messages_unroutable_returned_total => 1, publishers => 0}}, {value, #{consumers => 1,messages_confirmed_total => 2, messages_received_confirm_total => 2, messages_received_total => 5, messages_routed_total => 3, messages_unroutable_dropped_total => 1, messages_unroutable_returned_total => 1, publishers => 1}}]} in function shared_SUITE:global_counters/2 (shared_SUITE.erl, line 590) in call from test_server:ts_tc/3 (test_server.erl, line 1782) in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1291) in call from test_server:run_test_case_eval/9 (test_server.erl, line 1223) ``` The DISCONNECT packet is sent one-way from client to server.
2023-02-13 22:14:14 +08:00
eventually(?_assertEqual(#{publishers => 0,
consumers => 0,
messages_confirmed_total => 2,
messages_received_confirm_total => 2,
messages_received_total => 5,
messages_routed_total => 3,
messages_unroutable_dropped_total => 1,
messages_unroutable_returned_total => 1},
get_global_counters(Config, ProtoVer))).
Set common global counters for mqtt - protocols are set to mqtt301 or mqtt311 depends on protocal version set by client connection - added boolean isPublisher in proc state to track if connection was ever used to publish. This is used to set publisher_create and publisher_delete global counters. - tests added in integration_SUITE
2022-11-10 23:52:46 +08:00
Emit histogram metric for received message sizes per protocol (#12342) * Add global histogram metrics for received message sizes per-protocol fixup: add new files to bazel fixup: expose message_size_bytes as prometheus classic histogram type `rabbit_msg_size_metrics` does not use `seshat` any more, but `counters` directly. fixup: add msg_size_metrics unit test * Improve message size histogram 1. Avoid unnecessary time series emitted for stream protocol The stream protocol cannot observe message sizes. This commit ensures that the following time series are omitted: ``` rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="64"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="256"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="1024"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="4096"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="16384"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="65536"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="262144"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="1048576"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="4194304"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="16777216"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="67108864"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="268435456"} 0 rabbitmq_global_message_size_bytes_bucket{protocol="stream",le="+Inf"} 0 rabbitmq_global_message_size_bytes_count{protocol="stream"} 0 rabbitmq_global_message_size_bytes_sum{protocol="stream"} 0 ``` This reduces the number of time series by 15. 2. Further reduce the number of time series by reducing the number of buckets. Instead of 13 bucktes, emit only 9 buckets. Buckets are not free, each is an extra time series stored. Prior to this commit: ``` curl -s -u guest:guest localhost:15692/metrics | ag message_size | wc -l 92 ``` After this commit: ``` curl -s -u guest:guest localhost:15692/metrics | ag message_size | wc -l 57 ``` 3. The emitted metric should be called `rabbitmq_message_size_bytes_bucket` instead of `rabbitmq_global_message_size_bytes_bucket`. The latter is poor naming. There is no need to use `global` in the metric name given that this metric doesn't exist in the old flawed aggregated metrics. 4. This commit simplies module `rabbit_global_counters`. 5. Avoid garbage collecting the 10-elements list of buckets per message being received. --------- Co-authored-by: Péter Gömöri <peter@84codes.com>
2024-09-25 00:08:24 +08:00
message_size_metrics(Config) ->
Protocol = case ?config(mqtt_version, Config) of
v4 -> mqtt311;
v5 -> mqtt50
end,
BucketsBefore = rpc(Config, rabbit_msg_size_metrics, raw_buckets, [Protocol]),
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [0]} = emqtt:subscribe(C, Topic, qos0),
Payload1B = <<255>>,
Payload500B = binary:copy(Payload1B, 500),
Payload5KB = binary:copy(Payload1B, 5_000),
Payload2MB = binary:copy(Payload1B, 2_000_000),
Payloads = [Payload2MB, Payload5KB, Payload500B, Payload1B, Payload500B],
[ok = emqtt:publish(C, Topic, P, qos0) || P <- Payloads],
ok = expect_publishes(C, Topic, Payloads),
BucketsAfter = rpc(Config, rabbit_msg_size_metrics, raw_buckets, [Protocol]),
?assertEqual(
[{100, 1},
{1000, 2},
{10_000, 1},
{10_000_000, 1}],
rabbit_msg_size_metrics:diff_raw_buckets(BucketsAfter, BucketsBefore)),
ok = emqtt:disconnect(C).
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
pubsub(Config) ->
Topic0 = <<"t/0">>,
Topic1 = <<"t/1">>,
C1 = connect(<<"c1">>, Config, 1, []),
{ok, _, [1]} = emqtt:subscribe(C1, Topic1, qos1),
Fix Khepri flake Previously, test pubsub was flaky ``` {shared_SUITE,pubsub,766} {test_case_failed,missing m1} ``` because the binding wasn't present yet on node 0 when publishing to node 0.
2023-11-03 16:52:53 +08:00
C0 = connect(<<"c0">>, Config, 0, []),
{ok, _, [1]} = emqtt:subscribe(C0, Topic0, qos1),
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
{ok, _} = emqtt:publish(C0, Topic1, <<"m1">>, qos1),
receive {publish, #{client_pid := C1,
qos := 1,
payload := <<"m1">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing m1")
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
end,
ok = emqtt:publish(C0, Topic1, <<"m2">>, qos0),
receive {publish, #{client_pid := C1,
qos := 0,
payload := <<"m2">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing m2")
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
end,
{ok, _} = emqtt:publish(C1, Topic0, <<"m3">>, qos1),
receive {publish, #{client_pid := C0,
qos := 1,
payload := <<"m3">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing m3")
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
end,
ok = emqtt:publish(C1, Topic0, <<"m4">>, qos0),
receive {publish, #{client_pid := C0,
qos := 0,
payload := <<"m4">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing m4")
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
end,
ok = emqtt:disconnect(C0),
ok = emqtt:disconnect(C1).
Add tests
2022-11-11 02:55:47 +08:00
queue_down_qos1(Config) ->
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
{Conn1, Ch1} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 1),
Add tests
2022-11-11 02:55:47 +08:00
CQ = Topic = atom_to_binary(?FUNCTION_NAME),
declare_queue(Ch1, CQ, []),
bind(Ch1, CQ, Topic),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn1, Ch1),
Add tests
2022-11-11 02:55:47 +08:00
ok = rabbit_ct_broker_helpers:stop_node(Config, 1),
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(?FUNCTION_NAME, Config, [{retry_interval, 2}]),
Add tests
2022-11-11 02:55:47 +08:00
%% classic queue is down, therefore message is rejected
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
V = ?config(mqtt_version, Config),
if V =:= v3 orelse V =:= v4 ->
?assertEqual(puback_timeout, util:publish_qos1_timeout(C, Topic, <<"msg">>, 500)),
ok = rabbit_ct_broker_helpers:start_node(Config, 1),
%% Classic queue is up. Therefore, message should arrive.
eventually(?_assertEqual([[<<"1">>]],
rabbitmqctl_list(Config, 1, ["list_queues", "messages", "--no-table-headers"])),
500, 20);
V =:= v5 ->
?assertMatch({ok, #{reason_code_name := implementation_specific_error}},
emqtt:publish(C, Topic, <<"msg">>, qos1)),
ok = rabbit_ct_broker_helpers:start_node(Config, 1)
end,
Add tests
2022-11-11 02:55:47 +08:00
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch0} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
Add tests
2022-11-11 02:55:47 +08:00
delete_queue(Ch0, CQ),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch0).
Add tests
2022-11-11 02:55:47 +08:00
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
%% Consuming classic queue on a different node goes down.
consuming_classic_queue_down(Config) ->
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
[Server1, _Server2, Server3] = rabbit_ct_broker_helpers:get_node_configs(Config, nodename),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
ClientId = Topic = atom_to_binary(?FUNCTION_NAME),
%% Declare classic queue on Server1.
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C1 = connect(ClientId, Config, non_clean_sess_opts()),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
{ok, _, _} = emqtt:subscribe(C1, Topic, qos1),
ok = emqtt:disconnect(C1),
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
%% Consume from Server3.
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C2 = connect(ClientId, Config, Server3, non_clean_sess_opts()),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
ProtoVer = ?config(mqtt_version, Config),
Implement consumer global counter for clean sess false - remove has_subs from proc state; query datebase to check if a connection has subscription or not
2022-11-18 02:24:00 +08:00
?assertMatch(#{consumers := 1},
get_global_counters(Config, ProtoVer, Server3)),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
%% Let's stop the queue leader node.
process_flag(trap_exit, true),
ok = rabbit_ct_broker_helpers:stop_node(Config, Server1),
Remove classic mirror queues
2023-10-04 18:11:54 +08:00
%% When the dedicated MQTT connection queue goes down, it is reasonable
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
%% that the server closes the MQTT connection because the MQTT client cannot consume anymore.
eventually(?_assertMatch(#{consumers := 0},
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
get_global_counters(Config, ProtoVer, Server3)),
1000, 5),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
await_exit(C2),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
%% Cleanup
ok = rabbit_ct_broker_helpers:start_node(Config, Server1),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
C3 = connect(ClientId, Config, Server3, [{clean_start, true}]),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
ok = emqtt:disconnect(C3),
?assertEqual([],
Reset application env in MQTT flow tests so that they run independently of other tests
2022-11-17 19:48:56 +08:00
rpc(Config, Server1, rabbit_amqqueue, list, [])),
Add test for consuming MQTT classic queue going down
2022-11-17 18:22:31 +08:00
ok.
Support consuming classic mirrored queue failover Some users use classic mirrored queues for MQTT queues by applying a policy. Given that classic mirrored queues are deprecated, but still supported in RabbitMQ 3.x, native MQTT must support classic mirrored queues.
2022-11-17 00:20:17 +08:00
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
delete_create_queue(Config) ->
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
CQ1 = <<"classic-queue-1-delete-create">>,
CQ2 = <<"classic-queue-2-delete-create">>,
QQ = <<"quorum-queue-delete-create">>,
Topic = atom_to_binary(?FUNCTION_NAME),
DeclareQueues = fun() ->
declare_queue(Ch, CQ1, []),
bind(Ch, CQ1, Topic),
declare_queue(Ch, CQ2, []),
bind(Ch, CQ2, Topic),
declare_queue(Ch, QQ, [{<<"x-queue-type">>, longstr, <<"quorum">>}]),
bind(Ch, QQ, Topic)
end,
DeclareQueues(),
%% some large retry_interval to avoid re-sending
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(?FUNCTION_NAME, Config, [{retry_interval, 300}]),
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
NumMsgs = 50,
TestPid = self(),
spawn(
fun() ->
lists:foreach(
fun(N) ->
ok = emqtt:publish_async(C, Topic, integer_to_binary(N), qos1,
{fun(N0, {ok, #{reason_code_name := success}}) ->
TestPid ! {self(), N0}
end, [N]})
end, lists:seq(1, NumMsgs))
end),
%% Delete queues while sending to them.
%% We want to test the path where a queue is deleted while confirms are outstanding.
timer:sleep(2),
delete_queue(Ch, [CQ1, QQ]),
%% Give queues some time to be fully deleted
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
%% TODO: wait longer for quorum queues in mixed mode as it can take longer
%% for deletion to complete, delete timeout is 5s so we need to exceed that
timer:sleep(6000),
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
%% We expect confirms for all messages.
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
%% Confirm here does not mean that messages made it ever to the deleted queues.
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
%% It is valid for confirms to sporadically arrive out of order: This happens when the classic
%% queue is being deleted while the remaining messages are routed and confirmed to the 2nd and 3rd queues
%% before the monitor to the classic queue fires.
ok = await_confirms_unordered(C, NumMsgs),
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
%% Recreate the same queues.
DeclareQueues(),
%% Sending a message to each of them should work.
{ok, _} = emqtt:publish(C, Topic, <<"m">>, qos1),
eventually(?_assertEqual(lists:sort([[CQ1, <<"1">>],
%% This queue should have all messages because we did not delete it.
[CQ2, integer_to_binary(NumMsgs + 1)],
[QQ, <<"1">>]]),
lists:sort(rabbitmqctl_list(Config, 0, ["list_queues", "name", "messages", "--no-table-headers"]))),
1000, 10),
delete_queue(Ch, [CQ1, CQ2, QQ]),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
session_expiry(Config) ->
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
App = rabbitmq_mqtt,
Make mqtt.subscription_ttl unsupported Starting with RabbitMQ 3.13 mqtt.max_session_expiry_interval_seconds (set in seconds) will replace the previous setting mqtt.subscription_ttl. MQTT 5.0 introduces the Session Expiry Interval feature which does not only apply to subscribers, but also to publishers. The new config name mqtt.max_session_expiry_interval_seconds makes it clear that it also applies to publishers. Prior to this commit, when mqtt.subscription_ttl was set, a warning got logged and the value was ignored. This is dangerous if an operator does not see the warning but relies for example on `mqtt.subscription = infinity` to not expire non clean session. It's safer to make the boot fail if that unsupported config name is still set. A clear error message will be logged: ``` [error] <0.142.0> Error preparing configuration in phase apply_translations: [error] <0.142.0> - Translation for 'rabbitmq_mqtt.subscription_ttl' found invalid configuration: Since 3.13 mqtt.subscription_ttl (in milliseconds) is unsupported. Use mqtt.max_session_expiry_interval_seconds (in seconds) instead. ``` Alternatively, RabbitMQ could translate mqtt.subscription_ttl to mqtt.max_session_expiry_interval_seconds. However, forcing the new config option sounds the better way to go. Once we write MQTT 5.0 docs, this change must go into the 3.13 release notes. This commit also renames max_session_expiry_interval_secs to max_session_expiry_interval_seconds. The latter is clearer to users.
2023-07-13 22:38:47 +08:00
Par = max_session_expiry_interval_seconds,
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Seconds = 1,
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
{ok, DefaultVal} = rpc(Config, application, get_env, [App, Par]),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ok = rpc(Config, application, set_env, [App, Par, Seconds]),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C = connect(?FUNCTION_NAME, Config, non_clean_sess_opts()),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
{ok, _, [0, 1]} = emqtt:subscribe(C, [{<<"topic0">>, qos0},
{<<"topic1">>, qos1}]),
ok = emqtt:disconnect(C),
?assertEqual(2, rpc(Config, rabbit_amqqueue, count, [])),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
timer:sleep(timer:seconds(Seconds) + 100),
mqtt shared_SUITE: another case of a time dependent test Longer wait for expiry. On slow machines, as CI, we can't guarantee that the expiration will happen at the exact time
2023-08-03 16:37:53 +08:00
%% On a slow machine, this test might fail. Let's consider
%% the expiry on a longer time window
?awaitMatch(0, rpc(Config, rabbit_amqqueue, count, []), 15_000, 1000),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
ok = rpc(Config, application, set_env, [App, Par, DefaultVal]).
non_clean_sess_reconnect_qos1(Config) ->
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
non_clean_sess_reconnect(Config, 1).
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
non_clean_sess_reconnect_qos0(Config) ->
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
non_clean_sess_reconnect(Config, 0).
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
non_clean_sess_reconnect(Config, SubscriptionQoS) ->
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
Pub = connect(<<"publisher">>, Config),
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C1 = connect(ClientId, Config, non_clean_sess_opts()),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
{ok, _, [SubscriptionQoS]} = emqtt:subscribe(C1, Topic, SubscriptionQoS),
ok = await_consumer_count(1, ClientId, SubscriptionQoS, Config),
Implement consumer global counter for clean sess false - remove has_subs from proc state; query datebase to check if a connection has subscription or not
2022-11-18 02:24:00 +08:00
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
ok = emqtt:disconnect(C1),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
ok = await_consumer_count(0, ClientId, SubscriptionQoS, Config),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
ok = emqtt:publish(Pub, Topic, <<"msg-3-qos0">>, qos0),
{ok, _} = emqtt:publish(Pub, Topic, <<"msg-4-qos1">>, qos1),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C2 = connect(ClientId, Config, non_clean_sess_opts()),
%% Server should reply in CONNACK that it has session state.
?assertEqual({session_present, 1},
proplists:lookup(session_present, emqtt:info(C2))),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
ok = await_consumer_count(1, ClientId, SubscriptionQoS, Config),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
ok = emqtt:publish(Pub, Topic, <<"msg-5-qos0">>, qos0),
{ok, _} = emqtt:publish(Pub, Topic, <<"msg-6-qos1">>, qos1),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
%% shouldn't receive message after unsubscribe
{ok, _, _} = emqtt:unsubscribe(C2, Topic),
Implement consumer global counter for clean sess false - remove has_subs from proc state; query datebase to check if a connection has subscription or not
2022-11-18 02:24:00 +08:00
?assertMatch(#{consumers := 0},
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
get_global_counters(Config)),
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
{ok, _} = emqtt:publish(Pub, Topic, <<"msg-7-qos0">>, qos1),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
%% "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store
%% further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the
%% time of disconnection as part of the Session state [MQTT-3.1.2-5].
%% It MAY also store QoS 0 messages that meet the same criteria."
%% Starting with RabbitMQ v3.12 we store QoS 0 messages as well.
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
ok = expect_publishes(C2, Topic, [<<"msg-3-qos0">>, <<"msg-4-qos1">>,
<<"msg-5-qos0">>, <<"msg-6-qos1">>]),
{publish_not_received, <<"msg-7-qos0">>} = expect_publishes(C2, Topic, [<<"msg-7-qos0">>]),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(C2),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
%% connect with clean sess true to clean up
Expand clean_session=false test
2023-01-10 17:20:19 +08:00
C3 = connect(ClientId, Config, [{clean_start, true}]),
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
ok = emqtt:disconnect(C3).
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
non_clean_sess_reconnect_qos0_and_qos1(Config) ->
Pub = connect(<<"publisher">>, Config),
Topic0 = <<"t/0">>,
Topic1 = <<"t/1">>,
ClientId = ?FUNCTION_NAME,
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C1 = connect(ClientId, Config, non_clean_sess_opts()),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
{ok, _, [1, 0]} = emqtt:subscribe(C1, [{Topic1, qos1},
{Topic0, qos0}]),
ok = await_consumer_count(1, ClientId, 0, Config),
ok = await_consumer_count(1, ClientId, 1, Config),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
ok = emqtt:disconnect(C1),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
ok = await_consumer_count(0, ClientId, 0, Config),
ok = await_consumer_count(0, ClientId, 1, Config),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
{ok, _} = emqtt:publish(Pub, Topic0, <<"msg-0">>, qos1),
{ok, _} = emqtt:publish(Pub, Topic1, <<"msg-1">>, qos1),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C2 = connect(ClientId, Config, non_clean_sess_opts()),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
ok = await_consumer_count(1, ClientId, 0, Config),
ok = await_consumer_count(1, ClientId, 1, Config),
Fix "Clean Session" state for QoS 0 subscriptions Up to RabbitMQ 3.11, the following bug existed. The MQTT 3.1.1. protocol spec mandates: ``` The Session state in the Server consists of: * The Client’s subscriptions. * ... ``` However, because QoS 0 queues were auto-delete up to 3.11 (or exclusive prior to this commit), QoS 0 queues and therefore their bindings were deleted when a non-clean session terminated. When the same client re-connected, its QoS 0 subscription was lost. Note that storing **messages** for QoS 0 subscription is optional while the client is disconnected. However, storing the subscription itself (i.e. bindings in RabbitMQ terms) is not optional: The client must receive new messages for its QoS 0 subscriptions after it reconnects without having to send a SUBSCRIBE packet again. "After the disconnection of a Session that had CleanSession set to 0, the Server MUST store further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the Session state [MQTT-3.1.2-5]. It MAY also store QoS 0 messages that meet the same criteria." This commit additionally implements the last sentence.
2023-01-14 00:15:15 +08:00
ok = expect_publishes(C2, Topic0, [<<"msg-0">>]),
ok = expect_publishes(C2, Topic1, [<<"msg-1">>]),
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(C2),
C3 = connect(ClientId, Config, [{clean_start, true}]),
ok = emqtt:disconnect(C3).
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
Web MQTT: Send CONNACK error code before closing connection "If the Client supplies a zero-byte ClientId with CleanSession set to 0, the Server MUST respond to the CONNECT Packet with a CONNACK return code 0x02 (Identifier rejected) and then close the Network Connection" [MQTT-3.1.3-8]. In Web MQTT, the CONNACK was not sent to the client because the Web MQTT connection process terminated before being sending the CONNACK to the client.
2023-02-28 16:45:15 +08:00
non_clean_sess_empty_client_id(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
{C, Connect} = util:start_client(<<>>, Config, 0, non_clean_sess_opts()),
Return Assigned Client Identifier in CONNACK "If the Client connects using a zero length Client Identifier, the Server MUST respond with a CONNACK containing an Assigned Client Identifier."
2023-02-28 21:51:16 +08:00
case ?config(mqtt_version, Config) of
V when V =:= v3;
V =:= v4 ->
%% "If the Client supplies a zero-byte ClientId with CleanSession set to 0,
%% the Server MUST respond to the CONNECT Packet with a CONNACK return code 0x02
%% (Identifier rejected) and then close the Network Connection" [MQTT-3.1.3-8].
process_flag(trap_exit, true),
?assertMatch({error, {client_identifier_not_valid, _}}, Connect(C)),
ok = await_exit(C);
v5 ->
%% "If the Client connects using a zero length Client Identifier, the Server MUST respond with
%% a CONNACK containing an Assigned Client Identifier. The Assigned Client Identifier MUST be
%% a new Client Identifier not used by any other Session currently in the Server [MQTT-3.2.2-16]."
{ok, #{'Assigned-Client-Identifier' := ClientId}} = Connect(C),
{C2, Connect2} = util:start_client(<<>>, Config, 0, [{clean_start, true}]),
{ok, #{'Assigned-Client-Identifier' := ClientId2}} = Connect2(C2),
?assertNotEqual(ClientId, ClientId2),
ok = emqtt:disconnect(C),
ok = emqtt:disconnect(C2)
end.
Web MQTT: Send CONNACK error code before closing connection "If the Client supplies a zero-byte ClientId with CleanSession set to 0, the Server MUST respond to the CONNECT Packet with a CONNACK return code 0x02 (Identifier rejected) and then close the Network Connection" [MQTT-3.1.3-8]. In Web MQTT, the CONNACK was not sent to the client because the Web MQTT connection process terminated before being sending the CONNACK to the client.
2023-02-28 16:45:15 +08:00
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
subscribe_same_topic_same_qos(Config) ->
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(?FUNCTION_NAME, Config),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
Topic = <<"a/b">>,
{ok, _} = emqtt:publish(C, Topic, <<"retained">>, [{retain, true},
{qos, 1}]),
%% Subscribe with QoS 0
{ok, _, [0]} = emqtt:subscribe(C, Topic, qos0),
{ok, _} = emqtt:publish(C, Topic, <<"msg1">>, qos1),
%% Subscribe to same topic with same QoS
{ok, _, [0]} = emqtt:subscribe(C, Topic, qos0),
{ok, _} = emqtt:publish(C, Topic, <<"msg2">>, qos1),
%% "Any existing retained messages matching the Topic Filter MUST be re-sent" [MQTT-3.8.4-3]
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic, [<<"retained">>, <<"msg1">>,
<<"retained">>, <<"msg2">>
]),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
ok = emqtt:disconnect(C).
subscribe_same_topic_different_qos(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C = connect(?FUNCTION_NAME, Config, non_clean_sess_opts()),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
Topic = <<"b/c">>,
{ok, _} = emqtt:publish(C, Topic, <<"retained">>, [{retain, true},
{qos, 1}]),
%% Subscribe with QoS 0
{ok, _, [0]} = emqtt:subscribe(C, Topic, qos0),
{ok, _} = emqtt:publish(C, Topic, <<"msg1">>, qos1),
%% Subscribe to same topic with QoS 1
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"msg2">>, qos1),
%% Subscribe to same topic with QoS 0 again
{ok, _, [0]} = emqtt:subscribe(C, Topic, qos0),
{ok, _} = emqtt:publish(C, Topic, <<"msg3">>, qos1),
%% "Any existing retained messages matching the Topic Filter MUST be re-sent" [MQTT-3.8.4-3]
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic, [<<"retained">>, <<"msg1">>,
<<"retained">>, <<"msg2">>,
<<"retained">>, <<"msg3">>]),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
Consume from queue once Each MQTT connection consumes from its queue at most once (unless when failing over).
2022-11-22 01:26:40 +08:00
%% There should be exactly one consumer for each queue: qos0 and qos1
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Consumers = rpc(Config, rabbit_amqqueue, consumers_all, [<<"/">>]),
Consume from queue once Each MQTT connection consumes from its queue at most once (unless when failing over).
2022-11-22 01:26:40 +08:00
?assertEqual(2, length(Consumers)),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
ok = emqtt:disconnect(C),
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C1 = connect(?FUNCTION_NAME, Config, [{clean_start, true}]),
Replace existing subscription "If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3]."
2022-11-21 19:31:00 +08:00
ok = emqtt:disconnect(C1).
Avoid exceptions in mixed version cluster 1. Avoid following exceptions in mixed version clusters when new MQTT connections are created: ``` {{exception,{undef,[{rabbit_mqtt_util,remove_duplicate_clientid_connections, [{<<"/">>, <<"publish_to_all_queue_types">>}, <0.1447.0>], []}]}}, [{erpc,execute_cast,3,[{file,"erpc.erl"},{line,621}]}]} ``` If feature flag delete_ra_cluster_mqtt_node is disabled, let's still populate pg with MQTT client IDs such that we don't have to migrate them from the Ra cluster to pg when we enable the feature flag. However, for actually closing duplicate MQTT client ID connections, if that feature flag is disabled, let's rely on the Ra cluster to take care of it. 2. Write a test ensuring the QoS responses are in the right order when a single SUBSCRIBE packet contains multiple subscriptions.
2022-11-24 00:44:42 +08:00
subscribe_multiple(Config) ->
Use connect helper func in more mqtt tests - reduce code duplication - connect helper does not unlink the connection process by default
2022-11-29 19:41:50 +08:00
C = connect(?FUNCTION_NAME, Config),
Avoid exceptions in mixed version cluster 1. Avoid following exceptions in mixed version clusters when new MQTT connections are created: ``` {{exception,{undef,[{rabbit_mqtt_util,remove_duplicate_clientid_connections, [{<<"/">>, <<"publish_to_all_queue_types">>}, <0.1447.0>], []}]}}, [{erpc,execute_cast,3,[{file,"erpc.erl"},{line,621}]}]} ``` If feature flag delete_ra_cluster_mqtt_node is disabled, let's still populate pg with MQTT client IDs such that we don't have to migrate them from the Ra cluster to pg when we enable the feature flag. However, for actually closing duplicate MQTT client ID connections, if that feature flag is disabled, let's rely on the Ra cluster to take care of it. 2. Write a test ensuring the QoS responses are in the right order when a single SUBSCRIBE packet contains multiple subscriptions.
2022-11-24 00:44:42 +08:00
%% Subscribe to multiple topics at once
?assertMatch({ok, _, [0, 1]},
emqtt:subscribe(C, [{<<"topic0">>, qos0},
{<<"topic1">>, qos1}])),
ok = emqtt:disconnect(C).
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
large_message_mqtt_to_mqtt(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [1]} = emqtt:subscribe(C, {Topic, qos1}),
Increase MQTT test coverage and fix edge cases
2022-12-29 21:01:55 +08:00
Payload0 = binary:copy(<<"x">>, 8_000_000),
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
Payload = <<Payload0/binary, "y">>,
{ok, _} = emqtt:publish(C, Topic, Payload, qos1),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic, [Payload]),
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
ok = emqtt:disconnect(C).
large_message_amqp_to_mqtt(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [1]} = emqtt:subscribe(C, {Topic, qos1}),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Increase MQTT test coverage and fix edge cases
2022-12-29 21:01:55 +08:00
Payload0 = binary:copy(<<"x">>, 8_000_000),
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
Payload = <<Payload0/binary, "y">>,
Add test AMQP 0.9.1 to MQTT with QoS 0
2023-01-05 23:29:42 +08:00
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = Topic},
#amqp_msg{payload = Payload}),
ok = expect_publishes(C, Topic, [Payload]),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Add test AMQP 0.9.1 to MQTT with QoS 0
2023-01-05 23:29:42 +08:00
amqp_to_mqtt_qos0(Config) ->
Topic = ClientId = Payload = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [0]} = emqtt:subscribe(C, {Topic, qos0}),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = Topic},
#amqp_msg{payload = Payload}),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic, [Payload]),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
Increase MQTT test coverage and fix edge cases
2022-12-29 21:01:55 +08:00
%% Packet identifier is a non zero two byte integer.
%% Test that the server wraps around the packet identifier.
many_qos1_messages(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
NumMsgs = 16#ffff + 100,
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
C = connect(ClientId, Config, 0, [{retry_interval, 600},
{max_inflight, NumMsgs div 8}]),
{ok, _, [1]} = emqtt:subscribe(C, {Topic, qos1}),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Payloads = lists:map(fun integer_to_binary/1, lists:seq(1, NumMsgs)),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
Self = self(),
Target = lists:last(Payloads),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
lists:foreach(fun(P) ->
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
Cb = {fun(T, _) when T == Target ->
Self ! proceed;
(_, _) ->
ok
end, [P]},
ok = emqtt:publish_async(C, Topic, P, qos1, Cb)
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
end, Payloads),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
receive
proceed -> ok
Bump timeout since we've seen this test time out in CI.
2025-01-14 21:56:41 +08:00
after 300_000 ->
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
ct:fail("message to proceed never received")
end,
Add missing MQTT test assertion util:expect_publishes/3 returns 'ok' or {'publish_not_received', Payload}
2023-06-13 17:40:03 +08:00
ok = expect_publishes(C, Topic, Payloads),
Support iodata() when sending message to MQTT client When the MQTT connection receives an AMQP 0.9.1 message, it will contain a list of payload fragments. This commit avoids the expensive operation of turning that list into a binary. All I/O methods accept iodata(): * erlang:port_command/2 * ssl:send/2 * In Web MQTT, cowboy websockets accept iodata(): https://github.com/ninenines/cowlib/blob/0d04cfffa3bd49b10ccc783c381a261c15b7e7a3/src/cow_ws.erl#L58
2022-12-05 00:29:07 +08:00
ok = emqtt:disconnect(C).
Add feature flag rabbit_mqtt_qos0_queue Routing to a queue of type rabbit_mqtt_qos0_queue hosted on a remote node requires knowledge of that queue type on the local node.
2022-11-30 02:13:50 +08:00
%% This test is mostly interesting in mixed version mode where feature flag
%% rabbit_mqtt_qos0_queue is disabled and therefore a classic queue gets created.
rabbit_mqtt_qos0_queue(Config) ->
Topic = atom_to_binary(?FUNCTION_NAME),
%% Place MQTT subscriber process on new node in mixed version.
Sub = connect(<<"subscriber">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic, qos0),
%% Place MQTT publisher process on old node in mixed version.
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Pub = connect(<<"publisher">>, Config, 1, []),
Add feature flag rabbit_mqtt_qos0_queue Routing to a queue of type rabbit_mqtt_qos0_queue hosted on a remote node requires knowledge of that queue type on the local node.
2022-11-30 02:13:50 +08:00
Msg = <<"msg">>,
ok = emqtt:publish(Pub, Topic, Msg, qos0),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(Sub, Topic, [Msg]),
Add overflow drop-head to rabbit_mqtt_qos_queue type Traditionally, queue types implement flow control by keeping state in both sending and receiving Erlang processes (for example credit based flow control keeps the number of credits within the process dictionary). The rabbit_mqtt_qos0_queue cannot keep such state in sending or receiving Erlang process because such state would consume a large amount of memory in case of large fan-ins or large fan-outs. The whole idea of the rabbit_mqtt_qos_queue type is to not keep any state in the rabbit_queue_type client. This makes this new queue type scale so well. Therefore the new queue type cannot (easily) implement flow control throttling individual senders. In this commit, we take a different approach: Instead of implementing flow control throttling individual senders, the receiving MQTT connection process drops QoS 0 messages from the rabbit_mqtt_qos_queue if it is overflowed with messages AND its MQTT client is not capable of receiving messages fast enough. This is a simple and sufficient solution because it's better to drop QoS 0 (at most once) messages instead of causing cluster-wide memory alarms. The user can opt out of dropping messages by setting the new env setting mailbox_soft_limit to 0. Additionally, we reduce the send_timeout from 30 seconds default in Ranch to 15 seconds default in MQTT. This will detect hanging MQTT clients faster causing the MQTT connection to be closed.
2022-12-12 16:57:33 +08:00
ok = emqtt:disconnect(Sub),
ok = emqtt:disconnect(Pub).
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
rabbit_mqtt_qos0_queue_kill_node(Config) ->
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
Topic1 = <<"t/1">>,
Topic2 = <<"t/2">>,
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
Pub = connect(<<"publisher">>, Config, 2, []),
SubscriberId = <<"subscriber">>,
Sub0 = connect(SubscriberId, Config, 0, []),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
{ok, _, [0]} = emqtt:subscribe(Sub0, Topic1, qos0),
Reduce Khepri test flakes Test case rabbit_mqtt_qos0_queue_kill_node flaked because after an MQTT client subscribes on node 0, RabbitMQ returns success and replicated the new binding to node 0 and node 1, but not yet to node 2. Another MQTT client then publishes on node 2 without the binding being present yet on node 2, and the message therefore isn't routed. This commit attempts to eliminate this flake. It adds a function to rabbit_ct_broker_helpers which waits until a given node has caught up with the leader node. We can reuse that function in future to eliminate more test flakes.
2024-08-06 22:16:14 +08:00
ok = await_metadata_store_consistent(Config, 2),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
ok = emqtt:publish(Pub, Topic1, <<"m0">>, qos0),
ok = expect_publishes(Sub0, Topic1, [<<"m0">>]),
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
process_flag(trap_exit, true),
ok = rabbit_ct_broker_helpers:kill_node(Config, 0),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
ok = await_exit(Sub0),
%% Wait to run rabbit_amqqueue:on_node_down/1 on both live nodes.
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
timer:sleep(500),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
%% Re-connect to a live node with same MQTT client ID.
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
Sub1 = connect(SubscriberId, Config, 1, []),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
{ok, _, [0]} = emqtt:subscribe(Sub1, Topic2, qos0),
Attempt to eliminate test flake This commit attempts to eliminate the test flake described in https://github.com/rabbitmq/rabbitmq-server/issues/12413#issuecomment-2385449940 ``` rabbitmq_mqtt > parallel-ct-set-1 > mqtt_shared_SUITE > cluster_size_3 > v4 rabbit_mqtt_qos0_queue_kill_node === Ended at 2024-10-01 09:59:52 === Location: [{mqtt_shared_SUITE,rabbit_mqtt_qos0_queue_kill_node,[1165](https://github.com/rabbitmq/rabbitmq-server/issues/mqtt_shared_suite.src.html#1165)}, {test_server,ts_tc,1793}, {test_server,run_test_case_eval1,1302}, {test_server,run_test_case_eval,1234}] === === Reason: no match of right hand side value {publish_not_received, <<"m1">>} in function mqtt_shared_SUITE:rabbit_mqtt_qos0_queue_kill_node/1 (mqtt_shared_SUITE.erl, line 1165) in call from test_server:ts_tc/3 (test_server.erl, line 1793) in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1302) in call from test_server:run_test_case_eval/9 (test_server.erl, line 1234) ``` This flake could not be reproduced locally. This commit also assumes that this flake occurred under Khepri but not under Mnesia. The hypothesis is the following: * Node 0 is down * MQTT client creates binding on node 1 * Khepri commits since the binding is replicated and persisted on node 1 and node 2. However the binding isn't reflected yet in node 2's routing projecting table. * Publishing a message to node 2 routes to nowhere.
2024-10-24 01:36:58 +08:00
ok = await_metadata_store_consistent(Config, 2),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
ok = emqtt:publish(Pub, Topic2, <<"m1">>, qos0),
ok = expect_publishes(Sub1, Topic2, [<<"m1">>]),
%% Since we started a new clean session, previous subscription should have been deleted.
ok = emqtt:publish(Pub, Topic1, <<"m2">>, qos0),
receive {publish, _} = Publish -> ct:fail({unexpected, Publish})
after 300 -> ok
end,
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
ok = rabbit_ct_broker_helpers:start_node(Config, 0),
ok = rabbit_ct_broker_helpers:kill_node(Config, 1),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
%% This time, do not wait.
%% rabbit_amqqueue:on_node_down/1 may or may not have run.
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
Sub2 = connect(SubscriberId, Config, 2, []),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
{ok, _, [0]} = emqtt:subscribe(Sub2, Topic2, qos0),
ok = emqtt:publish(Pub, Topic2, <<"m3">>, qos0),
ok = expect_publishes(Sub2, Topic2, [<<"m3">>]),
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
ok = emqtt:disconnect(Sub2),
ok = emqtt:disconnect(Pub),
Allow MQTT QoS 0 subscribers to reconnect The solution in #10203 has the following issues: 1. Bindings can be left ofter in Mnesia table rabbit_durable_queue. One solution to 1. would be to first delete the old queue via `rabbit_amqqueue:internal_delete(Q, User, missing_owner)` and subsequently declare the new queue via `rabbit_amqqueue:internal_declare(Q, false)` However, even then, it suffers from: 2. Race conditions between `rabbit_amqqueue:on_node_down/1` and `rabbit_mqtt_qos0_queue:declare/2`: `rabbit_amqqueue:on_node_down/1` could first read the queue records that need to be deleted, thereafter `rabbit_mqtt_qos0_queue:declare/2` could re-create the queue owned by the new connection PID, and `rabbit_amqqueue:on_node_down/1` could subsequently delete the re-created queue. Unfortunately, `rabbit_amqqueue:on_node_down/1` does not delete transient queues in one isolated transaction. Instead it first reads queues and subsequenlty deletes queues in batches making it prone to race conditions. Ideally, this commit deletes all rabbit_mqtt_qos0_queue queues of the node that has crashed including their bindings. However, doing so in one transaction is risky as there may be millions of such queues and the current code path applies the same logic on all live nodes resulting in conflicting transactions and therefore a long database operation. Hence, this commit uses the simplest approach which should still be safe: Do not remove rabbit_mqtt_qos0_queue queues if a node crashes. Other live nodes will continue to route to these dead queues. That should be okay, given that the rabbit_mqtt_qos0_queue clients auto confirm. Continuing routing however has the effect of counting as routing result for AMQP 0.9.1 `mandatory` property. If an MQTT client re-connects to a live node with the same client ID, the new node will delete and then re-create the queue. Once the crashed node comes back online, it will clean up its leftover queues and bindings.
2023-12-27 23:40:52 +08:00
ok = rabbit_ct_broker_helpers:start_node(Config, 1),
?assertEqual([], rpc(Config, rabbit_db_binding, get_all, [])).
Overwrite rabbit_mqtt_qos0_queue record from crashed node When a node is shut down cleanly, the rabbit_mqtt_qos0_queue record is removed from Mnesia. When a node crashes and subsequently reboots the new node incarnation removes the old rabbit_mqtt_qos0_queue record from Mnesia (via rabbit_mqtt_qos0_queue:recover/2) However, when a node crashes, the rabbit_mqtt_qos0_queue will be removed from Mnesia table rabbit_queue, but will still be present in table rabbit_durable_queue on the other live nodes. Prior to this commit, when the same MQTT client (i.e. same MQTT client ID) re-connects from the crashed node to another live node and re-subscribes, the following error occurred: ``` [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic, [debug] <0.43155.0> <<"as923/gateway/+/command/#">>,0}] [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent, [error] <0.43155.0> {amqqueue, [error] <0.43155.0> {resource, [error] <0.43155.0> <<"/">>, [error] <0.43155.0> queue, [error] <0.43155.0> <<"mqtt-subscription-nodered_24e214feb018a232qos0">>}, [error] <0.43155.0> true, [error] <0.43155.0> false, [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> <15486.32690.0>, [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [], [error] <0.43155.0> [{vhost, [error] <0.43155.0> <<"/">>}, [error] <0.43155.0> {name, [error] <0.43155.0> <<"ha-all-mqtt">>}, [error] <0.43155.0> {pattern, [error] <0.43155.0> <<"^mqtt-">>}, [error] <0.43155.0> {'apply-to', [error] <0.43155.0> <<"all">>}, [error] <0.43155.0> {definition, [error] <0.43155.0> [{<<"ha-mode">>, [error] <0.43155.0> <<"all">>}]}, [error] <0.43155.0> {priority, [error] <0.43155.0> 0}], [error] <0.43155.0> undefined, [error] <0.43155.0> [], [error] <0.43155.0> undefined, [error] <0.43155.0> live, [error] <0.43155.0> 0, [error] <0.43155.0> [], [error] <0.43155.0> <<"/">>, [error] <0.43155.0> #{user => [error] <0.43155.0> <<"iottester">>}, [error] <0.43155.0> rabbit_mqtt_qos0_queue, [error] <0.43155.0> #{}}, [error] <0.43155.0> nodedown} [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error ``` This commit fixes this error allowing an MQTT client that connects with CleanSession=true and subscribes with QoS 0 to re-connect and re-subscribe to another live node if the original Rabbit node crashes. Reported in https://groups.google.com/g/rabbitmq-users/c/pxgy0QiwilM/m/LkJQ-3DyBgAJ
2023-12-22 00:15:27 +08:00
Simplify test cases 1. Only run the CLI tests on a single node cluster. The shared_SUITE is already very big. Testing the same CLI commands against node-0 on a 3-node cluster brings no benefit. 2. Move the two new CLI test cases in front of management_plugin_connection because they are similar in that all three tests close the MQTT connection. 3. There is no need to query the HTTP API for the two new CLI test cases. 4. There is no need to set keepalive in the two new CLI test cases.
2024-08-28 18:27:46 +08:00
cli_close_all_connections(Config) ->
ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
process_flag(trap_exit, true),
{ok, String} = rabbit_ct_broker_helpers:rabbitmqctl(
Config, 0, ["close_all_connections", "bye"]),
?assertEqual(match, re:run(String, "Closing .* reason: bye", [{capture, none}])),
ok = await_exit(C).
cli_close_all_user_connections(Config) ->
ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
process_flag(trap_exit, true),
{ok, String} = rabbit_ct_broker_helpers:rabbitmqctl(
Config, 0, ["close_all_user_connections","guest", "bye"]),
?assertEqual(match, re:run(String, "Closing .* reason: bye", [{capture, none}])),
ok = await_exit(C).
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
%% Test that MQTT connection can be listed and closed via the rabbitmq_management plugin.
management_plugin_connection(Config) ->
KeepaliveSecs = 99,
ClientId = atom_to_binary(?FUNCTION_NAME),
Node = atom_to_binary(get_node_config(Config, 0, nodename)),
Fix crash when closing connection Avoid the following crash ``` ** Reason for termination == ** {mqtt_unexpected_cast,{shutdown,"Closed via management plugin"}} crasher: initial call: rabbit_mqtt_reader:init/1 pid: <0.1096.0> registered_name: [] exception exit: {mqtt_unexpected_cast, {shutdown,"Closed via management plugin"}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1208) ``` when closing MQTT or Stream connections via HTTP API endpoint ``` /connections/username/:username ```
2023-12-14 19:35:51 +08:00
C1 = connect(ClientId, Config, [{keepalive, KeepaliveSecs}]),
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
FilterFun =
fun(#{client_properties := #{client_id := CId}})
when CId == ClientId -> true;
(_) -> false
end,
%% Sometimes connections remain open from other testcases,
%% let's match the one we're looking for
eventually(
?_assertMatch(
[_],
lists:filter(FilterFun, http_get(Config, "/connections"))),
1000, 10),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
[#{client_properties := #{client_id := ClientId},
timeout := KeepaliveSecs,
node := Node,
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
name := ConnectionName}] =
lists:filter(FilterFun, http_get(Config, "/connections")),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
process_flag(trap_exit, true),
http_delete(Config,
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
"/connections/" ++ binary_to_list(uri_string:quote(ConnectionName)),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
?NO_CONTENT),
Fix crash when closing connection Avoid the following crash ``` ** Reason for termination == ** {mqtt_unexpected_cast,{shutdown,"Closed via management plugin"}} crasher: initial call: rabbit_mqtt_reader:init/1 pid: <0.1096.0> registered_name: [] exception exit: {mqtt_unexpected_cast, {shutdown,"Closed via management plugin"}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1208) ``` when closing MQTT or Stream connections via HTTP API endpoint ``` /connections/username/:username ```
2023-12-14 19:35:51 +08:00
await_exit(C1),
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
eventually(
?_assertMatch(
[],
lists:filter(FilterFun, http_get(Config, "/connections"))),
1000, 10),
Fix crash when closing connection Avoid the following crash ``` ** Reason for termination == ** {mqtt_unexpected_cast,{shutdown,"Closed via management plugin"}} crasher: initial call: rabbit_mqtt_reader:init/1 pid: <0.1096.0> registered_name: [] exception exit: {mqtt_unexpected_cast, {shutdown,"Closed via management plugin"}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1208) ``` when closing MQTT or Stream connections via HTTP API endpoint ``` /connections/username/:username ```
2023-12-14 19:35:51 +08:00
eventually(?_assertEqual([], all_connection_pids(Config)), 500, 3),
Bump (c) line year There are no functional changes to this massive diff.
2025-01-02 06:54:10 +08:00
Fix crash when closing connection Avoid the following crash ``` ** Reason for termination == ** {mqtt_unexpected_cast,{shutdown,"Closed via management plugin"}} crasher: initial call: rabbit_mqtt_reader:init/1 pid: <0.1096.0> registered_name: [] exception exit: {mqtt_unexpected_cast, {shutdown,"Closed via management plugin"}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1208) ``` when closing MQTT or Stream connections via HTTP API endpoint ``` /connections/username/:username ```
2023-12-14 19:35:51 +08:00
C2 = connect(ClientId, Config, [{keepalive, KeepaliveSecs}]),
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
eventually(
?_assertMatch(
[_],
lists:filter(FilterFun, http_get(Config, "/connections"))),
1000, 10),
Fix crash when closing connection Avoid the following crash ``` ** Reason for termination == ** {mqtt_unexpected_cast,{shutdown,"Closed via management plugin"}} crasher: initial call: rabbit_mqtt_reader:init/1 pid: <0.1096.0> registered_name: [] exception exit: {mqtt_unexpected_cast, {shutdown,"Closed via management plugin"}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1208) ``` when closing MQTT or Stream connections via HTTP API endpoint ``` /connections/username/:username ```
2023-12-14 19:35:51 +08:00
http_delete(Config,
"/connections/username/guest",
?NO_CONTENT),
await_exit(C2),
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
eventually(
?_assertMatch(
[],
lists:filter(FilterFun, http_get(Config, "/connections"))),
1000, 10),
Fix dialyzer issues and add function specs Fix all dialyzer warnings in rabbitmq_mqtt and rabbitmq_web_mqtt. Add more function specs.
2022-12-27 18:00:33 +08:00
eventually(?_assertEqual([], all_connection_pids(Config)), 500, 3).
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
management_plugin_enable(Config) ->
ok = rabbit_ct_broker_helpers:disable_plugin(Config, 0, rabbitmq_management),
ok = rabbit_ct_broker_helpers:disable_plugin(Config, 0, rabbitmq_management_agent),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
%% If the (web) MQTT connection is established **before** the management plugin is enabled,
%% the management plugin should still list the (web) MQTT connection.
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
ok = rabbit_ct_broker_helpers:enable_plugin(Config, 0, rabbitmq_management_agent),
ok = rabbit_ct_broker_helpers:enable_plugin(Config, 0, rabbitmq_management),
Tests: mqtt_shared_SUITE match expected connection
2024-11-13 20:25:29 +08:00
FilterFun =
fun(#{client_properties := #{client_id := CId}})
when ClientId == CId -> true;
(_) -> false
end,
%% Sometimes connections remain open from other testcases,
%% let's match the one we're looking for
eventually(
?_assertMatch(
[_],
lists:filter(FilterFun, http_get(Config, "/connections"))),
1000, 10),
Fix MQTT in management plugin 1. Allow to inspect an (web) MQTT connection. 2. Show MQTT client ID on connection page as part of client_properties. 3. Handle force_event_refresh (when management_plugin gets enabled after (web) MQTT connections got created). 4. Reduce code duplication between protocol readers. 5. Display '?' instead of 'NaN' in UI for absent queue metrics. 6. Allow an (web) MQTT connection to be closed via management_plugin. For 6. this commit takes the same approach as already done for the stream plugin: The stream plugin registers neither with {type, network} nor {type, direct}. We cannot use gen_server:call/3 anymore to close the connection because the web MQTT connection cannot handle gen_server calls (only casts). Strictly speaking, this commit requires a feature flag to allow to force closing stream connections from the management plugin during a rolling update. However, given that this is rather an edge case, and there is a workaround (connect to the node directly hosting the stream connection), this commit will not introduce a new feature flag.
2022-12-20 02:03:18 +08:00
ok = emqtt:disconnect(C).
Allow CLI listing rabbit_mqtt_qos0_queue queues
2022-12-23 07:17:01 +08:00
%% Test that queues of type rabbit_mqtt_qos0_queue can be listed via rabbitmqctl.
cli_list_queues(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, _} = emqtt:subscribe(C, <<"a/b/c">>, qos0),
Qs = rabbit_ct_broker_helpers:rabbitmqctl_list(
Config, 1,
["list_queues", "--no-table-headers",
"type", "name", "state", "durable", "auto_delete",
"arguments", "pid", "owner_pid", "messages", "exclusive_consumer_tag"
]),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
?assertMatch([[<<"MQTT QoS 0">>, <<"mqtt-subscription-cli_list_queuesqos0">>,
Allow CLI listing rabbit_mqtt_qos0_queue queues
2022-12-23 07:17:01 +08:00
<<"running">>, <<"true">>, <<"false">>, <<"[]">>, _, _, <<"0">>, <<"">>]],
Qs),
Revert "Format MQTT code with `erlfmt`"
2023-01-28 02:25:57 +08:00
Allow CLI listing rabbit_mqtt_qos0_queue queues
2022-12-23 07:17:01 +08:00
?assertEqual([],
rabbit_ct_broker_helpers:rabbitmqctl_list(
Config, 1, ["list_queues", "--local", "--no-table-headers"])
),
ok = emqtt:disconnect(C).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
maintenance(Config) ->
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
C0 = connect(<<"client-0">>, Config, 0, []),
C1a = connect(<<"client-1a">>, Config, 1, []),
C1b = connect(<<"client-1b">>, Config, 1, []),
ClientsNode1 = [C1a, C1b],
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
timer:sleep(500),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
ok = drain_node(Config, 2),
ok = revive_node(Config, 2),
timer:sleep(500),
[?assert(erlang:is_process_alive(C)) || C <- [C0, C1a, C1b]],
process_flag(trap_exit, true),
ok = drain_node(Config, 1),
[await_exit(Pid) || Pid <- ClientsNode1],
[assert_v5_disconnect_reason_code(Config, ?RC_SERVER_SHUTTING_DOWN) || _ <- ClientsNode1],
ok = revive_node(Config, 1),
?assert(erlang:is_process_alive(C0)),
ok = drain_node(Config, 0),
await_exit(C0),
assert_v5_disconnect_reason_code(Config, ?RC_SERVER_SHUTTING_DOWN),
ok = revive_node(Config, 0).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
keepalive(Config) ->
KeepaliveSecs = 1,
KeepaliveMs = timer:seconds(KeepaliveSecs),
WillTopic = <<"will/topic">>,
WillPayload = <<"will-payload">>,
C1 = connect(?FUNCTION_NAME, Config, [{keepalive, KeepaliveSecs},
{will_topic, WillTopic},
{will_payload, WillPayload},
{will_retain, true},
{will_qos, 0}]),
ok = emqtt:publish(C1, <<"ignored">>, <<"msg">>),
%% Connection should stay up when client sends PING requests.
timer:sleep(KeepaliveMs),
?assertMatch(#{publishers := 1},
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
util:get_global_counters(Config)),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
%% Mock the server socket to not have received any bytes.
rabbit_ct_broker_helpers:setup_meck(Config),
Mod = rabbit_net,
ok = rpc(Config, meck, new, [Mod, [no_link, passthrough]]),
ok = rpc(Config, meck, expect, [Mod, getstat, 2, {ok, [{recv_oct, 999}]} ]),
process_flag(trap_exit, true),
%% We expect the server to respect the keepalive closing the connection.
eventually(?_assertMatch(#{publishers := 0},
Test MQTT v5 in existing MQTT suites
2023-02-22 02:24:15 +08:00
util:get_global_counters(Config)),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
KeepaliveMs, 4 * KeepaliveSecs),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
await_exit(C1),
assert_v5_disconnect_reason_code(Config, ?RC_KEEP_ALIVE_TIMEOUT),
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
?assert(rpc(Config, meck, validate, [Mod])),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = rpc(Config, meck, unload, [Mod]),
C2 = connect(<<"client2">>, Config),
{ok, _, [0]} = emqtt:subscribe(C2, WillTopic),
receive {publish, #{client_pid := C2,
dup := false,
qos := 0,
retain := true,
topic := WillTopic,
payload := WillPayload}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing will")
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
end,
ok = emqtt:disconnect(C2).
keepalive_turned_off(Config) ->
%% "A Keep Alive value of zero (0) has the effect of turning off the keep alive mechanism."
KeepaliveSecs = 0,
C = connect(?FUNCTION_NAME, Config, [{keepalive, KeepaliveSecs}]),
ok = emqtt:publish(C, <<"TopicB">>, <<"Payload">>),
%% Mock the server socket to not have received any bytes.
rabbit_ct_broker_helpers:setup_meck(Config),
Mod = rabbit_net,
ok = rpc(Config, meck, new, [Mod, [no_link, passthrough]]),
ok = rpc(Config, meck, expect, [Mod, getstat, 2, {ok, [{recv_oct, 999}]} ]),
rabbit_ct_helpers:consistently(?_assert(erlang:is_process_alive(C))),
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
?assert(rpc(Config, meck, validate, [Mod])),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = rpc(Config, meck, unload, [Mod]),
ok = emqtt:disconnect(C).
duplicate_client_id(Config) ->
Keep storing MQTT client IDs as lists in Ra Up to 3.11.x an MQTT client ID is tracked in Ra as a list of bytes as returned by binary_to_list/1 in https://github.com/rabbitmq/rabbitmq-server/blob/48467d6e1283b8d81e52cfd49c06ea4eaa31617d/deps/rabbitmq_mqtt/src/rabbit_mqtt_frame.erl#L137 This has two downsides: 1. Lists consume more memory than binaries (when tracking many clients). 2. It violates the MQTT spec which states "The ClientId MUST be a UTF-8 encoded string as defined in Section 1.5.3 [MQTT-3.1.3-4]." [v4 3.1.3.1] Therefore, the original idea was to always store MQTT client IDs as binaries starting with Native MQTT in 3.12. However, this leads to client ID tracking misbehaving in mixed version clusters since new nodes would register client IDs as binaries and old nodes would register client IDs as lists. This means that a client registering on a new node with the same client ID as a connection to the old node did not terminate the connection on the old node. Therefore, for backwards compatibility, we leave the client ID as a list of bytes in the Ra machine state because the feature flag delete_ra_cluster_mqtt_node introduced in v3.12 will delete the Ra cluster anyway and the new client ID tracking via pg local will store client IDs as binaries. An interesting side note learned here is that the compiled file rabbit_mqtt_collector must not be changed. This commit only modifies function specs. However as soon as the compiled code is changed, this module becomes a new version. The new version causes the anonymous ra query function to fail in mixed clusters: When the old node does a ra:leader_query where the leader is on the new node, the query function fails on the new node with `badfun` because the new node does not have the same module version. For more context, read: https://web.archive.org/web/20181017104411/http://www.javalimit.com/2010/05/passing-funs-to-other-erlang-nodes.html
2023-04-27 18:52:56 +08:00
[Server1, Server2, _] = rabbit_ct_broker_helpers:get_node_configs(Config, nodename),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% Test session takeover by both new and old node in mixed version clusters.
ClientId1 = <<"c1">>,
ClientId2 = <<"c2">>,
C1a = connect(ClientId1, Config, Server2, []),
C2a = connect(ClientId2, Config, Server1, []),
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
eventually(?_assertEqual(2, length(all_connection_pids(Config)))),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
process_flag(trap_exit, true),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
C1b = connect(ClientId1, Config, Server1, []),
C2b = connect(ClientId2, Config, Server2, []),
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
assert_v5_disconnect_reason_code(Config, ?RC_SESSION_TAKEN_OVER),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
assert_v5_disconnect_reason_code(Config, ?RC_SESSION_TAKEN_OVER),
await_exit(C1a),
await_exit(C2a),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
timer:sleep(200),
Do not skip test On `main` branch and v3.12.6 feature flag delete_ra_cluster_mqtt_node is supported. Instead of skipping the entire test if that feature flag is not enabled, enable the feature flag and run the test. More generally: "Instead of verifying if a feature flag is enabled or not, it's best to enable it and react from the return value (success or failure). Mixed version testing always turn off all feature flags by default. So in the future, even though all nodes supports the mentionned feature flag, the testcase will still be skipped." [JSP]
2023-10-27 21:49:05 +08:00
?assertEqual(2, length(all_connection_pids(Config))),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1b),
ok = emqtt:disconnect(C2b),
Keep storing MQTT client IDs as lists in Ra Up to 3.11.x an MQTT client ID is tracked in Ra as a list of bytes as returned by binary_to_list/1 in https://github.com/rabbitmq/rabbitmq-server/blob/48467d6e1283b8d81e52cfd49c06ea4eaa31617d/deps/rabbitmq_mqtt/src/rabbit_mqtt_frame.erl#L137 This has two downsides: 1. Lists consume more memory than binaries (when tracking many clients). 2. It violates the MQTT spec which states "The ClientId MUST be a UTF-8 encoded string as defined in Section 1.5.3 [MQTT-3.1.3-4]." [v4 3.1.3.1] Therefore, the original idea was to always store MQTT client IDs as binaries starting with Native MQTT in 3.12. However, this leads to client ID tracking misbehaving in mixed version clusters since new nodes would register client IDs as binaries and old nodes would register client IDs as lists. This means that a client registering on a new node with the same client ID as a connection to the old node did not terminate the connection on the old node. Therefore, for backwards compatibility, we leave the client ID as a list of bytes in the Ra machine state because the feature flag delete_ra_cluster_mqtt_node introduced in v3.12 will delete the Ra cluster anyway and the new client ID tracking via pg local will store client IDs as binaries. An interesting side note learned here is that the compiled file rabbit_mqtt_collector must not be changed. This commit only modifies function specs. However as soon as the compiled code is changed, this module becomes a new version. The new version causes the anonymous ra query function to fail in mixed clusters: When the old node does a ra:leader_query where the leader is on the new node, the query function fails on the new node with `badfun` because the new node does not have the same module version. For more context, read: https://web.archive.org/web/20181017104411/http://www.javalimit.com/2010/05/passing-funs-to-other-erlang-nodes.html
2023-04-27 18:52:56 +08:00
eventually(?_assertEqual(0, length(all_connection_pids(Config)))).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_reconnect(Config) ->
session_switch(Config, true).
session_takeover(Config) ->
session_switch(Config, false).
session_switch(Config, Disconnect) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
%% Connect to old node in mixed version cluster.
Fix MQTT test flakes Tests session_reconnect and session_takeover were flaky, specifically when run under Khepri. The issue was in the test itself that the connect properties didn't apply. Therefore, prior to this commit an exclusive queue got created.
2023-10-27 22:59:12 +08:00
C1 = connect(ClientId, Config, 1, non_clean_sess_opts()),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
{ok, _, [1]} = emqtt:subscribe(C1, Topic, qos1),
case Disconnect of
true -> ok = emqtt:disconnect(C1);
false -> unlink(C1)
end,
%% Connect to new node in mixed version cluster.
Fix MQTT test flakes Tests session_reconnect and session_takeover were flaky, specifically when run under Khepri. The issue was in the test itself that the connect properties didn't apply. Therefore, prior to this commit an exclusive queue got created.
2023-10-27 22:59:12 +08:00
C2 = connect(ClientId, Config, 0, non_clean_sess_opts()),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
case Disconnect of
true -> ok;
false -> assert_v5_disconnect_reason_code(Config, ?RC_SESSION_TAKEN_OVER)
end,
%% New connection should be able to modify subscription.
{ok, _, [0]} = emqtt:subscribe(C2, Topic, qos0),
{ok, _} = emqtt:publish(C2, Topic, <<"m1">>, qos1),
receive {publish, #{client_pid := C2,
payload := <<"m1">>,
qos := 0}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1 with QoS 0")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
%% New connection should be able to unsubscribe.
?assertMatch({ok, _, _}, emqtt:unsubscribe(C2, Topic)),
{ok, _} = emqtt:publish(C2, Topic, <<"m2">>, qos1),
receive Unexpected -> ct:fail({unexpected, Unexpected})
after 300 -> ok
end,
ok = emqtt:disconnect(C2),
C3 = connect(ClientId, Config, 0, [{clean_start, true}]),
ok = emqtt:disconnect(C3),
eventually(?_assertEqual([], all_connection_pids(Config))).
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
block(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, _} = emqtt:subscribe(C, Topic),
{ok, _} = emqtt:publish(C, Topic, <<"Not blocked yet">>, [{qos, 1}]),
Block only publishing (Web) MQTT connections When a cluster wide memory or disk alarm is fired, in AMQP 0.9.1 only connections that are publishing messages get blocked. Connections that only consume can continue to empty the queues. Prior to this commit, all MQTT connections got blocked during a memory or disk alarm. This has two downsides: 1. MQTT connections that only consume, but never published, cannot empty queues anymore. 2. If the memory or disk alarm takes long, the MQTT client does not receive a PINGRESP from the server when it sends a PINGREQ potentially leading to mass client disconnection (depending on the MQTT client implementation). This commit makes sure that an MQTT connection that never sent a single PUBLISH packet (e.g. "pure" MQTT subscribers) are not blocked during memory or disk alarms. In contrast to AMQP 0.9.1, new connections are still blocked from being accepted because accepting (many) new MQTT connections also lead to increased resource usage. The implemenation as done in this commit is simpler, but also more naive than the logic in rabbit_reader: rabbit_reader blocks connections more dynamically whereas rabbit_mqtt_reader and rabbit_web_mqtt_handler block a connection if the connection ever sent a single PUBLISH packet during its lifetime.
2023-01-05 02:29:28 +08:00
ok = rpc(Config, vm_memory_monitor, set_vm_memory_high_watermark, [0]),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
%% Let it block
timer:sleep(100),
%% Blocked, but still will publish when unblocked
puback_timeout = publish_qos1_timeout(C, Topic, <<"Now blocked">>, 1000),
puback_timeout = publish_qos1_timeout(C, Topic, <<"Still blocked">>, 1000),
%% Unblock
Set the default vm_memory_high_watermark to 0.6 (#12161) The default of 0.4 was very conservative even when it was set years ago. Since then: - we moved to CQv2, which have much more predictable memory usage than (non-lazy) CQv1 used to - we removed CQ mirroring which caused large sudden memory spikes in some situations - we removed the option to store message payload in memory in quorum queues For the past two years or so, we've been running all our internal tests and benchmarks using the value of 0.8 with no OOMkills at all (note: we do this on Kubernetes where the Cluster Operators overrides the available memory levaing some additional headroom, but effectively we are still using more than 0.6 of memory).
2024-08-29 18:10:49 +08:00
rpc(Config, vm_memory_monitor, set_vm_memory_high_watermark, [0.6]),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = expect_publishes(C, Topic, [<<"Not blocked yet">>,
<<"Now blocked">>,
<<"Still blocked">>]),
ok = emqtt:disconnect(C).
Block only publishing (Web) MQTT connections When a cluster wide memory or disk alarm is fired, in AMQP 0.9.1 only connections that are publishing messages get blocked. Connections that only consume can continue to empty the queues. Prior to this commit, all MQTT connections got blocked during a memory or disk alarm. This has two downsides: 1. MQTT connections that only consume, but never published, cannot empty queues anymore. 2. If the memory or disk alarm takes long, the MQTT client does not receive a PINGRESP from the server when it sends a PINGREQ potentially leading to mass client disconnection (depending on the MQTT client implementation). This commit makes sure that an MQTT connection that never sent a single PUBLISH packet (e.g. "pure" MQTT subscribers) are not blocked during memory or disk alarms. In contrast to AMQP 0.9.1, new connections are still blocked from being accepted because accepting (many) new MQTT connections also lead to increased resource usage. The implemenation as done in this commit is simpler, but also more naive than the logic in rabbit_reader: rabbit_reader blocks connections more dynamically whereas rabbit_mqtt_reader and rabbit_web_mqtt_handler block a connection if the connection ever sent a single PUBLISH packet during its lifetime.
2023-01-05 02:29:28 +08:00
block_only_publisher(Config) ->
Topic = atom_to_binary(?FUNCTION_NAME),
MQTT: speed up shared_SUITE:many_qos1_messages (#11477) * MQTT: speed up shared_SUITE:many_qos1_messages * speed up block_only_publisher * MQTT: reorganise tests groups To avoid starting a new broker for each protocol group (v3,v4,v5). Instead we run all protocol groups under a single cluster configuration group. * MQTT: speed up publish_to_all_queue_types_qos* tests. * Remove separate mnesia_store group to speed up the test suite * Fix wrong_shard_count * Remove unused field * Run subset of v3 tests The code being tested under v3 and v4 is almost identical. To save time in CI, we therefore run only a very small subset of tests in v3. This cuts the total time reported by CT for the shared_SUITE from 898 seconds to 614 seconds. Also, the java_SUITE tests in v3. * Fix wrong_shard_count * Fix mixed version failure --------- Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-06-20 02:02:33 +08:00
Opts = [{ack_timeout, 1}],
Block only publishing (Web) MQTT connections When a cluster wide memory or disk alarm is fired, in AMQP 0.9.1 only connections that are publishing messages get blocked. Connections that only consume can continue to empty the queues. Prior to this commit, all MQTT connections got blocked during a memory or disk alarm. This has two downsides: 1. MQTT connections that only consume, but never published, cannot empty queues anymore. 2. If the memory or disk alarm takes long, the MQTT client does not receive a PINGRESP from the server when it sends a PINGREQ potentially leading to mass client disconnection (depending on the MQTT client implementation). This commit makes sure that an MQTT connection that never sent a single PUBLISH packet (e.g. "pure" MQTT subscribers) are not blocked during memory or disk alarms. In contrast to AMQP 0.9.1, new connections are still blocked from being accepted because accepting (many) new MQTT connections also lead to increased resource usage. The implemenation as done in this commit is simpler, but also more naive than the logic in rabbit_reader: rabbit_reader blocks connections more dynamically whereas rabbit_mqtt_reader and rabbit_web_mqtt_handler block a connection if the connection ever sent a single PUBLISH packet during its lifetime.
2023-01-05 02:29:28 +08:00
Con = connect(<<"background-connection">>, Config, Opts),
Sub = connect(<<"subscriber-connection">>, Config, Opts),
Pub = connect(<<"publisher-connection">>, Config, Opts),
PubSub = connect(<<"publisher-and-subscriber-connection">>, Config, Opts),
{ok, _, [1]} = emqtt:subscribe(Sub, Topic, qos1),
{ok, _, [1]} = emqtt:subscribe(PubSub, Topic, qos1),
{ok, _} = emqtt:publish(Pub, Topic, <<"from Pub">>, [{qos, 1}]),
{ok, _} = emqtt:publish(PubSub, Topic, <<"from PubSub">>, [{qos, 1}]),
ok = expect_publishes(Sub, Topic, [<<"from Pub">>, <<"from PubSub">>]),
ok = expect_publishes(PubSub, Topic, [<<"from Pub">>, <<"from PubSub">>]),
ok = rpc(Config, vm_memory_monitor, set_vm_memory_high_watermark, [0]),
%% Let it block
timer:sleep(100),
%% We expect that the publishing connections are blocked.
[?assertEqual({error, ack_timeout}, emqtt:ping(Pid)) || Pid <- [Pub, PubSub]],
%% We expect that the non-publishing connections are not blocked.
[?assertEqual(pong, emqtt:ping(Pid)) || Pid <- [Con, Sub]],
%% While the memory alarm is on, let's turn a non-publishing connection
%% into a publishing connection.
{ok, _} = emqtt:publish(Con, Topic, <<"from Con 1">>, [{qos, 1}]),
%% The very first message still goes through.
ok = expect_publishes(Sub, Topic, [<<"from Con 1">>]),
%% But now the new publisher should be blocked as well.
?assertEqual({error, ack_timeout}, emqtt:ping(Con)),
?assertEqual(puback_timeout, publish_qos1_timeout(Con, Topic, <<"from Con 2">>, 500)),
?assertEqual(pong, emqtt:ping(Sub)),
Set the default vm_memory_high_watermark to 0.6 (#12161) The default of 0.4 was very conservative even when it was set years ago. Since then: - we moved to CQv2, which have much more predictable memory usage than (non-lazy) CQv1 used to - we removed CQ mirroring which caused large sudden memory spikes in some situations - we removed the option to store message payload in memory in quorum queues For the past two years or so, we've been running all our internal tests and benchmarks using the value of 0.8 with no OOMkills at all (note: we do this on Kubernetes where the Cluster Operators overrides the available memory levaing some additional headroom, but effectively we are still using more than 0.6 of memory).
2024-08-29 18:10:49 +08:00
rpc(Config, vm_memory_monitor, set_vm_memory_high_watermark, [0.6]),
Block only publishing (Web) MQTT connections When a cluster wide memory or disk alarm is fired, in AMQP 0.9.1 only connections that are publishing messages get blocked. Connections that only consume can continue to empty the queues. Prior to this commit, all MQTT connections got blocked during a memory or disk alarm. This has two downsides: 1. MQTT connections that only consume, but never published, cannot empty queues anymore. 2. If the memory or disk alarm takes long, the MQTT client does not receive a PINGRESP from the server when it sends a PINGREQ potentially leading to mass client disconnection (depending on the MQTT client implementation). This commit makes sure that an MQTT connection that never sent a single PUBLISH packet (e.g. "pure" MQTT subscribers) are not blocked during memory or disk alarms. In contrast to AMQP 0.9.1, new connections are still blocked from being accepted because accepting (many) new MQTT connections also lead to increased resource usage. The implemenation as done in this commit is simpler, but also more naive than the logic in rabbit_reader: rabbit_reader blocks connections more dynamically whereas rabbit_mqtt_reader and rabbit_web_mqtt_handler block a connection if the connection ever sent a single PUBLISH packet during its lifetime.
2023-01-05 02:29:28 +08:00
%% Let it unblock
timer:sleep(100),
%% All connections are unblocked.
[?assertEqual(pong, emqtt:ping(Pid)) || Pid <- [Con, Sub, Pub, PubSub]],
%% The publishing connections should be able to publish again.
{ok, _} = emqtt:publish(Con, Topic, <<"from Con 3">>, [{qos, 1}]),
ok = expect_publishes(Sub, Topic, [<<"from Con 2">>, <<"from Con 3">>]),
ok = expect_publishes(PubSub, Topic, [<<"from Con 1">>, <<"from Con 2">>, <<"from Con 3">>]),
[ok = emqtt:disconnect(Pid) || Pid <- [Con, Sub, Pub, PubSub]].
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
clean_session_disconnect_client(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, _} = emqtt:subscribe(C, <<"topic0">>, qos0),
{ok, _, _} = emqtt:subscribe(C, <<"topic1">>, qos1),
QsQos0 = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_mqtt_qos0_queue]),
QsClassic = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_classic_queue]),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
?assertEqual(1, length(QsQos0)),
?assertEqual(1, length(QsClassic)),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = emqtt:disconnect(C),
%% After terminating a clean session, we expect any session state to be cleaned up on the server.
timer:sleep(200), %% Give some time to clean up exclusive classic queue.
L = rpc(Config, rabbit_amqqueue, list, []),
?assertEqual(0, length(L)).
Clean up exclusive durable queues after unclean shutdown What: Delete bindings of exclusive durable queues after an unclean shutdown. Why: Native MQTT in 3.12 uses only durable queues to ease transition to Khepri. Since auto-delete queues are not deleted after an unclean shutdown, Native MQTT uses exclusive (instead of auto-delete) queues for clean sessions. While this bug is not specific to Native MQTT, this bug is most relevant for the upcoming 3.12 release since exclusive durable queues are rarely used otherwise. How: During queue recovery, not all bindings are recovered yet. Therefore, if, during queue recovery, an exclusive, durable queue need to be deleted, only durable bindings should be queried. Queue types need to make sure that their exclusive, durable queues including their bindings are deleted before starting with binding recovery. Otherwise binding deletion and binding recovery get interleaved leading to topic bindings being created and left behind. Therefore, a classic queue process replies to the recovery process after it deleted its queue record and associated bindings from the database.
2023-03-24 00:54:21 +08:00
clean_session_node_restart(Config) ->
clean_session_node_down(stop_node, Config).
clean_session_node_kill(Config) ->
clean_session_node_down(kill_node, Config).
clean_session_node_down(NodeDown, Config) ->
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
C = connect(?FUNCTION_NAME, Config),
{ok, _, _} = emqtt:subscribe(C, <<"topic0">>, qos0),
{ok, _, _} = emqtt:subscribe(C, <<"topic1">>, qos1),
QsQos0 = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_mqtt_qos0_queue]),
QsClassic = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_classic_queue]),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
?assertEqual(1, length(QsQos0)),
?assertEqual(1, length(QsClassic)),
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
?assertEqual(2, rpc(Config, rabbit_amqqueue, count, [])),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
unlink(C),
Clean up exclusive durable queues after unclean shutdown What: Delete bindings of exclusive durable queues after an unclean shutdown. Why: Native MQTT in 3.12 uses only durable queues to ease transition to Khepri. Since auto-delete queues are not deleted after an unclean shutdown, Native MQTT uses exclusive (instead of auto-delete) queues for clean sessions. While this bug is not specific to Native MQTT, this bug is most relevant for the upcoming 3.12 release since exclusive durable queues are rarely used otherwise. How: During queue recovery, not all bindings are recovered yet. Therefore, if, during queue recovery, an exclusive, durable queue need to be deleted, only durable bindings should be queried. Queue types need to make sure that their exclusive, durable queues including their bindings are deleted before starting with binding recovery. Otherwise binding deletion and binding recovery get interleaved leading to topic bindings being created and left behind. Therefore, a classic queue process replies to the recovery process after it deleted its queue record and associated bindings from the database.
2023-03-24 00:54:21 +08:00
ok = rabbit_ct_broker_helpers:NodeDown(Config, 0),
Share tests between MQTT and Web MQTT New test suite deps/rabbitmq_mqtt/test/shared_SUITE contains tests that are executed against both MQTT and Web MQTT. This has two major advantages: 1. Eliminates test code duplication between rabbitmq_mqtt and rabbitmq_web_mqtt making the tests easier to maintain and to understand. 2. Increases test coverage of Web MQTT. It's acceptable to add a **test** dependency from rabbitmq_mqtt to rabbitmq_web_mqtt. Obviously, there should be no such dependency for non-test code.
2023-01-03 01:02:43 +08:00
ok = rabbit_ct_broker_helpers:start_node(Config, 0),
Allow to use Khepri database to store metadata instead of Mnesia [Why] Mnesia is a very powerful and convenient tool for Erlang applications: it is a persistent disc-based database, it handles replication accross multiple Erlang nodes and it is available out-of-the-box from the Erlang/OTP distribution. RabbitMQ relies on Mnesia to manage all its metadata: * virtual hosts' properties * intenal users * queue, exchange and binding declarations (not queues data) * runtime parameters and policies * ... Unfortunately Mnesia makes it difficult to handle network partition and, as a consequence, the merge conflicts between Erlang nodes once the network partition is resolved. RabbitMQ provides several partition handling strategies but they are not bullet-proof. Users still hit situations where it is a pain to repair a cluster following a network partition. [How] @kjnilsson created Ra [1], a Raft consensus library that RabbitMQ already uses successfully to implement quorum queues and streams for instance. Those queues do not suffer from network partitions. We created Khepri [2], a new persistent and replicated database engine based on Ra and we want to use it in place of Mnesia in RabbitMQ to solve the problems with network partitions. This patch integrates Khepri as an experimental feature. When enabled, RabbitMQ will store all its metadata in Khepri instead of Mnesia. This change comes with behavior changes. While Khepri remains disabled, you should see no changes to the behavior of RabbitMQ. If there are changes, it is a bug. After Khepri is enabled, there are significant changes of behavior that you should be aware of. Because it is based on the Raft consensus algorithm, when there is a network partition, only the cluster members that are in the partition with at least `(Number of nodes in the cluster ÷ 2) + 1` number of nodes can "make progress". In other words, only those nodes may write to the Khepri database and read from the database and expect a consistent result. For instance in a cluster of 5 RabbitMQ nodes: * If there are two partitions, one with 3 nodes, one with 2 nodes, only the group of 3 nodes will be able to write to the database. * If there are three partitions, two with 2 nodes, one with 1 node, none of the group can write to the database. Because the Khepri database will be used for all kind of metadata, it means that RabbitMQ nodes that can't write to the database will be unable to perform some operations. A list of operations and what to expect is documented in the associated pull request and the RabbitMQ website. This requirement from Raft also affects the startup of RabbitMQ nodes in a cluster. Indeed, at least a quorum number of nodes must be started at once to allow nodes to become ready. To enable Khepri, you need to enable the `khepri_db` feature flag: rabbitmqctl enable_feature_flag khepri_db When the `khepri_db` feature flag is enabled, the migration code performs the following two tasks: 1. It synchronizes the Khepri cluster membership from the Mnesia cluster. It uses `mnesia_to_khepri:sync_cluster_membership/1` from the `khepri_mnesia_migration` application [3]. 2. It copies data from relevant Mnesia tables to Khepri, doing some conversion if necessary on the way. Again, it uses `mnesia_to_khepri:copy_tables/4` from `khepri_mnesia_migration` to do it. This can be performed on a running standalone RabbitMQ node or cluster. Data will be migrated from Mnesia to Khepri without any service interruption. Note that during the migration, the performance may decrease and the memory footprint may go up. Because this feature flag is considered experimental, it is not enabled by default even on a brand new RabbitMQ deployment. More about the implementation details below: In the past months, all accesses to Mnesia were isolated in a collection of `rabbit_db*` modules. This is where the integration of Khepri mostly takes place: we use a function called `rabbit_khepri:handle_fallback/1` which selects the database and perform the query or the transaction. Here is an example from `rabbit_db_vhost`: * Up until RabbitMQ 3.12.x: get(VHostName) when is_binary(VHostName) -> get_in_mnesia(VHostName). * Starting with RabbitMQ 3.13.0: get(VHostName) when is_binary(VHostName) -> rabbit_khepri:handle_fallback( #{mnesia => fun() -> get_in_mnesia(VHostName) end, khepri => fun() -> get_in_khepri(VHostName) end}). This `rabbit_khepri:handle_fallback/1` function relies on two things: 1. the fact that the `khepri_db` feature flag is enabled, in which case it always executes the Khepri-based variant. 4. the ability or not to read and write to Mnesia tables otherwise. Before the feature flag is enabled, or during the migration, the function will try to execute the Mnesia-based variant. If it succeeds, then it returns the result. If it fails because one or more Mnesia tables can't be used, it restarts from scratch: it means the feature flag is being enabled and depending on the outcome, either the Mnesia-based variant will succeed (the feature flag couldn't be enabled) or the feature flag will be marked as enabled and it will call the Khepri-based variant. The meat of this function really lives in the `khepri_mnesia_migration` application [3] and `rabbit_khepri:handle_fallback/1` is a wrapper on top of it that knows about the feature flag. However, some calls to the database do not depend on the existence of Mnesia tables, such as functions where we need to learn about the members of a cluster. For those, we can't rely on exceptions from Mnesia. Therefore, we just look at the state of the feature flag to determine which database to use. There are two situations though: * Sometimes, we need the feature flag state query to block because the function interested in it can't return a valid answer during the migration. Here is an example: case rabbit_khepri:is_enabled(RemoteNode) of true -> can_join_using_khepri(RemoteNode); false -> can_join_using_mnesia(RemoteNode) end * Sometimes, we need the feature flag state query to NOT block (for instance because it would cause a deadlock). Here is an example: case rabbit_khepri:get_feature_state() of enabled -> members_using_khepri(); _ -> members_using_mnesia() end Direct accesses to Mnesia still exists. They are limited to code that is specific to Mnesia such as classic queue mirroring or network partitions handling strategies. Now, to discover the Mnesia tables to migrate and how to migrate them, we use an Erlang module attribute called `rabbit_mnesia_tables_to_khepri_db` which indicates a list of Mnesia tables and an associated converter module. Here is an example in the `rabbitmq_recent_history_exchange` plugin: -rabbit_mnesia_tables_to_khepri_db( [{?RH_TABLE, rabbit_db_rh_exchange_m2k_converter}]). The converter module — `rabbit_db_rh_exchange_m2k_converter` in this example — is is fact a "sub" converter module called but `rabbit_db_m2k_converter`. See the documentation of a `mnesia_to_khepri` converter module to learn more about these modules. [1] https://github.com/rabbitmq/ra [2] https://github.com/rabbitmq/khepri [3] https://github.com/rabbitmq/khepri_mnesia_migration See #7206. Co-authored-by: Jean-Sébastien Pédron <jean-sebastien@rabbitmq.com> Co-authored-by: Diana Parra Corbacho <dparracorbac@vmware.com> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2023-01-05 20:57:50 +08:00
%% After terminating a clean session by a node crash, we expect any session
%% state to be cleaned up on the server once the server comes back up.
?assertEqual(0, rpc(Config, rabbit_amqqueue, count, [])).
Increase MQTT test coverage and fix edge cases
2022-12-29 21:01:55 +08:00
Include non-AMQP connections in connection count Prior to this commit: ``` rabbitmqctl status ... Totals Connection count: 0 Queue count: 64308 Virtual host count: 1 ... ``` only counted AMQP connections, but did not include MQTT or stream connections. Let's include the count of all connections in the output of `rabbitmqctl status`.
2023-01-12 19:34:57 +08:00
rabbit_status_connection_count(Config) ->
Add timeout for test AMQP 0.9.1 connection to open We see sporadic test failures where a test case hangs in the receive until the Bazel suite timeout is reached. There is no point in a test case to wait forever for an AMQP 0.9.1 connection to establish. Let's time out after 1 minute. This will make the test case fail faster.
2023-02-17 01:53:09 +08:00
_Pid = rabbit_ct_client_helpers:open_connection(Config, 0),
Include non-AMQP connections in connection count Prior to this commit: ``` rabbitmqctl status ... Totals Connection count: 0 Queue count: 64308 Virtual host count: 1 ... ``` only counted AMQP connections, but did not include MQTT or stream connections. Let's include the count of all connections in the output of `rabbitmqctl status`.
2023-01-12 19:34:57 +08:00
C = connect(?FUNCTION_NAME, Config),
{ok, String} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["status"]),
?assertNotEqual(nomatch, string:find(String, "Connection count: 2")),
ok = emqtt:disconnect(C).
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
trace(Config) ->
Server = atom_to_binary(get_node_config(Config, 0, nodename)),
Topic = Payload = TraceQ = atom_to_binary(?FUNCTION_NAME),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
declare_queue(Ch, TraceQ, []),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = TraceQ,
exchange = <<"amq.rabbitmq.trace">>,
routing_key = <<"#">>}),
%% We expect traced messages for connections created before and connections
%% created after tracing is enabled.
Pub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_publisher">>, Config),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_on"]),
Sub = connect(<<(atom_to_binary(?FUNCTION_NAME))/binary, "_subscriber">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic, qos0),
{ok, _} = emqtt:publish(Pub, Topic, Payload, qos1),
ok = expect_publishes(Sub, Topic, [Payload]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
timer:sleep(10),
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
{#'basic.get_ok'{routing_key = <<"publish.amq.topic">>},
#amqp_msg{props = #'P_basic'{headers = PublishHeaders},
payload = Payload}} =
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
amqp_channel:call(Ch, #'basic.get'{queue = TraceQ}),
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
?assertMatch(#{<<"exchange_name">> := <<"amq.topic">>,
<<"routing_keys">> := [Topic],
<<"connection">> := <<"127.0.0.1:", _/binary>>,
<<"node">> := Server,
<<"vhost">> := <<"/">>,
<<"channel">> := 0,
<<"user">> := <<"guest">>,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
<<"properties">> := #{<<"delivery_mode">> := 2},
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
<<"routed_queues">> := [<<"mqtt-subscription-trace_subscriberqos0">>]},
rabbit_misc:amqp_table(PublishHeaders)),
Revert "Format MQTT code with `erlfmt`"
2023-01-28 02:25:57 +08:00
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
{#'basic.get_ok'{routing_key = <<"deliver.mqtt-subscription-trace_subscriberqos0">>},
#amqp_msg{props = #'P_basic'{headers = DeliverHeaders},
payload = Payload}} =
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
amqp_channel:call(Ch, #'basic.get'{queue = TraceQ}),
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
?assertMatch(#{<<"exchange_name">> := <<"amq.topic">>,
<<"routing_keys">> := [Topic],
<<"connection">> := <<"127.0.0.1:", _/binary>>,
<<"node">> := Server,
<<"vhost">> := <<"/">>,
<<"channel">> := 0,
<<"user">> := <<"guest">>,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
<<"properties">> := #{<<"delivery_mode">> := 2},
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
<<"redelivered">> := 0},
rabbit_misc:amqp_table(DeliverHeaders)),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_off"]),
{ok, _} = emqtt:publish(Pub, Topic, Payload, qos1),
ok = expect_publishes(Sub, Topic, [Payload]),
?assertMatch(#'basic.get_empty'{},
Nack rejected messages to MQTT 5.0 client since MQTT 5.0 supports negative acknowledgements thanks to reason codes in the PUBACK packet. We could either choose reason code 128 or 131. The description code for 131 applies for rejected messages, hence this commit uses 131: > The PUBLISH is valid but the receiver is not willing to accept it.
2023-08-09 19:13:19 +08:00
amqp_channel:call(Ch, #'basic.get'{queue = TraceQ})),
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
delete_queue(Ch, TraceQ),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
[ok = emqtt:disconnect(C) || C <- [Pub, Sub]],
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Support tracing in Native MQTT
2023-01-25 00:35:00 +08:00
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
trace_large_message(Config) ->
TraceQ = <<"trace-queue">>,
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
declare_queue(Ch, TraceQ, []),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = TraceQ,
exchange = <<"amq.rabbitmq.trace">>,
routing_key = <<"deliver.*">>}),
C = connect(<<"my-client">>, Config),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_on"]),
{ok, _, [0]} = emqtt:subscribe(C, <<"/my/topic">>),
Payload0 = binary:copy(<<"x">>, 1_000_000),
Payload = <<Payload0/binary, "y">>,
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = <<".my.topic">>},
#amqp_msg{payload = Payload}),
ok = expect_publishes(C, <<"/my/topic">>, [Payload]),
timer:sleep(10),
?assertMatch(
{#'basic.get_ok'{routing_key = <<"deliver.mqtt-subscription-my-clientqos0">>},
#amqp_msg{payload = Payload}},
amqp_channel:call(Ch, #'basic.get'{queue = TraceQ})
),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_off"]),
delete_queue(Ch, TraceQ),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
Restrict MQTT CONNECT packet size In MQTT 3.1.1, the CONNECT packet consists of 1. 10 bytes variable header 2. ClientId (up to 23 bytes must be supported) 3. Will Topic 4. Will Message (maximum length 2^16 bytes) 5. User Name 6. Password Restricting the CONNECT packet size to 2^16 = 65,536 bytes seems to be a reasonalbe default. The value is configurable via the MQTT app parameter `max_packet_size_unauthenticated`. (Instead of being called `max_packet_size_connect`) the name `max_packet_size_unauthenticated` is generic because MQTT 5 introduces an AUTH packet type.
2023-01-29 22:46:26 +08:00
max_packet_size_unauthenticated(Config) ->
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
ClientId = ?FUNCTION_NAME,
Restrict MQTT CONNECT packet size In MQTT 3.1.1, the CONNECT packet consists of 1. 10 bytes variable header 2. ClientId (up to 23 bytes must be supported) 3. Will Topic 4. Will Message (maximum length 2^16 bytes) 5. User Name 6. Password Restricting the CONNECT packet size to 2^16 = 65,536 bytes seems to be a reasonalbe default. The value is configurable via the MQTT app parameter `max_packet_size_unauthenticated`. (Instead of being called `max_packet_size_connect`) the name `max_packet_size_unauthenticated` is generic because MQTT 5 introduces an AUTH packet type.
2023-01-29 22:46:26 +08:00
Opts = [{will_topic, <<"will/topic">>}],
{C1, Connect} = util:start_client(
ClientId, Config, 0,
[{will_payload, binary:copy(<<"a">>, 64_000)} | Opts]),
?assertMatch({ok, _}, Connect(C1)),
ok = emqtt:disconnect(C1),
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
Key = mqtt_max_packet_size_unauthenticated,
OldMaxSize = rpc(Config, persistent_term, get, [Key]),
Restrict MQTT CONNECT packet size In MQTT 3.1.1, the CONNECT packet consists of 1. 10 bytes variable header 2. ClientId (up to 23 bytes must be supported) 3. Will Topic 4. Will Message (maximum length 2^16 bytes) 5. User Name 6. Password Restricting the CONNECT packet size to 2^16 = 65,536 bytes seems to be a reasonalbe default. The value is configurable via the MQTT app parameter `max_packet_size_unauthenticated`. (Instead of being called `max_packet_size_connect`) the name `max_packet_size_unauthenticated` is generic because MQTT 5 introduces an AUTH packet type.
2023-01-29 22:46:26 +08:00
MaxSize = 500,
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
ok = rpc(Config, persistent_term, put, [Key, MaxSize]),
Restrict MQTT CONNECT packet size In MQTT 3.1.1, the CONNECT packet consists of 1. 10 bytes variable header 2. ClientId (up to 23 bytes must be supported) 3. Will Topic 4. Will Message (maximum length 2^16 bytes) 5. User Name 6. Password Restricting the CONNECT packet size to 2^16 = 65,536 bytes seems to be a reasonalbe default. The value is configurable via the MQTT app parameter `max_packet_size_unauthenticated`. (Instead of being called `max_packet_size_connect`) the name `max_packet_size_unauthenticated` is generic because MQTT 5 introduces an AUTH packet type.
2023-01-29 22:46:26 +08:00
{C2, Connect} = util:start_client(
ClientId, Config, 0,
[{will_payload, binary:copy(<<"b">>, MaxSize + 1)} | Opts]),
true = unlink(C2),
?assertMatch({error, _}, Connect(C2)),
{C3, Connect} = util:start_client(
ClientId, Config, 0,
[{will_payload, binary:copy(<<"c">>, round(MaxSize / 2))} | Opts]),
?assertMatch({ok, _}, Connect(C3)),
ok = emqtt:disconnect(C3),
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
ok = rpc(Config, persistent_term, put, [Key, OldMaxSize]).
max_packet_size_authenticated(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Key = mqtt_max_packet_size_authenticated,
OldMaxSize = rpc(Config, persistent_term, get, [Key]),
MaxSize = 500,
ok = rpc(Config, persistent_term, put, [Key, MaxSize]),
Return Assigned Client Identifier in CONNACK "If the Client connects using a zero length Client Identifier, the Server MUST respond with a CONNACK containing an Assigned Client Identifier."
2023-02-28 21:51:16 +08:00
{C, Connect} = util:start_client(ClientId, Config, 0, []),
{ok, ConnAckProps} = Connect(C),
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
process_flag(trap_exit, true),
ok = emqtt:publish(C, Topic, binary:copy(<<"x">>, MaxSize + 1), qos0),
await_exit(C),
case ?config(mqtt_version, Config) of
PUBACK with reason code "No matching subscribers" Support reason code "No matching subscribers" in PUBACK. This somewhat corresponds to the `mandatory` message property in AMQP 0.9.1.
2023-06-06 01:23:46 +08:00
v3 -> ok;
v4 -> ok;
v5 -> ?assertMatch(#{'Maximum-Packet-Size' := MaxSize}, ConnAckProps),
receive {disconnected, _ReasonCodePacketTooLarge = 149, _Props} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing DISCONNECT packet from server")
PUBACK with reason code "No matching subscribers" Support reason code "No matching subscribers" in PUBACK. This somewhat corresponds to the `mandatory` message property in AMQP 0.9.1.
2023-06-06 01:23:46 +08:00
end
Add MQTT v5 feature Maximum Packet Size set by server "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Server specifies the maximum packet size. "In the case of an error in a CONNECT packet it MAY send a CONNACK packet containing the Reason Code, before closing the Network Connection. In the case of an error in any other packet it SHOULD send a DISCONNECT packet containing the Reason Code before closing the Network Connection." This commit implements only the "SHOULD" (second) part, not the "MAY" (first) part. There are now 2 different global wide MQTT settings on the server: 1. max_packet_size_unauthenticated which applies to the CONNECT packet (and maybe AUTH packet in the future) 2. max_packet_size_authenticated which applies to all other MQTT packets (that is, after the client successfully authenticated). These two settings will apply to all MQTT versions. In MQTT v5, if a non-CONNECT packet is too large, the server will send a DISCONNECT packet to the client with Reason Code "Packet Too Large" before closing the network connection.
2023-02-25 02:23:41 +08:00
end,
ok = rpc(Config, persistent_term, put, [Key, OldMaxSize]).
Restrict MQTT CONNECT packet size In MQTT 3.1.1, the CONNECT packet consists of 1. 10 bytes variable header 2. ClientId (up to 23 bytes must be supported) 3. Will Topic 4. Will Message (maximum length 2^16 bytes) 5. User Name 6. Password Restricting the CONNECT packet size to 2^16 = 65,536 bytes seems to be a reasonalbe default. The value is configurable via the MQTT app parameter `max_packet_size_unauthenticated`. (Instead of being called `max_packet_size_connect`) the name `max_packet_size_unauthenticated` is generic because MQTT 5 introduces an AUTH packet type.
2023-01-29 22:46:26 +08:00
Add test that MQTT ingores configured default queue type The queue type being created for MQTT connections is solely determined by the rabbitmq_mqtt plugin, not by per vhost defaults. If the per vhost default queue type is configured to be a quorum queue, we still want to create classic queues for MQTT connections.
2023-02-10 01:24:52 +08:00
%% Test that the per vhost default queue type introduced in
%% https://github.com/rabbitmq/rabbitmq-server/pull/5305
%% does not apply to queues created for MQTT connections
%% because having millions of quorum queues is too expensive.
default_queue_type(Config) ->
Server = get_node_config(Config, 0, nodename),
QName = Vhost = ClientId = Topic = atom_to_binary(?FUNCTION_NAME),
ok = erpc:call(Server, rabbit_vhost, add, [Vhost,
#{default_queue_type => <<"quorum">>},
<<"acting-user">>]),
ok = rabbit_ct_broker_helpers:set_full_permissions(Config, <<"guest">>, Vhost),
?assertEqual([], rpc(Config, rabbit_amqqueue, list, [])),
%% Sanity check that the configured default queue type works with AMQP 0.9.1.
Conn = rabbit_ct_client_helpers:open_unmanaged_connection(Config, Server, Vhost),
{ok, Ch} = amqp_connection:open_channel(Conn),
declare_queue(Ch, QName, []),
QuorumQueues = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_quorum_queue]),
?assertEqual(1, length(QuorumQueues)),
delete_queue(Ch, QName),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch),
%% Test that the configured default queue type does not apply to MQTT.
Creds = [{username, <<Vhost/binary, ":guest">>},
{password, <<"guest">>}],
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C1 = connect(ClientId, Config, Creds ++ non_clean_sess_opts()),
Add test that MQTT ingores configured default queue type The queue type being created for MQTT connections is solely determined by the rabbitmq_mqtt plugin, not by per vhost defaults. If the per vhost default queue type is configured to be a quorum queue, we still want to create classic queues for MQTT connections.
2023-02-10 01:24:52 +08:00
{ok, _, [1]} = emqtt:subscribe(C1, Topic, qos1),
ClassicQueues = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_classic_queue]),
?assertEqual(1, length(ClassicQueues)),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, true} | Creds]),
ok = emqtt:disconnect(C2),
ok = rabbit_ct_broker_helpers:delete_vhost(Config, Vhost).
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
message_interceptors(Config) ->
Improve message interceptors 1. Force the config for timestamp and routing node message interceptors to be configured with the overwrite boolean() to avoid defining multiple default values throughout the code. 2. Add type specs 3. Extend existing test case for new MQTT client ID interceptor 4. routing node and timestamp should only set the annotation for incoming_message_interceptors group 5. Fix `rabbitmq.conf`. Prior to this commit there were several issue: a.) Setting the right configuration was too user unfriendly, e.g. the user has to set ``` message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id ``` just to enable the MQTT message interceptor. b.) The code that parses was too difficult to understand c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place This is now all fixed, the user sets the rabbitmq.conf as follows: ``` message_interceptors.incoming.set_header_timestamp.overwrite = true message_interceptors.incoming.set_header_routing_node.overwrite = false mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true ``` Note that the first two lines use the same format as for RabbitMQ 4.0 for backwards compatiblity. The last line (MQTT) follows a similar pattern.
2025-04-15 23:40:21 +08:00
ok = rpc(Config, persistent_term, put,
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
[message_interceptors,
[
{rabbit_mqtt_msg_interceptor_client_id, #{}},
{rabbit_msg_interceptor_timestamp, #{overwrite => false,
incoming => true,
outgoing => true}}
]]),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Improve message interceptors 1. Force the config for timestamp and routing node message interceptors to be configured with the overwrite boolean() to avoid defining multiple default values throughout the code. 2. Add type specs 3. Extend existing test case for new MQTT client ID interceptor 4. routing node and timestamp should only set the annotation for incoming_message_interceptors group 5. Fix `rabbitmq.conf`. Prior to this commit there were several issue: a.) Setting the right configuration was too user unfriendly, e.g. the user has to set ``` message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id ``` just to enable the MQTT message interceptor. b.) The code that parses was too difficult to understand c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place This is now all fixed, the user sets the rabbitmq.conf as follows: ``` message_interceptors.incoming.set_header_timestamp.overwrite = true message_interceptors.incoming.set_header_routing_node.overwrite = false mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true ``` Note that the first two lines use the same format as for RabbitMQ 4.0 for backwards compatiblity. The last line (MQTT) follows a similar pattern.
2025-04-15 23:40:21 +08:00
Payload = Topic = atom_to_binary(?FUNCTION_NAME),
ClientId = <<"🆔"/utf8>>,
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
CQName = <<"my classic queue">>,
Stream = <<"my stream">>,
declare_queue(Ch, CQName, [{<<"x-queue-type">>, longstr, <<"classic">>}]),
declare_queue(Ch, Stream, [{<<"x-queue-type">>, longstr, <<"stream">>}]),
bind(Ch, CQName, Topic),
bind(Ch, Stream, Topic),
Move plugin rabbitmq-message-timestamp to the core As reported in https://groups.google.com/g/rabbitmq-users/c/x8ACs4dBlkI/ plugins that implement rabbit_channel_interceptor break with Native MQTT in 3.12 because Native MQTT does not use rabbit_channel anymore. Specifically, these plugins don't work anymore in 3.12 when sending a message from an MQTT publisher to an AMQP 0.9.1 consumer. Two of these plugins are https://github.com/rabbitmq/rabbitmq-message-timestamp and https://github.com/rabbitmq/rabbitmq-routing-node-stamp This commit moves both plugins into rabbitmq-server. Therefore, these plugins are deprecated starting in 3.12. Instead of using these plugins, the user gets the same behaviour by configuring rabbitmq.conf as follows: ``` incoming_message_interceptors.set_header_timestamp.overwrite = false incoming_message_interceptors.set_header_routing_node.overwrite = false ``` While both plugins were incompatible to be used together, this commit allows setting both headers. We name the top level configuration key `incoming_message_interceptors` because only incoming messages are intercepted. Currently, only `set_header_timestamp` and `set_header_routing_node` are supported. (We might support more in the future.) Both can set `overwrite` to `false` or `true`. The meaning of `overwrite` is the same as documented in https://github.com/rabbitmq/rabbitmq-message-timestamp#always-overwrite-timestamps i.e. whether headers should be overwritten if they are already present in the message. Both `set_header_timestamp` and `set_header_routing_node` behave exactly to plugins `rabbitmq-message-timestamp` and `rabbitmq-routing-node-stamp`, respectively. Upon node boot, the configuration is put into persistent_term to not cause any performance penalty in the default case where these settings are disabled. The channel and MQTT connection process will intercept incoming messages and - if configured - add the desired AMQP 0.9.1 headers. For now, this allows using Native MQTT in 3.12 with the old plugins behaviour. In the future, once "message containers" are implemented, we can think about more generic message interceptors where plugins can be written to modify arbitrary headers or message contents for various protocols. Likewise, in the future, once MQTT 5.0 is implemented, we can think about an MQTT connection interceptor which could function similar to a `rabbit_channel_interceptor` allowing to modify any MQTT packet.
2023-05-12 22:12:50 +08:00
C = connect(ClientId, Config),
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
Move plugin rabbitmq-message-timestamp to the core As reported in https://groups.google.com/g/rabbitmq-users/c/x8ACs4dBlkI/ plugins that implement rabbit_channel_interceptor break with Native MQTT in 3.12 because Native MQTT does not use rabbit_channel anymore. Specifically, these plugins don't work anymore in 3.12 when sending a message from an MQTT publisher to an AMQP 0.9.1 consumer. Two of these plugins are https://github.com/rabbitmq/rabbitmq-message-timestamp and https://github.com/rabbitmq/rabbitmq-routing-node-stamp This commit moves both plugins into rabbitmq-server. Therefore, these plugins are deprecated starting in 3.12. Instead of using these plugins, the user gets the same behaviour by configuring rabbitmq.conf as follows: ``` incoming_message_interceptors.set_header_timestamp.overwrite = false incoming_message_interceptors.set_header_routing_node.overwrite = false ``` While both plugins were incompatible to be used together, this commit allows setting both headers. We name the top level configuration key `incoming_message_interceptors` because only incoming messages are intercepted. Currently, only `set_header_timestamp` and `set_header_routing_node` are supported. (We might support more in the future.) Both can set `overwrite` to `false` or `true`. The meaning of `overwrite` is the same as documented in https://github.com/rabbitmq/rabbitmq-message-timestamp#always-overwrite-timestamps i.e. whether headers should be overwritten if they are already present in the message. Both `set_header_timestamp` and `set_header_routing_node` behave exactly to plugins `rabbitmq-message-timestamp` and `rabbitmq-routing-node-stamp`, respectively. Upon node boot, the configuration is put into persistent_term to not cause any performance penalty in the default case where these settings are disabled. The channel and MQTT connection process will intercept incoming messages and - if configured - add the desired AMQP 0.9.1 headers. For now, this allows using Native MQTT in 3.12 with the old plugins behaviour. In the future, once "message containers" are implemented, we can think about more generic message interceptors where plugins can be written to modify arbitrary headers or message contents for various protocols. Likewise, in the future, once MQTT 5.0 is implemented, we can think about an MQTT connection interceptor which could function similar to a `rabbit_channel_interceptor` allowing to modify any MQTT packet.
2023-05-12 22:12:50 +08:00
NowSecs = os:system_time(second),
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
NowMillis = os:system_time(millisecond),
{ok, _} = emqtt:publish(C, Topic, Payload, qos1),
Move plugin rabbitmq-message-timestamp to the core As reported in https://groups.google.com/g/rabbitmq-users/c/x8ACs4dBlkI/ plugins that implement rabbit_channel_interceptor break with Native MQTT in 3.12 because Native MQTT does not use rabbit_channel anymore. Specifically, these plugins don't work anymore in 3.12 when sending a message from an MQTT publisher to an AMQP 0.9.1 consumer. Two of these plugins are https://github.com/rabbitmq/rabbitmq-message-timestamp and https://github.com/rabbitmq/rabbitmq-routing-node-stamp This commit moves both plugins into rabbitmq-server. Therefore, these plugins are deprecated starting in 3.12. Instead of using these plugins, the user gets the same behaviour by configuring rabbitmq.conf as follows: ``` incoming_message_interceptors.set_header_timestamp.overwrite = false incoming_message_interceptors.set_header_routing_node.overwrite = false ``` While both plugins were incompatible to be used together, this commit allows setting both headers. We name the top level configuration key `incoming_message_interceptors` because only incoming messages are intercepted. Currently, only `set_header_timestamp` and `set_header_routing_node` are supported. (We might support more in the future.) Both can set `overwrite` to `false` or `true`. The meaning of `overwrite` is the same as documented in https://github.com/rabbitmq/rabbitmq-message-timestamp#always-overwrite-timestamps i.e. whether headers should be overwritten if they are already present in the message. Both `set_header_timestamp` and `set_header_routing_node` behave exactly to plugins `rabbitmq-message-timestamp` and `rabbitmq-routing-node-stamp`, respectively. Upon node boot, the configuration is put into persistent_term to not cause any performance penalty in the default case where these settings are disabled. The channel and MQTT connection process will intercept incoming messages and - if configured - add the desired AMQP 0.9.1 headers. For now, this allows using Native MQTT in 3.12 with the old plugins behaviour. In the future, once "message containers" are implemented, we can think about more generic message interceptors where plugins can be written to modify arbitrary headers or message contents for various protocols. Likewise, in the future, once MQTT 5.0 is implemented, we can think about an MQTT connection interceptor which could function similar to a `rabbit_channel_interceptor` allowing to modify any MQTT packet.
2023-05-12 22:12:50 +08:00
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
{#'basic.get_ok'{},
#amqp_msg{payload = Payload,
props = #'P_basic'{
timestamp = Secs,
Improve message interceptors 1. Force the config for timestamp and routing node message interceptors to be configured with the overwrite boolean() to avoid defining multiple default values throughout the code. 2. Add type specs 3. Extend existing test case for new MQTT client ID interceptor 4. routing node and timestamp should only set the annotation for incoming_message_interceptors group 5. Fix `rabbitmq.conf`. Prior to this commit there were several issue: a.) Setting the right configuration was too user unfriendly, e.g. the user has to set ``` message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id ``` just to enable the MQTT message interceptor. b.) The code that parses was too difficult to understand c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place This is now all fixed, the user sets the rabbitmq.conf as follows: ``` message_interceptors.incoming.set_header_timestamp.overwrite = true message_interceptors.incoming.set_header_routing_node.overwrite = false mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true ``` Note that the first two lines use the same format as for RabbitMQ 4.0 for backwards compatiblity. The last line (MQTT) follows a similar pattern.
2025-04-15 23:40:21 +08:00
headers = Headers
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
}}
} = amqp_channel:call(Ch, #'basic.get'{queue = CQName}),
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
{_, long, ReceivedTs} = lists:keyfind(<<"timestamp_in_ms">>, 1, Headers),
?assert(Secs < NowSecs + 9),
?assert(Secs > NowSecs - 9),
?assert(ReceivedTs < NowMillis + 9000),
?assert(ReceivedTs > NowMillis - 9000),
{_, long, SentTs} = lists:keyfind(<<"x-opt-rabbitmq-sent-time">>, 1, Headers),
?assert(SentTs < NowMillis + 9000),
?assert(SentTs > NowMillis - 9000),
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
Improve message interceptors 1. Force the config for timestamp and routing node message interceptors to be configured with the overwrite boolean() to avoid defining multiple default values throughout the code. 2. Add type specs 3. Extend existing test case for new MQTT client ID interceptor 4. routing node and timestamp should only set the annotation for incoming_message_interceptors group 5. Fix `rabbitmq.conf`. Prior to this commit there were several issue: a.) Setting the right configuration was too user unfriendly, e.g. the user has to set ``` message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id ``` just to enable the MQTT message interceptor. b.) The code that parses was too difficult to understand c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place This is now all fixed, the user sets the rabbitmq.conf as follows: ``` message_interceptors.incoming.set_header_timestamp.overwrite = true message_interceptors.incoming.set_header_routing_node.overwrite = false mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true ``` Note that the first two lines use the same format as for RabbitMQ 4.0 for backwards compatiblity. The last line (MQTT) follows a similar pattern.
2025-04-15 23:40:21 +08:00
?assertEqual({<<"x-opt-mqtt-client-id">>, longstr, ClientId},
lists:keyfind(<<"x-opt-mqtt-client-id">>, 1, Headers)),
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
#'basic.qos_ok'{} = amqp_channel:call(Ch, #'basic.qos'{prefetch_count = 1}),
CTag = <<"my ctag">>,
#'basic.consume_ok'{} = amqp_channel:subscribe(
Ch,
#'basic.consume'{
queue = Stream,
consumer_tag = CTag,
arguments = [{<<"x-stream-offset">>, longstr, <<"first">>}]},
self()),
receive {#'basic.deliver'{consumer_tag = CTag},
#amqp_msg{payload = Payload,
props = #'P_basic'{
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
headers = [{<<"timestamp_in_ms">>, long, ReceivedTs} | XHeaders]
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
}}} ->
Improve message interceptors 1. Force the config for timestamp and routing node message interceptors to be configured with the overwrite boolean() to avoid defining multiple default values throughout the code. 2. Add type specs 3. Extend existing test case for new MQTT client ID interceptor 4. routing node and timestamp should only set the annotation for incoming_message_interceptors group 5. Fix `rabbitmq.conf`. Prior to this commit there were several issue: a.) Setting the right configuration was too user unfriendly, e.g. the user has to set ``` message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id ``` just to enable the MQTT message interceptor. b.) The code that parses was too difficult to understand c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place This is now all fixed, the user sets the rabbitmq.conf as follows: ``` message_interceptors.incoming.set_header_timestamp.overwrite = true message_interceptors.incoming.set_header_routing_node.overwrite = false mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true ``` Note that the first two lines use the same format as for RabbitMQ 4.0 for backwards compatiblity. The last line (MQTT) follows a similar pattern.
2025-04-15 23:40:21 +08:00
?assertEqual({<<"x-opt-mqtt-client-id">>, longstr, ClientId},
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
lists:keyfind(<<"x-opt-mqtt-client-id">>, 1, XHeaders)),
{_, long, SentTs1} = lists:keyfind(<<"x-opt-rabbitmq-sent-time">>, 1, XHeaders),
?assert(SentTs1 < NowMillis + 9000),
?assert(SentTs1 > NowMillis - 9000)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail(missing_deliver)
Respect message_interceptors.incoming.set_header_timestamp When feature flag message_containers is enabled, setting ``` message_interceptors.incoming.set_header_timestamp ``` wasn't respected anymore when a message is published via MQTT to a stream and subsequently consumed via AMQP 0.9.1. This commit ensures that AMQP 0.9.1 header timestamp_in_ms will be set. Note that we must not modify the AMQP 1.0 properties section when messages are received via AMQP 1.0 and consumed via AMQP 1.0. Also, message annoation keys not starting with "x-" are reserved.
2024-04-06 00:30:00 +08:00
end,
delete_queue(Ch, Stream),
delete_queue(Ch, CQName),
Support outgoing message interceptors
2025-04-17 16:15:32 +08:00
ok = rpc(Config, persistent_term, put, [message_interceptors, []]),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Move plugin rabbitmq-message-timestamp to the core As reported in https://groups.google.com/g/rabbitmq-users/c/x8ACs4dBlkI/ plugins that implement rabbit_channel_interceptor break with Native MQTT in 3.12 because Native MQTT does not use rabbit_channel anymore. Specifically, these plugins don't work anymore in 3.12 when sending a message from an MQTT publisher to an AMQP 0.9.1 consumer. Two of these plugins are https://github.com/rabbitmq/rabbitmq-message-timestamp and https://github.com/rabbitmq/rabbitmq-routing-node-stamp This commit moves both plugins into rabbitmq-server. Therefore, these plugins are deprecated starting in 3.12. Instead of using these plugins, the user gets the same behaviour by configuring rabbitmq.conf as follows: ``` incoming_message_interceptors.set_header_timestamp.overwrite = false incoming_message_interceptors.set_header_routing_node.overwrite = false ``` While both plugins were incompatible to be used together, this commit allows setting both headers. We name the top level configuration key `incoming_message_interceptors` because only incoming messages are intercepted. Currently, only `set_header_timestamp` and `set_header_routing_node` are supported. (We might support more in the future.) Both can set `overwrite` to `false` or `true`. The meaning of `overwrite` is the same as documented in https://github.com/rabbitmq/rabbitmq-message-timestamp#always-overwrite-timestamps i.e. whether headers should be overwritten if they are already present in the message. Both `set_header_timestamp` and `set_header_routing_node` behave exactly to plugins `rabbitmq-message-timestamp` and `rabbitmq-routing-node-stamp`, respectively. Upon node boot, the configuration is put into persistent_term to not cause any performance penalty in the default case where these settings are disabled. The channel and MQTT connection process will intercept incoming messages and - if configured - add the desired AMQP 0.9.1 headers. For now, this allows using Native MQTT in 3.12 with the old plugins behaviour. In the future, once "message containers" are implemented, we can think about more generic message interceptors where plugins can be written to modify arbitrary headers or message contents for various protocols. Likewise, in the future, once MQTT 5.0 is implemented, we can think about an MQTT connection interceptor which could function similar to a `rabbit_channel_interceptor` allowing to modify any MQTT packet.
2023-05-12 22:12:50 +08:00
Make retained message stores compatible with pre 3.13 The format of #mqtt_msg{} changes from 3.12 to 3.13. In 3.13 the record contains 2 additional fields: * props * timestamp The old #mqtt_msg{} might still be stored by the retained message store in ets or dets. This commit converts such an old message format when read from the database. The alternative would have been to run a migration function over the whole table which is slightly more complex to implement. Instead of giving the new message format a different record name, e.g. #mqtt_msg_v2{}, this commit decides to re-use the same name such that the new code only handles the record name #mqtt_msg{}.
2023-06-07 19:20:28 +08:00
%% This test makes sure that a retained message that got written in 3.12 or earlier
%% can be consumed in 3.13 or later.
retained_message_conversion(Config) ->
Topic = <<"a/b">>,
Payload = <<"my retained msg">>,
OldMqttMsgFormat = {mqtt_msg, _Retain = true, _QoS = 1, Topic, _Dup = false, _PktId = 1, Payload},
RetainerPid = rpc(Config, rabbit_mqtt_retainer_sup, start_child_for_vhost, [<<"/">>]),
{rabbit_mqtt_retainer, StoreState, _} = sys:get_state(RetainerPid),
ok = rpc(Config, rabbit_mqtt_retained_msg_store, insert, [Topic, OldMqttMsgFormat, StoreState]),
C = connect(?FUNCTION_NAME, Config),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
receive {publish, #{client_pid := C,
dup := false,
qos := 1,
retain := true,
topic := Topic,
payload := Payload}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing retained message")
Make retained message stores compatible with pre 3.13 The format of #mqtt_msg{} changes from 3.12 to 3.13. In 3.13 the record contains 2 additional fields: * props * timestamp The old #mqtt_msg{} might still be stored by the retained message store in ets or dets. This commit converts such an old message format when read from the database. The alternative would have been to run a migration function over the whole table which is slightly more complex to implement. Instead of giving the new message format a different record name, e.g. #mqtt_msg_v2{}, this commit decides to re-use the same name such that the new code only handles the record name #mqtt_msg{}.
2023-06-07 19:20:28 +08:00
end,
ok = emqtt:publish(C, Topic, <<>>, [{retain, true}]),
ok = emqtt:disconnect(C).
Serialise and parse MQTT 5.0 packets
2023-02-08 18:34:05 +08:00
%% Test that the server can handle UTF-8 encoded strings.
utf8(Config) ->
C = connect(?FUNCTION_NAME, Config),
% "The Topic Name MUST be present as the first field in the PUBLISH Packet Variable header.
% It MUST be a UTF-8 encoded string [MQTT-3.3.2-1] as defined in section 1.5.3."
Topic = <<"うさぎ"/utf8>>, %% Rabbit in Japanese
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"msg">>, qos1),
ok = expect_publishes(C, Topic, [<<"msg">>]),
ok = emqtt:disconnect(C).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
bind_exchange_to_exchange(Config) ->
SourceX = <<"amq.topic">>,
DestinationX = <<"destination">>,
Q = <<"q">>,
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
#'exchange.declare_ok'{} = amqp_channel:call(Ch, #'exchange.declare'{exchange = DestinationX,
durable = true,
auto_delete = true}),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = Q,
durable = true}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{exchange = DestinationX,
queue = Q,
routing_key = <<"a.b">>}),
#'exchange.bind_ok'{} = amqp_channel:call(Ch, #'exchange.bind'{destination = DestinationX,
source = SourceX,
routing_key = <<"*.b">>}),
C = connect(?FUNCTION_NAME, Config),
%% Message should be routed as follows: SourceX -> DestinationX -> Q
{ok, _} = emqtt:publish(C, <<"a/b">>, <<"msg">>, qos1),
eventually(?_assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"msg">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q}))),
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
#'queue.delete_ok'{message_count = 0} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
bind_exchange_to_exchange_single_message(Config) ->
SourceX = <<"amq.topic">>,
DestinationX = <<"destination">>,
Q = <<"q">>,
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
{Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
#'exchange.declare_ok'{} = amqp_channel:call(Ch, #'exchange.declare'{exchange = DestinationX,
durable = true,
auto_delete = true}),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = Q,
durable = true}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = Q,
exchange = DestinationX,
routing_key = <<"a.b">>}),
#'exchange.bind_ok'{} = amqp_channel:call(Ch, #'exchange.bind'{destination = DestinationX,
source = SourceX,
routing_key = <<"*.b">>}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = Q,
exchange = SourceX,
routing_key = <<"a.b">>}),
C = connect(?FUNCTION_NAME, Config),
%% Message should be routed as follows:
%% SourceX -> DestinationX -> Q and
%% SourceX -> Q
{ok, _} = emqtt:publish(C, <<"a/b">>, <<"msg">>, qos1),
%% However, since we publish only one time a single message and have a single destination queue,
%% we expect only one copy of the message to end up in the destination queue.
eventually(?_assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"msg">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q}))),
timer:sleep(10),
?assertEqual(#'queue.delete_ok'{message_count = 0},
amqp_channel:call(Ch, #'queue.delete'{queue = Q})),
Improve many testsuites to make them work with mixed versions of Khepri
2025-03-15 00:20:36 +08:00
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_connection_and_channel(Conn, Ch).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
notify_consumer_qos0_queue_deleted(Config) ->
Topic = atom_to_binary(?FUNCTION_NAME),
notify_consumer_queue_deleted(Config, Topic, <<"MQTT QoS 0">>, [{retry_interval, 1}], qos0).
notify_consumer_classic_queue_deleted(Config) ->
Topic = atom_to_binary(?FUNCTION_NAME),
notify_consumer_queue_deleted(Config, Topic, <<"classic">>, non_clean_sess_opts(), qos0).
notify_consumer_quorum_queue_deleted(Config) ->
set_durable_queue_type(Config),
Topic = atom_to_binary(?FUNCTION_NAME),
notify_consumer_queue_deleted(Config, Topic, <<"quorum">>, non_clean_sess_opts(), qos1),
unset_durable_queue_type(Config).
notify_consumer_queue_deleted(Config, Name = Topic, ExpectedType, ConnOpts, Qos) ->
C = connect(Name, Config, ConnOpts),
{ok, _, _} = emqtt:subscribe(C, Topic, Qos),
{ok, #{reason_code_name := success}} = emqtt:publish(C, Name, <<"m1">>, qos1),
{ok, #{reason_code_name := success}} = emqtt:publish(C, Name, <<"m2">>, qos1),
ok = expect_publishes(C, Topic, [<<"m1">>, <<"m2">>]),
[[QName, Type]] = rabbitmqctl_list(Config, 0, ["list_queues", "name", "type", "--no-table-headers"]),
?assertMatch(ExpectedType, Type),
process_flag(trap_exit, true),
{ok, _} = rabbitmqctl(Config, 0, ["delete_queue", QName]),
await_exit(C).
Remove subscriptions map from proc state in mqtt - subscriptions information can be retrieved directly from mnesia - when unsubscribe, we check if there is binding between topic name and queue (check for both qos0 queue name and qos1 queue name) to unbind - added a boolean value has_subs in proc state which will indicate if connection has any active subscriptions. Used for setting consumer global counter
2022-11-17 18:22:06 +08:00
%% -------------------------------------------------------------------
%% Internal helpers
%% -------------------------------------------------------------------
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
await_confirms_ordered(_, To, To) ->
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
ok;
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
await_confirms_ordered(From, N, To) ->
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
Expected = {From, N},
receive
Expected ->
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
await_confirms_ordered(From, N + 1, To);
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
Got ->
ct:fail("Received unexpected message. Expected: ~p Got: ~p", [Expected, Got])
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT ->
Handle queue deletion Handle deletion of queues correctly that an MQTT connection is publishing to with QoS 1.
2022-11-12 01:56:32 +08:00
ct:fail("Did not receive expected message: ~p", [Expected])
end.
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
await_confirms_unordered(_, 0) ->
ok;
await_confirms_unordered(From, Left) ->
receive
{From, _N} ->
await_confirms_unordered(From, Left - 1);
Other ->
ct:fail("Received unexpected message: ~p", [Other])
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT ->
Unblock queue when it gets deleted When a quorum queue or stream gets deleted while the MQTT connection process (or channel) is blocked by that deleted queue due to soft limit being exceeded, unblock that queue. In this commit, an unblock action is also returned with the eol.
2022-11-18 04:13:27 +08:00
ct:fail("~b confirms are missing", [Left])
end.
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
await_consumer_count(ConsumerCount, ClientId, QoS, Config) ->
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
{_Conn, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config),
Fix MQTT test flake Prior to this commit, test ``` make -C deps/rabbitmq_mqtt ct-mqtt_shared t=[mqtt,cluster_size_1,v4]:non_clean_sess_reconnect_qos0_and_qos1 ``` flaked in CI with error: ``` {mqtt_shared_SUITE,non_clean_sess_reconnect_qos0_and_qos1,972} {badmatch,{publish_not_received,<<"msg-0">>}} ``` The problem was the following race condition: * The MQTT v4 client sends an async DISCONNECT * The global MQTT consumer metric got decremented. However, the classic queue still has the MQTT connection proc registered as consumer. * The test case sends a message * The classic queue checks out the message to the old connection instead of checking out the message to the new connection. The solution in this commit is to check the consumer count of the classic queue before proceeding to send the message after disconnection.
2024-11-08 17:34:49 +08:00
QueueName = rabbit_mqtt_util:queue_name_bin(
rabbit_data_coercion:to_binary(ClientId), QoS),
eventually(
?_assertMatch(
#'queue.declare_ok'{consumer_count = ConsumerCount},
amqp_channel:call(Ch, #'queue.declare'{queue = QueueName,
passive = true})), 500, 10),
ok = rabbit_ct_client_helpers:close_channel(Ch).
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
declare_queue(Ch, QueueName, Args)
when is_pid(Ch), is_binary(QueueName), is_list(Args) ->
#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{
queue = QueueName,
durable = true,
arguments = Args}).
Revert "Format MQTT code with `erlfmt`"
2023-01-28 02:25:57 +08:00
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
delete_queue(Ch, QueueNames)
when is_pid(Ch), is_list(QueueNames) ->
lists:foreach(
fun(Q) ->
delete_queue(Ch, Q)
end, QueueNames);
delete_queue(Ch, QueueName)
when is_pid(Ch), is_binary(QueueName) ->
#'queue.delete_ok'{} = amqp_channel:call(
Ch, #'queue.delete'{
queue = QueueName}).
Revert "Format MQTT code with `erlfmt`"
2023-01-28 02:25:57 +08:00
Add test for publishing via MQTT to different queue types
2022-10-29 00:28:47 +08:00
bind(Ch, QueueName, Topic)
when is_pid(Ch), is_binary(QueueName), is_binary(Topic) ->
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = QueueName,
exchange = <<"amq.topic">>,
routing_key = Topic}).
Send disconnect packet from server - when clients connect with a duplicate client id; disconnect with reason code session taken over 142 - when keep alive has timed out; disconnect with reason code keep alive timeout 141
2023-05-02 22:08:15 +08:00
assert_v5_disconnect_reason_code(Config, ReasonCode) ->
case ?config(mqtt_version, Config) of
PUBACK with reason code "No matching subscribers" Support reason code "No matching subscribers" in PUBACK. This somewhat corresponds to the `mandatory` message property in AMQP 0.9.1.
2023-06-06 01:23:46 +08:00
v3 -> ok;
v4 -> ok;
v5 -> receive {disconnected, ReasonCode, _Props} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing DISCONNECT packet from server")
PUBACK with reason code "No matching subscribers" Support reason code "No matching subscribers" in PUBACK. This somewhat corresponds to the `mandatory` message property in AMQP 0.9.1.
2023-06-06 01:23:46 +08:00
end
Fix test decode_basic_properties This commit fixes 2 separate issues: 1. No quorum queue got created in v5 because Session Expiry Interval was 0. 2. Fix a function_clause error. Pass the decoded properties further to other functions looking up headers.
2023-05-21 18:32:14 +08:00
end.
MQTT: disconnect consumer when queue is deleted Queues are automatically declared for MQTT consumers, but they can be externally deleted. The consumer should be disconnected in such case, because it has no way of knowing this happened - from its perspective there are simply no messages to consume. In RabbitMQ 3.11 the consumer was disconnected in such situation. This behaviour changed with native MQTT, which doesn't use AMQP internally.
2025-06-01 22:43:39 +08:00
set_durable_queue_type(Config) ->
ok = rpc(Config, application, set_env, [rabbitmq_mqtt, durable_queue_type, quorum]).
unset_durable_queue_type(Config) ->
ok = rpc(Config, application, unset_env, [rabbitmq_mqtt, durable_queue_type]).