root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 4 Packages Projects Releases Wiki Activity
rabbitmq-server/deps/rabbitmq_mqtt/test/v5_SUITE.erl

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

2209 lines
101 KiB
Erlang
Raw Permalink Normal View History

Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%
(c) year bumps
2024-01-02 11:02:20 +08:00
%% Copyright (c) 2007-2025 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. All rights reserved.
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% This test suite covers MQTT 5.0 features.
-module(v5_SUITE).
-compile([export_all,
nowarn_export_all]).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
-include_lib("amqp_client/include/amqp_client.hrl").
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
-import(util,
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
[all_connection_pids/1,
Return Assigned Client Identifier in CONNACK "If the Client connects using a zero length Client Identifier, the Server MUST respond with a CONNACK containing an Assigned Client Identifier."
2023-02-28 21:51:16 +08:00
start_client/4,
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
connect/2, connect/3, connect/4,
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
assert_message_expiry_interval/2,
Implement client ReceiveMaximum - rename processor state prefetch to receive_maximum to better match property name for mqtt 5 - defaults to 10 (as previously) when not set not saved in session state, configuration is per connection
2023-03-29 18:40:40 +08:00
non_clean_sess_opts/0,
expect_publishes/3
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
]).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
-import(rabbit_ct_broker_helpers,
[rpc/4]).
-import(rabbit_ct_helpers,
[eventually/1]).
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
-define(APP, rabbitmq_mqtt).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
-define(QUEUE_TTL_KEY, <<"x-expires">>).
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
%% defined in MQTT v5 (not in v4 or v3)
-define(RC_SUCCESS, 16#00).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
-define(RC_NORMAL_DISCONNECTION, 16#00).
-define(RC_DISCONNECT_WITH_WILL, 16#04).
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
-define(RC_NO_SUBSCRIPTION_EXISTED, 16#11).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
-define(RC_UNSPECIFIED_ERROR, 16#80).
Support topic alias from client to broker
2023-06-08 22:12:43 +08:00
-define(RC_PROTOCOL_ERROR, 16#82).
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
-define(RC_SERVER_SHUTTING_DOWN, 16#8B).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
-define(RC_SESSION_TAKEN_OVER, 16#8E).
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
-define(RC_TOPIC_ALIAS_INVALID, 16#94).
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
-define(TIMEOUT, 30_000).
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
all() ->
mqtt tests: Move v5 web_mqtt tests out of rabbitmq_mqtt
2024-09-11 19:15:48 +08:00
[{group, mqtt}].
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
groups() ->
[
{mqtt, [],
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
[{cluster_size_1, [shuffle], cluster_size_1_tests()},
{cluster_size_3, [shuffle], cluster_size_3_tests()}
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
]}
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
].
cluster_size_1_tests() ->
[
client_set_max_packet_size_publish,
client_set_max_packet_size_connack,
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
client_set_max_packet_size_invalid,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
message_expiry,
message_expiry_will_message,
message_expiry_retained_message,
session_expiry_classic_queue_disconnect_decrease,
session_expiry_quorum_queue_disconnect_decrease,
session_expiry_disconnect_zero_to_non_zero,
session_expiry_disconnect_non_zero_to_zero,
session_expiry_disconnect_infinity_to_zero,
session_expiry_disconnect_to_infinity,
session_expiry_reconnect_non_zero,
session_expiry_reconnect_zero,
session_expiry_reconnect_infinity_to_zero,
Move test to v5 because it is a feature exclusive of v5
2025-06-04 22:14:46 +08:00
zero_session_expiry_disconnect_autodeletes_qos0_queue,
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
client_publish_qos2,
Close conn when will msg qos is 2 [MQTT-3.2.2-12] If a Server receives a CONNECT packet containing a Will QoS that exceeds its capabilities, it MUST reject the connection. It SHOULD use a CONNACK packet with Reason Code 0x9B (QoS not supported) as described in section 4.13 Handling errors, and MUST close the Network Connection
2023-03-06 21:26:00 +08:00
client_rejects_publish,
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
client_receive_maximum_min,
client_receive_maximum_large,
Simplify UNSUBACK reply Whether a payload is sent to the client is decided by the serialiser.
2023-04-19 20:53:01 +08:00
unsubscribe_success,
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
unsubscribe_topic_not_found,
subscription_option_no_local,
subscription_option_no_local_wildcards,
subscription_option_retain_as_published,
subscription_option_retain_as_published_wildcards,
Support MQTT 5.0 Subscription Option Retain Handling The MQTT v5 spec is a bit vague on Retain Handling 1: "If Retain Handling is set to 1 then if the subscription did not already exist, the Server MUST send all retained message matching the Topic Filter of the subscription to the Client, and if the subscription did exist the Server MUST NOT send the retained messages. [MQTT-3.3.1-10]." [v5 3.3.1.3] Does a subscription with the same topic filter but different subscription options mean that "the subscription did exist"? This commit interprets "subscription exists" as both topic filter and subscription options must be the same. Therefore, if a client creates a subscription with a topic filter that is identical to a previous subscription and subscription options that are different and Retain Handling 1, the server sends the retained message.
2023-05-09 17:59:10 +08:00
subscription_option_retain_handling,
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
subscription_identifier,
subscription_identifier_amqp091,
subscription_identifier_at_most_once_dead_letter,
at_most_once_dead_letter_detect_cycle,
subscription_options_persisted,
subscription_options_modify,
subscription_options_modify_qos1,
subscription_options_modify_qos0,
session_upgrade_v3_v5_qos1,
session_upgrade_v3_v5_qos0,
Fix MQTT crash This commit fixes the following crash: ``` 2388-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> exception error: bad argument 2389-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in function lists:keymember/3 2390-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> called as lists:keymember(<<"x-mqtt-publish-qos">>,1,undefined) 2391-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> *** argument 3: not a list 2392-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in call from rabbit_mqtt_processor:amqp_props_to_mqtt_props/2 (rabbit_mqtt_processor.erl, line 2219) ``` This crash occurs when a message without AMQP 0.9.1 #P_basic.headers is sent to the MQTT v4 connection process, but consumed by an MQTT v5 connection process. This is the case when an AMQP 0.9.1 client sends a message to a v4 MQTT client, and this same client subsequently upgrades to v5 consuming the message. When sending from AMQP 0.9.1 client directly to a v5 MQTT connection, the AMQP 0.9.1 header <<"x-binding-keys">> is set. When sending from AMQP 0.9.1 client directly to a v4 MQTT connection, no header is set, but this code branch was not evaluated.
2023-06-21 18:11:42 +08:00
session_upgrade_v3_v5_amqp091_pub,
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
compatibility_v3_v5,
session_upgrade_v3_v5_unsubscribe,
session_upgrade_v4_v5_no_queue_bind_permission,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
amqp091_cc_header,
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
publish_property_content_type,
publish_property_payload_format_indicator,
publish_property_response_topic_correlation_data,
publish_property_user_property,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
disconnect_with_will,
will_qos2,
will_delay_greater_than_session_expiry,
will_delay_less_than_session_expiry,
will_delay_equals_session_expiry,
will_delay_session_expiry_zero,
will_delay_reconnect_no_will,
will_delay_reconnect_with_will,
will_delay_session_takeover,
will_delay_message_expiry,
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
will_delay_message_expiry_publish_properties,
will_delay_properties,
will_properties,
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
retain_properties,
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
topic_alias_client_to_server,
topic_alias_server_to_client,
topic_alias_bidirectional,
topic_alias_invalid,
topic_alias_unknown,
CONNACK with Bad Authentication Method RabbitMQ MQTT already supports authenticating clients via username + password, OAuth tokens, and certificates. We could make use of RabbitMQ SASL mechanisms in the future, if needed. For now, if the client wants to perform extended authentication, we return Bad Authentication Method in the CONNACK packet.
2023-06-18 19:53:12 +08:00
topic_alias_disallowed,
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
topic_alias_retained_message,
topic_alias_disallowed_retained_message,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
extended_auth,
headers_exchange,
consistent_hash_exchange
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
].
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
cluster_size_3_tests() ->
[session_migrate_v3_v5,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_takeover_v3_v5,
will_delay_node_restart
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
].
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
suite() ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
[{timetrap, {minutes, 10}}].
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% -------------------------------------------------------------------
%% Testsuite setup/teardown.
%% -------------------------------------------------------------------
Mqtt test: Solve dependencies for web mqtt
2025-06-06 15:43:19 +08:00
init_per_suite(Config0) ->
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
rabbit_ct_helpers:log_environment(),
Mqtt test: Solve dependencies for web mqtt
2025-06-06 15:43:19 +08:00
Config = rabbit_ct_helpers:set_config(Config0, {test_plugins, [rabbitmq_mqtt]}),
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
rabbit_ct_helpers:run_setup_steps(Config).
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config).
init_per_group(mqtt, Config) ->
rabbit_ct_helpers:set_config(Config, {websocket, false});
init_per_group(Group, Config0) ->
Nodes = case Group of
cluster_size_1 -> 1;
cluster_size_3 -> 3
end,
Suffix = rabbit_ct_helpers:testcase_absname(Config0, "", "-"),
Config1 = rabbit_ct_helpers:set_config(
Config0,
[{mqtt_version, v5},
{rmq_nodes_count, Nodes},
Mqtt tests: start just required dependencies MQTT tests depend on a few plugins, which are just used in 1 or 2 suites each. These have caused issues in CI, triggering a bug in rabbitmq_federation where the mirrored supervisor submits a transaction while the cluster is being shut down. The transaction hangs and the whole rabbitmq_mqtt job times out. This bug has been addressed, however it is best to start just the required plugins on each SUITE.
2025-06-05 21:02:58 +08:00
{rmq_nodename_suffix, Suffix},
{start_rmq_with_plugins_disabled, true}
]),
Require MQTT feature flags in 4.0 Require all MQTT feature flags and remove their compatibility code: * delete_ra_cluster_mqtt_node * rabbit_mqtt_qos0_queue * mqtt_v5 These feature flags were introduced in or before 3.13.0.
2024-07-09 18:30:47 +08:00
Config = rabbit_ct_helpers:merge_app_env(
Config1,
{rabbit, [{quorum_tick_interval, 200}]}),
Mqtt tests: start just required dependencies MQTT tests depend on a few plugins, which are just used in 1 or 2 suites each. These have caused issues in CI, triggering a bug in rabbitmq_federation where the mirrored supervisor submits a transaction while the cluster is being shut down. The transaction hangs and the whole rabbitmq_mqtt job times out. This bug has been addressed, however it is best to start just the required plugins on each SUITE.
2025-06-05 21:02:58 +08:00
Config2 = rabbit_ct_helpers:run_steps(
Config,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
Mqtt test: Solve dependencies for web mqtt
2025-06-06 15:43:19 +08:00
[util:enable_plugin(Config2, Plugin) || Plugin <- ?config(test_plugins, Config2)],
Mqtt tests: start just required dependencies MQTT tests depend on a few plugins, which are just used in 1 or 2 suites each. These have caused issues in CI, triggering a bug in rabbitmq_federation where the mirrored supervisor submits a transaction while the cluster is being shut down. The transaction hangs and the whole rabbitmq_mqtt job times out. This bug has been addressed, however it is best to start just the required plugins on each SUITE.
2025-06-05 21:02:58 +08:00
Config2.
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
end_per_group(G, Config)
when G =:= cluster_size_1;
G =:= cluster_size_3 ->
Do not run_teardown_steps twice
2023-06-21 23:19:38 +08:00
rabbit_ct_helpers:run_steps(
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
Config,
rabbit_ct_client_helpers:teardown_steps() ++
rabbit_ct_broker_helpers:teardown_steps());
end_per_group(_, Config) ->
Config.
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
init_per_testcase(T, Config)
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
when T =:= session_expiry_disconnect_infinity_to_zero;
T =:= session_expiry_disconnect_to_infinity;
T =:= session_expiry_reconnect_infinity_to_zero ->
Make mqtt.subscription_ttl unsupported Starting with RabbitMQ 3.13 mqtt.max_session_expiry_interval_seconds (set in seconds) will replace the previous setting mqtt.subscription_ttl. MQTT 5.0 introduces the Session Expiry Interval feature which does not only apply to subscribers, but also to publishers. The new config name mqtt.max_session_expiry_interval_seconds makes it clear that it also applies to publishers. Prior to this commit, when mqtt.subscription_ttl was set, a warning got logged and the value was ignored. This is dangerous if an operator does not see the warning but relies for example on `mqtt.subscription = infinity` to not expire non clean session. It's safer to make the boot fail if that unsupported config name is still set. A clear error message will be logged: ``` [error] <0.142.0> Error preparing configuration in phase apply_translations: [error] <0.142.0> - Translation for 'rabbitmq_mqtt.subscription_ttl' found invalid configuration: Since 3.13 mqtt.subscription_ttl (in milliseconds) is unsupported. Use mqtt.max_session_expiry_interval_seconds (in seconds) instead. ``` Alternatively, RabbitMQ could translate mqtt.subscription_ttl to mqtt.max_session_expiry_interval_seconds. However, forcing the new config option sounds the better way to go. Once we write MQTT 5.0 docs, this change must go into the 3.13 release notes. This commit also renames max_session_expiry_interval_secs to max_session_expiry_interval_seconds. The latter is clearer to users.
2023-07-13 22:38:47 +08:00
Par = max_session_expiry_interval_seconds,
Fix two small bugs
2023-04-28 23:15:19 +08:00
{ok, Default} = rpc(Config, application, get_env, [?APP, Par]),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
ok = rpc(Config, application, set_env, [?APP, Par, infinity]),
Config1 = rabbit_ct_helpers:set_config(Config, {Par, Default}),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
init_per_testcase0(T, Config1);
Move test to v5 because it is a feature exclusive of v5
2025-06-04 22:14:46 +08:00
init_per_testcase(T, Config)
when T =:= zero_session_expiry_disconnect_autodeletes_qos0_queue ->
rpc(Config, rabbit_registry, register, [queue, <<"qos0">>, rabbit_mqtt_qos0_queue]),
init_per_testcase0(T, Config);
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
init_per_testcase(T, Config) ->
init_per_testcase0(T, Config).
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
init_per_testcase0(Testcase, Config) ->
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
rabbit_ct_helpers:testcase_started(Config, Testcase).
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
end_per_testcase(T, Config)
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
when T =:= session_expiry_disconnect_infinity_to_zero;
T =:= session_expiry_disconnect_to_infinity;
T =:= session_expiry_reconnect_infinity_to_zero ->
Make mqtt.subscription_ttl unsupported Starting with RabbitMQ 3.13 mqtt.max_session_expiry_interval_seconds (set in seconds) will replace the previous setting mqtt.subscription_ttl. MQTT 5.0 introduces the Session Expiry Interval feature which does not only apply to subscribers, but also to publishers. The new config name mqtt.max_session_expiry_interval_seconds makes it clear that it also applies to publishers. Prior to this commit, when mqtt.subscription_ttl was set, a warning got logged and the value was ignored. This is dangerous if an operator does not see the warning but relies for example on `mqtt.subscription = infinity` to not expire non clean session. It's safer to make the boot fail if that unsupported config name is still set. A clear error message will be logged: ``` [error] <0.142.0> Error preparing configuration in phase apply_translations: [error] <0.142.0> - Translation for 'rabbitmq_mqtt.subscription_ttl' found invalid configuration: Since 3.13 mqtt.subscription_ttl (in milliseconds) is unsupported. Use mqtt.max_session_expiry_interval_seconds (in seconds) instead. ``` Alternatively, RabbitMQ could translate mqtt.subscription_ttl to mqtt.max_session_expiry_interval_seconds. However, forcing the new config option sounds the better way to go. Once we write MQTT 5.0 docs, this change must go into the 3.13 release notes. This commit also renames max_session_expiry_interval_secs to max_session_expiry_interval_seconds. The latter is clearer to users.
2023-07-13 22:38:47 +08:00
Par = max_session_expiry_interval_seconds,
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
Default = ?config(Par, Config),
ok = rpc(Config, application, set_env, [?APP, Par, Default]),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end_per_testcase0(T, Config);
Move test to v5 because it is a feature exclusive of v5
2025-06-04 22:14:46 +08:00
end_per_testcase(T, Config)
when T =:= zero_session_expiry_disconnect_autodeletes_qos0_queue ->
ok = rpc(Config, rabbit_registry, unregister, [queue, <<"qos0">>]),
init_per_testcase0(T, Config);
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end_per_testcase(T, Config) ->
end_per_testcase0(T, Config).
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end_per_testcase0(Testcase, Config) ->
v5_SUITE: Close all connections in `end_per_testcase/2` [Why] Many tests do not clean up their connections if they encounter a failure. This affects subsequent testcases negatively.
2025-03-03 23:56:36 +08:00
%% Terminate all connections and wait for sessions to terminate before
%% starting the next test case.
_ = rabbit_ct_broker_helpers:rpc(
Config, 0,
rabbit_networking, close_all_connections, [<<"test finished">>]),
_ = rabbit_ct_broker_helpers:rpc_all(
Config,
rabbit_mqtt, close_local_client_connections, [normal]),
eventually(?_assertEqual(
[],
rpc(Config, rabbit_mqtt, local_connection_pids, []))),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% Assert that every testcase cleaned up their MQTT sessions.
v5_SUITE: Close all connections in `end_per_testcase/2` [Why] Many tests do not clean up their connections if they encounter a failure. This affects subsequent testcases negatively.
2025-03-03 23:56:36 +08:00
rabbit_ct_broker_helpers:rpc(Config, 0, ?MODULE, delete_queues, []),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
eventually(?_assertEqual([], rpc(Config, rabbit_amqqueue, list, []))),
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
rabbit_ct_helpers:testcase_finished(Config, Testcase).
v5_SUITE: Close all connections in `end_per_testcase/2` [Why] Many tests do not clean up their connections if they encounter a failure. This affects subsequent testcases negatively.
2025-03-03 23:56:36 +08:00
delete_queues() ->
_ = [catch rabbit_amqqueue:delete(Q, false, false, <<"test finished">>)
|| Q <- rabbit_amqqueue:list()],
ok.
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% -------------------------------------------------------------------
%% Testsuite cases
%% -------------------------------------------------------------------
client_set_max_packet_size_publish(Config) ->
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
NumRejectedBefore = dead_letter_metric(messages_dead_lettered_rejected_total, Config),
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
MaxPacketSize = 500,
C = connect(ClientId, Config, [{properties, #{'Maximum-Packet-Size' => MaxPacketSize}}]),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
PayloadTooLarge = binary:copy(<<"x">>, MaxPacketSize + 1),
%% We expect the PUBLISH from client to server to succeed.
?assertMatch({ok, _}, emqtt:publish(C, Topic, PayloadTooLarge, [{qos, 1}])),
%% We expect the server to drop the PUBLISH packet prior to sending to the client
%% because the packet is larger than what the client is able to receive.
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
assert_nothing_received(),
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
NumRejected = dead_letter_metric(messages_dead_lettered_rejected_total, Config) - NumRejectedBefore,
?assertEqual(1, NumRejected),
MC: various changes and improvements To refine conversion behaviour add additional tests and ensure it matches the documentation. mc: optionally capture source environment And pass target environment to mc:convert This allows environmental data and configuration to be captured and used to modify and complete conversion logic whilst allowing conversion code to remain pure and portable.
2023-10-26 20:32:37 +08:00
ok = emqtt:disconnect(C),
ok.
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
client_set_max_packet_size_connack(Config) ->
Return Assigned Client Identifier in CONNACK "If the Client connects using a zero length Client Identifier, the Server MUST respond with a CONNACK containing an Assigned Client Identifier."
2023-02-28 21:51:16 +08:00
{C, Connect} = start_client(?FUNCTION_NAME, Config, 0,
[{properties, #{'Maximum-Packet-Size' => 2}},
{connect_timeout, 1}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
unlink(C),
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
%% We expect the server to drop the CONNACK packet because it's larger than 2 bytes.
?assertEqual({error, connack_timeout}, Connect(C)).
%% "It is a Protocol Error to include the Receive Maximum
%% value more than once or for it to have the value 0."
client_set_max_packet_size_invalid(Config) ->
Return Assigned Client Identifier in CONNACK "If the Client connects using a zero length Client Identifier, the Server MUST respond with a CONNACK containing an Assigned Client Identifier."
2023-02-28 21:51:16 +08:00
{C, Connect} = start_client(?FUNCTION_NAME, Config, 0,
[{properties, #{'Maximum-Packet-Size' => 0}}]),
Add MQTT v5 feature Maximum Packet Size set by client "Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet." This commit implements the part where the Client specifies the maximum packet size. As per protocol spec, instead of sending, the server drops the MQTT packet if it's too large. A debug message is logged for "infrequent" packet types. For PUBLISH packets, the messages is rejected to the queue such that it will be dead lettered, if dead lettering is configured. At the very least, Prometheus metrics for dead lettered messages will be increased, even if dead lettering is not configured.
2023-02-24 00:49:33 +08:00
unlink(C),
?assertMatch({error, _}, Connect(C)).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
message_expiry(Config) ->
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
NumExpiredBefore = dead_letter_metric(messages_dead_lettered_expired_total, Config),
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Pub = connect(<<"publisher">>, Config),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Sub1 = connect(ClientId, Config, non_clean_sess_opts()),
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
{ok, _, [1]} = emqtt:subscribe(Sub1, Topic, qos1),
ok = emqtt:disconnect(Sub1),
{ok, _} = emqtt:publish(Pub, Topic, #{'Message-Expiry-Interval' => 1}, <<"m1">>, [{qos, 1}]),
{ok, _} = emqtt:publish(Pub, Topic, #{}, <<"m2">>, [{qos, 1}]),
{ok, _} = emqtt:publish(Pub, Topic, #{'Message-Expiry-Interval' => 10}, <<"m3">>, [{qos, 1}]),
{ok, _} = emqtt:publish(Pub, Topic, #{'Message-Expiry-Interval' => 2}, <<"m4">>, [{qos, 1}]),
timer:sleep(2001),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Sub2 = connect(ClientId, Config, non_clean_sess_opts()),
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
receive {publish, #{client_pid := Sub2,
topic := Topic,
payload := <<"m2">>,
properties := Props}}
when map_size(Props) =:= 0 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
end,
receive {publish, #{client_pid := Sub2,
topic := Topic,
payload := <<"m3">>,
%% "The PUBLISH packet sent to a Client by the Server MUST contain a Message
%% Expiry Interval set to the received value minus the time that the
%% Application Message has been waiting in the Server" [MQTT-3.3.2-6]
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
properties := #{'Message-Expiry-Interval' := MEI}}} ->
assert_message_expiry_interval(10 - 2, MEI)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m3")
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
end,
assert_nothing_received(),
NumExpired = dead_letter_metric(messages_dead_lettered_expired_total, Config) - NumExpiredBefore,
?assertEqual(2, NumExpired),
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(Sub2),
Sub3 = connect(ClientId, Config, [{clean_start, true}]),
ok = emqtt:disconnect(Sub3).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
message_expiry_will_message(Config) ->
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
NumExpiredBefore = dead_letter_metric(messages_dead_lettered_expired_total, Config),
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Opts = [{will_topic, Topic},
{will_payload, <<"will payload">>},
{will_qos, 1},
{will_props, #{'Message-Expiry-Interval' => 1}}
],
Pub = connect(<<"will-publisher">>, Config, Opts),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Sub1 = connect(ClientId, Config, non_clean_sess_opts()),
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
{ok, _, [1]} = emqtt:subscribe(Sub1, Topic, qos1),
ok = emqtt:disconnect(Sub1),
unlink(Pub),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
erlang:exit(Pub, trigger_will_message),
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
%% Wait for will message to expire.
timer:sleep(1100),
NumExpired = dead_letter_metric(messages_dead_lettered_expired_total, Config) - NumExpiredBefore,
?assertEqual(1, NumExpired),
Sub2 = connect(ClientId, Config, [{clean_start, true}]),
assert_nothing_received(),
ok = emqtt:disconnect(Sub2).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
message_expiry_retained_message(Config) ->
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
Pub = connect(<<"publisher">>, Config),
{ok, _} = emqtt:publish(Pub, <<"topic1">>, #{'Message-Expiry-Interval' => 100},
<<"m1.1">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic2">>, #{'Message-Expiry-Interval' => 2},
<<"m2">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic3">>, #{'Message-Expiry-Interval' => 100},
<<"m3.1">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic4">>, #{'Message-Expiry-Interval' => 100},
<<"m4">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic1">>, #{'Message-Expiry-Interval' => 2},
<<"m1.2">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic2">>, #{'Message-Expiry-Interval' => 2},
<<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Pub, <<"topic3">>, #{},
<<"m3.2">>, [{retain, true}, {qos, 1}]),
timer:sleep(2001),
%% Expectations:
%% topic1 expired because 2 seconds elapsed
%% topic2 is not retained because it got deleted
%% topic3 is retained because its new message does not have an Expiry-Interval set
%% topic4 is retained because 100 seconds have not elapsed
Sub = connect(<<"subscriber">>, Config),
{ok, _, [1,1,1,1]} = emqtt:subscribe(Sub, [{<<"topic1">>, qos1},
{<<"topic2">>, qos1},
{<<"topic3">>, qos1},
{<<"topic4">>, qos1}]),
receive {publish, #{client_pid := Sub,
retain := true,
topic := <<"topic3">>,
payload := <<"m3.2">>,
properties := Props}}
when map_size(Props) =:= 0 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not topic3")
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
end,
receive {publish, #{client_pid := Sub,
retain := true,
topic := <<"topic4">>,
payload := <<"m4">>,
properties := #{'Message-Expiry-Interval' := MEI}}} ->
assert_message_expiry_interval(100 - 2, MEI)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive topic4")
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
end,
assert_nothing_received(),
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(Sub).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_classic_queue_disconnect_decrease(Config) ->
ok = session_expiry_disconnect_decrease(rabbit_classic_queue, Config).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_quorum_queue_disconnect_decrease(Config) ->
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
ok = rpc(Config, application, set_env, [?APP, durable_queue_type, quorum]),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = session_expiry_disconnect_decrease(rabbit_quorum_queue, Config),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
ok = rpc(Config, application, unset_env, [?APP, durable_queue_type]).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Move test to v5 because it is a feature exclusive of v5
2025-06-04 22:14:46 +08:00
zero_session_expiry_disconnect_autodeletes_qos0_queue(Config) ->
ClientId = ?FUNCTION_NAME,
C = connect(ClientId, Config, [
{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 0}}]),
{ok, _, _} = emqtt:subscribe(C, <<"topic0">>, qos0),
QsQos0 = rpc(Config, rabbit_amqqueue, list_by_type, [rabbit_mqtt_qos0_queue]),
?assertEqual(1, length(QsQos0)),
ok = emqtt:disconnect(C),
%% After terminating a clean session, we expect any session state to be cleaned up on the server.
MQTT: correct a comment in v5_SUITE #14006
2025-06-04 22:30:31 +08:00
%% Give the node some time to clean up the MQTT QoS 0 queue.
timer:sleep(200),
Move test to v5 because it is a feature exclusive of v5
2025-06-04 22:14:46 +08:00
L = rpc(Config, rabbit_amqqueue, list, []),
?assertEqual(0, length(L)).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_disconnect_decrease(QueueType, Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 100}}]),
{ok, _, [1]} = emqtt:subscribe(C1, <<"t/1">>, qos1),
[Q1] = rpc(Config, rabbit_amqqueue, list, []),
?assertEqual(QueueType,
amqqueue:get_type(Q1)),
?assertEqual({long, 100_000},
rabbit_misc:table_lookup(amqqueue:get_arguments(Q1), ?QUEUE_TTL_KEY)),
%% DISCONNECT decreases Session Expiry Interval from 100 seconds to 1 second.
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1, ?RC_NORMAL_DISCONNECTION, #{'Session-Expiry-Interval' => 1}),
Fix MQTT test flake
2023-09-12 16:22:30 +08:00
%% Wait a bit since DISCONNECT is async.
timer:sleep(50),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
assert_queue_ttl(1, 1, Config),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
timer:sleep(1500),
C2 = connect(ClientId, Config, [{clean_start, false}]),
%% Server should reply in CONNACK that it does not have session state for our client ID.
?assertEqual({session_present, 0},
proplists:lookup(session_present, emqtt:info(C2))),
ok = emqtt:disconnect(C2).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_disconnect_zero_to_non_zero(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 0}}]),
{ok, _, [1]} = emqtt:subscribe(C1, <<"t/1">>, qos1),
%% "If the Session Expiry Interval in the CONNECT packet was zero, then it is a Protocol
%% Error to set a non-zero Session Expiry Interval in the DISCONNECT packet sent by the Client.
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1, ?RC_NORMAL_DISCONNECTION, #{'Session-Expiry-Interval' => 60}),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C2 = connect(ClientId, Config, [{clean_start, false}]),
%% Due to the prior protocol error, we expect the requested session expiry interval of
%% 60 seconds not to be applied. Therefore, the server should reply in CONNACK that
%% it does not have session state for our client ID.
?assertEqual({session_present, 0},
proplists:lookup(session_present, emqtt:info(C2))),
ok = emqtt:disconnect(C2).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_disconnect_non_zero_to_zero(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 60}}]),
{ok, _, [0, 1]} = emqtt:subscribe(C1, [{<<"t/0">>, qos0},
{<<"t/1">>, qos1}]),
?assertEqual(2, rpc(Config, rabbit_amqqueue, count, [])),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1, ?RC_NORMAL_DISCONNECTION, #{'Session-Expiry-Interval' => 0}),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
eventually(?_assertEqual(0, rpc(Config, rabbit_amqqueue, count, []))),
C2 = connect(ClientId, Config, [{clean_start, false}]),
?assertEqual({session_present, 0},
proplists:lookup(session_present, emqtt:info(C2))),
ok = emqtt:disconnect(C2).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_disconnect_infinity_to_zero(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 16#FFFFFFFF}}]),
{ok, _, [1, 0]} = emqtt:subscribe(C1, [{<<"t/1">>, qos1},
{<<"t/0">>, qos0}]),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
assert_no_queue_ttl(2, Config),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1, ?RC_NORMAL_DISCONNECTION, #{'Session-Expiry-Interval' => 0}),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
eventually(?_assertEqual(0, rpc(Config, rabbit_amqqueue, count, []))).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_disconnect_to_infinity(Config) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
ClientId = ?FUNCTION_NAME,
%% Connect with a non-zero and non-infinity Session Expiry Interval.
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 1}}]),
{ok, _, [0, 1]} = emqtt:subscribe(C1, [{<<"t/0">>, qos0},
{<<"t/1">>, qos1}]),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
assert_queue_ttl(1, 2, Config),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
%% Disconnect with infinity should remove queue TTL from both queues.
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = emqtt:disconnect(C1, ?RC_NORMAL_DISCONNECTION, #{'Session-Expiry-Interval' => 16#FFFFFFFF}),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
timer:sleep(100),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
assert_no_queue_ttl(2, Config),
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
C2 = connect(ClientId, Config, [{clean_start, true}]),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
ok = emqtt:disconnect(C2).
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_reconnect_non_zero(Config) ->
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 60}}]),
{ok, _, [0, 1]} = emqtt:subscribe(C1, [{<<"t/0">>, qos0},
{<<"t/1">>, qos1}]),
assert_queue_ttl(60, 2, Config),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 1}}]),
?assertEqual({session_present, 1},
proplists:lookup(session_present, emqtt:info(C2))),
assert_queue_ttl(1, 2, Config),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
ok = emqtt:disconnect(C2),
C3 = connect(ClientId, Config, [{clean_start, true}]),
ok = emqtt:disconnect(C3).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_reconnect_zero(Config) ->
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 60}}]),
{ok, _, [0, 1]} = emqtt:subscribe(C1, [{<<"t/0">>, qos0},
{<<"t/1">>, qos1}]),
assert_queue_ttl(60, 2, Config),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 0}}]),
?assertEqual({session_present, 1},
proplists:lookup(session_present, emqtt:info(C2))),
assert_queue_ttl(0, 2, Config),
ok = emqtt:disconnect(C2).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
session_expiry_reconnect_infinity_to_zero(Config) ->
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 16#FFFFFFFF}}]),
{ok, _, [0, 1]} = emqtt:subscribe(C1, [{<<"t/0">>, qos0},
{<<"t/1">>, qos1}]),
assert_no_queue_ttl(2, Config),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, false}]),
?assertEqual({session_present, 1},
proplists:lookup(session_present, emqtt:info(C2))),
assert_queue_ttl(0, 2, Config),
ok = emqtt:disconnect(C2).
Disconnect at pub qos > server max qos - "If the Server included a Maximum QoS in its CONNACK response to a Client and it receives a PUBLISH packet with a QoS greater than this then it uses DISCONNECT with Reason Code 0x9B (QoS not supported)" - only affects mqtt v5, server max qos is 1
2023-03-03 01:48:43 +08:00
client_publish_qos2(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
{C, Connect} = start_client(ClientId, Config, 0, []),
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
?assertMatch({ok, #{'Maximum-QoS' := 1}}, Connect(C)),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
unlink(C),
Add Message Expiry Interval for retained messages MQTT v5 spec: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic." This commit also supports Message Expiry Interval for retained messages when a node is restarted. Therefore, the insertion timestamp needs to be stored on disk. Upon recovery, the Erlang timers are re-created.
2023-03-04 00:09:36 +08:00
?assertEqual({error, {disconnected, _RcQosNotSupported = 155, #{}}},
Remove code duplication
2023-03-16 16:58:46 +08:00
emqtt:publish(C, Topic, <<"msg">>, qos2)).
Disconnect at pub qos > server max qos - "If the Server included a Maximum QoS in its CONNACK response to a Client and it receives a PUBLISH packet with a QoS greater than this then it uses DISCONNECT with Reason Code 0x9B (QoS not supported)" - only affects mqtt v5, server max qos is 1
2023-03-03 01:48:43 +08:00
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
client_rejects_publish(Config) ->
NumRejectedBefore = dead_letter_metric(messages_dead_lettered_rejected_total, Config),
Payload = Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config, [{auto_ack, false}]),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
{ok, _} = emqtt:publish(C, Topic, Payload, qos1),
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
receive {publish, #{payload := Payload,
packet_id := PacketId}} ->
%% Negatively ack the PUBLISH.
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
emqtt:puback(C, PacketId, ?RC_UNSPECIFIED_ERROR)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT ->
Dead letter negatively ACKed MQTT v5 messages MQTT v5 allows client and server to negatively ack a message by setting a reason code of 128 or greater indicating failure. "If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2]." Even though the spec prohibits resending such messages, if a client does not accept a message, RabbitMQ can still dead letter the message.
2023-03-04 02:16:20 +08:00
ct:fail("did not receive PUBLISH")
end,
%% Even though we nacked the PUBLISH, we expect the server to not re-send the same message:
%% "If PUBACK [...] is received containing a Reason Code of 0x80 or greater the corresponding
%% PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted" [MQTT-4.4.0-2].
assert_nothing_received(),
%% However, we expect RabbitMQ to dead letter negatively acknowledged messages.
NumRejected = dead_letter_metric(messages_dead_lettered_rejected_total, Config) - NumRejectedBefore,
?assertEqual(1, NumRejected),
ok = emqtt:disconnect(C).
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
client_receive_maximum_min(Config) ->
Implement client ReceiveMaximum - rename processor state prefetch to receive_maximum to better match property name for mqtt 5 - defaults to 10 (as previously) when not set not saved in session state, configuration is per connection
2023-03-29 18:40:40 +08:00
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
C = connect(ClientId, Config, [{auto_ack, false},
%% Minimum allowed Receive Maximum is 1.
{properties, #{'Receive-Maximum' => 1}}]),
Implement client ReceiveMaximum - rename processor state prefetch to receive_maximum to better match property name for mqtt 5 - defaults to 10 (as previously) when not set not saved in session state, configuration is per connection
2023-03-29 18:40:40 +08:00
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
{ok, _} = emqtt:publish(C, Topic, <<"m1">>, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"m2">>, qos1),
%% Since client set Receive Maximum is 1, at most 1 QoS 1 message should be in
%% flight from server to client at any given point in time.
PacketId1 = receive {publish, #{payload := <<"m1">>,
packet_id := Id}} ->
Id
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT ->
Add a test case for large Receive Maximum value
2023-04-19 16:34:46 +08:00
ct:fail("did not receive m1")
end,
assert_nothing_received(),
%% Only when we ack the 1st message, we expect to receive the 2nd message.
emqtt:puback(C, PacketId1),
ok = expect_publishes(C, Topic, [<<"m2">>]),
ok = emqtt:disconnect(C).
client_receive_maximum_large(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config, [{auto_ack, false},
{properties, #{'Receive-Maximum' => 1_000}}]),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
%% We know that the configured mqtt.prefetch is 10.
Prefetch = 10,
Payloads = [integer_to_binary(N) || N <- lists:seq(1, Prefetch)],
[{ok, _} = emqtt:publish(C, Topic, P, qos1) || P <- Payloads],
{ok, _} = emqtt:publish(C, Topic, <<"I wait in the queue">>, qos1),
ok = expect_publishes(C, Topic, Payloads),
%% We expect the server to cap the number of in flight QoS 1 messages sent to the
%% client to the configured mqtt.prefetch value even though the client set a larger
%% Receive Maximum value.
assert_nothing_received(),
Implement client ReceiveMaximum - rename processor state prefetch to receive_maximum to better match property name for mqtt 5 - defaults to 10 (as previously) when not set not saved in session state, configuration is per connection
2023-03-29 18:40:40 +08:00
ok = emqtt:disconnect(C).
Simplify UNSUBACK reply Whether a payload is sent to the client is decided by the serialiser.
2023-04-19 20:53:01 +08:00
unsubscribe_success(Config) ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
C = connect(?FUNCTION_NAME, Config),
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
{ok, _, [1]} = emqtt:subscribe(C, <<"topic/1">>, qos1),
{ok, _, [0]} = emqtt:subscribe(C, <<"topic/0">>, qos0),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertMatch({ok, _, [?RC_SUCCESS, ?RC_SUCCESS]},
emqtt:unsubscribe(C, [<<"topic/1">>, <<"topic/0">>])),
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
ok = emqtt:disconnect(C).
Simplify UNSUBACK reply Whether a payload is sent to the client is decided by the serialiser.
2023-04-19 20:53:01 +08:00
unsubscribe_topic_not_found(Config) ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
C = connect(?FUNCTION_NAME, Config),
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
{ok, _, [1]} = emqtt:subscribe(C, <<"topic/1">>, qos1),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertMatch({ok, _, [?RC_SUCCESS, ?RC_NO_SUBSCRIPTION_EXISTED]},
emqtt:unsubscribe(C, [<<"topic/1">>, <<"topic/0">>])),
ok = emqtt:disconnect(C).
subscription_option_no_local(Config) ->
C = connect(?FUNCTION_NAME, Config),
Other = connect(<<"other">>, Config),
{ok, _, [0, 0]} = emqtt:subscribe(C, [{<<"t/1">>, [{nl, true}]},
{<<"t/2">>, [{nl, false}]}]),
{ok, _, [0]} = emqtt:subscribe(Other, [{<<"t/1">>, [{nl, true}]}]),
ok = emqtt:publish(C, <<"t/1">>, <<"m1">>),
ok = emqtt:publish(C, <<"t/2">>, <<"m2">>),
ok = expect_publishes(Other, <<"t/1">>, [<<"m1">>]),
ok = expect_publishes(C, <<"t/2">>, [<<"m2">>]),
%% We expect C to not receive m1.
assert_nothing_received(),
ok = emqtt:disconnect(C),
ok = emqtt:disconnect(Other).
subscription_option_no_local_wildcards(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, [0, 0, 0, 0]} =
emqtt:subscribe(C, [{<<"+/1">>, [{nl, true}]},
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
{<<"t/1/#">>, [{nl, false}]},
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
{<<"+/2">>, [{nl, true}]},
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
{<<"t/2/#">>, [{nl, true}]}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
%% Matches the first two subscriptions.
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
%% Not all matching subscriptions have the No Local option set.
%% Therefore, we should receive m1.
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:publish(C, <<"t/1">>, <<"m1">>),
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
ok = expect_publishes(C, <<"t/1">>, [<<"m1">>]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
%% Matches the last two subscriptions.
Return matched binding keys faster For MQTT 5.0 destination queues, the topic exchange does not only have to return the destination queue names, but also the matched binding keys. This is needed to implement MQTT 5.0 subscription options No Local, Retain As Published and Subscription Identifiers. Prior to this commit, as the trie was walked down, we remembered the edges being walked and assembled the final binding key with list_to_binary/1. list_to_binary/1 is very expensive with long lists (long topic names), even in OTP 26. The CPU flame graph showed ~3% of CPU usage was spent only in list_to_binary/1. Unfortunately and unnecessarily, the current topic exchange implementation stores topic levels as lists. It would be better to store topic levels as binaries: split_topic_key/1 should ideally use binary:split/3 similar as follows: ``` 1> P = binary:compile_pattern(<<".">>). {bm,#Ref<0.1273071188.1488322568.63736>} 2> Bin = <<"aaa.bbb..ccc">>. <<"aaa.bbb..ccc">> 3> binary:split(Bin, P, [global]). [<<"aaa">>,<<"bbb">>,<<>>,<<"ccc">>] ``` The compiled pattern could be placed into persistent term. This commit decided to avoid migrating Mnesia tables to use binaries instead of lists. Mnesia migrations are non-trivial, especially with the current feature flag subsystem. Furthermore the Mnesia topic tables are already getting migrated to their Khepri counterparts in 3.13. Adding additional migration only for Mnesia does not make sense. So, instead of assembling the binding key as we walk down the trie and then calling list_to_binary/1 in the leaf, it would be better to just fetch the binding key from the database in the leaf. As we reach the leaf of the trie, we know both source and destination. Unfortunately, we cannot fetch the binding key efficiently with the current rabbit_route (sorted by source exchange) and rabbit_reverse_route (sorted by destination) tables as the key is in the middle between source and destination. If there are a huge number of bindings for a given sourc exchange (very realistic in MQTT use cases) or a large number of bindings for a given destination (also realistic), it would require scanning these large number of bindings. Therefore this commit takes the simplest possible solution: The solution leverages the fact that binding arguments are already part of table rabbit_topic_trie_binding. So, if we simply include the binding key into the binding arguments, we can fetch and return it efficiently in the topic exchange implementation. The following patch omitting fetching the empty list binding argument (the default) makes routing slower because function `analyze_pattern.constprop.0` requires significantly more (~2.5%) CPU time ``` @@ -273,7 +273,11 @@ trie_bindings(X, Node) -> node_id = Node, destination = '$1', arguments = '$2'}}, - mnesia:select(?MNESIA_BINDING_TABLE, [{MatchHead, [], [{{'$1', '$2'}}]}]). + mnesia:select( + ?MNESIA_BINDING_TABLE, + [{MatchHead, [{'andalso', {'is_list', '$2'}, {'=/=', '$2', []}}], [{{'$1', '$2'}}]}, + {MatchHead, [], ['$1']} + ]). ``` Hence, this commit always fetches the binding arguments. All MQTT 5.0 destination queues will create a binding that contains the binding key in the binding arguments. Not only does this solution avoid expensive list_to_binay/1 calls, but it also means that Erlang app rabbit (specifically the topic exchange implementation) does not need to be aware of MQTT anymore: It just returns the binding key when the binding args tell to do so. In future, once the Khepri migration completed, we should be able to relatively simply remove the binding key from the binding arguments again to free up some storage space. Note that one of the advantages of a trie data structue is its space efficiency that you don't have to store the same prefixes multiple times. However, for RabbitMQ the binding key is already stored at least N times in various routing tables, so storing it a few times more via the binding arguments should be acceptable. The speed improvements are favoured over a few more MBs ETS usage.
2023-06-09 18:52:34 +08:00
%% All matching subscriptions have the No Local option set.
%% Therefore, we should not receive m2.
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:publish(C, <<"t/2">>, <<"m2">>),
assert_nothing_received(),
ok = emqtt:disconnect(C).
subscription_option_retain_as_published(Config) ->
C1 = connect(<<"c1">>, Config),
C2 = connect(<<"c2">>, Config),
{ok, _, [0, 0]} = emqtt:subscribe(C1, [{<<"t/1">>, [{rap, true}]},
{<<"t/2">>, [{rap, false}]}]),
{ok, _, [0]} = emqtt:subscribe(C2, [{<<"t/1">>, [{rap, true}]}]),
ok = emqtt:publish(C1, <<"t/1">>, <<"m1">>, [{retain, true}]),
ok = emqtt:publish(C1, <<"t/2">>, <<"m2">>, [{retain, true}]),
receive {publish, #{client_pid := C1,
topic := <<"t/1">>,
payload := <<"m1">>,
retain := true}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish, #{client_pid := C1,
topic := <<"t/2">>,
payload := <<"m2">>,
retain := false}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish, #{client_pid := C2,
topic := <<"t/1">>,
payload := <<"m1">>,
retain := true}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(C1, <<"t/1">>, <<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C1, <<"t/2">>, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:disconnect(C1),
ok = emqtt:disconnect(C2).
subscription_option_retain_as_published_wildcards(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, [0, 0, 0, 0]} = emqtt:subscribe(C, [{<<"+/1">>, [{rap, false}]},
{<<"t/1/#">>, [{rap, false}]},
{<<"+/2">>, [{rap, false}]},
{<<"t/2/#">>, [{rap, true}]}]),
%% Matches the first two subscriptions.
ok = emqtt:publish(C, <<"t/1">>, <<"m1">>, [{retain, true}]),
%% Matches the last two subscriptions.
ok = emqtt:publish(C, <<"t/2">>, <<"m2">>, [{retain, true}]),
receive {publish, #{topic := <<"t/1">>,
payload := <<"m1">>,
%% No matching subscription has the
%% Retain As Published option set.
retain := false}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish, #{topic := <<"t/2">>,
payload := <<"m2">>,
%% (At least) one matching subscription has the
%% Retain As Published option set.
retain := true}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(C, <<"t/1">>, <<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C, <<"t/2">>, <<>>, [{retain, true}, {qos, 1}]),
Add return codes in unsuback packet
2023-04-12 00:01:48 +08:00
ok = emqtt:disconnect(C).
Support MQTT 5.0 Subscription Option Retain Handling The MQTT v5 spec is a bit vague on Retain Handling 1: "If Retain Handling is set to 1 then if the subscription did not already exist, the Server MUST send all retained message matching the Topic Filter of the subscription to the Client, and if the subscription did exist the Server MUST NOT send the retained messages. [MQTT-3.3.1-10]." [v5 3.3.1.3] Does a subscription with the same topic filter but different subscription options mean that "the subscription did exist"? This commit interprets "subscription exists" as both topic filter and subscription options must be the same. Therefore, if a client creates a subscription with a topic filter that is identical to a previous subscription and subscription options that are different and Retain Handling 1, the server sends the retained message.
2023-05-09 17:59:10 +08:00
subscription_option_retain_handling(Config) ->
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, non_clean_sess_opts()),
{ok, _} = emqtt:publish(C1, <<"t/1">>, <<"m1">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C1, <<"t/2">>, <<"m2">>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C1, <<"t/3">>, <<"m3">>, [{retain, true}, {qos, 1}]),
{ok, _, [1, 1, 1]} = emqtt:subscribe(C1, [{<<"t/1">>, [{rh, 0}, {qos, 1}]},
%% Subscription does not exist.
{<<"t/2">>, [{rh, 1}, {qos, 1}]},
{<<"t/3">>, [{rh, 2}, {qos, 1}]}]),
ok = expect_publishes(C1, <<"t/1">>, [<<"m1">>]),
ok = expect_publishes(C1, <<"t/2">>, [<<"m2">>]),
assert_nothing_received(),
{ok, _, [1, 1, 1]} = emqtt:subscribe(C1, [{<<"t/1">>, [{rh, 0}, {qos, 1}]},
%% Subscription exists.
{<<"t/2">>, [{rh, 1}, {qos, 1}]},
{<<"t/3">>, [{rh, 2}, {qos, 1}]}]),
ok = expect_publishes(C1, <<"t/1">>, [<<"m1">>]),
assert_nothing_received(),
{ok, _, [0, 0, 0]} = emqtt:subscribe(C1, [{<<"t/1">>, [{rh, 0}, {qos, 0}]},
%% That specific subscription does not exist.
{<<"t/2">>, [{rh, 1}, {qos, 0}]},
{<<"t/3">>, [{rh, 2}, {qos, 0}]}]),
ok = expect_publishes(C1, <<"t/1">>, [<<"m1">>]),
ok = expect_publishes(C1, <<"t/2">>, [<<"m2">>]),
assert_nothing_received(),
ok = emqtt:disconnect(C1),
C2 = connect(ClientId, Config, [{clean_start, false}]),
{ok, _, [0, 0, 0]} = emqtt:subscribe(C2, [{<<"t/1">>, [{rh, 0}, {qos, 0}]},
%% Subscription exists.
{<<"t/2">>, [{rh, 1}, {qos, 0}]},
{<<"t/3">>, [{rh, 2}, {qos, 0}]}]),
ok = expect_publishes(C2, <<"t/1">>, [<<"m1">>]),
assert_nothing_received(),
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(C2, <<"t/1">>, <<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C2, <<"t/2">>, <<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C2, <<"t/3">>, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 Subscription Option Retain Handling The MQTT v5 spec is a bit vague on Retain Handling 1: "If Retain Handling is set to 1 then if the subscription did not already exist, the Server MUST send all retained message matching the Topic Filter of the subscription to the Client, and if the subscription did exist the Server MUST NOT send the retained messages. [MQTT-3.3.1-10]." [v5 3.3.1.3] Does a subscription with the same topic filter but different subscription options mean that "the subscription did exist"? This commit interprets "subscription exists" as both topic filter and subscription options must be the same. Therefore, if a client creates a subscription with a topic filter that is identical to a previous subscription and subscription options that are different and Retain Handling 1, the server sends the retained message.
2023-05-09 17:59:10 +08:00
ok = emqtt:disconnect(C2).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
subscription_identifier(Config) ->
C1 = connect(<<"c1">>, Config),
C2 = connect(<<"c2">>, Config),
{ok, _, [0, 0]} = emqtt:subscribe(C1, #{'Subscription-Identifier' => 1}, [{<<"t/1">>, []},
{<<"+/1">>, []}]),
{ok, _, [0]} = emqtt:subscribe(C2, #{'Subscription-Identifier' => 1}, [{<<"t/2">>, []}]),
{ok, _, [0, 0]} = emqtt:subscribe(C2, #{'Subscription-Identifier' => 16#fffffff}, [{<<"+/2">>, []},
{<<"t/3">>, []}]),
{ok, _, [0]} = emqtt:subscribe(C2, #{}, [{<<"t/2/#">>, []}]),
ok = emqtt:publish(C1, <<"t/1">>, <<"m1">>),
ok = emqtt:publish(C1, <<"t/2">>, <<"m2">>),
ok = emqtt:publish(C1, <<"t/3">>, <<"m3">>),
ok = emqtt:publish(C1, <<"t/2/xyz">>, <<"m4">>),
receive {publish,
#{client_pid := C1,
topic := <<"t/1">>,
payload := <<"m1">>,
%% "It is possible that the Client made several subscriptions which match a publication
%% and that it used the same identifier for more than one of them. In this case the
%% PUBLISH packet will carry multiple identical Subscription Identifiers." [v5 3.3.4]
properties := #{'Subscription-Identifier' := [1, 1]}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish,
#{client_pid := C2,
topic := <<"t/2">>,
payload := <<"m2">>,
properties := #{'Subscription-Identifier' := Ids}}} ->
%% "If the Server sends a single copy of the message it MUST include in the PUBLISH
%% packet the Subscription Identifiers for all matching subscriptions which have a
%% Subscription Identifiers, their order is not significant [MQTT-3.3.4-4]." [v5 3.3.4]
?assertEqual([1, 16#fffffff], lists:sort(Ids))
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish,
#{client_pid := C2,
topic := <<"t/3">>,
payload := <<"m3">>,
properties := #{'Subscription-Identifier' := 16#fffffff}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m3")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish,
#{client_pid := C2,
topic := <<"t/2/xyz">>,
payload := <<"m4">>,
properties := Props}} ->
?assertNot(maps:is_key('Subscription-Identifier', Props))
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m4")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
assert_nothing_received(),
ok = emqtt:disconnect(C1),
ok = emqtt:disconnect(C2).
subscription_identifier_amqp091(Config) ->
C1 = connect(<<"sub 1">>, Config),
C2 = connect(<<"sub 2">>, Config),
{ok, _, [1]} = emqtt:subscribe(C1, #{'Subscription-Identifier' => 1}, [{<<"a/+">>, [{qos, 1}]}]),
{ok, _, [1]} = emqtt:subscribe(C2, #{'Subscription-Identifier' => 16#fffffff}, [{<<"a/b">>, [{qos, 1}]}]),
Ch = rabbit_ct_client_helpers:open_channel(Config),
%% Test routing to a single queue.
amqp_channel:call(Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = <<"a.a">>},
#amqp_msg{payload = <<"m1">>}),
receive {publish,
#{client_pid := C1,
topic := <<"a/a">>,
payload := <<"m1">>,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message m1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% Test routing to multiple queues.
amqp_channel:call(Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = <<"a.b">>},
#amqp_msg{payload = <<"m2">>}),
receive {publish,
#{client_pid := C1,
topic := <<"a/b">>,
payload := <<"m2">>,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
receive {publish,
#{client_pid := C2,
topic := <<"a/b">>,
payload := <<"m2">>,
properties := #{'Subscription-Identifier' := 16#fffffff}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ok = emqtt:disconnect(C1),
ok = emqtt:disconnect(C2),
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
subscription_identifier_at_most_once_dead_letter(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 1}, [{<<"dead letter/#">>, [{qos, 1}]}]),
Ch = rabbit_ct_client_helpers:open_channel(Config),
QArgs = [{<<"x-dead-letter-exchange">>, longstr, <<"amq.topic">>},
{<<"x-dead-letter-routing-key">>, longstr, <<"dead letter.a">>}],
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = <<"source queue">>,
durable = true,
exclusive = true,
arguments = QArgs}),
amqp_channel:call(Ch, #'basic.publish'{routing_key = <<"source queue">>},
#amqp_msg{payload = <<"msg">>,
props = #'P_basic'{expiration = <<"0">>}}),
receive {publish,
#{client_pid := C,
topic := <<"dead letter/a">>,
payload := <<"msg">>,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive msg")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ok = emqtt:disconnect(C),
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
at_most_once_dead_letter_detect_cycle(Config) ->
NumExpiredBefore = dead_letter_metric(messages_dead_lettered_expired_total, Config, at_most_once),
SubClientId = Payload = PolicyName = atom_to_binary(?FUNCTION_NAME),
ok = rabbit_ct_broker_helpers:set_policy(
Config, 0, PolicyName, <<"mqtt-subscription-", SubClientId/binary, "qos1">>, <<"queues">>,
%% Create dead letter cycle: qos1 queue -> topic exchange -> qos1 queue
[{<<"dead-letter-exchange">>, <<"amq.topic">>},
{<<"message-ttl">>, 1}]),
Sub1 = connect(SubClientId, Config, non_clean_sess_opts()),
{ok, _, [1]} = emqtt:subscribe(Sub1, #{'Subscription-Identifier' => 10}, [{<<"+/b">>, [{qos, 1}]}]),
ok = emqtt:disconnect(Sub1),
Pub = connect(<<"publisher">>, Config),
{ok, _} = emqtt:publish(Pub, <<"a/b">>, Payload, qos1),
ok = emqtt:disconnect(Pub),
%% Given our subscribing client is disconnected, the message should be dead lettered after 1 ms.
%% However, due to the dead letter cycle, we expect the message to be dropped.
Fix flake in at_most_once_dead_letter_detect_cycle The test case was flaky: ``` *** CT Error Notification 2023-08-15 14:25:51.016 ***🔗 v5_SUITE:at_most_once_dead_letter_detect_cycle failed on line 871 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.227.0>,dup => false, packet_id => 1, payload => <<"at_most_once_dead_letter_detect_cycle">>, properties => #{'Subscription-Identifier' => 10}, qos => 1,retain => false, topic => <<"a/b">>, via => #Port<0.76>}}} ```
2023-08-15 22:43:05 +08:00
timer:sleep(20),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
Sub2 = connect(SubClientId, Config, [{clean_start, false}]),
assert_nothing_received(),
%% Double check that the message was indeed (exactly once) dead lettered.
NumExpired = dead_letter_metric(messages_dead_lettered_expired_total,
Config, at_most_once) - NumExpiredBefore,
?assertEqual(1, NumExpired),
ok = emqtt:disconnect(Sub2),
ok = rabbit_ct_broker_helpers:clear_policy(Config, 0, PolicyName).
%% Tests that the session state in the server includes subscription options
%% and subscription identifiers and that this session state is persisted.
subscription_options_persisted(Config) ->
ClientId = ?FUNCTION_NAME,
C1 = connect(ClientId, Config, non_clean_sess_opts()),
{ok, _, [0, 1]} = emqtt:subscribe(C1, #{'Subscription-Identifier' => 99},
[{<<"t1">>, [{nl, true}, {rap, false}, {qos, 0}]},
{<<"t2">>, [{nl, false}, {rap, true}, {qos, 1}]}]),
unlink(C1),
ok = rabbit_ct_broker_helpers:restart_node(Config, 0),
Mqtt test: Solve dependencies for web mqtt
2025-06-06 15:43:19 +08:00
[util:enable_plugin(Config, Plugin) || Plugin <- ?config(test_plugins, Config)],
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
C2 = connect(ClientId, Config, [{clean_start, false}]),
ok = emqtt:publish(C2, <<"t1">>, <<"m1">>),
ok = emqtt:publish(C2, <<"t2">>, <<"m2">>, [{retain, true}]),
receive {publish,
#{client_pid := C2,
payload := <<"m2">>,
retain := true,
qos := 0,
properties := #{'Subscription-Identifier' := 99}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
assert_nothing_received(),
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(C2, <<"t2">>, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:disconnect(C2).
%% "If a Server receives a SUBSCRIBE packet containing a Topic Filter that is identical to a Non‑shared
%% Subscription’s Topic Filter for the current Session, then it MUST replace that existing Subscription
%% with a new Subscription [MQTT-3.8.4-3]. The Topic Filter in the new Subscription will be identical
%% to that in the previous Subscription, although its Subscription Options could be different."
%%
%% "The Subscription Identifiers are part of the Session State in the Server and are returned to the
%% Client receiving a matching PUBLISH packet. They are removed from the Server’s Session State when the
%% Server receives an UNSUBSCRIBE packet, when the Server receives a SUBSCRIBE packet from the Client for
%% the same Topic Filter but with a different Subscription Identifier or with no Subscription Identifier,
%% or when the Server sends Session Present 0 in a CONNACK packet" [v5 3.8.4]
subscription_options_modify(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [0]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 1}, Topic, [{nl, true}]),
{ok, _} = emqtt:publish(C, Topic, <<"m1">>, qos1),
assert_nothing_received(),
%% modify No Local
{ok, _, [0]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 1}, Topic, [{nl, false}]),
{ok, _} = emqtt:publish(C, Topic, <<"m2">>, qos1),
receive {publish, #{payload := <<"m2">>,
qos := 0 }} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% modify QoS
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 1}, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"m3">>, qos1),
receive {publish, #{payload := <<"m3">>,
qos := 1,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m3")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% modify Subscription Identifier
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 2}, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"m4">>, qos1),
receive {publish, #{payload := <<"m4">>,
properties := #{'Subscription-Identifier' := 2}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m4")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% remove Subscription Identifier
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, <<"m5">>, [{retain, true}, {qos, 1}]),
receive {publish, #{payload := <<"m5">>,
retain := false,
properties := Props}} when map_size(Props) =:= 0 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m5")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% modify Retain As Published
{ok, _, [1]} = emqtt:subscribe(C, Topic, [{rap, true}, {qos, 1}]),
receive {publish, #{payload := <<"m5">>,
retain := true}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive retained m5")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
{ok, _} = emqtt:publish(C, Topic, <<"m6">>, [{retain, true}, {qos, 1}]),
receive {publish, #{payload := <<"m6">>,
retain := true}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m6")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
assert_nothing_received(),
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(C, Topic, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:disconnect(C).
%% "If a Server receives a SUBSCRIBE packet containing a Topic Filter that is identical to a
%% Non‑shared Subscription’s Topic Filter for the current Session, then it MUST replace that
%% existing Subscription with a new Subscription [MQTT-3.8.4-3]. The Topic Filter in the new
%% Subscription will be identical to that in the previous Subscription, although its
%% Subscription Options could be different. [...] Applicaton Messages MUST NOT be lost due
%% to replacing the Subscription [MQTT-3.8.4-4]." [v5 3.8.4]
%%
%% This test ensures that messages are not lost when replacing a QoS 1 subscription.
subscription_options_modify_qos1(Config) ->
subscription_options_modify_qos(1, Config).
%% This test ensures that messages are received at most once
%% when replacing a QoS 0 subscription.
subscription_options_modify_qos0(Config) ->
subscription_options_modify_qos(0, Config).
subscription_options_modify_qos(Qos, Config) ->
Topic = atom_to_binary(?FUNCTION_NAME),
Pub = connect(<<"publisher">>, Config),
Sub = connect(<<"subscriber">>, Config),
{ok, _, [Qos]} = emqtt:subscribe(Sub, Topic, Qos),
Sender = spawn_link(?MODULE, send, [self(), Pub, Topic, 0]),
receive {publish, #{payload := <<"1">>,
properties := Props}} ->
?assertEqual(0, maps:size(Props))
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive 1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% Replace subscription while another client is sending messages.
{ok, _, [Qos]} = emqtt:subscribe(Sub, #{'Subscription-Identifier' => 1}, Topic, Qos),
Sender ! stop,
NumSent = receive {N, Sender} -> N
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("could not stop publisher")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ct:pal("Publisher sent ~b messages", [NumSent]),
LastExpectedPayload = integer_to_binary(NumSent),
receive {publish, #{payload := LastExpectedPayload,
qos := Qos,
client_pid := Sub,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive ~s", [LastExpectedPayload])
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
case Qos of
0 ->
assert_received_no_duplicates();
1 ->
ExpectedPayloads = [integer_to_binary(I) || I <- lists:seq(2, NumSent - 1)],
Support MQTT 5.0 Subscription Option Retain Handling The MQTT v5 spec is a bit vague on Retain Handling 1: "If Retain Handling is set to 1 then if the subscription did not already exist, the Server MUST send all retained message matching the Topic Filter of the subscription to the Client, and if the subscription did exist the Server MUST NOT send the retained messages. [MQTT-3.3.1-10]." [v5 3.3.1.3] Does a subscription with the same topic filter but different subscription options mean that "the subscription did exist"? This commit interprets "subscription exists" as both topic filter and subscription options must be the same. Therefore, if a client creates a subscription with a topic filter that is identical to a previous subscription and subscription options that are different and Retain Handling 1, the server sends the retained message.
2023-05-09 17:59:10 +08:00
ok = expect_publishes(Sub, Topic, ExpectedPayloads)
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(Sub).
%% Tests that no message is lost when upgrading a session
%% with QoS 1 subscription from v3 to v5.
session_upgrade_v3_v5_qos1(Config) ->
session_upgrade_v3_v5_qos(1, Config).
%% Tests that each message is received at most once
%% when upgrading a session with QoS 0 subscription from v3 to v5.
session_upgrade_v3_v5_qos0(Config) ->
session_upgrade_v3_v5_qos(0, Config).
session_upgrade_v3_v5_qos(Qos, Config) ->
ClientId = Topic = atom_to_binary(?FUNCTION_NAME),
Pub = connect(<<"publisher">>, Config),
Fix flake in test case session_upgrade_v3_v5_qos1 CI sometimes failed with the following error: ``` v5_SUITE:session_upgrade_v3_v5_qos failed on line 1068 Reason: {test_case_failed,Received unexpected PUBLISH payload. Expected: <<"2">> Got: <<"3">>} ``` The emqtt client auto acks by default. Therefore, if Subv3 client was able to successfully auto ack message 2 before Subv3 disconnected, Subv5 client did not receive message 2. This commit fixes this flake by making sure that Subv3 does not ack message 2.
2025-03-11 23:58:59 +08:00
Subv3 = connect(ClientId, Config,
[{proto_ver, v3},
{auto_ack, false}] ++
non_clean_sess_opts()),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(Subv3))),
{ok, _, [Qos]} = emqtt:subscribe(Subv3, Topic, Qos),
Sender = spawn_link(?MODULE, send, [self(), Pub, Topic, 0]),
receive {publish, #{payload := <<"1">>,
Fix flake in test case session_upgrade_v3_v5_qos1 CI sometimes failed with the following error: ``` v5_SUITE:session_upgrade_v3_v5_qos failed on line 1068 Reason: {test_case_failed,Received unexpected PUBLISH payload. Expected: <<"2">> Got: <<"3">>} ``` The emqtt client auto acks by default. Therefore, if Subv3 client was able to successfully auto ack message 2 before Subv3 disconnected, Subv5 client did not receive message 2. This commit fixes this flake by making sure that Subv3 does not ack message 2.
2025-03-11 23:58:59 +08:00
client_pid := Subv3,
packet_id := PacketId}} ->
case Qos of
0 -> ok;
1 -> emqtt:puback(Subv3, PacketId)
end
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive 1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% Upgrade session from v3 to v5 while another client is sending messages.
ok = emqtt:disconnect(Subv3),
Fix flake in test case session_upgrade_v3_v5_qos1 CI sometimes failed with the following error: ``` v5_SUITE:session_upgrade_v3_v5_qos failed on line 1068 Reason: {test_case_failed,Received unexpected PUBLISH payload. Expected: <<"2">> Got: <<"3">>} ``` The emqtt client auto acks by default. Therefore, if Subv3 client was able to successfully auto ack message 2 before Subv3 disconnected, Subv5 client did not receive message 2. This commit fixes this flake by making sure that Subv3 does not ack message 2.
2025-03-11 23:58:59 +08:00
Subv5 = connect(ClientId, Config, [{proto_ver, v5},
{clean_start, false},
{auto_ack, true}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(Subv5))),
Sender ! stop,
NumSent = receive {N, Sender} -> N
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("could not stop publisher")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ct:pal("Publisher sent ~b messages", [NumSent]),
LastExpectedPayload = integer_to_binary(NumSent),
receive {publish, #{payload := LastExpectedPayload,
qos := Qos,
client_pid := Subv5}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive ~s", [LastExpectedPayload])
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
case Qos of
0 ->
assert_received_no_duplicates();
1 ->
ExpectedPayloads = [integer_to_binary(I) || I <- lists:seq(2, NumSent - 1)],
Support MQTT 5.0 Subscription Option Retain Handling The MQTT v5 spec is a bit vague on Retain Handling 1: "If Retain Handling is set to 1 then if the subscription did not already exist, the Server MUST send all retained message matching the Topic Filter of the subscription to the Client, and if the subscription did exist the Server MUST NOT send the retained messages. [MQTT-3.3.1-10]." [v5 3.3.1.3] Does a subscription with the same topic filter but different subscription options mean that "the subscription did exist"? This commit interprets "subscription exists" as both topic filter and subscription options must be the same. Therefore, if a client creates a subscription with a topic filter that is identical to a previous subscription and subscription options that are different and Retain Handling 1, the server sends the retained message.
2023-05-09 17:59:10 +08:00
ok = expect_publishes(Subv5, Topic, ExpectedPayloads)
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
ok = emqtt:disconnect(Pub),
ok = emqtt:disconnect(Subv5).
send(Parent, Client, Topic, NumSent) ->
receive stop ->
Parent ! {NumSent, self()}
after 0 ->
N = NumSent + 1,
{ok, _} = emqtt:publish(Client, Topic, integer_to_binary(N), qos1),
send(Parent, Client, Topic, N)
end.
assert_received_no_duplicates() ->
tests: v5_SUITE wait longer to receive messages
2024-11-18 22:24:51 +08:00
assert_received_no_duplicates0(#{}, 30000).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
tests: v5_SUITE wait longer to receive messages
2024-11-18 22:24:51 +08:00
assert_received_no_duplicates0(Received, Timeout) ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
receive {publish, #{payload := P}} ->
case maps:is_key(P, Received) of
true -> ct:fail("Received ~p twice", [P]);
tests: v5_SUITE wait longer to receive messages
2024-11-18 22:24:51 +08:00
false -> assert_received_no_duplicates0(maps:put(P, ok, Received), 500)
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end
tests: v5_SUITE wait longer to receive messages
2024-11-18 22:24:51 +08:00
after Timeout ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
%% Check that we received at least one message.
?assertNotEqual(0, maps:size(Received))
end.
Fix MQTT crash This commit fixes the following crash: ``` 2388-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> exception error: bad argument 2389-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in function lists:keymember/3 2390-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> called as lists:keymember(<<"x-mqtt-publish-qos">>,1,undefined) 2391-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> *** argument 3: not a list 2392-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in call from rabbit_mqtt_processor:amqp_props_to_mqtt_props/2 (rabbit_mqtt_processor.erl, line 2219) ``` This crash occurs when a message without AMQP 0.9.1 #P_basic.headers is sent to the MQTT v4 connection process, but consumed by an MQTT v5 connection process. This is the case when an AMQP 0.9.1 client sends a message to a v4 MQTT client, and this same client subsequently upgrades to v5 consuming the message. When sending from AMQP 0.9.1 client directly to a v5 MQTT connection, the AMQP 0.9.1 header <<"x-binding-keys">> is set. When sending from AMQP 0.9.1 client directly to a v4 MQTT connection, no header is set, but this code branch was not evaluated.
2023-06-21 18:11:42 +08:00
session_upgrade_v3_v5_amqp091_pub(Config) ->
Payload = ClientId = Topic = atom_to_binary(?FUNCTION_NAME),
Subv3 = connect(ClientId, Config, [{proto_ver, v3} | non_clean_sess_opts()]),
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(Subv3))),
{ok, _, [1]} = emqtt:subscribe(Subv3, Topic, 1),
ok = emqtt:disconnect(Subv3),
Ch = rabbit_ct_client_helpers:open_channel(Config),
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = Topic},
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
#amqp_msg{payload = Payload,
props = #'P_basic'{delivery_mode = 2}}),
Fix MQTT crash This commit fixes the following crash: ``` 2388-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> exception error: bad argument 2389-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in function lists:keymember/3 2390-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> called as lists:keymember(<<"x-mqtt-publish-qos">>,1,undefined) 2391-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> *** argument 3: not a list 2392-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in call from rabbit_mqtt_processor:amqp_props_to_mqtt_props/2 (rabbit_mqtt_processor.erl, line 2219) ``` This crash occurs when a message without AMQP 0.9.1 #P_basic.headers is sent to the MQTT v4 connection process, but consumed by an MQTT v5 connection process. This is the case when an AMQP 0.9.1 client sends a message to a v4 MQTT client, and this same client subsequently upgrades to v5 consuming the message. When sending from AMQP 0.9.1 client directly to a v5 MQTT connection, the AMQP 0.9.1 header <<"x-binding-keys">> is set. When sending from AMQP 0.9.1 client directly to a v4 MQTT connection, no header is set, but this code branch was not evaluated.
2023-06-21 18:11:42 +08:00
Subv5 = connect(ClientId, Config, [{proto_ver, v5}, {clean_start, false}]),
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(Subv5))),
receive {publish, #{payload := Payload,
qos := 1,
client_pid := Subv5}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message")
Fix MQTT crash This commit fixes the following crash: ``` 2388-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> exception error: bad argument 2389-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in function lists:keymember/3 2390-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> called as lists:keymember(<<"x-mqtt-publish-qos">>,1,undefined) 2391-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> *** argument 3: not a list 2392-2023-06-21 08:53:28.189519+00:00 [error] <0.2191.0> in call from rabbit_mqtt_processor:amqp_props_to_mqtt_props/2 (rabbit_mqtt_processor.erl, line 2219) ``` This crash occurs when a message without AMQP 0.9.1 #P_basic.headers is sent to the MQTT v4 connection process, but consumed by an MQTT v5 connection process. This is the case when an AMQP 0.9.1 client sends a message to a v4 MQTT client, and this same client subsequently upgrades to v5 consuming the message. When sending from AMQP 0.9.1 client directly to a v5 MQTT connection, the AMQP 0.9.1 header <<"x-binding-keys">> is set. When sending from AMQP 0.9.1 client directly to a v4 MQTT connection, no header is set, but this code branch was not evaluated.
2023-06-21 18:11:42 +08:00
end,
ok = emqtt:disconnect(Subv5),
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
compatibility_v3_v5(Config) ->
Cv3 = connect(<<"client v3">>, Config, [{proto_ver, v3}]),
Cv5 = connect(<<"client v5">>, Config, [{proto_ver, v5}]),
%% Sanity check that versions were set correctly.
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(Cv3))),
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(Cv5))),
{ok, _, [1]} = emqtt:subscribe(Cv3, <<"v3/#">>, qos1),
{ok, _, [1]} = emqtt:subscribe(Cv5, #{'Subscription-Identifier' => 99},
[{<<"v5/#">>, [{rap, true}, {qos, 1}]}]),
%% Send message in either direction.
{ok, _} = emqtt:publish(Cv5, <<"v3">>, <<"from v5">>, qos1),
{ok, _} = emqtt:publish(Cv3, <<"v5">>, <<"from v3">>, [{retain, true}, {qos, 1}]),
ok = expect_publishes(Cv3, <<"v3">>, [<<"from v5">>]),
receive {publish,
#{client_pid := Cv5,
topic := <<"v5">>,
payload := <<"from v3">>,
%% v5 features should work even when message comes from a v3 client.
retain := true,
properties := #{'Subscription-Identifier' := 99}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive from v3")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(Cv3, <<"v5">>, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:disconnect(Cv3),
ok = emqtt:disconnect(Cv5).
session_upgrade_v3_v5_unsubscribe(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C1 = connect(ClientId, Config, [{proto_ver, v3} | non_clean_sess_opts()]),
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(C1))),
{ok, _, [0]} = emqtt:subscribe(C1, Topic),
ok = emqtt:disconnect(C1),
%% Upgrade the session from v3 to v5.
C2 = connect(ClientId, Config, [{proto_ver, v5}, {clean_start, false}]),
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(C2))),
ok = emqtt:publish(C2, Topic, <<"m1">>),
ok = expect_publishes(C2, Topic, [<<"m1">>]),
%% Unsubscribing in v5 should work.
?assertMatch({ok, _, [?RC_SUCCESS]}, emqtt:unsubscribe(C2, Topic)),
ok = emqtt:publish(C2, Topic, <<"m2">>),
assert_nothing_received(),
ok = emqtt:disconnect(C2).
session_upgrade_v4_v5_no_queue_bind_permission(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C1 = connect(ClientId, Config, [{proto_ver, v4} | non_clean_sess_opts()]),
?assertEqual(4, proplists:get_value(proto_ver, emqtt:info(C1))),
{ok, _, [0]} = emqtt:subscribe(C1, Topic),
ok = emqtt:disconnect(C1),
%% Revoking write access to queue will cause queue.bind to fail.
rabbit_ct_broker_helpers:set_permissions(Config, <<"guest">>, <<"/">>, <<".*">>, <<"">>, <<".*">>),
%% Upgrading the session from v4 to v5 should fail because it causes
%% queue.bind and queue.unbind (to change the binding arguments).
{C2, Connect} = start_client(ClientId, Config, 0, [{proto_ver, v5}, {clean_start, false}]),
unlink(C2),
?assertEqual({error, {not_authorized, #{}}}, Connect(C2)),
%% Cleanup
rabbit_ct_broker_helpers:set_full_permissions(Config, <<"guest">>, <<"/">>),
C3 = connect(ClientId, Config),
ok = emqtt:disconnect(C3).
amqp091_cc_header(Config) ->
C = connect(?FUNCTION_NAME, Config),
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 1}, [{<<"#">>, [{qos, 1}]}]),
Ch = rabbit_ct_client_helpers:open_channel(Config),
amqp_channel:call(
Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = <<"first.key">>},
#amqp_msg{payload = <<"msg">>,
props = #'P_basic'{
headers = [{<<"CC">>, array,
[{longstr, <<"second.key">>}]}]}}),
%% Even though both routing key and CC header match the topic filter,
%% we expect to receive a single message (because only one message is sent)
%% and a single subscription identifier (because we created only one subscription).
receive {publish,
#{topic := <<"first/key">>,
payload := <<"msg">>,
properties := #{'Subscription-Identifier' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive msg")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
assert_nothing_received(),
ok = emqtt:disconnect(C).
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
publish_property_content_type(Config) ->
Topic = ClientId = Payload = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
%% "The Content Type MUST be a UTF-8 Encoded String" [v5 3.3.2.3.9]
{ok, _} = emqtt:publish(C, Topic, #{'Content-Type' => <<"text/plain😎;charset=UTF-8"/utf8>>}, Payload, [{qos, 1}]),
receive {publish, #{payload := Payload,
properties := #{'Content-Type' := <<"text/plain😎;charset=UTF-8"/utf8>>}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
ok = emqtt:disconnect(C).
publish_property_payload_format_indicator(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
{ok, _} = emqtt:publish(C, Topic, #{'Payload-Format-Indicator' => 0}, <<"m1">>, [{qos, 1}]),
{ok, _} = emqtt:publish(C, Topic, #{'Payload-Format-Indicator' => 1}, <<"m2">>, [{qos, 1}]),
receive {publish, #{payload := <<"m1">>,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
properties := #{'Payload-Format-Indicator' := 0}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
receive {publish, #{payload := <<"m2">>,
properties := #{'Payload-Format-Indicator' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
ok = emqtt:disconnect(C).
publish_property_response_topic_correlation_data(Config) ->
%% "The Response Topic MUST be a UTF-8 Encoded String" [v5 3.3.2.3.5]
Requester = connect(<<"requester">>, Config),
FrenchResponder = connect(<<"French responder">>, Config),
ItalianResponder = connect(<<"Italian responder">>, Config),
ResponseTopic = <<"🗣️/response/for/English/request"/utf8>>,
{ok, _, [0]} = emqtt:subscribe(Requester, ResponseTopic),
{ok, _, [0]} = emqtt:subscribe(FrenchResponder, <<"greet/French">>),
{ok, _, [0]} = emqtt:subscribe(ItalianResponder, <<"greet/Italian">>),
CorrelationFrench = <<"French">>,
%% "the length of Binary Data is limited to the range of 0 to 65,535 Bytes" [v5 1.5.6]
%% Let's also test with large correlation data.
CorrelationItalian = <<"Italian", (binary:copy(<<"x">>, 65_500))/binary>>,
ok = emqtt:publish(Requester, <<"greet/French">>,
#{'Response-Topic' => ResponseTopic,
'Correlation-Data' => CorrelationFrench},
<<"Harry">>, [{qos, 0}]),
ok = emqtt:publish(Requester, <<"greet/Italian">>,
#{'Response-Topic' => ResponseTopic,
'Correlation-Data' => CorrelationItalian},
<<"Harry">>, [{qos, 0}]),
receive {publish, #{client_pid := FrenchResponder,
payload := <<"Harry">>,
properties := #{'Response-Topic' := ResponseTopic,
'Correlation-Data' := Corr0}}} ->
ok = emqtt:publish(FrenchResponder, ResponseTopic,
#{'Correlation-Data' => Corr0},
<<"Bonjour Henri">>, [{qos, 0}])
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("French responder did not receive request")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
receive {publish, #{client_pid := ItalianResponder,
payload := <<"Harry">>,
properties := #{'Response-Topic' := ResponseTopic,
'Correlation-Data' := Corr1}}} ->
ok = emqtt:publish(ItalianResponder, ResponseTopic,
#{'Correlation-Data' => Corr1},
<<"Buongiorno Enrico">>, [{qos, 0}])
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Italian responder did not receive request")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
receive {publish, #{client_pid := Requester,
properties := #{'Correlation-Data' := CorrelationItalian},
payload := Payload0
}} ->
?assertEqual(<<"Buongiorno Enrico">>, Payload0)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive Italian response")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
receive {publish, #{client_pid := Requester,
properties := #{'Correlation-Data' := CorrelationFrench},
payload := Payload1
}} ->
?assertEqual(<<"Bonjour Henri">>, Payload1)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive French response")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
[ok = emqtt:disconnect(C) || C <- [Requester, FrenchResponder, ItalianResponder]].
publish_property_user_property(Config) ->
Payload = Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config),
{ok, _, [1]} = emqtt:subscribe(C, Topic, qos1),
%% Same keys and values are allowed. Order must be maintained.
UserProperty = [{<<"k1">>, <<"v2">>},
{<<"k1">>, <<"v2">>},
{<<"k1">>, <<"v1">>},
{<<"k0">>, <<"v0">>},
%% "UTF-8 encoded strings can have any length in the range 0 to 65,535 bytes"
%% [v5 1.5.4]
{<<>>, <<>>},
{<<(binary:copy(<<"k">>, 65_000))/binary, "🐇"/utf8>>,
<<(binary:copy(<<"v">>, 65_000))/binary, "🐇"/utf8>>}],
{ok, _} = emqtt:publish(C, Topic, #{'User-Property' => UserProperty}, Payload, [{qos, 1}]),
receive {publish, #{payload := Payload,
properties := #{'User-Property' := UserProperty}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive message")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
ok = emqtt:disconnect(C).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
disconnect_with_will(Config) ->
Topic = Payload = ClientId = atom_to_binary(?FUNCTION_NAME),
Sub = connect(<<"subscriber">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
C = connect(ClientId, Config, [{will_topic, Topic},
{will_payload, Payload}]),
ok = emqtt:disconnect(C, ?RC_DISCONNECT_WITH_WILL),
ok = expect_publishes(Sub, Topic, [Payload]),
ok = emqtt:disconnect(Sub).
will_qos2(Config) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
Opts = [{will_topic, Topic},
{will_payload, <<"msg">>},
{will_qos, 2}],
{C, Connect} = start_client(ClientId, Config, 0, Opts),
unlink(C),
?assertEqual({error, {qos_not_supported, #{}}}, Connect(C)).
will_delay_less_than_session_expiry(Config) ->
Provide more debug output for flaky test In https://github.com/rabbitmq/rabbitmq-server/actions/runs/6735035912/job/18308551274?pr=9865 test ``` bazel test //deps/rabbitmq_mqtt:v5_SUITE -t- --test_sharding_strategy=disabled \ --test_env FOCUS="-group [mqtt,cluster_size_1] -case will_delay_equals_session_expiry" \ --test_env RABBITMQ_METADATA_STORE=mnesia --config=rbe-26 --runs_per_test=100 ``` seems to flake.
2023-11-03 01:40:38 +08:00
will_delay(1, 5, ?FUNCTION_NAME, Config).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
will_delay_equals_session_expiry(Config) ->
Provide more debug output for flaky test In https://github.com/rabbitmq/rabbitmq-server/actions/runs/6735035912/job/18308551274?pr=9865 test ``` bazel test //deps/rabbitmq_mqtt:v5_SUITE -t- --test_sharding_strategy=disabled \ --test_env FOCUS="-group [mqtt,cluster_size_1] -case will_delay_equals_session_expiry" \ --test_env RABBITMQ_METADATA_STORE=mnesia --config=rbe-26 --runs_per_test=100 ``` seems to flake.
2023-11-03 01:40:38 +08:00
will_delay(1, 1, ?FUNCTION_NAME, Config).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
will_delay_greater_than_session_expiry(Config) ->
Provide more debug output for flaky test In https://github.com/rabbitmq/rabbitmq-server/actions/runs/6735035912/job/18308551274?pr=9865 test ``` bazel test //deps/rabbitmq_mqtt:v5_SUITE -t- --test_sharding_strategy=disabled \ --test_env FOCUS="-group [mqtt,cluster_size_1] -case will_delay_equals_session_expiry" \ --test_env RABBITMQ_METADATA_STORE=mnesia --config=rbe-26 --runs_per_test=100 ``` seems to flake.
2023-11-03 01:40:38 +08:00
will_delay(5, 1, ?FUNCTION_NAME, Config).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% "The Server delays publishing the Client’s Will Message until the Will Delay
%% Interval has passed or the Session ends, whichever happens first." [v5 3.1.3.2.2]
Provide more debug output for flaky test In https://github.com/rabbitmq/rabbitmq-server/actions/runs/6735035912/job/18308551274?pr=9865 test ``` bazel test //deps/rabbitmq_mqtt:v5_SUITE -t- --test_sharding_strategy=disabled \ --test_env FOCUS="-group [mqtt,cluster_size_1] -case will_delay_equals_session_expiry" \ --test_env RABBITMQ_METADATA_STORE=mnesia --config=rbe-26 --runs_per_test=100 ``` seems to flake.
2023-11-03 01:40:38 +08:00
will_delay(WillDelay, SessionExpiry, ClientId, Config)
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
when WillDelay =:= 1 orelse
SessionExpiry =:= 1->
Topic = <<"a/b">>,
Msg = <<"msg">>,
Opts = [{properties, #{'Session-Expiry-Interval' => SessionExpiry}},
{will_props, #{'Will-Delay-Interval' => WillDelay}},
{will_topic, Topic},
{will_payload, Msg}],
C1 = connect(ClientId, Config, Opts),
Sub = connect(<<"subscriber">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
unlink(C1),
erlang:exit(C1, trigger_will_message),
receive TooEarly -> ct:fail(TooEarly)
after 800 -> ok
end,
Provide more debug output for flaky test In https://github.com/rabbitmq/rabbitmq-server/actions/runs/6735035912/job/18308551274?pr=9865 test ``` bazel test //deps/rabbitmq_mqtt:v5_SUITE -t- --test_sharding_strategy=disabled \ --test_env FOCUS="-group [mqtt,cluster_size_1] -case will_delay_equals_session_expiry" \ --test_env RABBITMQ_METADATA_STORE=mnesia --config=rbe-26 --runs_per_test=100 ``` seems to flake.
2023-11-03 01:40:38 +08:00
receive {publish, #{payload := Msg}} -> ok;
Unexpected -> ct:fail({unexpected_message, Unexpected})
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail(will_message_timeout)
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
%% Cleanup
C2 = connect(ClientId, Config),
ok = emqtt:disconnect(C2),
ok = emqtt:disconnect(Sub).
will_delay_session_expiry_zero(Config) ->
Topic = <<"a/b">>,
Msg = <<"msg">>,
Opts = [{will_props, #{'Will-Delay-Interval' => 1}},
{will_topic, Topic},
{will_payload, Msg}],
C = connect(?FUNCTION_NAME, Config, Opts),
Sub = connect(<<"subscriber">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
unlink(C),
erlang:exit(C, trigger_will_message),
%% Since default Session Expiry Interval is 0, we expect Will Message immediately.
receive {publish, #{payload := Msg}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail(will_message_timeout)
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
ok = emqtt:disconnect(Sub).
will_delay_reconnect_no_will(Config) ->
Topic = <<"my/topic">>,
ClientId = Payload = atom_to_binary(?FUNCTION_NAME),
Sub = connect(<<"sub">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
Opts = [{properties, #{'Session-Expiry-Interval' => 16#FFFFFFFF}},
{will_props, #{'Will-Delay-Interval' => 1}},
{will_topic, Topic},
{will_payload, Payload}],
C1 = connect(ClientId, Config, Opts),
unlink(C1),
erlang:exit(C1, trigger_will_message),
%% Should not receive anything because Will Delay is 1 second.
assert_nothing_received(200),
%% Reconnect with same ClientId, this time without a Will Message.
C2 = connect(ClientId, Config, [{clean_start, false}]),
%% Should not receive anything because client reconnected within Will Delay Interval.
assert_nothing_received(1100),
ok = emqtt:disconnect(C2, ?RC_DISCONNECT_WITH_WILL),
%% Should not receive anything because new client did not set Will Message.
assert_nothing_received(1100),
ok = emqtt:disconnect(Sub).
will_delay_reconnect_with_will(Config) ->
Topic = <<"my/topic">>,
ClientId = atom_to_binary(?FUNCTION_NAME),
Sub = connect(<<"sub">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
C1 = connect(ClientId, Config,
[{properties, #{'Session-Expiry-Interval' => 16#FFFFFFFF}},
{will_props, #{'Will-Delay-Interval' => 1}},
{will_topic, Topic},
{will_payload, <<"will-1">>}]),
unlink(C1),
erlang:exit(C1, trigger_will_message),
%% Should not receive anything because Will Delay is 1 second.
assert_nothing_received(300),
%% Reconnect with same ClientId, again with a delayed will message.
C2 = connect(ClientId, Config,
[{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 16#FFFFFFFF}},
{will_props, #{'Will-Delay-Interval' => 1}},
{will_topic, Topic},
{will_payload, <<"will-2">>}]),
ok = emqtt:disconnect(C2, ?RC_DISCONNECT_WITH_WILL),
%% The second will message should be sent after 1 second.
assert_nothing_received(700),
ok = expect_publishes(Sub, Topic, [<<"will-2">>]),
%% The first will message should not be sent.
assert_nothing_received(),
%% Cleanup
C3 = connect(ClientId, Config),
ok = emqtt:disconnect(C3),
ok = emqtt:disconnect(Sub).
%% "If a Network Connection uses a Client Identifier of an existing Network Connection to the Server,
%% the Will Message for the exiting connection is sent unless the new connection specifies Clean
%% Start of 0 and the Will Delay is greater than zero." [v5 3.1.3.2.2]
will_delay_session_takeover(Config) ->
Topic = <<"my/topic">>,
Sub = connect(<<"sub">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
C1a = connect(<<"c1">>, Config,
[{properties, #{'Session-Expiry-Interval' => 120}},
{will_props, #{'Will-Delay-Interval' => 30}},
{will_topic, Topic},
{will_payload, <<"will-1a">>}]),
C2a = connect(<<"c2">>, Config,
[{will_topic, Topic},
{will_payload, <<"will-2a">>}]),
C3a = connect(<<"c3">>, Config,
[{will_topic, Topic},
{will_payload, <<"will-3a">>}]),
C4a = connect(<<"c4">>, Config,
[{will_topic, Topic},
{will_payload, <<"will-4a">>}]),
Clients = [C1a, C2a, C3a, C4a],
[true = unlink(C) || C <- Clients],
C1b = connect(<<"c1">>, Config,
[{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 120}},
{will_props, #{'Will-Delay-Interval' => 30}},
{will_topic, Topic},
{will_payload, <<"will-1b">>}]),
C2b = connect(<<"c2">>, Config,
[{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 120}},
{will_props, #{'Will-Delay-Interval' => 30}},
{will_topic, Topic},
{will_payload, <<"will-2b">>}]),
C3b = connect(<<"c3">>, Config,
[{clean_start, true},
{properties, #{'Session-Expiry-Interval' => 120}},
{will_props, #{'Will-Delay-Interval' => 30}},
{will_topic, Topic},
{will_payload, <<"will-3b">>}]),
C4b = connect(<<"c4">>, Config,
[{clean_start, false},
{properties, #{'Session-Expiry-Interval' => 0}},
{will_topic, Topic},
{will_payload, <<"will-4b">>}]),
[receive {disconnected, ?RC_SESSION_TAKEN_OVER, #{}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("server did not disconnect us")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end || _ <- Clients],
Fix flake will_delay_session_takeover Prior to this commit, the following flake occurred in CI for ``` make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_1:will_delay_session_takeover ``` ``` === Location: [{v5_SUITE,will_delay_session_takeover,1473}, {test_server,ts_tc,1793}, {test_server,run_test_case_eval1,1302}, {test_server,run_test_case_eval,1234}] === === Reason: {test_case_failed,"Received unexpected PUBLISH payload. Expected: <<\"will-3a\">> Got: <<\"will-4a\">>"} ``` The RabbitMQ logs for this single node test show: ``` 2024-11-04 14:43:35.039196+00:00 [debug] <0.1334.0> MQTT accepting TCP connection <0.1334.0> (127.0.0.1:42576 -> 127.0.0.1:27005) 2024-11-04 14:43:35.039336+00:00 [debug] <0.1334.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.039438+00:00 [debug] <0.1334.0> MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.039537+00:00 [debug] <0.1334.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.039729+00:00 [info] <0.1334.0> Accepted MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.040297+00:00 [debug] <0.1337.0> MQTT accepting TCP connection <0.1337.0> (127.0.0.1:42580 -> 127.0.0.1:27005) 2024-11-04 14:43:35.040442+00:00 [debug] <0.1337.0> Received a CONNECT, client ID: c4, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.040534+00:00 [debug] <0.1337.0> MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.040597+00:00 [debug] <0.1337.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.040793+00:00 [info] <0.1337.0> Accepted MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.041463+00:00 [debug] <0.1340.0> MQTT accepting TCP connection <0.1340.0> (127.0.0.1:42596 -> 127.0.0.1:27005) 2024-11-04 14:43:35.041715+00:00 [debug] <0.1340.0> Received a CONNECT, client ID: c1, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.041806+00:00 [debug] <0.1340.0> MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.041881+00:00 [debug] <0.1340.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.041982+00:00 [warning] <0.1328.0> MQTT disconnecting client <<"127.0.0.1:42560 -> 127.0.0.1:27005">> with duplicate id 'c1' 2024-11-04 14:43:35.042062+00:00 [info] <0.1340.0> Accepted MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 for client ID c1 2024-11-04 14:43:35.045624+00:00 [debug] <0.1345.0> MQTT accepting TCP connection <0.1345.0> (127.0.0.1:42602 -> 127.0.0.1:27005) 2024-11-04 14:43:35.045781+00:00 [debug] <0.1345.0> Received a CONNECT, client ID: c2, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.045874+00:00 [debug] <0.1345.0> MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.045943+00:00 [debug] <0.1345.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.046032+00:00 [warning] <0.1331.0> MQTT disconnecting client <<"127.0.0.1:42566 -> 127.0.0.1:27005">> with duplicate id 'c2' 2024-11-04 14:43:35.046281+00:00 [info] <0.1345.0> Accepted MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 for client ID c2 2024-11-04 14:43:35.047063+00:00 [debug] <0.1350.0> MQTT accepting TCP connection <0.1350.0> (127.0.0.1:42614 -> 127.0.0.1:27005) 2024-11-04 14:43:35.047702+00:00 [debug] <0.1350.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.047910+00:00 [debug] <0.1350.0> MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.048467+00:00 [debug] <0.1350.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.049701+00:00 [info] <0.1350.0> Accepted MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.050907+00:00 [warning] <0.1334.0> MQTT disconnecting client <<"127.0.0.1:42576 -> 127.0.0.1:27005">> with duplicate id 'c3' 2024-11-04 14:43:35.051248+00:00 [debug] <0.1353.0> MQTT accepting TCP connection <0.1353.0> (127.0.0.1:42626 -> 127.0.0.1:27005) 2024-11-04 14:43:35.051395+00:00 [debug] <0.1353.0> Received a CONNECT, client ID: c4, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.051519+00:00 [debug] <0.1353.0> MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.051590+00:00 [debug] <0.1353.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.051871+00:00 [info] <0.1353.0> Accepted MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.051960+00:00 [warning] <0.1337.0> MQTT disconnecting client <<"127.0.0.1:42580 -> 127.0.0.1:27005">> with duplicate id 'c4' 2024-11-04 14:43:35.052689+00:00 [debug] <0.1337.0> sent Will Message to topic my/topic for MQTT client ID c4 2024-11-04 14:43:35.054119+00:00 [debug] <0.1334.0> sent Will Message to topic my/topic for MQTT client ID c3 ``` We see nicely how RabbitMQ sends the will message for both c3 and c4. However, the order in which RabbitMQ sends is not guaranteed. Hence, we adapt the test expectation to not depend on the order of Will messages being received.
2024-11-08 23:12:52 +08:00
receive {publish, #{client_pid := Sub,
payload := <<"will-3a">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail({missing_msg, ?LINE})
Fix flake will_delay_session_takeover Prior to this commit, the following flake occurred in CI for ``` make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_1:will_delay_session_takeover ``` ``` === Location: [{v5_SUITE,will_delay_session_takeover,1473}, {test_server,ts_tc,1793}, {test_server,run_test_case_eval1,1302}, {test_server,run_test_case_eval,1234}] === === Reason: {test_case_failed,"Received unexpected PUBLISH payload. Expected: <<\"will-3a\">> Got: <<\"will-4a\">>"} ``` The RabbitMQ logs for this single node test show: ``` 2024-11-04 14:43:35.039196+00:00 [debug] <0.1334.0> MQTT accepting TCP connection <0.1334.0> (127.0.0.1:42576 -> 127.0.0.1:27005) 2024-11-04 14:43:35.039336+00:00 [debug] <0.1334.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.039438+00:00 [debug] <0.1334.0> MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.039537+00:00 [debug] <0.1334.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.039729+00:00 [info] <0.1334.0> Accepted MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.040297+00:00 [debug] <0.1337.0> MQTT accepting TCP connection <0.1337.0> (127.0.0.1:42580 -> 127.0.0.1:27005) 2024-11-04 14:43:35.040442+00:00 [debug] <0.1337.0> Received a CONNECT, client ID: c4, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.040534+00:00 [debug] <0.1337.0> MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.040597+00:00 [debug] <0.1337.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.040793+00:00 [info] <0.1337.0> Accepted MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.041463+00:00 [debug] <0.1340.0> MQTT accepting TCP connection <0.1340.0> (127.0.0.1:42596 -> 127.0.0.1:27005) 2024-11-04 14:43:35.041715+00:00 [debug] <0.1340.0> Received a CONNECT, client ID: c1, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.041806+00:00 [debug] <0.1340.0> MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.041881+00:00 [debug] <0.1340.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.041982+00:00 [warning] <0.1328.0> MQTT disconnecting client <<"127.0.0.1:42560 -> 127.0.0.1:27005">> with duplicate id 'c1' 2024-11-04 14:43:35.042062+00:00 [info] <0.1340.0> Accepted MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 for client ID c1 2024-11-04 14:43:35.045624+00:00 [debug] <0.1345.0> MQTT accepting TCP connection <0.1345.0> (127.0.0.1:42602 -> 127.0.0.1:27005) 2024-11-04 14:43:35.045781+00:00 [debug] <0.1345.0> Received a CONNECT, client ID: c2, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.045874+00:00 [debug] <0.1345.0> MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.045943+00:00 [debug] <0.1345.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.046032+00:00 [warning] <0.1331.0> MQTT disconnecting client <<"127.0.0.1:42566 -> 127.0.0.1:27005">> with duplicate id 'c2' 2024-11-04 14:43:35.046281+00:00 [info] <0.1345.0> Accepted MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 for client ID c2 2024-11-04 14:43:35.047063+00:00 [debug] <0.1350.0> MQTT accepting TCP connection <0.1350.0> (127.0.0.1:42614 -> 127.0.0.1:27005) 2024-11-04 14:43:35.047702+00:00 [debug] <0.1350.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.047910+00:00 [debug] <0.1350.0> MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.048467+00:00 [debug] <0.1350.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.049701+00:00 [info] <0.1350.0> Accepted MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.050907+00:00 [warning] <0.1334.0> MQTT disconnecting client <<"127.0.0.1:42576 -> 127.0.0.1:27005">> with duplicate id 'c3' 2024-11-04 14:43:35.051248+00:00 [debug] <0.1353.0> MQTT accepting TCP connection <0.1353.0> (127.0.0.1:42626 -> 127.0.0.1:27005) 2024-11-04 14:43:35.051395+00:00 [debug] <0.1353.0> Received a CONNECT, client ID: c4, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.051519+00:00 [debug] <0.1353.0> MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.051590+00:00 [debug] <0.1353.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.051871+00:00 [info] <0.1353.0> Accepted MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.051960+00:00 [warning] <0.1337.0> MQTT disconnecting client <<"127.0.0.1:42580 -> 127.0.0.1:27005">> with duplicate id 'c4' 2024-11-04 14:43:35.052689+00:00 [debug] <0.1337.0> sent Will Message to topic my/topic for MQTT client ID c4 2024-11-04 14:43:35.054119+00:00 [debug] <0.1334.0> sent Will Message to topic my/topic for MQTT client ID c3 ``` We see nicely how RabbitMQ sends the will message for both c3 and c4. However, the order in which RabbitMQ sends is not guaranteed. Hence, we adapt the test expectation to not depend on the order of Will messages being received.
2024-11-08 23:12:52 +08:00
end,
receive {publish, #{client_pid := Sub,
payload := <<"will-4a">>}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail({missing_msg, ?LINE})
Fix flake will_delay_session_takeover Prior to this commit, the following flake occurred in CI for ``` make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_1:will_delay_session_takeover ``` ``` === Location: [{v5_SUITE,will_delay_session_takeover,1473}, {test_server,ts_tc,1793}, {test_server,run_test_case_eval1,1302}, {test_server,run_test_case_eval,1234}] === === Reason: {test_case_failed,"Received unexpected PUBLISH payload. Expected: <<\"will-3a\">> Got: <<\"will-4a\">>"} ``` The RabbitMQ logs for this single node test show: ``` 2024-11-04 14:43:35.039196+00:00 [debug] <0.1334.0> MQTT accepting TCP connection <0.1334.0> (127.0.0.1:42576 -> 127.0.0.1:27005) 2024-11-04 14:43:35.039336+00:00 [debug] <0.1334.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.039438+00:00 [debug] <0.1334.0> MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.039537+00:00 [debug] <0.1334.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.039729+00:00 [info] <0.1334.0> Accepted MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.040297+00:00 [debug] <0.1337.0> MQTT accepting TCP connection <0.1337.0> (127.0.0.1:42580 -> 127.0.0.1:27005) 2024-11-04 14:43:35.040442+00:00 [debug] <0.1337.0> Received a CONNECT, client ID: c4, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: [] 2024-11-04 14:43:35.040534+00:00 [debug] <0.1337.0> MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.040597+00:00 [debug] <0.1337.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.040793+00:00 [info] <0.1337.0> Accepted MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.041463+00:00 [debug] <0.1340.0> MQTT accepting TCP connection <0.1340.0> (127.0.0.1:42596 -> 127.0.0.1:27005) 2024-11-04 14:43:35.041715+00:00 [debug] <0.1340.0> Received a CONNECT, client ID: c1, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.041806+00:00 [debug] <0.1340.0> MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.041881+00:00 [debug] <0.1340.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.041982+00:00 [warning] <0.1328.0> MQTT disconnecting client <<"127.0.0.1:42560 -> 127.0.0.1:27005">> with duplicate id 'c1' 2024-11-04 14:43:35.042062+00:00 [info] <0.1340.0> Accepted MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 for client ID c1 2024-11-04 14:43:35.045624+00:00 [debug] <0.1345.0> MQTT accepting TCP connection <0.1345.0> (127.0.0.1:42602 -> 127.0.0.1:27005) 2024-11-04 14:43:35.045781+00:00 [debug] <0.1345.0> Received a CONNECT, client ID: c2, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.045874+00:00 [debug] <0.1345.0> MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.045943+00:00 [debug] <0.1345.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.046032+00:00 [warning] <0.1331.0> MQTT disconnecting client <<"127.0.0.1:42566 -> 127.0.0.1:27005">> with duplicate id 'c2' 2024-11-04 14:43:35.046281+00:00 [info] <0.1345.0> Accepted MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 for client ID c2 2024-11-04 14:43:35.047063+00:00 [debug] <0.1350.0> MQTT accepting TCP connection <0.1350.0> (127.0.0.1:42614 -> 127.0.0.1:27005) 2024-11-04 14:43:35.047702+00:00 [debug] <0.1350.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.047910+00:00 [debug] <0.1350.0> MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.048467+00:00 [debug] <0.1350.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.049701+00:00 [info] <0.1350.0> Accepted MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 for client ID c3 2024-11-04 14:43:35.050907+00:00 [warning] <0.1334.0> MQTT disconnecting client <<"127.0.0.1:42576 -> 127.0.0.1:27005">> with duplicate id 'c3' 2024-11-04 14:43:35.051248+00:00 [debug] <0.1353.0> MQTT accepting TCP connection <0.1353.0> (127.0.0.1:42626 -> 127.0.0.1:27005) 2024-11-04 14:43:35.051395+00:00 [debug] <0.1353.0> Received a CONNECT, client ID: c4, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2024-11-04 14:43:35.051519+00:00 [debug] <0.1353.0> MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2024-11-04 14:43:35.051590+00:00 [debug] <0.1353.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2024-11-04 14:43:35.051871+00:00 [info] <0.1353.0> Accepted MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 for client ID c4 2024-11-04 14:43:35.051960+00:00 [warning] <0.1337.0> MQTT disconnecting client <<"127.0.0.1:42580 -> 127.0.0.1:27005">> with duplicate id 'c4' 2024-11-04 14:43:35.052689+00:00 [debug] <0.1337.0> sent Will Message to topic my/topic for MQTT client ID c4 2024-11-04 14:43:35.054119+00:00 [debug] <0.1334.0> sent Will Message to topic my/topic for MQTT client ID c3 ``` We see nicely how RabbitMQ sends the will message for both c3 and c4. However, the order in which RabbitMQ sends is not guaranteed. Hence, we adapt the test expectation to not depend on the order of Will messages being received.
2024-11-08 23:12:52 +08:00
end,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
assert_nothing_received(),
[ok = emqtt:disconnect(C) || C <- [Sub, C1b, C2b, C3b, C4b]].
will_delay_message_expiry(Config) ->
Q1 = <<"dead-letter-queue-1">>,
Q2 = <<"dead-letter-queue-2">>,
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{
queue = Q1,
arguments = [{<<"x-dead-letter-exchange">>, longstr, <<"">>},
{<<"x-dead-letter-routing-key">>, longstr, Q2}]}),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q1,
exchange = <<"amq.topic">>,
routing_key = <<"my.topic">>}),
#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{queue = Q2}),
C = connect(<<"my-client">>, Config,
[{properties, #{'Session-Expiry-Interval' => 1}},
{will_props, #{'Will-Delay-Interval' => 1,
'Message-Expiry-Interval' => 1}},
{will_topic, <<"my/topic">>},
{will_payload, <<"msg">>}]),
ok = emqtt:disconnect(C, ?RC_DISCONNECT_WITH_WILL),
%% After 1 second, Will Message is published, i.e. dead lettered the 1st time,
%% after 2 seconds, Will Message is dead lettered the 2nd time:
%% mqtt-will-my-client -> dead-letter-queue-1 -> dead-letter-queue-2
%% Wait for 2 more seconds to validate that the Will Message does not expire again in dead-letter-queue-2.
timer:sleep(4000),
?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"msg">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q2})),
[#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}) || Q <- [Q1, Q2]].
%% "The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval
%% set to the received value minus the time that the message has been waiting in the Server."
%% [v5 MQTT-3.3.2-6]
%% Test that requirement for a Will Message with both Will Delay Interval and Message Expiry Interval.
will_delay_message_expiry_publish_properties(Config) ->
Topic = <<"my/topic">>,
ClientId = Payload = atom_to_binary(?FUNCTION_NAME),
Sub1 = connect(ClientId, Config, [{properties, #{'Session-Expiry-Interval' => 60}}]),
{ok, _, [1]} = emqtt:subscribe(Sub1, Topic, qos1),
ok = emqtt:disconnect(Sub1),
C = connect(<<"will">>, Config,
[{properties, #{'Session-Expiry-Interval' => 2}},
{will_props, #{'Will-Delay-Interval' => 2,
'Message-Expiry-Interval' => 20}},
{will_topic, Topic},
{will_qos, 1},
{will_payload, Payload}]),
ok = emqtt:disconnect(C, ?RC_DISCONNECT_WITH_WILL),
%% After 2 seconds, Will Message is published.
%% Wait for 2 more seconds to check the Message Expiry Interval sent to the client
%% is adjusted correctly.
timer:sleep(4000),
Sub2 = connect(ClientId, Config, [{clean_start, false}]),
receive {publish, #{client_pid := Sub2,
topic := Topic,
payload := Payload,
properties := #{'Message-Expiry-Interval' := MEI}}} ->
%% Since the point in time the Message was published,
%% it has been waiting in the Server for 2 seconds to be consumed.
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
assert_message_expiry_interval(20 - 2, MEI);
Other -> ct:fail("received unexpected message: ~p", [Other])
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive Will Message")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
ok = emqtt:disconnect(Sub2).
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
%% Test all Will Properties (v5 3.1.3.2) that are forwarded unaltered by the server.
will_properties(Config) ->
will_properties0(Config, 0).
%% Test all Will Properties (v5 3.1.3.2) that are forwarded unaltered by the server
%% when Will Delay Interval is set.
will_delay_properties(Config) ->
will_properties0(Config, 1).
will_properties0(Config, WillDelayInterval) ->
Topic = Payload = atom_to_binary(?FUNCTION_NAME),
Sub = connect(<<"sub">>, Config),
{ok, _, [0]} = emqtt:subscribe(Sub, Topic),
UserProperty = [{<<"k1">>, <<"v2">>},
{<<>>, <<>>},
{<<"k1">>, <<"v1">>}],
CorrelationData = binary:copy(<<"x">>, 65_000),
C = connect(<<"will">>, Config,
[{properties, #{'Session-Expiry-Interval' => 1}},
{will_props, #{'Will-Delay-Interval' => WillDelayInterval,
'User-Property' => UserProperty,
'Content-Type' => <<"text/plain😎;charset=UTF-8"/utf8>>,
'Payload-Format-Indicator' => 1,
'Response-Topic' => <<"response/topic">>,
'Correlation-Data' => CorrelationData}},
{will_topic, Topic},
{will_qos, 0},
{will_payload, Payload}]),
ok = emqtt:disconnect(C, ?RC_DISCONNECT_WITH_WILL),
if WillDelayInterval > 0 ->
receive Unexpected -> ct:fail(Unexpected)
after 700 -> ok
end;
WillDelayInterval =:= 0 ->
ok
end,
receive {publish,
#{client_pid := Sub,
topic := Topic,
payload := Payload,
properties := #{'User-Property' := UserProperty,
'Content-Type' := <<"text/plain😎;charset=UTF-8"/utf8>>,
'Payload-Format-Indicator' := 1,
'Response-Topic' := <<"response/topic">>,
'Correlation-Data' := CorrelationData} = Props}}
when map_size(Props) =:= 5 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive Will Message")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
ok = emqtt:disconnect(Sub).
%% "When an Application Message is transported by MQTT it contains payload data,
%% a Quality of Service (QoS), a collection of Properties, and a Topic Name" [v5 1.2]
%% Since a retained message is an Application Message, it must also include the Properties.
%% This test checks that the whole Application Message, especially Properties, are forwarded
%% to future subscribers.
retain_properties(Config) ->
Props = #{'Content-Type' => <<"text/plain;charset=UTF-8">>,
'User-Property' => [{<<"k1">>, <<"v2">>},
{<<>>, <<>>},
{<<"k1">>, <<"v1">>}],
'Payload-Format-Indicator' => 1,
'Response-Topic' => <<"response/topic">>,
'Correlation-Data' => <<"some correlation data">>},
%% Let's test both ways to retain messages:
%% 1. a Will Message that is retained, and
%% 2. a PUBLISH message that is retained
Pub = connect(<<"publisher">>, Config,
[{will_retain, true},
{will_topic, <<"t/1">>},
{will_payload, <<"m1">>},
{will_qos, 1},
{will_props, Props}]),
{ok, _} = emqtt:publish(Pub, <<"t/2">>, Props, <<"m2">>, [{retain, true}, {qos, 1}]),
ok = emqtt:disconnect(Pub, ?RC_DISCONNECT_WITH_WILL),
%% Both messages are now retained.
Sub = connect(<<"subscriber">>, Config),
{ok, _, [1, 1]} = emqtt:subscribe(Sub, [{<<"t/1">>, qos1},
{<<"t/2">>, qos1}]),
receive {publish,
#{client_pid := Sub,
topic := <<"t/1">>,
payload := <<"m1">>,
retain := true,
qos := 1,
properties := Props}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m1")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
receive {publish,
#{client_pid := Sub,
topic := <<"t/2">>,
payload := <<"m2">>,
retain := true,
qos := 1,
properties := Props}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m2")
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
end,
Clear retained messages synchronously due to the following flake: ``` v5_SUITE:subscription_identifier failed on line 783 Reason: {test_case_failed,Received unexpected message: {publish,#{client_pid => <0.495.0>,dup => false, packet_id => undefined, payload => <<"m3">>,properties => #{}, qos => 0,retain => true, topic => <<"t/3">>, via => #Port<0.164>}}} ``` Also, log if unexpected message received due to flake in ``` === Ended at 2023-06-22 14:30:07 === Location: [{v5_SUITE,will_delay_message_expiry_publish_properties,1597}, {test_server,ts_tc,1782}, {test_server,run_test_case_eval1,1291}, {test_server,run_test_case_eval,1223}] === === Reason: {test_case_failed,"did not receive Will Message"} ```
2023-06-23 18:09:27 +08:00
{ok, _} = emqtt:publish(Sub, <<"t/1">>, <<>>, [{retain, true}, {qos, 1}]),
{ok, _} = emqtt:publish(Sub, <<"t/2">>, <<>>, [{retain, true}, {qos, 1}]),
Support MQTT 5.0 Properties The following PUBLISH and Will properties are forwarded unaltered by the server: * Payload Format Indicator * Content Type * Response Topic * Correlation Data * User Property Not only must these properties be forwarded unaltered from an MQTT publishing client to an MQTT receiving client, but it would also be nice to allow for protocol interoperability: Think about RPC request-response style patterns where the requester is an MQTT client and the responder is an AMQP 0.9.1 or STOMP client. We reuse the P_basic fields where possible: * content_type (if <= 255 bytes) * correlation_id (if <= 255 bytes) Otherwise, we add custom AMQP 0.9.1 headers. The headers follow the naming pattern "x-mqtt-<property>" where <property> is the MQTT v5 property if that property makes only (mainly) sense in the MQTT world: * x-mqtt-user-property * x-mqtt-payload-format-indicator If the MQTT v5 property makes also sense outside of the MQTT world, we name it more generic: * x-correlation (if > 255 bytes) * x-reply-to-topic (since P_basic.reply_to assumes a queue name) In the future, we can think about adding a header x-reply-to-exchange and have the MQTT plugin set its value to the configured mqtt.exchange such that clients don't have to assume the default topic exchange amq.topic.
2023-06-02 22:37:34 +08:00
ok = emqtt:disconnect(Sub).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% "In the case of a Server shutdown or failure, the Server MAY defer publication of Will Messages
%% until a subsequent restart. If this happens, there might be a delay between the time the Server
%% experienced failure and when the Will Message is published." [v5 3.1.2.5]
%%
%% This test ensures that if a server is drained, shut down, the delayed Will Message expires,
%% and the server restarts, the delayed Will Message will still be published.
%% Publishing delayed Will Messages for unclean server shutdowns is currently not supported.
will_delay_node_restart(Config) ->
Topic = <<"my/topic">>,
Payload = <<"my-will">>,
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
Sub0a = connect(<<"sub0">>, Config, 0, [{properties, #{'Session-Expiry-Interval' => 900}}]),
{ok, _, [0]} = emqtt:subscribe(Sub0a, Topic),
Sub1 = connect(<<"sub1">>, Config, 1, []),
{ok, _, [0]} = emqtt:subscribe(Sub1, Topic),
Fix MQTT test flake in Khepri mixed version mode The following test flaked in CI under Khepri in mixed version mode: ``` make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_3:will_delay_node_restart RABBITMQ_METADATA_STORE=khepri SECONDARY_DIST=rabbitmq_server-4.0.5 FULL=1 ``` The first node took exactly 30 seconds for draining: ``` 2025-02-10 15:00:09.550824+00:00 [debug] <0.1449.0> MQTT accepting TCP connection <0.1449.0> (127.0.0.1:33376 -> 127.0.0.1:27005) 2025-02-10 15:00:09.550992+00:00 [debug] <0.1449.0> Received a CONNECT, client ID: sub0, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2025-02-10 15:00:09.551134+00:00 [debug] <0.1449.0> MQTT connection 127.0.0.1:33376 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2025-02-10 15:00:09.551219+00:00 [debug] <0.1449.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2025-02-10 15:00:09.551530+00:00 [info] <0.1449.0> Accepted MQTT connection 127.0.0.1:33376 -> 127.0.0.1:27005 for client ID sub0 2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> Received a SUBSCRIBE with subscription(s) [{mqtt_subscription,<<"my/topic">>, 2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> {mqtt_subscription_opts,0,false, 2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> false,0,undefined}}] 2025-02-10 15:00:09.556233+00:00 [debug] <0.896.0> RabbitMQ metadata store: follower leader cast - redirecting to {rabbitmq_metadata,'rmq-ct-mqtt-cluster_size_3-2-27054@localhost'} 2025-02-10 15:00:09.561518+00:00 [debug] <0.1456.0> MQTT accepting TCP connection <0.1456.0> (127.0.0.1:33390 -> 127.0.0.1:27005) 2025-02-10 15:00:09.561634+00:00 [debug] <0.1456.0> Received a CONNECT, client ID: will, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval'] 2025-02-10 15:00:09.561715+00:00 [debug] <0.1456.0> MQTT connection 127.0.0.1:33390 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost 2025-02-10 15:00:09.561828+00:00 [debug] <0.1456.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal 2025-02-10 15:00:09.562596+00:00 [info] <0.1456.0> Accepted MQTT connection 127.0.0.1:33390 -> 127.0.0.1:27005 for client ID will 2025-02-10 15:00:09.565743+00:00 [warning] <0.1460.0> This node is being put into maintenance (drain) mode 2025-02-10 15:00:09.565833+00:00 [debug] <0.1460.0> Marking the node as undergoing maintenance 2025-02-10 15:00:09.570772+00:00 [info] <0.1460.0> Marked this node as undergoing maintenance 2025-02-10 15:00:09.570904+00:00 [info] <0.1460.0> Asked to suspend 9 client connection listeners. No new client connections will be accepted until these listeners are resumed! 2025-02-10 15:00:09.572268+00:00 [warning] <0.1460.0> Suspended all listeners and will no longer accept client connections 2025-02-10 15:00:09.572317+00:00 [warning] <0.1460.0> Closed 0 local client connections 2025-02-10 15:00:09.572418+00:00 [warning] <0.1449.0> MQTT disconnecting client <<"127.0.0.1:33376 -> 127.0.0.1:27005">> with client ID 'sub0', reason: maintenance 2025-02-10 15:00:09.572414+00:00 [warning] <0.1000.0> Closed 2 local (Web) MQTT client connections 2025-02-10 15:00:09.572499+00:00 [warning] <0.1456.0> MQTT disconnecting client <<"127.0.0.1:33390 -> 127.0.0.1:27005">> with client ID 'will', reason: maintenance 2025-02-10 15:00:09.572866+00:00 [alert] <0.1000.0> Closed 0 local STOMP client connections 2025-02-10 15:00:09.577432+00:00 [debug] <0.1456.0> scheduled delayed Will Message to topic my/topic for MQTT client ID will to be sent in 10000 ms 2025-02-10 15:00:12.991328+00:00 [debug] <0.1469.0> Will reconcile virtual host processes on all cluster members... 2025-02-10 15:00:12.991443+00:00 [debug] <0.1469.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes 2025-02-10 15:00:12.992497+00:00 [debug] <0.1469.0> Done with virtual host processes reconciliation (run 3) 2025-02-10 15:00:16.511733+00:00 [debug] <0.1476.0> Will reconcile virtual host processes on all cluster members... 2025-02-10 15:00:16.511864+00:00 [debug] <0.1476.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes 2025-02-10 15:00:16.514293+00:00 [debug] <0.1476.0> Done with virtual host processes reconciliation (run 4) 2025-02-10 15:00:24.897477+00:00 [debug] <0.1479.0> Will reconcile virtual host processes on all cluster members... 2025-02-10 15:00:24.897607+00:00 [debug] <0.1479.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes 2025-02-10 15:00:24.898483+00:00 [debug] <0.1479.0> Done with virtual host processes reconciliation (run 5) 2025-02-10 15:00:24.898527+00:00 [debug] <0.1479.0> Will reschedule virtual host process reconciliation after 30 seconds 2025-02-10 15:00:32.994347+00:00 [debug] <0.1484.0> Will reconcile virtual host processes on all cluster members... 2025-02-10 15:00:32.994474+00:00 [debug] <0.1484.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes 2025-02-10 15:00:32.996539+00:00 [debug] <0.1484.0> Done with virtual host processes reconciliation (run 6) 2025-02-10 15:00:32.996585+00:00 [debug] <0.1484.0> Will reschedule virtual host process reconciliation after 30 seconds 2025-02-10 15:00:39.576325+00:00 [info] <0.1460.0> Will transfer leadership of 0 quorum queues with current leader on this node 2025-02-10 15:00:39.576456+00:00 [info] <0.1460.0> Leadership transfer for quorum queues hosted on this node has been initiated 2025-02-10 15:00:39.576948+00:00 [info] <0.1460.0> Will stop local follower replicas of 0 quorum queues on this node 2025-02-10 15:00:39.576990+00:00 [info] <0.1460.0> Stopped all local replicas of quorum queues hosted on this node 2025-02-10 15:00:39.577120+00:00 [info] <0.1460.0> Will transfer leadership of metadata store with current leader on this node 2025-02-10 15:00:39.577282+00:00 [info] <0.1460.0> Khepri clustering: transferring leadership to node 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost' 2025-02-10 15:00:39.577424+00:00 [info] <0.1460.0> Khepri clustering: skipping leadership transfer, leader is already in node 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost' 2025-02-10 15:00:39.577547+00:00 [info] <0.1460.0> Leadership transfer for metadata store on this node has been done. The new leader is 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost' 2025-02-10 15:00:39.577674+00:00 [info] <0.1460.0> Node is ready to be shut down for maintenance or upgrade 2025-02-10 15:00:39.595638+00:00 [notice] <0.64.0> SIGTERM received - shutting down 2025-02-10 15:00:39.595638+00:00 [notice] <0.64.0> 2025-02-10 15:00:39.595758+00:00 [debug] <0.44.0> Running rabbit_prelaunch:shutdown_func() as part of `kernel` shutdown ``` Running the same test locally revealed that [rabbit_maintenance:status_consistent_read/1](https://github.com/rabbitmq/rabbitmq-server/blob/55ae91809433d9e6edfcc98563bcb2f0736ee79e/deps/rabbit/src/rabbit_maintenance.erl#L131) takes exactly 30 seconds to complete. The test case assumes a Will Delay higher than the time it takes to drain and shut down the node. Hence, this commit increases the Will Delay time from 10 seconds to 40 seconds.
2025-02-12 01:06:01 +08:00
%% In mixed version mode with Khepri, draining the node can take 30 seconds.
WillDelaySecs = 40,
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
C0a = connect(<<"will">>, Config, 0,
[{properties, #{'Session-Expiry-Interval' => 900}},
{will_props, #{'Will-Delay-Interval' => WillDelaySecs}},
{will_topic, Topic},
{will_qos, 0},
{will_payload, Payload}]),
ClientsNode0 = [Sub0a, C0a],
[unlink(C) || C <- ClientsNode0],
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
T = erlang:monotonic_time(millisecond),
ok = rabbit_ct_broker_helpers:drain_node(Config, 0),
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
[receive {disconnected, ?RC_SERVER_SHUTTING_DOWN, #{}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("server did not disconnect us")
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
end || _ <- ClientsNode0],
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
ok = rabbit_ct_broker_helpers:stop_node(Config, 0),
ElapsedMs = erlang:monotonic_time(millisecond) - T,
SleepMs = max(0, timer:seconds(WillDelaySecs) - ElapsedMs),
ct:pal("Sleeping for ~b ms waiting for Will Message to expire while node 0 is down...", [SleepMs]),
timer:sleep(SleepMs),
assert_nothing_received(),
ok = rabbit_ct_broker_helpers:start_node(Config, 0),
Mqtt test: Solve dependencies for web mqtt
2025-06-06 15:43:19 +08:00
[util:enable_plugin(Config, Plugin) || Plugin <- ?config(test_plugins, Config)],
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% After node 0 restarts, we should receive the Will Message promptly on both nodes 0 and 1.
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
receive {publish, #{client_pid := Sub1,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
payload := Payload}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive Will Message on node 1")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
Sub0b = connect(<<"sub0">>, Config, 0, [{clean_start, false}]),
receive {publish, #{client_pid := Sub0b,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
payload := Payload}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive Will Message on node 0")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
DISCONNECT v5 clients with Server Shutting Down When RabbitMQ enters maintenance mode / is being drained, all client connections are closed. This commit sends a DISCONNECT packet to (Web) MQTT 5.0 clients with Reason Code "Server shutting down" before the connection is closed.
2023-06-05 21:15:58 +08:00
ok = emqtt:disconnect(Sub0b),
ok = emqtt:disconnect(Sub1),
C0b = connect(<<"will">>, Config),
ok = emqtt:disconnect(C0b).
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
session_migrate_v3_v5(Config) ->
session_switch_v3_v5(Config, true).
session_takeover_v3_v5(Config) ->
session_switch_v3_v5(Config, false).
session_switch_v3_v5(Config, Disconnect) ->
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
%% Connect to old node in mixed version cluster.
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
C1 = connect(ClientId, Config, 1, [{proto_ver, v3} | non_clean_sess_opts()]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(C1))),
{ok, _, [1]} = emqtt:subscribe(C1, Topic, qos1),
case Disconnect of
true -> ok = emqtt:disconnect(C1);
false -> unlink(C1)
end,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% Upgrade session from v3 to v5 (on new node in mixed version cluster).
C2 = connect(ClientId, Config, 0, [{proto_ver, v5} | non_clean_sess_opts()]),
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(C2))),
%% Subscription created with old v3 client should work.
{ok, _} = emqtt:publish(C2, Topic, <<"m1">>, qos1),
receive {publish,
#{client_pid := C2,
payload := <<"m1">>,
qos := 1}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive from m1")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end,
%% Modifying subscription with v5 specific feature should work.
{ok, _, [1]} = emqtt:subscribe(C2, Topic, [{nl, true}, {qos, 1}]),
{ok, _} = emqtt:publish(C2, Topic, <<"m2">>, qos1),
receive {publish, P} -> ct:fail("Unexpected local PUBLISH: ~p", [P])
after 500 -> ok
end,
case Disconnect of
true -> ok = emqtt:disconnect(C2);
false -> unlink(C2)
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
%% Downgrade session from v5 to v3.
C3 = connect(ClientId, Config, 0, [{proto_ver, v3} | non_clean_sess_opts()]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
?assertEqual(3, proplists:get_value(proto_ver, emqtt:info(C3))),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
case Disconnect of
true -> ok;
false -> receive {disconnected, ?RC_SESSION_TAKEN_OVER, #{}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("missing DISCONNECT packet for C2")
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
end
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% We expect that v5 specific subscription feature does not apply
%% anymore when downgrading the session.
{ok, _} = emqtt:publish(C3, Topic, <<"m3">>, qos1),
receive {publish,
#{client_pid := C3,
payload := <<"m3">>,
qos := 1}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m3 with QoS 1")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% Modifying the subscription once more with v3 client should work.
{ok, _, [0]} = emqtt:subscribe(C3, Topic, qos0),
{ok, _} = emqtt:publish(C3, Topic, <<"m4">>, qos1),
receive {publish,
#{client_pid := C3,
payload := <<"m4">>,
qos := 0}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("did not receive m3 with QoS 0")
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
end,
%% Unsubscribing in v3 should work.
?assertMatch({ok, _, _}, emqtt:unsubscribe(C3, Topic)),
{ok, _} = emqtt:publish(C3, Topic, <<"m5">>, qos1),
assert_nothing_received(),
ok = emqtt:disconnect(C3),
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
C4 = connect(ClientId, Config, 0, [{proto_ver, v3}, {clean_start, true}]),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
ok = emqtt:disconnect(C4),
eventually(?_assertEqual([], all_connection_pids(Config))).
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
topic_alias_client_to_server(Config) ->
ClientId = atom_to_binary(?FUNCTION_NAME),
Topic1 = <<"t/1">>,
Sub = connect(<<ClientId/binary, "_sub">>, Config),
QoS1 = [{qos, 1}],
{ok, _, [1]} = emqtt:subscribe(Sub, Topic1, QoS1),
Pub = connect(<<ClientId/binary, "_pub">>, Config),
{ok, _} = emqtt:publish(Pub, Topic1, #{'Topic-Alias' => 5}, <<"m1">>, QoS1),
{ok, _} = emqtt:publish(Pub, <<>>, #{'Topic-Alias' => 5}, <<"m2">>, QoS1),
{ok, _} = emqtt:publish(Pub, <<>>, #{'Topic-Alias' => 5}, <<"m3">>, QoS1),
{ok, _} = emqtt:publish(Pub, Topic1, #{'Topic-Alias' => 7}, <<"m4">>, QoS1),
{ok, _} = emqtt:publish(Pub, <<>>, #{'Topic-Alias' => 7}, <<"m5">>, QoS1),
ok = expect_publishes(Sub, Topic1, [<<"m1">>, <<"m2">>, <<"m3">>, <<"m4">>, <<"m5">>]),
Topic2 = <<"t/2">>,
{ok, _, [0]} = emqtt:subscribe(Sub, Topic2, [{qos, 0}]),
{ok, _} = emqtt:publish(Pub, Topic2, #{'Topic-Alias' => 2}, <<"m6">>, QoS1),
{ok, _} = emqtt:publish(Pub, <<>>, #{'Topic-Alias' => 2}, <<"m7">>, QoS1),
{ok, _} = emqtt:publish(Pub, <<>>, #{'Topic-Alias' => 7}, <<"m8">>, QoS1),
ok = expect_publishes(Sub, Topic2, [<<"m6">>, <<"m7">>]),
ok = expect_publishes(Sub, Topic1, [<<"m8">>]),
ok = emqtt:disconnect(Sub),
ok = emqtt:disconnect(Pub).
topic_alias_server_to_client(Config) ->
Key = mqtt_topic_alias_maximum,
DefaultMax = rpc(Config, persistent_term, get, [Key]),
%% The Topic Alias Maximum configured on the server:
%% 1. defines the Topic Alias Maximum for messages from client to server, and
%% 2. serves as an upper bound for the Topic Alias Maximum for messages from server to client
TopicAliasMaximum = 2,
ok = rpc(Config, persistent_term, put, [Key, TopicAliasMaximum]),
ClientId = ?FUNCTION_NAME,
{C1, Connect} = start_client(ClientId, Config, 0, [{properties, #{'Topic-Alias-Maximum' => 16#ffff}}]),
%% Validate 2.
?assertMatch({ok, #{'Topic-Alias-Maximum' := TopicAliasMaximum}}, Connect(C1)),
{ok, _, [1]} = emqtt:subscribe(C1, <<"#">>, qos1),
{ok, _} = emqtt:publish(C1, <<"t/1">>, <<"m1">>, qos1),
A1 = receive {publish, #{payload := <<"m1">>,
topic := <<"t/1">>,
properties := #{'Topic-Alias' := A1a}}} -> A1a
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m1")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
%% We don't expect a Topic Alias when the Topic Name consists of a single byte.
{ok, _} = emqtt:publish(C1, <<"t">>, <<"m2">>, qos1),
receive {publish, #{payload := <<"m2">>,
topic := <<"t">>,
properties := Props1}}
when map_size(Props1) =:= 0 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m2")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
{ok, _} = emqtt:publish(C1, <<"t/2">>, <<"m3">>, qos1),
A2 = receive {publish, #{payload := <<"m3">>,
topic := <<"t/2">>,
properties := #{'Topic-Alias' := A2a}}} -> A2a
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m3")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
?assertEqual([1, 2], lists:sort([A1, A2])),
{ok, _} = emqtt:publish(C1, <<"t/3">>, <<"m4">>, qos1),
%% In the current server implementation, once the Topic Alias cache is full,
%% existing aliases won't be replaced. So, we expect to get the Topic Name instead.
receive {publish, #{payload := <<"m4">>,
topic := <<"t/3">>,
properties := Props2}}
when map_size(Props2) =:= 0 -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m4")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
%% Existing topic aliases should still be sent.
ok = emqtt:publish(C1, <<"t/1">>, <<"m5">>, qos0),
ok = emqtt:publish(C1, <<"t/2">>, <<"m6">>, qos0),
receive {publish, #{payload := <<"m5">>,
topic := <<>>,
properties := #{'Topic-Alias' := A1b}}} ->
?assertEqual(A1, A1b)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m5")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
receive {publish, #{payload := <<"m6">>,
topic := <<>>,
properties := #{'Topic-Alias' := A2b}}} ->
?assertEqual(A2, A2b)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m6")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
ok = emqtt:disconnect(C1),
ok = rpc(Config, persistent_term, put, [Key, DefaultMax]).
%% "The Topic Alias mappings used by the Client and Server are independent from each other.
%% Thus, when a Client sends a PUBLISH containing a Topic Alias value of 1 to a Server and
%% the Server sends a PUBLISH with a Topic Alias value of 1 to that Client they will in
%% general be referring to different Topics." [v5 3.3.2.3.4]
topic_alias_bidirectional(Config) ->
C1 = connect(<<"client 1">>, Config, [{properties, #{'Topic-Alias-Maximum' => 1}}]),
C2 = connect(<<"client 2">>, Config),
Topic1 = <<"/a/a">>,
Topic2 = <<"/b/b">>,
{ok, _, [0]} = emqtt:subscribe(C1, Topic1),
{ok, _, [0]} = emqtt:subscribe(C2, Topic2),
ok = emqtt:publish(C1, Topic2, #{'Topic-Alias' => 1}, <<"m1">>, [{qos, 0}]),
ok = emqtt:publish(C2, Topic1, <<"m2">>),
ok = emqtt:publish(C1, <<>>, #{'Topic-Alias' => 1}, <<"m3">>, [{qos, 0}]),
ok = emqtt:publish(C2, Topic1, <<"m4">>),
ok = expect_publishes(C2, Topic2, [<<"m1">>, <<"m3">>]),
receive {publish, #{client_pid := C1,
payload := <<"m2">>,
topic := Topic1,
properties := #{'Topic-Alias' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m2")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
receive {publish, #{client_pid := C1,
payload := <<"m4">>,
topic := <<>>,
properties := #{'Topic-Alias' := 1}}} -> ok
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive m4")
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
end,
ok = emqtt:disconnect(C1),
ok = emqtt:disconnect(C2).
topic_alias_invalid(Config) ->
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
{C1, Connect} = start_client(ClientId, Config, 0, []),
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
?assertMatch({ok, #{'Topic-Alias-Maximum' := 16}}, Connect(C1)),
process_flag(trap_exit, true),
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
?assertMatch({error, {disconnected, ?RC_TOPIC_ALIAS_INVALID, _}},
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
emqtt:publish(C1, Topic, #{'Topic-Alias' => 17}, <<"msg">>, [{qos, 1}])),
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
C2 = connect(ClientId, Config),
?assertMatch({error, {disconnected, ?RC_TOPIC_ALIAS_INVALID, _}},
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
emqtt:publish(C2, Topic, #{'Topic-Alias' => 0}, <<"msg">>, [{qos, 1}])).
Make Topic Alias Maximum configurable - default to 20, configurable through cuttlefish config - add test to v5 suite for invalid topic alias in publish
2023-06-05 23:06:44 +08:00
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
topic_alias_unknown(Config) ->
C = connect(?FUNCTION_NAME, Config),
Support topic alias from client to broker
2023-06-08 22:12:43 +08:00
unlink(C),
?assertMatch({error, {disconnected, ?RC_PROTOCOL_ERROR, _}},
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
emqtt:publish(C, <<>>, #{'Topic-Alias' => 1}, <<"msg">>, [{qos, 1}])).
Support topic alias from client to broker
2023-06-08 22:12:43 +08:00
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
%% A RabbitMQ operator should be able to disallow topic aliases.
topic_alias_disallowed(Config) ->
Key = mqtt_topic_alias_maximum,
DefaultMax = rpc(Config, persistent_term, get, [Key]),
TopicAliasMaximum = 0,
ok = rpc(Config, persistent_term, put, [Key, TopicAliasMaximum]),
Support topic alias from client to broker
2023-06-08 22:12:43 +08:00
Add Topic Aliases from server to client Once the server's Topic Alias cache for messages from server to client is full, this commit does not replace any existing aliases. So, the first topics "win" and stay in the cache forever. This matches the behaviour of VerneMQ and EMQX. For now that's good enough. In the future, we can easily change that behaviour to some smarter strategy, for example 1. Hash the TopicName to an Topic Alias and replace the old alias, or 2. For the Topic Alias Cache from server to client, keep 2 Maps: #{TopicName => TopicAlias} and #{TopicAlias => TopicName} and a counter that wraps to 1 once the Topic Alias Maximum is reached and just replace an existing Alias if the TopicName is not cached. Also, refactor Topic Alias Maximum: * Remove code duplication * Allow an operator to prohibit Topic Aliases by allowing value 0 to be configured * Change config name to topic_alias_maximum to that it matches exactly the MQTT feture name * Fix wrong code formatting * Add the invalid or unkown Topic Alias to log message for easier troubleshooting
2023-06-12 22:31:50 +08:00
{C, Connect} = start_client(?FUNCTION_NAME, Config, 0, [{properties, #{'Topic-Alias-Maximum' => 10}}]),
?assertMatch({ok, #{'Topic-Alias-Maximum' := 0}}, Connect(C)),
unlink(C),
?assertMatch({error, {disconnected, ?RC_TOPIC_ALIAS_INVALID, _}},
emqtt:publish(C, <<"t">>, #{'Topic-Alias' => 1}, <<"msg">>, [{qos, 1}])),
ok = rpc(Config, persistent_term, put, [Key, DefaultMax]).
Support topic alias from client to broker
2023-06-08 22:12:43 +08:00
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
topic_alias_retained_message(Config) ->
topic_alias_in_retained_message0(Config, 1, 10, #{'Topic-Alias' => 1}).
Set topic alias when publish retained message - remove topic alias from message props when storing retained msgs - set topic alias for outbound before sending retained msgs
2023-06-19 22:46:48 +08:00
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
topic_alias_disallowed_retained_message(Config) ->
topic_alias_in_retained_message0(Config, 0, 1, #{}).
Set topic alias when publish retained message - remove topic alias from message props when storing retained msgs - set topic alias for outbound before sending retained msgs
2023-06-19 22:46:48 +08:00
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
topic_alias_in_retained_message0(Config, TopicAliasMax, TopicAlias, ExpectedProps) ->
Payload = Topic = ClientId = atom_to_binary(?FUNCTION_NAME),
C = connect(ClientId, Config, [{properties, #{'Topic-Alias-Maximum' => TopicAliasMax}}]),
{ok, _} = emqtt:publish(C, Topic, #{'Topic-Alias' => TopicAlias}, Payload, [{retain, true}, {qos, 1}]),
Set topic alias when publish retained message - remove topic alias from message props when storing retained msgs - set topic alias for outbound before sending retained msgs
2023-06-19 22:46:48 +08:00
{ok, _, [1]} = emqtt:subscribe(C, Topic, [{qos, 1}]),
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
receive {publish, #{payload := Payload,
Set topic alias when publish retained message - remove topic alias from message props when storing retained msgs - set topic alias for outbound before sending retained msgs
2023-06-19 22:46:48 +08:00
topic := Topic,
retain := true,
Fix test expectation Matching against #{} does not validate that the map is empty.
2023-06-20 15:15:04 +08:00
properties := Props}} ->
?assertEqual(ExpectedProps, Props)
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after ?TIMEOUT -> ct:fail("Did not receive retained message")
Set topic alias when publish retained message - remove topic alias from message props when storing retained msgs - set topic alias for outbound before sending retained msgs
2023-06-19 22:46:48 +08:00
end,
ok = emqtt:disconnect(C).
CONNACK with Bad Authentication Method RabbitMQ MQTT already supports authenticating clients via username + password, OAuth tokens, and certificates. We could make use of RabbitMQ SASL mechanisms in the future, if needed. For now, if the client wants to perform extended authentication, we return Bad Authentication Method in the CONNACK packet.
2023-06-18 19:53:12 +08:00
extended_auth(Config) ->
{C, Connect} = start_client(?FUNCTION_NAME, Config, 0,
[{properties, #{'Authentication-Method' => <<"OTP">>,
'Authentication-Data' => <<"123456">>}}]),
unlink(C),
?assertEqual({error, {bad_authentication_method, #{}}}, Connect(C)).
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
%% Binding a headers exchange to the MQTT topic exchange should support
%% routing based on (topic and) User Property in the PUBLISH packet.
headers_exchange(Config) ->
HeadersX = <<"my-headers-exchange">>,
Q1 = <<"q1">>,
Q2 = <<"q2">>,
Qs = [Q1, Q2],
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'exchange.declare_ok'{} = amqp_channel:call(
Ch, #'exchange.declare'{exchange = HeadersX,
type = <<"headers">>,
durable = true,
auto_delete = true}),
#'exchange.bind_ok'{} = amqp_channel:call(
Ch, #'exchange.bind'{destination = HeadersX,
source = <<"amq.topic">>,
routing_key = <<"my.topic">>}),
[#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{queue = Q,
durable = true}) || Q <- Qs],
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q1,
exchange = HeadersX,
arguments = [{<<"x-match">>, longstr, <<"any">>},
{<<"k1">>, longstr, <<"v1">>},
{<<"k2">>, longstr, <<"v2">>}]
}),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q2,
exchange = HeadersX,
arguments = [{<<"x-match">>, longstr, <<"all-with-x">>},
{<<"k1">>, longstr, <<"v1">>},
{<<"k2">>, longstr, <<"v2">>},
{<<"x-k3">>, longstr, <<"🐇"/utf8>>}]
}),
C = connect(?FUNCTION_NAME, Config),
Topic = <<"my/topic">>,
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k1">>, <<"v1">>},
{<<"k2">>, <<"v2">>},
{<<"x-k3">>, unicode:characters_to_binary("🐇")}
]},
<<"m1">>, [{qos, 1}]),
[?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m1">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
ok = emqtt:publish(C, Topic, <<"m2">>),
[?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k1">>, <<"nope">>}]},
<<"m3">>, [{qos, 1}]),
[?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k2">>, <<"v2">>}]},
<<"m4">>, [{qos, 1}]),
?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m4">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q1})),
?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q2})),
ok = emqtt:disconnect(C),
[#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}) || Q <- Qs],
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
%% Binding a consistent hash exchange to the MQTT topic exchange should support
%% consistent routing based on Correlation-Data in the PUBLISH packet.
consistent_hash_exchange(Config) ->
ok = rabbit_ct_broker_helpers:enable_plugin(Config, 0, rabbitmq_consistent_hash_exchange),
HashX = <<"my-consistent-hash-exchange">>,
Q1 = <<"q1">>,
Q2 = <<"q2">>,
Qs = [Q1, Q2],
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'exchange.declare_ok'{} = amqp_channel:call(
Ch, #'exchange.declare'{
exchange = HashX,
type = <<"x-consistent-hash">>,
arguments = [{<<"hash-property">>, longstr, <<"correlation_id">>}],
durable = true,
auto_delete = true}),
#'exchange.bind_ok'{} = amqp_channel:call(
Ch, #'exchange.bind'{destination = HashX,
source = <<"amq.topic">>,
routing_key = <<"a.*">>}),
[#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{queue = Q,
durable = true}) || Q <- Qs],
[#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q,
exchange = HashX,
%% weight
routing_key = <<"1">>}) || Q <- Qs],
Rands = [integer_to_binary(rand:uniform(1000)) || _ <- lists:seq(1, 30)],
UniqRands = lists:uniq(Rands),
NumMsgs = 150,
C = connect(?FUNCTION_NAME, Config),
[begin
N = integer_to_binary(rand:uniform(1_000_000)),
Topic = <<"a/", N/binary>>,
{ok, _} = emqtt:publish(C, Topic,
#{'Correlation-Data' => lists:nth(rand:uniform(length(UniqRands)), UniqRands)},
N, [{qos, 1}])
end || _ <- lists:seq(1, NumMsgs)],
#'basic.consume_ok'{consumer_tag = Ctag1} = amqp_channel:subscribe(
Ch, #'basic.consume'{queue = Q1,
no_ack = true}, self()),
#'basic.consume_ok'{consumer_tag = Ctag2} = amqp_channel:subscribe(
Ch, #'basic.consume'{queue = Q2,
no_ack = true}, self()),
{N1, Corrs1} = receive_correlations(Ctag1, 0, sets:new([{version, 2}])),
{N2, Corrs2} = receive_correlations(Ctag2, 0, sets:new([{version, 2}])),
ct:pal("q1: ~b messages, ~b unique correlation-data", [N1, sets:size(Corrs1)]),
ct:pal("q2: ~b messages, ~b unique correlation-data", [N2, sets:size(Corrs2)]),
%% All messages should be routed.
?assertEqual(NumMsgs, N1 + N2),
%% Each of the 2 queues should have received at least 1 message.
?assert(sets:size(Corrs1) > 0),
?assert(sets:size(Corrs2) > 0),
%% Assert that the consistent hash exchange routed the given Correlation-Data consistently.
%% The same Correlation-Data should never be present in both queues.
Intersection = sets:intersection(Corrs1, Corrs2),
?assert(sets:is_empty(Intersection)),
ok = emqtt:disconnect(C),
[#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}) || Q <- Qs],
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
%% -------------------------------------------------------------------
%% Helpers
%% -------------------------------------------------------------------
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
receive_correlations(Ctag, N, Set) ->
receive {#'basic.deliver'{consumer_tag = Ctag},
#amqp_msg{props = #'P_basic'{correlation_id = Corr}}} ->
?assert(is_binary(Corr)),
receive_correlations(Ctag, N + 1, sets:add_element(Corr, Set))
Tests: Increase receive-after timeout in all mqtt test suites
2024-12-10 23:19:34 +08:00
after 1000 ->
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
{N, Set}
end.
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
assert_no_queue_ttl(NumQs, Config) ->
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
Qs = rpc(Config, rabbit_amqqueue, list, []),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
?assertEqual(NumQs, length(Qs)),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
?assertNot(lists:any(fun(Q) ->
proplists:is_defined(?QUEUE_TTL_KEY, amqqueue:get_arguments(Q))
end, Qs)).
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
assert_queue_ttl(TTLSecs, NumQs, Config) ->
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
Qs = rpc(Config, rabbit_amqqueue, list, []),
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
?assertEqual(NumQs, length(Qs)),
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
?assert(lists:all(fun(Q) ->
Add a test for Session Expiry Interval
2023-03-22 23:49:29 +08:00
{long, timer:seconds(TTLSecs)} =:= rabbit_misc:table_lookup(
amqqueue:get_arguments(Q), ?QUEUE_TTL_KEY)
Update sess exp interval when client reconnect - when client reconnecting with clean start false, server respects the new session expiry interval provided by the client
2023-03-22 19:54:22 +08:00
end, Qs)).
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
dead_letter_metric(Metric, Config) ->
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
dead_letter_metric(Metric, Config, disabled).
dead_letter_metric(Metric, Config, Strategy) ->
Support Session Expiry Interval Allow Session Expiry Interval to be changed when client DISCONNECTs. Deprecate config subscription_ttl in favour of max_session_expiry_interval_secs because the Session Expiry Interval will also apply to publishers that connect with a will message and will delay interval. "The additional session state of an MQTT v5 server includes: * The Will Message and the Will Delay Interval * If the Session is currently not connected, the time at which the Session will end and Session State will be discarded." The Session Expiry Interval picked by the server and sent to the client in the CONNACK is the minimum of max_session_expiry_interval_secs and the requested Session Expiry Interval by the client in CONNECT. This commit favours dynamically changing the queue argument x-expires over creating millions of different policies since that many policies will come with new scalability issues. Dynamically changing queue arguments is not allowed by AMQP 0.9.1 clients. However, it should be perfectly okay for the MQTT plugin to do so for the queues it manages. MQTT clients are not aware that these queues exist.
2023-03-17 20:48:26 +08:00
Counters = rpc(Config, rabbit_global_counters, overview, []),
Support MQTT 5.0 features No Local, RAP, Subscription IDs Support subscription options "No Local" and "Retain As Published" as well as Subscription Identifiers. All three MQTT 5.0 features can be set on a per subscription basis. Due to wildcards in topic filters, multiple subscriptions can match a given topic. Therefore, to implement Retain As Published and Subscription Identifiers, the destination MQTT connection process needs to know what subscription(s) caused it to receive the message. There are a few ways how this could be implemented: 1. The destination MQTT connection process is aware of all its subscriptions. Whenever, it receives a message, it can match the message's routing key / topic against all its known topic filters. However, to iteratively match the routing key against all topic filters for every received message can become very expensive in the worst case when the MQTT client creates many subscriptions containing wildcards. This could be the case for an MQTT client that acts as a bridge or proxy or dispatcher: It could subscribe via a wildcard for each of its own clients. 2. Instead of interatively matching the topic of the received message against all topic filters that contain wildcards, a better approach would be for every MQTT subscriber connection process to maintain a local trie datastructure (similar to how topic exchanges are implemented) and perform matching therefore more efficiently. However, this does not sound optimal either because routing is effectively performed twice: in the topic exchange and again against a much smaller trie in each destination connection process. 3. Given that the topic exchange already perform routing, a much more sensible way would be to send the matched binding key(s) to the destination MQTT connection process. A subscription (topic filter) maps to a binding key in AMQP 0.9.1 routing. Therefore, for the first time in RabbitMQ, the routing function should not only output a list of unique destination queues, but also the binding keys (subscriptions) that caused the message to be routed to the destination queue. This commit therefore implements the 3rd approach. The downside of the 3rd approach is that it requires API changes to the routing function and topic exchange. Specifically, this commit adds a new function rabbit_exchange:route/3 that accepts a list of routing options. If that list contains version 2, the caller of the routing function knows how to handle the return value that could also contain binding keys. This commits allows an MQTT connection process, the channel process, and at-most-once dead lettering to handle binding keys. Binding keys are included as AMQP 0.9.1 headers into the basic message. Therefore, whenever a message is sent from an MQTT client or AMQP 0.9.1 client or AMQP 1.0 client or STOMP client, the MQTT receiver will know the subscription identifier that caused the message to be received. Note that due to the low number of allowed wildcard characters (# and +), the cardinality of matched binding keys shouldn't be high even if the topic contains for example 3 levels and the message is sent to for example 5 million destination queues. In other words, sending multiple distinct basic messages to the destination shouldn't hurt the delegate optimisation too much. The delegate optimisation implemented for classic queues and rabbit_mqtt_qos0_queue(s) still takes place for all basic messages that contain the same set of matched binding keys. The topic exchange returns all matched binding keys by remembering the edges walked down to the leaves. As an optimisation, only for MQTT queues are binding keys being returned. This does add a small dependency from app rabbit to app rabbitmq_mqtt which is not optimal. However, this dependency should be simple to remove when omitting this optimisation. Another important feature of this commit is persisting subscription options and subscription identifiers because they are part of the MQTT 5.0 session state. In MQTT v3 and v4, the only subscription information that were part of the session state was the topic filter and the QoS level. Both information were implicitly stored in the form of bindings: The topic filter as the binding key and the QoS level as the destination queue name of the binding. For MQTT v5 we need to persist more subscription information. From a domain perspective, it makes sense to store subscription options as part of subscriptions, i.e. bindings, even though they are currently not used in routing. Therefore, this commits stores subscription options as binding arguments. Storing subscription options as binding arguments comes in turn with new challenges: How to handle mixed version clusters and upgrading an MQTT session from v3 or v4 to v5? Imagine an MQTT client connects via v5 with Session Expiry Interval > 0 to a new node in a mixed version cluster, creates a subscription, disconnects, and subsequently connects via v3 to an old node. The client should continue to receive messages. To simplify such edge cases, this commit introduces a new feature flag called mqtt_v5. If mqtt_v5 is disabled, clients cannot connect to RabbitMQ via MQTT 5.0. This still doesn't entirely solve the problem of MQTT session upgrades (v4 to v5 client) or session downgrades (v5 to v4 client). Ideally, once mqtt_v5 is enabled, all MQTT bindings contain non-empty binding arguments. However, this will require a feature flag migration function to modify all MQTT bindings. To be more precise, all MQTT bindings need to be deleted and added because the binding argument is part of the Mnesia table key. Since feature flag migration functions are non-trivial to implement in RabbitMQ (they can run on every node multiple times and concurrently), this commit takes a simpler approach: All v3 / v4 sessions keep the empty binding argument []. All v5 sessions use the new binding argument [#mqtt_subscription_opts{}]. This requires only handling a session upgrade / downgrade by creating a binding (with the new binding arg) and deleting the old binding (with the old binding arg) when processing the CONNECT packet. Note that such session upgrades or downgrades should be rather rare in practice. Therefore these binding transactions shouldn't hurt peformance. The No Local option is implemented within the MQTT publishing connection process: The message is not sent to the MQTT destination if the destination queue name matches the current MQTT client ID and the message was routed due to a subscription that has the No Local flag set. This avoids unnecessary traffic on the MQTT queue. The alternative would have been that the "receiving side" (same process) filters the message out - which would have been more consistent in how Retain As Published and Subscription Identifiers are implemented, but would have caused unnecessary load on the MQTT queue.
2023-04-19 21:32:34 +08:00
Map = maps:get([{queue_type, rabbit_classic_queue}, {dead_letter_strategy, Strategy}], Counters),
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
maps:get(Metric, Map).
assert_nothing_received() ->
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
assert_nothing_received(500).
assert_nothing_received(Timeout) ->
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
receive Unexpected -> ct:fail("Received unexpected message: ~p", [Unexpected])
Support Will Delay Interval Previously, the Will Message could be kept in memory in the MQTT connection process state. Upon termination, the Will Message is sent. The new MQTT 5.0 feature Will Delay Interval requires storing the Will Message outside of the MQTT connection process state. The Will Message should not be stored node local because the client could reconnect to a different node. Storing the Will Message in Mnesia is not an option because we want to get rid of Mnesia. Storing the Will Message in a Ra cluster or in Khepri is only an option if the Will Payload is small as there is currently no way in Ra to **efficiently** snapshot large binary data (Note that these Will Messages are not consumed in a FIFO style workload like messages in quorum queues. A Will Message needs to be stored for as long as the Session lasts - up to 1 day by default, but could also be much longer if RabbitMQ is configured with a higher maximum session expiry interval.) Usually Will Payloads are small: They are just a notification that its MQTT session ended abnormally. However, we don't know how users leverage the Will Message feature. The MQTT protocol allows for large Will Payloads. Therefore, the solution implemented in this commit - which should work good enough - is storing the Will Message in a queue. Each MQTT session which has a Session Expiry Interval and Will Delay Interval of > 0 seconds will create a queue if the current Network Connection ends where it stores its Will Message. The Will Message has a message TTL set (corresponds to the Will Delay Interval) and the queue has a queue TTL set (corresponds to the Session Expiry Interval). If the client does not reconnect within the Will Delay Interval, the message is dead lettered to the configured MQTT topic exchange (amq.topic by default). The Will Delay Interval can be set by both publishers and subscribers. Therefore, the Will Message is the 1st session state that RabbitMQ keeps for publish-only MQTT clients. One current limitation of this commit is that a Will Message that is delayed (i.e. Will Delay Interval is set) and retained (i.e. Will Retain flag set) will not be retained. One solution to retain delayed Will Messages is that the retainer process consumes from a queue and the queue binds to the topic exchange with a topic starting with `$`, for example `$retain/#`. The AMQP 0.9.1 Will Message that is dead lettered could then be added a CC header such that it won't not only be published with the Will Topic, but also with `$retain` topic. For example, if the Will Topic is `a/b`, it will publish with routing key `a/b` and CC header `$retain/a/b`. The reason this is not implemented in this commit is that to keep the currently broken retained message store behaviour, we would require creating at least one queue per node and publishing only to that local queue. In future, once we have a replicated retained message store based on a Stream for example, we could just publish all retained messages to the `$retain` topic and thefore into the Stream. So, for now, we list "retained and delayed Will Messages" as a limitation that they actually won't be retained.
2023-05-18 23:36:25 +08:00
after Timeout -> ok
Add MQTT v5 feature Message Expiry Interval This commit does not yet implement Message Expiry Interval of * retained messages: "If the current retained message for a Topic expires, it is discarded and there will be no retained message for that topic."
2023-03-02 00:44:56 +08:00
end.