rabbitmq-server/deps/amqp10_common/test/binary_parser_SUITE.erl

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

88 lines
2.8 KiB
Erlang
Raw Permalink Normal View History

-module(binary_parser_SUITE).
-compile(export_all).
-export([
]).
-include_lib("eunit/include/eunit.hrl").
%%%===================================================================
%%% Common Test callbacks
%%%===================================================================
all() ->
[
{group, tests}
].
all_tests() ->
[
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
roundtrip,
array_with_extra_input,
unsupported_type
].
groups() ->
[
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{tests, [parallel], all_tests()}
].
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
roundtrip(_Config) ->
Terms = [
null,
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{described,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
{symbol, <<"URL">>},
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{utf8, <<"http://example.org/hello-world">>}},
{described,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
{symbol, <<"URL">>},
{binary, <<"https://rabbitmq.com">>}},
%% "The descriptor portion of a described format code is itself
%% any valid AMQP encoded value, including other described values." [1.2]
{described,
{described,
{symbol, <<"inner constructor">>},
{binary, <<"inner value">>}},
{binary, <<"outer value">>}},
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{array, ubyte, [{ubyte, 1}, {ubyte, 255}]},
true,
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{list, [{utf8, <<"hi">>},
{described,
Message Containers (#5077) This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception. Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol. This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point. The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed. This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt. COMMIT HISTORY: * Refactor away from using the delivery{} record In many places including exchange types. This should make it easier to move towards using a message container type instead of basic_message. Add mc module and move direct replies outside of exchange Lots of changes incl classic queues Implement stream support incl amqp conversions simplify mc state record move mc.erl mc dlx stuff recent history exchange Make tracking work But doesn't take a protocol agnostic approach as we just convert everything into AMQP legacy and back. Might be good enough for now. Tracing as a whole may want a bit of a re-vamp at some point. tidy make quorum queue peek work by legacy conversion dead lettering fixes dead lettering fixes CMQ fixes rabbit_trace type fixes fixes fix Fix classic queue props test assertion fix feature flag and backwards compat Enable message_container feature flag in some SUITEs Dialyzer fixes fixes fix test fixes Various Manually update a gazelle generated file until a gazelle enhancement can be made https://github.com/rabbitmq/rules_erlang/issues/185 Add message_containers_SUITE to bazel and regen bazel files with gazelle from rules_erlang@main Simplify essential proprty access Such as durable, ttl and priority by extracting them into annotations at message container init time. Move type to remove dependenc on amqp10 stuff in mc.erl mostly because I don't know how to make bazel do the right thing add more stuff Refine routing header stuff wip Cosmetics Do not use "maybe" as type name as "maybe" is a keyword since OTP 25 which makes Erlang LS complain. * Dedup death queue names * Fix function clause crashes Fix failing tests in the MQTT shared_SUITE: A classic queue message ID can be undefined as set in https://github.com/rabbitmq/rabbitmq-server/blob/fbe79ff47b4edbc0fd95457e623d6593161ad198/deps/rabbit/src/rabbit_classic_queue_index_v2.erl#L1048 Fix failing tests in the MQTT shared_SUITE-mixed: When feature flag message_containers is disabled, the message is not an #mc{} record, but a #basic_message{} record. * Fix is_utf8_no_null crash Prior to this commit, the function crashed if invalid UTF-8 was provided, e.g.: ``` 1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>). ** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481) ``` * Implement mqtt mc behaviour For now via amqp translation. This is still work in progress, but the following SUITEs pass: ``` make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1 make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1 ``` * Shorten mc file names Module name length matters because for each persistent message the #mc{} record is persisted to disk. ``` 1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})). 30 2> iolist_size(term_to_iovec({mc, mc_amqpl})). 17 ``` This commit renames the mc modules: ``` ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g' ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g' ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g' ``` * mc: make deaths an annotation + fixes * Fix mc_mqtt protocol_state callback * Fix test will_delay_node_restart ``` make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1 ``` * Bazel run gazelle * mix format rabbitmqctl.ex * Ensure ttl annotation is refelected in amqp legacy protocol state * Fix id access in message store * Fix rabbit_message_interceptor_SUITE * dializer fixes * Fix rabbit:rabbit_message_interceptor_SUITE-mixed set_annotation/3 should not result in duplicate keys * Fix MQTT shared_SUITE-mixed Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode. https://github.com/rabbitmq/rabbitmq-server/blob/75a953ce286a10aca910c098805a4f545989af38/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L2075-L2076 From now on, non-MQTT publishes are QoS 1 if durable. This makes more sense. The MQTT plugin must send a #basic_message{} to an old node that does not understand message containers. * Field content of 'v1_0.data' can be binary Fix ``` bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \ --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \ -t- --test_sharding_strategy=disabled ``` * Remove route/2 and implement route/3 for all exchange types. This removes the route/2 callback from rabbit_exchange_type and makes route/3 mandatory instead. This is a breaking change and will require all implementations of exchange types to update their code, however this is necessary anyway for them to correctly handle the mc type. stream filtering fixes * Translate directly from MQTT to AMQP 0.9.1 * handle undecoded properties in mc_compat amqpl: put clause in right order recover death deatails from amqp data * Replace callback init_amqp with convert_from * Fix return value of lists:keyfind/3 * Translate directly from AMQP 0.9.1 to MQTT * Fix MQTT payload size MQTT payload can be a list when converted from AMQP 0.9.1 for example First conversions tests Plus some other conversion related fixes. bazel bazel translate amqp 1.0 null to undefined mc: property/2 and correlation_id/message_id return type tagged values. To ensure we can support a variety of types better. The type type tags are AMQP 1.0 flavoured. fix death recovery mc_mqtt: impl new api Add callbacks to allow protocols to compact data before storage And make readable if needing to query things repeatedly. bazel fix * more decoding * tracking mixed versions compat * mc: flip default of `durable` annotation to save some data. Assuming most messages are durable and that in memory messages suffer less from persistence overhead it makes sense for a non existent `durable` annotation to mean durable=true. * mc conversion tests and tidy up * mc make x_header unstrict again * amqpl: death record fixes * bazel * amqp -> amqpl conversion test * Fix crash in mc_amqp:size/1 Body can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0/ ct-system t=java ``` on branch native-amqp. * Fix crash in lists:flatten/1 Data can be a single amqp-value section (instead of being a list) as shown by test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp. * Fix crash in rabbit_writer Running test ``` make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091 ``` on branch native-amqp resulted in the following crash: ``` crasher: initial call: rabbit_writer:enter_mainloop/2 pid: <0.711.0> registered_name: [] exception error: bad argument in function size/1 called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]]) *** argument 1: not tuple or binary in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89) in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61) in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334) in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365) in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265) in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232) in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223) ``` because #content.payload_fragments_rev is currently supposed to be a flat list of binaries instead of being an iolist. This commit fixes this crash inefficiently by calling iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev to be an iolist. * Add accidentally deleted line back * mc: optimise mc_amqp internal format By removint the outer records for message and delivery annotations as well as application properties and footers. * mc: optimis mc_amqp map_add by using upsert * mc: refactoring and bug fixes * mc_SUITE routingheader assertions * mc remove serialize/1 callback as only used by amqp * mc_amqp: avoid returning a nested list from protocol_state * test and bug fix * move infer_type to mc_util * mc fixes and additiona assertions * Support headers exchange routing for MQTT messages When a headers exchange is bound to the MQTT topic exchange, routing will be performend based on both MQTT topic (by the topic exchange) and MQTT User Property (by the headers exchange). This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and enables powerful routing topologies. When the User Property contains the same name multiple times, only the last name (and value) will be considered by the headers exchange. * Fix crash when sending from stream to amqpl When publishing a message via the stream protocol and consuming it via AMQP 0.9.1, the following crash occurred prior to this commit: ``` crasher: initial call: rabbit_channel:init/1 pid: <0.818.0> registered_name: [] exception exit: {{badmatch,undefined}, [{rabbit_channel,handle_deliver0,4, [{file,"rabbit_channel.erl"}, {line,2728}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}, {rabbit_channel,handle_cast,2, [{file,"rabbit_channel.erl"}, {line,728}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,wake_up,3, [{file,"proc_lib.erl"},{line,251}]}]} ``` This commit first gives `mc:init/3` the chance to set exchange and routing_keys annotations. If not set, `rabbit_stream_queue` will set these annotations assuming the message was originally published via the stream protocol. * Support consistent hash exchange routing for MQTT 5.0 When a consistent hash exchange is bound to the MQTT topic exchange, MQTT 5.0 messages can be routed to queues consistently based on the Correlation-Data in the PUBLISH packet. * Convert MQTT 5.0 User Property * to AMQP 0.9.1 headers * from AMQP 0.9.1 headers * to AMQP 1.0 application properties and message annotations * from AMQP 1.0 application properties and message annotations * Make use of Annotations in mc_mqtt:protocol_state/2 mc_mqtt:protocol_state/2 includes Annotations as parameter. It's cleaner to make use of these Annotations when computing the protocol state instead of relying on the caller (rabbitmq_mqtt_processor) to compute the protocol state. * Enforce AMQP 0.9.1 field name length limit The AMQP 0.9.1 spec prohibits field names longer than 128 characters. Therefore, when converting AMQP 1.0 message annotations, application properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any names longer than 128 characters. * Fix type specs Apply feedback from Michael Davis Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * Add mc_mqtt unit test suite Implement mc_mqtt:x_header/2 * Translate indicator that payload is UTF-8 encoded when converting between MQTT 5.0 and AMQP 1.0 * Translate single amqp-value section from AMQP 1.0 to MQTT Convert to a text representation, if possible, and indicate to MQTT client that the payload is UTF-8 encoded. This way, the MQTT client will be able to parse the payload. If conversion to text representation is not possible, encode the payload using the AMQP 1.0 type system and indiate the encoding via Content-Type message/vnd.rabbitmq.amqp. This Content-Type is not registered. Type "message" makes sense since it's a message. Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not registered. * Fix payload conversion * Translate Response Topic between MQTT and AMQP Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice versa. The Response Topic must be a UTF-8 encoded string. This commit re-uses the already defined RabbitMQ target addresses: ``` "/topic/" RK Publish to amq.topic with routing key RK "/exchange/" X "/" RK Publish to exchange X with routing key RK ``` By default, the MQTT topic exchange is configure dto be amq.topic using the 1st target address. When an operator modifies the mqtt.exchange, the 2nd target address is used. * Apply PR feedback and fix formatting Co-authored-by: Michael Davis <mcarsondavis@gmail.com> * tidy up * Add MQTT message_containers test * consistent hash exchange: avoid amqp legacy conversion When hashing on a header value. * Avoid converting to amqp legacy when using exchange federation * Fix test flake * test and dialyzer fixes * dialyzer fix * Add MQTT protocol interoperability tests Test receiving from and sending to MQTT 5.0 and * AMQP 0.9.1 * AMQP 1.0 * STOMP * Streams * Regenerate portions of deps/rabbit/app.bzl with gazelle I'm not exactly sure how this happened, but gazell seems to have been run with an older version of the rules_erlang gazelle extension at some point. This caused generation of a structure that is no longer used. This commit updates the structure to the current pattern. * mc: refactoring * mc_amqpl: handle delivery annotations Just in case they are included. Also use iolist_to_iovec to create flat list of binaries when converting from amqp with amqp encoded payload. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
2023-08-31 18:27:13 +08:00
{symbol, <<"URL">>},
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
{utf8, <<"http://example.org/hello-world">>}}
]},
{list, [{int, 123},
{array, int, [{int, 1}, {int, 2}, {int, 3}]}
]},
{map, [
{{utf8, <<"key1">>}, {utf8, <<"value1">>}},
{{utf8, <<"key2">>}, {int, 33}}
]},
{array, {described, {utf8, <<"URL">>}, utf8}, []},
false
],
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
Bin = lists:foldl(
fun(T, Acc) ->
B = iolist_to_binary(amqp10_binary_generator:generate(T)),
<<Acc/binary, B/binary>>
end, <<>>, Terms),
2024-04-14 21:04:38 +08:00
?assertEqual(Terms, amqp10_binary_parser:parse_many(Bin, [])).
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
array_with_extra_input(_Config) ->
2021-04-07 16:34:27 +08:00
Bin = <<83,16,192,85,10,177,0,0,0,1,48,161,12,114,97,98,98,105,116, 109,113,45,98,111,120,112,255,255,0,0,96,0,50,112,0,0,19,136,163,5,101,110,45,85,83,224,14,2,65,5,102,105,45,70,73,5,101,110,45,85,83,64,64,193,24,2,163,20,68,69,70,69,78,83,73,67,83,46,84,69,83,84,46,83,85,73,84,69,65>>,
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
Expected = {failed_to_parse_array_extra_input_remaining,
%% element type, input, accumulated result
65, <<105,45,70,73,5,101,110,45,85,83>>, [true,true]},
2024-04-14 21:04:38 +08:00
?assertExit(Expected, amqp10_binary_parser:parse_many(Bin, [])).
Increase receiving throughput from a stream via AMQP This commit increases consumption throughput from a stream via AMQP 0.9.1 for 1 consumer by 83k msg/s or 55%, for 4 consumers by 140k msg/s or 44%. This commit tries to follow https://www.erlang.org/doc/efficiency_guide/binaryhandling.html by reusing match contexts instead of creating new sub-binaries. The CPU and mmap() memory flame graphs show that when producing and consuming from a stream via AMQP 0.9.1 module amqp10_binary_parser requires before this commit: 10.1% CPU time and 8.0% of mmap system calls after this commit: 2.6% CPU time 2.5% of mmap system calls Performance tests Start rabbitmq-server without any plugins enabled and with 4 schedulers: ``` make run-broker PLUGINS="" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4" ``` Test 1 Perf test client: ``` -x 1 -y 2 -qa x-queue-type=stream -ad false -f persistent -u s1 --qos 10000 --multi-ack-every 1000 -z 30 ``` master branch: sending rate avg msg/s 143k - 146k receiving rate avg msg/s 188k - 194k PR: sending rate avg 133k - 138k receiving rate avg 266k - 276k This shows that with AMQP 0.9.1 and a stream, prior to this commit the broker could not deliver messages to consumers as fast as they were published. After this commit, it can. Test 2 First, produce a few millions messages: ``` -x 1 -y 0 -qa x-queue-type=stream -ad false -f persistent -u s2 ``` Then, consume them: ``` -x 0 -y 1 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 ``` receving rate avg msg/s master branch: 147k - 156k PR: 230k - 237k Improvement: 83k / 55% Test 3 -x 0 -y 4 -qa x-queue-type=stream -ad false -f persistent -u s2 --qos 10000 --multi-ack-every 1000 -ca x-stream-offset=first -z 30 receving rate avg msg/s master branch: 313k - 319k PR: 450k - 461k Improvement: 140k / 44%
2022-05-16 16:46:33 +08:00
unsupported_type(_Config) ->
2024-04-14 21:04:38 +08:00
UnsupportedType = 16#02,
Bin = <<UnsupportedType, "hey">>,
2024-04-16 22:08:27 +08:00
Expected = {primitive_type_unsupported, UnsupportedType, {position, 0}},
?assertThrow(Expected, amqp10_binary_parser:parse_many(Bin, [])).