root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 6 Packages Projects Releases Wiki Activity

Message Containers (#5077) 

This PR implements an approach for a "protocol (data format) agnostic core" where the format of the message isn't converted at point of reception.

Currently all non AMQP 0.9.1 originating messages are converted into a AMQP 0.9.1 flavoured basic_message record before sent to a queue. If the messages are then consumed by the originating protocol they are converted back from AMQP 0.9.1. For some protocols such as MQTT 3.1 this isn't too expensive as MQTT is mostly a fairly easily mapped subset of AMQP 0.9.1 but for others such as AMQP 1.0 the conversions are awkward and in some cases lossy even if consuming from the originating protocol.

This PR instead wraps all incoming messages in their originating form into a generic, extensible message container type (mc). The container module exposes an API to get common message details such as size and various properties (ttl, priority etc) directly from the source data type. Each protocol needs to implement the mc behaviour such that when a message originating form one protocol is consumed by another protocol we convert it to the target protocol at that point.

The message container also contains annotations, dead letter records and other meta data we need to record during the lifetime of a message. The original protocol message is never modified unless it is consumed.

This includes conversion modules to and from amqp, amqpl (AMQP 0.9.1) and mqtt.


COMMIT HISTORY:

* Refactor away from using the delivery{} record

In many places including exchange types. This should make it
easier to move towards using a message container type instead of
basic_message.

Add mc module and move direct replies outside of exchange

Lots of changes incl classic queues

Implement stream support incl amqp conversions

simplify mc state record

move mc.erl

mc dlx stuff

recent history exchange

Make tracking work

But doesn't take a protocol agnostic approach as we just convert
everything into AMQP legacy and back. Might be good enough for now.

Tracing as a whole may want a bit of a re-vamp at some point.

tidy

make quorum queue peek work by legacy conversion

dead lettering fixes

dead lettering fixes

CMQ fixes

rabbit_trace type fixes

fixes

fix

Fix classic queue props

test assertion fix

feature flag and backwards compat

Enable message_container feature flag in some SUITEs

Dialyzer fixes

fixes

fix

test fixes

Various

Manually update a gazelle generated file

until a gazelle enhancement can be made
https://github.com/rabbitmq/rules_erlang/issues/185

Add message_containers_SUITE to bazel

and regen bazel files with gazelle from rules_erlang@main

Simplify essential proprty access

Such as durable, ttl and priority by extracting them into annotations
at message container init time.

Move type

to remove dependenc on amqp10 stuff in mc.erl

mostly because I don't know how to make bazel do the right thing

add more stuff

Refine routing header stuff

wip

Cosmetics

Do not use "maybe" as type name as "maybe" is a keyword since OTP 25
which makes Erlang LS complain.

* Dedup death queue names

* Fix function clause crashes

Fix failing tests in the MQTT shared_SUITE:
A classic queue message ID can be undefined as set in
fbe79ff47b/deps/rabbit/src/rabbit_classic_queue_index_v2.erl (L1048)

Fix failing tests in the MQTT shared_SUITE-mixed:
When feature flag message_containers is disabled, the
message is not an #mc{} record, but a #basic_message{} record.

* Fix is_utf8_no_null crash

Prior to this commit, the function crashed if invalid UTF-8 was
provided, e.g.:
```
1> rabbit_misc:is_valid_shortstr(<<"😇"/utf16>>).
** exception error: no function clause matching rabbit_misc:is_utf8_no_null(<<216,61,222,7>>) (rabbit_misc.erl, line 1481)
```

* Implement mqtt mc behaviour

For now via amqp translation.

This is still work in progress, but the following SUITEs pass:
```
make -C deps/rabbitmq_mqtt ct-shared t=[mqtt,v5,cluster_size_1] FULL=1
make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_1] FULL=1
```

* Shorten mc file names

Module name length matters because for each persistent message the #mc{}
record is persisted to disk.

```
1> iolist_size(term_to_iovec({mc, rabbit_mc_amqp_legacy})).
30
2> iolist_size(term_to_iovec({mc, mc_amqpl})).
17
```

This commit renames the mc modules:
```
ag -l rabbit_mc_amqp_legacy | xargs sed -i 's/rabbit_mc_amqp_legacy/mc_amqpl/g'
ag -l rabbit_mc_amqp | xargs sed -i 's/rabbit_mc_amqp/mc_amqp/g'
ag -l rabbit_mqtt_mc | xargs sed -i 's/rabbit_mqtt_mc/mc_mqtt/g'
```

* mc: make deaths an annotation + fixes

* Fix mc_mqtt protocol_state callback

* Fix test will_delay_node_restart

```
make -C deps/rabbitmq_mqtt ct-v5 t=[mqtt,cluster_size_3]:will_delay_node_restart FULL=1
```

* Bazel run gazelle

* mix format rabbitmqctl.ex

* Ensure ttl annotation is refelected in amqp legacy protocol state

* Fix id access in message store

* Fix rabbit_message_interceptor_SUITE

* dializer fixes

* Fix rabbit:rabbit_message_interceptor_SUITE-mixed

set_annotation/3 should not result in duplicate keys

* Fix MQTT shared_SUITE-mixed

Up to 3.12 non-MQTT publishes were always QoS 1 regardless of delivery_mode.
75a953ce28/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl (L2075-L2076)
From now on, non-MQTT publishes are QoS 1 if durable.
This makes more sense.

The MQTT plugin must send a #basic_message{} to an old node that does
not understand message containers.

* Field content of 'v1_0.data' can be binary

Fix
```
bazel test //deps/rabbitmq_mqtt:shared_SUITE-mixed \
    --test_env FOCUS="-group [mqtt,v4,cluster_size_1] -case trace" \
    -t- --test_sharding_strategy=disabled
```

* Remove route/2 and implement route/3 for all exchange types.

This removes the route/2 callback from rabbit_exchange_type and
makes route/3 mandatory instead. This is a breaking change and
will require all implementations of exchange types to update their
code, however this is necessary anyway for them to correctly handle
the mc type.

stream filtering fixes

* Translate directly from MQTT to AMQP 0.9.1

* handle undecoded properties in mc_compat

amqpl: put clause in right order

recover death deatails from amqp data

* Replace callback init_amqp with convert_from

* Fix return value of lists:keyfind/3

* Translate directly from AMQP 0.9.1 to MQTT

* Fix MQTT payload size

MQTT payload can be a list when converted from AMQP 0.9.1 for example

First conversions tests

Plus some other conversion related fixes.

bazel

bazel

translate amqp 1.0 null to undefined

mc: property/2 and correlation_id/message_id return type tagged values.

To ensure we can support a variety of types better.

The type type tags are AMQP 1.0 flavoured.

fix death recovery

mc_mqtt: impl new api

Add callbacks to allow protocols to compact data before storage

And make readable if needing to query things repeatedly.

bazel fix

* more decoding

* tracking mixed versions compat

* mc: flip default of `durable` annotation to save some data.

Assuming most messages are durable and that in memory messages suffer less
from persistence overhead it makes sense for a non existent `durable`
annotation to mean durable=true.

* mc conversion tests and tidy up

* mc make x_header unstrict again

* amqpl: death record fixes

* bazel

* amqp -> amqpl conversion test

* Fix crash in mc_amqp:size/1

Body can be a single amqp-value section (instead of
being a list) as shown by test
```
make -C deps/rabbitmq_amqp1_0/ ct-system t=java
```
on branch native-amqp.

* Fix crash in lists:flatten/1

Data can be a single amqp-value section (instead of
being a list) as shown by test
```
make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091
```
on branch native-amqp.

* Fix crash in rabbit_writer

Running test
```
make -C deps/rabbitmq_amqp1_0 ct-system t=dotnet:roundtrip_to_amqp_091
```
on branch native-amqp resulted in the following crash:
```
crasher:
  initial call: rabbit_writer:enter_mainloop/2
  pid: <0.711.0>
  registered_name: []
  exception error: bad argument
    in function  size/1
       called as size([<<0>>,<<"Sw">>,[<<160,2>>,<<"hi">>]])
       *** argument 1: not tuple or binary
    in call from rabbit_binary_generator:build_content_frames/7 (rabbit_binary_generator.erl, line 89)
    in call from rabbit_binary_generator:build_simple_content_frames/4 (rabbit_binary_generator.erl, line 61)
    in call from rabbit_writer:assemble_frames/5 (rabbit_writer.erl, line 334)
    in call from rabbit_writer:internal_send_command_async/3 (rabbit_writer.erl, line 365)
    in call from rabbit_writer:handle_message/2 (rabbit_writer.erl, line 265)
    in call from rabbit_writer:handle_message/3 (rabbit_writer.erl, line 232)
    in call from rabbit_writer:mainloop1/2 (rabbit_writer.erl, line 223)
```
because #content.payload_fragments_rev is currently supposed to
be a flat list of binaries instead of being an iolist.

This commit fixes this crash inefficiently by calling
iolist_to_binary/1. A better solution would be to allow AMQP legacy's #content.payload_fragments_rev
to be an iolist.

* Add accidentally deleted line back

* mc: optimise mc_amqp internal format

By removint the outer records for message and delivery annotations
as well as application properties and footers.

* mc: optimis mc_amqp map_add by using upsert

* mc: refactoring and bug fixes

* mc_SUITE routingheader assertions

* mc remove serialize/1 callback as only used by amqp

* mc_amqp: avoid returning a nested list from protocol_state

* test and bug fix

* move infer_type to mc_util

* mc fixes and additiona assertions

* Support headers exchange routing for MQTT messages

When a headers exchange is bound to the MQTT topic exchange, routing
will be performend based on both MQTT topic (by the topic exchange) and
MQTT User Property (by the headers exchange).

This combines the best worlds of both MQTT 5.0 and AMQP 0.9.1 and
enables powerful routing topologies.

When the User Property contains the same name multiple times, only the
last name (and value) will be considered by the headers exchange.

* Fix crash when sending from stream to amqpl

When publishing a message via the stream protocol and consuming it via
AMQP 0.9.1, the following crash occurred prior to this commit:
```
crasher:
  initial call: rabbit_channel:init/1
  pid: <0.818.0>
  registered_name: []
  exception exit: {{badmatch,undefined},
                   [{rabbit_channel,handle_deliver0,4,
                                    [{file,"rabbit_channel.erl"},
                                     {line,2728}]},
                    {lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
                    {rabbit_channel,handle_cast,2,
                                    [{file,"rabbit_channel.erl"},
                                     {line,728}]},
                    {gen_server2,handle_msg,2,
                                 [{file,"gen_server2.erl"},{line,1056}]},
                    {proc_lib,wake_up,3,
                              [{file,"proc_lib.erl"},{line,251}]}]}
```

This commit first gives `mc:init/3` the chance to set exchange and
routing_keys annotations.
If not set, `rabbit_stream_queue` will set these annotations assuming
the message was originally published via the stream protocol.

* Support consistent hash exchange routing for MQTT 5.0

When a consistent hash exchange is bound to the MQTT topic exchange,
MQTT 5.0 messages can be routed to queues consistently based on the
Correlation-Data in the PUBLISH packet.

* Convert MQTT 5.0 User Property

* to AMQP 0.9.1 headers
* from AMQP 0.9.1 headers
* to AMQP 1.0 application properties and message annotations
* from AMQP 1.0 application properties and message annotations

* Make use of Annotations in mc_mqtt:protocol_state/2

mc_mqtt:protocol_state/2 includes Annotations as parameter.
It's cleaner to make use of these Annotations when computing the
protocol state instead of relying on the caller (rabbitmq_mqtt_processor)
to compute the protocol state.

* Enforce AMQP 0.9.1 field name length limit

The AMQP 0.9.1 spec prohibits field names longer than 128 characters.
Therefore, when converting AMQP 1.0 message annotations, application
properties or MQTT 5.0 User Property to AMQP 0.9.1 headers, drop any
names longer than 128 characters.

* Fix type specs

Apply feedback from Michael Davis

Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

* Add mc_mqtt unit test suite

Implement mc_mqtt:x_header/2

* Translate indicator that payload is UTF-8 encoded

when converting between MQTT 5.0 and AMQP 1.0

* Translate single amqp-value section from AMQP 1.0 to MQTT

Convert to a text representation, if possible, and indicate to MQTT
client that the payload is UTF-8 encoded. This way, the MQTT client will
be able to parse the payload.

If conversion to text representation is not possible, encode the payload
using the AMQP 1.0 type system and indiate the encoding via Content-Type
message/vnd.rabbitmq.amqp.

This Content-Type is not registered.
Type "message" makes sense since it's a message.
Vendor tree "vnd.rabbitmq.amqp" makes sense since merely subtype "amqp" is not
registered.

* Fix payload conversion

* Translate Response Topic between MQTT and AMQP

Translate MQTT 5.0 Response Topic to AMQP 1.0 reply-to address and vice
versa.

The Response Topic must be a UTF-8 encoded string.

This commit re-uses the already defined RabbitMQ target addresses:
```
"/topic/"     RK        Publish to amq.topic with routing key RK
"/exchange/"  X "/" RK  Publish to exchange X with routing key RK
```

By default, the MQTT topic exchange is configure dto be amq.topic using
the 1st target address.

When an operator modifies the mqtt.exchange, the 2nd target address is
used.

* Apply PR feedback

and fix formatting

Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

* tidy up

* Add MQTT message_containers test

* consistent hash exchange: avoid amqp legacy conversion

When hashing on a header value.

* Avoid converting to amqp legacy when using exchange federation

* Fix test flake

* test and dialyzer fixes

* dialyzer fix

* Add MQTT protocol interoperability tests

Test receiving from and sending to MQTT 5.0 and
* AMQP 0.9.1
* AMQP 1.0
* STOMP
* Streams

* Regenerate portions of deps/rabbit/app.bzl with gazelle

I'm not exactly sure how this happened, but gazell seems to have been
run with an older version of the rules_erlang gazelle extension at
some point. This caused generation of a structure that is no longer
used. This commit updates the structure to the current pattern.

* mc: refactoring

* mc_amqpl: handle delivery annotations

Just in case they are included.

Also use iolist_to_iovec to create flat list of binaries when
converting from amqp with amqp encoded payload.

---------

Co-authored-by: David Ansari <david.ansari@gmx.de>
Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
Co-authored-by: Rin Kuryloski <kuryloskip@vmware.com>
This commit is contained in:
Karl Nilsson 2023-08-31 11:27:13 +01:00 committed by GitHub
parent 17d51d6ce6
commit 119f034406
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
102 changed files with 5956 additions and 1888 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
deps/amqp10_common/test/binary_generator_SUITE.erl vendored
View File

@ -1,11 +1,11 @@
-module(binary_generator_SUITE).
-compile(nowarn_export_all).
-compile(export_all).
-export([
]).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
%%%===================================================================

8
deps/amqp10_common/test/binary_parser_SUITE.erl vendored
View File

@ -55,16 +55,16 @@ end_per_testcase(_TestCase, _Config) ->
roundtrip(_Config) ->
Terms = [null,
{described,
{utf8, <<"URL">>},
{symbol, <<"URL">>},
{utf8, <<"http://example.org/hello-world">>}},
{described,
{utf8, <<"URL">>},
{utf8, <<"https://rabbitmq.com">>}},
{symbol, <<"URL">>},
{binary, <<"https://rabbitmq.com">>}},
{array, ubyte, [{ubyte, 1}, {ubyte, 255}]},
{boolean, false},
{list, [{utf8, <<"hi">>},
{described,
{utf8, <<"URL">>},
{symbol, <<"URL">>},
{utf8, <<"http://example.org/hello-world">>}}
]},
{list, [{int, 123},

14
deps/rabbit/BUILD.bazel vendored
View File

@ -535,6 +535,11 @@ rabbitmq_integration_suite(
size = "medium",
)
rabbitmq_integration_suite(
name = "message_containers_SUITE",
size = "medium",
)
rabbitmq_integration_suite(
name = "metrics_SUITE",
size = "medium",
@ -778,6 +783,15 @@ rabbitmq_suite(
],
)
rabbitmq_suite(
name = "mc_SUITE",
size = "small",
deps = [
"//deps/amqp10_common:erlang_app",
"//deps/rabbit_common:erlang_app",
],
)
rabbitmq_suite(
name = "rabbit_stream_coordinator_SUITE",
deps = [

35
deps/rabbit/app.bzl vendored
View File

@ -10,6 +10,7 @@ def all_beam_files(name = "all_beam_files"):
name = "behaviours",
srcs = [
"src/gm.erl",
"src/mc.erl",
"src/rabbit_backing_queue.erl",
"src/rabbit_credential_validator.erl",
"src/rabbit_exchange_type.erl",
@ -34,6 +35,10 @@ def all_beam_files(name = "all_beam_files"):
"src/gatherer.erl",
"src/internal_user.erl",
"src/lqueue.erl",
"src/mc_amqp.erl",
"src/mc_amqpl.erl",
"src/mc_compat.erl",
"src/mc_util.erl",
"src/mirrored_supervisor.erl",
"src/mirrored_supervisor_sups.erl",
"src/pg_local.erl",
@ -254,6 +259,7 @@ def all_test_beam_files(name = "all_test_beam_files"):
testonly = True,
srcs = [
"src/gm.erl",
"src/mc.erl",
"src/rabbit_backing_queue.erl",
"src/rabbit_credential_validator.erl",
"src/rabbit_exchange_type.erl",
@ -279,6 +285,10 @@ def all_test_beam_files(name = "all_test_beam_files"):
"src/gatherer.erl",
"src/internal_user.erl",
"src/lqueue.erl",
"src/mc_amqp.erl",
"src/mc_amqpl.erl",
"src/mc_compat.erl",
"src/mc_util.erl",
"src/mirrored_supervisor.erl",
"src/mirrored_supervisor_sups.erl",
"src/pg_local.erl",
@ -503,6 +513,7 @@ def all_srcs(name = "all_srcs"):
"include/amqqueue.hrl",
"include/amqqueue_v2.hrl",
"include/gm_specs.hrl",
"include/mc.hrl",
"include/rabbit_global_counters.hrl",
"include/vhost.hrl",
"include/vhost_v2.hrl",
@ -535,6 +546,11 @@ def all_srcs(name = "all_srcs"):
"src/gm.erl",
"src/internal_user.erl",
"src/lqueue.erl",
"src/mc.erl",
"src/mc_amqp.erl",
"src/mc_amqpl.erl",
"src/mc_compat.erl",
"src/mc_util.erl",
"src/mirrored_supervisor.erl",
"src/mirrored_supervisor_sups.erl",
"src/pg_local.erl",
@ -1985,3 +2001,22 @@ def test_suite_beam_files(name = "test_suite_beam_files"):
erlc_opts = "//:test_erlc_opts",
deps = ["//deps/amqp_client:erlang_app", "//deps/rabbitmq_ct_helpers:erlang_app"],
)
erlang_bytecode(
name = "message_containers_SUITE_beam_files",
testonly = True,
srcs = ["test/message_containers_SUITE.erl"],
outs = ["test/message_containers_SUITE.beam"],
app_name = "rabbit",
erlc_opts = "//:test_erlc_opts",
deps = ["//deps/amqp_client:erlang_app"],
)
erlang_bytecode(
name = "mc_SUITE_beam_files",
testonly = True,
srcs = ["test/mc_SUITE.erl"],
outs = ["test/mc_SUITE.beam"],
hdrs = ["include/mc.hrl"],
app_name = "rabbit",
erlc_opts = "//:test_erlc_opts",
deps = ["//deps/amqp10_common:erlang_app", "//deps/rabbit_common:erlang_app"],
)

23
deps/rabbit/include/mc.hrl vendored Normal file
View File

@ -0,0 +1,23 @@
-type death_key() :: {Queue :: rabbit_misc:resource_name(), rabbit_dead_letter:reason()}.
-type death_anns() :: #{first_time := non_neg_integer(), %% the timestamp of the first
last_time := non_neg_integer(), %% the timestamp of the last
ttl => non_neg_integer()}.
-record(death, {
exchange :: rabbit_misc:resource_name(),
routing_keys = [] :: [rabbit_types:routing_key()],
count = 0 :: non_neg_integer(),
anns :: death_anns()
}).
-record(deaths, {first :: death_key(),
last :: death_key(),
records = #{} :: #{death_key() := #death{}}}).
%% good enough for most use cases
-define(IS_MC(Msg), element(1, Msg) == mc andalso tuple_size(Msg) == 5).
%% "Field names MUST start with a letter, '$' or '#' and may continue with letters, '$' or '#', digits, or
%% underlines, to a maximum length of 128 characters." [AMQP 0.9.1 4.2.5.5 Field Tables]
%% Given that the valid chars are ASCII chars, 1 char is encoded as 1 byte.
-define(AMQP_LEGACY_FIELD_NAME_MAX_LEN, 128).

404
deps/rabbit/src/mc.erl vendored Normal file
View File

@ -0,0 +1,404 @@
-module(mc).
-export([
init/3,
size/1,
is/1,
get_annotation/2,
take_annotation/2,
set_annotation/3,
%% properties
is_persistent/1,
ttl/1,
correlation_id/1,
message_id/1,
timestamp/1,
priority/1,
set_ttl/2,
x_header/2,
routing_headers/2,
%%
convert/2,
protocol_state/1,
prepare/2,
record_death/3,
is_death_cycle/2,
last_death/1,
death_queue_names/1
]).
-include("mc.hrl").
-type str() :: atom() | string() | binary().
-type internal_ann_key() :: atom().
-type x_ann_key() :: binary(). %% should begin with x- or ideally x-opt-
-type x_ann_value() :: str() | integer() | float() | [x_ann_value()].
-type protocol() :: module().
-type annotations() :: #{internal_ann_key() => term(),
x_ann_key() => x_ann_value()}.
-type ann_key() :: internal_ann_key() | x_ann_key().
-type ann_value() :: term().
%% the protocol module must implement the mc behaviour
-record(?MODULE, {protocol :: protocol(),
%% protocol specific data term
data :: proto_state(),
%% any annotations done by the broker itself
%% such as recording the exchange / routing keys used
annotations = #{} :: annotations()
}).
-opaque state() :: #?MODULE{} | mc_compat:state().
-export_type([
state/0,
ann_key/0,
ann_value/0
]).
-type proto_state() :: term().
-type property_value() :: undefined |
string() |
binary() |
integer() |
float() |
boolean().
-type tagged_prop() :: {uuid, binary()} |
{utf8, binary()} |
{binary, binary()} |
{boolean, boolean()} |
{long, integer()} |
{ulong, non_neg_integer() } |
{list, [tagged_prop()]} |
{map, [{tagged_prop(), tagged_prop()}]} |
undefined.
%% behaviour callbacks for protocol specific implementation
%% protocol specific init function
%% returns a map of additional annotations to merge into the
%% protocol generic annotations map, e.g. ttl, priority and durable
-callback init(term()) ->
{proto_state(), annotations()}.
%% the size of the payload and other meta data respectively
-callback size(proto_state()) ->
{MetadataSize :: non_neg_integer(),
PayloadSize :: non_neg_integer()}.
%% retrieve and x- header from the protocol data
%% the return value should be tagged with an AMQP 1.0 type
-callback x_header(binary(), proto_state()) ->
tagged_prop().
%% retrieve a property field from the protocol data
%% e.g. message_id, correlation_id
-callback property(atom(), proto_state()) ->
tagged_prop().
%% return a map of header values used for message routing,
%% optionally include x- headers and / or complex types (i.e. tables, arrays etc)
-callback routing_headers(proto_state(), [x_headers | complex_types]) ->
#{binary() => term()}.
%% Convert state to another protocol
%% all protocols must be able to convert to mc_amqp (AMQP 1.0)
-callback convert_to(Target :: protocol(), proto_state()) ->
proto_state() | not_implemented.
%% Convert from another protocol
%% all protocols must be able to convert from mc_amqp (AMQP 1.0)
-callback convert_from(Source :: protocol(), proto_state()) ->
proto_state() | not_implemented.
%% emit a protocol specific state package
%% typically used by connection / channel type process at consumer delivery
%% time
-callback protocol_state(proto_state(), annotations()) ->
term().
%% prepare the data for either reading or storage
-callback prepare(read | store, proto_state()) ->
proto_state().
%%% API
-spec init(protocol(), term(), annotations()) -> state().
init(Proto, Data, Anns)
when is_atom(Proto)
andalso is_map(Anns) ->
{ProtoData, ProtoAnns} = Proto:init(Data),
#?MODULE{protocol = Proto,
data = ProtoData,
annotations = maps:merge(ProtoAnns, Anns)}.
-spec size(state()) ->
{MetadataSize :: non_neg_integer(),
PayloadSize :: non_neg_integer()}.
size(#?MODULE{protocol = Proto,
data = Data}) ->
Proto:size(Data);
size(BasicMsg) ->
mc_compat:size(BasicMsg).
-spec is(term()) -> boolean().
is(#?MODULE{}) ->
true;
is(Term) ->
mc_compat:is(Term).
-spec get_annotation(ann_key(), state()) -> ann_value() | undefined.
get_annotation(Key, #?MODULE{annotations = Anns}) ->
maps:get(Key, Anns, undefined);
get_annotation(Key, BasicMessage) ->
mc_compat:get_annotation(Key, BasicMessage).
-spec take_annotation(ann_key(), state()) -> {ann_value() | undefined, state()}.
take_annotation(Key, #?MODULE{annotations = Anns} = State) ->
case maps:take(Key, Anns) of
{Val, Anns1} ->
{Val, State#?MODULE{annotations = Anns1}};
error ->
{undefined, State}
end;
take_annotation(_Key, BasicMessage) ->
{undefined, BasicMessage}.
-spec set_annotation(ann_key(), ann_value(), state()) ->
state().
set_annotation(Key, Value, #?MODULE{annotations = Anns} = State) ->
State#?MODULE{annotations = maps:put(Key, Value, Anns)};
set_annotation(Key, Value, BasicMessage) ->
mc_compat:set_annotation(Key, Value, BasicMessage).
-spec x_header(Key :: binary(), state()) ->
tagged_prop().
x_header(Key, #?MODULE{protocol = Proto,
annotations = Anns,
data = Data}) ->
%% x-headers may be have been added to the annotations map so
%% we need to check that first
case Anns of
#{Key := Value} ->
mc_util:infer_type(Value);
_ ->
%% if not we have to call into the protocol specific handler
Proto:x_header(Key, Data)
end;
x_header(Key, BasicMsg) ->
mc_compat:x_header(Key, BasicMsg).
-spec routing_headers(state(), [x_headers | complex_types]) ->
#{binary() => property_value()}.
routing_headers(#?MODULE{protocol = Proto,
annotations = Anns,
data = Data}, Options) ->
New = case lists:member(x_headers, Options) of
true ->
maps:filter(fun (Key, _) ->
mc_util:is_x_header(Key)
end, Anns);
false ->
#{}
end,
maps:merge(Proto:routing_headers(Data, Options), New);
routing_headers(BasicMsg, Opts) ->
mc_compat:routing_headers(BasicMsg, Opts).
-spec is_persistent(state()) -> boolean().
is_persistent(#?MODULE{annotations = Anns}) ->
maps:get(durable, Anns, true);
is_persistent(BasicMsg) ->
mc_compat:is_persistent(BasicMsg).
-spec ttl(state()) -> undefined | non_neg_integer().
ttl(#?MODULE{annotations = Anns}) ->
maps:get(ttl, Anns, undefined);
ttl(BasicMsg) ->
mc_compat:ttl(BasicMsg).
-spec timestamp(state()) -> undefined | non_neg_integer().
timestamp(#?MODULE{annotations = Anns}) ->
maps:get(timestamp, Anns, undefined);
timestamp(BasicMsg) ->
mc_compat:timestamp(BasicMsg).
-spec priority(state()) -> undefined | non_neg_integer().
priority(#?MODULE{annotations = Anns}) ->
maps:get(priority, Anns, undefined);
priority(BasicMsg) ->
mc_compat:priority(BasicMsg).
-spec correlation_id(state()) ->
{uuid, binary()} |
{utf8, binary()} |
{binary, binary()} |
{ulong, non_neg_integer()} |
undefined.
correlation_id(#?MODULE{protocol = Proto,
data = Data}) ->
Proto:property(?FUNCTION_NAME, Data);
correlation_id(BasicMsg) ->
mc_compat:correlation_id(BasicMsg).
-spec message_id(state()) ->
{uuid, binary()} |
{utf8, binary()} |
{binary, binary()} |
{ulong, non_neg_integer()} |
undefined.
message_id(#?MODULE{protocol = Proto,
data = Data}) ->
Proto:property(?FUNCTION_NAME, Data);
message_id(BasicMsg) ->
mc_compat:message_id(BasicMsg).
-spec set_ttl(undefined | non_neg_integer(), state()) -> state().
set_ttl(Value, #?MODULE{annotations = Anns} = State) ->
State#?MODULE{annotations = maps:put(ttl, Value, Anns)};
set_ttl(Value, BasicMsg) ->
mc_compat:set_ttl(Value, BasicMsg).
-spec convert(protocol(), state()) -> state().
convert(Proto, #?MODULE{protocol = Proto} = State) ->
State;
convert(TargetProto, #?MODULE{protocol = SourceProto,
data = Data0} = State) ->
Data = SourceProto:prepare(read, Data0),
TargetState =
case SourceProto:convert_to(TargetProto, Data) of
not_implemented ->
case TargetProto:convert_from(SourceProto, Data) of
not_implemented ->
AmqpData = SourceProto:convert_to(mc_amqp, Data),
mc_amqp:convert_to(TargetProto, AmqpData);
TargetState0 ->
TargetState0
end;
TargetState0 ->
TargetState0
end,
State#?MODULE{protocol = TargetProto,
data = TargetState};
convert(Proto, BasicMsg) ->
mc_compat:convert_to(Proto, BasicMsg).
-spec protocol_state(state()) -> term().
protocol_state(#?MODULE{protocol = Proto,
annotations = Anns,
data = Data}) ->
Proto:protocol_state(Data, Anns);
protocol_state(BasicMsg) ->
mc_compat:protocol_state(BasicMsg).
-spec record_death(rabbit_dead_letter:reason(),
SourceQueue :: rabbit_misc:resource_name(),
state()) -> state().
record_death(Reason, SourceQueue,
#?MODULE{protocol = _Mod,
data = _Data,
annotations = Anns0} = State)
when is_atom(Reason) andalso is_binary(SourceQueue) ->
Key = {SourceQueue, Reason},
Exchange = maps:get(exchange, Anns0),
RoutingKeys = maps:get(routing_keys, Anns0),
Timestamp = os:system_time(millisecond),
Ttl = maps:get(ttl, Anns0, undefined),
DeathAnns = rabbit_misc:maps_put_truthy(ttl, Ttl, #{first_time => Timestamp,
last_time => Timestamp}),
case maps:get(deaths, Anns0, undefined) of
undefined ->
Ds = #deaths{last = Key,
first = Key,
records = #{Key => #death{count = 1,
exchange = Exchange,
routing_keys = RoutingKeys,
anns = DeathAnns}}},
Anns = Anns0#{<<"x-first-death-reason">> => atom_to_binary(Reason),
<<"x-first-death-queue">> => SourceQueue,
<<"x-first-death-exchange">> => Exchange,
<<"x-last-death-reason">> => atom_to_binary(Reason),
<<"x-last-death-queue">> => SourceQueue,
<<"x-last-death-exchange">> => Exchange
},
State#?MODULE{annotations = Anns#{deaths => Ds}};
#deaths{records = Rs} = Ds0 ->
Death = #death{count = C,
anns = DA} = maps:get(Key, Rs,
#death{exchange = Exchange,
routing_keys = RoutingKeys,
anns = DeathAnns}),
Ds = Ds0#deaths{last = Key,
records = Rs#{Key =>
Death#death{count = C + 1,
anns = DA#{last_time => Timestamp}}}},
Anns = Anns0#{deaths => Ds,
<<"x-last-death-reason">> => atom_to_binary(Reason),
<<"x-last-death-queue">> => SourceQueue,
<<"x-last-death-exchange">> => Exchange},
State#?MODULE{annotations = Anns}
end;
record_death(Reason, SourceQueue, BasicMsg) ->
mc_compat:record_death(Reason, SourceQueue, BasicMsg).
-spec is_death_cycle(rabbit_misc:resource_name(), state()) -> boolean().
is_death_cycle(TargetQueue, #?MODULE{annotations = #{deaths := Deaths}}) ->
is_cycle(TargetQueue, maps:keys(Deaths#deaths.records));
is_death_cycle(_TargetQueue, #?MODULE{}) ->
false;
is_death_cycle(TargetQueue, BasicMsg) ->
mc_compat:is_death_cycle(TargetQueue, BasicMsg).
-spec death_queue_names(state()) -> [rabbit_misc:resource_name()].
death_queue_names(#?MODULE{annotations = Anns}) ->
case maps:get(deaths, Anns, undefined) of
undefined ->
[];
#deaths{records = Records} ->
proplists:get_keys(maps:keys(Records))
end;
death_queue_names(BasicMsg) ->
mc_compat:death_queue_names(BasicMsg).
-spec last_death(state()) ->
undefined | {death_key(), #death{}}.
last_death(#?MODULE{annotations = Anns})
when not is_map_key(deaths, Anns) ->
undefined;
last_death(#?MODULE{annotations = #{deaths := #deaths{last = Last,
records = Rs}}}) ->
{Last, maps:get(Last, Rs)};
last_death(BasicMsg) ->
mc_compat:last_death(BasicMsg).
-spec prepare(read | store, state()) -> state().
prepare(For, #?MODULE{protocol = Proto,
data = Data} = State) ->
State#?MODULE{data = Proto:prepare(For, Data)};
prepare(For, State) ->
mc_compat:prepare(For, State).
%% INTERNAL
%% if there is a death with a source queue that is the same as the target
%% queue name and there are no newer deaths with the 'rejected' reason then
%% consider this a cycle
is_cycle(_Queue, []) ->
false;
is_cycle(_Queue, [{_Q, rejected} | _]) ->
%% any rejection breaks the cycle
false;
is_cycle(Queue, [{Queue, Reason} | _])
when Reason =/= rejected ->
true;
is_cycle(Queue, [_ | Rem]) ->
is_cycle(Queue, Rem).
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
-endif.

461
deps/rabbit/src/mc_amqp.erl vendored Normal file
View File

@ -0,0 +1,461 @@
-module(mc_amqp).
-behaviour(mc).
-include_lib("amqp10_common/include/amqp10_framing.hrl").
-include("mc.hrl").
-export([
init/1,
size/1,
x_header/2,
property/2,
routing_headers/2,
get_property/2,
convert_to/2,
convert_from/2,
protocol_state/2,
serialize/1,
prepare/2
]).
-import(rabbit_misc,
[maps_put_truthy/3,
maps_put_falsy/3
]).
-type message_section() ::
#'v1_0.header'{} |
#'v1_0.delivery_annotations'{} |
#'v1_0.message_annotations'{} |
#'v1_0.properties'{} |
#'v1_0.application_properties'{} |
#'v1_0.data'{} |
#'v1_0.amqp_sequence'{} |
#'v1_0.amqp_value'{} |
#'v1_0.footer'{}.
-define(SIMPLE_VALUE(V), is_binary(V) orelse
is_number(V) orelse
is_boolean(V)).
-type opt(T) :: T | undefined.
-type amqp10_data() :: [#'v1_0.amqp_sequence'{} | #'v1_0.data'{}] |
#'v1_0.amqp_value'{}.
-record(msg,
{
header :: opt(#'v1_0.header'{}),
delivery_annotations = []:: list(),
message_annotations = [] :: list(),
properties :: opt(#'v1_0.properties'{}),
application_properties = [] :: list(),
data = [] :: amqp10_data(),
footer = [] :: list()
}).
-opaque state() :: #msg{}.
-export_type([
state/0,
message_section/0
]).
%% mc implementation
init(Sections) when is_list(Sections) ->
Msg = decode(Sections, #msg{}),
init(Msg);
init(#msg{} = Msg) ->
%% TODO: as the essential annotations, durable, priority, ttl and delivery_count
%% is all we are interested in it isn't necessary to keep hold of the
%% incoming AMQP header inside the state
Anns = essential_properties(Msg),
{Msg, Anns}.
convert_from(?MODULE, Sections) ->
element(1, init(Sections));
convert_from(_SourceProto, _) ->
not_implemented.
size(#msg{data = Body}) ->
%% TODO how to estimate anything but data sections?
BodySize = if is_list(Body) ->
lists:foldl(
fun(#'v1_0.data'{content = Data}, Acc) ->
iolist_size(Data) + Acc;
(#'v1_0.amqp_sequence'{content = _}, Acc) ->
Acc
end, 0, Body);
is_record(Body, 'v1_0.amqp_value') ->
0
end,
{_MetaSize = 0, BodySize}.
x_header(Key, Msg) ->
message_annotation(Key, Msg, undefined).
property(correlation_id, #msg{properties = #'v1_0.properties'{correlation_id = Corr}}) ->
Corr;
property(message_id, #msg{properties = #'v1_0.properties'{message_id = MsgId}}) ->
MsgId;
property(_Prop, #msg{}) ->
undefined.
routing_headers(Msg, Opts) ->
IncludeX = lists:member(x_headers, Opts),
X = case IncludeX of
true ->
message_annotations_as_simple_map(Msg);
false ->
#{}
end,
application_properties_as_simple_map(Msg, X).
get_property(durable, Msg) ->
case Msg of
#msg{header = #'v1_0.header'{durable = Durable}}
when is_atom(Durable) ->
Durable;
#msg{header = #'v1_0.header'{durable = {boolean, Durable}}} ->
Durable;
_ ->
%% fallback in case the source protocol was old AMQP 0.9.1
case message_annotation(<<"x-basic-delivery-mode">>, Msg, 2) of
{ubyte, 2} ->
true;
_ ->
false
end
end;
get_property(timestamp, Msg) ->
case Msg of
#msg{properties = #'v1_0.properties'{creation_time = {timestamp, Ts}}} ->
Ts;
_ ->
undefined
end;
get_property(correlation_id, Msg) ->
case Msg of
#msg{properties = #'v1_0.properties'{correlation_id = {_Type, CorrId}}} ->
CorrId;
_ ->
undefined
end;
get_property(message_id, Msg) ->
case Msg of
#msg{properties = #'v1_0.properties'{message_id = {_Type, CorrId}}} ->
CorrId;
_ ->
undefined
end;
get_property(ttl, Msg) ->
case Msg of
#msg{header = #'v1_0.header'{ttl = {_, Ttl}}} ->
Ttl;
_ ->
%% fallback in case the source protocol was AMQP 0.9.1
case message_annotation(<<"x-basic-expiration">>, Msg, undefined) of
{utf8, Expiration} ->
{ok, Ttl} = rabbit_basic:parse_expiration(Expiration),
Ttl;
_ ->
undefined
end
end;
get_property(priority, Msg) ->
case Msg of
#msg{header = #'v1_0.header'{priority = {ubyte, Priority}}} ->
Priority;
_ ->
%% fallback in case the source protocol was AMQP 0.9.1
case message_annotation(<<"x-basic-priority">>, Msg, undefined) of
{_, Priority} ->
Priority;
_ ->
undefined
end
end;
get_property(_P, _Msg) ->
undefined.
convert_to(?MODULE, Msg) ->
Msg;
convert_to(TargetProto, Msg) ->
TargetProto:convert_from(?MODULE, msg_to_sections(Msg, fun (X) -> X end)).
serialize(Sections) ->
encode_bin(Sections).
protocol_state(Msg, Anns) ->
Exchange = maps:get(exchange, Anns),
[RKey | _] = maps:get(routing_keys, Anns),
%% any x-* annotations get added as message annotations
AnnsToAdd = maps:filter(fun (Key, _) -> mc_util:is_x_header(Key) end, Anns),
MACFun = fun(MAC) ->
add_message_annotations(
AnnsToAdd#{<<"x-exchange">> => wrap(utf8, Exchange),
<<"x-routing-key">> => wrap(utf8, RKey)}, MAC)
end,
msg_to_sections(Msg, MACFun).
prepare(_For, Msg) ->
Msg.
%% internal
msg_to_sections(#msg{header = H,
delivery_annotations = DAC,
message_annotations = MAC0,
properties = P,
application_properties = APC,
data = Data,
footer = FC}, MacFun) ->
Tail = case FC of
[] -> [];
_ ->
[#'v1_0.footer'{content = FC}]
end,
S0 = case Data of
#'v1_0.amqp_value'{} ->
[Data | Tail];
_ when is_list(Data) ->
Data ++ Tail
end,
S1 = case APC of
[] -> S0;
_ ->
[#'v1_0.application_properties'{content = APC} | S0]
end,
S2 = case P of
undefined -> S1;
_ ->
[P | S1]
end,
S3 = case MacFun(MAC0) of
[] -> S2;
MAC ->
[#'v1_0.message_annotations'{content = MAC} | S2]
end,
S4 = case DAC of
[] -> S3;
_ ->
[#'v1_0.delivery_annotations'{content = DAC} | S3]
end,
case H of
undefined -> S4;
_ ->
[H | S4]
end.
encode_bin(undefined) ->
<<>>;
encode_bin(Sections) when is_list(Sections) ->
[amqp10_framing:encode_bin(Section) || Section <- Sections,
not is_empty(Section)];
encode_bin(Section) ->
case is_empty(Section) of
true ->
<<>>;
false ->
amqp10_framing:encode_bin(Section)
end.
is_empty(undefined) ->
true;
is_empty(#'v1_0.properties'{message_id = undefined,
user_id = undefined,
to = undefined,
subject = undefined,
reply_to = undefined,
correlation_id = undefined,
content_type = undefined,
content_encoding = undefined,
absolute_expiry_time = undefined,
creation_time = undefined,
group_id = undefined,
group_sequence = undefined,
reply_to_group_id = undefined}) ->
true;
is_empty(#'v1_0.application_properties'{content = []}) ->
true;
is_empty(#'v1_0.message_annotations'{content = []}) ->
true;
is_empty(#'v1_0.delivery_annotations'{content = []}) ->
true;
is_empty(#'v1_0.footer'{content = []}) ->
true;
is_empty(#'v1_0.header'{durable = undefined,
priority = undefined,
ttl = undefined,
first_acquirer = undefined,
delivery_count = undefined}) ->
true;
is_empty(_) ->
false.
message_annotation(_Key, #msg{message_annotations = []},
Default) ->
Default;
message_annotation(Key, #msg{message_annotations = Content},
Default)
when is_binary(Key) ->
mc_util:amqp_map_get(Key, Content, Default).
message_annotations_as_simple_map(#msg{message_annotations = []}) ->
#{};
message_annotations_as_simple_map(#msg{message_annotations = Content}) ->
%% the section record format really is terrible
lists:foldl(fun ({{symbol, K}, {_T, V}}, Acc)
when ?SIMPLE_VALUE(V) ->
Acc#{K => V};
(_, Acc)->
Acc
end, #{}, Content).
application_properties_as_simple_map(#msg{application_properties = []}, M) ->
M;
application_properties_as_simple_map(#msg{application_properties = Content},
M) ->
%% the section record format really is terrible
lists:foldl(fun
({{utf8, K}, {_T, V}}, Acc)
when ?SIMPLE_VALUE(V) ->
Acc#{K => V};
({{utf8, K}, undefined}, Acc) ->
Acc#{K => undefined};
(_, Acc)->
Acc
end, M, Content).
decode([], Acc) ->
Acc;
decode([#'v1_0.header'{} = H | Rem], Msg) ->
decode(Rem, Msg#msg{header = H});
decode([#'v1_0.message_annotations'{content = MAC} | Rem], Msg) ->
decode(Rem, Msg#msg{message_annotations = MAC});
decode([#'v1_0.properties'{} = P | Rem], Msg) ->
decode(Rem, Msg#msg{properties = P});
decode([#'v1_0.application_properties'{content = APC} | Rem], Msg) ->
decode(Rem, Msg#msg{application_properties = APC});
decode([#'v1_0.delivery_annotations'{content = DAC} | Rem], Msg) ->
decode(Rem, Msg#msg{delivery_annotations = DAC});
decode([#'v1_0.data'{} = D | Rem], #msg{data = Body} = Msg)
when is_list(Body) ->
decode(Rem, Msg#msg{data = Body ++ [D]});
decode([#'v1_0.amqp_sequence'{} = D | Rem], #msg{data = Body} = Msg)
when is_list(Body) ->
decode(Rem, Msg#msg{data = Body ++ [D]});
decode([#'v1_0.footer'{content = FC} | Rem], Msg) ->
decode(Rem, Msg#msg{footer = FC});
decode([#'v1_0.amqp_value'{} = B | Rem], #msg{} = Msg) ->
%% an amqp value can only be a singleton
decode(Rem, Msg#msg{data = B}).
add_message_annotations(Anns, MA0) ->
maps:fold(fun (K, V, Acc) ->
map_add(symbol, K, mc_util:infer_type(V), Acc)
end, MA0, Anns).
map_add(_T, _Key, undefined, Acc) ->
Acc;
map_add(KeyType, Key, TaggedValue, Acc0) ->
TaggedKey = wrap(KeyType, Key),
lists_upsert({TaggedKey, TaggedValue}, Acc0).
wrap(_Type, undefined) ->
undefined;
wrap(Type, Val) ->
{Type, Val}.
key_find(K, [{{_, K}, {_, V}} | _]) ->
V;
key_find(K, [_ | Rem]) ->
key_find(K, Rem);
key_find(_K, []) ->
undefined.
recover_deaths([], Acc) ->
Acc;
recover_deaths([{map, Kvs} | Rem], Acc) ->
Queue = key_find(<<"queue">>, Kvs),
Reason = binary_to_atom(key_find(<<"reason">>, Kvs)),
DA0 = case key_find(<<"original-expiration">>, Kvs) of
undefined ->
#{};
Exp ->
#{ttl => binary_to_integer(Exp)}
end,
RKeys = [RK || {_, RK} <- key_find(<<"routing-keys">>, Kvs)],
Ts = key_find(<<"time">>, Kvs),
DA = DA0#{first_time => Ts,
last_time => Ts},
recover_deaths(Rem,
Acc#{{Queue, Reason} =>
#death{anns = DA,
exchange = key_find(<<"exchange">>, Kvs),
count = key_find(<<"count">>, Kvs),
routing_keys = RKeys}}).
essential_properties(#msg{message_annotations = MA} = Msg) ->
Durable = get_property(durable, Msg),
Priority = get_property(priority, Msg),
Timestamp = get_property(timestamp, Msg),
Ttl = get_property(ttl, Msg),
Deaths = case message_annotation(<<"x-death">>, Msg, undefined) of
{list, DeathMaps} ->
%% TODO: make more correct?
Def = {utf8, <<>>},
{utf8, FstQ} = message_annotation(<<"x-first-death-queue">>, Msg, Def),
{utf8, FstR} = message_annotation(<<"x-first-death-reason">>, Msg, Def),
{utf8, LastQ} = message_annotation(<<"x-last-death-queue">>, Msg, Def),
{utf8, LastR} = message_annotation(<<"x-last-death-reason">>, Msg, Def),
#deaths{first = {FstQ, binary_to_atom(FstR)},
last = {LastQ, binary_to_atom(LastR)},
records = recover_deaths(DeathMaps, #{})};
_ ->
undefined
end,
Anns = maps_put_falsy(
durable, Durable,
maps_put_truthy(
priority, Priority,
maps_put_truthy(
timestamp, Timestamp,
maps_put_truthy(
ttl, Ttl,
maps_put_truthy(
deaths, Deaths,
#{}))))),
case MA of
[] ->
Anns;
_ ->
lists:foldl(
fun ({{symbol, <<"x-routing-key">>},
{utf8, Key}}, Acc) ->
Acc#{routing_keys => [Key]};
({{symbol, <<"x-exchange">>},
{utf8, Exchange}}, Acc) ->
Acc#{exchange => Exchange};
(_, Acc) ->
Acc
end, Anns, MA)
end.
lists_upsert(New, L) ->
lists_upsert(New, L, [], L).
lists_upsert({Key, _} = New, [{Key, _} | Rem], Pref, _All) ->
lists:reverse(Pref, [New | Rem]);
lists_upsert(New, [Item | Rem], Pref, All) ->
lists_upsert(New, Rem, [Item | Pref], All);
lists_upsert(New, [], _Pref, All) ->
[New | All].

677
deps/rabbit/src/mc_amqpl.erl vendored Normal file
View File

@ -0,0 +1,677 @@
-module(mc_amqpl).
-behaviour(mc).
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("amqp10_common/include/amqp10_framing.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include("mc.hrl").
%% mc
-export([
init/1,
size/1,
x_header/2,
routing_headers/2,
convert_to/2,
convert_from/2,
protocol_state/2,
property/2,
set_property/3,
prepare/2
]).
%% utility functions
-export([
message/3,
message/4,
message/5,
from_basic_message/1
]).
-import(rabbit_misc,
[maps_put_truthy/3,
maps_put_falsy/3
]).
-define(HEADER_GUESS_SIZE, 100). %% see determine_persist_to/2
-define(AMQP10_TYPE, <<"amqp-1.0">>).
-define(AMQP10_PROPERTIES_HEADER, <<"x-amqp-1.0-properties">>).
-define(AMQP10_APP_PROPERTIES_HEADER, <<"x-amqp-1.0-app-properties">>).
-define(AMQP10_MESSAGE_ANNOTATIONS_HEADER, <<"x-amqp-1.0-message-annotations">>).
-define(PROTOMOD, rabbit_framing_amqp_0_9_1).
-define(CLASS_ID, 60).
-opaque state() :: #content{}.
-export_type([
state/0
]).
%% mc implementation
init(#content{} = Content0) ->
Content = rabbit_binary_parser:ensure_content_decoded(Content0),
%% project essential properties into annotations
Anns = essential_properties(Content),
{strip_header(Content, ?DELETED_HEADER), Anns}.
convert_from(mc_amqp, Sections) ->
{H, MAnn, Prop, AProp, BodyRev} =
lists:foldl(
fun
(#'v1_0.header'{} = S, Acc) ->
setelement(1, Acc, S);
(#'v1_0.message_annotations'{} = S, Acc) ->
setelement(2, Acc, S);
(#'v1_0.properties'{} = S, Acc) ->
setelement(3, Acc, S);
(#'v1_0.application_properties'{} = S, Acc) ->
setelement(4, Acc, S);
(#'v1_0.delivery_annotations'{}, Acc) ->
%% delivery annotations not currently used
Acc;
(#'v1_0.footer'{}, Acc) ->
%% footer not currently used
Acc;
(undefined, Acc) ->
Acc;
(BodySection, Acc) ->
Body = element(5, Acc),
setelement(5, Acc, [BodySection | Body])
end, {undefined, undefined, undefined, undefined, []},
Sections),
{PayloadRev, Type0} =
case BodyRev of
[#'v1_0.data'{content = Bin}] when is_binary(Bin) ->
{[Bin], undefined};
[#'v1_0.data'{content = Bin}] when is_list(Bin) ->
{lists:reverse(Bin), undefined};
_ ->
%% anything else needs to be encoded
%% TODO: This is potentially inefficient, but #content.payload_fragments_rev expects
%% currently a flat list of binaries. Can we make rabbit_writer work
%% with an iolist instead?
{[erlang:iolist_to_iovec(amqp10_framing:encode_bin(X))
|| X <- BodyRev], ?AMQP10_TYPE}
end,
#'v1_0.properties'{message_id = MsgId,
user_id = UserId0,
reply_to = ReplyTo0,
correlation_id = CorrId,
content_type = ContentType,
content_encoding = ContentEncoding,
creation_time = Timestamp,
group_id = GroupId} = case Prop of
undefined ->
#'v1_0.properties'{};
_ ->
Prop
end,
AP = case AProp of
#'v1_0.application_properties'{content = AC} -> AC;
_ -> []
end,
MA = case MAnn of
#'v1_0.message_annotations'{content = MC} -> MC;
_ -> []
end,
DelMode = case H of
#'v1_0.header'{durable = true} -> 2;
#'v1_0.header'{durable = false} -> 1;
_ -> amqp10_map_get(symbol(<<"x-basic-delivery-mode">>), MA)
end,
Priority = case H of
#'v1_0.header'{priority = {_, P}} -> P;
_ -> amqp10_map_get(symbol(<<"x-basic-priority">>), MA)
end,
%% check amqp header first for priority, ttl
Expiration = case H of
#'v1_0.header'{ttl = {_, T}} ->
integer_to_binary(T);
_ ->
amqp10_map_get(symbol(<<"x-basic-expiration">>), MA)
end,
Type = case Type0 of
undefined ->
amqp10_map_get(symbol(<<"x-basic-type">>), MA);
_ ->
Type0
end,
Headers0 = [to_091(K, V) || {{utf8, K}, V} <- AP,
byte_size(K) =< ?AMQP_LEGACY_FIELD_NAME_MAX_LEN],
%% Add remaining message annotations as headers?
XHeaders = [to_091(K, V) || {{symbol, K}, V} <- MA,
not is_internal_header(K),
byte_size(K) =< ?AMQP_LEGACY_FIELD_NAME_MAX_LEN],
{Headers1, MsgId091} = message_id(MsgId, <<"x-message-id">>, Headers0),
{Headers, CorrId091} = message_id(CorrId, <<"x-correlation-id">>, Headers1),
UserId1 = unwrap(UserId0),
UserId = case mc_util:is_valid_shortstr(UserId1) of
true ->
UserId1;
false ->
%% drop it, what else can we do?
undefined
end,
BP = #'P_basic'{message_id = MsgId091,
delivery_mode = DelMode,
expiration = Expiration,
user_id = UserId,
headers = case XHeaders ++ Headers of
[] -> undefined;
AllHeaders -> AllHeaders
end,
reply_to = unwrap(ReplyTo0),
type = Type,
app_id = unwrap(GroupId),
priority = Priority,
correlation_id = CorrId091,
content_type = unwrap(ContentType),
content_encoding = unwrap(ContentEncoding),
timestamp = unwrap(Timestamp)
},
#content{class_id = ?CLASS_ID,
properties = BP,
properties_bin = none,
payload_fragments_rev = PayloadRev};
convert_from(_SourceProto, _) ->
not_implemented.
size(#content{properties_bin = PropsBin,
properties = Props,
payload_fragments_rev = Payload}) ->
MetaSize = case is_binary(PropsBin) of
true ->
byte_size(PropsBin);
false ->
#'P_basic'{headers = Hs} = Props,
case Hs of
undefined -> 0;
_ -> length(Hs)
end * ?HEADER_GUESS_SIZE
end,
{MetaSize, iolist_size(Payload)}.
x_header(_Key, #content{properties = #'P_basic'{headers = undefined}}) ->
undefined;
x_header(Key, #content{properties = #'P_basic'{headers = Headers}}) ->
case rabbit_misc:table_lookup(Headers, Key) of
undefined ->
undefined;
{Type, Value} ->
from_091(Type, Value)
end;
x_header(Key, #content{properties = none} = Content0) ->
Content = rabbit_binary_parser:ensure_content_decoded(Content0),
x_header(Key, Content).
property(Prop, Content) ->
mc_util:infer_type(mc_compat:get_property(Prop, Content)).
routing_headers(#content{properties = #'P_basic'{headers = undefined}}, _Opts) ->
#{};
routing_headers(#content{properties = #'P_basic'{headers = Headers}}, Opts) ->
IncludeX = lists:member(x_headers, Opts),
%% Complex AMQP values such as array and table are hard to match on but
%% should still be included as routing headers as users may use a `void'
%% match which would only check for the presence of the key
lists:foldl(
fun({<<"x-", _/binary>> = Key, T, Value}, Acc) ->
case IncludeX of
true ->
Acc#{Key => routing_value(T, Value)};
false ->
Acc
end;
({Key, T, Value}, Acc) ->
Acc#{Key => routing_value(T, Value)}
end, #{}, Headers);
routing_headers(#content{properties = none} = Content, Opts) ->
routing_headers(prepare(read, Content), Opts).
routing_value(timestamp, V) ->
V * 1000;
routing_value(_, V) ->
V.
set_property(ttl, undefined, #content{properties = Props} = C) ->
%% only ttl is ever modified atm and only unset during dead lettering
C#content{properties = Props#'P_basic'{expiration = undefined},
properties_bin = none};
set_property(_P, _V, Msg) ->
Msg.
prepare(read, Content) ->
rabbit_binary_parser:ensure_content_decoded(Content);
prepare(store, Content) ->
rabbit_binary_parser:clear_decoded_content(
rabbit_binary_generator:ensure_content_encoded(Content, ?PROTOMOD)).
convert_to(?MODULE, Content) ->
Content;
convert_to(mc_amqp, #content{payload_fragments_rev = Payload} = Content) ->
#content{properties = Props} = prepare(read, Content),
#'P_basic'{message_id = MsgId,
expiration = Expiration,
delivery_mode = DelMode,
headers = Headers0,
user_id = UserId,
reply_to = ReplyTo,
type = Type,
priority = Priority,
app_id = AppId,
correlation_id = CorrId,
content_type = ContentType,
content_encoding = ContentEncoding,
timestamp = Timestamp} = Props,
ConvertedTs = case Timestamp of
undefined ->
undefined;
_ ->
Timestamp * 1000
end,
Headers = case Headers0 of
undefined -> [];
_ -> Headers0
end,
%% TODO: only add header section if at least one of the fields
%% needs to be set
H = #'v1_0.header'{durable = DelMode =:= 2,
%% TODO: check Priority is a ubyte?
priority = wrap(ubyte, Priority)},
P = case amqp10_section_header(?AMQP10_PROPERTIES_HEADER, Headers) of
undefined ->
#'v1_0.properties'{message_id = wrap(utf8, MsgId),
user_id = wrap(binary, UserId),
to = undefined,
% subject = wrap(utf8, RKey),
reply_to = wrap(utf8, ReplyTo),
correlation_id = wrap(utf8, CorrId),
content_type = wrap(symbol, ContentType),
content_encoding = wrap(symbol, ContentEncoding),
creation_time = wrap(timestamp, ConvertedTs),
%% this is semantically not the best idea but you
%% could imagine these having similar behaviour
group_id = wrap(utf8, AppId)
};
V10Prop ->
V10Prop
end,
AP = case amqp10_section_header(?AMQP10_APP_PROPERTIES_HEADER, Headers) of
undefined ->
%% non x- headers are stored as application properties when the type allows
APC = [{wrap(utf8, K), from_091(T, V)}
|| {K, T, V} <- Headers,
supported_header_value_type(T),
not mc_util:is_x_header(K)],
#'v1_0.application_properties'{content = APC};
A ->
A
end,
%% x- headers are stored as message annotations
MA = case amqp10_section_header(?AMQP10_MESSAGE_ANNOTATIONS_HEADER, Headers) of
undefined ->
MAC0 = [{{symbol, K}, from_091(T, V)}
|| {K, T, V} <- Headers,
mc_util:is_x_header(K),
%% all message annotation keys need to be either a symbol or ulong
%% but 0.9.1 field-table names are always strings
is_binary(K)
],
%% properties that _are_ potentially used by the broker
%% are stored as message annotations
%% an alternative woud be to store priority and delivery mode in
%% the amqp (1.0) header section using the dura
MAC = map_add(symbol, <<"x-basic-type">>, utf8, Type,
map_add(symbol, <<"x-basic-priority">>, ubyte, Priority,
map_add(symbol, <<"x-basic-delivery-mode">>, ubyte, DelMode,
map_add(symbol, <<"x-basic-expiration">>, utf8, Expiration,
MAC0)))),
#'v1_0.message_annotations'{content = MAC};
Section ->
Section
end,
Sections = [H, P, AP, MA, #'v1_0.data'{content = lists:reverse(Payload)}],
mc_amqp:convert_from(mc_amqp, Sections);
convert_to(_TargetProto, _Content) ->
not_implemented.
protocol_state(#content{properties = #'P_basic'{headers = H00} = B0} = C,
Anns) ->
%% Add any x- annotations as headers
H0 = case H00 of
undefined -> [];
_ ->
H00
end,
Deaths = maps:get(deaths, Anns, undefined),
Headers0 = deaths_to_headers(Deaths, H0),
Headers1 = maps:fold(
fun (<<"x-", _/binary>> = Key, Val, H) when is_integer(Val) ->
[{Key, long, Val} | H];
(<<"x-", _/binary>> = Key, Val, H) when is_binary(Val) ->
[{Key, longstr, Val} | H];
(<<"x-", _/binary>> = Key, Val, H) when is_boolean(Val) ->
[{Key, bool, Val} | H];
(<<"timestamp_in_ms">> = Key, Val, H) when is_integer(Val) ->
[{Key, long, Val} | H];
(_, _, Acc) ->
Acc
end, Headers0, Anns),
Headers = case Headers1 of
[] ->
undefined;
_ ->
%% Dedup
lists:usort(fun({Key1, _, _}, {Key2, _, _}) ->
Key1 =< Key2
end, Headers1)
end,
Timestamp = case Anns of
#{timestamp := Ts} ->
Ts div 1000;
_ ->
undefined
end,
Expiration = case Anns of
#{ttl := undefined} ->
%% this resets the TTL, only done bt dead lettering
%% publishes
undefined;
#{ttl := Ttl} ->
%% not sure this will ever happen
%% as we only ever unset the expiry
integer_to_binary(Ttl);
_ ->
B0#'P_basic'.expiration
end,
B = B0#'P_basic'{timestamp = Timestamp,
expiration = Expiration,
headers = Headers},
C#content{properties = B,
properties_bin = none};
protocol_state(Content0, Anns) ->
%% TODO: refactor to detect _if_ the properties even need decoding
%% It is possible that no additional annotations or properties need to be
%% changed
protocol_state(prepare(read, Content0), Anns).
-spec message(rabbit_types:exchange_name(), rabbit_types:routing_key(), #content{}) -> mc:state().
message(ExchangeName, RoutingKey, Content) ->
message(ExchangeName, RoutingKey, Content, #{}).
-spec message(rabbit_types:exchange_name(), rabbit_types:routing_key(), #content{}, map()) ->
mc:state().
message(XName, RoutingKey, Content, Anns) ->
message(XName, RoutingKey, Content, Anns,
rabbit_feature_flags:is_enabled(message_containers)).
%% helper for creating message container from messages received from
%% AMQP legacy
message(#resource{name = ExchangeNameBin}, RoutingKey,
#content{properties = Props} = Content, Anns, true)
when is_binary(RoutingKey) andalso
is_map(Anns) ->
HeaderRoutes = rabbit_basic:header_routes(Props#'P_basic'.headers),
mc:init(?MODULE,
rabbit_basic:strip_bcc_header(Content),
Anns#{routing_keys => [RoutingKey | HeaderRoutes],
exchange => ExchangeNameBin});
message(#resource{} = XName, RoutingKey,
#content{} = Content, Anns, false) ->
{ok, Msg} = rabbit_basic:message(XName, RoutingKey, Content),
case Anns of
#{id := Id} ->
Msg#basic_message{id = Id};
_ ->
Msg
end.
from_basic_message(#basic_message{content = Content,
id = Id,
exchange_name = Ex,
routing_keys = [RKey | _]}) ->
Anns = case Id of
undefined ->
#{};
_ ->
#{id => Id}
end,
message(Ex, RKey, prepare(read, Content), Anns, true).
%% Internal
deaths_to_headers(undefined, Headers) ->
Headers;
deaths_to_headers(#deaths{records = Records}, Headers0) ->
%% sort records by the last timestamp
List = lists:sort(
fun({_, #death{anns = #{last_time := L1}}},
{_, #death{anns = #{last_time := L2}}}) ->
L1 < L2
end, maps:to_list(Records)),
Infos = lists:foldl(
fun ({{QName, Reason}, #death{anns = #{first_time := Ts} = DA,
exchange = Ex,
count = Count,
routing_keys = RoutingKeys}},
Acc) ->
%% The first routing key is the one specified in the
%% basic.publish; all others are CC or BCC keys.
RKs = [hd(RoutingKeys) | rabbit_basic:header_routes(Headers0)],
RKeys = [{longstr, Key} || Key <- RKs],
ReasonBin = atom_to_binary(Reason, utf8),
PerMsgTTL = case maps:get(ttl, DA, undefined) of
undefined -> [];
Ttl when is_integer(Ttl) ->
Expiration = integer_to_binary(Ttl),
[{<<"original-expiration">>, longstr,
Expiration}]
end,
[{table, [{<<"count">>, long, Count},
{<<"reason">>, longstr, ReasonBin},
{<<"queue">>, longstr, QName},
{<<"time">>, timestamp, Ts div 1000},
{<<"exchange">>, longstr, Ex},
{<<"routing-keys">>, array, RKeys}] ++ PerMsgTTL}
| Acc]
end, [], List),
rabbit_misc:set_table_value(Headers0, <<"x-death">>, array, Infos).
strip_header(#content{properties = #'P_basic'{headers = undefined}}
= DecodedContent, _Key) ->
DecodedContent;
strip_header(#content{properties = Props0 = #'P_basic'{headers = Headers0}}
= Content, Key) ->
case lists:keytake(Key, 1, Headers0) of
false ->
Content;
{value, _Found, Headers} ->
Props = Props0#'P_basic'{headers = Headers},
rabbit_binary_generator:clear_encoded_content(
Content#content{properties = Props})
end.
wrap(_Type, undefined) ->
undefined;
wrap(Type, Val) ->
{Type, Val}.
from_091(longstr, V) ->
case mc_util:is_valid_shortstr(V) of
true ->
{utf8, V};
false ->
%% if a string is longer than 255 bytes we just assume it is binary
%% it _may_ still be valid utf8 but checking this is going to be
%% excessively slow
{binary, V}
end;
from_091(long, V) -> {long, V};
from_091(unsignedbyte, V) -> {ubyte, V};
from_091(short, V) -> {short, V};
from_091(unsignedshort, V) -> {ushort, V};
from_091(unsignedint, V) -> {uint, V};
from_091(signedint, V) -> {int, V};
from_091(double, V) -> {double, V};
from_091(float, V) -> {float, V};
from_091(bool, V) -> {boolean, V};
from_091(binary, V) -> {binary, V};
from_091(timestamp, V) -> {timestamp, V * 1000};
from_091(byte, V) -> {byte, V};
from_091(void, _V) -> null;
from_091(array, L) ->
{list, [from_091(T, V) || {T, V} <- L]};
from_091(table, L) ->
{map, [{wrap(symbol, K), from_091(T, V)} || {K, T, V} <- L]}.
map_add(_T, _Key, _Type, undefined, Acc) ->
Acc;
map_add(KeyType, Key, Type, Value, Acc) ->
[{wrap(KeyType, Key), wrap(Type, Value)} | Acc].
supported_header_value_type(array) ->
false;
supported_header_value_type(table) ->
false;
supported_header_value_type(_) ->
true.
amqp10_map_get(_K, []) ->
undefined;
amqp10_map_get(K, Tuples) ->
case lists:keyfind(K, 1, Tuples) of
false ->
undefined;
{_, V} ->
unwrap(V)
end.
symbol(T) -> {symbol, T}.
unwrap(undefined) ->
undefined;
unwrap({timestamp, V}) ->
V div 1000;
unwrap({_Type, V}) ->
V.
to_091(Key, {utf8, V}) when is_binary(V) -> {Key, longstr, V};
to_091(Key, {long, V}) -> {Key, long, V};
to_091(Key, {ulong, V}) -> {Key, long, V}; %% TODO: we could try to constrain this
to_091(Key, {byte, V}) -> {Key, byte, V};
to_091(Key, {ubyte, V}) -> {Key, unsignedbyte, V};
to_091(Key, {short, V}) -> {Key, short, V};
to_091(Key, {ushort, V}) -> {Key, unsignedshort, V};
to_091(Key, {uint, V}) -> {Key, unsignedint, V};
to_091(Key, {int, V}) -> {Key, signedint, V};
to_091(Key, {double, V}) -> {Key, double, V};
to_091(Key, {float, V}) -> {Key, float, V};
to_091(Key, {timestamp, V}) -> {Key, timestamp, V div 1000};
to_091(Key, {binary, V}) -> {Key, binary, V};
to_091(Key, {boolean, V}) -> {Key, bool, V};
to_091(Key, true) -> {Key, bool, true};
to_091(Key, false) -> {Key, bool, false};
to_091(Key, undefined) -> {Key, void, undefined};
to_091(Key, null) -> {Key, void, undefined};
to_091(Key, {list, L}) ->
{Key, array, [to_091(V) || V <- L]};
to_091(Key, {map, M}) ->
{Key, table, [to_091(unwrap(K), V) || {K, V} <- M]}.
to_091({utf8, V}) -> {longstr, V};
to_091({long, V}) -> {long, V};
to_091({byte, V}) -> {byte, V};
to_091({ubyte, V}) -> {unsignedbyte, V};
to_091({short, V}) -> {short, V};
to_091({ushort, V}) -> {unsignedshort, V};
to_091({uint, V}) -> {unsignedint, V};
to_091({int, V}) -> {signedint, V};
to_091({double, V}) -> {double, V};
to_091({float, V}) -> {float, V};
to_091({timestamp, V}) -> {timestamp, V div 1000};
to_091({binary, V}) -> {binary, V};
to_091({boolean, V}) -> {bool, V};
to_091(true) -> {bool, true};
to_091(false) -> {bool, false};
to_091(undefined) -> {void, undefined};
to_091(null) -> {void, undefined};
to_091({list, L}) ->
{array, [to_091(V) || V <- L]};
to_091({map, M}) ->
{table, [to_091(unwrap(K), V) || {K, V} <- M]}.
message_id({uuid, UUID}, _HKey, H0) ->
{H0, mc_util:uuid_to_string(UUID)};
message_id({ulong, N}, _HKey, H0) ->
{H0, erlang:integer_to_binary(N)};
message_id({binary, B}, HKey, H0) ->
{[{HKey, longstr, B} | H0], undefined};
message_id({utf8, S}, HKey, H0) ->
case byte_size(S) > 255 of
true ->
{[{HKey, longstr, S} | H0], undefined};
false ->
{H0, S}
end;
message_id(undefined, _HKey, H) ->
{H, undefined}.
essential_properties(#content{} = C) ->
#'P_basic'{delivery_mode = Mode,
priority = Priority,
timestamp = TimestampRaw} = Props = C#content.properties,
{ok, MsgTTL} = rabbit_basic:parse_expiration(Props),
Timestamp = case TimestampRaw of
undefined ->
undefined;
_ ->
%% timestamp should be in ms
TimestampRaw * 1000
end,
Durable = Mode == 2,
maps_put_truthy(
priority, Priority,
maps_put_truthy(
ttl, MsgTTL,
maps_put_truthy(
timestamp, Timestamp,
maps_put_falsy(
durable, Durable,
#{})))).
%% headers that are added as annotations during conversions
is_internal_header(<<"x-basic-", _/binary>>) ->
true;
is_internal_header(<<"x-routing-key">>) ->
true;
is_internal_header(<<"x-exchange">>) ->
true;
is_internal_header(<<"x-death">>) ->
true;
is_internal_header(_) ->
false.
amqp10_section_header(Header, Headers) ->
case lists:keyfind(Header, 1, Headers) of
{_, _, Data} when is_binary(Data) ->
[Section] = amqp10_framing:decode_bin(Data),
Section ;
_ ->
undefined
end.

407
deps/rabbit/src/mc_compat.erl vendored Normal file
View File

@ -0,0 +1,407 @@
-module(mc_compat).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include("mc.hrl").
-export([
%init/3,
size/1,
is/1,
get_annotation/2,
set_annotation/3,
%%% properties
is_persistent/1,
ttl/1,
correlation_id/1,
message_id/1,
timestamp/1,
priority/1,
set_ttl/2,
x_header/2,
routing_headers/2,
%%%
convert_to/2,
protocol_state/1,
%serialize/1,
prepare/2,
record_death/3,
is_death_cycle/2,
%deaths/1,
last_death/1,
death_queue_names/1
]).
-export([get_property/2]).
-type state() :: rabbit_types:message().
-export_type([
state/0
]).
size(#basic_message{content = Content}) ->
mc_amqpl:size(Content).
is(#basic_message{}) ->
true;
is(_) ->
false.
-spec get_annotation(mc:ann_key(), state()) -> mc:ann_value() | undefined.
get_annotation(routing_keys, #basic_message{routing_keys = RKeys}) ->
RKeys;
get_annotation(exchange, #basic_message{exchange_name = Ex}) ->
Ex#resource.name;
get_annotation(id, #basic_message{id = Id}) ->
Id.
set_annotation(id, Value, #basic_message{} = Msg) ->
Msg#basic_message{id = Value};
set_annotation(routing_keys, Value, #basic_message{} = Msg) ->
Msg#basic_message{routing_keys = Value};
set_annotation(exchange, Value, #basic_message{exchange_name = Ex} = Msg) ->
Msg#basic_message{exchange_name = Ex#resource{name = Value}};
set_annotation(<<"x-", _/binary>> = Key, Value,
#basic_message{content =
#content{properties =
#'P_basic'{headers = H0} = B} = C0} = Msg) ->
T = case Value of
_ when is_integer(Value) ->
long;
_ when is_binary(Value) ->
longstr
end,
H1 = case H0 of
undefined ->
[];
_ ->
H0
end,
H2 = [{Key, T, Value} | H1],
H = lists:usort(fun({Key1, _, _}, {Key2, _, _}) ->
Key1 =< Key2
end, H2),
C = C0#content{properties = B#'P_basic'{headers = H},
properties_bin = none},
Msg#basic_message{content = C};
set_annotation(<<"timestamp_in_ms">> = Name, Value, #basic_message{} = Msg) ->
rabbit_basic:add_header(Name, long, Value, Msg);
set_annotation(timestamp, Millis,
#basic_message{content = #content{properties = B} = C0} = Msg) ->
C = C0#content{properties = B#'P_basic'{timestamp = Millis div 1000},
properties_bin = none},
Msg#basic_message{content = C}.
is_persistent(#basic_message{content = Content}) ->
get_property(durable, Content).
ttl(#basic_message{content = Content}) ->
get_property(?FUNCTION_NAME, Content).
timestamp(#basic_message{content = Content}) ->
get_property(?FUNCTION_NAME, Content).
priority(#basic_message{content = Content}) ->
get_property(?FUNCTION_NAME, Content).
correlation_id(#basic_message{content = Content}) ->
case get_property(?FUNCTION_NAME, Content) of
undefined ->
undefined;
Corr ->
{utf8, Corr}
end.
message_id(#basic_message{content = Content}) ->
case get_property(?FUNCTION_NAME, Content) of
undefined ->
undefined;
MsgId ->
{utf8, MsgId}
end.
set_ttl(Value, #basic_message{content = Content0} = Msg) ->
Content = mc_amqpl:set_property(ttl, Value, Content0),
Msg#basic_message{content = Content}.
x_header(Key,#basic_message{content = Content}) ->
mc_amqpl:x_header(Key, Content).
routing_headers(#basic_message{content = Content}, Opts) ->
mc_amqpl:routing_headers(Content, Opts).
convert_to(mc_amqpl, #basic_message{} = BasicMsg) ->
BasicMsg;
convert_to(Proto, #basic_message{} = BasicMsg) ->
%% at this point we have to assume this message will no longer travel between nodes
%% and potentially end up on a node that doesn't yet understand message containers
%% create legacy mc, then convert and return this
mc:convert(Proto, mc_amqpl:from_basic_message(BasicMsg)).
protocol_state(#basic_message{content = Content}) ->
rabbit_binary_parser:ensure_content_decoded(Content).
prepare(read, #basic_message{content = Content} = Msg) ->
Msg#basic_message{content =
rabbit_binary_parser:ensure_content_decoded(Content)};
prepare(store, Msg) ->
Msg.
record_death(Reason, SourceQueue,
#basic_message{content = Content,
exchange_name = Exchange,
routing_keys = RoutingKeys} = Msg) ->
% HeadersFun1 = fun (H) -> lists:keydelete(<<"CC">>, 1, H) end,
ReasonBin = atom_to_binary(Reason),
TimeSec = os:system_time(seconds),
PerMsgTTL = per_msg_ttl_header(Content#content.properties),
HeadersFun2 =
fun (Headers) ->
%% The first routing key is the one specified in the
%% basic.publish; all others are CC or BCC keys.
RKs = [hd(RoutingKeys) | rabbit_basic:header_routes(Headers)],
RKs1 = [{longstr, Key} || Key <- RKs],
Info = [{<<"reason">>, longstr, ReasonBin},
{<<"queue">>, longstr, SourceQueue},
{<<"time">>, timestamp, TimeSec},
{<<"exchange">>, longstr, Exchange#resource.name},
{<<"routing-keys">>, array, RKs1}] ++ PerMsgTTL,
update_x_death_header(Info, Headers)
end,
Content1 = #content{properties = Props} =
rabbit_basic:map_headers(HeadersFun2, Content),
Content2 = Content1#content{properties =
Props#'P_basic'{expiration = undefined}},
Msg#basic_message{id = rabbit_guid:gen(),
content = Content2}.
x_death_event_key(Info, Key) ->
x_death_event_key(Info, Key, undefined).
x_death_event_key(Info, Key, Def) ->
case lists:keysearch(Key, 1, Info) of
false -> Def;
{value, {Key, _KeyType, Val}} -> Val
end.
maybe_append_to_event_group(Table, _Key, _SeenKeys, []) ->
[Table];
maybe_append_to_event_group(Table, {_Queue, _Reason} = Key, SeenKeys, Acc) ->
case sets:is_element(Key, SeenKeys) of
true -> Acc;
false -> [Table | Acc]
end.
group_by_queue_and_reason([]) ->
[];
group_by_queue_and_reason([Table]) ->
[Table];
group_by_queue_and_reason(Tables) ->
{_, Grouped} =
lists:foldl(
fun ({table, Info}, {SeenKeys, Acc}) ->
Q = x_death_event_key(Info, <<"queue">>),
R = x_death_event_key(Info, <<"reason">>),
Matcher = queue_and_reason_matcher(Q, R),
{Matches, _} = lists:partition(Matcher, Tables),
{Augmented, N} = case Matches of
[X] -> {X, 1};
[X|_] = Xs -> {X, length(Xs)}
end,
Key = {Q, R},
Acc1 = maybe_append_to_event_group(
ensure_xdeath_event_count(Augmented, N),
Key, SeenKeys, Acc),
{sets:add_element(Key, SeenKeys), Acc1}
end, {sets:new([{version, 2}]), []}, Tables),
Grouped.
update_x_death_header(Info, undefined) ->
update_x_death_header(Info, []);
update_x_death_header(Info, Headers) ->
X = x_death_event_key(Info, <<"exchange">>),
Q = x_death_event_key(Info, <<"queue">>),
R = x_death_event_key(Info, <<"reason">>),
case rabbit_basic:header(<<"x-death">>, Headers) of
undefined ->
%% First x-death event gets its own top-level headers.
%% See rabbitmq/rabbitmq-server#1332.
Headers2 = rabbit_misc:set_table_value(Headers, <<"x-first-death-reason">>,
longstr, R),
Headers3 = rabbit_misc:set_table_value(Headers2, <<"x-first-death-queue">>,
longstr, Q),
Headers4 = rabbit_misc:set_table_value(Headers3, <<"x-first-death-exchange">>,
longstr, X),
rabbit_basic:prepend_table_header(
<<"x-death">>,
[{<<"count">>, long, 1} | Info], Headers4);
{<<"x-death">>, array, Tables} ->
%% group existing x-death headers in case we have some from
%% before rabbitmq-server#78
GroupedTables = group_by_queue_and_reason(Tables),
{Matches, Others} = lists:partition(
queue_and_reason_matcher(Q, R),
GroupedTables),
Info1 = case Matches of
[] ->
[{<<"count">>, long, 1} | Info];
[{table, M}] ->
increment_xdeath_event_count(M)
end,
rabbit_misc:set_table_value(
Headers, <<"x-death">>, array,
[{table, rabbit_misc:sort_field_table(Info1)} | Others]);
{<<"x-death">>, InvalidType, Header} ->
rabbit_log:warning("Message has invalid x-death header (type: ~tp)."
" Resetting header ~tp",
[InvalidType, Header]),
%% if x-death is something other than an array (list)
%% then we reset it: this happens when some clients consume
%% a message and re-publish is, converting header values
%% to strings, intentionally or not.
%% See rabbitmq/rabbitmq-server#767 for details.
rabbit_misc:set_table_value(
Headers, <<"x-death">>, array,
[{table, [{<<"count">>, long, 1} | Info]}])
end.
ensure_xdeath_event_count({table, Info}, InitialVal) when InitialVal >= 1 ->
{table, ensure_xdeath_event_count(Info, InitialVal)};
ensure_xdeath_event_count(Info, InitialVal) when InitialVal >= 1 ->
case x_death_event_key(Info, <<"count">>) of
undefined ->
[{<<"count">>, long, InitialVal} | Info];
_ ->
Info
end.
increment_xdeath_event_count(Info) ->
case x_death_event_key(Info, <<"count">>) of
undefined ->
[{<<"count">>, long, 1} | Info];
N ->
lists:keyreplace(
<<"count">>, 1, Info,
{<<"count">>, long, N + 1})
end.
queue_and_reason_matcher(Q, R) ->
F = fun(Info) ->
x_death_event_key(Info, <<"queue">>) =:= Q
andalso x_death_event_key(Info, <<"reason">>) =:= R
end,
fun({table, Info}) ->
F(Info);
(Info) when is_list(Info) ->
F(Info)
end.
per_msg_ttl_header(#'P_basic'{expiration = undefined}) ->
[];
per_msg_ttl_header(#'P_basic'{expiration = Expiration}) ->
[{<<"original-expiration">>, longstr, Expiration}];
per_msg_ttl_header(_) ->
[].
is_death_cycle(Queue, #basic_message{content = Content}) ->
#content{properties = #'P_basic'{headers = Headers0}} =
rabbit_binary_parser:ensure_content_decoded(Content),
Headers = case Headers0 of
undefined ->
[];
_ ->
Headers0
end,
case rabbit_misc:table_lookup(Headers, <<"x-death">>) of
{array, Deaths} ->
{Cycle, Rest} =
lists:splitwith(
fun ({table, D}) ->
{longstr, Queue} =/=
rabbit_misc:table_lookup(D, <<"queue">>);
(_) ->
true
end, Deaths),
%% Is there a cycle, and if so, is it "fully automatic", i.e. with
%% no reject in it?
case Rest of
[] ->
false;
[H|_] ->
lists:all(fun ({table, D}) ->
{longstr, <<"rejected">>} =/=
rabbit_misc:table_lookup(D, <<"reason">>);
(_) ->
%% There was something we didn't expect, therefore
%% a client must have put it there, therefore the
%% cycle was not "fully automatic".
false
end, Cycle ++ [H])
end;
_ ->
false
end.
death_queue_names(#basic_message{content = Content}) ->
#content{properties = #'P_basic'{headers = Headers}} =
rabbit_binary_parser:ensure_content_decoded(Content),
case rabbit_misc:table_lookup(Headers, <<"x-death">>) of
{array, Deaths} ->
[begin
{_, N} = rabbit_misc:table_lookup(D, <<"queue">>),
N
end || {table, D} <- Deaths];
_ ->
[]
end.
last_death(#basic_message{content = Content}) ->
#content{properties = #'P_basic'{headers = Headers}} =
rabbit_binary_parser:ensure_content_decoded(Content),
%% TODO: review this conversion and/or change the API
case rabbit_misc:table_lookup(Headers, <<"x-death">>) of
{array, [{table, Info} | _]} ->
X = x_death_event_key(Info, <<"exchange">>),
Q = x_death_event_key(Info, <<"queue">>),
T = x_death_event_key(Info, <<"time">>, 0),
Keys = x_death_event_key(Info, <<"routing_keys">>),
Count = x_death_event_key(Info, <<"count">>),
{Q, #death{exchange = X,
anns = #{first_time => T * 1000,
last_time => T * 1000},
routing_keys = Keys,
count = Count}};
_ ->
undefined
end.
get_property(P, #content{properties = none} = Content) ->
%% this is inefficient but will only apply to old messages that are
%% not containerized
get_property(P, rabbit_binary_parser:ensure_content_decoded(Content));
get_property(durable,
#content{properties = #'P_basic'{delivery_mode = Mode}}) ->
Mode == 2;
get_property(ttl, #content{properties = Props}) ->
{ok, MsgTTL} = rabbit_basic:parse_expiration(Props),
MsgTTL;
get_property(priority, #content{properties = #'P_basic'{priority = P}}) ->
P;
get_property(timestamp, #content{properties = Props}) ->
#'P_basic'{timestamp = Timestamp} = Props,
case Timestamp of
undefined ->
undefined;
_ ->
%% timestamp should be in ms
Timestamp * 1000
end;
get_property(correlation_id,
#content{properties = #'P_basic'{correlation_id = Corr}}) ->
Corr;
get_property(message_id,
#content{properties = #'P_basic'{message_id = MsgId}}) ->
MsgId;
get_property(_P, _C) ->
undefined.

70
deps/rabbit/src/mc_util.erl vendored Normal file
View File

@ -0,0 +1,70 @@
-module(mc_util).
-export([is_valid_shortstr/1,
is_utf8_no_null/1,
uuid_to_string/1,
infer_type/1,
utf8_string_is_ascii/1,
amqp_map_get/3,
is_x_header/1
]).
-spec is_valid_shortstr(term()) -> boolean().
is_valid_shortstr(Bin) when byte_size(Bin) < 256 ->
is_utf8_no_null(Bin);
is_valid_shortstr(_) ->
false.
is_utf8_no_null(<<>>) ->
true;
is_utf8_no_null(<<0, _/binary>>) ->
false;
is_utf8_no_null(<<_/utf8, Rem/binary>>) ->
is_utf8_no_null(Rem);
is_utf8_no_null(_) ->
false.
-spec uuid_to_string(binary()) -> binary().
uuid_to_string(<<TL:32, TM:16, THV:16, CSR:8, CSL:8, N:48>>) ->
list_to_binary(
io_lib:format(<<"urn:uuid:~8.16.0b-~4.16.0b-~4.16.0b-~2.16.0b~2.16.0b-~12.16.0b">>,
[TL, TM, THV, CSR, CSL, N])).
infer_type(undefined) ->
undefined;
infer_type(V) when is_binary(V) ->
{utf8, V};
infer_type(V) when is_integer(V) ->
{long, V};
infer_type(V) when is_boolean(V) ->
{boolean, V};
infer_type({T, _} = V) when is_atom(T) ->
%% looks like a pre-tagged type
V.
utf8_string_is_ascii(UTF8String)
when is_binary(UTF8String) ->
List = unicode:characters_to_list(UTF8String),
lists:all(fun(Char) ->
Char >= 0 andalso
Char < 128
end, List).
amqp_map_get(Key, {map, List}, Default) ->
amqp_map_get(Key, List, Default);
amqp_map_get(Key, List, Default) when is_list(List) ->
case lists:search(fun ({{_, K}, _}) -> K == Key end, List) of
{value, {_K, V}} ->
V;
false ->
Default
end;
amqp_map_get(_, _, Default) ->
Default.
-spec is_x_header(binary()) -> boolean().
is_x_header(<<"x-", _/binary>>) ->
true;
is_x_header(_) ->
false.

55
deps/rabbit/src/rabbit_amqqueue.erl vendored
View File

@ -11,15 +11,15 @@
-export([recover/1, stop/1, start/1, declare/6, declare/7,
delete_immediately/1, delete_exclusive/2, delete/4, purge/1,
forget_all_durable/1]).
-export([pseudo_queue/2, pseudo_queue/3, immutable/1]).
-export([pseudo_queue/2, pseudo_queue/3]).
-export([exists/1, lookup/1, lookup/2, lookup_many/1, lookup_durable_queue/1,
not_found_or_absent_dirty/1,
with/2, with/3, with_or_die/2,
assert_equivalence/5,
augment_declare_args/5,
check_exclusive_access/2, with_exclusive_access_or_die/3,
stat/1, deliver/2,
requeue/3, ack/3, reject/4]).
stat/1
]).
-export([not_found/1, absent/2]).
-export([list/0, list_durable/0, list/1, info_keys/0, info/1, info/2, info_all/1, info_all/2,
emit_info_all/5, list_local/1, info_local/1,
@ -69,7 +69,7 @@
-export([deactivate_limit_all/2]).
-export([prepend_extra_bcc/1]).
-export([queue/1, queue_name/1, queue_names/1]).
-export([queue/1, queue_names/1]).
%% internal
-export([internal_declare/2, internal_delete/2, run_backing_queue/3,
@ -95,7 +95,7 @@
-type qlen() :: rabbit_types:ok(non_neg_integer()).
-type qfun(A) :: fun ((amqqueue:amqqueue()) -> A | no_return()).
-type qmsg() :: {name(), pid() | {atom(), pid()}, msg_id(),
boolean(), rabbit_types:message()}.
boolean(), mc:state()}.
-type msg_id() :: non_neg_integer().
-type ok_or_errors() ::
'ok' | {'error', [{'error' | 'exit' | 'throw', any()}]}.
@ -1616,33 +1616,6 @@ delete_crashed_internal(Q, ActingUser) when ?amqqueue_is_classic(Q) ->
purge(Q) when ?is_amqqueue(Q) ->
rabbit_queue_type:purge(Q).
-spec requeue(name(),
{rabbit_fifo:consumer_tag(), [msg_id()]},
rabbit_queue_type:state()) ->
{ok, rabbit_queue_type:state(), rabbit_queue_type:actions()}.
requeue(QRef, {CTag, MsgIds}, QStates) ->
reject(QRef, true, {CTag, MsgIds}, QStates).
-spec ack(name(),
{rabbit_fifo:consumer_tag(), [msg_id()]},
rabbit_queue_type:state()) ->
{ok, rabbit_queue_type:state(), rabbit_queue_type:actions()}.
ack(QPid, {CTag, MsgIds}, QueueStates) ->
rabbit_queue_type:settle(QPid, complete, CTag, MsgIds, QueueStates).
-spec reject(name(),
boolean(),
{rabbit_fifo:consumer_tag(), [msg_id()]},
rabbit_queue_type:state()) ->
{ok, rabbit_queue_type:state(), rabbit_queue_type:actions()}.
reject(QRef, Requeue, {CTag, MsgIds}, QStates) ->
Op = case Requeue of
true -> requeue;
false -> discard
end,
rabbit_queue_type:settle(QRef, Op, CTag, MsgIds, QStates).
-spec notify_down_all(qpids(), pid()) -> ok_or_errors().
notify_down_all(QPids, ChPid) ->
notify_down_all(QPids, ChPid, ?CHANNEL_OPERATION_TIMEOUT).
@ -2001,16 +1974,6 @@ pseudo_queue(#resource{kind = queue} = QueueName, Pid, Durable)
rabbit_classic_queue % Type
).
-spec immutable(amqqueue:amqqueue()) -> amqqueue:amqqueue().
immutable(Q) -> amqqueue:set_immutable(Q).
-spec deliver([amqqueue:amqqueue()], rabbit_types:delivery()) -> 'ok'.
deliver(Qs, Delivery) ->
_ = rabbit_queue_type:deliver(Qs, Delivery, stateless),
ok.
get_quorum_nodes(Q) ->
case amqqueue:get_type_state(Q) of
#{nodes := Nodes} ->
@ -2065,14 +2028,6 @@ queue({Q, RouteInfos})
when ?is_amqqueue(Q) andalso is_map(RouteInfos) ->
Q.
-spec queue_name(name() | {name(), route_infos()}) ->
name().
queue_name(QName = #resource{kind = queue}) ->
QName;
queue_name({QName = #resource{kind = queue}, RouteInfos})
when is_map(RouteInfos) ->
QName.
-spec queue_names([Q | {Q, route_infos()}]) ->
[name()] when Q :: amqqueue:amqqueue().
queue_names(Queues)

54
deps/rabbit/src/rabbit_amqqueue_process.erl vendored
View File

@ -611,20 +611,20 @@ send_or_record_confirm(#delivery{confirm = false}, State) ->
send_or_record_confirm(#delivery{confirm = true,
sender = SenderPid,
msg_seq_no = MsgSeqNo,
message = #basic_message {
is_persistent = true,
id = MsgId}},
message = Message
},
State = #q{q = Q,
msg_id_to_channel = MTC})
when ?amqqueue_is_durable(Q) ->
MTC1 = maps:put(MsgId, {SenderPid, MsgSeqNo}, MTC),
{eventually, State#q{msg_id_to_channel = MTC1}};
send_or_record_confirm(#delivery{confirm = true,
sender = SenderPid,
msg_seq_no = MsgSeqNo},
#q{q = Q} = State) ->
confirm_to_sender(SenderPid, amqqueue:get_name(Q), [MsgSeqNo]),
{immediately, State}.
msg_id_to_channel = MTC}) ->
Persistent = mc:is_persistent(Message),
MsgId = mc:get_annotation(id, Message),
case Persistent of
true when ?amqqueue_is_durable(Q) ->
MTC1 = maps:put(MsgId, {SenderPid, MsgSeqNo}, MTC),
{eventually, State#q{msg_id_to_channel = MTC1}};
_ ->
confirm_to_sender(SenderPid, amqqueue:get_name(Q), [MsgSeqNo]),
{immediately, State}
end.
%% This feature was used by `rabbit_amqqueue_process` and
%% `rabbit_mirror_queue_slave` up-to and including RabbitMQ 3.7.x. It is
@ -641,7 +641,8 @@ send_mandatory(#delivery{mandatory = true,
discard(#delivery{confirm = Confirm,
sender = SenderPid,
flow = Flow,
message = #basic_message{id = MsgId}}, BQ, BQS, MTC, QName) ->
message = Msg}, BQ, BQS, MTC, QName) ->
MsgId = mc:get_annotation(id, Msg),
MTC1 = case Confirm of
true -> confirm_messages([MsgId], MTC, QName);
false -> MTC
@ -809,12 +810,13 @@ send_reject_publish(#delivery{confirm = true,
sender = SenderPid,
flow = Flow,
msg_seq_no = MsgSeqNo,
message = #basic_message{id = MsgId}},
message = Msg},
_Delivered,
State = #q{ q = Q,
backing_queue = BQ,
backing_queue_state = BQS,
msg_id_to_channel = MTC}) ->
MsgId = mc:get_annotation(id, Msg),
ok = rabbit_classic_queue:send_rejection(SenderPid,
amqqueue:get_name(Q), MsgSeqNo),
@ -834,9 +836,8 @@ will_overflow(#delivery{message = Message},
backing_queue_state = BQS}) ->
ExpectedQueueLength = BQ:len(BQS) + 1,
#basic_message{content = #content{payload_fragments_rev = PFR}} = Message,
MessageSize = iolist_size(PFR),
ExpectedQueueSizeBytes = BQ:info(message_bytes_ready, BQS) + MessageSize,
{_, PayloadSize} = mc:size(Message),
ExpectedQueueSizeBytes = BQ:info(message_bytes_ready, BQS) + PayloadSize,
ExpectedQueueLength > MaxLen orelse ExpectedQueueSizeBytes > MaxBytes.
@ -977,21 +978,18 @@ subtract_acks(ChPid, AckTags, State = #q{consumers = Consumers}, Fun) ->
run_message_queue(true, Fun(State1))
end.
message_properties(Message = #basic_message{content = Content},
Confirm, #q{ttl = TTL}) ->
#content{payload_fragments_rev = PFR} = Content,
message_properties(Message, Confirm, #q{ttl = TTL}) ->
{_, Size} = mc:size(Message),
#message_properties{expiry = calculate_msg_expiry(Message, TTL),
needs_confirming = Confirm == eventually,
size = iolist_size(PFR)}.
size = Size}.
calculate_msg_expiry(#basic_message{content = Content}, TTL) ->
#content{properties = Props} =
rabbit_binary_parser:ensure_content_decoded(Content),
%% We assert that the expiration must be valid - we check in the channel.
{ok, MsgTTL} = rabbit_basic:parse_expiration(Props),
calculate_msg_expiry(Msg, TTL) ->
MsgTTL = mc:ttl(Msg),
case lists:min([TTL, MsgTTL]) of
undefined -> undefined;
T -> os:system_time(microsecond) + T * 1000
T ->
os:system_time(microsecond) + T * 1000
end.
%% Logically this function should invoke maybe_send_drained/2.

16
deps/rabbit/src/rabbit_backing_queue.erl vendored
View File

@ -23,12 +23,12 @@
-type flow() :: 'flow' | 'noflow'.
-type msg_ids() :: [rabbit_types:msg_id()].
-type publish() :: {rabbit_types:basic_message(),
-type publish() :: {mc:state(),
rabbit_types:message_properties(), boolean()}.
-type delivered_publish() :: {rabbit_types:basic_message(),
-type delivered_publish() :: {mc:state(),
rabbit_types:message_properties()}.
-type fetch_result(Ack) ::
('empty' | {rabbit_types:basic_message(), boolean(), Ack}).
('empty' | {mc:state(), boolean(), Ack}).
-type drop_result(Ack) ::
('empty' | {rabbit_types:msg_id(), Ack}).
-type recovery_terms() :: [term()] | 'non_clean_shutdown'.
@ -38,7 +38,7 @@
fun ((atom(), fun ((atom(), state()) -> state())) -> 'ok').
-type duration() :: ('undefined' | 'infinity' | number()).
-type msg_fun(A) :: fun ((rabbit_types:basic_message(), ack(), A) -> A).
-type msg_fun(A) :: fun ((mc:state(), ack(), A) -> A).
-type msg_pred() :: fun ((rabbit_types:message_properties()) -> boolean()).
-type queue_mode() :: atom().
@ -95,7 +95,7 @@
-callback purge_acks(state()) -> state().
%% Publish a message.
-callback publish(rabbit_types:basic_message(),
-callback publish(mc:state(),
rabbit_types:message_properties(), boolean(), pid(), flow(),
state()) -> state().
@ -105,7 +105,7 @@
%% Called for messages which have already been passed straight
%% out to a client. The queue will be empty for these calls
%% (i.e. saves the round trip through the backing queue).
-callback publish_delivered(rabbit_types:basic_message(),
-callback publish_delivered(mc:state(),
rabbit_types:message_properties(), pid(), flow(),
state())
-> {ack(), state()}.
@ -187,7 +187,7 @@
%% Fold over all the messages in a queue and return the accumulated
%% results, leaving the queue undisturbed.
-callback fold(fun((rabbit_types:basic_message(),
-callback fold(fun((mc:state(),
rabbit_types:message_properties(),
boolean(), A) -> {('stop' | 'cont'), A}),
A, state()) -> {A, state()}.
@ -246,7 +246,7 @@
%% Called prior to a publish or publish_delivered call. Allows the BQ
%% to signal that it's already seen this message, (e.g. it was published
%% or discarded previously) specifying whether to drop the message or reject it.
-callback is_duplicate(rabbit_types:basic_message(), state())
-callback is_duplicate(mc:state(), state())
-> {{true, drop} | {true, reject} | boolean(), state()}.
-callback set_queue_mode(queue_mode(), state()) -> state().

86
deps/rabbit/src/rabbit_basic.erl vendored
View File

@ -9,76 +9,41 @@
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-export([publish/4, publish/5, publish/1,
message/3, message_no_id/3, message/4, properties/1, prepend_table_header/3,
extract_headers/1, extract_timestamp/1, map_headers/2, delivery/4,
-export([message/3,
message_no_id/3,
message/4,
properties/1,
prepend_table_header/3,
extract_headers/1,
extract_timestamp/1,
map_headers/2,
delivery/4,
delivery/5,
header_routes/1, parse_expiration/1, header/2, header/3]).
-export([build_content/2, from_content/1, msg_size/1,
maybe_gc_large_msg/1, maybe_gc_large_msg/2]).
-export([add_header/4,
peek_fmt_message/1]).
-export([strip_bcc_header/1,
add_header/4,
peek_fmt_message/3]).
%%----------------------------------------------------------------------------
-type properties_input() ::
rabbit_framing:amqp_property_record() | [{atom(), any()}].
-type publish_result() ::
ok | rabbit_types:error('not_found').
-type header() :: any().
-type headers() :: rabbit_framing:amqp_table() | 'undefined'.
-type exchange_input() :: rabbit_types:exchange() | rabbit_exchange:name().
-type body_input() :: binary() | [binary()].
%%----------------------------------------------------------------------------
%% Convenience function, for avoiding round-trips in calls across the
%% erlang distributed network.
-spec publish
(exchange_input(), rabbit_router:routing_key(), properties_input(),
body_input()) ->
publish_result().
publish(Exchange, RoutingKeyBin, Properties, Body) ->
publish(Exchange, RoutingKeyBin, false, Properties, Body).
%% Convenience function, for avoiding round-trips in calls across the
%% erlang distributed network.
-spec publish
(exchange_input(), rabbit_router:routing_key(), boolean(),
properties_input(), body_input()) ->
publish_result().
publish(X = #exchange{name = XName}, RKey, Mandatory, Props, Body) ->
Message = message(XName, RKey, properties(Props), Body),
publish(X, delivery(Mandatory, false, Message, undefined));
publish(XName, RKey, Mandatory, Props, Body) ->
Message = message(XName, RKey, properties(Props), Body),
publish(delivery(Mandatory, false, Message, undefined)).
-spec publish(rabbit_types:delivery()) -> publish_result().
publish(Delivery = #delivery{
message = #basic_message{exchange_name = XName}}) ->
case rabbit_exchange:lookup(XName) of
{ok, X} -> publish(X, Delivery);
Err -> Err
end.
publish(X, Delivery) ->
Qs = rabbit_amqqueue:lookup_many(rabbit_exchange:route(X, Delivery)),
_ = rabbit_queue_type:deliver(Qs, Delivery, stateless),
ok.
-spec delivery
(boolean(), boolean(), rabbit_types:message(), undefined | integer()) ->
(boolean(), boolean(), mc:state(), undefined | integer()) ->
rabbit_types:delivery().
delivery(Mandatory, Confirm, Message, MsgSeqNo) ->
delivery(Mandatory, Confirm, Message, MsgSeqNo, noflow).
delivery(Mandatory, Confirm, Message, MsgSeqNo, Flow) ->
#delivery{mandatory = Mandatory, confirm = Confirm, sender = self(),
message = Message, msg_seq_no = MsgSeqNo, flow = noflow}.
message = Message, msg_seq_no = MsgSeqNo, flow = Flow}.
-spec build_content
(rabbit_framing:amqp_property_record(), binary() | [binary()]) ->
@ -111,6 +76,9 @@ from_content(Content) ->
rabbit_framing_amqp_0_9_1:method_id('basic.publish'),
{Props, list_to_binary(lists:reverse(FragmentsRev))}.
strip_bcc_header(#content{} = Content) ->
strip_header(Content, ?DELETED_HEADER).
%% This breaks the spec rule forbidding message modification
strip_header(#content{properties = #'P_basic'{headers = undefined}}
= DecodedContent, _Key) ->
@ -146,7 +114,6 @@ message(XName, RoutingKey, Content) ->
(rabbit_exchange:name(), rabbit_router:routing_key(), properties_input(),
binary()) ->
rabbit_types:message().
message(XName, RoutingKey, RawProperties, Body) ->
Properties = properties(RawProperties),
Content = build_content(Properties, Body),
@ -288,6 +255,8 @@ header_routes(HeadersTable) ->
parse_expiration(#'P_basic'{expiration = undefined}) ->
{ok, undefined};
parse_expiration(#'P_basic'{expiration = Expiration}) ->
parse_expiration(Expiration);
parse_expiration(Expiration) when is_binary(Expiration) ->
case string:to_integer(binary_to_list(Expiration)) of
{error, no_integer} = E ->
E;
@ -320,11 +289,8 @@ add_header(Name, Type, Value, #basic_message{content = Content0} = Msg) ->
end, Content0),
Msg#basic_message{content = Content}.
peek_fmt_message(#basic_message{exchange_name = Ex,
routing_keys = RKeys,
content =
#content{payload_fragments_rev = Payl0,
properties = Props}}) ->
peek_fmt_message(Ex, RKeys, #content{payload_fragments_rev = Payl0,
properties = Props}) ->
Fields = [atom_to_binary(F, utf8) || F <- record_info(fields, 'P_basic')],
T = lists:zip(Fields, tl(tuple_to_list(Props))),
lists:foldl(
@ -342,7 +308,7 @@ peek_fmt_message(#basic_message{exchange_name = Ex,
end, [], [{<<"payload (max 64 bytes)">>,
%% restric payload to 64 bytes
binary_prefix_64(iolist_to_binary(lists:reverse(Payl0)), 64)},
{<<"exchange">>, Ex#resource.name},
{<<"exchange">>, Ex},
{<<"routing_keys">>, RKeys} | T]).
header_key(A) ->

277
deps/rabbit/src/rabbit_channel.erl vendored
View File

@ -68,6 +68,8 @@
%% Mgmt HTTP API refactor
-export([handle_method/6]).
-import(rabbit_misc, [maps_put_truthy/3]).
-record(conf, {
%% starting | running | flow | closing
state,
@ -300,22 +302,25 @@ shutdown(Pid) ->
send_command(Pid, Msg) ->
gen_server2:cast(Pid, {command, Msg}).
-spec deliver_reply(binary(), rabbit_types:delivery()) -> 'ok'.
deliver_reply(<<"amq.rabbitmq.reply-to.", EncodedBin/binary>>, Delivery) ->
case rabbit_direct_reply_to:decode_reply_to_v2(EncodedBin, rabbit_nodes:all_running_with_hashes()) of
-spec deliver_reply(binary(), mc:state()) -> 'ok'.
deliver_reply(<<"amq.rabbitmq.reply-to.", EncodedBin/binary>>, Message) ->
case rabbit_direct_reply_to:decode_reply_to_v2(EncodedBin,
rabbit_nodes:all_running_with_hashes()) of
{ok, Pid, Key} ->
delegate:invoke_no_result(Pid, {?MODULE, deliver_reply_local, [Key, Delivery]});
delegate:invoke_no_result(Pid, {?MODULE, deliver_reply_local,
[Key, Message]});
{error, _} ->
deliver_reply_v1(EncodedBin, Delivery)
deliver_reply_v1(EncodedBin, Message)
end.
-spec deliver_reply_v1(binary(), rabbit_types:delivery()) -> 'ok'.
deliver_reply_v1(EncodedBin, Delivery) ->
-spec deliver_reply_v1(binary(), mc:state()) -> 'ok'.
deliver_reply_v1(EncodedBin, Message) ->
%% the the original encoding function
case rabbit_direct_reply_to:decode_reply_to_v1(EncodedBin) of
{ok, V1Pid, V1Key} ->
delegate:invoke_no_result(V1Pid, {?MODULE, deliver_reply_local, [V1Key, Delivery]});
delegate:invoke_no_result(V1Pid,
{?MODULE, deliver_reply_local, [V1Key, Message]});
{error, _} ->
ok
end.
@ -323,11 +328,11 @@ deliver_reply_v1(EncodedBin, Delivery) ->
%% We want to ensure people can't use this mechanism to send a message
%% to an arbitrary process and kill it!
-spec deliver_reply_local(pid(), binary(), rabbit_types:delivery()) -> 'ok'.
-spec deliver_reply_local(pid(), binary(), mc:state()) -> 'ok'.
deliver_reply_local(Pid, Key, Delivery) ->
deliver_reply_local(Pid, Key, Message) ->
case pg_local:in_group(rabbit_channels, Pid) of
true -> gen_server2:cast(Pid, {deliver_reply, Key, Delivery});
true -> gen_server2:cast(Pid, {deliver_reply, Key, Message});
false -> ok
end.
@ -676,22 +681,22 @@ handle_cast({command, Msg}, State) ->
handle_cast({deliver_reply, _K, _Del},
State = #ch{cfg = #conf{state = closing}}) ->
noreply(State);
handle_cast({deliver_reply, _K, _Del}, State = #ch{reply_consumer = none}) ->
handle_cast({deliver_reply, _K, _Msg}, State = #ch{reply_consumer = none}) ->
noreply(State);
handle_cast({deliver_reply, Key, #delivery{message =
#basic_message{exchange_name = ExchangeName,
routing_keys = [RoutingKey | _CcRoutes],
content = Content}}},
handle_cast({deliver_reply, Key, Msg},
State = #ch{cfg = #conf{writer_pid = WriterPid},
next_tag = DeliveryTag,
reply_consumer = {ConsumerTag, _Suffix, Key}}) ->
Content = mc:protocol_state(mc:convert(mc_amqpl, Msg)),
ExchName = mc:get_annotation(exchange, Msg),
[RoutingKey | _] = mc:get_annotation(routing_keys, Msg),
ok = rabbit_writer:send_command(
WriterPid,
#'basic.deliver'{consumer_tag = ConsumerTag,
delivery_tag = DeliveryTag,
redelivered = false,
exchange = ExchangeName#resource.name,
routing_key = RoutingKey},
redelivered = false,
exchange = ExchName,
routing_key = RoutingKey},
Content),
noreply(State);
handle_cast({deliver_reply, _K1, _}, State=#ch{reply_consumer = {_, _, _K2}}) ->
@ -1271,38 +1276,43 @@ handle_method(#'basic.publish'{exchange = ExchangeNameBin,
check_write_permitted_on_topic(Exchange, User, RoutingKey, AuthzContext),
%% We decode the content's properties here because we're almost
%% certain to want to look at delivery-mode and priority.
DecodedContent0 = #content {properties = Props} =
DecodedContent = #content {properties = Props} =
maybe_set_fast_reply_to(
rabbit_binary_parser:ensure_content_decoded(Content), State),
DecodedContent = rabbit_message_interceptor:intercept(DecodedContent0),
check_user_id_header(Props, State),
check_expiration_header(Props),
DoConfirm = Tx =/= none orelse ConfirmEnabled,
{MsgSeqNo, State1} =
{DeliveryOptions, SeqNum, State1} =
case DoConfirm of
false -> {undefined, State0};
true -> rabbit_global_counters:messages_received_confirm(amqp091, 1),
SeqNo = State0#ch.publish_seqno,
{SeqNo, State0#ch{publish_seqno = SeqNo + 1}}
false ->
{maps_put_truthy(flow, Flow, #{}), undefined, State0};
true ->
rabbit_global_counters:messages_received_confirm(amqp091, 1),
SeqNo = State0#ch.publish_seqno,
Opts = maps_put_truthy(flow, Flow, #{correlation => SeqNo}),
{Opts, SeqNo, State0#ch{publish_seqno = SeqNo + 1}}
end,
case rabbit_basic:message_no_id(ExchangeName, RoutingKey, DecodedContent) of
{ok, Message} ->
Delivery = rabbit_basic:delivery(
Mandatory, DoConfirm, Message, MsgSeqNo),
QNames = rabbit_exchange:route(Exchange, Delivery, #{return_binding_keys => true}),
rabbit_trace:tap_in(Message, QNames, ConnName, ChannelNum,
Username, TraceState),
DQ = {Delivery#delivery{flow = Flow}, QNames},
{noreply, case Tx of
none ->
deliver_to_queues(DQ, State1);
{Msgs, Acks} ->
Msgs1 = ?QUEUE:in(DQ, Msgs),
State1#ch{tx = {Msgs1, Acks}}
end};
{error, Reason} ->
precondition_failed("invalid message: ~tp", [Reason])
end;
% rabbit_feature_flags:is_enabled(message_containers),
Message0 = mc_amqpl:message(ExchangeName,
RoutingKey,
DecodedContent),
Message = rabbit_message_interceptor:intercept(Message0),
QNames = rabbit_exchange:route(Exchange, Message, #{return_binding_keys => true}),
[rabbit_channel:deliver_reply(RK, Message) ||
{virtual_reply_queue, RK} <- QNames],
Queues = rabbit_amqqueue:lookup_many(QNames),
ok = process_routing_mandatory(Mandatory, Queues, SeqNum, Message, ExchangeName, State0),
rabbit_trace:tap_in(Message, QNames, ConnName, ChannelNum,
Username, TraceState),
%% TODO: call delivery_to_queues with plain args
Delivery = {Message, DeliveryOptions, Queues},
{noreply, case Tx of
none ->
deliver_to_queues(ExchangeName, Delivery, State1);
{Msgs, Acks} ->
Msgs1 = ?QUEUE:in(Delivery, Msgs),
State1#ch{tx = {Msgs1, Acks}}
end};
handle_method(#'basic.nack'{delivery_tag = DeliveryTag,
multiple = Multiple,
@ -1573,13 +1583,15 @@ handle_method(#'basic.recover_async'{requeue = true},
queue_states = QueueStates0}) ->
OkFun = fun () -> ok end,
UAMQL = ?QUEUE:to_list(UAMQ),
{QueueStates, Actions} =
foreach_per_queue(
fun ({QPid, CTag}, MsgIds, {Acc0, Actions0}) ->
fun ({QRef, CTag}, MsgIds, {Acc0, Actions0}) ->
rabbit_misc:with_exit_handler(
OkFun,
fun () ->
{ok, Acc, Act} = rabbit_amqqueue:requeue(QPid, {CTag, MsgIds}, Acc0),
{ok, Acc, Act} = rabbit_queue_type:settle(
QRef, requeue, CTag, MsgIds, Acc0),
{Acc, Act ++ Actions0}
end)
end, lists:reverse(UAMQL), {QueueStates0, []}),
@ -1914,20 +1926,17 @@ binding_action(Fun, SourceNameBin0, DestinationType, DestinationNameBin0,
ok
end.
basic_return(Content, #basic_message{exchange_name = ExchangeName,
routing_keys = [RoutingKey | _CcRoutes]},
State = #ch{cfg = #conf{protocol = Protocol,
writer_pid = WriterPid}},
basic_return(Content, RoutingKey, XNameBin,
#ch{cfg = #conf{protocol = Protocol,
writer_pid = WriterPid}},
Reason) ->
?INCR_STATS(exchange_stats, ExchangeName, 1, return_unroutable, State),
{_Close, ReplyCode, ReplyText} = Protocol:lookup_amqp_exception(Reason),
ok = rabbit_writer:send_command(
WriterPid,
#'basic.return'{reply_code = ReplyCode,
reply_text = ReplyText,
exchange = ExchangeName#resource.name,
routing_key = RoutingKey},
Content).
ok = rabbit_writer:send_command(WriterPid,
#'basic.return'{reply_code = ReplyCode,
reply_text = ReplyText,
exchange = XNameBin,
routing_key = RoutingKey},
Content).
reject(DeliveryTag, Requeue, Multiple,
State = #ch{unacked_message_q = UAMQ, tx = Tx}) ->
@ -2139,38 +2148,41 @@ notify_limiter(Limiter, Acked) ->
end
end.
deliver_to_queues({#delivery{message = #basic_message{exchange_name = XName},
confirm = false,
mandatory = false},
_RoutedToQueueNames = []}, State) -> %% optimisation when there are no queues
deliver_to_queues({Message, _Options, _RoutedToQueues = []} = Delivery,
#ch{cfg = #conf{virtual_host = VHost}} = State) ->
XNameBin = mc:get_annotation(exchange, Message),
XName = rabbit_misc:r(VHost, exchange, XNameBin),
deliver_to_queues(XName, Delivery, State).
deliver_to_queues(XName,
{_Message, Options, _RoutedToQueues = []},
State)
when not is_map_key(correlation, Options) -> %% optimisation when there are no queues
?INCR_STATS(exchange_stats, XName, 1, publish, State),
rabbit_global_counters:messages_unroutable_dropped(amqp091, 1),
?INCR_STATS(exchange_stats, XName, 1, drop_unroutable, State),
State;
deliver_to_queues({Delivery = #delivery{message = Message = #basic_message{exchange_name = XName},
mandatory = Mandatory,
confirm = Confirm,
msg_seq_no = MsgSeqNo},
RoutedToQueueNames = [QName]},
State0 = #ch{cfg = #conf{extended_return_callback = ExtendedReturnCallback},
queue_states = QueueStates0}) -> %% optimisation when there is one queue
Qs0 = rabbit_amqqueue:lookup_many(RoutedToQueueNames),
Qs = rabbit_amqqueue:prepend_extra_bcc(Qs0),
case rabbit_queue_type:deliver(Qs, Delivery, QueueStates0) of
{ok, QueueStates, Actions} ->
rabbit_global_counters:messages_routed(amqp091, erlang:min(1, length(Qs))),
deliver_to_queues(XName,
{Message, Options, RoutedToQueues},
State0 = #ch{queue_states = QueueStates0}) ->
Qs = rabbit_amqqueue:prepend_extra_bcc(RoutedToQueues),
case rabbit_queue_type:deliver(Qs, Message, Options, QueueStates0) of
{ok, QueueStates, Actions} ->
rabbit_global_counters:messages_routed(amqp091, length(Qs)),
QueueNames = rabbit_amqqueue:queue_names(Qs),
MsgSeqNo = maps:get(correlation, Options, undefined),
%% NB: the order here is important since basic.returns must be
%% sent before confirms.
ok = process_routing_mandatory(ExtendedReturnCallback, Mandatory, Qs, MsgSeqNo, Message, State0),
QueueNames = rabbit_amqqueue:queue_names(Qs),
State1 = process_routing_confirm(Confirm, QueueNames, MsgSeqNo, XName, State0),
State1 = process_routing_confirm(MsgSeqNo, QueueNames, XName, State0),
%% Actions must be processed after registering confirms as actions may
%% contain rejections of publishes
State = handle_queue_actions(Actions, State1#ch{queue_states = QueueStates}),
case rabbit_event:stats_level(State, #ch.stats_timer) of
fine ->
?INCR_STATS(exchange_stats, XName, 1, publish),
?INCR_STATS(queue_exchange_stats, {rabbit_amqqueue:queue_name(QName), XName}, 1, publish);
lists:foreach(fun(QName) ->
?INCR_STATS(queue_exchange_stats, {QName, XName}, 1, publish)
end, QueueNames);
_ ->
ok
end,
@ -2185,79 +2197,47 @@ deliver_to_queues({Delivery = #delivery{message = Message = #basic_message{ex
resource_error,
"Stream coordinator unavailable for ~ts",
[rabbit_misc:rs(Resource)])
end;
deliver_to_queues({Delivery = #delivery{message = Message = #basic_message{exchange_name = XName},
mandatory = Mandatory,
confirm = Confirm,
msg_seq_no = MsgSeqNo},
RoutedToQueueNames},
State0 = #ch{cfg = #conf{extended_return_callback = ExtendedReturnCallback},
queue_states = QueueStates0}) ->
Qs0 = rabbit_amqqueue:lookup_many(RoutedToQueueNames),
Qs = rabbit_amqqueue:prepend_extra_bcc(Qs0),
case rabbit_queue_type:deliver(Qs, Delivery, QueueStates0) of
{ok, QueueStates, Actions} ->
rabbit_global_counters:messages_routed(amqp091, length(Qs)),
%% NB: the order here is important since basic.returns must be
%% sent before confirms.
ok = process_routing_mandatory(ExtendedReturnCallback, Mandatory, Qs, MsgSeqNo, Message, State0),
QueueNames = rabbit_amqqueue:queue_names(Qs),
State1 = process_routing_confirm(Confirm, QueueNames,
MsgSeqNo, XName, State0),
%% Actions must be processed after registering confirms as actions may
%% contain rejections of publishes
State = handle_queue_actions(Actions, State1#ch{queue_states = QueueStates}),
_ = case rabbit_event:stats_level(State, #ch.stats_timer) of
fine ->
?INCR_STATS(exchange_stats, XName, 1, publish),
[?INCR_STATS(queue_exchange_stats, {QName, XName}, 1, publish)
|| QName <- QueueNames];
_ ->
ok
end,
State;
{error, {coordinator_unavailable, Resource}} ->
rabbit_misc:protocol_error(
resource_error,
"Stream coordinator unavailable for ~ts",
[rabbit_misc:rs(Resource)])
end.
process_routing_mandatory(_ExtendedReturnCallback = false,
_Mandatory = true,
_RoutedToQs = [],
_MsgSeqNo,
#basic_message{content = Content} = Msg, State) ->
rabbit_global_counters:messages_unroutable_returned(amqp091, 1),
ok = basic_return(Content, Msg, State, no_route),
ok;
process_routing_mandatory(_ExtendedReturnCallback = true,
_Mandatory = true,
process_routing_mandatory(_Mandatory = true,
_RoutedToQs = [],
MsgSeqNo,
#basic_message{content = Content} = Msg, State) ->
Msg,
XName,
State = #ch{cfg = #conf{extended_return_callback = ExtRetCallback}}) ->
rabbit_global_counters:messages_unroutable_returned(amqp091, 1),
%% providing the publishing sequence for AMQP 1.0
ok = basic_return({MsgSeqNo, Content}, Msg, State, no_route),
ok;
process_routing_mandatory(_ExtendedReturnCallback,
_Mandatory = false,
?INCR_STATS(exchange_stats, XName, 1, return_unroutable, State),
Content0 = mc:protocol_state(Msg),
Content = case ExtRetCallback of
true ->
%% providing the publishing sequence for AMQP 1.0
{MsgSeqNo, Content0};
false ->
Content0
end,
[RoutingKey | _] = mc:get_annotation(routing_keys, Msg),
ok = basic_return(Content, RoutingKey, XName#resource.name, State, no_route);
process_routing_mandatory(_Mandatory = false,
_RoutedToQs = [],
_MsgSeqNo,
#basic_message{exchange_name = ExchangeName}, State) ->
_Msg,
XName,
State) ->
rabbit_global_counters:messages_unroutable_dropped(amqp091, 1),
?INCR_STATS(exchange_stats, ExchangeName, 1, drop_unroutable, State),
?INCR_STATS(exchange_stats, XName, 1, drop_unroutable, State),
ok;
process_routing_mandatory(_, _, _, _, _, _) ->
ok.
process_routing_confirm(false, _, _, _, State) ->
process_routing_confirm(undefined, _, _, State) ->
State;
process_routing_confirm(true, [], MsgSeqNo, XName, State) ->
process_routing_confirm(MsgSeqNo, [], XName, State)
when is_integer(MsgSeqNo) ->
record_confirms([{MsgSeqNo, XName}], State);
process_routing_confirm(true, QRefs, MsgSeqNo, XName, State) ->
process_routing_confirm(MsgSeqNo, QRefs, XName, State)
when is_integer(MsgSeqNo) ->
State#ch{unconfirmed =
rabbit_confirms:insert(MsgSeqNo, QRefs, XName, State#ch.unconfirmed)}.
rabbit_confirms:insert(MsgSeqNo, QRefs, XName, State#ch.unconfirmed)}.
confirm(MsgSeqNos, QRef, State = #ch{unconfirmed = UC}) ->
%% NOTE: if queue name does not exist here it's likely that the ref also
@ -2739,18 +2719,19 @@ handle_deliver(CTag, Ack, Msgs, State) when is_list(Msgs) ->
end, State, Msgs).
handle_deliver0(ConsumerTag, AckRequired,
Msg = {QName, QPid, _MsgId, Redelivered,
#basic_message{exchange_name = ExchangeName,
routing_keys = [RoutingKey | _CcRoutes],
content = Content}},
{QName, QPid, _MsgId, Redelivered, MsgCont0} = Msg,
State = #ch{cfg = #conf{writer_pid = WriterPid,
writer_gc_threshold = GCThreshold},
next_tag = DeliveryTag,
queue_states = Qs}) ->
[RoutingKey | _] = mc:get_annotation(routing_keys, MsgCont0),
ExchangeNameBin = mc:get_annotation(exchange, MsgCont0),
MsgCont = mc:convert(mc_amqpl, MsgCont0),
Content = mc:protocol_state(MsgCont),
Deliver = #'basic.deliver'{consumer_tag = ConsumerTag,
delivery_tag = DeliveryTag,
redelivered = Redelivered,
exchange = ExchangeName#resource.name,
exchange = ExchangeNameBin,
routing_key = RoutingKey},
{ok, QueueType} = rabbit_queue_type:module(QName, Qs),
case QueueType of
@ -2767,20 +2748,21 @@ handle_deliver0(ConsumerTag, AckRequired,
record_sent(deliver, QueueType, ConsumerTag, AckRequired, Msg, State).
handle_basic_get(WriterPid, DeliveryTag, NoAck, MessageCount,
Msg = {_QName, _QPid, _MsgId, Redelivered,
#basic_message{exchange_name = ExchangeName,
routing_keys = [RoutingKey | _CcRoutes],
content = Content}},
Msg0 = {_QName, _QPid, _MsgId, Redelivered, MsgCont0},
QueueType, State) ->
[RoutingKey | _] = mc:get_annotation(routing_keys, MsgCont0),
ExchangeName = mc:get_annotation(exchange, MsgCont0),
MsgCont = mc:convert(mc_amqpl, MsgCont0),
Content = mc:protocol_state(MsgCont),
ok = rabbit_writer:send_command(
WriterPid,
#'basic.get_ok'{delivery_tag = DeliveryTag,
redelivered = Redelivered,
exchange = ExchangeName#resource.name,
exchange = ExchangeName,
routing_key = RoutingKey,
message_count = MessageCount},
Content),
{noreply, record_sent(get, QueueType, DeliveryTag, not(NoAck), Msg, State)}.
{noreply, record_sent(get, QueueType, DeliveryTag, not(NoAck), Msg0, State)}.
init_tick_timer(State = #ch{tick_timer = undefined}) ->
{ok, Interval} = application:get_env(rabbit, channel_tick_interval),
@ -2916,3 +2898,4 @@ maybe_decrease_global_publishers(#ch{publishing_mode = true}) ->
is_global_qos_permitted() ->
rabbit_deprecated_features:is_permitted(global_qos).

21
deps/rabbit/src/rabbit_classic_queue.erl vendored
View File

@ -38,7 +38,7 @@
consume/3,
cancel/5,
handle_event/3,
deliver/2,
deliver/3,
settle/5,
credit/5,
dequeue/5,
@ -326,18 +326,21 @@ settlement_action(Type, QRef, MsgSeqs, Acc) ->
[{Type, QRef, MsgSeqs} | Acc].
-spec deliver([{amqqueue:amqqueue(), state()}],
Delivery :: term()) ->
Delivery :: mc:state(),
rabbit_queue_type:delivery_options()) ->
{[{amqqueue:amqqueue(), state()}], rabbit_queue_type:actions()}.
deliver(Qs0, #delivery{flow = Flow,
msg_seq_no = MsgNo,
message = #basic_message{} = Msg0,
confirm = Confirm} = Delivery0) ->
deliver(Qs0, Msg0, Options) ->
%% add guid to content here instead of in rabbit_basic:message/3,
%% as classic queues are the only ones that need it
Msg = Msg0#basic_message{id = rabbit_guid:gen()},
Delivery = Delivery0#delivery{message = Msg},
Msg = mc:prepare(store, mc:set_annotation(id, rabbit_guid:gen(), Msg0)),
Mandatory = maps:get(mandatory, Options, false),
MsgSeqNo = maps:get(correlation, Options, undefined),
Flow = maps:get(flow, Options, noflow),
Confirm = MsgSeqNo /= undefined,
{MPids, SPids, Qs} = qpids(Qs0, Confirm, MsgSeqNo),
Delivery = rabbit_basic:delivery(Mandatory, Confirm, Msg, MsgSeqNo, Flow),
{MPids, SPids, Qs} = qpids(Qs0, Confirm, MsgNo),
case Flow of
%% Here we are tracking messages sent by the rabbit_channel
%% process. We are accessing the rabbit_channel process

8
deps/rabbit/src/rabbit_classic_queue_store_v2.erl vendored
View File

@ -71,7 +71,7 @@
-type buffer() :: #{
%% SeqId => {Offset, Size, Msg}
rabbit_variable_queue:seq_id() => {non_neg_integer(), non_neg_integer(), #basic_message{}}
rabbit_variable_queue:seq_id() => {non_neg_integer(), non_neg_integer(), mc:state()}
}.
-record(qs, {
@ -142,7 +142,7 @@ maybe_close_fd(Fd) ->
info(#qs{ write_buffer = WriteBuffer }) ->
[{qs_buffer_size, map_size(WriteBuffer)}].
-spec write(rabbit_variable_queue:seq_id(), rabbit_types:basic_message(),
-spec write(rabbit_variable_queue:seq_id(), mc:state(),
rabbit_types:message_properties(), State)
-> {msg_location(), State} when State::state().
@ -283,7 +283,7 @@ build_data({_, Size, Msg}, CheckCRC32) ->
].
-spec read(rabbit_variable_queue:seq_id(), msg_location(), State)
-> {rabbit_types:basic_message(), State} when State::state().
-> {mc:state(), State} when State::state().
read(SeqId, DiskLocation, State = #qs{ write_buffer = WriteBuffer,
cache = Cache }) ->
@ -328,7 +328,7 @@ read_from_disk(SeqId, {?MODULE, Offset, Size}, State0) ->
{Msg, State}.
-spec read_many([{rabbit_variable_queue:seq_id(), msg_location()}], State)
-> {[rabbit_types:basic_message()], State} when State::state().
-> {[mc:state()], State} when State::state().
read_many([], State) ->
{[], State};

10
deps/rabbit/src/rabbit_core_ff.erl vendored
View File

@ -99,13 +99,12 @@
-rabbit_feature_flag(
{restart_streams,
#{desc => "Support for restarting streams with optional preferred next leader argument. "
#{desc => "Support for restarting streams with optional preferred next leader argument."
"Used to implement stream leader rebalancing",
stability => stable,
depends_on => [stream_queue]
}}).
-rabbit_feature_flag(
{stream_sac_coordinator_unblock_group,
#{desc => "Bug fix to unblock a group of consumers in a super stream partition",
@ -120,3 +119,10 @@
stability => stable,
depends_on => [stream_queue]
}}).
-rabbit_feature_flag(
{message_containers,
#{desc => "Message containers.",
stability => stable,
depends_on => [feature_flags_v2]
}}).

281
deps/rabbit/src/rabbit_dead_letter.erl vendored
View File

@ -8,261 +8,68 @@
-module(rabbit_dead_letter).
-export([publish/5,
make_msg/5,
detect_cycles/3]).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
%%----------------------------------------------------------------------------
-type reason() :: 'expired' | 'rejected' | 'maxlen' | delivery_limit.
-type reason() :: expired | rejected | maxlen | delivery_limit.
-export_type([reason/0]).
%%----------------------------------------------------------------------------
-spec publish(rabbit_types:message(), reason(), rabbit_types:exchange(),
'undefined' | binary(), rabbit_amqqueue:name()) -> 'ok'.
publish(Msg, Reason, X, RK, SourceQName) ->
DLMsg = make_msg(Msg, Reason, X#exchange.name, RK, SourceQName),
Delivery = rabbit_basic:delivery(false, false, DLMsg, undefined),
QNames0 = rabbit_exchange:route(X, Delivery, #{return_binding_keys => true}),
{QNames, Cycles} = detect_cycles(Reason, DLMsg, QNames0),
-spec publish(mc:state(), reason(), rabbit_types:exchange(),
undefined | rabbit_types:routing_key(), rabbit_amqqueue:name()) ->
ok.
publish(Msg0, Reason, #exchange{name = XName} = DLX, RK,
#resource{name = SourceQName}) ->
DLRKeys = case RK of
undefined ->
mc:get_annotation(routing_keys, Msg0);
_ ->
[RK]
end,
Msg1 = mc:record_death(Reason, SourceQName, Msg0),
{Ttl, Msg2} = mc:take_annotation(dead_letter_ttl, Msg1),
Msg3 = mc:set_ttl(Ttl, Msg2),
Msg4 = mc:set_annotation(routing_keys, DLRKeys, Msg3),
DLMsg = mc:set_annotation(exchange, XName#resource.name, Msg4),
Routed = rabbit_exchange:route(DLX, DLMsg, #{return_binding_keys => true}),
{QNames, Cycles} = detect_cycles(Reason, DLMsg, Routed),
lists:foreach(fun log_cycle_once/1, Cycles),
Qs0 = rabbit_amqqueue:lookup_many(QNames),
Qs = rabbit_amqqueue:prepend_extra_bcc(Qs0),
_ = rabbit_queue_type:deliver(Qs, Delivery, stateless),
_ = rabbit_queue_type:deliver(Qs, DLMsg, #{}, stateless),
ok.
make_msg(Msg = #basic_message{content = Content,
exchange_name = Exchange,
routing_keys = RoutingKeys},
Reason, DLX, RK, #resource{name = QName}) ->
#content{properties = #'P_basic'{headers = Headers} = Props} = Content1 =
rabbit_binary_parser:ensure_content_decoded(Content),
Headers1 = if Headers =:= undefined -> [];
is_list(Headers) -> Headers
end,
ReasonBin = atom_to_binary(Reason),
TimeSec = os:system_time(second),
PerMsgTTL = per_msg_ttl_header(Props),
%% The first routing key is the one specified in the
%% basic.publish; all others are CC or BCC keys.
RKs = [hd(RoutingKeys) | rabbit_basic:header_routes(Headers1)],
RKs1 = [{longstr, Key} || Key <- RKs],
Info = [{<<"reason">>, longstr, ReasonBin},
{<<"queue">>, longstr, QName},
{<<"time">>, timestamp, TimeSec},
{<<"exchange">>, longstr, Exchange#resource.name},
{<<"routing-keys">>, array, RKs1}] ++ PerMsgTTL,
{DeathRoutingKeys, Headers2} =
case RK of
undefined -> {RoutingKeys, Headers1};
_ -> {[RK], lists:keydelete(<<"CC">>, 1, Headers1)}
end,
Headers3 = update_x_death_header(Info, Headers2),
{Expiration, Headers5} =
case lists:keytake(<<"x-dead-letter-expiration">>, 1, Headers3) of
false ->
{undefined, Headers3};
{value, {<<"x-dead-letter-expiration">>, longstr, Expiration0}, Headers4} ->
{Expiration0, Headers4}
end,
Props1 = Props#'P_basic'{headers = Headers5,
expiration = Expiration},
Content2 = rabbit_binary_generator:clear_encoded_content(
Content1#content{properties = Props1}),
Msg#basic_message{exchange_name = DLX,
id = rabbit_guid:gen(),
routing_keys = DeathRoutingKeys,
content = Content2}.
x_death_event_key(Info, Key) ->
case lists:keysearch(Key, 1, Info) of
false -> undefined;
{value, {Key, _KeyType, Val}} -> Val
end.
maybe_append_to_event_group(Table, _Key, _SeenKeys, []) ->
[Table];
maybe_append_to_event_group(Table, {_Queue, _Reason} = Key, SeenKeys, Acc) ->
case sets:is_element(Key, SeenKeys) of
true -> Acc;
false -> [Table | Acc]
end.
group_by_queue_and_reason([]) ->
[];
group_by_queue_and_reason([Table]) ->
[Table];
group_by_queue_and_reason(Tables) ->
{_, Grouped} =
lists:foldl(
fun ({table, Info}, {SeenKeys, Acc}) ->
Q = x_death_event_key(Info, <<"queue">>),
R = x_death_event_key(Info, <<"reason">>),
Matcher = queue_and_reason_matcher(Q, R),
{Matches, _} = lists:partition(Matcher, Tables),
{Augmented, N} = case Matches of
[X] -> {X, 1};
[X|_] = Xs -> {X, length(Xs)}
end,
Key = {Q, R},
Acc1 = maybe_append_to_event_group(
ensure_xdeath_event_count(Augmented, N),
Key, SeenKeys, Acc),
{sets:add_element(Key, SeenKeys), Acc1}
end, {sets:new([{version, 2}]), []}, Tables),
Grouped.
update_x_death_header(Info, Headers) ->
X = x_death_event_key(Info, <<"exchange">>),
Q = x_death_event_key(Info, <<"queue">>),
R = x_death_event_key(Info, <<"reason">>),
case rabbit_basic:header(<<"x-death">>, Headers) of
undefined ->
%% First x-death event gets its own top-level headers.
%% See rabbitmq/rabbitmq-server#1332.
Headers2 = rabbit_misc:set_table_value(Headers, <<"x-first-death-reason">>,
longstr, R),
Headers3 = rabbit_misc:set_table_value(Headers2, <<"x-first-death-queue">>,
longstr, Q),
Headers4 = rabbit_misc:set_table_value(Headers3, <<"x-first-death-exchange">>,
longstr, X),
rabbit_basic:prepend_table_header(
<<"x-death">>,
[{<<"count">>, long, 1} | Info], Headers4);
{<<"x-death">>, array, Tables} ->
%% group existing x-death headers in case we have some from
%% before rabbitmq-server#78
GroupedTables = group_by_queue_and_reason(Tables),
{Matches, Others} = lists:partition(
queue_and_reason_matcher(Q, R),
GroupedTables),
Info1 = case Matches of
[] ->
[{<<"count">>, long, 1} | Info];
[{table, M}] ->
increment_xdeath_event_count(M)
end,
rabbit_misc:set_table_value(
Headers, <<"x-death">>, array,
[{table, rabbit_misc:sort_field_table(Info1)} | Others]);
{<<"x-death">>, InvalidType, Header} ->
rabbit_log:warning("Message has invalid x-death header (type: ~tp)."
" Resetting header ~tp",
[InvalidType, Header]),
%% if x-death is something other than an array (list)
%% then we reset it: this happens when some clients consume
%% a message and re-publish is, converting header values
%% to strings, intentionally or not.
%% See rabbitmq/rabbitmq-server#767 for details.
rabbit_misc:set_table_value(
Headers, <<"x-death">>, array,
[{table, [{<<"count">>, long, 1} | Info]}])
end.
ensure_xdeath_event_count({table, Info}, InitialVal) when InitialVal >= 1 ->
{table, ensure_xdeath_event_count(Info, InitialVal)};
ensure_xdeath_event_count(Info, InitialVal) when InitialVal >= 1 ->
case x_death_event_key(Info, <<"count">>) of
undefined ->
[{<<"count">>, long, InitialVal} | Info];
_ ->
Info
end.
increment_xdeath_event_count(Info) ->
case x_death_event_key(Info, <<"count">>) of
undefined ->
[{<<"count">>, long, 1} | Info];
N ->
lists:keyreplace(
<<"count">>, 1, Info,
{<<"count">>, long, N + 1})
end.
queue_and_reason_matcher(Q, R) ->
F = fun(Info) ->
x_death_event_key(Info, <<"queue">>) =:= Q
andalso x_death_event_key(Info, <<"reason">>) =:= R
end,
fun({table, Info}) ->
F(Info);
(Info) when is_list(Info) ->
F(Info)
end.
per_msg_ttl_header(#'P_basic'{expiration = Expiration})
when is_binary(Expiration) ->
[{<<"original-expiration">>, longstr, Expiration}];
per_msg_ttl_header(_) ->
[].
detect_cycles(rejected, _Msg, Queues) ->
{Queues, []};
detect_cycles(_Reason, #basic_message{content = Content}, Queues) ->
#content{properties = #'P_basic'{headers = Headers}} =
rabbit_binary_parser:ensure_content_decoded(Content),
NoCycles = {Queues, []},
case Headers of
undefined ->
NoCycles;
_ ->
case rabbit_misc:table_lookup(Headers, <<"x-death">>) of
{array, Deaths} ->
{Cycling, NotCycling} =
lists:partition(fun (#resource{name = Queue}) ->
is_cycle(Queue, Deaths);
({#resource{name = Queue}, _RouteInfos}) ->
is_cycle(Queue, Deaths)
end, Queues),
OldQueues = [rabbit_misc:table_lookup(D, <<"queue">>) ||
{table, D} <- Deaths],
OldQueues1 = [QName || {longstr, QName} <- OldQueues],
Cycling1 = lists:map(fun(#resource{name = QName}) ->
[QName | OldQueues1];
({#resource{name = QName}, _RouteInfos}) ->
[QName | OldQueues1]
end, Cycling),
{NotCycling, Cycling1};
_ ->
NoCycles
end
end.
is_cycle(Queue, Deaths) ->
{Cycle, Rest} =
lists:splitwith(
fun ({table, D}) ->
{longstr, Queue} =/= rabbit_misc:table_lookup(D, <<"queue">>);
(_) ->
true
end, Deaths),
%% Is there a cycle, and if so, is it "fully automatic", i.e. with
%% no reject in it?
case Rest of
[] -> false;
[H|_] -> lists:all(
fun ({table, D}) ->
{longstr, <<"rejected">>} =/=
rabbit_misc:table_lookup(D, <<"reason">>);
(_) ->
%% There was something we didn't expect, therefore
%% a client must have put it there, therefore the
%% cycle was not "fully automatic".
false
end, Cycle ++ [H])
end.
detect_cycles(_Reason, Msg, Queues) ->
{Cycling, NotCycling} =
lists:partition(fun (#resource{name = Queue}) ->
mc:is_death_cycle(Queue, Msg);
({#resource{name = Queue}, _RouteInfos}) ->
mc:is_death_cycle(Queue, Msg)
end, Queues),
DeathQueues = mc:death_queue_names(Msg),
CycleKeys = lists:foldl(fun(#resource{name = Q}, Acc) ->
[Q | Acc];
({#resource{name = Q}, _RouteInfos}, Acc) ->
[Q | Acc]
end, DeathQueues, Cycling),
{NotCycling, CycleKeys}.
log_cycle_once(Queues) ->
Key = {queue_cycle, Queues},
%% using a hash won't eliminate this as a potential memory leak but it will
%% reduce the potential amount of memory used whilst probably being
%% "good enough"
Key = {queue_cycle, erlang:phash2(Queues)},
case get(Key) of
true -> ok;
undefined -> rabbit_log:warning(
"Message dropped. Dead-letter queues cycle detected" ++
": ~tp~nThis cycle will NOT be reported again.",
[Queues]),
put(Key, true)
true -> ok;
undefined ->
rabbit_log:warning("Message dropped. Dead-letter queues cycle detected"
": ~tp~nThis cycle will NOT be reported again.",
[Queues]),
put(Key, true)
end.

80
deps/rabbit/src/rabbit_exchange.erl vendored
View File

@ -20,17 +20,22 @@
%%----------------------------------------------------------------------------
-deprecated([{route, 2, "Use route/3 instead"}]).
-export_type([name/0, type/0, route_opts/0, route_infos/0, route_return/0]).
-type name() :: rabbit_types:exchange_name().
-type type() :: rabbit_types:exchange_type().
-type route_opts() :: #{return_binding_keys => boolean()}.
-type route_infos() :: #{binding_keys => #{rabbit_types:binding_key() => true}}.
-type route_return() :: [rabbit_amqqueue:name() | {rabbit_amqqueue:name(), route_infos()}].
-type route_return() :: list(rabbit_amqqueue:name() |
{rabbit_amqqueue:name(), route_infos()} |
{virtual_reply_queue, binary()}).
%%----------------------------------------------------------------------------
-define(INFO_KEYS, [name, type, durable, auto_delete, internal, arguments,
policy, user_who_performed_action]).
-define(DEFAULT_EXCHANGE_NAME, <<>>).
-spec recover(rabbit_types:vhost()) -> [name()].
@ -339,38 +344,39 @@ info_all(VHostPath, Items, Ref, AggregatorPid) ->
rabbit_control_misc:emitting_map(
AggregatorPid, Ref, fun(X) -> info(X, Items) end, list(VHostPath)).
%% rabbit_types:delivery() is more strict than #delivery{}, some
%% fields can't be undefined. But there are places where
%% rabbit_exchange:route/2 is called with the absolutely bare delivery
%% like #delivery{message = #basic_message{routing_keys = [...]}}
-spec route(rabbit_types:exchange(), #delivery{}) -> [rabbit_amqqueue:name()].
route(Exchange, Delivery) ->
route(Exchange, Delivery, #{}).
-spec route(rabbit_types:exchange(), mc:state()) ->
[rabbit_amqqueue:name() | {virtual_reply_queue, binary()}].
route(Exchange, Message) ->
route(Exchange, Message, #{}).
-spec route(rabbit_types:exchange(), #delivery{}, route_opts()) ->
-spec route(rabbit_types:exchange(), mc:state(), route_opts()) ->
route_return().
route(#exchange{name = #resource{virtual_host = VHost, name = RName} = XName,
decorators = Decorators} = X,
#delivery{message = #basic_message{routing_keys = RKs}} = Delivery,
Opts) ->
case RName of
<<>> ->
RKsSorted = lists:usort(RKs),
[rabbit_channel:deliver_reply(RK, Delivery) ||
RK <- RKsSorted, virtual_reply_queue(RK)],
[rabbit_misc:r(VHost, queue, RK) || RK <- RKsSorted,
not virtual_reply_queue(RK)];
route(#exchange{name = #resource{name = ?DEFAULT_EXCHANGE_NAME,
virtual_host = VHost}},
Message, _Opts) ->
RKs0 = mc:get_annotation(routing_keys, Message),
RKs = lists:usort(RKs0),
[begin
case virtual_reply_queue(RK) of
false ->
rabbit_misc:r(VHost, queue, RK);
true ->
{virtual_reply_queue, RK}
end
end
|| RK <- RKs];
route(X = #exchange{name = XName,
decorators = Decorators},
Message, Opts) ->
Decs = rabbit_exchange_decorator:select(route, Decorators),
QNamesToBKeys = route1(Message, Decs, Opts, {[X], XName, #{}}),
case Opts of
#{return_binding_keys := true} ->
maps:fold(fun(QName, BindingKeys, L) ->
[{QName, #{binding_keys => BindingKeys}} | L]
end, [], QNamesToBKeys);
_ ->
Decs = rabbit_exchange_decorator:select(route, Decorators),
QNamesToBKeys = route1(Delivery, Decs, Opts, {[X], XName, #{}}),
case Opts of
#{return_binding_keys := true} ->
maps:fold(fun(QName, BindingKeys, L) ->
[{QName, #{binding_keys => BindingKeys}} | L]
end, [], QNamesToBKeys);
_ ->
maps:keys(QNamesToBKeys)
end
maps:keys(QNamesToBKeys)
end.
virtual_reply_queue(<<"amq.rabbitmq.reply-to.", _/binary>>) -> true;
@ -378,16 +384,16 @@ virtual_reply_queue(_) -> false.
route1(_, _, _, {[], _, QNames}) ->
QNames;
route1(Delivery, Decorators, Opts,
route1(Message, Decorators, Opts,
{[X = #exchange{type = Type} | WorkList], SeenXs, QNames}) ->
{Route, Arity} = type_to_route_fun(Type),
ExchangeDests = case Arity of
2 -> Route(X, Delivery);
3 -> Route(X, Delivery, Opts)
2 -> Route(X, Message);
3 -> Route(X, Message, Opts)
end,
DecorateDests = process_decorators(X, Decorators, Delivery),
DecorateDests = process_decorators(X, Decorators, Message),
AlternateDests = process_alternate(X, ExchangeDests),
route1(Delivery, Decorators, Opts,
route1(Message, Decorators, Opts,
lists:foldl(fun process_route/2, {WorkList, SeenXs, QNames},
AlternateDests ++ DecorateDests ++ ExchangeDests)).
@ -402,8 +408,8 @@ process_alternate(_X, _Results) ->
process_decorators(_, [], _) -> %% optimisation
[];
process_decorators(X, Decorators, Delivery) ->
lists:append([Decorator:route(X, Delivery) || Decorator <- Decorators]).
process_decorators(X, Decorators, Message) ->
lists:append([Decorator:route(X, Message) || Decorator <- Decorators]).
process_route(#resource{kind = exchange} = XName,
{_WorkList, XName, _QNames} = Acc) ->

2
deps/rabbit/src/rabbit_exchange_decorator.erl vendored
View File

@ -54,7 +54,7 @@
[rabbit_types:binding()]) -> 'ok'.
%% Allows additional destinations to be added to the routing decision.
-callback route(rabbit_types:exchange(), rabbit_types:delivery()) ->
-callback route(rabbit_types:exchange(), rabbit_types:message()) ->
[rabbit_amqqueue:name() | rabbit_exchange:name()].
%% Whether the decorator wishes to receive callbacks for the exchange

10
deps/rabbit/src/rabbit_exchange_type.erl vendored
View File

@ -23,10 +23,12 @@
%% The no_return is there so that we can have an "invalid" exchange
%% type (see rabbit_exchange_type_invalid).
-callback route(rabbit_types:exchange(), rabbit_types:delivery()) ->
rabbit_router:match_result().
%% NB: This callback is deprecated in favour of route/3
%% and will be removed in the future
% -callback route(rabbit_types:exchange(), mc:state()) ->
% rabbit_router:match_result().
-callback route(rabbit_types:exchange(), rabbit_types:delivery(), rabbit_exchange:route_opts()) ->
-callback route(rabbit_types:exchange(), mc:state(), rabbit_exchange:route_opts()) ->
[rabbit_types:binding_destination() |
{rabbit_amqqueue:name(), rabbit_types:binding_key()}].
@ -67,8 +69,6 @@
-callback info(rabbit_types:exchange(), [atom()]) -> [{atom(), term()}].
-optional_callbacks([route/3]).
added_to_rabbit_registry(Type, _ModuleName) ->
persistent_term:erase(Type),
ok.

15
deps/rabbit/src/rabbit_exchange_type_direct.erl vendored
View File

@ -10,7 +10,7 @@
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/2, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2, add_binding/3,
remove_bindings/3, assert_args_equivalence/2]).
@ -31,11 +31,16 @@ description() ->
serialise_events() -> false.
route(#exchange{name = Name, type = Type},
#delivery{message = #basic_message{routing_keys = Routes}}) ->
route(#exchange{name = Name, type = Type}, Msg) ->
route(#exchange{name = Name, type = Type}, Msg, #{}).
route(#exchange{name = Name, type = Type}, Msg, _Opts) ->
Routes = mc:get_annotation(routing_keys, Msg),
case Type of
direct -> route_v2(Name, Routes);
_ -> rabbit_router:match_routing_key(Name, Routes)
direct ->
route_v2(Name, Routes);
_ ->
rabbit_router:match_routing_key(Name, Routes)
end.
validate(_X) -> ok.

7
deps/rabbit/src/rabbit_exchange_type_fanout.erl vendored
View File

@ -10,7 +10,7 @@
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/2, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2, add_binding/3,
remove_bindings/3, assert_args_equivalence/2]).
@ -31,7 +31,10 @@ description() ->
serialise_events() -> false.
route(#exchange{name = Name}, _Delivery) ->
route(#exchange{name = Name}, _Message) ->
route(#exchange{name = Name}, _Message, #{}).
route(#exchange{name = Name}, _Message, _Opts) ->
rabbit_router:match_routing_key(Name, ['_']).
validate(_X) -> ok.

165
deps/rabbit/src/rabbit_exchange_type_headers.erl vendored
View File

@ -3,15 +3,13 @@
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%
%% Copyright (c) 2007-2023 VMware, Inc. or its affiliates. All rights reserved.
%%
-module(rabbit_exchange_type_headers).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/2, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2, add_binding/3,
remove_bindings/3, assert_args_equivalence/2]).
@ -32,14 +30,61 @@ description() ->
serialise_events() -> false.
route(#exchange{name = Name},
#delivery{message = #basic_message{content = Content}}) ->
Headers = case (Content#content.properties)#'P_basic'.headers of
undefined -> [];
H -> rabbit_misc:sort_field_table(H)
end,
route(#exchange{name = Name}, Msg) ->
route(#exchange{name = Name}, Msg, #{}).
route(#exchange{name = Name}, Msg, _Opts) ->
%% TODO: find a way not to extract x-headers unless necessary
Headers = mc:routing_headers(Msg, [x_headers]),
rabbit_router:match_bindings(
Name, fun (#binding{args = Spec}) -> headers_match(Spec, Headers) end).
Name, fun(#binding{args = Args}) ->
case rabbit_misc:table_lookup(Args, <<"x-match">>) of
{longstr, <<"any">>} ->
match_any(Args, Headers, fun match/2);
{longstr, <<"any-with-x">>} ->
match_any(Args, Headers, fun match_x/2);
{longstr, <<"all-with-x">>} ->
match_all(Args, Headers, fun match_x/2);
_ ->
match_all(Args, Headers, fun match/2)
end
end).
match_x({<<"x-match">>, _, _}, _M) ->
skip;
match_x({K, void, _}, M) ->
maps:is_key(K, M);
match_x({K, _, V}, M) ->
maps:get(K, M, undefined) =:= V.
match({<<"x-", _/binary>>, _, _}, _M) ->
skip;
match({K, void, _}, M) ->
maps:is_key(K, M);
match({K, _, V}, M) ->
maps:get(K, M, undefined) =:= V.
match_all([], _, _MatchFun) ->
true;
match_all([Arg | Rem], M, Fun) ->
case Fun(Arg, M) of
false ->
false;
_ ->
match_all(Rem, M, Fun)
end.
match_any([], _, _Fun) ->
false;
match_any([Arg | Rem], M, Fun) ->
case Fun(Arg, M) of
true ->
true;
_ ->
match_any(Rem, M, Fun)
end.
validate_binding(_X, #binding{args = Args}) ->
case rabbit_misc:table_lookup(Args, <<"x-match">>) of
@ -47,98 +92,16 @@ validate_binding(_X, #binding{args = Args}) ->
{longstr, <<"any">>} -> ok;
{longstr, <<"all-with-x">>} -> ok;
{longstr, <<"any-with-x">>} -> ok;
{longstr, Other} -> {error,
{binding_invalid,
"Invalid x-match field value ~tp; "
"expected all, any, all-with-x, or any-with-x", [Other]}};
{Type, Other} -> {error,
{binding_invalid,
"Invalid x-match field type ~tp (value ~tp); "
"expected longstr", [Type, Other]}};
undefined -> ok %% [0]
{longstr, Other} ->
{error, {binding_invalid,
"Invalid x-match field value ~tp; "
"expected all, any, all-with-x, or any-with-x", [Other]}};
{Type, Other} ->
{error, {binding_invalid,
"Invalid x-match field type ~tp (value ~tp); "
"expected longstr", [Type, Other]}};
undefined -> ok %% [0]
end.
%% [0] spec is vague on whether it can be omitted but in practice it's
%% useful to allow people to do this
parse_x_match({longstr, <<"all">>}) -> all;
parse_x_match({longstr, <<"any">>}) -> any;
parse_x_match({longstr, <<"all-with-x">>}) -> all_with_x;
parse_x_match({longstr, <<"any-with-x">>}) -> any_with_x;
parse_x_match(_) -> all. %% legacy; we didn't validate
%% Horrendous matching algorithm. Depends for its merge-like
%% (linear-time) behaviour on the lists:keysort
%% (rabbit_misc:sort_field_table) that route/1 and
%% rabbit_binding:{add,remove}/2 do.
%%
%% !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
%% In other words: REQUIRES BOTH PATTERN AND DATA TO BE SORTED ASCENDING BY KEY.
%% !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
%%
-spec headers_match
(rabbit_framing:amqp_table(), rabbit_framing:amqp_table()) ->
boolean().
headers_match(Args, Data) ->
MK = parse_x_match(rabbit_misc:table_lookup(Args, <<"x-match">>)),
headers_match(Args, Data, true, false, MK).
% A bit less horrendous algorithm :)
headers_match(_, _, false, _, all) -> false;
headers_match(_, _, false, _, all_with_x) -> false;
headers_match(_, _, _, true, any) -> true;
headers_match(_, _, _, true, any_with_x) -> true;
% No more bindings, return current state
headers_match([], _Data, AllMatch, _AnyMatch, all) -> AllMatch;
headers_match([], _Data, AllMatch, _AnyMatch, all_with_x) -> AllMatch;
headers_match([], _Data, _AllMatch, AnyMatch, any) -> AnyMatch;
headers_match([], _Data, _AllMatch, AnyMatch, any_with_x) -> AnyMatch;
%% Always delete binding x-match
headers_match([{<<"x-match">>, _PT, _PV} | PRest], Data,
AllMatch, AnyMatch, MatchKind) ->
headers_match(PRest, Data, AllMatch, AnyMatch, MatchKind);
% Delete all other bindings starting with x-
% unless x-match is set to all-with-x or any-with-x
headers_match([{<<"x-", _/binary>>, _PT, _PV} | PRest], Data,
AllMatch, AnyMatch, MatchKind)
when MatchKind =/= all_with_x, MatchKind =/= any_with_x ->
headers_match(PRest, Data, AllMatch, AnyMatch, MatchKind);
% No more data, but still bindings, false with all
headers_match(_Pattern, [], _AllMatch, AnyMatch, MatchKind) ->
headers_match([], [], false, AnyMatch, MatchKind);
% Data key header not in binding, go next data
headers_match(Pattern = [{PK, _PT, _PV} | _], [{DK, _DT, _DV} | DRest],
AllMatch, AnyMatch, MatchKind) when PK > DK ->
headers_match(Pattern, DRest, AllMatch, AnyMatch, MatchKind);
% Binding key header not in data, false with all, go next binding
headers_match([{PK, _PT, _PV} | PRest], Data = [{DK, _DT, _DV} | _],
_AllMatch, AnyMatch, MatchKind) when PK < DK ->
headers_match(PRest, Data, false, AnyMatch, MatchKind);
%% It's not properly specified, but a "no value" in a
%% pattern field is supposed to mean simple presence of
%% the corresponding data field. I've interpreted that to
%% mean a type of "void" for the pattern field.
headers_match([{PK, void, _PV} | PRest], [{DK, _DT, _DV} | DRest],
AllMatch, _AnyMatch, MatchKind) when PK == DK ->
headers_match(PRest, DRest, AllMatch, true, MatchKind);
% Complete match, true with any, go next
headers_match([{PK, _PT, PV} | PRest], [{DK, _DT, DV} | DRest],
AllMatch, _AnyMatch, MatchKind) when PK == DK andalso PV == DV ->
headers_match(PRest, DRest, AllMatch, true, MatchKind);
% Value does not match, false with all, go next
headers_match([{PK, _PT, _PV} | PRest], [{DK, _DT, _DV} | DRest],
_AllMatch, AnyMatch, MatchKind) when PK == DK ->
headers_match(PRest, DRest, false, AnyMatch, MatchKind).
validate(_X) -> ok.
create(_Tx, _X) -> ok.

9
deps/rabbit/src/rabbit_exchange_type_invalid.erl vendored
View File

@ -10,7 +10,7 @@
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/2, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2, add_binding/3,
remove_bindings/3, assert_args_equivalence/2]).
@ -26,9 +26,12 @@ description() ->
serialise_events() -> false.
-spec route(rabbit_types:exchange(), rabbit_types:delivery()) -> no_return().
-spec route(rabbit_types:exchange(), mc:state()) -> no_return().
route(Exchange, Msg) ->
route(Exchange, Msg, #{}).
route(#exchange{name = Name, type = Type}, _) ->
-spec route(rabbit_types:exchange(), mc:state(), map()) -> no_return().
route(#exchange{name = Name, type = Type}, _, _Opts) ->
rabbit_misc:protocol_error(
precondition_failed,
"Cannot route message through ~ts: exchange type ~ts not found",

11
deps/rabbit/src/rabbit_exchange_type_topic.erl vendored
View File

@ -36,13 +36,12 @@ serialise_events() -> false.
%% route/2 and route/3 can return duplicate destinations (and duplicate binding keys).
%% The caller of these functions is responsible for deduplication.
route(Exchange, Delivery) ->
route(Exchange, Delivery, #{}).
route(Exchange, Msg) ->
route(Exchange, Msg, #{}).
route(#exchange{name = XName},
#delivery{message = #basic_message{routing_keys = Routes}},
Opts) ->
lists:append([rabbit_db_topic_exchange:match(XName, RKey, Opts) || RKey <- Routes]).
route(#exchange{name = XName}, Msg, Opts) ->
RKeys = mc:get_annotation(routing_keys, Msg),
lists:append([rabbit_db_topic_exchange:match(XName, RKey, Opts) || RKey <- RKeys]).
validate(_X) -> ok.
validate_binding(_X, _B) -> ok.

26
deps/rabbit/src/rabbit_fifo.erl vendored
View File

@ -1531,10 +1531,6 @@ drop_head(#?MODULE{ra_indexes = Indexes0} = State0, Effects) ->
{State0, Effects}
end.
maybe_set_msg_ttl(#basic_message{content = #content{properties = none}},
_, Header,
#?MODULE{cfg = #cfg{msg_ttl = undefined}}) ->
Header;
maybe_set_msg_ttl(#basic_message{content = #content{properties = none}},
RaCmdTs, Header,
#?MODULE{cfg = #cfg{msg_ttl = PerQueueMsgTTL}}) ->
@ -1548,9 +1544,15 @@ maybe_set_msg_ttl(#basic_message{content = #content{properties = Props}},
{ok, PerMsgMsgTTL} = rabbit_basic:parse_expiration(Props),
TTL = min(PerMsgMsgTTL, PerQueueMsgTTL),
update_expiry_header(RaCmdTs, TTL, Header);
maybe_set_msg_ttl(_, _, Header,
#?MODULE{cfg = #cfg{}}) ->
Header.
maybe_set_msg_ttl(Msg, RaCmdTs, Header,
#?MODULE{cfg = #cfg{msg_ttl = MsgTTL}}) ->
case mc:is(Msg) of
true ->
TTL = min(MsgTTL, mc:ttl(Msg)),
update_expiry_header(RaCmdTs, TTL, Header);
false ->
Header
end.
update_expiry_header(_, undefined, Header) ->
Header;
@ -2393,8 +2395,14 @@ message_size(#basic_message{content = Content}) ->
message_size(B) when is_binary(B) ->
byte_size(B);
message_size(Msg) ->
%% probably only hit this for testing so ok to use erts_debug
erts_debug:size(Msg).
case mc:is(Msg) of
true ->
{_, PayloadSize} = mc:size(Msg),
PayloadSize;
false ->
%% probably only hit this for testing so ok to use erts_debug
erts_debug:size(Msg)
end.
all_nodes(#?MODULE{consumers = Cons0,

20
deps/rabbit/src/rabbit_fifo_client.erl vendored
View File

@ -30,11 +30,10 @@
pending_size/1,
stat/1,
stat/2,
query_single_active_consumer/1
query_single_active_consumer/1,
cluster_name/1
]).
-include_lib("rabbit_common/include/rabbit.hrl").
-define(SOFT_LIMIT, 32).
-define(TIMER_TIME, 10000).
-define(COMMAND_TIMEOUT, 30000).
@ -225,12 +224,14 @@ dequeue(QueueName, ConsumerTag, Settlement,
Err
end.
add_delivery_count_header(#basic_message{} = Msg0, Count)
when is_integer(Count) andalso
Count > 0 ->
rabbit_basic:add_header(<<"x-delivery-count">>, long, Count, Msg0);
add_delivery_count_header(Msg, _Count) ->
Msg.
add_delivery_count_header(Msg, Count) ->
case mc:is(Msg) of
true when is_integer(Count) andalso
Count > 0 ->
mc:set_annotation(<<"x-delivery-count">>, Count, Msg);
_ ->
Msg
end.
%% @doc Settle a message. Permanently removes message from the queue.
@ -780,6 +781,7 @@ transform_msgs(QName, QRef, Msgs) ->
_ ->
{Msg0, false}
end,
{QName, QRef, MsgId, Redelivered, Msg}
end, Msgs).

133
deps/rabbit/src/rabbit_fifo_dlx_worker.erl vendored
View File

@ -23,8 +23,9 @@
-module(rabbit_fifo_dlx_worker).
-include("mc.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
% -include_lib("rabbit_common/include/rabbit_framing.hrl").
-behaviour(gen_server).
@ -43,7 +44,7 @@
%% This rabbit_fifo_dlx_worker does not have the concept of delivery tags because it settles (acks)
%% message IDs directly back to the queue (and there is no AMQP consumer).
consumed_msg_id :: non_neg_integer(),
delivery :: rabbit_types:delivery(),
delivery :: mc:state(),
reason :: rabbit_dead_letter:reason(),
%% target queues for which publisher confirm has not been received yet
unsettled = [] :: [rabbit_amqqueue:name()],
@ -303,7 +304,7 @@ lookup_dlx(#state{exchange_ref = DLXRef} = State0) ->
{X, State0}
end.
-spec forward(rabbit_types:message(), non_neg_integer(), rabbit_amqqueue:name(),
-spec forward(mc:state(), non_neg_integer(), rabbit_amqqueue:name(),
rabbit_types:exchange() | not_found, rabbit_dead_letter:reason(), state()) ->
state().
forward(ConsumedMsg, ConsumedMsgId, ConsumedQRef, DLX, Reason,
@ -311,34 +312,42 @@ forward(ConsumedMsg, ConsumedMsgId, ConsumedQRef, DLX, Reason,
pendings = Pendings,
exchange_ref = DLXRef,
routing_key = RKey} = State0) ->
#basic_message{routing_keys = RKeys} = Msg = rabbit_dead_letter:make_msg(ConsumedMsg, Reason,
DLXRef, RKey, ConsumedQRef),
%% Field 'mandatory' is set to false because we check ourselves whether the message is routable.
Delivery = rabbit_basic:delivery(_Mandatory = false, _Confirm = true, Msg, OutSeq),
{TargetQs, State3} = case DLX of
not_found ->
{[], State0};
_ ->
RouteToQs0 = rabbit_exchange:route(DLX, Delivery),
{RouteToQs1, Cycles} = rabbit_dead_letter:detect_cycles(Reason, Msg, RouteToQs0),
State1 = log_cycles(Cycles, RKeys, State0),
RouteToQs2 = rabbit_amqqueue:lookup_many(RouteToQs1),
RouteToQs = rabbit_amqqueue:prepend_extra_bcc(RouteToQs2),
State2 = case RouteToQs of
[] ->
log_no_route_once(State1);
_ ->
State1
end,
{RouteToQs, State2}
end,
Now = os:system_time(millisecond),
Pend0 = #pending{
consumed_msg_id = ConsumedMsgId,
consumed_at = Now,
delivery = Delivery,
reason = Reason
},
#resource{name = SourceQName} = ConsumedQRef,
#resource{name = DLXName} = DLXRef,
DLRKeys = case RKey of
undefined ->
mc:get_annotation(routing_keys, ConsumedMsg);
_ ->
[RKey]
end,
Msg0 = mc:record_death(Reason, SourceQName, ConsumedMsg),
Msg1 = mc:set_ttl(undefined, Msg0),
Msg2 = mc:set_annotation(routing_keys, DLRKeys, Msg1),
Msg = mc:set_annotation(exchange, DLXName, Msg2),
{TargetQs, State3} =
case DLX of
not_found ->
{[], State0};
_ ->
RouteToQs0 = rabbit_exchange:route(DLX, Msg),
{RouteToQs1, Cycles} = rabbit_dead_letter:detect_cycles(
Reason, Msg, RouteToQs0),
State1 = log_cycles(Cycles, [RKey], State0),
RouteToQs2 = rabbit_amqqueue:lookup_many(RouteToQs1),
RouteToQs = rabbit_amqqueue:prepend_extra_bcc(RouteToQs2),
State2 = case RouteToQs of
[] ->
log_no_route_once(State1);
_ ->
State1
end,
{RouteToQs, State2}
end,
Pend0 = #pending{consumed_msg_id = ConsumedMsgId,
consumed_at = Now,
delivery = Msg,
reason = Reason},
case TargetQs of
[] ->
%% We can't deliver this message since there is no target queue we can route to.
@ -351,26 +360,29 @@ forward(ConsumedMsg, ConsumedMsgId, ConsumedQRef, DLX, Reason,
unsettled = queue_names(TargetQs)},
State = State3#state{next_out_seq = OutSeq + 1,
pendings = maps:put(OutSeq, Pend, Pendings)},
deliver_to_queues(Delivery, TargetQs, State)
Options = #{correlation => OutSeq},
deliver_to_queues(Msg, Options, TargetQs, State)
end.
-spec deliver_to_queues(rabbit_types:delivery(), [amqqueue:amqqueue()], state()) ->
state().
deliver_to_queues(#delivery{msg_seq_no = SeqNo} = Delivery, Qs, #state{queue_type_state = QTypeState0,
pendings = Pendings} = State0) ->
{State, Actions} = case rabbit_queue_type:deliver(Qs, Delivery, QTypeState0) of
{ok, QTypeState, Actions0} ->
{State0#state{queue_type_state = QTypeState}, Actions0};
{error, Reason} ->
%% rabbit_queue_type:deliver/3 does not tell us which target queue failed.
%% Therefore, reject all target queues. We need to reject them such that
%% we won't rely on rabbit_fifo_client to re-deliver on behalf of us
%% (and therefore preventing messages to get stuck in our 'unsettled' state).
QNames = queue_names(Qs),
rabbit_log:debug("Failed to deliver message with seq_no ~b to queues ~tp: ~tp",
[SeqNo, QNames, Reason]),
{State0#state{pendings = rejected(SeqNo, QNames, Pendings)}, []}
end,
deliver_to_queues(Msg, Options, Qs, #state{queue_type_state = QTypeState0,
pendings = Pendings} = State0) ->
SeqNo = maps:get(correlation, Options),
{State, Actions} =
case rabbit_queue_type:deliver(Qs, Msg, Options, QTypeState0) of
{ok, QTypeState, Actions0} ->
{State0#state{queue_type_state = QTypeState}, Actions0};
{error, Reason} ->
%% rabbit_queue_type:deliver/3 does not tell us which target queue failed.
%% Therefore, reject all target queues. We need to reject them such that
%% we won't rely on rabbit_fifo_client to re-deliver on behalf of us
%% (and therefore preventing messages to get stuck in our 'unsettled' state).
QNames = queue_names(Qs),
rabbit_log:debug("Failed to deliver message with seq_no ~b to "
"queues ~tp: ~tp",
[SeqNo, QNames, Reason]),
{State0#state{pendings = rejected(SeqNo, QNames, Pendings)}, []}
end,
handle_queue_actions(Actions, State).
handle_settled(QRef, MsgSeqs, State) ->
@ -441,22 +453,18 @@ redeliver_messages(#state{pendings = Pendings,
end, State, Pendings)
end.
redeliver(#pending{delivery = #delivery{message = #basic_message{content = Content}}} = Pend,
redeliver(#pending{delivery = Msg} = Pend,
DLX, OutSeq, #state{routing_key = undefined} = State) ->
%% No dead-letter-routing-key defined for source quorum queue.
%% Therefore use all of messages's original routing keys (which can include CC and BCC recipients).
%% This complies with the behaviour of the rabbit_dead_letter module.
%% We stored these original routing keys in the 1st (i.e. most recent) x-death entry.
#content{properties = #'P_basic'{headers = Headers}} =
rabbit_binary_parser:ensure_content_decoded(Content),
{array, [{table, MostRecentDeath}|_]} = rabbit_misc:table_lookup(Headers, <<"x-death">>),
{<<"routing-keys">>, array, Routes0} = lists:keyfind(<<"routing-keys">>, 1, MostRecentDeath),
Routes = [Route || {longstr, Route} <- Routes0],
{_, #death{routing_keys = Routes}} = mc:last_death(Msg),
redeliver0(Pend, DLX, Routes, OutSeq, State);
redeliver(Pend, DLX, OutSeq, #state{routing_key = DLRKey} = State) ->
redeliver0(Pend, DLX, [DLRKey], OutSeq, State).
redeliver0(#pending{delivery = #delivery{message = BasicMsg} = Delivery0,
redeliver0(#pending{delivery = Msg0,
unsettled = Unsettled0,
settled = Settled,
publish_count = PublishCount,
@ -468,14 +476,16 @@ redeliver0(#pending{delivery = #delivery{message = BasicMsg} = Delivery0,
exchange_ref = DLXRef,
queue_type_state = QTypeState} = State0)
when is_list(DLRKeys) ->
Delivery = Delivery0#delivery{message = BasicMsg#basic_message{exchange_name = DLXRef,
routing_keys = DLRKeys}},
#resource{name = DLXName} = DLXRef,
Msg1 = mc:set_ttl(undefined, Msg0),
Msg2 = mc:set_annotation(routing_keys, DLRKeys, Msg1),
Msg = mc:set_annotation(exchange, DLXName, Msg2),
%% Because of implicit default bindings rabbit_exchange:route/2 can route to target
%% queues that do not exist. Therefore, filter out non-existent target queues.
RouteToQs0 = queue_names(
rabbit_amqqueue:prepend_extra_bcc(
rabbit_amqqueue:lookup_many(
rabbit_exchange:route(DLX, Delivery)))),
rabbit_exchange:route(DLX, Msg)))),
case {RouteToQs0, Settled} of
{[], [_|_]} ->
%% Routes changed dynamically so that we don't await any publisher confirms anymore.
@ -491,7 +501,7 @@ redeliver0(#pending{delivery = #delivery{message = BasicMsg} = Delivery0,
%% Note that a quorum queue client does not redeliver on our behalf if it previously
%% rejected the message. This is why we always redeliver rejected messages here.
RouteToQs1 = Unsettled -- clients_redeliver(Unsettled0, QTypeState),
{RouteToQs, Cycles} = rabbit_dead_letter:detect_cycles(Reason, BasicMsg, RouteToQs1),
{RouteToQs, Cycles} = rabbit_dead_letter:detect_cycles(Reason, Msg, RouteToQs1),
State1 = log_cycles(Cycles, DLRKeys, State0),
case RouteToQs of
[] ->
@ -499,14 +509,15 @@ redeliver0(#pending{delivery = #delivery{message = BasicMsg} = Delivery0,
_ ->
Pend = Pend0#pending{publish_count = PublishCount + 1,
last_published_at = os:system_time(millisecond),
delivery = Delivery,
delivery = Msg,
%% Override 'unsettled' because topology could have changed.
unsettled = Unsettled,
%% Any target queue that rejected previously and still need
%% to be routed to is moved back to 'unsettled'.
rejected = []},
State = State0#state{pendings = maps:update(OutSeq, Pend, Pendings)},
deliver_to_queues(Delivery, rabbit_amqqueue:lookup_many(RouteToQs), State)
Options = #{correlation => OutSeq},
deliver_to_queues(Msg, Options, rabbit_amqqueue:lookup_many(RouteToQs), State)
end
end.

10
deps/rabbit/src/rabbit_logger_exchange_h.erl vendored
View File

@ -51,10 +51,14 @@ log(LogEvent, Config) ->
do_log(LogEvent, #{config := #{exchange := Exchange}} = Config) ->
RoutingKey = make_routing_key(LogEvent, Config),
AmqpMsg = log_event_to_amqp_msg(LogEvent, Config),
PBasic = log_event_to_amqp_msg(LogEvent, Config),
Body = try_format_body(LogEvent, Config),
case rabbit_basic:publish(Exchange, RoutingKey, AmqpMsg, Body) of
ok -> ok;
Content = rabbit_basic:build_content(PBasic, Body),
Anns = #{exchange => Exchange#resource.name,
routing_keys => [RoutingKey]},
Msg = mc:init(mc_amqpl, Content, Anns),
case rabbit_queue_type:publish_at_most_once(Exchange, Msg) of
ok -> ok;
{error, not_found} -> ok
end.

60
deps/rabbit/src/rabbit_message_interceptor.erl vendored
View File

@ -12,54 +12,38 @@
-export([intercept/1]).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-define(HEADER_TIMESTAMP, <<"timestamp_in_ms">>).
-define(HEADER_ROUTING_NODE, <<"x-routed-by">>).
-type content() :: rabbit_types:decoded_content().
-spec intercept(content()) -> content().
intercept(Content) ->
-spec intercept(mc:state()) -> mc:state().
intercept(Msg) ->
Interceptors = persistent_term:get({rabbit, incoming_message_interceptors}, []),
lists:foldl(fun(I, C) ->
intercept(C, I)
end, Content, Interceptors).
lists:foldl(fun({InterceptorName, Overwrite}, M) ->
intercept(M, InterceptorName, Overwrite)
end, Msg, Interceptors).
intercept(Content, {set_header_routing_node, Overwrite}) ->
intercept(Msg, set_header_routing_node, Overwrite) ->
Node = atom_to_binary(node()),
set_header(Content, {?HEADER_ROUTING_NODE, longstr, Node}, Overwrite);
intercept(Content0, {set_header_timestamp, Overwrite}) ->
NowMs = os:system_time(millisecond),
NowSecs = NowMs div 1_000,
Content = set_header(Content0, {?HEADER_TIMESTAMP, long, NowMs}, Overwrite),
set_property_timestamp(Content, NowSecs, Overwrite).
set_annotation(Msg, ?HEADER_ROUTING_NODE, Node, Overwrite);
intercept(Msg0, set_header_timestamp, Overwrite) ->
Millis = os:system_time(millisecond),
Msg = set_annotation(Msg0, ?HEADER_TIMESTAMP, Millis, Overwrite),
set_timestamp(Msg, Millis, Overwrite).
-spec set_header(content(),
{binary(), rabbit_framing:amqp_field_type(), rabbit_framing:amqp_value()},
boolean()) ->
content().
set_header(Content = #content{properties = Props = #'P_basic'{headers = Headers0}},
Header = {Key, Type, Value}, Overwrite) ->
case {rabbit_basic:header(Key, Headers0), Overwrite} of
-spec set_annotation(mc:state(), mc:ann_key(), mc:ann_value(), boolean()) -> mc:state().
set_annotation(Msg, Key, Value, Overwrite) ->
case {mc:x_header(Key, Msg), Overwrite} of
{Val, false} when Val =/= undefined ->
Content;
Msg;
_ ->
Headers = if Headers0 =:= undefined -> [Header];
true -> rabbit_misc:set_table_value(Headers0, Key, Type, Value)
end,
Content#content{properties = Props#'P_basic'{headers = Headers},
properties_bin = none}
mc:set_annotation(Key, Value, Msg)
end.
-spec set_property_timestamp(content(), pos_integer(), boolean()) -> content().
set_property_timestamp(Content = #content{properties = Props = #'P_basic'{timestamp = Ts}},
Timestamp, Overwrite) ->
case {Ts, Overwrite} of
{Secs, false} when is_integer(Secs) ->
Content;
-spec set_timestamp(mc:state(), pos_integer(), boolean()) -> mc:state().
set_timestamp(Msg, Timestamp, Overwrite) ->
case {mc:timestamp(Msg), Overwrite} of
{Ts, false} when is_integer(Ts) ->
Msg;
_ ->
Content#content{properties = Props#'P_basic'{timestamp = Timestamp},
properties_bin = none}
mc:set_annotation(timestamp, Timestamp, Msg)
end.

29
deps/rabbit/src/rabbit_mirror_queue_master.erl vendored
View File

@ -26,7 +26,6 @@
-behaviour(rabbit_backing_queue).
-include_lib("rabbit_common/include/rabbit.hrl").
-include("amqqueue.hrl").
-record(state, { name,
@ -232,14 +231,17 @@ purge(State = #state { gm = GM,
-spec purge_acks(_) -> no_return().
purge_acks(_State) -> exit({not_implemented, {?MODULE, purge_acks}}).
publish(Msg = #basic_message { id = MsgId }, MsgProps, IsDelivered, ChPid, Flow,
publish(Msg, MsgProps, IsDelivered, ChPid, Flow,
State = #state { gm = GM,
seen_status = SS,
backing_queue = BQ,
backing_queue_state = BQS }) ->
MsgId = mc:get_annotation(id, Msg),
{_, Size} = mc:size(Msg),
false = maps:is_key(MsgId, SS), %% ASSERTION
ok = gm:broadcast(GM, {publish, ChPid, Flow, MsgProps, Msg},
rabbit_basic:msg_size(Msg)),
Size),
BQS1 = BQ:publish(Msg, MsgProps, IsDelivered, ChPid, Flow, BQS),
ensure_monitoring(ChPid, State #state { backing_queue_state = BQS1 }).
@ -249,11 +251,13 @@ batch_publish(Publishes, ChPid, Flow,
backing_queue = BQ,
backing_queue_state = BQS }) ->
{Publishes1, false, MsgSizes} =
lists:foldl(fun ({Msg = #basic_message { id = MsgId },
lists:foldl(fun ({Msg,
MsgProps, _IsDelivered}, {Pubs, false, Sizes}) ->
MsgId = mc:get_annotation(id, Msg),
{_, Size} = mc:size(Msg),
{[{Msg, MsgProps, true} | Pubs], %% [0]
false = maps:is_key(MsgId, SS), %% ASSERTION
Sizes + rabbit_basic:msg_size(Msg)}
Sizes + Size}
end, {[], false, 0}, Publishes),
Publishes2 = lists:reverse(Publishes1),
ok = gm:broadcast(GM, {batch_publish, ChPid, Flow, Publishes2},
@ -264,14 +268,16 @@ batch_publish(Publishes, ChPid, Flow,
%% IsDelivered flag to true, so to avoid iterating over the messages
%% again at the mirror, we do it here.
publish_delivered(Msg = #basic_message { id = MsgId }, MsgProps,
publish_delivered(Msg, MsgProps,
ChPid, Flow, State = #state { gm = GM,
seen_status = SS,
backing_queue = BQ,
backing_queue_state = BQS }) ->
MsgId = mc:get_annotation(id, Msg),
{_, Size} = mc:size(Msg),
false = maps:is_key(MsgId, SS), %% ASSERTION
ok = gm:broadcast(GM, {publish_delivered, ChPid, Flow, MsgProps, Msg},
rabbit_basic:msg_size(Msg)),
Size),
{AckTag, BQS1} = BQ:publish_delivered(Msg, MsgProps, ChPid, Flow, BQS),
State1 = State #state { backing_queue_state = BQS1 },
{AckTag, ensure_monitoring(ChPid, State1)}.
@ -282,10 +288,12 @@ batch_publish_delivered(Publishes, ChPid, Flow,
backing_queue = BQ,
backing_queue_state = BQS }) ->
{false, MsgSizes} =
lists:foldl(fun ({Msg = #basic_message { id = MsgId }, _MsgProps},
lists:foldl(fun ({Msg, _MsgProps},
{false, Sizes}) ->
MsgId = mc:get_annotation(id, Msg),
{_, Size} = mc:size(Msg),
{false = maps:is_key(MsgId, SS), %% ASSERTION
Sizes + rabbit_basic:msg_size(Msg)}
Sizes + Size}
end, {false, 0}, Publishes),
ok = gm:broadcast(GM, {batch_publish_delivered, ChPid, Flow, Publishes},
MsgSizes),
@ -439,11 +447,12 @@ invoke(Mod, Fun, State = #state { backing_queue = BQ,
backing_queue_state = BQS }) ->
State #state { backing_queue_state = BQ:invoke(Mod, Fun, BQS) }.
is_duplicate(Message = #basic_message { id = MsgId },
is_duplicate(Message,
State = #state { seen_status = SS,
backing_queue = BQ,
backing_queue_state = BQS,
confirmed = Confirmed }) ->
MsgId = mc:get_annotation(id, Message),
%% Here, we need to deal with the possibility that we're about to
%% receive a message that we've already seen when we were a mirror
%% (we received it via gm). Thus if we do receive such message now

74
deps/rabbit/src/rabbit_mirror_queue_slave.erl vendored
View File

@ -563,21 +563,24 @@ send_mandatory(#delivery{mandatory = true,
send_or_record_confirm(_, #delivery{ confirm = false }, MS, _State) ->
MS;
send_or_record_confirm(published, #delivery { sender = ChPid,
send_or_record_confirm(Status, #delivery { sender = ChPid,
confirm = true,
msg_seq_no = MsgSeqNo,
message = #basic_message {
id = MsgId,
is_persistent = true } },
MS, #state{q = Q}) when ?amqqueue_is_durable(Q) ->
maps:put(MsgId, {published, ChPid, MsgSeqNo} , MS);
send_or_record_confirm(_Status, #delivery { sender = ChPid,
confirm = true,
msg_seq_no = MsgSeqNo },
MS, #state{q = Q} = _State) ->
ok = rabbit_classic_queue:confirm_to_sender(ChPid,
amqqueue:get_name(Q), [MsgSeqNo]),
MS.
message = Msg
},
MS, #state{q = Q}) ->
MsgId = mc:get_annotation(id, Msg),
IsPersistent = mc:is_persistent(Msg),
case IsPersistent of
true when ?amqqueue_is_durable(Q) andalso
Status == published ->
maps:put(MsgId, {published, ChPid, MsgSeqNo}, MS);
_ ->
ok = rabbit_classic_queue:confirm_to_sender(ChPid,
amqqueue:get_name(Q),
[MsgSeqNo]),
MS
end.
confirm_messages(MsgIds, State = #state{q = Q, msg_id_status = MS}) ->
QName = amqqueue:get_name(Q),
@ -847,9 +850,10 @@ maybe_forget_sender(ChPid, ChState, State = #state { sender_queues = SQ,
end.
maybe_enqueue_message(
Delivery = #delivery { message = #basic_message { id = MsgId },
Delivery = #delivery { message = Msg,
sender = ChPid },
State = #state { sender_queues = SQ, msg_id_status = MS }) ->
MsgId = mc:get_annotation(id, Msg),
send_mandatory(Delivery), %% must do this before confirms
State1 = ensure_monitoring(ChPid, State),
%% We will never see {published, ChPid, MsgSeqNo} here.
@ -897,25 +901,28 @@ publish_or_discard(Status, ChPid, MsgId,
{MQ, sets:add_element(MsgId, PendingCh),
maps:put(MsgId, Status, MS)};
{{value, Delivery = #delivery {
message = #basic_message { id = MsgId } }}, MQ2} ->
{MQ2, PendingCh,
%% We received the msg from the channel first. Thus
%% we need to deal with confirms here.
send_or_record_confirm(Status, Delivery, MS, State1)};
{{value, #delivery {}}, _MQ2} ->
%% The instruction was sent to us before we were
%% within the slave_pids within the #amqqueue{}
%% record. We'll never receive the message directly
%% from the channel. And the channel will not be
%% expecting any confirms from us.
{MQ, PendingCh, MS}
message = Msg }}, MQ2} ->
case mc:get_annotation(id, Msg) of
MsgId ->
{MQ2, PendingCh,
%% We received the msg from the channel first. Thus
%% we need to deal with confirms here.
send_or_record_confirm(Status, Delivery, MS, State1)};
_ ->
%% The instruction was sent to us before we were
%% within the slave_pids within the #amqqueue{}
%% record. We'll never receive the message directly
%% from the channel. And the channel will not be
%% expecting any confirms from us.
{MQ, PendingCh, MS}
end
end,
SQ1 = maps:put(ChPid, {MQ1, PendingCh1, ChState}, SQ),
State1 #state { sender_queues = SQ1, msg_id_status = MS1 }.
process_instruction({publish, ChPid, Flow, MsgProps,
Msg = #basic_message { id = MsgId }}, State) ->
process_instruction({publish, ChPid, Flow, MsgProps, Msg}, State) ->
MsgId = mc:get_annotation(id, Msg),
maybe_flow_ack(ChPid, Flow),
State1 = #state { backing_queue = BQ, backing_queue_state = BQS } =
publish_or_discard(published, ChPid, MsgId, State),
@ -924,14 +931,14 @@ process_instruction({publish, ChPid, Flow, MsgProps,
process_instruction({batch_publish, ChPid, Flow, Publishes}, State) ->
maybe_flow_ack(ChPid, Flow),
State1 = #state { backing_queue = BQ, backing_queue_state = BQS } =
lists:foldl(fun ({#basic_message { id = MsgId },
_MsgProps, _IsDelivered}, St) ->
lists:foldl(fun ({Msg, _MsgProps, _IsDelivered}, St) ->
MsgId = mc:get_annotation(id, Msg),
publish_or_discard(published, ChPid, MsgId, St)
end, State, Publishes),
BQS1 = BQ:batch_publish(Publishes, ChPid, Flow, BQS),
{ok, State1 #state { backing_queue_state = BQS1 }};
process_instruction({publish_delivered, ChPid, Flow, MsgProps,
Msg = #basic_message { id = MsgId }}, State) ->
process_instruction({publish_delivered, ChPid, Flow, MsgProps, Msg}, State) ->
MsgId = mc:get_annotation(id, Msg),
maybe_flow_ack(ChPid, Flow),
State1 = #state { backing_queue = BQ, backing_queue_state = BQS } =
publish_or_discard(published, ChPid, MsgId, State),
@ -943,8 +950,9 @@ process_instruction({batch_publish_delivered, ChPid, Flow, Publishes}, State) ->
maybe_flow_ack(ChPid, Flow),
{MsgIds,
State1 = #state { backing_queue = BQ, backing_queue_state = BQS }} =
lists:foldl(fun ({#basic_message { id = MsgId }, _MsgProps},
lists:foldl(fun ({Msg, _MsgProps},
{MsgIds, St}) ->
MsgId = mc:get_annotation(id, Msg),
{[MsgId | MsgIds],
publish_or_discard(published, ChPid, MsgId, St)}
end, {[], State}, Publishes),

3
deps/rabbit/src/rabbit_mirror_queue_sync.erl vendored
View File

@ -133,7 +133,8 @@ bq_fold(FoldFun, FoldAcc, Args, BQ, BQS) ->
append_to_acc(Msg, MsgProps, Unacked, {Batch, I, {_, _, 0}, {Curr, Len}, T}) ->
{[{Msg, MsgProps, Unacked} | Batch], I, {0, 0, 0}, {Curr + 1, Len}, T};
append_to_acc(Msg, MsgProps, Unacked, {Batch, I, {TotalBytes, LastCheck, SyncThroughput}, {Curr, Len}, T}) ->
{[{Msg, MsgProps, Unacked} | Batch], I, {TotalBytes + rabbit_basic:msg_size(Msg), LastCheck, SyncThroughput}, {Curr + 1, Len}, T}.
{_, MsgSize} = mc:size(Msg),
{[{Msg, MsgProps, Unacked} | Batch], I, {TotalBytes + MsgSize, LastCheck, SyncThroughput}, {Curr + 1, Len}, T}.
master_send_receive(SyncMsg, NewAcc, Syncer, Ref, Parent) ->
receive

8
deps/rabbit/src/rabbit_msg_store.erl vendored
View File

@ -553,9 +553,11 @@ read_many_file2(MsgIds0, CState = #client_msstate{ dir = Dir,
%% Before we continue the read_many calls we must remove the
%% MsgIds we have read from the list and add the messages to
%% the Acc.
{Acc, MsgIdsRead} = lists:foldl(fun(Msg = #basic_message{id = MsgIdRead}, {Acc1, MsgIdsAcc}) ->
{Acc1#{MsgIdRead => Msg}, [MsgIdRead|MsgIdsAcc]}
end, {Acc0, []}, Msgs),
{Acc, MsgIdsRead} = lists:foldl(
fun(Msg, {Acc1, MsgIdsAcc}) ->
MsgIdRead = mc:get_annotation(id, Msg),
{Acc1#{MsgIdRead => Msg}, [MsgIdRead|MsgIdsAcc]}
end, {Acc0, []}, Msgs),
MsgIds = MsgIds0 -- MsgIdsRead,
%% Unmark opened files and continue.
read_many_file3(MsgIds, CState#client_msstate{ reader = Reader }, Acc, File)

13
deps/rabbit/src/rabbit_priority_queue.erl vendored
View File

@ -8,7 +8,6 @@
-module(rabbit_priority_queue).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include("amqqueue.hrl").
-behaviour(rabbit_backing_queue).
@ -628,11 +627,8 @@ priority(Priority, MaxP) when is_integer(Priority), Priority =< MaxP ->
Priority;
priority(Priority, MaxP) when is_integer(Priority), Priority > MaxP ->
MaxP;
priority(#basic_message{content = Content}, MaxP) ->
priority(rabbit_binary_parser:ensure_content_decoded(Content), MaxP);
priority(#content{properties = Props}, MaxP) ->
#'P_basic'{priority = Priority0} = Props,
priority(Priority0, MaxP).
priority(Msg, MaxP) ->
priority(mc:priority(Msg), MaxP).
add_maybe_infinity(infinity, _) -> infinity;
add_maybe_infinity(_, infinity) -> infinity;
@ -689,6 +685,7 @@ find_head_message_timestamp(_, [], Timestamp) ->
zip_msgs_and_acks(Pubs, AckTags) ->
lists:zipwith(
fun ({#basic_message{ id = Id }, _Props}, AckTag) ->
{Id, AckTag}
fun ({Msg, _Props}, AckTag) ->
Id = mc:get_annotation(id, Msg),
{Id, AckTag}
end, Pubs, AckTags).

15
deps/rabbit/src/rabbit_queue_index.erl vendored
View File

@ -428,8 +428,9 @@ maybe_needs_confirming(MsgProps, MsgOrId,
State = #qistate{unconfirmed = UC,
unconfirmed_msg = UCM}) ->
MsgId = case MsgOrId of
#basic_message{id = Id} -> Id;
Id when is_binary(Id) -> Id
Id when is_binary(Id) -> Id;
Msg ->
mc:get_annotation(id, Msg)
end,
?MSG_ID_BYTES = size(MsgId),
case {MsgProps#message_properties.needs_confirming, MsgOrId} of
@ -874,7 +875,8 @@ create_pub_record_body(MsgOrId, #message_properties { expiry = Expiry,
case MsgOrId of
MsgId when is_binary(MsgId) ->
{<<MsgId/binary, ExpiryBin/binary, Size:?SIZE_BITS>>, <<>>};
#basic_message{id = MsgId} ->
Msg ->
MsgId = mc:get_annotation(id, Msg),
MsgBin = term_to_binary(MsgOrId),
{<<MsgId/binary, ExpiryBin/binary, Size:?SIZE_BITS>>, MsgBin}
end.
@ -894,8 +896,11 @@ parse_pub_record_body(<<MsgIdNum:?MSG_ID_BITS, Expiry:?EXPIRY_BITS,
size = Size},
case MsgBin of
<<>> -> {MsgId, Props};
_ -> Msg = #basic_message{id = MsgId} = binary_to_term(MsgBin),
{Msg, Props}
_ ->
Msg = binary_to_term(MsgBin),
%% assertion
MsgId = mc:get_annotation(id, Msg),
{Msg, Props}
end.
%%----------------------------------------------------------------------------

64
deps/rabbit/src/rabbit_queue_type.erl vendored
View File

@ -40,7 +40,7 @@
handle_down/4,
handle_event/3,
module/2,
deliver/3,
deliver/4,
settle/5,
credit/5,
dequeue/5,
@ -49,7 +49,8 @@
is_server_named_allowed/1,
arguments/1,
arguments/2,
notify_decorators/1
notify_decorators/1,
publish_at_most_once/2
]).
-export([
@ -119,12 +120,14 @@
ok_msg := term(),
acting_user := rabbit_types:username()}.
-type delivery_options() :: #{correlation => term(), %% sequence no typically
atom() => term()}.
-type settle_op() :: 'complete' | 'requeue' | 'discard'.
-export_type([state/0,
consume_spec/0,
delivery_options/0,
action/0,
actions/0,
settle_op/0]).
@ -194,7 +197,8 @@
{protocol_error, Type :: atom(), Reason :: string(), Args :: term()}.
-callback deliver([{amqqueue:amqqueue(), queue_state()}],
Delivery :: term()) ->
Message :: mc:state(),
Options :: delivery_options()) ->
{[{amqqueue:amqqueue(), queue_state()}], actions()}.
-callback settle(queue_name(), settle_op(), rabbit_types:ctag(),
@ -497,20 +501,40 @@ module(QRef, State) ->
{error, not_found}
end.
%% convenience function for throwaway publishes
-spec publish_at_most_once(rabbit_types:exchange() |
rabbit_exchange:name(),
mc:state()) ->
ok | {error, not_found}.
publish_at_most_once(#resource{} = XName, Msg) ->
case rabbit_exchange:lookup(XName) of
{ok, X} ->
publish_at_most_once(X, Msg);
Err ->
Err
end;
publish_at_most_once(X, Msg)
when element(1, X) == exchange -> % hacky but good enough
QNames = rabbit_exchange:route(X, Msg, #{return_binding_keys => true}),
Qs = rabbit_amqqueue:lookup_many(QNames),
_ = deliver(Qs, Msg, #{}, stateless),
ok.
-spec deliver([amqqueue:amqqueue() |
{amqqueue:amqqueue(), rabbit_exchange:route_infos()}],
rabbit_types:delivery(),
Message :: mc:state(),
delivery_options(),
stateless | state()) ->
{ok, state(), actions()} | {error, Reason :: term()}.
deliver(Qs, Delivery, State) ->
deliver(Qs, Message, Options, State) ->
try
deliver0(Qs, Delivery, State)
deliver0(Qs, Message, Options, State)
catch
exit:Reason ->
{error, Reason}
end.
deliver0(Qs, Delivery0, stateless) ->
deliver0(Qs, Message0, Options, stateless) ->
ByTypeAndBindingKeys =
lists:foldl(fun(Elem, Acc) ->
{Q, BKeys} = queue_binding_keys(Elem),
@ -521,11 +545,11 @@ deliver0(Qs, Delivery0, stateless) ->
end, [{Q, stateless}], Acc)
end, #{}, Qs),
maps:foreach(fun({Mod, BKeys}, QSs) ->
Delivery = add_binding_keys(Delivery0, BKeys),
_ = Mod:deliver(QSs, Delivery)
Message = add_binding_keys(Message0, BKeys),
_ = Mod:deliver(QSs, Message, Options)
end, ByTypeAndBindingKeys),
{ok, stateless, []};
deliver0(Qs, Delivery0, #?STATE{} = State0) ->
deliver0(Qs, Message0, Options, #?STATE{} = State0) ->
%% TODO: optimise single queue case?
%% sort by queue type - then dispatch each group
ByTypeAndBindingKeys =
@ -548,8 +572,8 @@ deliver0(Qs, Delivery0, #?STATE{} = State0) ->
%%% dispatch each group to queue type interface?
{Xs, Actions} = maps:fold(
fun({Mod, BKeys}, QSs, {X0, A0}) ->
Delivery = add_binding_keys(Delivery0, BKeys),
{X, A} = Mod:deliver(QSs, Delivery),
Message = add_binding_keys(Message0, BKeys),
{X, A} = Mod:deliver(QSs, Message, Options),
{X0 ++ X, A0 ++ A}
end, {[], []}, ByTypeAndBindingKeys),
State = lists:foldl(
@ -569,17 +593,11 @@ queue_binding_keys({Q, _RouteInfos})
when ?is_amqqueue(Q) ->
{Q, #{}}.
add_binding_keys(Delivery, BindingKeys)
add_binding_keys(Message, BindingKeys)
when map_size(BindingKeys) =:= 0 ->
Delivery;
add_binding_keys(Delivery, BindingKeys) ->
L = maps:fold(fun(B, true, Acc) ->
[{longstr, B} | Acc]
end, [], BindingKeys),
BasicMsg = rabbit_basic:add_header(
<<"x-binding-keys">>, array, L,
Delivery#delivery.message),
Delivery#delivery{message = BasicMsg}.
Message;
add_binding_keys(Message, BindingKeys) ->
mc:set_annotation(binding_keys, maps:keys(BindingKeys), Message).
-spec settle(queue_name(), settle_op(), rabbit_types:ctag(),
[non_neg_integer()], state()) ->

64
deps/rabbit/src/rabbit_quorum_queue.erl vendored
View File

@ -27,7 +27,7 @@
-export([settle/5, dequeue/5, consume/3, cancel/5]).
-export([credit/5]).
-export([purge/1]).
-export([stateless_deliver/2, deliver/2]).
-export([stateless_deliver/2, deliver/3]).
-export([dead_letter_publish/5]).
-export([cluster_state/1, status/2]).
-export([update_consumer_handler/8, update_consumer/9]).
@ -77,11 +77,11 @@
-include_lib("stdlib/include/qlc.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include("amqqueue.hrl").
-type msg_id() :: non_neg_integer().
-type qmsg() :: {rabbit_types:r('queue'), pid(), msg_id(), boolean(), rabbit_types:message()}.
-type qmsg() :: {rabbit_types:r('queue'), pid(), msg_id(), boolean(),
mc:state()}.
-define(RA_SYSTEM, quorum_queues).
-define(RA_WAL_NAME, ra_log_wal).
@ -894,39 +894,30 @@ stateless_deliver(ServerId, Delivery) ->
ok = rabbit_fifo_client:untracked_enqueue([ServerId],
Delivery#delivery.message).
-spec deliver(rabbit_amqqueue:name(), Confirm :: boolean(),
rabbit_types:delivery(), rabbit_fifo_client:state()) ->
{ok, rabbit_fifo_client:state(), rabbit_queue_type:actions()} |
{reject_publish, rabbit_fifo_client:state()}.
deliver(QName, false, Delivery, QState0) ->
case rabbit_fifo_client:enqueue(QName, Delivery#delivery.message, QState0) of
{ok, _State, _Actions} = Res ->
Res;
deliver0(QName, undefined, Msg, QState0) ->
case rabbit_fifo_client:enqueue(QName, Msg, QState0) of
{ok, _, _} = Res -> Res;
{reject_publish, State} ->
{ok, State, []}
end;
deliver(QName, true, Delivery, QState0) ->
rabbit_fifo_client:enqueue(QName,
Delivery#delivery.msg_seq_no,
Delivery#delivery.message, QState0).
deliver0(QName, Correlation, Msg, QState0) ->
rabbit_fifo_client:enqueue(QName, Correlation,
Msg, QState0).
deliver(QSs, #delivery{message = #basic_message{content = Content0} = Msg,
confirm = Confirm} = Delivery0) ->
%% TODO: we could also consider clearing out the message id here
Content = prepare_content(Content0),
Delivery = Delivery0#delivery{message = Msg#basic_message{content = Content}},
deliver(QSs, Msg0, Options) ->
Correlation = maps:get(correlation, Options, undefined),
Msg = mc:prepare(store, Msg0),
lists:foldl(
fun({Q, stateless}, {Qs, Actions}) ->
QRef = amqqueue:get_pid(Q),
ok = rabbit_fifo_client:untracked_enqueue(
[QRef], Delivery#delivery.message),
ok = rabbit_fifo_client:untracked_enqueue([QRef], Msg),
{Qs, Actions};
({Q, S0}, {Qs, Actions}) ->
QName = amqqueue:get_name(Q),
case deliver(QName, Confirm, Delivery, S0) of
case deliver0(QName, Correlation, Msg, S0) of
{reject_publish, S} ->
Seq = Delivery#delivery.msg_seq_no,
{[{Q, S} | Qs], [{rejected, QName, [Seq]} | Actions]};
{[{Q, S} | Qs],
[{rejected, QName, [Correlation]} | Actions]};
{ok, S, As} ->
{[{Q, S} | Qs], As ++ Actions}
end
@ -1579,9 +1570,12 @@ peek(Pos, Q) when ?is_amqqueue(Q) andalso ?amqqueue_is_quorum(Q) ->
#{delivery_count := C} -> C;
_ -> 0
end,
Msg = rabbit_basic:add_header(<<"x-delivery-count">>, long,
Count, Msg0),
{ok, rabbit_basic:peek_fmt_message(Msg)};
Msg = mc:set_annotation(<<"x-delivery-count">>, Count, Msg0),
XName = mc:get_annotation(exchange, Msg),
RoutingKeys = mc:get_annotation(routing_keys, Msg),
AmqpLegacyMsg = mc:prepare(read, mc:convert(mc_amqpl, Msg)),
Content = mc:protocol_state(AmqpLegacyMsg),
{ok, rabbit_basic:peek_fmt_message(XName, RoutingKeys, Content)};
{error, Err} ->
{error, Err};
Err ->
@ -1715,20 +1709,6 @@ notify_decorators(QName, F, A) ->
ok
end.
%% remove any data that a quorum queue doesn't need
prepare_content(#content{properties = none} = Content) ->
Content;
prepare_content(#content{protocol = none} = Content) ->
Content;
prepare_content(#content{properties = #'P_basic'{expiration = undefined} = Props,
protocol = Proto} = Content) ->
Content#content{properties = none,
properties_bin = Proto:encode_properties(Props)};
prepare_content(Content) ->
%% expiration is set. Therefore, leave properties decoded so that
%% rabbit_fifo can directly parse it without having to decode again.
Content.
ets_lookup_element(Tbl, Key, Pos, Default) ->
try ets:lookup_element(Tbl, Key, Pos) of
V -> V

113
deps/rabbit/src/rabbit_stream_queue.erl vendored
View File

@ -20,7 +20,7 @@
consume/3,
cancel/5,
handle_event/3,
deliver/2,
deliver/3,
settle/5,
credit/5,
dequeue/5,
@ -417,30 +417,33 @@ credit(QName, CTag, Credit, Drain, #stream_client{readers = Readers0,
false ->
{Readers1, []}
end,
{State#stream_client{readers = Readers}, [{send_credit_reply, length(Msgs)},
{deliver, CTag, true, Msgs}] ++ Actions}.
{State#stream_client{readers = Readers},
[{send_credit_reply, length(Msgs)},
{deliver, CTag, true, Msgs}] ++ Actions}.
deliver(QSs, #delivery{message = Msg, confirm = Confirm} = Delivery) ->
deliver(QSs, Msg, Options) ->
lists:foldl(
fun({Q, stateless}, {Qs, Actions}) ->
LeaderPid = amqqueue:get_pid(Q),
ok = osiris:write(LeaderPid,
stream_message(Msg, filtering_supported())),
{Qs, Actions};
({Q, S0}, {Qs, Actions}) ->
{S, As} = deliver(Confirm, Delivery, S0),
{[{Q, S} | Qs], As ++ Actions}
({Q, S0}, {Qs, Actions0}) ->
{S, Actions} = deliver0(maps:get(correlation, Options, undefined),
Msg, S0, Actions0),
{[{Q, S} | Qs], Actions}
end, {[], []}, QSs).
deliver(_Confirm, #delivery{message = Msg, msg_seq_no = MsgId},
#stream_client{name = Name,
leader = LeaderPid,
writer_id = WriterId,
next_seq = Seq,
correlation = Correlation0,
soft_limit = SftLmt,
slow = Slow0,
filtering_supported = FilteringSupported} = State) ->
deliver0(MsgId, Msg,
#stream_client{name = Name,
leader = LeaderPid,
writer_id = WriterId,
next_seq = Seq,
correlation = Correlation0,
soft_limit = SftLmt,
slow = Slow0,
filtering_supported = FilteringSupported} = State,
Actions0) ->
ok = osiris:write(LeaderPid, WriterId, Seq,
stream_message(Msg, FilteringSupported)),
Correlation = case MsgId of
@ -451,9 +454,9 @@ deliver(_Confirm, #delivery{message = Msg, msg_seq_no = MsgId},
end,
{Slow, Actions} = case maps:size(Correlation) >= SftLmt of
true when not Slow0 ->
{true, [{block, Name}]};
{true, [{block, Name} | Actions0]};
Bool ->
{Bool, []}
{Bool, Actions0}
end,
{State#stream_client{next_seq = Seq + 1,
correlation = Correlation,
@ -461,23 +464,15 @@ deliver(_Confirm, #delivery{message = Msg, msg_seq_no = MsgId},
stream_message(Msg, _FilteringSupported = true) ->
MsgData = msg_to_iodata(Msg),
case filter_header(Msg) of
{_, longstr, Value} ->
{Value, MsgData};
_ ->
MsgData
case mc:x_header(<<"x-stream-filter-value">>, Msg) of
undefined ->
MsgData;
{utf8, Value} ->
{Value, MsgData}
end;
stream_message(Msg, _FilteringSupported = false) ->
msg_to_iodata(Msg).
filter_header(Msg) ->
basic_header(<<"x-stream-filter-value">>, Msg).
basic_header(Key, #basic_message{content = Content}) ->
Headers = rabbit_basic:extract_headers(Content),
rabbit_basic:header(Key, Headers).
-spec dequeue(_, _, _, _, client()) -> no_return().
dequeue(_, _, _, _, #stream_client{name = Name}) ->
{protocol_error, not_implemented, "basic.get not supported by stream queues ~ts",
@ -559,8 +554,8 @@ recover(_VHost, Queues) ->
end, {[], []}, Queues).
settle(QName, complete, CTag, MsgIds, #stream_client{readers = Readers0,
local_pid = LocalPid,
name = Name} = State) ->
local_pid = LocalPid,
name = Name} = State) ->
Credit = length(MsgIds),
{Readers, Msgs} = case Readers0 of
#{CTag := #stream{credit = Credit0} = Str0} ->
@ -1038,8 +1033,7 @@ stream_entries(QName, Name, LocalPid,
{Records, Seg} ->
Msgs = [begin
Msg0 = binary_to_msg(QName, B),
Msg = rabbit_basic:add_header(<<"x-stream-offset">>,
long, O, Msg0),
Msg = mc:set_annotation(<<"x-stream-offset">>, O, Msg0),
{Name, LocalPid, O, false, Msg}
end || {O, B} <- Records,
O >= StartOffs],
@ -1061,41 +1055,24 @@ stream_entries(QName, Name, LocalPid,
stream_entries(_QName, _Name, _LocalPid, Str, Msgs) ->
{Str, Msgs}.
binary_to_msg(#resource{virtual_host = VHost,
kind = queue,
binary_to_msg(#resource{kind = queue,
name = QName}, Data) ->
R0 = rabbit_msg_record:init(Data),
%% if the message annotation isn't present the data most likely came from
%% the rabbitmq-stream plugin so we'll choose defaults that simulate use
%% of the direct exchange
{utf8, Exchange} = rabbit_msg_record:message_annotation(<<"x-exchange">>,
R0, {utf8, <<>>}),
{utf8, RoutingKey} = rabbit_msg_record:message_annotation(<<"x-routing-key">>,
R0, {utf8, QName}),
{Props, Payload} = rabbit_msg_record:to_amqp091(R0),
XName = #resource{kind = exchange,
virtual_host = VHost,
name = Exchange},
Content = #content{class_id = 60,
properties = Props,
properties_bin = none,
payload_fragments_rev = [Payload]},
{ok, Msg} = rabbit_basic:message(XName, RoutingKey, Content),
Msg.
Mc0 = mc:init(mc_amqp, amqp10_framing:decode_bin(Data), #{}),
%% If exchange or routing_keys annotation isn't present the data most likely came
%% from the rabbitmq-stream plugin so we'll choose defaults that simulate use
%% of the direct exchange.
Mc = case mc:get_annotation(exchange, Mc0) of
undefined -> mc:set_annotation(exchange, <<>>, Mc0);
_ -> Mc0
end,
case mc:get_annotation(routing_keys, Mc) of
undefined -> mc:set_annotation(routing_keys, [QName], Mc);
_ -> Mc
end.
msg_to_iodata(#basic_message{exchange_name = #resource{name = Exchange},
routing_keys = [RKey | _],
content = Content}) ->
#content{properties = Props,
payload_fragments_rev = Payload} =
rabbit_binary_parser:ensure_content_decoded(Content),
R0 = rabbit_msg_record:from_amqp091(Props, lists:reverse(Payload)),
%% TODO durable?
R = rabbit_msg_record:add_message_annotations(
#{<<"x-exchange">> => {utf8, Exchange},
<<"x-routing-key">> => {utf8, RKey}}, R0),
rabbit_msg_record:to_iodata(R).
msg_to_iodata(Msg0) ->
Sections = mc:protocol_state(mc:convert(mc_amqp, Msg0)),
mc_amqp:serialize(Sections).
capabilities() ->
#{unsupported_policies => [%% Classic policies

55
deps/rabbit/src/rabbit_trace.erl vendored
View File

@ -47,19 +47,18 @@ enabled(none) ->
enabled(#exchange{}) ->
true.
-spec tap_in(rabbit_types:basic_message(), rabbit_exchange:route_return(),
-spec tap_in(mc:state(), rabbit_exchange:route_return(),
binary(), rabbit_types:username(), state()) -> 'ok'.
tap_in(Msg, QNames, ConnName, Username, State) ->
tap_in(Msg, QNames, ConnName, ?CONNECTION_GLOBAL_CHANNEL_NUM, Username, State).
-spec tap_in(rabbit_types:basic_message(), rabbit_exchange:route_return(),
-spec tap_in(mc:state(), rabbit_exchange:route_return(),
binary(), rabbit_channel:channel_number(),
rabbit_types:username(), state()) -> 'ok'.
tap_in(_Msg, _QNames, _ConnName, _ChannelNum, _Username, none) -> ok;
tap_in(Msg = #basic_message{exchange_name = #resource{name = XName,
virtual_host = VHost}},
QNames, ConnName, ChannelNum, Username, TraceX) ->
tap_in(Msg, QNames, ConnName, ChannelNum, Username, TraceX) ->
XName = mc:get_annotation(exchange, Msg),
#exchange{name = #resource{virtual_host = VHost}} = TraceX,
RoutedQs = lists:map(fun(#resource{kind = queue, name = Name}) ->
{longstr, Name};
({#resource{kind = queue, name = Name}, _}) ->
@ -78,9 +77,8 @@ tap_out(Msg, ConnName, Username, State) ->
tap_out(Msg, ConnName, ?CONNECTION_GLOBAL_CHANNEL_NUM, Username, State).
-spec tap_out(rabbit_amqqueue:qmsg(), binary(),
rabbit_channel:channel_number(),
rabbit_types:username(), state()) -> 'ok'.
rabbit_channel:channel_number(),
rabbit_types:username(), state()) -> 'ok'.
tap_out(_Msg, _ConnName, _ChannelNum, _Username, none) -> ok;
tap_out({#resource{name = QName, virtual_host = VHost},
_QPid, _QMsgId, Redelivered, Msg},
@ -127,7 +125,6 @@ update_config(Fun) ->
VHosts0 = vhosts_with_tracing_enabled(),
VHosts = Fun(VHosts0),
application:set_env(rabbit, ?TRACE_VHOSTS, VHosts),
NonAmqpPids = rabbit_networking:local_non_amqp_connections(),
rabbit_log:debug("Will now refresh state of channels and of ~b non AMQP 0.9.1 "
"connections after virtual host tracing changes",
@ -142,21 +139,31 @@ vhosts_with_tracing_enabled() ->
%%----------------------------------------------------------------------------
trace(#exchange{name = Name}, #basic_message{exchange_name = Name},
_RKPrefix, _RKSuffix, _Extra) ->
ok;
trace(X, Msg = #basic_message{content = #content{payload_fragments_rev = PFR}},
RKPrefix, RKSuffix, Extra) ->
ok = rabbit_basic:publish(
X, <<RKPrefix/binary, ".", RKSuffix/binary>>,
#'P_basic'{headers = msg_to_table(Msg) ++ Extra}, PFR),
ok.
trace(X, Msg0, RKPrefix, RKSuffix, Extra) ->
XName = mc:get_annotation(exchange, Msg0),
case X of
#exchange{name = #resource{name = XName}} ->
ok;
#exchange{name = SourceXName} ->
RoutingKeys = mc:get_annotation(routing_keys, Msg0),
%% for now convert into amqp legacy
Msg = mc:prepare(read, mc:convert(mc_amqpl, Msg0)),
%% check exchange name in case it is same as target
#content{properties = Props} = Content0 =
mc:protocol_state(Msg),
msg_to_table(#basic_message{exchange_name = #resource{name = XName},
routing_keys = RoutingKeys,
content = Content}) ->
#content{properties = Props} =
rabbit_binary_parser:ensure_content_decoded(Content),
Key = <<RKPrefix/binary, ".", RKSuffix/binary>>,
Content = Content0#content{properties =
#'P_basic'{headers = msg_to_table(XName, RoutingKeys, Props)
++ Extra},
properties_bin = none},
TargetXName = SourceXName#resource{name = ?XNAME},
TraceMsg = mc_amqpl:message(TargetXName, Key, Content),
ok = rabbit_queue_type:publish_at_most_once(X, TraceMsg),
ok
end.
msg_to_table(XName, RoutingKeys, Props) ->
{PropsTable, _Ix} =
lists:foldl(fun (K, {L, Ix}) ->
V = element(Ix, Props),

99
deps/rabbit/src/rabbit_variable_queue.erl vendored
View File

@ -28,6 +28,7 @@
%% exported for testing only
-export([start_msg_store/3, stop_msg_store/1, init/5]).
-include("mc.hrl").
-include_lib("stdlib/include/qlc.hrl").
-define(QUEUE_MIGRATION_BATCH_SIZE, 100).
@ -458,8 +459,8 @@ init(Q, Terms, MsgOnDiskFun, MsgIdxOnDiskFun, MsgAndIdxOnDiskFun) when ?is_amqqu
MsgOnDiskFun, VHost),
{C, fun (MsgId) when is_binary(MsgId) ->
rabbit_msg_store:contains(MsgId, C);
(#basic_message{is_persistent = Persistent}) ->
Persistent
(Msg) ->
mc:is_persistent(Msg)
end};
false -> {undefined, fun(_MsgId) -> false end}
end,
@ -887,7 +888,8 @@ set_queue_mode(_, State) ->
State.
zip_msgs_and_acks(Msgs, AckTags, Accumulator, _State) ->
lists:foldl(fun ({{#basic_message{ id = Id }, _Props}, AckTag}, Acc) ->
lists:foldl(fun ({{Msg, _Props}, AckTag}, Acc) ->
Id = mc:get_annotation(id, Msg),
[{Id, AckTag} | Acc]
end, Accumulator, lists:zip(Msgs, AckTags)).
@ -992,8 +994,9 @@ convert_from_v1_to_v2_loop(QueueName, V1Index0, V2Index0, V2Store0,
garbage_collect(),
{V2Index3, V2Store3} = lists:foldl(fun
%% Move embedded messages to the per-queue store.
({Msg = #basic_message{id = MsgId}, SeqId, rabbit_queue_index, Props, IsPersistent},
({Msg, SeqId, rabbit_queue_index, Props, IsPersistent},
{V2Index1, V2Store1}) ->
MsgId = mc:get_annotation(id, Msg),
{MsgLocation, V2Store2} = rabbit_classic_queue_store_v2:write(SeqId, Msg, Props, V2Store1),
V2Index2 = case SkipFun(SeqId, V2Index1) of
{skip, V2Index1a} ->
@ -1197,10 +1200,10 @@ head_message_timestamp(Q3, RPA) ->
HeadMsgStatus#msg_status.msg /= undefined ],
Timestamps =
[Timestamp || HeadMsg <- HeadMsgs,
Timestamp <- [rabbit_basic:extract_timestamp(
HeadMsg#basic_message.content)],
Timestamp /= undefined
[Timestamp div 1000
|| HeadMsg <- HeadMsgs,
Timestamp <- [mc:timestamp(HeadMsg)],
Timestamp /= undefined
],
case Timestamps == [] of
@ -1268,7 +1271,8 @@ cons_if(true, E, L) -> [E | L];
cons_if(false, _E, L) -> L.
msg_status(Version, IsPersistent, IsDelivered, SeqId,
Msg = #basic_message {id = MsgId}, MsgProps, IndexMaxSize) ->
Msg, MsgProps, IndexMaxSize) ->
MsgId = mc:get_annotation(id, Msg),
#msg_status{seq_id = SeqId,
msg_id = MsgId,
msg = Msg,
@ -1281,8 +1285,19 @@ msg_status(Version, IsPersistent, IsDelivered, SeqId,
persist_to = determine_persist_to(Version, Msg, MsgProps, IndexMaxSize),
msg_props = MsgProps}.
beta_msg_status({Msg = #basic_message{id = MsgId},
SeqId, MsgLocation, MsgProps, IsPersistent}) ->
beta_msg_status({MsgId, SeqId, MsgLocation, MsgProps, IsPersistent})
when is_binary(MsgId) orelse
MsgId =:= undefined ->
MS0 = beta_msg_status0(SeqId, MsgProps, IsPersistent),
MS0#msg_status{msg_id = MsgId,
msg = undefined,
persist_to = case is_tuple(MsgLocation) of
true -> queue_store; %% @todo I'm not sure this clause is triggered anymore.
false -> msg_store
end,
msg_location = MsgLocation};
beta_msg_status({Msg, SeqId, MsgLocation, MsgProps, IsPersistent}) ->
MsgId = mc:get_annotation(id, Msg),
MS0 = beta_msg_status0(SeqId, MsgProps, IsPersistent),
MS0#msg_status{msg_id = MsgId,
msg = Msg,
@ -1294,17 +1309,7 @@ beta_msg_status({Msg = #basic_message{id = MsgId},
msg_location = case MsgLocation of
rabbit_queue_index -> memory;
_ -> MsgLocation
end};
beta_msg_status({MsgId, SeqId, MsgLocation, MsgProps, IsPersistent}) ->
MS0 = beta_msg_status0(SeqId, MsgProps, IsPersistent),
MS0#msg_status{msg_id = MsgId,
msg = undefined,
persist_to = case is_tuple(MsgLocation) of
true -> queue_store; %% @todo I'm not sure this clause is triggered anymore.
false -> msg_store
end,
msg_location = MsgLocation}.
end}.
beta_msg_status0(SeqId, MsgProps, IsPersistent) ->
#msg_status{seq_id = SeqId,
@ -1548,7 +1553,7 @@ read_msg(SeqId, _, _, MsgLocation, State = #vqstate{ store_state = StoreState0 }
{Msg, State#vqstate{ store_state = StoreState }};
read_msg(_, MsgId, IsPersistent, rabbit_msg_store, State = #vqstate{msg_store_clients = MSCState,
disk_read_count = Count}) ->
{{ok, Msg = #basic_message {}}, MSCState1} =
{{ok, Msg}, MSCState1} =
msg_store_read(MSCState, IsPersistent, MsgId),
{Msg, State #vqstate {msg_store_clients = MSCState1,
disk_read_count = Count + 1}}.
@ -1940,7 +1945,7 @@ process_delivers_and_acks_fun(_) ->
%% Internal gubbins for publishing
%%----------------------------------------------------------------------------
publish1(Msg = #basic_message { is_persistent = IsPersistent, id = MsgId },
publish1(Msg,
MsgProps = #message_properties { needs_confirming = NeedsConfirming },
IsDelivered, _ChPid, _Flow, PersistFun,
State = #vqstate { q3 = Q3, delta = Delta = #delta { count = DeltaCount },
@ -1954,6 +1959,8 @@ publish1(Msg = #basic_message { is_persistent = IsPersistent, id = MsgId },
unconfirmed = UC,
unconfirmed_simple = UCS,
rates = #rates{ out = OutRate }}) ->
MsgId = mc:get_annotation(id, Msg),
IsPersistent = mc:is_persistent(Msg),
IsPersistent1 = IsDurable andalso IsPersistent,
MsgStatus = msg_status(Version, IsPersistent1, IsDelivered, SeqId, Msg, MsgProps, IndexMaxSize),
%% We allow from 1 to 2048 messages in memory depending on the consume rate. The lower
@ -1990,8 +1997,9 @@ batch_publish1({Msg, MsgProps, IsDelivered}, {ChPid, Flow, State}) ->
{ChPid, Flow, publish1(Msg, MsgProps, IsDelivered, ChPid, Flow,
fun maybe_prepare_write_to_disk/4, State)}.
publish_delivered1(Msg = #basic_message { is_persistent = IsPersistent, id = MsgId },
MsgProps = #message_properties { needs_confirming = NeedsConfirming },
publish_delivered1(Msg,
MsgProps = #message_properties {
needs_confirming = NeedsConfirming },
_ChPid, _Flow, PersistFun,
State = #vqstate { version = Version,
qi_embed_msgs_below = IndexMaxSize,
@ -2002,6 +2010,8 @@ publish_delivered1(Msg = #basic_message { is_persistent = IsPersistent, id = Msg
durable = IsDurable,
unconfirmed = UC,
unconfirmed_simple = UCS }) ->
MsgId = mc:get_annotation(id, Msg),
IsPersistent = mc:is_persistent(Msg),
IsPersistent1 = IsDurable andalso IsPersistent,
MsgStatus = msg_status(Version, IsPersistent1, true, SeqId, Msg, MsgProps, IndexMaxSize),
{MsgStatus1, State1} = PersistFun(false, false, MsgStatus, State),
@ -2165,9 +2175,7 @@ maybe_prepare_write_to_disk(ForceMsg, ForceIndex0, MsgStatus, State = #vqstate{
maybe_batch_write_index_to_disk(ForceIndex, MsgStatus1, State1).
determine_persist_to(Version,
#basic_message{
content = #content{properties = Props,
properties_bin = PropsBin}},
Msg,
#message_properties{size = BodySize},
IndexMaxSize) ->
%% The >= is so that you can set the env to 0 and never persist
@ -2183,30 +2191,23 @@ determine_persist_to(Version,
%% case) we can just check their size. If we don't (message came
%% via the direct client), we make a guess based on the number of
%% headers.
case BodySize >= IndexMaxSize of
true -> msg_store;
false -> Est = case is_binary(PropsBin) of
true -> BodySize + size(PropsBin);
false -> #'P_basic'{headers = Hs} = Props,
case Hs of
undefined -> 0;
_ -> length(Hs)
end * ?HEADER_GUESS_SIZE + BodySize
end,
case Est >= IndexMaxSize of
true -> msg_store;
false when Version =:= 1 -> queue_index;
false when Version =:= 2 -> queue_store
end
end.
{MetaSize, _BodySize} = mc:size(Msg),
case BodySize >= IndexMaxSize of
true -> msg_store;
false ->
Est = MetaSize + BodySize,
case Est >= IndexMaxSize of
true -> msg_store;
false when Version =:= 1 -> queue_index;
false when Version =:= 2 -> queue_store
end
end.
persist_to(#msg_status{persist_to = To}) -> To.
prepare_to_store(Msg) ->
Msg#basic_message{
%% don't persist any recoverable decoded properties
content = rabbit_binary_parser:clear_decoded_content(
Msg #basic_message.content)}.
mc:prepare(store, Msg).
%%----------------------------------------------------------------------------
%% Internal gubbins for acks

58
deps/rabbit/test/backing_queue_SUITE.erl vendored
View File

@ -11,6 +11,7 @@
-include_lib("amqp_client/include/amqp_client.hrl").
-include("amqqueue.hrl").
-compile(nowarn_export_all).
-compile(export_all).
-define(PERSISTENT_MSG_STORE, msg_store_persistent).
@ -1479,24 +1480,12 @@ pub_res(VQS) ->
VQS.
make_publish(IsPersistent, PayloadFun, PropFun, N) ->
{rabbit_basic:message(
rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = case IsPersistent of
true -> 2;
false -> 1
end},
PayloadFun(N)),
{message(IsPersistent, PayloadFun, N),
PropFun(N, #message_properties{size = 10}),
false}.
make_publish_delivered(IsPersistent, PayloadFun, PropFun, N) ->
{rabbit_basic:message(
rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = case IsPersistent of
true -> 2;
false -> 1
end},
PayloadFun(N)),
{message(IsPersistent, PayloadFun, N),
PropFun(N, #message_properties{size = 10})}.
queue_name(Config, Name) ->
@ -1615,13 +1604,15 @@ publish_and_confirm(Q, Payload, Count) ->
QTState =
lists:foldl(
fun (Seq, Acc0) ->
Msg = rabbit_basic:message(rabbit_misc:r(<<>>, exchange, <<>>),
BMsg = rabbit_basic:message(rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = 2},
Payload),
Delivery = #delivery{mandatory = false, sender = self(),
confirm = true, message = Msg, msg_seq_no = Seq,
flow = noflow},
{ok, Acc, _Actions} = rabbit_queue_type:deliver([Q], Delivery, Acc0),
Content = BMsg#basic_message.content,
Ex = BMsg#basic_message.exchange_name,
Msg = mc_amqpl:message(Ex, <<>>, Content),
Options = #{correlation => Seq},
{ok, Acc, _Actions} = rabbit_queue_type:deliver([Q], Msg,
Options, Acc0),
Acc
end, QTState0, Seqs),
wait_for_confirms(sets:from_list(Seqs, [{version, 2}])),
@ -1687,15 +1678,9 @@ variable_queue_publish(IsPersistent, Start, Count, PropFun, PayloadFun, VQ) ->
variable_queue_wait_for_shuffling_end(
lists:foldl(
fun (N, VQN) ->
Msg = message(IsPersistent, PayloadFun, N),
rabbit_variable_queue:publish(
rabbit_basic:message(
rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = case IsPersistent of
true -> 2;
false -> 1
end},
PayloadFun(N)),
Msg,
PropFun(N, #message_properties{size = 10}),
false, self(), noflow, VQN)
end, VQ, lists:seq(Start, Start + Count - 1))).
@ -1738,9 +1723,9 @@ variable_queue_batch_publish0(IsPersistent, Start, Count, PropFun, PayloadFun,
variable_queue_fetch(Count, IsPersistent, IsDelivered, Len, VQ) ->
lists:foldl(fun (N, {VQN, AckTagsAcc}) ->
Rem = Len - N,
{{#basic_message { is_persistent = IsPersistent },
IsDelivered, AckTagN}, VQM} =
{{Msg, IsDelivered, AckTagN}, VQM} =
rabbit_variable_queue:fetch(true, VQN),
IsPersistent = mc:is_persistent(Msg),
Rem = rabbit_variable_queue:len(VQM),
{VQM, [AckTagN | AckTagsAcc]}
end, {VQ, []}, lists:seq(1, Count)).
@ -1792,6 +1777,9 @@ variable_queue_wait_for_shuffling_end(VQ) ->
end.
msg2int(#basic_message{content = #content{ payload_fragments_rev = P}}) ->
binary_to_term(list_to_binary(lists:reverse(P)));
msg2int(Msg) ->
#content{payload_fragments_rev = P} = mc:protocol_state(Msg),
binary_to_term(list_to_binary(lists:reverse(P))).
ack_subset(AckSeqs, Interval, Rem) ->
@ -1864,3 +1852,15 @@ flush() ->
after 0 ->
ok
end.
message(IsPersistent, PayloadFun, N) ->
#basic_message{content = Content,
exchange_name = Ex,
id = Id} =
rabbit_basic:message(rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = case IsPersistent of
true -> 2;
false -> 1
end},
PayloadFun(N)),
mc_amqpl:message(Ex, <<>>, Content, #{id => Id}).

22
deps/rabbit/test/classic_queue_prop_SUITE.erl vendored
View File

@ -795,14 +795,17 @@ do_check_queue_version(AMQ, Version, N) ->
cmd_publish_msg(St=#cq{amq=AMQ}, PayloadSize, DeliveryMode, Mandatory, Expiration) ->
?DEBUG("~0p ~0p ~0p ~0p ~0p", [St, PayloadSize, DeliveryMode, Mandatory, Expiration]),
Payload = do_rand_payload(PayloadSize),
Msg = rabbit_basic:message(rabbit_misc:r(<<>>, exchange, <<>>),
<<>>, #'P_basic'{delivery_mode = DeliveryMode,
expiration = do_encode_expiration(Expiration)},
Payload),
Delivery = #delivery{mandatory = Mandatory, sender = self(),
confirm = false, message = Msg, flow = noflow},
ok = rabbit_amqqueue:deliver([AMQ], Delivery),
{MsgProps, MsgPayload} = rabbit_basic_common:from_content(Msg#basic_message.content),
Ex = rabbit_misc:r(<<>>, exchange, <<>>),
BasicMsg = rabbit_basic:message(Ex, <<>>,
#'P_basic'{delivery_mode = DeliveryMode,
expiration = do_encode_expiration(Expiration)},
Payload),
Msg0 = mc_amqpl:message(Ex, <<>>, BasicMsg#basic_message.content),
Msg = mc:set_annotation(id, BasicMsg#basic_message.id, Msg0),
{ok, _, _} = rabbit_queue_type:deliver([AMQ], Msg, #{}, stateless),
Content = mc:protocol_state(Msg),
{MsgProps, MsgPayload} = rabbit_basic_common:from_content(Content),
#amqp_msg{props=MsgProps, payload=MsgPayload}.
cmd_basic_get_msg(St=#cq{amq=AMQ, limiter=LimiterPid}) ->
@ -815,7 +818,8 @@ cmd_basic_get_msg(St=#cq{amq=AMQ, limiter=LimiterPid}) ->
{empty, _} ->
empty;
{ok, _CountMinusOne, {_QName, _QPid, _AckTag, _IsDelivered, Msg}, _} ->
{MsgProps, MsgPayload} = rabbit_basic_common:from_content(Msg#basic_message.content),
Content = mc:protocol_state(Msg),
{MsgProps, MsgPayload} = rabbit_basic_common:from_content(Content),
#amqp_msg{props=MsgProps, payload=MsgPayload}
end.

1
deps/rabbit/test/cluster_SUITE.erl vendored
View File

@ -13,6 +13,7 @@
-include_lib("amqp_client/include/amqp_client.hrl").
-include("amqqueue.hrl").
-compile(nowarn_export_all).
-compile(export_all).
-define(TIMEOUT, 30000).

53
deps/rabbit/test/dead_lettering_SUITE.erl vendored
View File

@ -152,9 +152,13 @@ init_per_group(Group, Config) ->
{rmq_nodename_suffix, Group},
{rmq_nodes_count, ClusterSize}
]),
rabbit_ct_helpers:run_steps(Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps());
Config2 = rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2,
message_containers),
Config2;
false ->
rabbit_ct_helpers:run_steps(Config, [])
end.
@ -1034,8 +1038,10 @@ dead_letter_headers_cycle(Config) ->
QName = ?config(queue_name, Config),
DeadLetterArgs = [{<<"x-dead-letter-exchange">>, longstr, <<>>}],
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = QName, arguments = DeadLetterArgs ++ Args, durable = Durable}),
#'queue.declare_ok'{} =
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
arguments = DeadLetterArgs ++ Args,
durable = Durable}),
P = <<"msg1">>,
%% Publish message
@ -1047,8 +1053,9 @@ dead_letter_headers_cycle(Config) ->
multiple = false,
requeue = false}),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
{#'basic.get_ok'{delivery_tag = DTag1}, #amqp_msg{payload = P,
props = #'P_basic'{headers = Headers1}}} =
{#'basic.get_ok'{delivery_tag = DTag1},
#amqp_msg{payload = P,
props = #'P_basic'{headers = Headers1}}} =
amqp_channel:call(Ch, #'basic.get'{queue = QName}),
{array, [{table, Death1}]} = rabbit_misc:table_lookup(Headers1, <<"x-death">>),
?assertEqual({long, 1}, rabbit_misc:table_lookup(Death1, <<"count">>)),
@ -1130,10 +1137,12 @@ dead_letter_headers_CC_with_routing_key(Config) ->
DeadLetterArgs = [{<<"x-dead-letter-routing-key">>, longstr, DLXQName},
{<<"x-dead-letter-exchange">>, longstr, DLXExchange}],
#'exchange.declare_ok'{} = amqp_channel:call(Ch, #'exchange.declare'{exchange = DLXExchange}),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = QName, arguments = DeadLetterArgs ++ Args, durable = Durable}),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = DLXQName, durable = Durable}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = DLXQName,
exchange = DLXExchange,
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = QName,
arguments = DeadLetterArgs ++ Args, durable = Durable}),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = DLXQName,
durable = Durable}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = DLXQName,
exchange = DLXExchange,
routing_key = DLXQName}),
P1 = <<"msg1">>,
@ -1161,7 +1170,10 @@ dead_letter_headers_CC_with_routing_key(Config) ->
props = #'P_basic'{headers = Headers3}}} =
amqp_channel:call(Ch, #'basic.get'{queue = DLXQName}),
consume_empty(Ch, QName),
?assertEqual(undefined, rabbit_misc:table_lookup(Headers3, <<"CC">>)),
%% TODO: commented out assert,
%% this only checks that the message was mutated, which is bad not that
%% it wasn't included in routing
% ?assertEqual(undefined, rabbit_misc:table_lookup(Headers3, <<"CC">>)),
?assertMatch({array, _}, rabbit_misc:table_lookup(Headers3, <<"x-death">>)).
%% 16) the BCC header will always be removed
@ -1195,8 +1207,8 @@ dead_letter_headers_BCC(Config) ->
props = #'P_basic'{headers = Headers2}}} =
amqp_channel:call(Ch, #'basic.get'{queue = DLXQName}),
%% We check the headers to ensure no dead lettering has happened
?assertEqual(undefined, rabbit_misc:table_lookup(Headers1, <<"x-death">>)),
?assertEqual(undefined, rabbit_misc:table_lookup(Headers2, <<"x-death">>)),
?assertEqual(undefined, header_lookup(Headers1, <<"x-death">>)),
?assertEqual(undefined, header_lookup(Headers2, <<"x-death">>)),
%% Nack the message so it now gets dead lettered
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DTag1,
@ -1468,7 +1480,8 @@ declare_dead_letter_queues(Ch, Config, QName, DLXQName, ExtraArgs) ->
routing_key = DLXQName}).
publish(Ch, QName, Payloads) ->
[amqp_channel:call(Ch, #'basic.publish'{routing_key = QName}, #amqp_msg{payload = Payload})
[amqp_channel:call(Ch, #'basic.publish'{routing_key = QName},
#amqp_msg{payload = Payload})
|| Payload <- Payloads].
publish(Ch, QName, Payloads, Headers) ->
@ -1480,7 +1493,7 @@ publish(Ch, QName, Payloads, Headers) ->
consume(Ch, QName, Payloads) ->
[begin
{#'basic.get_ok'{delivery_tag = DTag}, #amqp_msg{payload = Payload}} =
amqp_channel:call(Ch, #'basic.get'{queue = QName}),
amqp_channel:call(Ch, #'basic.get'{queue = QName}),
DTag
end || Payload <- Payloads].
@ -1503,8 +1516,9 @@ counted(Metric, Config) ->
Strategy = group_name(Config),
OldCounters = ?config(counters, Config),
Counters = get_global_counters(Config),
ct:pal("Counters ~p", [Counters]),
metric(QueueType, Strategy, Metric, Counters) -
metric(QueueType, Strategy, Metric, OldCounters).
metric(QueueType, Strategy, Metric, OldCounters).
metric(QueueType, Strategy, Metric, Counters) ->
Metrics = maps:get([{queue_type, QueueType}, {dead_letter_strategy, Strategy}], Counters),
@ -1521,3 +1535,8 @@ queue_type(quorum_queue) ->
rabbit_quorum_queue;
queue_type(_) ->
rabbit_classic_queue.
header_lookup(undefined, _Key) ->
undefined;
header_lookup(Headers, Key) ->
rabbit_misc:table_lookup(Headers, Key).

14
deps/rabbit/test/dynamic_ha_SUITE.erl vendored
View File

@ -27,6 +27,7 @@
-include_lib("amqp_client/include/amqp_client.hrl").
-include_lib("rabbitmq_ct_helpers/include/rabbit_assert.hrl").
-compile(nowarn_export_all).
-compile(export_all).
-define(QNAME, <<"ha.test">>).
@ -106,9 +107,18 @@ init_per_testcase(Testcase, Config) ->
{rmq_nodename_suffix, Testcase},
{tcp_ports_base, {skip_n_nodes, TestNumber * ClusterSize}}
]),
rabbit_ct_helpers:run_steps(Config1,
Config2 = rabbit_ct_helpers:run_steps(Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()).
rabbit_ct_client_helpers:setup_steps()),
case Testcase of
change_cluster ->
%% do not enable message_containers feature flag as it will stop
%% nodes in mixed versions joining later
ok;
_ ->
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers)
end,
Config2.
end_per_testcase(Testcase, Config) ->
Config1 = rabbit_ct_helpers:run_steps(Config,

10
deps/rabbit/test/dynamic_qq_SUITE.erl vendored
View File

@ -74,10 +74,12 @@ init_per_testcase(Testcase, Config) ->
{queue_name, Q},
{queue_args, [{<<"x-queue-type">>, longstr, <<"quorum">>}]}
]),
rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps())
Config2 = rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers),
Config2
end.
end_per_testcase(Testcase, Config) ->

15
deps/rabbit/test/eager_sync_SUITE.erl vendored
View File

@ -65,12 +65,15 @@ init_per_testcase(Testcase, Config) ->
{rmq_nodename_suffix, Testcase},
{tcp_ports_base, {skip_n_nodes, TestNumber * ClusterSize}}
]),
rabbit_ct_helpers:run_steps(Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps() ++ [
fun rabbit_ct_broker_helpers:set_ha_policy_two_pos/1,
fun rabbit_ct_broker_helpers:set_ha_policy_two_pos_batch_sync/1
]).
Config2 = rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps() ++ [
fun rabbit_ct_broker_helpers:set_ha_policy_two_pos/1,
fun rabbit_ct_broker_helpers:set_ha_policy_two_pos_batch_sync/1
]),
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers),
Config2.
end_per_testcase(Testcase, Config) ->
Config1 = rabbit_ct_helpers:run_steps(Config,

656
deps/rabbit/test/mc_SUITE.erl vendored Normal file
View File

@ -0,0 +1,656 @@
-module(mc_SUITE).
-compile([export_all, nowarn_export_all]).
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp10_common/include/amqp10_framing.hrl").
-include_lib("rabbit/include/mc.hrl").
%%%===================================================================
%%% Common Test callbacks
%%%===================================================================
all() ->
[
{group, tests}
].
all_tests() ->
[
amqpl_defaults,
amqpl_compat,
amqpl_table_x_header,
amqpl_table_x_header_array_of_tbls,
amqpl_death_records,
amqpl_amqp_bin_amqpl,
amqp_amqpl,
amqp_to_amqpl_data_body
].
groups() ->
[
{tests, [], all_tests()}
].
%%%===================================================================
%%% Test cases
%%%===================================================================
amqpl_defaults(_Config) ->
Props = #'P_basic'{},
Payload = [<<"data">>],
Content = #content{properties = Props,
payload_fragments_rev = Payload},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg = mc:init(mc_amqpl, Content, Anns),
?assertEqual(undefined, mc:priority(Msg)),
?assertEqual(false, mc:is_persistent(Msg)),
?assertEqual(undefined, mc:timestamp(Msg)),
?assertEqual(undefined, mc:correlation_id(Msg)),
?assertEqual(undefined, mc:message_id(Msg)),
?assertEqual(undefined, mc:ttl(Msg)),
?assertEqual(undefined, mc:x_header("x-fruit", Msg)),
ok.
amqpl_compat(_Config) ->
Props = #'P_basic'{content_type = <<"text/plain">>,
content_encoding = <<"gzip">>,
headers = [{<<"a-stream-offset">>, long, 99},
{<<"a-string">>, longstr, <<"a string">>},
{<<"a-bool">>, bool, false},
{<<"a-unsignedbyte">>, unsignedbyte, 1},
{<<"a-unsignedshort">>, unsignedshort, 1},
{<<"a-unsignedint">>, unsignedint, 1},
{<<"a-signedint">>, signedint, 1},
{<<"a-timestamp">>, timestamp, 1},
{<<"a-double">>, double, 1.0},
{<<"a-float">>, float, 1.0},
{<<"a-void">>, void, undefined},
{<<"a-binary">>, binary, <<"data">>},
{<<"x-stream-filter">>, longstr, <<"apple">>}
],
delivery_mode = 1,
priority = 98,
correlation_id = <<"corr">> ,
reply_to = <<"reply-to">>,
expiration = <<"1">>,
message_id = <<"msg-id">>,
timestamp = 99,
type = <<"45">>,
user_id = <<"banana">>,
app_id = <<"rmq">>
},
Payload = [<<"data">>],
Content = #content{properties = Props,
payload_fragments_rev = Payload},
XName= <<"exch">>,
RoutingKey = <<"apple">>,
{ok, Msg} = rabbit_basic:message_no_id(XName, RoutingKey, Content),
?assertEqual(98, mc:priority(Msg)),
?assertEqual(false, mc:is_persistent(Msg)),
?assertEqual(99000, mc:timestamp(Msg)),
?assertEqual({utf8, <<"corr">>}, mc:correlation_id(Msg)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(Msg)),
?assertEqual(1, mc:ttl(Msg)),
?assertEqual({utf8, <<"apple">>}, mc:x_header(<<"x-stream-filter">>, Msg)),
RoutingHeaders = mc:routing_headers(Msg, []),
ct:pal("routing headers ~p", [RoutingHeaders]),
?assertMatch(#{<<"a-binary">> := <<"data">>,
<<"a-bool">> := false,
<<"a-double">> := 1.0,
<<"a-float">> := 1.0,
<<"a-signedint">> := 1,
<<"a-stream-offset">> := 99,
<<"a-string">> := <<"a string">>,
<<"a-timestamp">> := 1000,
<<"a-unsignedbyte">> := 1,
<<"a-unsignedint">> := 1,
<<"a-unsignedshort">> := 1,
<<"a-void">> := undefined}, RoutingHeaders),
RoutingHeadersX = mc:routing_headers(Msg, [x_headers]),
?assertMatch(#{<<"a-binary">> := <<"data">>,
<<"a-bool">> := false,
<<"a-double">> := 1.0,
<<"a-float">> := 1.0,
<<"a-signedint">> := 1,
<<"a-stream-offset">> := 99,
<<"a-string">> := <<"a string">>,
<<"a-timestamp">> := 1000,
<<"a-unsignedbyte">> := 1,
<<"a-unsignedint">> := 1,
<<"a-unsignedshort">> := 1,
<<"a-void">> := undefined,
<<"x-stream-filter">> := <<"apple">>}, RoutingHeadersX),
ok.
amqpl_table_x_header(_Config) ->
Tbl = [{<<"type">>, longstr, <<"apple">>},
{<<"count">>, long, 99}],
Props = #'P_basic'{headers = [
{<<"x-fruit">>, table, Tbl},
{<<"fruit">>, table, Tbl}
]},
Payload = [<<"data">>],
Content = #content{properties = Props,
payload_fragments_rev = Payload},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg = mc:init(mc_amqpl, Content, Anns),
%% x-header values come back AMQP 1.0 ish formatted
?assertMatch({map,
[{{symbol, <<"type">>}, {utf8, <<"apple">>}},
{{symbol, <<"count">>}, {long, 99}}]},
mc:x_header(<<"x-fruit">>, Msg)),
%% non x-headers should not show up
% ?assertEqual(undefined, mc:x_header(<<"fruit">>, Msg)),
?assertMatch(#{<<"fruit">> := _,
<<"x-fruit">> := _},
mc:routing_headers(Msg, [x_headers])),
ok.
amqpl_table_x_header_array_of_tbls(_Config) ->
Tbl1 = [{<<"type">>, longstr, <<"apple">>},
{<<"count">>, long, 99}],
Tbl2 = [{<<"type">>, longstr, <<"orange">>},
{<<"count">>, long, 45}],
Props = #'P_basic'{headers = [
{<<"x-fruit">>, array, [{table, Tbl1},
{table, Tbl2}]}
]},
Payload = [<<"data">>],
Content = #content{properties = Props,
payload_fragments_rev = Payload},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg = mc:init(mc_amqpl, Content, Anns),
?assertMatch({list,
[{map,
[{{symbol, <<"type">>}, {utf8, <<"apple">>}},
{{symbol, <<"count">>}, {long, 99}}]},
{map,
[{{symbol, <<"type">>}, {utf8, <<"orange">>}},
{{symbol, <<"count">>}, {long, 45}}]}
]},
mc:x_header(<<"x-fruit">>, Msg)),
ok.
amqpl_death_records(_Config) ->
Content = #content{class_id = 60,
properties = #'P_basic'{headers = []},
payload_fragments_rev = [<<"data">>]},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg0 = mc:prepare(store, mc:init(mc_amqpl, Content, Anns)),
Msg1 = mc:record_death(rejected, <<"q1">>, Msg0),
?assertEqual([<<"q1">>], mc:death_queue_names(Msg1)),
?assertMatch({{<<"q1">>, rejected},
#death{exchange = <<"exch">>,
routing_keys = [<<"apple">>],
count = 1}}, mc:last_death(Msg1)),
?assertEqual(false, mc:is_death_cycle(<<"q1">>, Msg1)),
#content{properties = #'P_basic'{headers = H1}} = mc:protocol_state(Msg1),
?assertMatch({_, array, [_]}, header(<<"x-death">>, H1)),
?assertMatch({_, longstr, <<"q1">>}, header(<<"x-first-death-queue">>, H1)),
?assertMatch({_, longstr, <<"q1">>}, header(<<"x-last-death-queue">>, H1)),
?assertMatch({_, longstr, <<"exch">>}, header(<<"x-first-death-exchange">>, H1)),
?assertMatch({_, longstr, <<"exch">>}, header(<<"x-last-death-exchange">>, H1)),
?assertMatch({_, longstr, <<"rejected">>}, header(<<"x-first-death-reason">>, H1)),
?assertMatch({_, longstr, <<"rejected">>}, header(<<"x-last-death-reason">>, H1)),
{_, array, [{table, T1}]} = header(<<"x-death">>, H1),
?assertMatch({_, long, 1}, header(<<"count">>, T1)),
?assertMatch({_, longstr, <<"rejected">>}, header(<<"reason">>, T1)),
?assertMatch({_, longstr, <<"q1">>}, header(<<"queue">>, T1)),
?assertMatch({_, longstr, <<"exch">>}, header(<<"exchange">>, T1)),
?assertMatch({_, timestamp, _}, header(<<"time">>, T1)),
?assertMatch({_, array, [{longstr, <<"apple">>}]}, header(<<"routing-keys">>, T1)),
%% second dead letter, e.g. a ttl reason returning to source queue
%% record_death uses a timestamp for death record ordering, ensure
%% it is definitely higher than the last timestamp taken
timer:sleep(2),
Msg2 = mc:record_death(ttl, <<"dl">>, Msg1),
#content{properties = #'P_basic'{headers = H2}} = mc:protocol_state(Msg2),
{_, array, [{table, T2a}, {table, T2b}]} = header(<<"x-death">>, H2),
?assertMatch({_, longstr, <<"dl">>}, header(<<"queue">>, T2a)),
?assertMatch({_, longstr, <<"q1">>}, header(<<"queue">>, T2b)),
ct:pal("H2 ~p", [T2a]),
ct:pal("routing headers ~p", [mc:routing_headers(Msg2, [x_headers])]),
ok.
header(K, H) ->
rabbit_basic:header(K, H).
amqpl_amqp_bin_amqpl(_Config) ->
%% incoming amqpl converted to amqp, serialized / deserialized then converted
%% back to amqpl.
%% simulates a legacy message published then consumed to a stream
Props = #'P_basic'{content_type = <<"text/plain">>,
content_encoding = <<"gzip">>,
headers = [{<<"a-stream-offset">>, long, 99},
{<<"a-string">>, longstr, <<"a string">>},
{<<"a-bool">>, bool, false},
{<<"a-unsignedbyte">>, unsignedbyte, 1},
{<<"a-unsignedshort">>, unsignedshort, 1},
{<<"a-unsignedint">>, unsignedint, 1},
{<<"a-signedint">>, signedint, 1},
{<<"a-timestamp">>, timestamp, 1},
{<<"a-double">>, double, 1.0},
{<<"a-float">>, float, 1.0},
{<<"a-void">>, void, undefined},
{<<"a-binary">>, binary, <<"data">>},
{<<"x-stream-filter">>, longstr, <<"apple">>}
],
delivery_mode = 2,
priority = 98,
correlation_id = <<"corr">> ,
reply_to = <<"reply-to">>,
expiration = <<"1">>,
message_id = <<"msg-id">>,
timestamp = 99,
type = <<"45">>,
user_id = <<"banana">>,
app_id = <<"rmq">>
},
Payload = [<<"data">>],
Content = #content{properties = Props,
payload_fragments_rev = Payload},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg = mc:init(mc_amqpl, Content, Anns),
?assertEqual(<<"exch">>, mc:get_annotation(exchange, Msg)),
?assertEqual([<<"apple">>], mc:get_annotation(routing_keys, Msg)),
?assertEqual(98, mc:priority(Msg)),
?assertEqual(true, mc:is_persistent(Msg)),
?assertEqual(99000, mc:timestamp(Msg)),
?assertEqual({utf8, <<"corr">>}, mc:correlation_id(Msg)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(Msg)),
?assertEqual(1, mc:ttl(Msg)),
?assertEqual({utf8, <<"apple">>}, mc:x_header(<<"x-stream-filter">>, Msg)),
RoutingHeaders = mc:routing_headers(Msg, []),
%% roundtrip to binary
Msg10Pre = mc:convert(mc_amqp, Msg),
Sections = amqp10_framing:decode_bin(
iolist_to_binary(amqp_serialize(Msg10Pre))),
Msg10 = mc:init(mc_amqp, Sections, #{}),
?assertEqual(<<"exch">>, mc:get_annotation(exchange, Msg10)),
?assertEqual([<<"apple">>], mc:get_annotation(routing_keys, Msg10)),
?assertEqual(98, mc:priority(Msg10)),
?assertEqual(true, mc:is_persistent(Msg10)),
?assertEqual(99000, mc:timestamp(Msg10)),
?assertEqual({utf8, <<"corr">>}, mc:correlation_id(Msg10)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(Msg10)),
?assertEqual(1, mc:ttl(Msg10)),
?assertEqual({utf8, <<"apple">>}, mc:x_header(<<"x-stream-filter">>, Msg10)),
?assertEqual(RoutingHeaders, mc:routing_headers(Msg10, [])),
MsgL2 = mc:convert(mc_amqpl, Msg10),
?assertEqual(<<"exch">>, mc:get_annotation(exchange, MsgL2)),
?assertEqual([<<"apple">>], mc:get_annotation(routing_keys, MsgL2)),
?assertEqual(98, mc:priority(MsgL2)),
?assertEqual(true, mc:is_persistent(MsgL2)),
?assertEqual(99000, mc:timestamp(MsgL2)),
?assertEqual({utf8, <<"corr">>}, mc:correlation_id(MsgL2)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(MsgL2)),
?assertEqual(1, mc:ttl(MsgL2)),
?assertEqual({utf8, <<"apple">>}, mc:x_header(<<"x-stream-filter">>, MsgL2)),
?assertEqual(RoutingHeaders, mc:routing_headers(MsgL2, [])),
ok.
thead2(T, Value) ->
{symbol(atom_to_binary(T)), {T, Value}}.
thead(T, Value) ->
{utf8(atom_to_binary(T)), {T, Value}}.
amqp_amqpl(_Config) ->
H = #'v1_0.header'{priority = {ubyte, 3},
ttl = {uint, 20000},
durable = true},
MAC = [
{{symbol, <<"x-stream-filter">>}, {utf8, <<"apple">>}},
thead2(list, [utf8(<<"1">>)]),
thead2(map, [{utf8(<<"k">>), utf8(<<"v">>)}])
],
M = #'v1_0.message_annotations'{content = MAC},
P = #'v1_0.properties'{content_type = {symbol, <<"ctype">>},
content_encoding = {symbol, <<"cenc">>},
message_id = {utf8, <<"msg-id">>},
correlation_id = {utf8, <<"corr-id">>},
user_id = {binary, <<"user-id">>},
reply_to = {utf8, <<"reply-to">>},
group_id = {utf8, <<"group-id">>},
creation_time = {timestamp, 10000}
},
AC = [
thead(long, 5),
thead(ulong, 5),
thead(utf8, <<"a-string">>),
thead(binary, <<"data">>),
thead(ubyte, 1),
thead(short, 2),
thead(ushort, 3),
thead(uint, 4),
thead(int, 4),
thead(double, 5.0),
thead(float, 6.0),
thead(timestamp, 7000),
thead(byte, 128),
thead(boolean, true),
{utf8(<<"null">>), null}
],
A = #'v1_0.application_properties'{content = AC},
D = #'v1_0.data'{content = <<"data">>},
Anns = #{exchange => <<"exch">>,
routing_keys => [<<"apple">>]},
Msg = mc:init(mc_amqp, [H, M, P, A, D], Anns),
%% validate source data is serialisable
_ = amqp_serialize(Msg),
?assertEqual(3, mc:priority(Msg)),
?assertEqual(true, mc:is_persistent(Msg)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(Msg)),
?assertEqual({utf8, <<"corr-id">>}, mc:correlation_id(Msg)),
MsgL = mc:convert(mc_amqpl, Msg),
?assertEqual(3, mc:priority(MsgL)),
?assertEqual(true, mc:is_persistent(MsgL)),
?assertEqual({utf8, <<"msg-id">>}, mc:message_id(MsgL)),
#content{properties = #'P_basic'{headers = HL} = Props} = Content = mc:protocol_state(MsgL),
?assertMatch(#'P_basic'{user_id = <<"user-id">>}, Props),
?assertMatch(#'P_basic'{reply_to = <<"reply-to">>}, Props),
?assertMatch(#'P_basic'{content_type = <<"ctype">>}, Props),
?assertMatch(#'P_basic'{content_encoding = <<"cenc">>}, Props),
?assertMatch(#'P_basic'{app_id = <<"group-id">>}, Props),
?assertMatch(#'P_basic'{timestamp = 10}, Props),
?assertMatch(#'P_basic'{delivery_mode = 2}, Props),
?assertMatch(#'P_basic'{priority = 3}, Props),
?assertMatch(#'P_basic'{expiration = <<"20000">>}, Props),
?assertMatch({_, longstr, <<"apple">>}, header(<<"x-stream-filter">>, HL)),
?assertMatch({_, long, 5}, header(<<"long">>, HL)),
?assertMatch({_, long, 5}, header(<<"ulong">>, HL)),
?assertMatch({_, longstr, <<"a-string">>}, header(<<"utf8">>, HL)),
?assertMatch({_, binary, <<"data">>}, header(<<"binary">>, HL)),
?assertMatch({_, unsignedbyte, 1}, header(<<"ubyte">>, HL)),
?assertMatch({_, short, 2}, header(<<"short">>, HL)),
?assertMatch({_, unsignedshort, 3}, header(<<"ushort">>, HL)),
?assertMatch({_, unsignedint, 4}, header(<<"uint">>, HL)),
?assertMatch({_, signedint, 4}, header(<<"int">>, HL)),
?assertMatch({_, double, 5.0}, header(<<"double">>, HL)),
?assertMatch({_, float, 6.0}, header(<<"float">>, HL)),
?assertMatch({_, timestamp, 7}, header(<<"timestamp">>, HL)),
?assertMatch({_, byte, 128}, header(<<"byte">>, HL)),
?assertMatch({_, bool, true}, header(<<"boolean">>, HL)),
?assertMatch({_, void, undefined}, header(<<"null">>, HL)),
%% validate content is serialisable
_ = rabbit_binary_generator:build_simple_content_frames(1, Content,
1000000,
rabbit_framing_amqp_0_9_1),
ok.
amqp_to_amqpl_data_body(_Config) ->
Cases = [#'v1_0.data'{content = <<"helloworld">>},
#'v1_0.data'{content = [<<"hello">>, <<"world">>]}],
lists:foreach(
fun(Section) ->
Sections = case is_list(Section) of
true -> Section;
false -> [Section]
end,
Mc0 = mc:init(mc_amqp, Sections, #{}),
Mc = mc:convert(mc_amqpl, Mc0),
#content{payload_fragments_rev = PayFragRev} = mc:protocol_state(Mc),
PayFrag = lists:reverse(PayFragRev),
?assertEqual(<<"helloworld">>,
iolist_to_binary(PayFrag))
end, Cases).
amqp10_non_single_data_bodies(_Config) ->
Props = #'P_basic'{type = <<"amqp-1.0">>},
Payloads = [
[#'v1_0.data'{content = <<"hello">>},
#'v1_0.data'{content = <<"brave">>},
#'v1_0.data'{content = <<"new">>},
#'v1_0.data'{content = <<"world">>}
],
#'v1_0.amqp_value'{content = {utf8, <<"hello world">>}},
[#'v1_0.amqp_sequence'{content = [{utf8, <<"one">>},
{utf8, <<"blah">>}]},
#'v1_0.amqp_sequence'{content = [{utf8, <<"two">>}]}
]
],
[begin
EncodedPayload = amqp10_encode_bin(Payload),
MsgRecord0 = rabbit_msg_record:from_amqp091(Props, EncodedPayload),
MsgRecord = rabbit_msg_record:init(
iolist_to_binary(rabbit_msg_record:to_iodata(MsgRecord0))),
{PropsOut, PayloadEncodedOut} = rabbit_msg_record:to_amqp091(MsgRecord),
PayloadOut = case amqp10_framing:decode_bin(iolist_to_binary(PayloadEncodedOut)) of
L when length(L) =:= 1 ->
lists:nth(1, L);
L ->
L
end,
?assertEqual(Props, PropsOut),
?assertEqual(iolist_to_binary(EncodedPayload),
iolist_to_binary(PayloadEncodedOut)),
?assertEqual(Payload, PayloadOut)
end || Payload <- Payloads],
ok.
unsupported_091_header_is_dropped(_Config) ->
Props = #'P_basic'{
headers = [
{<<"x-received-from">>, array, []}
]
},
MsgRecord0 = rabbit_msg_record:from_amqp091(Props, <<"payload">>),
MsgRecord = rabbit_msg_record:init(
iolist_to_binary(rabbit_msg_record:to_iodata(MsgRecord0))),
% meck:unload(),
{PropsOut, <<"payload">>} = rabbit_msg_record:to_amqp091(MsgRecord),
?assertMatch(#'P_basic'{headers = undefined}, PropsOut),
ok.
message_id_ulong(_Config) ->
Num = 9876789,
ULong = erlang:integer_to_binary(Num),
P = #'v1_0.properties'{message_id = {ulong, Num},
correlation_id = {ulong, Num}},
D = #'v1_0.data'{content = <<"data">>},
Bin = [amqp10_framing:encode_bin(P),
amqp10_framing:encode_bin(D)],
R = rabbit_msg_record:init(iolist_to_binary(Bin)),
{Props, _} = rabbit_msg_record:to_amqp091(R),
?assertMatch(#'P_basic'{message_id = ULong,
correlation_id = ULong,
headers =
[
%% ordering shouldn't matter
{<<"x-correlation-id-type">>, longstr, <<"ulong">>},
{<<"x-message-id-type">>, longstr, <<"ulong">>}
]},
Props),
ok.
message_id_uuid(_Config) ->
%% fake a uuid
UUId = erlang:md5(term_to_binary(make_ref())),
TextUUId = rabbit_data_coercion:to_binary(rabbit_guid:to_string(UUId)),
P = #'v1_0.properties'{message_id = {uuid, UUId},
correlation_id = {uuid, UUId}},
D = #'v1_0.data'{content = <<"data">>},
Bin = [amqp10_framing:encode_bin(P),
amqp10_framing:encode_bin(D)],
R = rabbit_msg_record:init(iolist_to_binary(Bin)),
{Props, _} = rabbit_msg_record:to_amqp091(R),
?assertMatch(#'P_basic'{message_id = TextUUId,
correlation_id = TextUUId,
headers =
[
%% ordering shouldn't matter
{<<"x-correlation-id-type">>, longstr, <<"uuid">>},
{<<"x-message-id-type">>, longstr, <<"uuid">>}
]},
Props),
ok.
message_id_binary(_Config) ->
%% fake a uuid
Orig = <<"asdfasdf">>,
Text = base64:encode(Orig),
P = #'v1_0.properties'{message_id = {binary, Orig},
correlation_id = {binary, Orig}},
D = #'v1_0.data'{content = <<"data">>},
Bin = [amqp10_framing:encode_bin(P),
amqp10_framing:encode_bin(D)],
R = rabbit_msg_record:init(iolist_to_binary(Bin)),
{Props, _} = rabbit_msg_record:to_amqp091(R),
?assertMatch(#'P_basic'{message_id = Text,
correlation_id = Text,
headers =
[
%% ordering shouldn't matter
{<<"x-correlation-id-type">>, longstr, <<"binary">>},
{<<"x-message-id-type">>, longstr, <<"binary">>}
]},
Props),
ok.
message_id_large_binary(_Config) ->
%% cannot fit in a shortstr
Orig = crypto:strong_rand_bytes(500),
P = #'v1_0.properties'{message_id = {binary, Orig},
correlation_id = {binary, Orig}},
D = #'v1_0.data'{content = <<"data">>},
Bin = [amqp10_framing:encode_bin(P),
amqp10_framing:encode_bin(D)],
R = rabbit_msg_record:init(iolist_to_binary(Bin)),
{Props, _} = rabbit_msg_record:to_amqp091(R),
?assertMatch(#'P_basic'{message_id = undefined,
correlation_id = undefined,
headers =
[
%% ordering shouldn't matter
{<<"x-correlation-id">>, longstr, Orig},
{<<"x-message-id">>, longstr, Orig}
]},
Props),
ok.
message_id_large_string(_Config) ->
%% cannot fit in a shortstr
Orig = base64:encode(crypto:strong_rand_bytes(500)),
P = #'v1_0.properties'{message_id = {utf8, Orig},
correlation_id = {utf8, Orig}},
D = #'v1_0.data'{content = <<"data">>},
Bin = [amqp10_framing:encode_bin(P),
amqp10_framing:encode_bin(D)],
R = rabbit_msg_record:init(iolist_to_binary(Bin)),
{Props, _} = rabbit_msg_record:to_amqp091(R),
?assertMatch(#'P_basic'{message_id = undefined,
correlation_id = undefined,
headers =
[
%% ordering shouldn't matter
{<<"x-correlation-id">>, longstr, Orig},
{<<"x-message-id">>, longstr, Orig}
]},
Props),
ok.
reuse_amqp10_binary_chunks(_Config) ->
Amqp10MsgAnnotations = #'v1_0.message_annotations'{content =
[{{symbol, <<"x-route">>}, {utf8, <<"dummy">>}}]},
Amqp10MsgAnnotationsBin = amqp10_encode_bin(Amqp10MsgAnnotations),
Amqp10Props = #'v1_0.properties'{group_id = {utf8, <<"my-group">>},
group_sequence = {uint, 42}},
Amqp10PropsBin = amqp10_encode_bin(Amqp10Props),
Amqp10AppProps = #'v1_0.application_properties'{content = [{{utf8, <<"foo">>}, {utf8, <<"bar">>}}]},
Amqp10AppPropsBin = amqp10_encode_bin(Amqp10AppProps),
Amqp091Headers = [{<<"x-amqp-1.0-message-annotations">>, longstr, Amqp10MsgAnnotationsBin},
{<<"x-amqp-1.0-properties">>, longstr, Amqp10PropsBin},
{<<"x-amqp-1.0-app-properties">>, longstr, Amqp10AppPropsBin}],
Amqp091Props = #'P_basic'{type= <<"amqp-1.0">>, headers = Amqp091Headers},
Body = #'v1_0.amqp_value'{content = {utf8, <<"hello world">>}},
EncodedBody = amqp10_encode_bin(Body),
R = rabbit_msg_record:from_amqp091(Amqp091Props, EncodedBody),
RBin = rabbit_msg_record:to_iodata(R),
Amqp10DecodedMsg = amqp10_framing:decode_bin(iolist_to_binary(RBin)),
[Amqp10DecodedMsgAnnotations, Amqp10DecodedProps,
Amqp10DecodedAppProps, DecodedBody] = Amqp10DecodedMsg,
?assertEqual(Amqp10MsgAnnotations, Amqp10DecodedMsgAnnotations),
?assertEqual(Amqp10Props, Amqp10DecodedProps),
?assertEqual(Amqp10AppProps, Amqp10DecodedAppProps),
?assertEqual(Body, DecodedBody),
ok.
amqp10_encode_bin(L) when is_list(L) ->
iolist_to_binary([amqp10_encode_bin(X) || X <- L]);
amqp10_encode_bin(X) ->
iolist_to_binary(amqp10_framing:encode_bin(X)).
%% Utility
test_amqp091_roundtrip(Props, Payload) ->
MsgRecord0 = rabbit_msg_record:from_amqp091(Props, Payload),
MsgRecord = rabbit_msg_record:init(
iolist_to_binary(rabbit_msg_record:to_iodata(MsgRecord0))),
% meck:unload(),
{PropsOut, PayloadOut} = rabbit_msg_record:to_amqp091(MsgRecord),
?assertEqual(Props, PropsOut),
?assertEqual(iolist_to_binary(Payload),
iolist_to_binary(PayloadOut)),
ok.
utf8(V) ->
{utf8, V}.
symbol(V) ->
{symbol, V}.
amqp_serialize(Msg) ->
mc_amqp:serialize(mc:protocol_state(Msg)).

226
deps/rabbit/test/message_containers_SUITE.erl vendored Normal file
View File

@ -0,0 +1,226 @@
-module(message_containers_SUITE).
-compile([export_all, nowarn_export_all]).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-define(FEATURE_FLAG, message_containers).
%%%===================================================================
%%% Common Test callbacks
%%%===================================================================
all() ->
[
{group, classic},
{group, quorum},
{group, stream}
].
groups() ->
[
{classic, [], all_tests()},
{quorum, [], all_tests()},
{stream, [], all_tests()}
].
all_tests() ->
[
enable_ff
].
init_per_suite(Config0) ->
rabbit_ct_helpers:log_environment(),
Config = rabbit_ct_helpers:merge_app_env(
Config0, {rabbit, [{quorum_tick_interval, 1000}]}),
rabbit_ct_helpers:run_setup_steps(Config).
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config),
ok.
init_per_group(Group, Config) ->
ct:pal("init per group ~p", [Group]),
ClusterSize = 3,
Config1 = rabbit_ct_helpers:set_config(Config,
[{rmq_nodes_count, ClusterSize},
{rmq_nodename_suffix, Group},
{tcp_ports_base}]),
Config1b = rabbit_ct_helpers:set_config(Config1,
[{queue_type, atom_to_binary(Group, utf8)},
{net_ticktime, 10}]),
Config1c = rabbit_ct_helpers:merge_app_env(
Config1b, {rabbit, [{forced_feature_flags_on_init, []}]}),
Config2 = rabbit_ct_helpers:run_steps(Config1c,
[fun merge_app_env/1 ] ++
rabbit_ct_broker_helpers:setup_steps()),
ok = rabbit_ct_broker_helpers:rpc(
Config2, 0, application, set_env,
[rabbit, channel_tick_interval, 100]),
AllFFs = rabbit_ct_broker_helpers:rpc(Config2, rabbit_feature_flags, list, [all, stable]),
FFs = maps:keys(maps:remove(?FEATURE_FLAG, AllFFs)),
ct:pal("FFs ~p", [FFs]),
rabbit_ct_broker_helpers:set_policy(Config2, 0,
<<"ha-policy">>, <<".*">>, <<"queues">>,
[{<<"ha-mode">>, <<"all">>}]),
Config2.
merge_app_env(Config) ->
rabbit_ct_helpers:merge_app_env(
rabbit_ct_helpers:merge_app_env(Config,
{rabbit,
[{core_metrics_gc_interval, 100},
{log, [{file, [{level, debug}]}]}]}),
{ra, [{min_wal_roll_over_interval, 30000}]}).
end_per_group(_Group, Config) ->
rabbit_ct_helpers:run_steps(Config,
rabbit_ct_broker_helpers:teardown_steps()).
init_per_testcase(Testcase, Config) ->
case rabbit_ct_broker_helpers:is_feature_flag_supported(Config, ?FEATURE_FLAG) of
false ->
{skip, "feature flag message_containers is unsupported"};
true ->
Config1 = rabbit_ct_helpers:testcase_started(Config, Testcase),
?assertNot(rabbit_ct_broker_helpers:is_feature_flag_enabled(Config, ?FEATURE_FLAG)),
Q = rabbit_data_coercion:to_binary(Testcase),
Config2 = rabbit_ct_helpers:set_config(Config1,
[{queue_name, Q},
{alt_queue_name, <<Q/binary, "_alt">>}
]),
rabbit_ct_helpers:run_steps(Config2,
rabbit_ct_client_helpers:setup_steps())
end.
end_per_testcase(Testcase, Config) ->
rabbit_ct_broker_helpers:rpc(Config, 0, ?MODULE, delete_queues, []),
Config1 = rabbit_ct_helpers:run_steps(
Config,
rabbit_ct_client_helpers:teardown_steps()),
rabbit_ct_helpers:testcase_finished(Config1, Testcase).
%%%===================================================================
%%% Test cases
%%%===================================================================
enable_ff(Config) ->
Server = rabbit_ct_broker_helpers:get_node_config(Config, 0, nodename),
Ch = rabbit_ct_client_helpers:open_channel(Config, Server),
QName = ?config(queue_name, Config),
?assertEqual({'queue.declare_ok', QName, 0, 0},
declare(Ch, QName, [{<<"x-queue-type">>, longstr,
?config(queue_type, Config)}])),
#'confirm.select_ok'{} = amqp_channel:call(Ch, #'confirm.select'{}),
amqp_channel:register_confirm_handler(Ch, self()),
timer:sleep(100),
ConsumerTag1 = <<"ctag1">>,
Ch2 = rabbit_ct_client_helpers:open_channel(Config, 2),
qos(Ch2, 2),
ok = subscribe(Ch2, QName, ConsumerTag1),
publish_and_confirm(Ch, QName, <<"msg1">>),
receive_and_ack(Ch2),
%% consume
publish(Ch, QName, <<"msg2">>),
ok = rabbit_ct_broker_helpers:enable_feature_flag(Config, ?FEATURE_FLAG),
confirm(),
publish_and_confirm(Ch, QName, <<"msg3">>),
receive_and_ack(Ch2),
receive_and_ack(Ch2).
receive_and_ack(Ch) ->
receive
{#'basic.deliver'{delivery_tag = DeliveryTag,
redelivered = false},
#amqp_msg{}} ->
basic_ack(Ch, DeliveryTag)
after 5000 ->
flush(),
exit(basic_deliver_timeout)
end.
%% Utility
delete_queues() ->
[{ok, 0} = rabbit_amqqueue:delete(Q, false, false, <<"dummy">>)
|| Q <- rabbit_amqqueue:list()].
declare(Ch, Q, Args) ->
amqp_channel:call(Ch, #'queue.declare'{queue = Q,
durable = true,
auto_delete = false,
arguments = Args}).
delete(Ch, Q) ->
amqp_channel:call(Ch, #'queue.delete'{queue = Q}).
publish(Ch, Queue, Msg) ->
ok = amqp_channel:cast(Ch,
#'basic.publish'{routing_key = Queue},
#amqp_msg{props = #'P_basic'{delivery_mode = 2},
payload = Msg}).
publish_and_confirm(Ch, Queue, Msg) ->
publish(Ch, Queue, Msg),
ct:pal("waiting for ~ts message confirmation from ~ts", [Msg, Queue]),
confirm().
confirm() ->
ok = receive
#'basic.ack'{} -> ok;
#'basic.nack'{} -> fail
after 2500 ->
flush(),
exit(confirm_timeout)
end.
subscribe(Ch, Queue, CTag) ->
amqp_channel:subscribe(Ch, #'basic.consume'{queue = Queue,
no_ack = false,
consumer_tag = CTag},
self()),
receive
#'basic.consume_ok'{consumer_tag = CTag} ->
ok
after 5000 ->
exit(basic_consume_timeout)
end.
basic_ack(Ch, DTag) ->
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DTag,
multiple = false}).
basic_cancel(Ch, CTag) ->
#'basic.cancel_ok'{} =
amqp_channel:call(Ch, #'basic.cancel'{consumer_tag = CTag}).
basic_nack(Ch, DTag) ->
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DTag,
requeue = true,
multiple = false}).
flush() ->
receive
Any ->
ct:pal("flush ~tp", [Any]),
flush()
after 0 ->
ok
end.
get_global_counters(Config) ->
rabbit_ct_broker_helpers:rpc(Config, 0, rabbit_global_counters, overview, []).
qos(Ch, Prefetch) ->
?assertMatch(#'basic.qos_ok'{},
amqp_channel:call(Ch, #'basic.qos'{prefetch_count = Prefetch})).

35
deps/rabbit/test/priority_queue_SUITE.erl vendored
View File

@ -11,6 +11,7 @@
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-compile(nowarn_export_all).
-compile(export_all).
all() ->
@ -58,9 +59,12 @@ init_per_group(single_node, Config) ->
{rmq_nodes_count, 1},
{rmq_nodename_suffix, Suffix}
]),
rabbit_ct_helpers:run_steps(Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps());
Config2 = rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers),
Config2;
init_per_group(overflow_reject_publish, Config) ->
rabbit_ct_helpers:set_config(Config, [
{overflow, <<"reject-publish">>}
@ -351,6 +355,7 @@ info_head_message_timestamp(Config) ->
info_head_message_timestamp1(_Config) ->
QName = rabbit_misc:r(<<"/">>, queue,
<<"info_head_message_timestamp-queue">>),
ExName = rabbit_misc:r(<<"/">>, exchange, <<>>),
Q0 = rabbit_amqqueue:pseudo_queue(QName, self()),
Q1 = amqqueue:set_arguments(Q0, [{<<"x-max-priority">>, long, 2}]),
PQ = rabbit_priority_queue,
@ -359,27 +364,17 @@ info_head_message_timestamp1(_Config) ->
true = PQ:is_empty(BQS1),
'' = PQ:info(head_message_timestamp, BQS1),
%% Publish one message with timestamp 1000.
Msg1 = #basic_message{
id = <<"msg1">>,
content = #content{
properties = #'P_basic'{
priority = 1,
timestamp = 1000
}},
is_persistent = false
},
Content1 = #content{properties = #'P_basic'{priority = 1,
timestamp = 1000},
payload_fragments_rev = []},
Msg1 = mc_amqpl:message(ExName, <<>>, Content1, #{id => <<"msg1">>}),
BQS2 = PQ:publish(Msg1, #message_properties{size = 0}, false, self(),
noflow, BQS1),
1000 = PQ:info(head_message_timestamp, BQS2),
%% Publish a higher priority message with no timestamp.
Msg2 = #basic_message{
id = <<"msg2">>,
content = #content{
properties = #'P_basic'{
priority = 2
}},
is_persistent = false
},
Content2 = #content{properties = #'P_basic'{priority = 2},
payload_fragments_rev = []},
Msg2 = mc_amqpl:message(ExName, <<>>, Content2, #{id => <<"msg2">>}),
BQS3 = PQ:publish(Msg2, #message_properties{size = 0}, false, self(),
noflow, BQS2),
'' = PQ:info(head_message_timestamp, BQS3),

11
deps/rabbit/test/publisher_confirms_parallel_SUITE.erl vendored
View File

@ -265,7 +265,7 @@ confirm_nack1(Config) ->
rabbit_channel:do(Ch, #'queue.bind'{
queue = QName,
exchange = <<"amq.direct">>,
routing_key = "confirms-magic" }),
routing_key = <<"confirms-magic">>}),
receive #'queue.bind_ok'{} -> ok
after ?TIMEOUT -> throw(failed_to_bind_queue)
end
@ -286,7 +286,7 @@ confirm_nack1(Config) ->
end,
%% Publish a message
rabbit_channel:do(Ch, #'basic.publish'{exchange = <<"amq.direct">>,
routing_key = "confirms-magic"
routing_key = <<"confirms-magic">>
},
rabbit_basic:build_content(
#'P_basic'{delivery_mode = 2}, <<"">>)),
@ -332,10 +332,13 @@ confirm_minority(Config) ->
publish(Ch, QName, [<<"msg2">>]),
receive
#'basic.nack'{} -> throw(unexpected_nack);
#'basic.ack'{} -> ok
#'basic.ack'{} ->
ok
after 60000 ->
throw(missing_ack)
end.
end,
ok = rabbit_ct_broker_helpers:start_node(Config, C),
ok.
%%%%%%%%%%%%%%%%%%%%%%%%
%% Test helpers

3
deps/rabbit/test/queue_parallel_SUITE.erl vendored
View File

@ -8,13 +8,12 @@
-module(queue_parallel_SUITE).
-include_lib("common_test/include/ct.hrl").
-include_lib("kernel/include/file.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-include_lib("eunit/include/eunit.hrl").
-compile(nowarn_export_all).
-compile(export_all).
-define(TIMEOUT, 30_000).
all() ->
[

61
deps/rabbit/test/queue_type_SUITE.erl vendored
View File

@ -13,7 +13,8 @@
all() ->
[
{group, classic},
{group, quorum}
{group, quorum},
{group, stream}
].
@ -26,7 +27,11 @@ all_tests() ->
groups() ->
[
{classic, [], all_tests()},
{quorum, [], all_tests()}
{quorum, [], all_tests()},
{stream, [],
[
stream
]}
].
init_per_suite(Config0) ->
@ -40,6 +45,7 @@ end_per_suite(Config) ->
ok.
init_per_group(Group, Config) ->
ct:pal("init per group ~p", [Group]),
ClusterSize = 3,
Config1 = rabbit_ct_helpers:set_config(Config,
[{rmq_nodes_count, ClusterSize},
@ -63,6 +69,9 @@ init_per_group(Group, Config) ->
_ ->
Config2
end,
EnableFF = rabbit_ct_broker_helpers:enable_feature_flag(Config3,
message_containers),
ct:pal("message_containers ff ~p", [EnableFF]),
rabbit_ct_broker_helpers:set_policy(
Config3, 0,
@ -154,7 +163,7 @@ smoke(Config) ->
%% get and ack
basic_ack(Ch, basic_get(Ch, QName)),
%% global counters
publish_and_confirm(Ch, <<"non-existent_queue">>, <<"msg4">>),
ok = publish_and_confirm(Ch, <<"non-existent_queue">>, <<"msg4">>),
ConsumerTag3 = <<"ctag3">>,
ok = subscribe(Ch, QName, ConsumerTag3),
ProtocolCounters = maps:get([{protocol, amqp091}], get_global_counters(Config)),
@ -221,8 +230,45 @@ ack_after_queue_delete(Config) ->
flush(),
ok.
stream(Config) ->
Server = rabbit_ct_broker_helpers:get_node_config(Config, 0, nodename),
Ch = rabbit_ct_client_helpers:open_channel(Config, Server),
QName = ?config(queue_name, Config),
?assertEqual({'queue.declare_ok', QName, 0, 0},
declare(Ch, QName, [{<<"x-queue-type">>, longstr,
?config(queue_type, Config)}])),
#'confirm.select_ok'{} = amqp_channel:call(Ch, #'confirm.select'{}),
amqp_channel:register_confirm_handler(Ch, self()),
publish_and_confirm(Ch, QName, <<"msg1">>),
Args = [{<<"x-stream-offset">>, longstr, <<"last">>}],
SubCh = rabbit_ct_client_helpers:open_channel(Config, 2),
qos(SubCh, 10, false),
try
amqp_channel:subscribe(
SubCh, #'basic.consume'{queue = QName,
consumer_tag = <<"ctag">>,
arguments = Args},
self()),
receive
{#'basic.deliver'{delivery_tag = T,
redelivered = false},
#amqp_msg{}} ->
basic_ack(SubCh, T)
after 5000 ->
exit(basic_deliver_timeout)
end
catch
_:Err ->
ct:pal("basic.consume error ~p", [Err]),
exit(Err)
end,
ok.
%% Utility
%%
delete_queues() ->
[rabbit_amqqueue:delete(Q, false, false, <<"dummy">>)
|| Q <- rabbit_amqqueue:list()].
@ -244,7 +290,7 @@ publish(Ch, Queue, Msg) ->
publish_and_confirm(Ch, Queue, Msg) ->
publish(Ch, Queue, Msg),
ct:pal("waiting for ~ts message confirmation from ~ts", [Msg, Queue]),
ct:pal("xwaiting for ~ts message confirmation from ~ts", [Msg, Queue]),
ok = receive
#'basic.ack'{} -> ok;
#'basic.nack'{} -> fail
@ -300,3 +346,8 @@ flush() ->
get_global_counters(Config) ->
rabbit_ct_broker_helpers:rpc(Config, 0, rabbit_global_counters, overview, []).
qos(Ch, Prefetch, Global) ->
?assertMatch(#'basic.qos_ok'{},
amqp_channel:call(Ch, #'basic.qos'{global = Global,
prefetch_count = Prefetch})).

7
deps/rabbit/test/quorum_queue_SUITE.erl vendored
View File

@ -190,10 +190,11 @@ init_per_group(clustered_with_partitions, Config0) ->
true ->
{skip, "clustered_with_partitions is too unreliable in mixed mode"};
false ->
Config = rabbit_ct_helpers:run_setup_steps(
Config1 = rabbit_ct_helpers:run_setup_steps(
Config0,
[fun rabbit_ct_broker_helpers:configure_dist_proxy/1]),
rabbit_ct_helpers:set_config(Config, [{net_ticktime, 10}])
Config2 = rabbit_ct_helpers:set_config(Config1, [{net_ticktime, 10}]),
Config2
end;
init_per_group(Group, Config) ->
ClusterSize = case Group of
@ -220,6 +221,7 @@ init_per_group(Group, Config) ->
{skip, _} ->
Ret;
Config2 ->
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers),
ok = rabbit_ct_broker_helpers:rpc(
Config2, 0, application, set_env,
[rabbit, channel_tick_interval, 100]),
@ -2186,6 +2188,7 @@ subscribe_redelivery_count(Config) ->
{#'basic.deliver'{delivery_tag = DeliveryTag1,
redelivered = true},
#amqp_msg{props = #'P_basic'{headers = H1}}} ->
ct:pal("H1 ~p", [H1]),
?assertMatch({DCHeader, _, 1}, rabbit_basic:header(DCHeader, H1)),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag1,
multiple = false,

4
deps/rabbit/test/rabbit_confirms_SUITE.erl vendored
View File

@ -1,6 +1,8 @@
-module(rabbit_confirms_SUITE).
-compile([export_all, nowarn_export_all]).
-compile([export_all,
nowarn_export_all]).
-include_lib("eunit/include/eunit.hrl").
%%%===================================================================

2
deps/rabbit/test/rabbit_core_metrics_gc_SUITE.erl vendored
View File

@ -7,10 +7,10 @@
-module(rabbit_core_metrics_gc_SUITE).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-compile(nowarn_export_all).
-compile(export_all).
all() ->

1
deps/rabbit/test/rabbit_fifo_dlx_integration_SUITE.erl vendored
View File

@ -96,6 +96,7 @@ init_per_group(Group, Config, NodesCount) ->
Config2 = rabbit_ct_helpers:run_steps(Config1,
[fun merge_app_env/1 ] ++
rabbit_ct_broker_helpers:setup_steps()),
_ = rabbit_ct_broker_helpers:enable_feature_flag(Config2, message_containers),
ok = rpc(Config2, 0, application, set_env,
[rabbit, channel_tick_interval, 100]),
Config2.

2
deps/rabbit/test/rabbit_msg_record_SUITE.erl vendored
View File

@ -1,12 +1,12 @@
-module(rabbit_msg_record_SUITE).
-compile(nowarn_export_all).
-compile(export_all).
-export([
]).
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp10_common/include/amqp10_framing.hrl").

1
deps/rabbit/test/rabbit_stream_queue_SUITE.erl vendored
View File

@ -1027,6 +1027,7 @@ consume_timestamp_offset(Config) ->
#'basic.consume_ok'{consumer_tag = <<"ctag">>} ->
ok
after 5000 ->
flush(),
exit(consume_ok_timeout)
end,

10
deps/rabbit/test/unit_access_control_SUITE.erl vendored
View File

@ -1,4 +1,4 @@
%% This Source Code Form is subject to the terms of the Mozilla Public
% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%
@ -427,10 +427,10 @@ test_topic_expect_match(X, List) ->
BinKey = list_to_binary(Key),
Message = rabbit_basic:message(X#exchange.name, BinKey,
#'P_basic'{}, <<>>),
Res = rabbit_exchange_type_topic:route(
X, #delivery{mandatory = false,
sender = self(),
message = Message}),
Msg = mc_amqpl:message(X#exchange.name,
BinKey,
Message#basic_message.content),
Res = rabbit_exchange_type_topic:route(X, Msg),
ExpectedRes = lists:map(
fun (Q) -> #resource{virtual_host = <<"/">>,
kind = queue,

130
deps/rabbit/test/unit_classic_mirrored_queue_sync_throttling_SUITE.erl vendored
View File

@ -9,80 +9,76 @@
-compile(export_all).
all() ->
[
maybe_master_batch_send,
get_time_diff,
append_to_acc
].
[
maybe_master_batch_send,
get_time_diff,
append_to_acc
].
maybe_master_batch_send(_Config) ->
SyncBatchSize = 4096,
SyncThroughput = 2000,
QueueLen = 10000,
?assertEqual(
true, %% Message reach the last one in the queue
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {0, 0, SyncThroughput}, {QueueLen, QueueLen}, 0}, SyncBatchSize)),
?assertEqual(
true, %% # messages batched is less than batch size; and total message size has reached the batch size
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {0, 0, SyncThroughput}, {SyncBatchSize, QueueLen}, 0}, SyncBatchSize)),
TotalBytes0 = SyncThroughput + 1,
Curr0 = 1,
?assertEqual(
true, %% Total batch size exceed max sync throughput
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {TotalBytes0, 0, SyncThroughput}, {Curr0, QueueLen}, 0}, SyncBatchSize)),
TotalBytes1 = 1,
Curr1 = 1,
?assertEqual(
false, %% # messages batched is less than batch size; and total bytes is less than sync throughput
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {TotalBytes1, 0, SyncThroughput}, {Curr1, QueueLen}, 0}, SyncBatchSize)),
ok.
SyncBatchSize = 4096,
SyncThroughput = 2000,
QueueLen = 10000,
?assertEqual(
true, %% Message reach the last one in the queue
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {0, 0, SyncThroughput}, {QueueLen, QueueLen}, 0}, SyncBatchSize)),
?assertEqual(
true, %% # messages batched is less than batch size; and total message size has reached the batch size
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {0, 0, SyncThroughput}, {SyncBatchSize, QueueLen}, 0}, SyncBatchSize)),
TotalBytes0 = SyncThroughput + 1,
Curr0 = 1,
?assertEqual(
true, %% Total batch size exceed max sync throughput
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {TotalBytes0, 0, SyncThroughput}, {Curr0, QueueLen}, 0}, SyncBatchSize)),
TotalBytes1 = 1,
Curr1 = 1,
?assertEqual(
false, %% # messages batched is less than batch size; and total bytes is less than sync throughput
rabbit_mirror_queue_sync:maybe_master_batch_send({[], 0, {TotalBytes1, 0, SyncThroughput}, {Curr1, QueueLen}, 0}, SyncBatchSize)),
ok.
get_time_diff(_Config) ->
TotalBytes0 = 100,
Interval0 = 1000, %% ms
MaxSyncThroughput0 = 100, %% bytes/s
?assertEqual(%% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; matched max throughput
0, %% => no need to pause queue sync
rabbit_mirror_queue_sync:get_time_diff(TotalBytes0, Interval0, MaxSyncThroughput0)),
TotalBytes0 = 100,
Interval0 = 1000, %% ms
MaxSyncThroughput0 = 100, %% bytes/s
?assertEqual(%% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; matched max throughput
0, %% => no need to pause queue sync
rabbit_mirror_queue_sync:get_time_diff(TotalBytes0, Interval0, MaxSyncThroughput0)),
TotalBytes1 = 100,
Interval1 = 1000, %% ms
MaxSyncThroughput1 = 200, %% bytes/s
?assertEqual( %% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; less than max throughput
0, %% => no need to pause queue sync
rabbit_mirror_queue_sync:get_time_diff(TotalBytes1, Interval1, MaxSyncThroughput1)),
TotalBytes1 = 100,
Interval1 = 1000, %% ms
MaxSyncThroughput1 = 200, %% bytes/s
?assertEqual( %% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; less than max throughput
0, %% => no need to pause queue sync
rabbit_mirror_queue_sync:get_time_diff(TotalBytes1, Interval1, MaxSyncThroughput1)),
TotalBytes2 = 100,
Interval2 = 1000, %% ms
MaxSyncThroughput2 = 50, %% bytes/s
?assertEqual( %% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; greater than max throughput
1000, %% => pause queue sync for 1000 ms
rabbit_mirror_queue_sync:get_time_diff(TotalBytes2, Interval2, MaxSyncThroughput2)),
ok.
TotalBytes2 = 100,
Interval2 = 1000, %% ms
MaxSyncThroughput2 = 50, %% bytes/s
?assertEqual( %% Used throughput = 100 / 1000 * 1000 = 100 bytes/s; greater than max throughput
1000, %% => pause queue sync for 1000 ms
rabbit_mirror_queue_sync:get_time_diff(TotalBytes2, Interval2, MaxSyncThroughput2)),
ok.
append_to_acc(_Config) ->
Msg = #basic_message{
id = 1,
content = #content{
properties = #'P_basic'{
priority = 2
},
payload_fragments_rev = [[<<"1234567890">>]] %% 10 bytes
},
is_persistent = true
},
BQDepth = 10,
SyncThroughput_0 = 0,
FoldAcc1 = {[], 0, {0, erlang:monotonic_time(), SyncThroughput_0}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes1, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc1),
?assertEqual(0, TotalBytes1), %% Skipping calculating TotalBytes for the pending batch as SyncThroughput is 0.
Content = #content{properties = #'P_basic'{delivery_mode = 2,
priority = 2},
payload_fragments_rev = [[<<"1234567890">>]] %% 10 bytes
},
ExName = rabbit_misc:r(<<>>, exchange, <<>>),
Msg = mc_amqpl:message(ExName, <<>>, Content, #{id => 1}, true),
BQDepth = 10,
SyncThroughput_0 = 0,
FoldAcc1 = {[], 0, {0, erlang:monotonic_time(), SyncThroughput_0}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes1, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc1),
?assertEqual(0, TotalBytes1), %% Skipping calculating TotalBytes for the pending batch as SyncThroughput is 0.
SyncThroughput = 100,
FoldAcc2 = {[], 0, {0, erlang:monotonic_time(), SyncThroughput}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes2, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc2),
?assertEqual(10, TotalBytes2), %% Message size is added to existing TotalBytes
SyncThroughput = 100,
FoldAcc2 = {[], 0, {0, erlang:monotonic_time(), SyncThroughput}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes2, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc2),
?assertEqual(10, TotalBytes2), %% Message size is added to existing TotalBytes
FoldAcc3 = {[], 0, {TotalBytes2, erlang:monotonic_time(), SyncThroughput}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes3, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc3),
?assertEqual(TotalBytes2 + 10, TotalBytes3), %% Message size is added to existing TotalBytes
ok.
FoldAcc3 = {[], 0, {TotalBytes2, erlang:monotonic_time(), SyncThroughput}, {0, BQDepth}, erlang:monotonic_time()},
{_, _, {TotalBytes3, _, _}, _, _} = rabbit_mirror_queue_sync:append_to_acc(Msg, {}, false, FoldAcc3),
?assertEqual(TotalBytes2 + 10, TotalBytes3), %% Message size is added to existing TotalBytes
ok.

2
deps/rabbit_common/include/rabbit.hrl vendored
View File

@ -121,7 +121,7 @@
{mandatory, %% Whether the message was published as mandatory
confirm, %% Whether the message needs confirming
sender, %% The pid of the process that created the delivery
message, %% The #basic_message record
message, %% The message container
msg_seq_no, %% Msg Sequence Number from the channel publish_seqno field
flow}). %% Should flow control be used for this delivery

41
deps/rabbit_common/src/rabbit_misc.erl vendored
View File

@ -10,7 +10,6 @@
-ignore_xref([{maps, get, 2}]).
-include("rabbit.hrl").
-include("rabbit_framing.hrl").
-include("rabbit_misc.hrl").
-include_lib("kernel/include/file.hrl").
@ -79,10 +78,15 @@
-export([raw_read_file/1]).
-export([find_child/2]).
-export([is_regular_file/1]).
-export([maps_any/2]).
-export([safe_ets_update_counter/3, safe_ets_update_counter/4, safe_ets_update_counter/5,
safe_ets_update_element/3, safe_ets_update_element/4, safe_ets_update_element/5]).
-export([is_even/1, is_odd/1]).
-export([is_valid_shortstr/1]).
-export([maps_any/2,
maps_put_truthy/3,
maps_put_falsy/3
]).
%% Horrible macro to use in guards
-define(IS_BENIGN_EXIT(R),
@ -1594,3 +1598,36 @@ is_even(N) ->
-spec is_odd(integer()) -> boolean().
is_odd(N) ->
(N band 1) =:= 1.
-spec is_valid_shortstr(term()) -> boolean().
is_valid_shortstr(Bin) when byte_size(Bin) < 256 ->
is_utf8_no_null(Bin);
is_valid_shortstr(_) ->
false.
is_utf8_no_null(<<>>) ->
true;
is_utf8_no_null(<<0, _/binary>>) ->
false;
is_utf8_no_null(<<_/utf8, Rem/binary>>) ->
is_utf8_no_null(Rem);
is_utf8_no_null(_) ->
false.
-spec maps_put_truthy(Key, Value, Map) -> Map when
Map :: #{Key => Value}.
maps_put_truthy(_K, undefined, M) ->
M;
maps_put_truthy(_K, false, M) ->
M;
maps_put_truthy(K, V, M) ->
maps:put(K, V, M).
-spec maps_put_falsy(Key, Value, Map) -> Map when
Map :: #{Key => Value}.
maps_put_falsy(K, undefined, M) ->
maps:put(K, undefined, M);
maps_put_falsy(K, false, M) ->
maps:put(K, false, M);
maps_put_falsy(_K, _V, M) ->
M.

1
deps/rabbitmq_amqp1_0/src/rabbit_amqp1_0_reader.erl vendored
View File

@ -420,6 +420,7 @@ handle_1_0_connection_frame(#'v1_0.open'{ max_frame_size = ClientFrameMax,
end,
HostnameVal = case Hostname of
undefined -> undefined;
null -> undefined;
{utf8, Val} -> Val
end,
rabbit_log:debug("AMQP 1.0 connection.open frame: hostname = ~ts, extracted vhost = ~ts, idle_timeout = ~tp" ,

57
deps/rabbitmq_consistent_hash_exchange/src/rabbit_exchange_type_consistent_hash.erl vendored
View File

@ -7,13 +7,12 @@
-module(rabbit_exchange_type_consistent_hash).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include("rabbitmq_consistent_hash_exchange.hrl").
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2,
add_binding/3, remove_bindings/3, assert_args_equivalence/2]).
@ -59,9 +58,9 @@ description() ->
serialise_events() -> false.
route(#exchange {name = Name,
arguments = Args},
#delivery {message = Msg}) ->
route(#exchange{name = Name,
arguments = Args},
Msg, _Options) ->
case rabbit_db_ch_exchange:get(Name) of
undefined ->
[];
@ -69,13 +68,14 @@ route(#exchange {name = Name,
case maps:size(BM) of
0 -> [];
N ->
K = value_to_hash(hash_on(Args), Msg),
K = value_to_hash(hash_on(Args), Msg),
SelectedBucket = jump_consistent_hash(K, N),
case maps:get(SelectedBucket, BM, undefined) of
undefined ->
rabbit_log:warning("Bucket ~tp not found", [SelectedBucket]),
[];
Queue -> [Queue]
Queue ->
[Queue]
end
end
end.
@ -259,26 +259,22 @@ jump_consistent_hash_value(_B0, J0, NumberOfBuckets, SeedState0) ->
J = trunc((B + 1) / R),
jump_consistent_hash_value(B, J, NumberOfBuckets, SeedState).
value_to_hash(undefined, #basic_message { routing_keys = Routes }) ->
Routes;
value_to_hash({header, Header}, #basic_message { content = Content }) ->
Headers = rabbit_basic:extract_headers(Content),
case Headers of
undefined -> undefined;
_ -> rabbit_misc:table_lookup(Headers, Header)
end;
value_to_hash({property, Property}, #basic_message { content = Content }) ->
#content{properties = #'P_basic'{ correlation_id = CorrId,
message_id = MsgId,
timestamp = Timestamp }} =
rabbit_binary_parser:ensure_content_decoded(Content),
value_to_hash(undefined, Msg) ->
mc:get_annotation(routing_keys, Msg);
value_to_hash({header, Header}, Msg0) ->
maps:get(Header, mc:routing_headers(Msg0, [x_headers]));
value_to_hash({property, Property}, Msg) ->
case Property of
<<"correlation_id">> -> CorrId;
<<"message_id">> -> MsgId;
<<"timestamp">> ->
case Timestamp of
undefined -> undefined;
_ -> integer_to_binary(Timestamp)
<<"correlation_id">> ->
unwrap(mc:correlation_id(Msg));
<<"message_id">> ->
unwrap(mc:message_id(Msg));
<<"timestamp">> ->
case mc:timestamp(Msg) of
undefined ->
undefined;
Timestamp ->
integer_to_binary(Timestamp div 1000)
end
end.
@ -298,8 +294,8 @@ hash_args(Args) ->
hash_on(Args) ->
case hash_args(Args) of
{undefined, undefined} -> undefined;
{Header, undefined} -> Header;
{undefined, Property} -> Property
{Header, undefined} -> Header;
{undefined, Property} -> Property
end.
-spec map_has_value(#{bucket() => rabbit_types:binding_destination()},
@ -320,3 +316,8 @@ map_has_value0({_Bucket, SameVal, _I}, SameVal) ->
true;
map_has_value0({_Bucket, _OtherVal, I}, Val) ->
map_has_value0(maps:next(I), Val).
unwrap(undefined) ->
undefined;
unwrap({_, V}) ->
V.

10
deps/rabbitmq_ct_helpers/app.bzl vendored
View File

@ -24,10 +24,7 @@ def all_beam_files(name = "all_beam_files"):
app_name = "rabbitmq_ct_helpers",
dest = "ebin",
erlc_opts = "//:erlc_opts",
deps = [
"//deps/rabbit_common:erlang_app",
"@proper//:erlang_app",
],
deps = ["//deps/rabbit_common:erlang_app", "@proper//:erlang_app"],
)
def all_test_beam_files(name = "all_test_beam_files"):
@ -53,10 +50,7 @@ def all_test_beam_files(name = "all_test_beam_files"):
app_name = "rabbitmq_ct_helpers",
dest = "test",
erlc_opts = "//:test_erlc_opts",
deps = [
"//deps/rabbit_common:erlang_app",
"@proper//:erlang_app",
],
deps = ["//deps/rabbit_common:erlang_app", "@proper//:erlang_app"],
)
def all_srcs(name = "all_srcs"):

33
deps/rabbitmq_event_exchange/src/rabbit_exchange_type_event.erl vendored
View File

@ -51,7 +51,7 @@ exchange() ->
exchange(VHost) ->
_ = ensure_vhost_exists(VHost),
rabbit_misc:r(VHost, exchange, ?EXCH_NAME).
rabbit_misc:r(VHost, exchange, ?EXCH_NAME).
%%----------------------------------------------------------------------------
@ -74,21 +74,22 @@ handle_event(#event{type = Type,
timestamp = TS,
reference = none}, #state{vhost = VHost} = State) ->
_ = case key(Type) of
ignore -> ok;
Key ->
Props2 = [{<<"timestamp_in_ms">>, TS} | Props],
PBasic = #'P_basic'{delivery_mode = 2,
headers = fmt_proplist(Props2),
%% 0-9-1 says the timestamp is a
%% "64 bit POSIX
%% timestamp". That's second
%% resolution, not millisecond.
timestamp = erlang:convert_time_unit(
TS, milli_seconds, seconds)},
Msg = rabbit_basic:message(exchange(VHost), Key, PBasic, <<>>),
rabbit_basic:publish(
rabbit_basic:delivery(false, false, Msg, undefined))
end,
ignore -> ok;
Key ->
Props2 = [{<<"timestamp_in_ms">>, TS} | Props],
PBasic = #'P_basic'{delivery_mode = 2,
headers = fmt_proplist(Props2),
%% 0-9-1 says the timestamp is a
%% "64 bit POSIX
%% timestamp". That's second
%% resolution, not millisecond.
timestamp = erlang:convert_time_unit(
TS, milli_seconds, seconds)},
Content = rabbit_basic:build_content(PBasic, <<>>),
XName = exchange(VHost),
Msg = mc_amqpl:message(XName, Key, Content),
rabbit_queue_type:publish_at_most_once(XName, Msg)
end,
{ok, State};
handle_event(_Event, State) ->
{ok, State}.

2
deps/rabbitmq_federation/src/rabbit_federation_queue_link_sup_sup.erl vendored
View File

@ -87,5 +87,5 @@ init([]) ->
%% we don't clear it out here and can trust it.
id(Q) when ?is_amqqueue(Q) ->
Policy = amqqueue:get_policy(Q),
Q1 = rabbit_amqqueue:immutable(Q),
Q1 = amqqueue:set_immutable(Q),
amqqueue:set_policy(Q1, Policy).

27
deps/rabbitmq_federation/src/rabbit_federation_upstream_exchange.erl vendored
View File

@ -21,7 +21,7 @@
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/3]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2,
add_binding/3, remove_bindings/3, assert_args_equivalence/2]).
@ -38,8 +38,7 @@ description() ->
serialise_events() -> false.
route(X = #exchange{arguments = Args},
D = #delivery{message = #basic_message{content = Content}}) ->
route(X = #exchange{arguments = Args}, Msg, _Opts) ->
%% This arg was introduced in the same release as this exchange type;
%% it must be set
{long, MaxHops} = rabbit_misc:table_lookup(Args, ?MAX_HOPS_ARG),
@ -53,12 +52,28 @@ route(X = #exchange{arguments = Args},
{longstr, Val1} -> Val1;
_ -> unknown
end,
Headers = rabbit_basic:extract_headers(Content),
case rabbit_federation_util:should_forward(Headers, MaxHops, DName, DVhost) of
true -> rabbit_exchange_type_fanout:route(X, D);
case should_forward(Msg, MaxHops, DName, DVhost) of
true -> rabbit_exchange_type_fanout:route(X, Msg);
false -> []
end.
should_forward(Msg, MaxHops, DName, DVhost) ->
case mc:x_header(?ROUTING_HEADER, Msg) of
{list, A} ->
length(A) < MaxHops andalso
not already_seen(DName, DVhost, A);
_ ->
true
end.
already_seen(DName, DVhost, List) ->
lists:any(fun (Map) ->
{utf8, DName} =:= mc_util:amqp_map_get(<<"cluster-name">>, Map, undefined) andalso
{utf8, DVhost} =:= mc_util:amqp_map_get(<<"vhost">>, Map, undefined)
end, List).
validate(#exchange{arguments = Args}) ->
rabbit_federation_util:validate_arg(?MAX_HOPS_ARG, long, Args).

10
deps/rabbitmq_jms_topic_exchange/src/rabbit_jms_topic_exchange.erl vendored
View File

@ -19,7 +19,7 @@
%% Rabbit exchange type functions:
-export([ description/0
, serialise_events/0
, route/2
, route/3
, validate/1
, create/2
, delete/2
@ -89,14 +89,14 @@ description() -> [ {name, <<"jms-selector">>}
serialise_events() -> false.
% Route messages
route( #exchange{name = XName}
, #delivery{message = #basic_message{content = MessageContent, routing_keys = RKs}}
) ->
route(#exchange{name = XName}, Msg, _Opts) ->
RKs = mc:get_annotation(routing_keys, Msg),
Content = mc:protocol_state(mc:convert(mc_amqpl, Msg)),
case get_binding_funs_x(XName) of
not_found ->
[];
BindingFuns ->
match_bindings(XName, RKs, MessageContent, BindingFuns)
match_bindings(XName, RKs, Content, BindingFuns)
end.

5
deps/rabbitmq_management/app.bzl vendored
View File

@ -571,10 +571,7 @@ def test_suite_beam_files(name = "test_suite_beam_files"):
outs = ["test/rabbit_mgmt_stats_SUITE.beam"],
app_name = "rabbitmq_management",
erlc_opts = "//:test_erlc_opts",
deps = [
"//deps/rabbitmq_management_agent:erlang_app",
"@proper//:erlang_app",
],
deps = ["//deps/rabbitmq_management_agent:erlang_app", "@proper//:erlang_app"],
)
erlang_bytecode(
name = "rabbit_mgmt_test_db_SUITE_beam_files",

45
deps/rabbitmq_mqtt/BUILD.bazel vendored
View File

@ -55,7 +55,8 @@ APP_ENV = """[
%% 256 MB is upper limit defined by MQTT spec
{max_packet_size_authenticated, 268435455},
{topic_alias_maximum, 16}
]"""
]
"""
all_beam_files(name = "all_beam_files")
@ -81,6 +82,7 @@ rabbitmq_app(
license_files = [":license_files"],
priv = [":priv"],
deps = [
"//deps/amqp10_common:erlang_app",
"//deps/rabbit:erlang_app",
"//deps/rabbit_common:erlang_app",
"@ra//:erlang_app",
@ -127,6 +129,10 @@ broker_for_integration_suites(
extra_plugins = [
"//deps/rabbitmq_management:erlang_app",
"//deps/rabbitmq_web_mqtt:erlang_app",
"//deps/rabbitmq_consistent_hash_exchange:erlang_app",
"//deps/rabbitmq_amqp1_0:erlang_app",
"//deps/rabbitmq_stomp:erlang_app",
"//deps/rabbitmq_stream:erlang_app",
],
)
@ -253,14 +259,6 @@ rabbitmq_integration_suite(
],
)
rabbitmq_suite(
name = "util_SUITE",
size = "small",
data = [
"test/rabbitmq_mqtt.app",
],
)
rabbitmq_integration_suite(
name = "v5_SUITE",
size = "large",
@ -274,6 +272,19 @@ rabbitmq_integration_suite(
],
)
rabbitmq_integration_suite(
name = "protocol_interop_SUITE",
size = "medium",
additional_beam = [
":test_util_beam",
],
runtime_deps = [
"//deps/amqp10_client:erlang_app",
"//deps/rabbitmq_stomp:erlang_app",
"@emqtt//:erlang_app",
],
)
rabbitmq_suite(
name = "packet_prop_SUITE",
deps = [
@ -289,6 +300,22 @@ rabbitmq_suite(
],
)
rabbitmq_suite(
name = "util_SUITE",
size = "small",
data = [
"test/rabbitmq_mqtt.app",
],
)
rabbitmq_suite(
name = "mc_mqtt_SUITE",
size = "small",
deps = [
"//deps/amqp10_common:erlang_app",
],
)
assert_suites()
alias(

2
deps/rabbitmq_mqtt/Makefile vendored
View File

@ -45,7 +45,7 @@ export BUILD_WITHOUT_QUIC
LOCAL_DEPS = ssl
DEPS = ranch rabbit_common rabbit ra
TEST_DEPS = emqtt ct_helper rabbitmq_ct_helpers rabbitmq_ct_client_helpers rabbitmq_management rabbitmq_web_mqtt amqp_client
TEST_DEPS = emqtt ct_helper rabbitmq_ct_helpers rabbitmq_ct_client_helpers rabbitmq_management rabbitmq_web_mqtt amqp_client rabbitmq_consistent_hash_exchange rabbitmq_amqp1_0 amqp10_client rabbitmq_stomp rabbitmq_stream
dep_ct_helper = git https://github.com/extend/ct_helper.git master
dep_emqtt = git https://github.com/rabbitmq/emqtt.git master

25
deps/rabbitmq_mqtt/app.bzl vendored
View File

@ -19,6 +19,7 @@ def all_beam_files(name = "all_beam_files"):
srcs = [
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.DecommissionMqttNodeCommand.erl",
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.ListMqttConnectionsCommand.erl",
"src/mc_mqtt.erl",
"src/mqtt_machine.erl",
"src/mqtt_machine_v0.erl",
"src/mqtt_node.erl",
@ -45,7 +46,7 @@ def all_beam_files(name = "all_beam_files"):
beam = [":behaviours"],
dest = "ebin",
erlc_opts = "//:erlc_opts",
deps = ["//deps/rabbit:erlang_app", "//deps/rabbit_common:erlang_app", "//deps/rabbitmq_cli:erlang_app", "@ra//:erlang_app", "@ranch//:erlang_app"],
deps = ["//deps/amqp10_common:erlang_app", "//deps/rabbit:erlang_app", "//deps/rabbit_common:erlang_app", "//deps/rabbitmq_cli:erlang_app", "@ra//:erlang_app", "@ranch//:erlang_app"],
)
def all_test_beam_files(name = "all_test_beam_files"):
@ -69,6 +70,7 @@ def all_test_beam_files(name = "all_test_beam_files"):
srcs = [
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.DecommissionMqttNodeCommand.erl",
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.ListMqttConnectionsCommand.erl",
"src/mc_mqtt.erl",
"src/mqtt_machine.erl",
"src/mqtt_machine_v0.erl",
"src/mqtt_node.erl",
@ -96,6 +98,7 @@ def all_test_beam_files(name = "all_test_beam_files"):
dest = "test",
erlc_opts = "//:test_erlc_opts",
deps = [
"//deps/amqp10_common:erlang_app",
"//deps/rabbit:erlang_app",
"//deps/rabbit_common:erlang_app",
"//deps/rabbitmq_cli:erlang_app",
@ -126,6 +129,7 @@ def all_srcs(name = "all_srcs"):
srcs = [
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.DecommissionMqttNodeCommand.erl",
"src/Elixir.RabbitMQ.CLI.Ctl.Commands.ListMqttConnectionsCommand.erl",
"src/mc_mqtt.erl",
"src/mqtt_machine.erl",
"src/mqtt_machine_v0.erl",
"src/mqtt_node.erl",
@ -339,3 +343,22 @@ def test_suite_beam_files(name = "test_suite_beam_files"):
app_name = "rabbitmq_mqtt",
erlc_opts = "//:test_erlc_opts",
)
erlang_bytecode(
name = "mc_mqtt_SUITE_beam_files",
testonly = True,
srcs = ["test/mc_mqtt_SUITE.erl"],
outs = ["test/mc_mqtt_SUITE.beam"],
hdrs = ["include/rabbit_mqtt_packet.hrl"],
app_name = "rabbitmq_mqtt",
erlc_opts = "//:test_erlc_opts",
deps = ["//deps/amqp10_common:erlang_app"],
)
erlang_bytecode(
name = "protocol_interop_SUITE_beam_files",
testonly = True,
srcs = ["test/protocol_interop_SUITE.erl"],
outs = ["test/protocol_interop_SUITE.beam"],
app_name = "rabbitmq_mqtt",
erlc_opts = "//:test_erlc_opts",
deps = ["//deps/amqp_client:erlang_app", "//deps/rabbitmq_stomp:erlang_app"],
)

1
deps/rabbitmq_mqtt/include/rabbit_mqtt.hrl vendored
View File

@ -9,6 +9,7 @@
-define(PG_SCOPE, pg_scope_rabbitmq_mqtt_clientid).
-define(QUEUE_TYPE_QOS_0, rabbit_mqtt_qos0_queue).
-define(PERSISTENT_TERM_MAILBOX_SOFT_LIMIT, mqtt_mailbox_soft_limit).
-define(PERSISTENT_TERM_EXCHANGE, mqtt_exchange).
-define(MQTT_GUIDE_URL, <<"https://rabbitmq.com/mqtt.html">>).
-define(MQTT_PROTO_V3, mqtt310).

4
deps/rabbitmq_mqtt/include/rabbit_mqtt_packet.hrl vendored
View File

@ -245,10 +245,10 @@
%% MQTT application message starting in 3.13
-record(mqtt_msg, {retain :: boolean(),
qos :: qos(),
topic :: topic(),
topic :: option(topic()),
dup :: boolean(),
packet_id :: option(packet_id()) | ?WILL_MSG_QOS_1_CORRELATION,
payload :: binary(),
payload :: iodata(),
%% PUBLISH or Will properties
props :: properties(),
timestamp :: option(integer())

546
deps/rabbitmq_mqtt/src/mc_mqtt.erl vendored Normal file
View File

@ -0,0 +1,546 @@
-module(mc_mqtt).
-behaviour(mc).
-include("rabbit_mqtt_packet.hrl").
-include("rabbit_mqtt.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("amqp10_common/include/amqp10_framing.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit/include/mc.hrl").
-define(CONTENT_TYPE_AMQP, <<"message/vnd.rabbitmq.amqp">>).
-define(DEFAULT_MQTT_EXCHANGE, <<"amq.topic">>).
-export([
init/1,
size/1,
x_header/2,
property/2,
routing_headers/2,
convert_to/2,
convert_from/2,
protocol_state/2,
prepare/2
]).
init(Msg = #mqtt_msg{qos = Qos,
props = Props})
when is_integer(Qos) ->
Anns0 = case Qos > 0 of
true ->
#{durable => true};
false ->
#{}
end,
Anns1 = case Props of
#{'Message-Expiry-Interval' := Seconds} ->
Anns0#{ttl => timer:seconds(Seconds),
timestamp => os:system_time(millisecond)};
_ ->
Anns0
end,
Anns = case Props of
#{'Correlation-Data' := Corr} ->
case mc_util:is_valid_shortstr(Corr) of
true ->
Anns1#{correlation_id => Corr};
false ->
Anns1
end;
_ ->
Anns1
end,
{Msg, Anns}.
convert_from(mc_amqp, Sections) ->
{Header, MsgAnns, AmqpProps, AppProps, PayloadRev,
PayloadFormatIndicator, ContentType} =
lists:foldl(
fun(#'v1_0.header'{} = S, Acc) ->
setelement(1, Acc, S);
(#'v1_0.message_annotations'{content = List}, Acc) ->
setelement(2, Acc, List);
(#'v1_0.properties'{} = S, Acc) ->
setelement(3, Acc, S);
(#'v1_0.application_properties'{content = List}, Acc) ->
setelement(4, Acc, List);
(#'v1_0.footer'{}, Acc) ->
Acc;
(#'v1_0.data'{content = C}, Acc) ->
setelement(5, Acc, [C | element(5, Acc)]);
(#'v1_0.amqp_value'{content = {binary, Bin}}, Acc) ->
setelement(5, Acc, [Bin]);
(#'v1_0.amqp_value'{content = C} = Val, Acc) ->
case amqp_to_utf8_string(C) of
cannot_convert ->
amqp_encode(Val, Acc);
String ->
Acc1 = setelement(5, Acc, [String]),
setelement(6, Acc1, true)
end;
(#'v1_0.amqp_sequence'{} = Seq, Acc) ->
amqp_encode(Seq, Acc)
end, {undefined, [], undefined, [], [], false, undefined}, Sections),
Qos = case Header of
#'v1_0.header'{durable = true} ->
?QOS_1;
_ ->
?QOS_0
end,
Props0 = case PayloadFormatIndicator of
true -> #{'Payload-Format-Indicator' => 1};
false -> #{}
end,
Props1 = case AmqpProps of
#'v1_0.properties'{reply_to = {utf8, Address}} ->
MqttX = persistent_term:get(?PERSISTENT_TERM_EXCHANGE),
case Address of
<<"/topic/", Topic/binary>>
when MqttX =:= ?DEFAULT_MQTT_EXCHANGE ->
add_response_topic(Topic, Props0);
<<"/exchange/", MqttX:(byte_size(MqttX))/binary, "/", RoutingKey/binary>> ->
add_response_topic(RoutingKey, Props0);
_ ->
Props0
end;
_ ->
Props0
end,
Props2 = case AmqpProps of
#'v1_0.properties'{correlation_id = {_Type, _Val} = Corr} ->
Props1#{'Correlation-Data' => correlation_id(Corr)};
_ ->
Props1
end,
Props3 = case ContentType of
undefined ->
case AmqpProps of
#'v1_0.properties'{content_type = {symbol, ContentType1}} ->
Props2#{'Content-Type' => rabbit_data_coercion:to_binary(ContentType1)};
_ ->
Props2
end;
_ ->
Props2#{'Content-Type' => ContentType}
end,
UserProp0 = lists:filtermap(fun({{symbol, <<"x-", _/binary>> = Key}, Val}) ->
filter_map_amqp_to_utf8_string(Key, Val);
(_) ->
false
end, MsgAnns),
%% "The keys of this map are restricted to be of type string" [AMQP 1.0 3.2.5]
UserProp1 = lists:filtermap(fun({{utf8, Key}, Val}) ->
filter_map_amqp_to_utf8_string(Key, Val)
end, AppProps),
Props = case UserProp0 ++ UserProp1 of
[] -> Props3;
UserProp -> Props3#{'User-Property' => UserProp}
end,
Payload = lists:flatten(lists:reverse(PayloadRev)),
#mqtt_msg{retain = false,
qos = Qos,
dup = false,
props = Props,
payload = Payload};
convert_from(mc_amqpl, #content{properties = PBasic,
payload_fragments_rev = Payload}) ->
#'P_basic'{expiration = Expiration,
delivery_mode = DelMode,
headers = H0,
correlation_id = CorrId,
content_type = ContentType} = PBasic,
Qos = case DelMode of
2 -> ?QOS_1;
_ -> ?QOS_0
end,
P0 = case is_binary(ContentType) of
true -> #{'Content-Type' => ContentType};
false -> #{}
end,
H1 = case H0 of
undefined -> [];
_ -> H0
end,
{P1, H3} = case lists:keytake(<<"x-reply-to-topic">>, 1, H1) of
{value, {_, longstr, Topic}, H2} ->
{P0#{'Response-Topic' => rabbit_mqtt_util:amqp_to_mqtt(Topic)}, H2};
false ->
{P0, H1}
end,
{P2, H} = case is_binary(CorrId) of
true ->
{P1#{'Correlation-Data' => CorrId}, H3};
false ->
case lists:keytake(<<"x-correlation-id">>, 1, H3) of
{value, {_, longstr, Corr}, H4} ->
{P1#{'Correlation-Data' => Corr}, H4};
false ->
{P1, H3}
end
end,
P3 = case amqpl_header_to_user_property(H) of
[] ->
P2;
UserProperty ->
P2#{'User-Property' => UserProperty}
end,
P = case is_binary(Expiration) of
true ->
Millis = binary_to_integer(Expiration),
P3#{'Message-Expiry-Interval' => Millis div 1000};
false ->
P3
end,
#mqtt_msg{retain = false,
qos = Qos,
dup = false,
payload = lists:reverse(Payload),
props = P};
convert_from(_SourceProto, _) ->
not_implemented.
convert_to(?MODULE, Msg) ->
Msg;
convert_to(mc_amqp, #mqtt_msg{qos = Qos,
props = Props,
payload = Payload}) ->
Body = case Props of
#{'Payload-Format-Indicator' := 1}
when is_binary(Payload) ->
#'v1_0.amqp_value'{content = {utf8, Payload}};
_ ->
#'v1_0.data'{content = Payload}
end,
S0 = [Body],
%% x- prefixed MQTT User Properties go into Message Annotations.
%% All other MQTT User Properties go into Application Properties.
%% MQTT User Property allows duplicate keys, while AMQP maps don't.
%% Order is semantically important in both MQTT User Property and AMQP maps.
%% Therefore, we must dedup the keys and must maintain order.
{MsgAnns, AppProps} =
case Props of
#{'User-Property' := UserProps} ->
{MsgAnnsRev, AppPropsRev, _} =
lists:foldl(fun({Name, _}, Acc = {_, _, M})
when is_map_key(Name, M) ->
Acc;
({<<"x-", _/binary>> = Name, Val}, Acc = {MAnns, AProps, M}) ->
case mc_util:utf8_string_is_ascii(Name) of
true ->
{[{{symbol, Name}, {utf8, Val}} | MAnns], AProps, M#{Name => true}};
false ->
Acc
end;
({Name, Val}, {MAnns, AProps, M}) ->
{MAnns, [{{utf8, Name}, {utf8, Val}} | AProps], M#{Name => true}}
end, {[], [], #{}}, UserProps),
{lists:reverse(MsgAnnsRev), lists:reverse(AppPropsRev)};
_ ->
{[], []}
end,
S1 = case AppProps of
[] -> S0;
_ -> [#'v1_0.application_properties'{content = AppProps} | S0]
end,
ContentType = case Props of
#{'Content-Type' := ContType} ->
case mc_util:utf8_string_is_ascii(ContType) of
true ->
{symbol, ContType};
false ->
undefined
end;
_ ->
undefined
end,
CorrId = case Props of
#{'Correlation-Data' := Corr} ->
{binary, Corr};
_ ->
undefined
end,
ReplyTo = case Props of
#{'Response-Topic' := MqttTopic} ->
Topic = rabbit_mqtt_util:mqtt_to_amqp(MqttTopic),
Address = case persistent_term:get(?PERSISTENT_TERM_EXCHANGE) of
?DEFAULT_MQTT_EXCHANGE ->
<<"/topic/", Topic/binary>>;
Exchange ->
<<"/exchange/", Exchange/binary, "/", Topic/binary>>
end,
{utf8, Address};
_ ->
undefined
end,
S2 = case {ContentType, CorrId, ReplyTo} of
{undefined, undefined, undefined} ->
S1;
_ ->
[#'v1_0.properties'{content_type = ContentType,
correlation_id = CorrId,
reply_to = ReplyTo} | S1]
end,
S3 = case MsgAnns of
[] -> S2;
_ -> [#'v1_0.message_annotations'{content = MsgAnns} | S2]
end,
S = [#'v1_0.header'{durable = Qos > 0} | S3],
mc_amqp:convert_from(mc_amqp, S);
convert_to(mc_amqpl, #mqtt_msg{qos = Qos,
props = Props,
payload = Payload}) ->
DelMode = case Qos of
?QOS_0 -> 1;
?QOS_1 -> 2
end,
ContentType = case Props of
#{'Content-Type' := ContType} -> ContType;
_ -> undefined
end,
Hs0 = case Props of
#{'User-Property' := UserProperty} ->
lists:filtermap(
fun({Name, Value})
when byte_size(Name) =< ?AMQP_LEGACY_FIELD_NAME_MAX_LEN ->
{true, {Name, longstr, Value}};
(_) ->
false
end, UserProperty);
_ ->
[]
end,
Hs1 = case Props of
#{'Response-Topic' := Topic} ->
[{<<"x-reply-to-topic">>, longstr, rabbit_mqtt_util:mqtt_to_amqp(Topic)} | Hs0];
_ ->
Hs0
end,
{CorrId, Hs2} = case Props of
#{'Correlation-Data' := Corr} ->
case mc_util:is_valid_shortstr(Corr) of
true ->
{Corr, Hs1};
false ->
{undefined, [{<<"x-correlation-id">>, longstr, Corr} | Hs1]}
end;
_ ->
{undefined, Hs1}
end,
Expiration = case Props of
#{'Message-Expiry-Interval' := Seconds} ->
integer_to_binary(timer:seconds(Seconds));
_ ->
undefined
end,
%% "Duplicate fields are illegal." [4.2.5.5 Field Tables]
%% RabbitMQ sorts field tables by keys.
Hs = lists:usort(fun({Key1, _Type1, _Val1},
{Key2, _Type2, _Val2}) ->
Key1 =< Key2
end, Hs2),
BP = #'P_basic'{content_type = ContentType,
headers = if Hs =:= [] -> undefined;
Hs =/= [] -> Hs
end,
delivery_mode = DelMode,
correlation_id = CorrId,
expiration = Expiration},
PFR = case is_binary(Payload) of
true -> [Payload];
false -> lists:reverse(Payload)
end,
#content{class_id = 60,
properties = BP,
properties_bin = none,
payload_fragments_rev = PFR};
convert_to(_TargetProto, #mqtt_msg{}) ->
not_implemented.
size(#mqtt_msg{payload = Payload,
topic = Topic,
props = Props}) ->
PropsSize = maps:fold(fun size_prop/3, 0, Props),
MetadataSize = PropsSize + byte_size(Topic),
{MetadataSize, iolist_size(Payload)}.
size_prop(K, Val, Sum)
when K =:= 'Content-Type' orelse
K =:= 'Response-Topic' orelse
K =:= 'Correlation-Data' ->
byte_size(Val) + Sum;
size_prop('User-Property', L, Sum) ->
lists:foldl(fun({Name, Val}, Acc) ->
byte_size(Name) + byte_size(Val) + Acc
end, Sum, L);
size_prop(_, _, Sum) ->
Sum.
x_header(Key, #mqtt_msg{props = #{'User-Property' := UserProp}}) ->
case proplists:get_value(Key, UserProp) of
undefined -> undefined;
Val -> {utf8, Val}
end;
x_header(_Key, #mqtt_msg{}) ->
undefined.
property(correlation_id, #mqtt_msg{props = #{'Correlation-Data' := Corr}}) ->
{binary, Corr};
property(_Key, #mqtt_msg{}) ->
undefined.
routing_headers(#mqtt_msg{props = #{'User-Property' := UserProperty}}, Opts) ->
IncludeX = lists:member(x_headers, Opts),
lists:foldl(fun({<<"x-", _/binary>> = K, V}, M) ->
case IncludeX of
true -> M#{K => V};
false -> M
end;
({K, V}, M) ->
M#{K => V}
end, #{}, UserProperty);
routing_headers(#mqtt_msg{}, _Opts) ->
#{}.
protocol_state(Msg = #mqtt_msg{props = Props0,
topic = Topic}, Anns) ->
%% Remove any PUBLISH or Will Properties that are not forwarded unaltered.
Props1 = maps:remove('Message-Expiry-Interval', Props0),
{WillDelay, Props2} = case maps:take('Will-Delay-Interval', Props1) of
error -> {0, Props1};
ValMap -> ValMap
end,
Props = case maps:get(ttl, Anns, undefined) of
undefined ->
Props2;
Ttl ->
case maps:get(timestamp, Anns) of
undefined ->
Props2;
Timestamp ->
SourceProtocolIsMqtt = Topic =/= undefined,
%% Only if source protocol is MQTT we know that timestamp was set by the server.
case SourceProtocolIsMqtt of
false ->
Props2;
true ->
%% "The PUBLISH packet sent to a Client by the Server MUST contain a
%% Message Expiry Interval set to the received value minus the time that
%% the Application Message has been waiting in the Server" [MQTT-3.3.2-6]
WaitingMillis0 = os:system_time(millisecond) - Timestamp,
%% For a delayed Will Message, the waiting time starts
%% when the Will Message was published.
WaitingMillis = WaitingMillis0 - WillDelay * 1000,
MEIMillis = max(0, Ttl - WaitingMillis),
Props2#{'Message-Expiry-Interval' => MEIMillis div 1000}
end
end
end,
[RoutingKey | _] = maps:get(routing_keys, Anns),
Msg#mqtt_msg{topic = rabbit_mqtt_util:amqp_to_mqtt(RoutingKey),
props = Props}.
prepare(_For, #mqtt_msg{} = Msg) ->
Msg.
correlation_id({uuid, UUID}) ->
mc_util:uuid_to_string(UUID);
correlation_id({_T, Corr}) ->
rabbit_data_coercion:to_binary(Corr).
%% Translates AMQP 0.9.1 headers to MQTT 5.0 User Properties if
%% the value is convertible to a UTF-8 String.
-spec amqpl_header_to_user_property(rabbit_framing:amqp_table()) ->
user_property().
amqpl_header_to_user_property(Table) ->
lists:filtermap(fun amqpl_field_to_utf8_string_pair/1, Table).
amqpl_field_to_utf8_string_pair({K, longstr, V}) ->
case mc_util:is_utf8_no_null(V) of
true -> {true, {K, V}};
false -> false
end;
amqpl_field_to_utf8_string_pair({K, T, V})
when T =:= byte;
T =:= unsignedbyte;
T =:= short;
T =:= unsignedshort;
T =:= signedint;
T =:= unsignedint;
T =:= long;
T =:= timestamp ->
{true, {K, integer_to_binary(V)}};
amqpl_field_to_utf8_string_pair({K, T, V})
when T =:= float;
T =:= double ->
{true, {K, float_to_binary(V)}};
amqpl_field_to_utf8_string_pair({K, void, _V}) ->
{true, {K, <<>>}};
amqpl_field_to_utf8_string_pair({K, bool, V}) ->
{true, {K, atom_to_binary(V)}};
amqpl_field_to_utf8_string_pair({_K, T, _V})
when T =:= array;
T =:= table;
%% Raw binary data is not UTF-8 encoded.
T =:= binary ->
false.
filter_map_amqp_to_utf8_string(Key, TypeVal) ->
case amqp_to_utf8_string(TypeVal) of
cannot_convert ->
false;
String ->
{true, {Key, String}}
end.
amqp_to_utf8_string({utf8, Val})
when is_binary(Val) ->
Val;
amqp_to_utf8_string(Val)
when Val =:= null;
Val =:= undefined ->
<<>>;
amqp_to_utf8_string({T, Val})
when T =:= byte;
T =:= ubyte;
T =:= short;
T =:= ushort;
T =:= int;
T =:= uint;
T =:= long;
T =:= ulong ->
integer_to_binary(Val);
amqp_to_utf8_string({timestamp, Millis}) ->
%% MQTT 5.0 defines all intervals (e.g. Keep Alive, Message Expiry Interval,
%% Session Expiry Interval) in seconds. Therefore let's convert to seconds.
integer_to_binary(Millis div 1000);
amqp_to_utf8_string({T, Val})
when T =:= double;
T =:= float ->
float_to_binary(Val);
amqp_to_utf8_string(Val)
when Val =:= true;
Val =:= {boolean, true} ->
<<"true">>;
amqp_to_utf8_string(Val)
when Val =:= false;
Val =:= {boolean, false} ->
<<"false">>;
amqp_to_utf8_string({T, _Val})
when T =:= map;
T =:= list;
T =:= array;
%% Raw binary data is not UTF-8 encoded.
T =:= binary ->
cannot_convert.
amqp_encode(Data, Acc0) ->
Bin = amqp10_framing:encode_bin(Data),
Acc = setelement(5, Acc0, [Bin | element(5, Acc0)]),
setelement(7, Acc, ?CONTENT_TYPE_AMQP).
add_response_topic(AmqpTopic, PublishProperties) ->
MqttTopic = rabbit_mqtt_util:amqp_to_mqtt(AmqpTopic),
PublishProperties#{'Response-Topic' => MqttTopic}.

4
deps/rabbitmq_mqtt/src/rabbit_mqtt.erl vendored
View File

@ -110,6 +110,10 @@ init_global_counters(ProtoVer) ->
persist_static_configuration() ->
rabbit_mqtt_util:init_sparkplug(),
{ok, Exchange} = application:get_env(?APP_NAME, exchange),
?assert(is_binary(Exchange)),
ok = persistent_term:put(?PERSISTENT_TERM_EXCHANGE, Exchange),
{ok, MailboxSoftLimit} = application:get_env(?APP_NAME, mailbox_soft_limit),
?assert(is_integer(MailboxSoftLimit)),
ok = persistent_term:put(?PERSISTENT_TERM_MAILBOX_SOFT_LIMIT, MailboxSoftLimit),

11
deps/rabbitmq_mqtt/src/rabbit_mqtt_ff.erl vendored
View File

@ -39,10 +39,13 @@
%% available on all nodes to support Will Delay Interval.
-rabbit_feature_flag(
{mqtt_v5,
#{desc => "Support MQTT 5.0",
stability => stable,
%% MQTT 5.0 feature Will Delay Interval depends on client ID tracking in pg local.
depends_on => [delete_ra_cluster_mqtt_node]
#{desc => "Support MQTT 5.0",
stability => stable,
depends_on => [
%% MQTT 5.0 feature Will Delay Interval depends on client ID tracking in pg local.
delete_ra_cluster_mqtt_node,
message_containers
]
}}).
-spec track_client_id_in_ra() -> boolean().

501
deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl vendored
View File

@ -17,7 +17,9 @@
update_trace/2, send_disconnect/2]).
-ifdef(TEST).
-export([get_vhost_username/1, get_vhost/3, get_vhost_from_user_mapping/2]).
-export([get_vhost_username/1,
get_vhost/3,
get_vhost_from_user_mapping/2]).
-endif.
-export_type([state/0,
@ -26,10 +28,10 @@
-import(rabbit_mqtt_util, [mqtt_to_amqp/1,
amqp_to_mqtt/1,
ip_address_to_binary/1]).
-import(rabbit_misc, [maps_put_truthy/3]).
-include_lib("kernel/include/logger.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("rabbit/include/amqqueue.hrl").
-include("rabbit_mqtt.hrl").
-include("rabbit_mqtt_packet.hrl").
@ -37,7 +39,7 @@
-define(MAX_PERMISSION_CACHE_SIZE, 12).
-define(CONSUMER_TAG, <<"mqtt">>).
-define(QUEUE_TTL_KEY, <<"x-expires">>).
-define(AMQP_091_SHORT_STR_MAX_SIZE, 255).
-define(DEFAULT_EXCHANGE_NAME, <<>>).
-type send_fun() :: fun((iodata()) -> ok).
-type session_expiry_interval() :: non_neg_integer() | infinity.
@ -198,6 +200,7 @@ process_connect(
{ok, WillMsg} ?= make_will_msg(Packet),
{TraceState, ConnName} = init_trace(VHost, ConnName0),
ok = rabbit_mqtt_keepalive:start(KeepaliveSecs, Socket),
Exchange = rabbit_misc:r(VHost, exchange, persistent_term:get(?PERSISTENT_TERM_EXCHANGE)),
S = #state{
cfg = #cfg{socket = Socket,
proto_ver = proto_integer_to_atom(ProtoVer),
@ -213,7 +216,7 @@ process_connect(
peer_ip_addr = PeerIp,
peer_port = PeerPort,
send_fun = SendFun,
exchange = rabbit_misc:r(VHost, exchange, rabbit_mqtt_util:env(exchange)),
exchange = Exchange,
retainer_pid = rabbit_mqtt_retainer_sup:start_child_for_vhost(VHost),
vhost = VHost,
client_id = ClientId,
@ -1238,10 +1241,6 @@ creds(User, Pass, SSLLoginName) ->
auth_attempt_failed(PeerIp, Username) ->
rabbit_core_metrics:auth_attempt_failed(PeerIp, Username, mqtt).
delivery_mode(?QOS_0) -> 1;
delivery_mode(?QOS_1) -> 2;
delivery_mode(?QOS_2) -> 2.
maybe_downgrade_qos(?QOS_0) -> ?QOS_0;
maybe_downgrade_qos(?QOS_1) -> ?QOS_1;
maybe_downgrade_qos(?QOS_2) -> ?QOS_1.
@ -1543,62 +1542,25 @@ binding_action(ExchangeName, TopicFilter, QName, BindingArgs,
BindingFun(Binding, Username).
publish_to_queues(
#mqtt_msg{retain = Retain,
qos = Qos,
topic = Topic,
packet_id = PacketId,
payload = Payload,
props = Props},
#state{cfg = #cfg{exchange = ExchangeName,
#mqtt_msg{topic = Topic,
packet_id = PacketId} = MqttMsg,
#state{cfg = #cfg{exchange = ExchangeName = #resource{name = ExchangeNameBin},
delivery_flow = Flow,
conn_name = ConnName,
trace_state = TraceState},
auth_state = #auth_state{user = #user{username = Username}}
} = State) ->
RoutingKey = mqtt_to_amqp(Topic),
Confirm = Qos > ?QOS_0,
{Expiration, Timestamp} = case Props of
#{'Message-Expiry-Interval' := ExpirySecs} ->
{integer_to_binary(timer:seconds(ExpirySecs)),
os:system_time(second)};
_ ->
{undefined, undefined}
end,
PBasic0 = mqtt_props_to_amqp_props(Props, Qos, Retain),
PBasic = PBasic0#'P_basic'{
delivery_mode = delivery_mode(Qos),
expiration = Expiration,
timestamp = Timestamp},
{ClassId, _MethodId} = rabbit_framing_amqp_0_9_1:method_id('basic.publish'),
Content0 = #content{
class_id = ClassId,
properties = PBasic,
properties_bin = none,
protocol = none,
payload_fragments_rev = [Payload]
},
Content = rabbit_message_interceptor:intercept(Content0),
BasicMessage = #basic_message{
exchange_name = ExchangeName,
routing_keys = [RoutingKey],
content = Content,
id = <<>>, %% GUID set in rabbit_classic_queue
is_persistent = Confirm
},
Delivery = #delivery{
mandatory = false,
confirm = Confirm,
sender = self(),
message = BasicMessage,
msg_seq_no = PacketId,
flow = Flow
},
auth_state = #auth_state{user = #user{username = Username}}} = State) ->
Anns = #{exchange => ExchangeNameBin,
routing_keys => [mqtt_to_amqp(Topic)]},
Msg0 = mc:init(mc_mqtt, MqttMsg, Anns),
Msg = rabbit_message_interceptor:intercept(Msg0),
case rabbit_exchange:lookup(ExchangeName) of
{ok, Exchange} ->
QNames0 = rabbit_exchange:route(Exchange, Delivery, #{return_binding_keys => true}),
QNames0 = rabbit_exchange:route(Exchange, Msg, #{return_binding_keys => true}),
QNames = drop_local(QNames0, State),
rabbit_trace:tap_in(BasicMessage, QNames, ConnName, Username, TraceState),
deliver_to_queues(Delivery, QNames, State);
rabbit_trace:tap_in(Msg, QNames, ConnName, Username, TraceState),
Opts = maps_put_truthy(flow, Flow, maps_put_truthy(correlation, PacketId, #{})),
deliver_to_queues(Msg, Opts, QNames, State);
{error, not_found} ->
?LOG_ERROR("~s not found", [rabbit_misc:rs(ExchangeName)]),
{error, exchange_not_found, State}
@ -1637,54 +1599,55 @@ drop_local(QNames, #state{subscriptions = Subs,
drop_local(QNames, _) ->
QNames.
deliver_to_queues(Delivery,
deliver_to_queues(Message0,
Options,
RoutedToQNames,
State0 = #state{queue_states = QStates0,
cfg = #cfg{proto_ver = ProtoVer}}) ->
Qs0 = rabbit_amqqueue:lookup_many(RoutedToQNames),
Qs = rabbit_amqqueue:prepend_extra_bcc(Qs0),
case rabbit_queue_type:deliver(Qs, Delivery, QStates0) of
Message = compat(Message0, State0),
case rabbit_queue_type:deliver(Qs, Message, Options, QStates0) of
{ok, QStates, Actions} ->
rabbit_global_counters:messages_routed(ProtoVer, length(Qs)),
State = process_routing_confirm(Delivery, Qs,
State = process_routing_confirm(Options, Qs,
State0#state{queue_states = QStates}),
%% Actions must be processed after registering confirms as actions may
%% contain rejections of publishes.
{ok, handle_queue_actions(Actions, State)};
{error, Reason} ->
Corr = maps:get(correlation, Options, undefined),
?LOG_ERROR("Failed to deliver message with packet_id=~p to queues: ~p",
[Delivery#delivery.msg_seq_no, Reason]),
[Corr, Reason]),
{error, Reason, State0}
end.
process_routing_confirm(#delivery{confirm = false},
[], State = #state{cfg = #cfg{proto_ver = ProtoVer}}) ->
process_routing_confirm(Options,
[], State = #state{cfg = #cfg{proto_ver = ProtoVer}})
when not is_map_key(correlation, Options) ->
rabbit_global_counters:messages_unroutable_dropped(ProtoVer, 1),
State;
process_routing_confirm(#delivery{confirm = true,
msg_seq_no = ?WILL_MSG_QOS_1_CORRELATION},
process_routing_confirm(#{correlation := ?WILL_MSG_QOS_1_CORRELATION},
[], State = #state{cfg = #cfg{proto_ver = ProtoVer}}) ->
%% unroutable will message with QoS 1
rabbit_global_counters:messages_unroutable_dropped(ProtoVer, 1),
State;
process_routing_confirm(#delivery{confirm = true,
msg_seq_no = PktId},
process_routing_confirm(#{correlation := PktId},
[], State = #state{cfg = #cfg{proto_ver = ProtoVer}}) ->
rabbit_global_counters:messages_unroutable_returned(ProtoVer, 1),
send_puback(PktId, ?RC_NO_MATCHING_SUBSCRIBERS, State),
State;
process_routing_confirm(#delivery{confirm = false}, _, State) ->
State;
process_routing_confirm(#delivery{confirm = true,
msg_seq_no = ?WILL_MSG_QOS_1_CORRELATION}, [_|_], State) ->
process_routing_confirm(#{correlation := ?WILL_MSG_QOS_1_CORRELATION},
[_|_], State) ->
%% routable will message with QoS 1
State;
process_routing_confirm(#delivery{confirm = true,
msg_seq_no = PktId},
process_routing_confirm(#{correlation := PktId},
Qs, State = #state{unacked_client_pubs = U0}) ->
QNames = rabbit_amqqueue:queue_names(Qs),
U = rabbit_mqtt_confirms:insert(PktId, QNames, U0),
State#state{unacked_client_pubs = U}.
State#state{unacked_client_pubs = U};
process_routing_confirm(#{}, _, State) ->
State.
-spec send_puback(packet_id() | list(packet_id()), reason_code(), state()) -> ok.
send_puback(PktIds0, ReasonCode, State)
@ -1754,77 +1717,71 @@ terminate(SendWill, Infos, State) ->
-spec maybe_send_will(boolean(), state()) -> ok.
maybe_send_will(
true, #state{cfg = #cfg{will_msg = #mqtt_msg{
props = Props = #{'Will-Delay-Interval' := Delay},
retain = Retain,
qos = Qos,
topic = Topic,
payload = Payload},
session_expiry_interval_secs = SessionExpiry,
exchange = #resource{name = XName},
client_id = ClientId,
vhost = Vhost}} = State)
true,
#state{cfg = #cfg{will_msg = #mqtt_msg{
props = Props = #{'Will-Delay-Interval' := Delay},
topic = Topic} = MqttMsg,
session_expiry_interval_secs = SessionExpiry,
exchange = #resource{name = XName},
client_id = ClientId,
vhost = Vhost
}} = State)
when is_integer(Delay) andalso Delay > 0 andalso SessionExpiry > 0 ->
QArgs0 = queue_ttl_args(SessionExpiry),
QArgs = QArgs0 ++ [{<<"x-dead-letter-exchange">>, longstr, XName},
{<<"x-dead-letter-routing-key">>, longstr, mqtt_to_amqp(Topic)}],
{<<"x-dead-letter-routing-key">>, longstr, mqtt_to_amqp(Topic)}],
T = erlang:monotonic_time(millisecond),
case create_queue(will, none, QArgs, rabbit_queue_type:default(), State) of
{ok, Q} ->
#resource{name = QNameBin} = amqqueue:get_name(Q),
DefaultX = #resource{virtual_host = Vhost,
kind = exchange,
name = <<"">>},
%% "The Server delays publishing the Client’s Will Message until the Will Delay
%% Interval has passed or the Session ends, whichever happens first." [v5 3.1.3.2.2]
MsgTTLSecs = min(Delay, SessionExpiry),
MsgTTL0 = timer:seconds(MsgTTLSecs),
MsgTTL = if SessionExpiry =:= infinity ->
MsgTTL0;
is_integer(SessionExpiry) ->
%% Queue creation could have taken several milliseconds.
Elapsed = erlang:monotonic_time(millisecond) - T,
SessionExpiryFromNow = timer:seconds(SessionExpiry) - Elapsed,
%% Ensure the Will Message is dead lettered BEFORE the queue expires.
%% 5 ms should be enough time to send out the Will Message.
%% The important bit is that, in the queue implementation, the
%% message expiry timer fires before the queue expiry timer.
%% From MQTT client perspective, the granularity of defined intervals
%% is seconds. So sending the Will Message a few milliseconds earlier
%% doesn't matter from the client's point of view.
%% However, we shouldn't send the Will Message too early because
%% "The Client can arrange for the Will Message to notify that Session
%% Expiry has occurred" [v5 3.1.2.5]
%% So, we don't want to send out a false positive session expiry
%% notification in case the client reconnects shortly after.
Interval0 = SessionExpiryFromNow - 5,
Interval = max(0, Interval0),
min(MsgTTL0, Interval)
end,
{Headers, Timestamp} = case Props of
#{'Message-Expiry-Interval' := ExpirySecs} ->
E = integer_to_binary(timer:seconds(ExpirySecs)),
{[{<<"x-dead-letter-expiration">>, longstr, E},
{<<"x-mqtt-will-delay-interval">>, long, Delay}],
os:system_time(second)};
_ ->
{[], undefined}
end,
PBasic0 = mqtt_props_to_amqp_props(Props, Qos, Retain),
PBasic = PBasic0#'P_basic'{
%% Persist message regardless of Will QoS since there is no noticable
%% performance benefit if that single message is transient. This ensures that
%% delayed Will Messages are not lost after a broker restart.
headers = Headers ++ PBasic0#'P_basic'.headers,
delivery_mode = 2,
expiration = integer_to_binary(MsgTTL),
timestamp = Timestamp},
Ttl0 = timer:seconds(min(Delay, SessionExpiry)),
Ttl = if SessionExpiry =:= infinity ->
Ttl0;
is_integer(SessionExpiry) ->
%% Queue creation could have taken several milliseconds.
Elapsed = erlang:monotonic_time(millisecond) - T,
SessionExpiryFromNow = timer:seconds(SessionExpiry) - Elapsed,
%% Ensure the Will Message is dead lettered BEFORE the queue expires.
%% 5 ms should be enough time to send out the Will Message.
%% The important bit is that, in the queue implementation, the
%% message expiry timer fires before the queue expiry timer.
%% From MQTT client perspective, the granularity of defined intervals
%% is seconds. So sending the Will Message a few milliseconds earlier
%% doesn't matter from the client's point of view.
%% However, we shouldn't send the Will Message too early because
%% "The Client can arrange for the Will Message to notify that Session
%% Expiry has occurred" [v5 3.1.2.5]
%% So, we don't want to send out a false positive session expiry
%% notification in case the client reconnects shortly after.
Interval0 = SessionExpiryFromNow - 5,
Interval = max(0, Interval0),
min(Ttl0, Interval)
end,
DefaultX = #resource{virtual_host = Vhost,
kind = exchange,
name = ?DEFAULT_EXCHANGE_NAME},
#resource{name = QNameBin} = amqqueue:get_name(Q),
Anns0 = #{exchange => ?DEFAULT_EXCHANGE_NAME,
routing_keys => [QNameBin],
ttl => Ttl,
%% Persist message regardless of Will QoS since there is no noticable
%% performance benefit if that single message is transient. This ensures that
%% delayed Will Messages are not lost after a broker restart.
durable => true},
Anns = case Props of
#{'Message-Expiry-Interval' := MEI} ->
Anns0#{dead_letter_ttl => timer:seconds(MEI)};
_ ->
Anns0
end,
Msg = mc:init(mc_mqtt, MqttMsg, Anns),
case check_publish_permitted(DefaultX, Topic, State) of
ok ->
ok = rabbit_basic:publish(DefaultX, QNameBin, PBasic, Payload),
?LOG_DEBUG("scheduled delayed Will Message to topic ~s for MQTT "
"client ID ~s to be sent in ~b ms",
[Topic, ClientId, MsgTTL]);
ok = rabbit_queue_type:publish_at_most_once(DefaultX, Msg),
?LOG_DEBUG("scheduled delayed Will Message to topic ~s "
"for MQTT client ID ~s to be sent in ~b ms",
[Topic, ClientId, Ttl]);
{error, access_refused = Reason} ->
log_delayed_will_failure(Topic, ClientId, Reason)
end;
@ -2064,27 +2021,16 @@ deliver_to_client(Msgs, Ack, State) ->
deliver_one_to_client(Msg, Ack, S)
end, State, Msgs).
deliver_one_to_client(Msg0 = {QNameOrType, QPid, QMsgId, _Redelivered,
BasicMsg = #basic_message{content = Content0}},
deliver_one_to_client({QNameOrType, QPid, QMsgId, _Redelivered, Mc} = Delivery,
AckRequired, State0) ->
Content = #content{properties = #'P_basic'{headers = Headers}} =
rabbit_binary_parser:ensure_content_decoded(Content0),
Msg = setelement(5, Msg0, BasicMsg#basic_message{content = Content}),
PublisherQoS = case rabbit_mqtt_util:table_lookup(Headers, <<"x-mqtt-publish-qos">>) of
{byte, QoS0} ->
QoS0;
undefined ->
%% non-MQTT publishes are assumed to be QoS 1 regardless of delivery_mode
?QOS_1
end,
SubscriberQoS = case AckRequired of
true ->
?QOS_1;
false ->
?QOS_0
true -> ?QOS_1;
false -> ?QOS_0
end,
QoS = effective_qos(PublisherQoS, SubscriberQoS),
{SettleOp, State1} = maybe_publish_to_client(Msg, QoS, State0),
McMqtt = mc:convert(mc_mqtt, Mc),
MqttMsg = #mqtt_msg{qos = PublisherQos} = mc:protocol_state(McMqtt),
QoS = effective_qos(PublisherQos, SubscriberQoS),
{SettleOp, State1} = maybe_publish_to_client(MqttMsg, Delivery, QoS, State0),
State = maybe_auto_settle(AckRequired, SettleOp, QoS, QNameOrType, QMsgId, State1),
ok = maybe_notify_sent(QNameOrType, QPid, State),
State.
@ -2096,37 +2042,34 @@ effective_qos(PublisherQoS, SubscriberQoS) ->
%% [MQTT-3.8.4-8]."
erlang:min(PublisherQoS, SubscriberQoS).
maybe_publish_to_client({_, _, _, _Redelivered = true, _}, ?QOS_0, State) ->
maybe_publish_to_client(_, {_, _, _, _Redelivered = true, _}, ?QOS_0, State) ->
%% Do not redeliver to MQTT subscriber who gets message at most once.
{complete, State};
maybe_publish_to_client(
Msg = {QNameOrType, _QPid, QMsgId, Redelivered,
#basic_message{
routing_keys = [RoutingKey | _CcRoutes],
content = #content{payload_fragments_rev = FragmentsRev,
properties = PBasic = #'P_basic'{headers = Headers}}}},
QoS, State0 = #state{cfg = #cfg{proto_ver = ProtoVer}}) ->
Props0 = amqp_props_to_mqtt_props(PBasic, ProtoVer),
MatchedTopicFilters = matched_topic_filters_v5(Headers, State0),
#mqtt_msg{retain = Retain,
topic = Topic0,
payload = Payload,
props = Props0},
{QNameOrType, _QPid, QMsgId, Redelivered, Mc} = Delivery,
QoS, State0) ->
MatchedTopicFilters = matched_topic_filters_v5(Mc, State0),
Props1 = maybe_add_subscription_ids(MatchedTopicFilters, Props0, State0),
Topic0 = amqp_to_mqtt(RoutingKey),
{Topic, Props, State1} = process_topic_alias_outbound(Topic0, Props1, State0),
{PacketId, State} = msg_id_to_packet_id(QMsgId, QoS, State1),
Packet =
#mqtt_packet{
fixed = #mqtt_packet_fixed{
type = ?PUBLISH,
qos = QoS,
dup = Redelivered,
retain = retain(Headers, MatchedTopicFilters, State)},
variable = #mqtt_packet_publish{
packet_id = PacketId,
topic_name = Topic,
props = Props},
payload = lists:reverse(FragmentsRev)},
Packet = #mqtt_packet{
fixed = #mqtt_packet_fixed{
type = ?PUBLISH,
qos = QoS,
dup = Redelivered,
retain = retain(Retain, MatchedTopicFilters, State)},
variable = #mqtt_packet_publish{
packet_id = PacketId,
topic_name = Topic,
props = Props},
payload = Payload},
SettleOp = case send(Packet, State) of
ok ->
trace_tap_out(Msg, State),
trace_tap_out(Delivery, State),
message_delivered(QNameOrType, Redelivered, QoS, State),
complete;
{error, packet_too_large} ->
@ -2134,173 +2077,10 @@ maybe_publish_to_client(
end,
{SettleOp, State}.
%% Convert MQTT v5 PUBLISH or Will properties to AMQP 0.9.1 properties.
-spec mqtt_props_to_amqp_props(properties(), qos(), boolean()) ->
rabbit_framing:amqp_property_record().
mqtt_props_to_amqp_props(Props, Qos, Retain) ->
P0 = #'P_basic'{headers = [{<<"x-mqtt-publish-qos">>, byte, Qos},
{<<"x-mqtt-retain">>, bool, Retain}]},
P1 = case Props of
#{'Content-Type' := T}
when byte_size(T) =< ?AMQP_091_SHORT_STR_MAX_SIZE ->
P0#'P_basic'{content_type = T};
_ ->
%% TODO if Content-Type is > 255 bytes (which seems unlikely), should we:
%% 1. silently ignore (as done right now), or
%% 2. close the network connection (i.e. prohibit), or
%% 3. add a custom AMQP 0.9.1 header?
P0
end,
P2 = case Props of
#{'Payload-Format-Indicator' := 1} ->
%% UTF-8 is not a MIME content encoding and therefore cannot be set as #'P_basic'.content_encoding.
%% Rather, it would match to #'P_basic'.content_type = <<"text/plain;charset=UTF-8">>.
%% However, we cannot set #'P_basic'.content_type because we don't know the subtype (wehther it's
%% 'plain') and that field is already set by MQTT 5.0 property Content-Type.
%% Therefore, we add a custom header.
P1#'P_basic'{headers = [{<<"x-mqtt-payload-format-indicator">>, bool, true} |
P1#'P_basic'.headers]};
_ ->
P1
end,
P3 = case Props of
#{'Response-Topic' := Topic} ->
%% Unfortunately, we cannot set #'P_basic'.reply_to because they are expected to hold
%% the binary queue name in AMQP 0.9.1: "One of the standard message properties is
%% Reply-To, which is designed specifically for carrying the name of reply queues."
%% Therefore, we add a custom header.
P2#'P_basic'{headers = [{<<"x-opt-reply-to-topic">>, longstr,
%% Convert such that an AMQP consumer can respond.
mqtt_to_amqp(Topic)} |
P2#'P_basic'.headers]};
_ ->
P2
end,
P4 = case Props of
#{'Correlation-Data' := Corr}
when byte_size(Corr) =< ?AMQP_091_SHORT_STR_MAX_SIZE ->
P3#'P_basic'{correlation_id = Corr};
#{'Correlation-Data' := Corr}
when byte_size(Corr) > ?AMQP_091_SHORT_STR_MAX_SIZE ->
P3#'P_basic'{headers = [{<<"x-correlation-id">>, longstr, Corr}
| P3#'P_basic'.headers]};
_ ->
P3
end,
P = case Props of
#{'User-Property' := PropList} ->
%% "The same name is allowed to appear more than once."
%% "The Server MUST maintain the order of User Properties
%% when forwarding the Application Message" [v5 3.3.2.3.7]
%% However, in AMQP 0.9.1 Field Tables: "Duplicate fields are illegal."
%% To allow duplicate names and to maintain order, we create a 2 element map:
%% The 1st element contains all names in order.
%% The 2nd element contains all values in order.
{Names, Values} = lists:unzip(PropList),
Header = {<<"x-mqtt-user-property">>,
table,
rabbit_misc:to_amqp_table(#{<<"names">> => Names,
<<"values">> => Values})},
P4#'P_basic'{headers = [Header | P4#'P_basic'.headers]};
_ ->
P4
end,
P.
%% Convert AMQP 0.9.1 properties to MQTT v5 PUBLISH properties.
-spec amqp_props_to_mqtt_props(rabbit_framing:amqp_property_record(), protocol_version_atom()) ->
properties().
%% Do not unnecessarily convert properties.
amqp_props_to_mqtt_props(_, ?MQTT_PROTO_V3) ->
#{};
amqp_props_to_mqtt_props(_, ?MQTT_PROTO_V4) ->
#{};
amqp_props_to_mqtt_props(
#'P_basic'{headers = Headers,
expiration = Expiration,
timestamp = TimestampSeconds,
content_type = ContentType,
correlation_id = CorrelationId
}, ?MQTT_PROTO_V5) ->
SourceProtocolIsMqtt = case rabbit_mqtt_util:table_lookup(Headers, <<"x-mqtt-publish-qos">>) of
{byte, _Qos} -> true;
undefined -> false
end,
P0 = if is_binary(Expiration) andalso
is_integer(TimestampSeconds) andalso
%% Only if source protocol is MQTT we know that timestamp was set by the server
SourceProtocolIsMqtt ->
ExpirationMs = binary_to_integer(Expiration),
ExpirationSeconds = ExpirationMs div 1000,
%% "The PUBLISH packet sent to a Client by the Server MUST contain a Message
%% Expiry Interval set to the received value minus the time that the
%% Application Message has been waiting in the Server" [MQTT-3.3.2-6]
WaitingSeconds0 = os:system_time(second) - TimestampSeconds,
%% For a delayed Will Message, the waiting time starts when the Will Message was published.
WaitingSeconds = case rabbit_basic:header(<<"x-mqtt-will-delay-interval">>, Headers) of
{<<"x-mqtt-will-delay-interval">>, long, Delay} ->
WaitingSeconds0 - Delay;
_ ->
WaitingSeconds0
end,
Expiry = max(0, ExpirationSeconds - WaitingSeconds),
#{'Message-Expiry-Interval' => Expiry};
true ->
#{}
end,
P1 = case ContentType of
T when is_binary(T) ->
P0#{'Content-Type' => T};
_ ->
P0
end,
P2 = case rabbit_basic:header(<<"x-mqtt-payload-format-indicator">>, Headers) of
{<<"x-mqtt-payload-format-indicator">>, bool, true} ->
P1#{'Payload-Format-Indicator' => 1};
_ ->
P1
end,
P3 = case rabbit_basic:header(<<"x-opt-reply-to-topic">>, Headers) of
{<<"x-opt-reply-to-topic">>, longstr, Topic}
when is_binary(Topic) ->
P2#{'Response-Topic' => amqp_to_mqtt(Topic)};
_ ->
P2
end,
P4 = case CorrelationId of
C when is_binary(C) ->
P3#{'Correlation-Data' => C};
C when is_list(C) ->
P3#{'Correlation-Data' => list_to_binary(C)};
_ ->
case rabbit_basic:header(<<"x-correlation-id">>, Headers) of
{<<"x-correlation-id">>, longstr, C}
when is_binary(C) ->
P3#{'Correlation-Data' => C};
_ ->
P3
end
end,
P = case rabbit_basic:header(<<"x-mqtt-user-property">>, Headers) of
{<<"x-mqtt-user-property">>, table, Table} ->
case rabbit_misc:amqp_table(Table) of
#{<<"names">> := Names,
<<"values">> := Values} ->
P4#{'User-Property' => lists:zip(Names, Values)};
_ ->
P4
end;
_ ->
P4
end,
P.
matched_topic_filters_v5(Headers, #state{cfg = #cfg{proto_ver = ?MQTT_PROTO_V5}}) ->
case rabbit_mqtt_util:table_lookup(Headers, <<"x-binding-keys">>) of
{array, BindingKeys} ->
[amqp_to_mqtt(BKey) || {longstr, BKey} <- BindingKeys];
undefined ->
[]
matched_topic_filters_v5(Msg, #state{cfg = #cfg{proto_ver = ?MQTT_PROTO_V5}}) ->
case mc:get_annotation(binding_keys, Msg) of
undefined -> [];
BKeys -> lists:map(fun rabbit_mqtt_util:amqp_to_mqtt/1, BKeys)
end;
matched_topic_filters_v5(_, _) ->
[].
@ -2323,17 +2103,14 @@ maybe_add_subscription_ids(TopicFilters, Props, #state{subscriptions = Subs}) ->
%% If 1, Application Messages forwarded using this subscription keep the RETAIN
%% flag they were published with. If 0, Application Messages forwarded using
%% this subscription have the RETAIN flag set to 0." [v5 3.8.3.1]
retain(Headers, TopicFilters, #state{subscriptions = Subs}) ->
case rabbit_mqtt_util:table_lookup(Headers, <<"x-mqtt-retain">>) of
{bool, true} ->
lists:any(fun(T) -> case maps:get(T, Subs, undefined) of
#mqtt_subscription_opts{retain_as_published = Rap} -> Rap;
undefined -> false
end
end, TopicFilters);
_ ->
false
end.
retain(false, _, _) ->
false;
retain(true, TopicFilters, #state{subscriptions = Subs}) ->
lists:any(fun(T) -> case maps:get(T, Subs, undefined) of
#mqtt_subscription_opts{retain_as_published = Rap} -> Rap;
undefined -> false
end
end, TopicFilters).
msg_id_to_packet_id(_, ?QOS_0, State) ->
%% "A PUBLISH packet MUST NOT contain a Packet Identifier if its QoS value is set to 0 [MQTT-2.2.1-2]."
@ -2753,3 +2530,17 @@ format_status(
ra_register_state => RaRegisterState,
queues_soft_limit_exceeded => QSLE,
qos0_messages_dropped => Qos0MsgsDropped}.
-spec compat(mc:state(), state()) -> mc:state().
compat(McMqtt, #state{cfg = #cfg{exchange = XName}}) ->
case rabbit_feature_flags:is_enabled(message_containers) of
true ->
McMqtt;
false = FFState ->
#mqtt_msg{qos = Qos} = mc:protocol_state(McMqtt),
[RoutingKey] = mc:get_annotation(routing_keys, McMqtt),
McLegacy = mc:convert(mc_amqpl, McMqtt),
Content = mc:protocol_state(McLegacy),
BasicMsg = mc_amqpl:message(XName, RoutingKey, Content, #{}, FFState),
rabbit_basic:add_header(<<"x-mqtt-publish-qos">>, byte, Qos, BasicMsg)
end.

63
deps/rabbitmq_mqtt/src/rabbit_mqtt_qos0_queue.erl vendored
View File

@ -27,7 +27,7 @@
is_stateful/0,
declare/2,
delete/4,
deliver/2,
deliver/3,
is_enabled/0,
is_compatible/3,
is_recoverable/1,
@ -107,38 +107,39 @@ delete(Q, _IfUnused, _IfEmpty, ActingUser) ->
ok = rabbit_amqqueue:internal_delete(Q, ActingUser),
{ok, 0}.
-spec deliver([{amqqueue:amqqueue(), stateless}], Delivery :: term()) ->
-spec deliver([{amqqueue:amqqueue(), stateless}],
Msg :: mc:state(),
rabbit_queue_type:delivery_options()) ->
{[], rabbit_queue_type:actions()}.
deliver(Qs, #delivery{message = BasicMessage,
confirm = Confirm,
msg_seq_no = SeqNo}) ->
Msg = {queue_event, ?MODULE,
{?MODULE, _QPid = none, _QMsgId = none, _Redelivered = false, BasicMessage}},
deliver(Qs, Msg, Options) ->
Evt = {queue_event, ?MODULE,
{?MODULE, _QPid = none, _QMsgId = none, _Redelivered = false, Msg}},
{Pids, Actions} =
case Confirm of
false ->
Pids0 = lists:map(fun({Q, stateless}) -> amqqueue:get_pid(Q) end, Qs),
{Pids0, []};
true ->
%% We confirm the message directly here in the queue client.
%% Alternatively, we could have the target MQTT connection process confirm the message.
%% However, given that this message might be lost anyway between target MQTT connection
%% process and MQTT subscriber, and we know that the MQTT subscriber wants to receive
%% this message at most once, we confirm here directly.
%% Benefits:
%% 1. We do not block sending the confirmation back to the publishing client just because a single
%% (at-most-once) target queue out of potentially many (e.g. million) queues might be unavailable.
%% 2. Memory usage in this (publishing) process is kept lower because the target queue name can be
%% directly removed from rabbit_mqtt_confirms and rabbit_confirms.
%% 3. Reduced network traffic across RabbitMQ nodes.
%% 4. Lower latency of sending publisher confirmation back to the publishing client.
SeqNos = [SeqNo],
lists:mapfoldl(fun({Q, stateless}, Actions) ->
{amqqueue:get_pid(Q),
[{settled, amqqueue:get_name(Q), SeqNos} | Actions]}
end, [], Qs)
end,
delegate:invoke_no_result(Pids, {gen_server, cast, [Msg]}),
case maps:get(correlation, Options, undefined) of
undefined ->
Pids0 = lists:map(fun({Q, stateless}) -> amqqueue:get_pid(Q) end, Qs),
{Pids0, []};
Corr ->
%% We confirm the message directly here in the queue client.
%% Alternatively, we could have the target MQTT connection process confirm the message.
%% However, given that this message might be lost anyway between target MQTT connection
%% process and MQTT subscriber, and we know that the MQTT subscriber wants to receive
%% this message at most once, we confirm here directly.
%% Benefits:
%% 1. We do not block sending the confirmation back to the publishing client just because a single
%% (at-most-once) target queue out of potentially many (e.g. million) queues might be unavailable.
%% 2. Memory usage in this (publishing) process is kept lower because the target queue name can be
%% directly removed from rabbit_mqtt_confirms and rabbit_confirms.
%% 3. Reduced network traffic across RabbitMQ nodes.
%% 4. Lower latency of sending publisher confirmation back to the publishing client.
Corrs = [Corr],
lists:mapfoldl(fun({Q, stateless}, Actions) ->
{amqqueue:get_pid(Q),
[{settled, amqqueue:get_name(Q), Corrs}
| Actions]}
end, [], Qs)
end,
delegate:invoke_no_result(Pids, {gen_server, cast, [Evt]}),
{[], Actions}.
-spec is_enabled() -> boolean().

1
deps/rabbitmq_mqtt/src/rabbit_mqtt_util.erl vendored
View File

@ -143,7 +143,6 @@ env(Key) ->
coerce_env_value(default_pass, Val) -> rabbit_data_coercion:to_binary(Val);
coerce_env_value(default_user, Val) -> rabbit_data_coercion:to_binary(Val);
coerce_env_value(exchange, Val) -> rabbit_data_coercion:to_binary(Val);
coerce_env_value(vhost, Val) -> rabbit_data_coercion:to_binary(Val);
coerce_env_value(_, Val) -> Val.

2
deps/rabbitmq_mqtt/test/cluster_SUITE.erl vendored
View File

@ -79,8 +79,6 @@ init_per_testcase(Testcase, Config) ->
rabbit_ct_helpers:log_environment(),
Config1 = rabbit_ct_helpers:set_config(Config, [
{rmq_nodename_suffix, Testcase},
{rmq_extra_tcp_ports, [tcp_port_mqtt_extra,
tcp_port_mqtt_tls_extra]},
{rmq_nodes_clustered, true}
]),
Config2 = rabbit_ct_helpers:run_setup_steps(Config1,

33
deps/rabbitmq_mqtt/test/ff_SUITE.erl vendored
View File

@ -137,12 +137,33 @@ rabbit_mqtt_qos0_queue(Config) ->
mqtt_v5(Config) ->
FeatureFlag = ?FUNCTION_NAME,
{Client1, Connect} = util:start_client(?FUNCTION_NAME, Config, 0, [{proto_ver, v5}]),
unlink(Client1),
?assertEqual({error, {unsupported_protocol_version, #{}}}, Connect(Client1)),
%% MQTT 5.0 is not yet supported.
{C1, Connect} = util:start_client(?FUNCTION_NAME, Config, 0, [{proto_ver, v5}]),
unlink(C1),
?assertEqual({error, {unsupported_protocol_version, #{}}}, Connect(C1)),
%% Send message from node 0.
%% Message is stored in old AMQP 0.9.1 format on node 1.
Topic = <<"my/topic">>,
C2 = connect(<<"sub-v4">>, Config, 1, util:non_clean_sess_opts()),
{ok, _, [1]} = emqtt:subscribe(C2, Topic, qos1),
ok = emqtt:disconnect(C2),
C3 = connect(<<"pub-v4">>, Config),
{ok, _} = emqtt:publish(C3, Topic, <<"msg">>, qos1),
ok = emqtt:disconnect(C3),
DependantFF = message_containers,
?assertNot(rabbit_ct_broker_helpers:is_feature_flag_enabled(Config, DependantFF)),
?assertEqual(ok, rabbit_ct_broker_helpers:enable_feature_flag(Config, FeatureFlag)),
?assert(rabbit_ct_broker_helpers:is_feature_flag_enabled(Config, DependantFF)),
{Client2, Connect} = util:start_client(?FUNCTION_NAME, Config, 0, [{proto_ver, v5}]),
?assertMatch({ok, _}, Connect(Client2)),
ok = emqtt:disconnect(Client2).
%% Translate from old AMQP 0.9.1 message format consuming from node 2.
C4 = connect(<<"sub-v4">>, Config, 2, [{clean_start, false}]),
ok = expect_publishes(C4, Topic, [<<"msg">>]),
ok = emqtt:disconnect(C4),
%% MQTT 5.0 is now supported.
{C5, Connect} = util:start_client(?FUNCTION_NAME, Config, 0, [{proto_ver, v5}]),
?assertMatch({ok, _}, Connect(C5)),
ok = emqtt:disconnect(C5).

311
deps/rabbitmq_mqtt/test/mc_mqtt_SUITE.erl vendored Normal file
View File

@ -0,0 +1,311 @@
-module(mc_mqtt_SUITE).
-compile([export_all,
nowarn_export_all]).
-include_lib("rabbitmq_mqtt/include/rabbit_mqtt_packet.hrl").
-include_lib("amqp10_common/include/amqp10_framing.hrl").
-include_lib("eunit/include/eunit.hrl").
all() ->
[
{group, lossless},
{group, lossy}
].
groups() ->
[
{lossless, [shuffle],
[roundtrip_amqp,
roundtrip_amqp_payload_format_indicator,
roundtrip_amqp_response_topic,
roundtrip_amqpl,
roundtrip_amqpl_correlation,
amqp_to_mqtt_amqp_value_section_binary,
amqp_to_mqtt_amqp_value_section_list,
amqp_to_mqtt_amqp_value_section_null,
amqp_to_mqtt_amqp_value_section_int,
amqp_to_mqtt_amqp_value_section_boolean
]
},
{lossy, [shuffle],
[roundtrip_amqp_user_property,
roundtrip_amqpl_user_property,
roundtrip_amqp_content_type,
amqp_to_mqtt_reply_to,
amqp_to_mqtt_footer
]
}
].
roundtrip_amqp(_Config) ->
Msg = #mqtt_msg{
qos = 1,
topic = <<"/my/topic">>,
payload = <<"my payload">>,
props = #{'Content-Type' => <<"text-plain">>,
'Correlation-Data' => <<0, 255, 0>>,
'User-Property' => [{<<"x-key-2">>, <<"val-2">>},
{<<"x-key-3">>, <<"val-3">>},
{<<"x-key-1">>, <<"val-1">>},
{<<"key-2">>, <<"val-2">>},
{<<"key-3">>, <<"val-3">>},
{<<"key-1">>, <<"val-1">>}]}},
Anns = #{routing_keys => [rabbit_mqtt_util:mqtt_to_amqp(Msg#mqtt_msg.topic)]},
Mc0 = mc:init(mc_mqtt, Msg, Anns),
BytesTopic = 9,
BytesContentType = 10,
BytesCorrelationData = 3,
BytesUserProperty = 66,
MetaDataSize = BytesTopic + BytesContentType + BytesCorrelationData + BytesUserProperty,
PayloadSize = 10,
ExpectedSize = {MetaDataSize, PayloadSize},
?assertEqual(ExpectedSize, mc:size(Mc0)),
?assertEqual(Msg, mc_mqtt:convert_to(mc_mqtt, Msg)),
?assertEqual(not_implemented, mc_mqtt:convert_to(mc_stomp, Msg)),
?assertEqual(Mc0, mc:convert(mc_mqtt, Mc0)),
%% roundtrip
Mc1 = mc:convert(mc_amqp, Mc0),
Mc = mc:convert(mc_mqtt, Mc1),
?assertEqual({binary, <<0, 255, 0>>}, mc:correlation_id(Mc)),
?assertEqual(undefined, mc:timestamp(Mc)),
?assert(mc:is_persistent(Mc)),
?assertEqual(#{<<"key-1">> => <<"val-1">>,
<<"key-2">> => <<"val-2">>,
<<"key-3">> => <<"val-3">>},
mc:routing_headers(Mc, [])),
?assertEqual(#{<<"key-1">> => <<"val-1">>,
<<"key-2">> => <<"val-2">>,
<<"key-3">> => <<"val-3">>,
<<"x-key-1">> => <<"val-1">>,
<<"x-key-2">> => <<"val-2">>,
<<"x-key-3">> => <<"val-3">>},
mc:routing_headers(Mc, [x_headers])),
?assertEqual({utf8, <<"val-3">>}, mc:x_header(<<"x-key-3">>, Mc)),
?assertEqual(undefined, mc:x_header(<<"x-key-4">>, Mc)),
#mqtt_msg{qos = Qos,
topic = Topic,
payload = Payload,
props = Props
} = mc:protocol_state(Mc),
?assertEqual(1, Qos),
?assertEqual(<<"/my/topic">>, Topic),
?assertEqual(<<"my payload">>, iolist_to_binary(Payload)),
?assertMatch(#{'Content-Type' := <<"text-plain">>,
'Correlation-Data' := <<0, 255, 0>>
}, Props),
ExpectedUserProperty = maps:get('User-Property', Msg#mqtt_msg.props),
%% We expect order to be maintained.
?assertMatch(#{'User-Property' := ExpectedUserProperty}, Props).
%% The indicator that the Payload is UTF-8 encoded should not be lost when translating
%% from MQTT 5.0 to AMQP 1.0 or vice versa.
roundtrip_amqp_payload_format_indicator(_Config) ->
Msg0 = mqtt_msg(),
Msg = Msg0#mqtt_msg{payload = <<"🐇"/utf8>>,
props = #{'Payload-Format-Indicator' => 1}},
#mqtt_msg{payload = Payload,
props = Props} = roundtrip(mc_amqp, Msg),
?assertEqual(unicode:characters_to_binary("🐇"),
iolist_to_binary(Payload)),
?assertMatch(#{'Payload-Format-Indicator' := 1}, Props).
roundtrip_amqp_response_topic(_Config) ->
Topic = <<"/rabbit/🐇"/utf8>>,
Msg0 = mqtt_msg(),
Key = mqtt_exchange,
MqttExchanges = [<<"amq.topic">>,
<<"some-other-topic-exchange">>],
[begin
ok = persistent_term:put(Key, X),
Msg = Msg0#mqtt_msg{props = #{'Response-Topic' => Topic}},
?assertMatch(#mqtt_msg{props = #{'Response-Topic' := Topic}},
roundtrip(mc_amqp, Msg)),
true = persistent_term:erase(Key)
end || X <- MqttExchanges].
roundtrip_amqpl(_Config) ->
Msg = #mqtt_msg{
qos = 1,
topic = <<"/my/topic">>,
payload = <<"my payload">>,
props = #{'Content-Type' => <<"text-plain">>,
'Correlation-Data' => <<"ABC-123">>,
'Response-Topic' => <<"my/response/topic">>,
'User-Property' => [{<<"x-key-2">>, <<"val-2">>},
{<<"x-key-3">>, <<"val-3">>},
{<<"x-key-1">>, <<"val-1">>},
{<<"key-2">>, <<"val-2">>},
{<<"key-3">>, <<"val-3">>},
{<<"key-1">>, <<"val-1">>}]}},
Anns = #{routing_keys => [rabbit_mqtt_util:mqtt_to_amqp(Msg#mqtt_msg.topic)]},
Mc0 = mc:init(mc_mqtt, Msg, Anns),
Mc1 = mc:convert(mc_amqpl, Mc0),
Mc = mc:convert(mc_mqtt, Mc1),
?assertEqual({binary, <<"ABC-123">>}, mc:correlation_id(Mc)),
?assert(mc:is_persistent(Mc)),
#mqtt_msg{qos = Qos,
topic = Topic,
payload = Payload,
props = Props
} = mc:protocol_state(Mc),
?assertEqual(1, Qos),
?assertEqual(<<"/my/topic">>, Topic),
?assertEqual(<<"my payload">>, iolist_to_binary(Payload)),
?assertMatch(#{'Content-Type' := <<"text-plain">>,
'Correlation-Data' := <<"ABC-123">>,
'Response-Topic' := <<"my/response/topic">>},
Props),
UserProperty = maps:get('User-Property', Msg#mqtt_msg.props),
%% We expect order to not be maintained since AMQP 0.9.1 sorts by key.
ExpectedUserProperty = lists:keysort(1, UserProperty),
?assertMatch(#{'User-Property' := ExpectedUserProperty}, Props).
%% Non-UTF-8 Correlation Data should also be converted (via AMQP 0.9.1 header x-correlation-id).
roundtrip_amqpl_correlation(_Config) ->
Msg0 = mqtt_msg(),
Correlation = binary:copy(<<0>>, 1024),
Msg = Msg0#mqtt_msg{
props = #{'Correlation-Data' => Correlation}},
?assertMatch(#mqtt_msg{props = #{'Correlation-Data' := Correlation}},
roundtrip(mc_amqpl, Msg)).
%% Binaries should be sent unmodified.
amqp_to_mqtt_amqp_value_section_binary(_Config) ->
Val = amqp_value({binary, <<0, 255>>}),
#mqtt_msg{props = Props,
payload = Payload} = amqp_to_mqtt([Val]),
?assertEqual(<<0, 255>>, iolist_to_binary(Payload)),
?assertEqual(#{}, Props).
%% Lists cannot be converted to a text representation.
%% They should be encoded using the AMQP 1.0 type system.
amqp_to_mqtt_amqp_value_section_list(_Config) ->
Val = amqp_value({list, [{uint, 3}]}),
#mqtt_msg{props = Props,
payload = Payload} = amqp_to_mqtt([Val]),
?assertEqual(#{'Content-Type' => <<"message/vnd.rabbitmq.amqp">>}, Props),
?assert(iolist_size(Payload) > 0).
amqp_to_mqtt_amqp_value_section_null(_Config) ->
Val = amqp_value(null),
#mqtt_msg{props = Props,
payload = Payload} = amqp_to_mqtt([Val]),
?assertEqual(#{'Payload-Format-Indicator' => 1}, Props),
?assertEqual(0, iolist_size(Payload)).
amqp_to_mqtt_amqp_value_section_int(_Config) ->
Val = amqp_value({int, -3}),
#mqtt_msg{props = Props,
payload = Payload} = amqp_to_mqtt([Val]),
?assertEqual(#{'Payload-Format-Indicator' => 1}, Props),
?assertEqual(<<"-3">>, iolist_to_binary(Payload)).
amqp_to_mqtt_amqp_value_section_boolean(_Config) ->
Val1 = amqp_value(true),
#mqtt_msg{props = Props1,
payload = Payload1} = amqp_to_mqtt([Val1]),
?assertEqual(#{'Payload-Format-Indicator' => 1}, Props1),
?assertEqual(<<"true">>, iolist_to_binary(Payload1)),
Val2 = amqp_value({boolean, false}),
#mqtt_msg{props = Props2,
payload = Payload2} = amqp_to_mqtt([Val2]),
?assertEqual(#{'Payload-Format-Indicator' => 1}, Props2),
?assertEqual(<<"false">>, iolist_to_binary(Payload2)).
%% When converting from MQTT 5.0 to AMQP 1.0, we expect to lose some User Property.
roundtrip_amqp_user_property(_Config) ->
Msg0 = mqtt_msg(),
Msg = Msg0#mqtt_msg{props = #{'User-Property' =>
[{<<"x-dup"/utf8>>, <<"val-2">>},
{<<"x-dup"/utf8>>, <<"val-3">>},
{<<"dup">>, <<"val-4">>},
{<<"dup">>, <<"val-5">>},
{<<"x-key-no-ascii🐇"/utf8>>, <<"val-1">>}
]}},
#mqtt_msg{props = Props} = roundtrip(mc_amqp, Msg),
Lost = [%% AMQP 1.0 maps disallow duplicate keys.
{<<"x-dup">>, <<"val-3">>},
{<<"dup">>, <<"val-5">>},
%% AMQP 1.0 annotations require keys to be symbols, i.e. ASCII
{<<"x-key-no-ascii🐇"/utf8>>, <<"val-1">>}
],
ExpectedUserProperty = maps:get('User-Property', Msg#mqtt_msg.props) -- Lost,
?assertMatch(#{'User-Property' := ExpectedUserProperty}, Props).
%% When converting from MQTT 5.0 to AMQP 0.9.1, we expect to lose any duplicates and
%% any User Property whose name is longer than 128 characters.
roundtrip_amqpl_user_property(_Config) ->
Msg0 = mqtt_msg(),
Msg = Msg0#mqtt_msg{
props = #{'User-Property' => [{<<"key-2">>, <<"val-2">>},
{<<"key-1">>, <<"val-1">>},
{binary:copy(<<"k">>, 129), <<"val-2">>},
{<<"key-1">>, <<"val-1">>}
]}},
?assertMatch(#mqtt_msg{props = #{'User-Property' := [{<<"key-1">>, <<"val-1">>},
{<<"key-2">>, <<"val-2">>}]}},
roundtrip(mc_amqpl, Msg)).
%% In MQTT 5.0 the Content Type is a UTF-8 encoded string.
%% In AMQP 1.0 the Content Type is a symbol.
%% We expect to lose the Content Type when converting from MQTT 5.0 to AMQP 1.0 if
%% the Content Type is not valid ASCII.
roundtrip_amqp_content_type(_Config) ->
Msg0 = mqtt_msg(),
Msg = Msg0#mqtt_msg{props = #{'Content-Type' => <<"no-ascii🐇"/utf8>>}},
#mqtt_msg{props = Props} = roundtrip(mc_amqp, Msg),
?assertNot(maps:is_key('Content-Type', Props)).
amqp_to_mqtt_reply_to(_Config) ->
Val = amqp_value({utf8, <<"hey">>}),
Key = mqtt_exchange,
ok = persistent_term:put(Key, <<"mqtt-topic-exchange">>),
AmqpProps1 = #'v1_0.properties'{reply_to = {utf8, <<"/exchange/mqtt-topic-exchange/my.routing.key">>}},
#mqtt_msg{props = Props1} = amqp_to_mqtt([AmqpProps1, Val]),
?assertEqual({ok, <<"my/routing/key">>},
maps:find('Response-Topic', Props1)),
AmqpProps2 = #'v1_0.properties'{reply_to = {utf8, <<"/exchange/NON-mqtt-topic-exchange/my.routing.key">>}},
#mqtt_msg{props = Props2} = amqp_to_mqtt([AmqpProps2, Val]),
?assertEqual(error,
maps:find('Response-Topic', Props2)),
true = persistent_term:erase(Key).
amqp_to_mqtt_footer(_Config) ->
Val = amqp_value({utf8, <<"hey">>}),
Footer = #'v1_0.footer'{content = [{symbol, <<"key">>}, {utf8, <<"value">>}]},
%% We can translate, but lose the footer.
#mqtt_msg{payload = Payload} = amqp_to_mqtt([Val, Footer]),
?assertEqual(<<"hey">>, iolist_to_binary(Payload)).
mqtt_msg() ->
#mqtt_msg{qos = 0,
topic = <<"my/topic">>,
payload = <<>>}.
roundtrip(Mod, MqttMsg) ->
Anns = #{routing_keys => [rabbit_mqtt_util:mqtt_to_amqp(MqttMsg#mqtt_msg.topic)]},
Mc0 = mc:init(mc_mqtt, MqttMsg, Anns),
Mc1 = mc:convert(Mod, Mc0),
Mc = mc:convert(mc_mqtt, Mc1),
mc:protocol_state(Mc).
amqp_to_mqtt(Sections) ->
Anns = #{routing_keys => [<<"apple">>]},
Mc0 = mc:init(mc_amqp, Sections, Anns),
Mc = mc:convert(mc_mqtt, Mc0),
mc:protocol_state(Mc).
amqp_value(Content) ->
#'v1_0.amqp_value'{content = Content}.

18
deps/rabbitmq_mqtt/test/mqtt_machine_SUITE.erl vendored
View File

@ -28,24 +28,6 @@ groups() ->
{tests, [], all_tests()}
].
init_per_suite(Config) ->
Config.
end_per_suite(_Config) ->
ok.
init_per_group(_Group, Config) ->
Config.
end_per_group(_Group, _Config) ->
ok.
init_per_testcase(_TestCase, Config) ->
Config.
end_per_testcase(_TestCase, _Config) ->
ok.
%%%===================================================================
%%% Test cases
%%%===================================================================

394
deps/rabbitmq_mqtt/test/protocol_interop_SUITE.erl vendored Normal file
View File

@ -0,0 +1,394 @@
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%
%% Copyright (c) 2007-2023 VMware, Inc. or its affiliates. All rights reserved.
%%
%% This test suite covers protocol interoperability publishing via MQTT 5.0,
%% receiving via AMQP 0.9.1, AMQP 1.0, STOMP 1.2, and Stream, and vice versa.
-module(protocol_interop_SUITE).
-compile([export_all,
nowarn_export_all]).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-include_lib("rabbitmq_stomp/include/rabbit_stomp_frame.hrl").
-import(util,
[connect/2]).
-import(rabbit_ct_broker_helpers,
[rpc/4]).
-import(rabbit_ct_helpers,
[eventually/3]).
all() ->
[{group, tests}].
groups() ->
[{tests, [shuffle],
[
amqpl,
amqp,
stomp,
stream
]
}].
%% -------------------------------------------------------------------
%% Testsuite setup/teardown.
%% -------------------------------------------------------------------
init_per_suite(Config) ->
{ok, _} = application:ensure_all_started(amqp10_client),
rabbit_ct_helpers:log_environment(),
rabbit_ct_helpers:run_setup_steps(Config).
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config).
init_per_group(_Group, Config0) ->
Config1 = rabbit_ct_helpers:set_config(
Config0,
{mqtt_version, v5}),
Config = rabbit_ct_helpers:run_steps(
Config1,
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps()),
ok = rabbit_ct_broker_helpers:enable_feature_flag(Config, mqtt_v5),
Plugins = [rabbitmq_amqp1_0,
rabbitmq_stomp,
rabbitmq_stream],
[ok = rabbit_ct_broker_helpers:enable_plugin(Config, 0, Plugin) || Plugin <- Plugins],
Config.
end_per_group(_Group, Config) ->
rabbit_ct_helpers:run_steps(
Config,
rabbit_ct_client_helpers:teardown_steps() ++
rabbit_ct_broker_helpers:teardown_steps()).
init_per_testcase(Testcase, Config) ->
rabbit_ct_helpers:testcase_started(Config, Testcase).
end_per_testcase(Testcase, Config) ->
%% Wait for exclusive or auto-delete queues being deleted.
timer:sleep(800),
rabbit_ct_broker_helpers:rpc(Config, ?MODULE, delete_queues, []),
rabbit_ct_helpers:testcase_finished(Config, Testcase).
%% -------------------------------------------------------------------
%% Testsuite cases
%% -------------------------------------------------------------------
amqpl(Config) ->
Q = ClientId = atom_to_binary(?FUNCTION_NAME),
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = Q}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = Q,
exchange = <<"amq.topic">>,
routing_key = <<"my.topic">>}),
%% MQTT 5.0 to AMQP 0.9.1
C = connect(ClientId, Config),
MqttResponseTopic = <<"response/topic">>,
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 999}, [{MqttResponseTopic, [{qos, 1}]}]),
Correlation = <<"some correlation ID">>,
RequestPayload = <<"my request">>,
UserProperty = [{<<"rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>},
{<<"key">>, <<"val">>},
{<<"key">>, <<"val">>}],
{ok, _} = emqtt:publish(C, <<"my/topic">>,
#{'Content-Type' => <<"text/plain">>,
'Correlation-Data' => Correlation,
'Response-Topic' => MqttResponseTopic,
'User-Property' => UserProperty},
RequestPayload, [{qos, 1}]),
{#'basic.get_ok'{},
#amqp_msg{payload = RequestPayload,
props = #'P_basic'{content_type = <<"text/plain">>,
correlation_id = Correlation,
delivery_mode = 2,
headers = Headers}}} = amqp_channel:call(Ch, #'basic.get'{queue = Q}),
%% AMQP 0.9.1 expects unique headers sorted by key.
[{<<"key">>, longstr, <<"val">>},
{<<"rabbit🐇"/utf8>>, longstr, <<"carrot🥕"/utf8>>},
{<<"x-reply-to-topic">>, longstr, AmqpResponseTopic}] = Headers,
%% AMQP 0.9.1 to MQTT 5.0
ReplyPayload = <<"{\"my\" : \"reply\"}">>,
amqp_channel:call(Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = AmqpResponseTopic},
#amqp_msg{payload = ReplyPayload,
props = #'P_basic'{correlation_id = Correlation,
content_type = <<"application/json">>,
headers = Headers ++ [{<<"a">>, unsignedint, 4},
{<<"b">>, bool, true},
{"c", binary, <<0, 255, 0>>}]}}),
receive {publish,
#{client_pid := C,
topic := MqttResponseTopic,
payload := ReplyPayload,
properties := #{'Content-Type' := <<"application/json">>,
'Correlation-Data' := Correlation,
'User-Property' := UserProperty1,
'Subscription-Identifier' := 999}}} ->
?assertEqual(
[{<<"a">>, <<"4">>},
{<<"b">>, <<"true">>},
{<<"key">>, <<"val">>},
{<<"rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>}],
lists:sort(UserProperty1))
after 1000 -> ct:fail("did not receive reply")
end,
ok = emqtt:disconnect(C).
amqp(Config) ->
Host = ?config(rmq_hostname, Config),
Port = rabbit_ct_broker_helpers:get_node_config(Config, 0, tcp_port_amqp),
ClientId = Container = atom_to_binary(?FUNCTION_NAME),
OpnConf = #{address => Host,
port => Port,
container_id => Container,
sasl => {plain, <<"guest">>, <<"guest">>}},
{ok, Connection1} = amqp10_client:open_connection(OpnConf),
{ok, Session1} = amqp10_client:begin_session(Connection1),
ReceiverLinkName = <<"test-receiver">>,
{ok, Receiver} = amqp10_client:attach_receiver_link(
Session1, ReceiverLinkName, <<"/topic/topic.1">>, unsettled),
%% MQTT 5.0 to AMQP 1.0
C = connect(ClientId, Config),
MqttResponseTopic = <<"response/topic">>,
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 999},
[{MqttResponseTopic, [{qos, 1}]}]),
Correlation = <<"some correlation ID">>,
ContentType = <<"text/plain">>,
RequestPayload = <<"my request">>,
UserProperty = [{<<"rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>},
{<<"x-rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>},
{<<"key">>, <<"val">>},
{<<"key">>, <<"val">>},
{<<"x-key">>, <<"val">>},
{<<"x-key">>, <<"val">>}],
{ok, _} = emqtt:publish(C, <<"topic/1">>,
#{'Content-Type' => ContentType,
'Correlation-Data' => Correlation,
'Response-Topic' => MqttResponseTopic,
'User-Property' => UserProperty},
RequestPayload, [{qos, 1}]),
%% As of 3.13, AMQP 1.0 is proxied via AMQP 0.9.1 and therefore the conversion from
%% mc_mqtt to mc_amqpl takes place. We therefore lose MQTT User Property and Response Topic
%% which gets converted to AMQP 0.9.1 headers. In the future, Native AMQP 1.0 will convert
%% from mc_mqtt to mc_amqp allowing us to do many more assertions here.
{ok, Msg1} = amqp10_client:get_msg(Receiver),
ct:pal("Received AMQP 1.0 message:~n~p", [Msg1]),
?assertEqual([RequestPayload], amqp10_msg:body(Msg1)),
?assertMatch(#{correlation_id := Correlation,
content_type := ContentType}, amqp10_msg:properties(Msg1)),
?assert(amqp10_msg:header(durable, Msg1)),
?assert(amqp10_msg:header(first_acquirer, Msg1)),
ok = amqp10_client:settle_msg(Receiver, Msg1, accepted),
ok = amqp10_client:detach_link(Receiver),
ok = amqp10_client:end_session(Session1),
ok = amqp10_client:close_connection(Connection1),
%% AMQP 1.0 to MQTT 5.0
{ok, Connection2} = amqp10_client:open_connection(OpnConf),
{ok, Session2} = amqp10_client:begin_session(Connection2),
SenderLinkName = <<"test-sender">>,
{ok, Sender} = amqp10_client:attach_sender_link(
%% With Native AMQP 1.0, address should be read from received reply-to
Session2, SenderLinkName, <<"/topic/response.topic">>, unsettled),
receive {amqp10_event, {link, Sender, credited}} -> ok
after 1000 -> ct:fail(credited_timeout)
end,
DTag = <<"my-dtag">>,
ReplyPayload = <<"my response">>,
Msg2a = amqp10_msg:new(DTag, ReplyPayload),
Msg2b = amqp10_msg:set_properties(
#{correlation_id => Correlation,
content_type => ContentType},
Msg2a),
Msg2 = amqp10_msg:set_headers(#{durable => true}, Msg2b),
ok = amqp10_client:send_msg(Sender, Msg2),
receive {amqp10_disposition, {accepted, DTag}} -> ok
after 1000 -> ct:fail(settled_timeout)
end,
ok = amqp10_client:detach_link(Sender),
ok = amqp10_client:end_session(Session2),
ok = amqp10_client:close_connection(Connection2),
receive {publish, MqttMsg} ->
ct:pal("Received MQTT message:~n~p", [MqttMsg]),
?assertMatch(
#{client_pid := C,
qos := 1,
topic := MqttResponseTopic,
payload := ReplyPayload,
properties := #{'Content-Type' := ContentType,
'Correlation-Data' := Correlation,
'Subscription-Identifier' := 999}},
MqttMsg)
after 1000 -> ct:fail("did not receive reply")
end,
ok = emqtt:disconnect(C).
stomp(Config) ->
{ok, StompC0} = stomp_connect(Config),
ok = stomp_send(StompC0, "SUBSCRIBE", [{"destination", "/topic/t.1"},
{"receipt", "my-receipt"},
{"id", "subscription-888"}]),
{#stomp_frame{command = "RECEIPT",
headers = [{"receipt-id","my-receipt"}]}, StompC1} = stomp_recv(StompC0),
%% MQTT 5.0 to STOMP 1.2
C = connect(<<"my-mqtt-client">>, Config),
MqttResponseTopic = <<"response/topic">>,
{ok, _, [1]} = emqtt:subscribe(C, #{'Subscription-Identifier' => 999},
[{MqttResponseTopic, [{qos, 1}]}]),
Correlation = <<"some correlation ID">>,
ContentType = <<"application/json">>,
RequestPayload = <<"{\"my\" : \"request\"}">>,
UserProperty = [{<<"rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>},
{<<"x-rabbit🐇"/utf8>>, <<"carrot🥕"/utf8>>},
%% "If a client or a server receives repeated frame header entries,
%% only the first header entry SHOULD be used as the value of header
%% entry. " [STOMP 1.2]
{<<"key">>, <<"val1">>},
{<<"key">>, <<"val2">>},
{<<"x-key">>, <<"val1">>},
{<<"x-key">>, <<"val2">>}],
{ok, _} = emqtt:publish(C, <<"t/1">>,
#{'Content-Type' => ContentType,
'Correlation-Data' => Correlation,
'Response-Topic' => MqttResponseTopic,
'User-Property' => UserProperty},
RequestPayload, [{qos, 1}]),
{#stomp_frame{command = "MESSAGE",
headers = Headers0,
body_iolist = Body} = Msg1, StompC2} = stomp_recv(StompC1),
?assertEqual(RequestPayload, iolist_to_binary(Body)),
Headers1 = maps:from_list(Headers0),
Headers = maps:map(fun(_K, V) -> unicode:characters_to_binary(V) end, Headers1),
ct:pal("Received STOMP 1.2 message:~n~p~n"
"with headers map:~n~p", [Msg1, Headers]),
?assertMatch(
#{"content-type" := ContentType,
"correlation-id" := Correlation,
"destination" := <<"/topic/t.1">>,
%% With Native STOMP, this should be translated to
%% reply-to: /topic/response.topic
"x-reply-to-topic" := <<"response.topic">>,
"subscription" := <<"subscription-888">>,
"persistent" := <<"true">>,
%% The STOMP spec mandates headers to be encoded as UTF-8, but unfortunately the RabbitMQ
%% STOMP implementation (as of 3.13) does not adhere and therefore does not provide UTF-8 support.
% "rabbit🐇" := <<"carrot🥕"/utf8>>,
% "x-rabbit🐇" := <<"carrot🥕"/utf8>>,
"key" := <<"val1">>,
"x-key" := <<"val1">>
},
Headers),
%% STOMP 1.2 to MQTT 5.0
ok = stomp_send(StompC2, "SEND",
[{"destination", "/topic/response.topic"},
{"persistent", "true"},
{"content-type", "application/json"},
{"correlation-id", binary_to_list(Correlation)},
{"x-key", "val4"}],
["{\"my\" : \"response\"}"]),
ok = stomp_disconnect(StompC2),
receive {publish, MqttMsg} ->
ct:pal("Received MQTT message:~n~p", [MqttMsg]),
#{client_pid := C,
qos := 1,
topic := MqttResponseTopic,
payload := <<"{\"my\" : \"response\"}">>,
properties := #{'Content-Type' := ContentType,
'Correlation-Data' := Correlation,
'User-Property' := UserProp}} = MqttMsg,
?assert(lists:member({<<"x-key">>, <<"val4">>}, UserProp))
after 1000 -> ct:fail("did not receive reply")
end,
ok = emqtt:disconnect(C).
stream(_Config) ->
{skip, "TODO write test"}.
%% -------------------------------------------------------------------
%% Helpers
%% -------------------------------------------------------------------
delete_queues() ->
[{ok, 0} = rabbit_amqqueue:delete(Q, false, false, <<"dummy">>) || Q <- rabbit_amqqueue:list()].
%% -------------------------------------------------------------------
%% STOMP client BEGIN
%% -------------------------------------------------------------------
%% Below STOMP client is a simplified version of deps/rabbitmq_stomp/test/src/rabbit_stomp_client.erl
%% It would be better to use rabbit_stomp_client directly.
stomp_connect(Config) ->
Port = rabbit_ct_broker_helpers:get_node_config(Config, 0, tcp_port_stomp),
{ok, Sock} = gen_tcp:connect(localhost, Port, [{active, false}, binary]),
Client0 = {Sock, []},
stomp_send(Client0, "CONNECT", [{"accept-version", "1.2"}]),
{#stomp_frame{command = "CONNECTED"}, Client1} = stomp_recv(Client0),
{ok, Client1}.
stomp_disconnect(Client = {Sock, _}) ->
stomp_send(Client, "DISCONNECT"),
gen_tcp:close(Sock).
stomp_send(Client, Command) ->
stomp_send(Client, Command, []).
stomp_send(Client, Command, Headers) ->
stomp_send(Client, Command, Headers, []).
stomp_send({Sock, _}, Command, Headers, Body) ->
Frame = rabbit_stomp_frame:serialize(
#stomp_frame{command = list_to_binary(Command),
headers = Headers,
body_iolist = Body}),
gen_tcp:send(Sock, Frame).
stomp_recv({_Sock, []} = Client) ->
stomp_recv(Client, rabbit_stomp_frame:initial_state(), 0);
stomp_recv({Sock, [Frame | Frames]}) ->
{Frame, {Sock, Frames}}.
stomp_recv(Client = {Sock, _}, FrameState, Length) ->
{ok, Payload} = gen_tcp:recv(Sock, Length, 1000),
stomp_parse(Payload, Client, FrameState, Length).
stomp_parse(Payload, Client = {Sock, FramesRev}, FrameState, Length) ->
case rabbit_stomp_frame:parse(Payload, FrameState) of
{ok, Frame, <<>>} ->
stomp_recv({Sock, lists:reverse([Frame | FramesRev])});
{ok, Frame, <<"\n">>} ->
stomp_recv({Sock, lists:reverse([Frame | FramesRev])});
{ok, Frame, Rest} ->
stomp_parse(Rest, {Sock, [Frame | FramesRev]},
rabbit_stomp_frame:initial_state(), Length);
{more, NewState} ->
stomp_recv(Client, NewState, 0)
end.
%% -------------------------------------------------------------------
%% STOMP client END
%% -------------------------------------------------------------------

6
deps/rabbitmq_mqtt/test/reader_SUITE.erl vendored
View File

@ -62,11 +62,7 @@ merge_app_env(Config) ->
init_per_suite(Config) ->
rabbit_ct_helpers:log_environment(),
Config1 = rabbit_ct_helpers:set_config(Config, [
{rmq_nodename_suffix, ?MODULE},
{rmq_extra_tcp_ports, [tcp_port_mqtt_extra,
tcp_port_mqtt_tls_extra]}
]),
Config1 = rabbit_ct_helpers:set_config(Config, {rmq_nodename_suffix, ?MODULE}),
Config2 = rabbit_ct_helpers:run_setup_steps(
Config1,
[fun merge_app_env/1] ++

7
deps/rabbitmq_mqtt/test/retainer_SUITE.erl vendored
View File

@ -64,12 +64,7 @@ init_per_group(G, Config)
init_per_group(Group, Config0) ->
Suffix = rabbit_ct_helpers:testcase_absname(Config0, "", "-"),
Config1 = rabbit_ct_helpers:set_config(
Config0,
[
{rmq_nodename_suffix, Suffix},
{rmq_extra_tcp_ports, [tcp_port_mqtt_extra,
tcp_port_mqtt_tls_extra]}
]),
Config0, {rmq_nodename_suffix, Suffix}),
Mod = list_to_atom("rabbit_mqtt_retained_msg_store_" ++ atom_to_list(Group)),
Env = [{rabbitmq_mqtt, [{retained_message_store, Mod}]},
{rabbit, [

80
deps/rabbitmq_mqtt/test/shared_SUITE.erl vendored
View File

@ -117,6 +117,7 @@ cluster_size_1_tests() ->
,clean_session_node_kill
,rabbit_status_connection_count
,trace
,trace_large_message
,max_packet_size_unauthenticated
,max_packet_size_authenticated
,default_queue_type
@ -129,6 +130,7 @@ cluster_size_1_tests() ->
cluster_size_3_tests() ->
[
pubsub,
queue_down_qos1,
consuming_classic_mirrored_queue_down,
consuming_classic_queue_down,
@ -180,9 +182,7 @@ init_per_group(Group, Config0) ->
Config1 = rabbit_ct_helpers:set_config(
Config0,
[{rmq_nodes_count, Nodes},
{rmq_nodename_suffix, Suffix},
{rmq_extra_tcp_ports, [tcp_port_mqtt_extra,
tcp_port_mqtt_tls_extra]}]),
{rmq_nodename_suffix, Suffix}]),
Config2 = rabbit_ct_helpers:merge_app_env(
Config1,
{rabbit, [{classic_queue_default_version, 2}]}),
@ -675,6 +675,45 @@ global_counters(Config) ->
messages_unroutable_returned_total => 1},
get_global_counters(Config, ProtoVer))).
pubsub(Config) ->
Topic0 = <<"t/0">>,
Topic1 = <<"t/1">>,
C0 = connect(<<"c0">>, Config, 0, []),
C1 = connect(<<"c1">>, Config, 1, []),
{ok, _, [1]} = emqtt:subscribe(C0, Topic0, qos1),
{ok, _, [1]} = emqtt:subscribe(C1, Topic1, qos1),
{ok, _} = emqtt:publish(C0, Topic1, <<"m1">>, qos1),
receive {publish, #{client_pid := C1,
qos := 1,
payload := <<"m1">>}} -> ok
after 1000 -> ct:fail("missing m1")
end,
ok = emqtt:publish(C0, Topic1, <<"m2">>, qos0),
receive {publish, #{client_pid := C1,
qos := 0,
payload := <<"m2">>}} -> ok
after 1000 -> ct:fail("missing m2")
end,
{ok, _} = emqtt:publish(C1, Topic0, <<"m3">>, qos1),
receive {publish, #{client_pid := C0,
qos := 1,
payload := <<"m3">>}} -> ok
after 1000 -> ct:fail("missing m3")
end,
ok = emqtt:publish(C1, Topic0, <<"m4">>, qos0),
receive {publish, #{client_pid := C0,
qos := 0,
payload := <<"m4">>}} -> ok
after 1000 -> ct:fail("missing m4")
end,
ok = emqtt:disconnect(C0),
ok = emqtt:disconnect(C1).
queue_down_qos1(Config) ->
{Conn1, Ch1} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 1),
CQ = Topic = atom_to_binary(?FUNCTION_NAME),
@ -1482,8 +1521,7 @@ trace(Config) ->
<<"vhost">> := <<"/">>,
<<"channel">> := 0,
<<"user">> := <<"guest">>,
<<"properties">> := #{<<"delivery_mode">> := 2,
<<"headers">> := #{<<"x-mqtt-publish-qos">> := 1}},
<<"properties">> := #{<<"delivery_mode">> := 2},
<<"routed_queues">> := [<<"mqtt-subscription-trace_subscriberqos0">>]},
rabbit_misc:amqp_table(PublishHeaders)),
@ -1498,8 +1536,7 @@ trace(Config) ->
<<"vhost">> := <<"/">>,
<<"channel">> := 0,
<<"user">> := <<"guest">>,
<<"properties">> := #{<<"delivery_mode">> := 2,
<<"headers">> := #{<<"x-mqtt-publish-qos">> := 1}},
<<"properties">> := #{<<"delivery_mode">> := 2},
<<"redelivered">> := 0},
rabbit_misc:amqp_table(DeliverHeaders)),
@ -1512,6 +1549,35 @@ trace(Config) ->
delete_queue(Ch, TraceQ),
[ok = emqtt:disconnect(C) || C <- [Pub, Sub]].
trace_large_message(Config) ->
TraceQ = <<"trace-queue">>,
Ch = rabbit_ct_client_helpers:open_channel(Config),
declare_queue(Ch, TraceQ, []),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = TraceQ,
exchange = <<"amq.rabbitmq.trace">>,
routing_key = <<"deliver.*">>}),
C = connect(<<"my-client">>, Config),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_on"]),
{ok, _, [0]} = emqtt:subscribe(C, <<"/my/topic">>),
Payload0 = binary:copy(<<"x">>, 1_000_000),
Payload = <<Payload0/binary, "y">>,
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = <<".my.topic">>},
#amqp_msg{payload = Payload}),
ok = expect_publishes(C, <<"/my/topic">>, [Payload]),
timer:sleep(10),
?assertMatch(
{#'basic.get_ok'{routing_key = <<"deliver.mqtt-subscription-my-clientqos0">>},
#amqp_msg{payload = Payload}},
amqp_channel:call(Ch, #'basic.get'{queue = TraceQ})
),
{ok, _} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, ["trace_off"]),
delete_queue(Ch, TraceQ),
ok = emqtt:disconnect(C).
max_packet_size_unauthenticated(Config) ->
ClientId = ?FUNCTION_NAME,
Opts = [{will_topic, <<"will/topic">>}],

13
deps/rabbitmq_mqtt/test/util_SUITE.erl vendored
View File

@ -7,7 +7,6 @@
-module(util_SUITE).
-compile([export_all, nowarn_export_all]).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
all() ->
@ -18,7 +17,6 @@ all() ->
groups() ->
[
{tests, [parallel], [
coerce_exchange,
coerce_vhost,
coerce_default_user,
coerce_default_pass,
@ -27,22 +25,13 @@ groups() ->
}
].
suite() ->
[{timetrap, {seconds, 60}}].
init_per_suite(Config) ->
ok = application:load(rabbitmq_mqtt),
Config.
end_per_suite(Config) ->
ok = application:unload(rabbitmq_mqtt),
Config.
init_per_group(_, Config) -> Config.
end_per_group(_, Config) -> Config.
init_per_testcase(_, Config) -> Config.
end_per_testcase(_, Config) -> Config.
coerce_exchange(_) ->
?assertEqual(<<"amq.topic">>, rabbit_mqtt_util:env(exchange)).
coerce_vhost(_) ->
?assertEqual(<<"/">>, rabbit_mqtt_util:env(vhost)).

204
deps/rabbitmq_mqtt/test/v5_SUITE.erl vendored
View File

@ -104,7 +104,6 @@ cluster_size_1_tests() ->
publish_property_payload_format_indicator,
publish_property_response_topic_correlation_data,
publish_property_user_property,
publish_property_mqtt_to_amqp091,
disconnect_with_will,
will_qos2,
will_delay_greater_than_session_expiry,
@ -127,7 +126,9 @@ cluster_size_1_tests() ->
topic_alias_disallowed,
topic_alias_retained_message,
topic_alias_disallowed_retained_message,
extended_auth
extended_auth,
headers_exchange,
consistent_hash_exchange
].
cluster_size_3_tests() ->
@ -165,9 +166,7 @@ init_per_group(Group, Config0) ->
Config0,
[{mqtt_version, v5},
{rmq_nodes_count, Nodes},
{rmq_nodename_suffix, Suffix},
{rmq_extra_tcp_ports, [tcp_port_mqtt_extra,
tcp_port_mqtt_tls_extra]}]),
{rmq_nodename_suffix, Suffix}]),
Config2 = rabbit_ct_helpers:merge_app_env(
Config1,
{rabbit, [{classic_queue_default_version, 2},
@ -1100,7 +1099,8 @@ session_upgrade_v3_v5_amqp091_pub(Config) ->
amqp_channel:call(Ch,
#'basic.publish'{exchange = <<"amq.topic">>,
routing_key = Topic},
#amqp_msg{payload = Payload}),
#amqp_msg{payload = Payload,
props = #'P_basic'{delivery_mode = 2}}),
Subv5 = connect(ClientId, Config, [{proto_ver, v5}, {clean_start, false}]),
?assertEqual(5, proplists:get_value(proto_ver, emqtt:info(Subv5))),
@ -1218,8 +1218,7 @@ publish_property_payload_format_indicator(Config) ->
{ok, _} = emqtt:publish(C, Topic, #{'Payload-Format-Indicator' => 0}, <<"m1">>, [{qos, 1}]),
{ok, _} = emqtt:publish(C, Topic, #{'Payload-Format-Indicator' => 1}, <<"m2">>, [{qos, 1}]),
receive {publish, #{payload := <<"m1">>,
properties := Props}} ->
?assertEqual(0, maps:size(Props))
properties := #{'Payload-Format-Indicator' := 0}}} -> ok
after 1000 -> ct:fail("did not receive m1")
end,
receive {publish, #{payload := <<"m2">>,
@ -1304,48 +1303,6 @@ publish_property_user_property(Config) ->
end,
ok = emqtt:disconnect(C).
%% Test Properties interoperability between MQTT and AMQP 0.9.1
publish_property_mqtt_to_amqp091(Config) ->
Q = ClientId = atom_to_binary(?FUNCTION_NAME),
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = Q}),
#'queue.bind_ok'{} = amqp_channel:call(Ch, #'queue.bind'{queue = Q,
exchange = <<"amq.topic">>,
routing_key = <<"my.topic">>}),
C = connect(ClientId, Config),
MqttResponseTopic = <<"response/topic">>,
{ok, _, [1]} = emqtt:subscribe(C, MqttResponseTopic, qos1),
Correlation = <<"some correlation ID">>,
RequestPayload = <<"my request">>,
{ok, _} = emqtt:publish(C, <<"my/topic">>,
#{'Content-Type' => <<"text/plain">>,
'Correlation-Data' => Correlation,
'Response-Topic' => MqttResponseTopic},
RequestPayload, [{qos, 1}]),
{#'basic.get_ok'{},
#amqp_msg{payload = RequestPayload,
props = #'P_basic'{content_type = <<"text/plain">>,
correlation_id = Correlation,
delivery_mode = 2,
headers = Headers}}} = amqp_channel:call(Ch, #'basic.get'{queue = Q}),
{<<"x-opt-reply-to-topic">>, longstr, AmqpResponseTopic} = rabbit_basic:header(<<"x-opt-reply-to-topic">>, Headers),
ReplyPayload = <<"{\"my\" : \"reply\"}">>,
amqp_channel:call(Ch, #'basic.publish'{exchange = <<"amq.topic">>,
routing_key = AmqpResponseTopic},
#amqp_msg{payload = ReplyPayload,
props = #'P_basic'{correlation_id = Correlation,
content_type = <<"application/json">>}}),
receive {publish,
#{client_pid := C,
topic := MqttResponseTopic,
payload := ReplyPayload,
properties := #{'Content-Type' := <<"application/json">>,
'Correlation-Data' := Correlation}}} -> ok
after 500 -> ct:fail("did not receive reply")
end,
#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}),
ok = emqtt:disconnect(C).
disconnect_with_will(Config) ->
Topic = Payload = ClientId = atom_to_binary(?FUNCTION_NAME),
Sub = connect(<<"subscriber">>, Config),
@ -2011,10 +1968,156 @@ extended_auth(Config) ->
unlink(C),
?assertEqual({error, {bad_authentication_method, #{}}}, Connect(C)).
%% Binding a headers exchange to the MQTT topic exchange should support
%% routing based on (topic and) User Property in the PUBLISH packet.
headers_exchange(Config) ->
HeadersX = <<"my-headers-exchange">>,
Q1 = <<"q1">>,
Q2 = <<"q2">>,
Qs = [Q1, Q2],
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'exchange.declare_ok'{} = amqp_channel:call(
Ch, #'exchange.declare'{exchange = HeadersX,
type = <<"headers">>,
durable = true,
auto_delete = true}),
#'exchange.bind_ok'{} = amqp_channel:call(
Ch, #'exchange.bind'{destination = HeadersX,
source = <<"amq.topic">>,
routing_key = <<"my.topic">>}),
[#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{queue = Q,
durable = true}) || Q <- Qs],
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q1,
exchange = HeadersX,
arguments = [{<<"x-match">>, longstr, <<"any">>},
{<<"k1">>, longstr, <<"v1">>},
{<<"k2">>, longstr, <<"v2">>}]
}),
#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q2,
exchange = HeadersX,
arguments = [{<<"x-match">>, longstr, <<"all-with-x">>},
{<<"k1">>, longstr, <<"v1">>},
{<<"k2">>, longstr, <<"v2">>},
{<<"x-k3">>, longstr, <<"🐇"/utf8>>}]
}),
C = connect(?FUNCTION_NAME, Config),
Topic = <<"my/topic">>,
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k1">>, <<"v1">>},
{<<"k2">>, <<"v2">>},
{<<"x-k3">>, unicode:characters_to_binary("🐇")}
]},
<<"m1">>, [{qos, 1}]),
[?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m1">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
ok = emqtt:publish(C, Topic, <<"m2">>),
[?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k1">>, <<"nope">>}]},
<<"m3">>, [{qos, 1}]),
[?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q})) || Q <- Qs],
{ok, _} = emqtt:publish(
C, Topic,
#{'User-Property' => [{<<"k2">>, <<"v2">>}]},
<<"m4">>, [{qos, 1}]),
?assertMatch({#'basic.get_ok'{}, #amqp_msg{payload = <<"m4">>}},
amqp_channel:call(Ch, #'basic.get'{queue = Q1})),
?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = Q2})),
ok = emqtt:disconnect(C),
[#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}) || Q <- Qs],
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
%% Binding a consistent hash exchange to the MQTT topic exchange should support
%% consistent routing based on Correlation-Data in the PUBLISH packet.
consistent_hash_exchange(Config) ->
ok = rabbit_ct_broker_helpers:enable_plugin(Config, 0, rabbitmq_consistent_hash_exchange),
HashX = <<"my-consistent-hash-exchange">>,
Q1 = <<"q1">>,
Q2 = <<"q2">>,
Qs = [Q1, Q2],
Ch = rabbit_ct_client_helpers:open_channel(Config),
#'exchange.declare_ok'{} = amqp_channel:call(
Ch, #'exchange.declare'{
exchange = HashX,
type = <<"x-consistent-hash">>,
arguments = [{<<"hash-property">>, longstr, <<"correlation_id">>}],
durable = true,
auto_delete = true}),
#'exchange.bind_ok'{} = amqp_channel:call(
Ch, #'exchange.bind'{destination = HashX,
source = <<"amq.topic">>,
routing_key = <<"a.*">>}),
[#'queue.declare_ok'{} = amqp_channel:call(
Ch, #'queue.declare'{queue = Q,
durable = true}) || Q <- Qs],
[#'queue.bind_ok'{} = amqp_channel:call(
Ch, #'queue.bind'{queue = Q,
exchange = HashX,
%% weight
routing_key = <<"1">>}) || Q <- Qs],
Rands = [integer_to_binary(rand:uniform(1000)) || _ <- lists:seq(1, 30)],
UniqRands = lists:uniq(Rands),
NumMsgs = 150,
C = connect(?FUNCTION_NAME, Config),
[begin
N = integer_to_binary(rand:uniform(1_000_000)),
Topic = <<"a/", N/binary>>,
{ok, _} = emqtt:publish(C, Topic,
#{'Correlation-Data' => lists:nth(rand:uniform(length(UniqRands)), UniqRands)},
N, [{qos, 1}])
end || _ <- lists:seq(1, NumMsgs)],
#'basic.consume_ok'{consumer_tag = Ctag1} = amqp_channel:subscribe(
Ch, #'basic.consume'{queue = Q1,
no_ack = true}, self()),
#'basic.consume_ok'{consumer_tag = Ctag2} = amqp_channel:subscribe(
Ch, #'basic.consume'{queue = Q2,
no_ack = true}, self()),
{N1, Corrs1} = receive_correlations(Ctag1, 0, sets:new([{version, 2}])),
{N2, Corrs2} = receive_correlations(Ctag2, 0, sets:new([{version, 2}])),
ct:pal("q1: ~b messages, ~b unique correlation-data", [N1, sets:size(Corrs1)]),
ct:pal("q2: ~b messages, ~b unique correlation-data", [N2, sets:size(Corrs2)]),
%% All messages should be routed.
?assertEqual(NumMsgs, N1 + N2),
%% Each of the 2 queues should have received at least 1 message.
?assert(sets:size(Corrs1) > 0),
?assert(sets:size(Corrs2) > 0),
%% Assert that the consistent hash exchange routed the given Correlation-Data consistently.
%% The same Correlation-Data should never be present in both queues.
Intersection = sets:intersection(Corrs1, Corrs2),
?assert(sets:is_empty(Intersection)),
ok = emqtt:disconnect(C),
[#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = Q}) || Q <- Qs],
ok = rabbit_ct_client_helpers:close_channels_and_connection(Config, 0).
%% -------------------------------------------------------------------
%% Helpers
%% -------------------------------------------------------------------
receive_correlations(Ctag, N, Set) ->
receive {#'basic.deliver'{consumer_tag = Ctag},
#amqp_msg{props = #'P_basic'{correlation_id = Corr}}} ->
?assert(is_binary(Corr)),
receive_correlations(Ctag, N + 1, sets:add_element(Corr, Set))
after 200 ->
{N, Set}
end.
assert_no_queue_ttl(NumQs, Config) ->
Qs = rpc(Config, rabbit_amqqueue, list, []),
?assertEqual(NumQs, length(Qs)),
@ -2045,4 +2148,3 @@ assert_nothing_received(Timeout) ->
receive Unexpected -> ct:fail("Received unexpected message: ~p", [Unexpected])
after Timeout -> ok
end.

4
deps/rabbitmq_random_exchange/src/rabbit_exchange_type_random.erl vendored
View File

@ -26,7 +26,7 @@
recover/2,
remove_bindings/3,
validate_binding/2,
route/2,
route/3,
serialise_events/0,
validate/1,
info/1,
@ -36,7 +36,7 @@
description() ->
[{name, <<"x-random">>}, {description, <<"Randomly picks a binding (queue) to route via (to).">>}].
route(_X=#exchange{name = Name}, _Delivery) ->
route(#exchange{name = Name}, _Message, _Options) ->
Matches = rabbit_router:match_routing_key(Name, ['_']),
case length(Matches) of
Len when Len < 2 -> Matches;

33
deps/rabbitmq_recent_history_exchange/src/rabbit_exchange_type_recent_history.erl vendored
View File

@ -7,14 +7,13 @@
-module(rabbit_exchange_type_recent_history).
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include("rabbit_recent_history.hrl").
-behaviour(rabbit_exchange_type).
-import(rabbit_misc, [table_lookup/2]).
-export([description/0, serialise_events/0, route/2]).
-export([description/0, serialise_events/0, route/3]).
-export([validate/1, validate_binding/2, create/2, delete/2, add_binding/3,
remove_bindings/3, assert_args_equivalence/2, policy_changed/2]).
-export([setup_schema/0, disable_plugin/0]).
@ -46,8 +45,7 @@ description() ->
serialise_events() -> false.
route(#exchange{name = XName,
arguments = Args},
#delivery{message = Message}) ->
arguments = Args}, Message, _Options) ->
Length = table_lookup(Args, <<"x-recent-history-length">>),
maybe_cache_msg(XName, Message, Length),
rabbit_router:match_routing_key(XName, ['_']).
@ -94,8 +92,7 @@ add_binding(none, #exchange{ name = XName },
{ok, X} ->
Msgs = get_msgs_from_cache(XName),
[begin
Delivery = rabbit_basic:delivery(false, false, Msg, undefined),
Qs = rabbit_exchange:route(X, Delivery),
Qs = rabbit_exchange:route(X, Msg),
case rabbit_amqqueue:lookup_many(Qs) of
[] ->
destination_not_found_error(Qs);
@ -125,23 +122,12 @@ disable_plugin() ->
%%----------------------------------------------------------------------------
%%private
maybe_cache_msg(XName,
#basic_message{content =
#content{properties =
#'P_basic'{headers = Headers}}}
= Message,
Length) ->
case Headers of
undefined ->
cache_msg(XName, Message, Length);
maybe_cache_msg(XName, Message, Length) ->
case mc:x_header(<<"x-recent-history-no-store">>, Message) of
{boolean, true} ->
ok;
_ ->
Store = table_lookup(Headers, <<"x-recent-history-no-store">>),
case Store of
{bool, true} ->
ok;
_ ->
cache_msg(XName, Message, Length)
end
cache_msg(XName, Message, Length)
end.
cache_msg(XName, Message, Length) ->
@ -153,8 +139,7 @@ get_msgs_from_cache(XName) ->
deliver_messages(Qs, Msgs) ->
lists:map(
fun (Msg) ->
Delivery = rabbit_basic:delivery(false, false, Msg, undefined),
rabbit_amqqueue:deliver(Qs, Delivery)
_ = rabbit_queue_type:deliver(Qs, Msg, #{}, stateless)
end, lists:reverse(Msgs)).
-spec destination_not_found_error(string()) -> no_return().

6
deps/rabbitmq_sharding/src/rabbit_sharding_exchange_type_modulus_hash.erl vendored
View File

@ -11,7 +11,7 @@
-behaviour(rabbit_exchange_type).
-export([description/0, serialise_events/0, route/2, info/1, info/2]).
-export([description/0, serialise_events/0, route/3, info/1, info/2]).
-export([validate/1, validate_binding/2,
create/2, delete/2, policy_changed/2,
add_binding/3, remove_bindings/3, assert_args_equivalence/2]).
@ -33,8 +33,8 @@ description() ->
serialise_events() -> false.
route(#exchange{name = Name},
#delivery{message = #basic_message{routing_keys = Routes}}) ->
route(#exchange{name = Name}, Msg, _Options) ->
Routes = mc:get_annotation(routing_keys, Msg),
Qs = rabbit_router:match_routing_key(Name, ['_']),
case length(Qs) of
0 -> [];

6
deps/rabbitmq_shovel/test/dynamic_SUITE.erl vendored
View File

@ -8,6 +8,7 @@
-module(dynamic_SUITE).
-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-compile(export_all).
@ -189,8 +190,9 @@ headers(Config) ->
<<"test">>,
[{<<"src-queue">>, <<"src">>},
{<<"dest-queue">>, <<"dest">>}]),
#amqp_msg{props = #'P_basic'{headers = undefined}} =
publish_expect(Ch, <<>>, <<"src">>, <<"dest">>, <<"hi1">>),
?assertMatch(#amqp_msg{props = #'P_basic'{headers = H0}}
when H0 == undefined orelse H0 == [],
publish_expect(Ch, <<>>, <<"src">>, <<"dest">>, <<"hi1">>)),
shovel_test_utils:set_param(Config,
<<"test">>,

10
deps/rabbitmq_stream/src/rabbit_stream_manager.erl vendored
View File

@ -18,6 +18,7 @@
-behaviour(gen_server).
-include_lib("rabbit_common/include/rabbit_framing.hrl").
-include_lib("rabbit_common/include/rabbit.hrl").
-include_lib("rabbit/include/amqqueue.hrl").
@ -429,10 +430,11 @@ handle_call({route, RoutingKey, VirtualHost, SuperStream}, _From,
ExchangeName = rabbit_misc:r(VirtualHost, exchange, SuperStream),
Res = try
Exchange = rabbit_exchange:lookup_or_die(ExchangeName),
Delivery =
#delivery{message =
#basic_message{routing_keys = [RoutingKey]}},
case rabbit_exchange:route(Exchange, Delivery) of
Content = #content{properties = #'P_basic'{}},
DummyMsg = mc_amqpl:message(ExchangeName,
RoutingKey,
Content),
case rabbit_exchange:route(Exchange, DummyMsg) of
[] ->
{ok, no_route};
Routes ->

Some files were not shown because too many files have changed in this diff Show More