Trigger a 4.3.x alpha release build / trigger_alpha_build (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 26) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 27) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 28) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Type check (1.18, 28) (push) Waiting to runDetails
`rabbit_amqqueue:declare/7` consults the `is_enabled/0` callback of
`rabbit_queue_type` before attempting to create a queue.
`rabbit_stream_manager` emulates `rabbit_amqqueue:declare/7` rather
than calling it directly. To align with the `is_enabled/0` check in
`rabbit_amqqueue:declare/7`, `rabbit_stream_manager` needs to check
the callback during stream creation.
A quorum queue tries to repair its record in a tick handler. This can
happen during a network partition and the metadata store may itself be
unavailable, making the update likely to time out.
The default metadata store timeout is usually higher than the tick
interval, so the tick handler may be stuck during several ticks. The
record takes some time to be updated (timeout + tick interval, 30 + 5
seconds by default), significantly longer than it takes the metadata
store to trigger an election and recover.
Client applications may rely on the quorum queue topology to connect to
an appropriate node, so making the system reflect the actual topology
faster is important to them.
This commit makes the record update operations use a timeout 1-second
lower than the tick interval. The tick handler process should finish
earlier in case of metadata datastore unavailability and it should not
take more than a couple of ticks once the datastore is available to
update the record.
... instead of as part of a parallel CT group.
[Why]
This testsuite takes quite a lot of time. It affects the time to run the
parallel CT set 2 significantly, especially if it has to be restarted.
[Why]
The `*_queue_after_partition_recovery_1` testcases rely on the fact that
the queue process retries to delete its queue record in its terminate
function.
RabbitMQ 4.1.x and before don't have that. Thus, depending on the timing
of the election of the Khepri leader after a network partition, the test
might fail.
As mentioned in discussion #14426, the way that `cacerts` is handled by
cuttlefish schemas simply will not work if set.
If `cacerts` were set to a string value containing one X509 certificate,
it would eventually result in a crash because the `cacerts` ssl option
must be of [this type](https://www.erlang.org/doc/apps/ssl/ssl.html#t:client_option_cert/0):
```
{cacerts, CACerts :: [public_key:der_encoded()] | [public_key:combined_cert()]}
```
Neither of those are strings, of course.
This PR removes all use of `cacerts` in cuttlefish schemas. In addition,
it filters out `cacerts` and `certs_keys` from being JSON-encoded by an
HTTP API call to `/api/overview`. It _is_ technically possible to set
`cacerts` via `advanced.config`, so, if set, it would crash this API
call, as would `certs_keys`.
[Why]
With Mnesia, when the network partition strategy is set to
`pause_minority`, nodes on the "minority side" are stopped.
Thus, the exclusive queues that were hosted by nodes on that minority
side are lost:
* Consumers connected on these nodes are disconnected because the nodes
are stopped.
* Queue records on the majority side are deleted from the metadata
store.
This was ok with Mnesia and how this network partition handling strategy
is implemented. However, it does not work with Khepri because the nodes
on the "minority side" continue to run and serve clients. Therefore the
cluster ends up in a weird situation:
1. The "majority side" deleted the queue records.
2. When the network partition is solved, the "minority side" gets the
record deletion, but the queue processes continue to run.
This was similar for auto-delete queues.
[How]
With Khepri, we stop to delete transient queue records in general, just
because there is a node going down. Thanks to this, an exclusive or an
auto-delete queue and its consumer(s) are not affected by a network
partition: they continue to work.
However, if a node is really lost, we need to clean up dead queue
records. This was already done for durable queues with both Mnesia and
Khepri. But with Khepri, transient queue records persist in the store
like durable queue records (unlike with Mnesia).
That's why this commit changes the clean-up function,
`rabbit_amqqueue:forget_all_durable/1` into
`rabbit_amqqueue:forget_all/1` which deletes all queue records of queues
that were hosted on the given node, regardless if they are transient or
durable.
In addition to this, the queue process will spawn a temporary process
who will try to delete the underlying record indefinitely if no other
processes are waiting for a reply from the queue process. That's the
case for queues that are deleted because of an internal event (like the
exclusive/auto-delete conditions). The queue process will exit, which
will notify connections that the queue is gone.
Thanks to this, the temporary process will do its best to delete the
record in case of a network partition, whether the consumers go away
during or after that partition. That said, the node monitor drives some
failsafe code that cleans up record if the queue process was killed
before it could delete its own record.
Fixes#12949, #12597, #14527.
[Why]
So far, when there was a network partition with Mnesia, the most popular
partition handling strategies restarted RabbitMQ nodes. Therefore,
`rabbit` would execute the boot steps and one of them would notify other
members of the cluster that "this RabbitMQ node is live".
With Khepri, nodes are not restarted anymore and thus, boot steps are
not executed at the end of a network partition. As a consequence, other
members are not notified that a member is back online.
[How]
When the node monitor receives the `nodeup` message (managed by Erlang,
meaning that "a remote Erlang node just connected to this node through
Erlang distribution"), a `node_up` message is sent to all cluster
members (meaning "RabbitMQ is now running on the originating node").
Yeah, very poor naming...
This lets the RabbitMQ node monitor know when other nodes running
RabbitMQ are back online and react accordingly.
If a node is restarted, it means that another node could receive the
`node_up` message twice. The actions behind it must be idempotent.
This change adds config options for controlling the `is_enabled/0`
queue type callback for classic queues, quorum queues and stream
queues. This is useful for operators to be able to control whether
users can declare these queues - i.e. to prevent users from declaring
queue types which cannot be supported operationally.
Setting
queue_types.stream.enabled = false
In a config file for example prevents stream queues from being declared.
'user_provided_plugins_data_dir' make it immediately
obvious that the data directory is for 3rd party
plugins (user-provided plugins), and it is
not set up or used internally by RabbitMQ.
This is mainly for community plugins which want to migrate away from
Mnesia into a new subdir. (They might not even use Khepri, or use
Khepri but a different store id.)
Fixes#11304
Follow-up to:
* #3442
* 93db480bc4
* #13899
* #14326
Simply changing `+=` to `?=` doesn't work as there are other important
statements added to `RMQ_ERLC_OPTS` that can't be left out when `make
RMQ_ERLC_OPTS=''` is run. This only became apparent when a completely
cleaned-out checkout of `rabbitmq/rabbitmq-server` was attempted to be
built with that command.
This adds `NON_DETERMINISTIC` which will be used like this:
```
make NON_DETERMINISTIC=true
```
...and `+deterministic` won't be passed to `erlc` or other commands.
* Reduce ETS copy overhead when delivering to target queues
## What?
This commit avoids copying the full amqqueue record from ETS per incoming message
and target queue.
The amqqueue record contains 21 elements and for some queue types,
especially streams, some elements are themselves nested terms.
## How?
In Khepri, use a new `rabbit_khepri_queue_target` projection which
contains a subset of the full amqqueue record.
This way all relevant information to deliver to a target queue can be
looked up in a single ets:lookup_element call.
Alternative approaches are described in https://github.com/erlang/otp/issues/10211
## Benchmark
Fanout to 3 streams
Start broker:
```
make run-broker TEST_TMPDIR="$HOME/scratch/rabbit/test" \
FULL=1 \
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \
RABBITMQ_CONFIG_FILE="$HOME/scratch/rabbit/high-credit.config" \
PLUGINS="rabbitmq_management"
```
`high-credit.config` contains:
```
[
{rabbit, [
%% Maximum incoming-window of AMQP 1.0 session.
%% Default: 400
{max_incoming_window, 5000},
%% Maximum link-credit RabbitMQ grants to AMQP 1.0 sender.
%% Default: 128
{max_link_credit, 2000},
%% Maximum link-credit RabbitMQ AMQP 1.0 session grants to sending queue.
%% Default: 256
{max_queue_credit, 5000},
{loopback_users, []}
]},
{rabbitmq_management_agent, [
{disable_metrics_collector, true}
]}
].
```
Create the 3 streams and bindings to the fanout exchange:
```
deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=stream durable=true name=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss1 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=stream durable=true name=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss2 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=stream durable=true name=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss3 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss1 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss2 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss3
```
Start the client:
```
quiver-arrow send //host.docker.internal//exchanges/amq.fanout --summary --count 1m --body-size 4
```
`main` branch:
```
Count ............................................. 1,000,000 messages
Duration ............................................... 16.3 seconds
Message rate ......................................... 61,237 messages/s
```
with this PR:
```
Count ............................................. 1,000,000 messages
Duration ............................................... 14.2 seconds
Message rate ......................................... 70,309 messages/s
```
Hence, this PR increases the throughput when sending to 3 streams via AMQP by ~14%.
* Avoid creating 5 elems tuple
* Simplify rabbit_queue_type callbacks
deliver should only take targets and init should only take the full record
* Fix flaky test
* Fix specs
A queue client that send a message during a network partition
that later caused a distribution disconnection would in some cases
never resend the lost message, even if kept in the pending buffer.
Subsequent sends would be accepted by the state machine but would
never be enqueued as there would be a missing sequence.
In the case of publishers that use pre-settled sends the pending
messages would have also been incorrectly removed from the
pending map.
To fix we removed timer resend aapproach and instead have the leader
send leader_change messages on node up to prompt any queue clients
to resend their pending buffer.
The whole point of introducing rabbitmq-diagnostics,
rabbitmq-queues, rabbitmq-streams is to stop
growing the ctl command list, which has been massive
even five years ago.
This command displays cluster-wide message size
statistics. It's less detailed than what can be
retrieved from the Prometheus endpoint, but
it'll be available to all users, regardless of their
monitoring setup, or lack thereof.
These changes add the minimum required changes and artifacts to enable
cuttlefish schema tests, which can be run with this command:
```
make -C deps/rabbitmq_aws ct-config_schema
```
* AWS peer discovery: ensure consistent hostname path ordering
AWS EC2 API returns networkInterfaceSet and privateIpAddressesSet in
arbitrary order, causing non-deterministic hostname resolution during
peer discovery. This leads to inconsistent cluster formation.
Changes:
- Sort network interfaces by deviceIndex (0 first for primary ENI)
- Sort private IP addresses by primary flag (primary=true first)
- Add debug logging to show hostname path selection and sorting results
- Add comprehensive unit tests for sorting behavior
The sorting ensures deviceIndex=0 and primary=true IPs are consistently
selected first, making peer discovery deterministic across deployments.
* AWS peer discovery: ensure consistent hostname path ordering (address feedback on debug logs and sorting helper functions)
AMQP10 shovels don't need the amqp10 message format, the binary
can be translated directly into a message container and also
the other way around. The new amqp10_raw_msg just stores the payload
and information required to create the transfer frame, skipping
a few unnecessary encoding/decoding operations of the AMQP10 sections.
See discussion #14244
These changes will allow a user to make an HTTP API request to...
```
/api/ldap/validate/simple-bind
```
...with an appropriate JSON body, and the plugin will attempt a
connection to the specified LDAP server using the provided credentials.
This allows validation that a connection can be made to an LDAP server
from a RabbitMQ cluster environment.
* Add code and tests for `eldap:simple_bind` validation.
* Add support for testing TLS connections to OpenLDAP
* Add support for validating TLS related configuration via `/ldap/validate/simple-bind`
* Add support for various TLS options:
* versions
* depth
* multiple CA cert pem data
* Fall back to system certs if neither `cacertfile` nor `cacerts_pem_data` are provided to the `simple-bind` validation.
* Add `ssl_hostname_verification` support.
* Return 422 when auth fails
* Add more informative information when connection fails
* Add more tests with invalid input
* Catch parsing errors
* Tests for edge-cases for password / user_dn
* Add test for use_ssl + use_starttls combo
* Add test for ssl_options.depth validation
* Add validation tests for server_name_indication
And `eof` crashes.
The problem is that we may end up trying to read more data
from the file when scanning, despite being at the end of the
file. This results in the current Acc to be returned instead
of the remaining data being parsed.
This results in some messages at the end of the file being
truncated off despite still being in memory (and still pointing
to the end of the original file, well past the truncation point).
Osiris can read ahead data in case of small chunks. This saves system
calls and increases consumption rate dramatically for some streams.
This is transparent for the stream protocol, but requires a small tweak
for the stream queue type implementation (passing in the previous
iterator when creating a new one).
The read ahead is on by default but can be deactivated with to the new
stream.read_ahead configuration entry (true / false).
Co-authored-by: Karl Nilsson <kjnilsson@gmail.com>
References rabbitmq/osiris#192
This uses Cowboy's new direct data delivery mechanism,
which provides more performance.
Cowboy is now pinned to 2.14.0 and Cowlib to 2.16.0.
HTTP/2 Websocket is enabled by default. It can be disabled
as needed by setting `#{enable_connect_protocol => false}`
in the plugin's `cowboy_opts`.
Web-STOMP did not have HTTP/2 enabled before, now it does.
Web-MQTT already had HTTP/2 enabled.
* Add test case for binding args Khepri regression
This commit adds a test case for a regression/bug that occurs in Khepri.
```
make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia
```
succeeds, but
```
make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri
```
fails.
The problem is that ETS table `rabbit_khepri_index_route` cannot
differentiate between two bindings with different binding arguments, and
therefore deletes entries too early, leading to wrong routing decisions.
The solution to this bug is to include the binding arguments in the
`rabbit_khepri_index_route` projection, similar to how the binding args
are also included in the `rabbit_index_route` Mnesia table.
This bug/regression is an edge case and exists if the source exchange
type is `direct` or `fanout` and if different bindings arguments are
used by client apps. Note that such binding arguments are entirely
ignored when RabbitMQ performs routing decisions for the `direct` or
`fanout` exchange. However, there might be client apps that use binding
arguments to add some metadata to the binding, for example `app-id` or
`user` or `purpose` and might use this metadata as a form of reference
counting in deciding when to delete `auto-delete` exchanges or just for
informational/operational purposes.
* Fix regression with Khepri binding args
Fix#14533
* Speed up fanout exchange
Resolves#14531
## What?
Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below).
In addition to the fanout exchange, a similar speed up is achieved for the following exchange types:
* modulus hash
* random
* recent history
This applies only if Khepri is enabled.
## How?
Use an additional routing table (projection) whose table key is the source exchange.
Looking up the destinations happens then by an ETS table key.
Prior to this commit, CPUs were busy compiling the same match spec for every incoming message.
## Benchmark
1. Start RabbitMQ:
```
make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \
RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management"
```
where `advanced.config` contains:
```
[
{rabbitmq_management_agent, [
{disable_metrics_collector, true}
]}
].
```
2. Create a queue and binding:
```
deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1
```
3. Create the load
```
java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60
```
Before this commit:
```
sending rate avg: 97394 msg/s
receiving rate avg: 97394 msg/s
```
After this commit:
```
sending rate avg: 138677 msg/s
receiving rate avg: 138677 msg/s
```
The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts:
* 13.5% before this commit
* 3.4% after this commit
## Downsides
Additional ETS memory usage for the new projection table.
However, the new table does not store any binding entries for the following
source exchange types:
* direct
* headers
* topic
* x-local-random
* Add exchange binding tests
Test that exchange bindings work correctly with the new projection
tables `rabbit_khepri_route_by_source` and
`rabbit_khepri_route_by_source_key`.
* Always register all projections
Khepri won’t modify a projection that is already registered (based on its name).
* Protect ets:lookup_element/4 in try catch
See https://github.com/rabbitmq/rabbitmq-server/pull/11667#issue-2401399413
for rationale.
The previous implementation did not take the unique `passthrough` record
into account when formatting state.
To test the priority and non-priority scenarios, add `exit(kaboom)` in
`rabbit_amqqueue_process` `handle_call({basic_consume...` then run
PerfTest with these arguments:
```
--producers 1 --consumers 1 --pmessages 100 --queue-args x-max-priority=10
--producers 1 --consumers 1 --pmessages 100
```
* Call `Mod:format_state/1` if exported to possibly truncate huge states
* Add more information about truncated ram_pending_ack and disk_pending_ack
* Add `log.error_logger_format_depth` cuttlefish schema value
* Add `format_state/1` to `rabbit_channel`
* Add `log.summarize_process_state`, default is `false`, to enable summarizing process state for crash logs.
* Added `format_state` to `rabbit_classic_queue_index_v2` and `rabbit_classic_queue_store_v2`
* Ensure `rabbit_channel:format_state/1` uses `summarize_process_state_when_logged`
* Do not set `summarize_process_state_when_logged` value by default.
* Type specs
This commit makes read operations for the following Khepri projections much cheaper:
* rabbit_khepri_queue
* rabbit_khepri_exchange
* rabbit_khepri_index_route
* rabbit_khepri_topic_trie
Entries in these ETS tables are read for every message entering RabbitMQ.
Some messages entering RabbitMQ will cause even multiple reads from
these ETS tables, e.g. multiple reads from `rabbit_khepri_queue` if a message
is routed to more than one queue or multiple reads from `rabbit_khepri_index_route`
if a message has multiple routing keys.
On a busy RabbitMQ node, these tables are read concurrently (on multiple physical processors)
hundreds of thousands of times per second.
There can be at most one consumer per volatile queue instance.
This consumer must also have attached on the same channel/session as the
creator of the queue.
Prior to this commit, it was possible for clients on other
connections or sessions to attach a receiving link to an existing volatile
queue name, even though no messages would be delivered.
It's better for RabbitMQ to directly refuse the link at attach time.
# What?
* Support Direct Reply-To for AMQP 1.0
* Compared to AMQP 0.9.1, this PR allows for multiple volatile queues on a single
AMQP 1.0 session. Use case: JMS clients can create multiple temporary queues on
the same JMS/AMQP session:
* https://jakarta.ee/specifications/messaging/3.1/apidocs/jakarta.messaging/jakarta/jms/session#createTemporaryQueue()
* https://jakarta.ee/specifications/messaging/3.1/apidocs/jakarta.messaging/jakarta/jms/jmscontext#createTemporaryQueue()
* Fix missing metrics in for Direct Reply-To in AMQP 0.9.1, e.g.
`messages_delivered_total`
* Fix missing metrics (even without using Direct Reply-To ) in AMQP 0.9.1:
If stats level is not `fine`, global metrics `rabbitmq_global_messages_delivered_*` should still be incremented.
# Why?
* Allow for scalable at-most-once RPC reply delivery
Example use case: thousands of requesters connect, send a single
request, wait for a single reply, and disconnect.
This PR won't create any queue and won't write to the metadata store.
Therefore, there's less pressure on the metadata store, less pressure
on the Management API when listing all queues, less pressure on the
metrics subsystem, etc.
* Feature parity with AMQP 0.9.1
# How?
This PR extracts the previously channel specific Direct Reply-To code
into a new queue type: `rabbit_volatile_queue`.
"Volatile" describes the semantics, not a use-case. It signals non-durable,
zero-buffer, at-most-once, may-drop, and "not stored in Khepri."
This new queue type is then used for AMQP 1.0 and AMQP 0.9.1.
Sending to the volatile queue is stateless like previously with Direct Reply-To in AMQP 0.9.1 and like done
for the MQTT QoS 0 queue.
This allows for use cases where a single responder replies to e.g. 100k different requesters.
RabbitMQ will automatically auto grant new link-credit to the responder because the new queue type confirms immediately.
The key gets implicitly checked by the channel/session:
If the queue name (including the key) doesn’t exist, the `handle_event` callback for this queue isn’t invoked and therefore
no delivery will be sent to the responder.
This commit supports Direct Reply-To across AMQP 1.0 and 0.9.1. In other
words, the requester can be an AMQP 1.0 client while the responder is an
AMQP 0.9.1 client or vice versa.
RabbitMQ will internally convert between AMQP 0.9.1 `reply_to` and AMQP
1.0 `/queues/<queue>` address. The AMQP 0.9.1 `reply_to` property is
expected to contain a queue name. That's in line with the AMQP 0.9.1
spec:
> One of the standard message properties is Reply-To, which is designed
specifically for carrying the name of reply queues.
Compared to AMQP 0.9.1 where the requester sets the `reply_to` property
to `amq.rabbitmq.reply-to` and RabbitMQ modifies this field when
forwarding the message to the request queue, in AMQP 1.0 the requester
learns about the queue name from the broker at link attachment time.
The requester has to set the reply-to property to the server generated
queue name. That's because the server isn't allowed to modify the bare
message.
During link attachment time, the client has to set certain fields.
These fields are expected to be set by the RabbitMQ client libraries.
Here is an Erlang example:
```erl
Source = #{address => undefined,
durable => none,
expiry_policy => <<"link-detach">>,
dynamic => true,
capabilities => [<<"rabbitmq:volatile-queue">>]},
AttachArgs = #{name => <<"receiver">>,
role => {receiver, Source, self()},
snd_settle_mode => settled,
rcv_settle_mode => first},
{ok, Receiver} = amqp10_client:attach_link(Session, AttachArgs),
AddressReplyQ = receive {amqp10_event, {link, Receiver, {attached, Attach}}} ->
#'v1_0.attach'{source = #'v1_0.source'{address = {utf8, Addr}}} = Attach,
Addr
end,
```
The client then sends the message by setting the reply-to address as
follows:
```erl
amqp10_client:send_msg(
SenderRequester,
amqp10_msg:set_properties(
#{message_id => <<"my ID">>,
reply_to => AddressReplyQ},
amqp10_msg:new(<<"tag">>, <<"request">>))),
```
If the responder attaches to the queue target in the reply-to field,
RabbitMQ will check if the requester link is still attached. If the
requester detached, the link will be refused.
The responder can also attach to the anonymous null target and set the
`to` field to the `reply-to` address.
If RabbitMQ cannot deliver a reply, instead of buffering the reply,
RabbitMQ will be drop the reply and increment the following Prometheus metric:
```
rabbitmq_global_messages_dead_lettered_maxlen_total{queue_type="rabbit_volatile_queue",dead_letter_strategy="disabled"} 0.0
```
That's in line with the MQTT QoS 0 queue type.
A reply message could be dropped for a variety of reasons:
1. The requester ran out of link-credit. It's therefore the requester's
responsibility to grant sufficient link-credit on its receiving link.
2. RabbitMQ isn't allowed to deliver any message to due session flow
control. It's the requster's responsibility to keep the session window
large enough.
3. The requester doesn't consume messages fast enough causing TCP
backpressure being applied or the RabbitMQ AMQP writer proc isn't
scheduled quickly enough. The latter can happen for example if
RabbitMQ runs with a single scheduler (is assigned a single CPU
core). In either case, RabbitMQ internal flow control causes the
volatile queue to drop messages.
Therefore, if high throughput is required while message loss is undesirable, a classic queue should be used
instead of a volatile queue since the former buffers messages while the
latter doesn't.
The main difference between the volatile queue and the MQTT QoS 0 queue
is that the former isn't written to the metadata store.
# Breaking Change
Prior to this PR the following [documented caveat](https://www.rabbitmq.com/docs/4.0/direct-reply-to#limitations) applied:
> If the RPC server publishes with the mandatory flag set then `amq.rabbitmq.reply-to.*`
is treated as **not** a queue; i.e. if the server only publishes to this name then the message
will be considered "not routed"; a `basic.return` will be sent if the mandatory flag was set.
This PR removes this caveat.
This PR introduces the following new behaviour:
> If the RPC server publishes with the mandatory flag set, then `amq.rabbitmq.reply-to.*`
is treated as a queue (assuming this queue name is encoded correctly). However,
whether the requester is still there to consume the reply is not checked at routing time.
In other words, if the RPC server only publishes to this name, then the message will be
considered "routed" and RabbitMQ will therefore not send a `basic.return`.
A list of binaries was incorrectly converted by
`unicode:characters_to_binary` into one big binary. Add a function head
to match this case.
Also, add tests for the values that were not correctly formatted prior
to #14101 and #14381
Speed up Direct Reply-To in AMQP 0.9.1 because prior to this PR sending each RPC reply
was bottlenecked by the slowest network link between the RabbitMQ node the responder
connected to and all other RabbitMQ nodes. This regression was introduced in RabbitMQ 3.12.0
via d65637190a
This is meant to be used as a debugging tool via an eval command. It can
be hard to tell from the outside why a QQ is not snapshotting. Looking
at the `overview/1` is a nice debugging tool. The `smallest_raft_index`
key (and other keys like the `release_cursors`) points out why a
snapshot isn't being taken. The `smallest_raft_index` is the minimum of
three criteria though:
* oldest ready message
* oldest checked out message
* oldest message in the dlx state
This change adds all of those indexes so that it is clear which criteria
is keeping the `smallest_raft_index` down.
The test suites need to be excluded at group level, so the end_per_suite
is always executed and the cluster stopped. Otherwise, clusters
remain running in CI and the following suites find the TCP ports busy.
The new client library version returns reason
`{shutdown, connack_timeout}` in the monitor message.
Delete the monitor since the test case already asserts on the
`{error, connack_timeout}` return value from emqtt:connect/1.
Namely, 'starting'.
Previously, a Prometheus scraping endpoint hit
just at the right moment (when a shovel was
in the 'starting' state) would produce
a function_clause exception.
Bumps the dev-deps group with 1 update in the /deps/rabbit/test/amqp_jms_SUITE_data directory: org.apache.qpid:qpid-jms-client.
Updates `org.apache.qpid:qpid-jms-client` from 2.7.0 to 2.8.0
---
updated-dependencies:
- dependency-name: org.apache.qpid:qpid-jms-client
dependency-version: 2.8.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: dev-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
These lines can be very useful for determining why a subset of feature
flags is/isn't being enabled on boot. They're currently at debug level.
However a first-time boot of an unclustered node is a rare event, so
these lines shouldn't be noisy.
Commit e435bc6886 introduced behavior in
creating the source-dist archive that sets the file modification time to
the same time as the most recent `git` commit. This facilitates
reproducible builds, but it also has an issue. The code sets `TZ=`,
which means that on systems running in time zones which aren't UTC, the
modification time may be in the future (or past). It depends on the
system's behavior.
Up until recently, this only resulted in GNU make printing warnings
about modification times being in the future. However, with recent
changes in how escripts are built, this results in an "infinite make
loop" that is described in https://github.com/rabbitmq/rabbitmq-server/discussions/14440
and https://github.com/rabbitmq/rabbitmq-server/discussions/14443
This solution is to set the `TZ=UTC` when calculating the time of the
most recent `git` commit, as well as when using `touch -t` to set the
file modification times.
Fixes#14440, #14443
## What?
Refuse or detach the link instead of ending the session for many
link-level errors, including the following:
* Source queue or target exchange doesn't exist during attach
* Trying to consume from an exclusive queue on a different connection
* Delivery of message to a target queue fails
* Wrong delivery-id
* Wrong settled flag
* Queue declaration fails for dynamic queues
* Publishing to internal exchange
## Why?
Because many errors are scoped to a single terminus, detaching just that link
preserves the rest of the session’s links - avoiding needless disruption of
other traffic. AMQP 1.0’s error model is hierarchical; RabbitMQ should escalate to
ending the session only for session-level faults or if the client keeps
using a destroyed link.
## How?
Refuse link as per figure 2.33
Sometimes you want a plugin to act as a library
and not an application. That is, for its modules
to be available at compile time and on a running
node but, say, the actual runtime parameter
handling and supervision of shovels to be
handled by another plugin.
Since we do not currently have a concept of
"library plugins" or "library dependencies",
this approach demonstrates one example of how
some plugins can be used as libraries.
Then wait for elections to complete before shutting further
members down.
This should help avoid election storms when enabling maintenance
mode.
Transfer khepri before queues to ensure meta data store is
ready to accept pid updates.
Some other state related tweaks.
In some environments it may be tricky to run a command
before RabbitMQ is started. Kubernetes For example, upgrading
to Containerd 2.0 accidentally lowered the value of available
descriptors in GKE
(https://cloud.google.com/kubernetes-engine/docs/troubleshooting/known-issues#containerd-ulimit-reduced).
While this particular issue is resolved in newer GKE versions,
it was a good example of why a variable like this would be helpful.
Kubernetes-level solutions (on the node or with a daemonset) would
require a different level of access for users.
[Why]
When the module wanted to create an exchange in Khepri, it used
`rabbit_khepri:create/2` which ensures that the tree node doesn't exist
before. If the tree node exists, it expects it to contain an exchange
record which is returned to the caller, indicating an exchange with that
name already exists.
However, there are several other resources stored under an exchange tree
node, like bindings and topic permissions.
In particular, during a definitions import, topic permissions used to be
imported before exchanges. This caused a crash because the write of the
topic permission automatically created a parent tree node for the
exchange it dpends on, but without an exchange record obviously (see
previous commit).
[How]
As an addition improvement to the previous commit, we change the
conditions: instead of matching on the fact the tree node doesn't exist,
the module also accepts that the tree node exists but has no payload.
Under any of these conditions, the exchange is considered to be new and
written to Khepri.
[Why]
Topic permissions depend on an exchange, in addition to a user and a
vhost like other permissions.
This fixes a bug where an exchange imported after a topic permission
that depends on it caused the following crash when Khepri is used:
{case_clause,{error,{khepri,mismatching_node,
#{node_name => <<"exchange_name">>,
node_props => #{payload_version => 1},
node_path =>
[rabbitmq,vhosts,<<"/">>,exchanges,
<<"exchange_name">>],
condition => {if_node_exists,false},
node_is_target => true}}}}
The crash comes from the fact that the exchange code expect to either
create the tree node in Khepri for that exchange, or there is an
existing tree node holding an exchange tree node. Here, there was a tree
node created implicitly when the topic permission was stored, but that
tree node didn't have an exchange record (because the exchange was not
imported yet).
[How]
We simply swap the import of topic permissions and exchanges.
[Why]
The testcase tries to replicate the steps described in issue #12934.
[How]
It uses intermediate Erlang nodes between the common_test control node
and the RabbitMQ nodes, using `peer` standard_io communication. The goal
is to make sure the common_test control node doesn't interfere with the
nodes the RabbitMQ nodes can see, despite the blocking of the Erlang
distribution connection.
So far, I couldn't reproduce the problem reported in #12934. @mkuratczyk
couldn't either, so it might have been fixed as a side effect of another
change...
References #12934.
This is a follow-up to commit 93db480bc4
`erlang.mk` supports the `YRL_ERLC_OPTS` variable to set `erlc`-specific
compiler options when processing `.yrl` and `.xrl` files. By using this
variable, it allows `make RMQ_ERLC_OPTS=` to disable the
`+deterministic` option. This allows using `c()` in the erl shell to
recompile modules on the fly when a cluster is running.
You can see that, when `make RMQ_ERLC_OPTS=` is run, these generated
files were produced with the `+deterministic` option, because their
`-file` directives use only basenames.
* `deps/rabbit/src/rabbit_amqp_sql_lexer.erl`
* `deps/rabbit/src/rabbit_amqp_sql_parser.erl`
```
-file("rabbit_amqp_sql_parser.yrl", 0).
-module(rabbit_amqp_sql_parser).
-file("rabbit_amqp_sql_parser.erl", 3).
-export([parse/1, parse_and_scan/1, format_error/1]).
-file("rabbit_amqp_sql_parser.yrl", 122).
```
This commit also ignores those two files, as they will always be
auto-generated.
[Why]
If a user configures an auth backend module, but doesn't enabled the
plugin that provides it, it will get a crash and a stacktrace when
authentication is performed. The error is not helpful to understand what
the problem is.
[How]
We add a boot step that go through the configured auth backends and
query the core of RabbitMQ and the plugins. If an auth backend is
provided by a plugin, the plugin must be enabled to consider the auth
backend to be valid.
In the end, at least one auth backend must be valid, otherwise the boot
is aborted.
If only some of the configured auth backends were filtered out, but
there are still some valid auth backends, we store the filtered list in
the application environment variable so that
authentication/authorization doesn't try to use them later.
We also report invalid auth backends in the logs:
* Info message for a single invalid auth backend:
[info] <0.213.0> The `rabbit_auth_backend_ldap` auth backend module is configured. However, the `rabbitmq_auth_backend_ldap` plugin must be enabled in order to use this auth backend. Until then it will be skipped during authentication/authorization
* Warning message when some auth backends were filtered out:
[warning] <0.213.0> Some configured backends were dropped because their corresponding plugins are disabled. Please look at the info messages above to learn which plugin(s) should be enabled. Here is the list of auth backends kept after filering:
[warning] <0.213.0> [rabbit_auth_backend_internal]
* Error message when no auth backends are valid:
[error] <0.213.0> None of the configured auth backends are usable because their corresponding plugins were not enabled. Please look at the info messages above to learn which plugin(s) should be enabled.
V2: In fact, `rabbit_plugins:is_enabled/1` indicates if a plugin is
running, not if it is enabled... The new check runs as a boot step
and thus is executed before plugins are started. Therefore we can't
use this API. Instead, we use `rabbit_plugins:enabled_plugins/0'
which lists explicitly enabled plugins. The drawback is that in the
auth backend is enabled implicitly because it is a dependency of
another explicitly enabled plugin, the check will still consider it
is disabled and thus abort the boot.
Fixes#13783.
[Why]
This will be used in a later commit to find the auth backend plugin that
provides a configured auth backend module.
[How]
We go through the list of available plugins, regardless if they are
enabled or not, then look up the given module in the list of modules
associated with each plugin's application.
... without having to pass a plugins path.
[Why]
It's painful to have to get the plugins path, then pass it to `list/1`
every time. It's also more difficult to discover how to use
`rabbit_plugins` to get that list of plugins.
The code path in question is executed every time
rabbit_plugins:list/2 (e.g. rabbit_plugins:is_enabled/1)
is used, which with some distributed plugins can
happen once or several times a minute.
Given the maturity of the plugins subsystem, we
arguably can drop those messages.
This elimiantes a race condition between the destination
granting the sender link credit and the rest of what
the test does.
Note: the amqp_utils module in server core cannot be easily
moved to, say, rabbit_ct_helpers because it combines
two kinds of helpers that belong to two of our
CT helper subprojects.
So we've copied two small functions from it for
the needs of this suite.
This implementation intentionally ignores the
drain property of AMQP 1.0 flow control because
local shovels does not use this feature, and no
external component can "enable" it, so we can
simply ignore it.
Pair: @dcorbacho.
Co-authored-by: Diana Parra Corbacho <diana.parra-corbacho@broadcom.com>
Co-authored-by: Michael Klishin <michaelklishin@icloud.com>
This version forces prefixed binaries
(such as encrypted:TkQbjiVWtUJw3Ed/hkJ5JIsFIyhruKII6uKPXogfvDyMXGH1qQK3hVqshFolLN0S)
to have alphanumeric prefixes ([a-zA-Z0-9_]+).
This allows us to tell a generated password value
with a colon from an tagged binary.
If a value of, say, default_pass or ssl_options.password
cannot be parsed as a tagged value, it will be
parsed as a regular binary, because rabbit.schema
specifies multiple types as supported.
References #14233.
The rabbitmq_stream.advertised_tls_host setting is not used in the
metadata frame of the stream protocol, even if it is set. This commit
makes sure the setting is used if set.
References rabbitmq/rabbitmq-stream-java-client#803
[Why]
I noticed the following error in a test case:
error sending frame
Traceback (most recent call last):
File "/home/runner/work/rabbitmq-server/rabbitmq-server/deps/rabbitmq_stomp/test/python_SUITE_data/src/deps/stomp/transport.py", line 623, in send
self.socket.sendall(encoded_frame)
OSError: [Errno 9] Bad file descriptor
When the test suite succeeds, this error is not present. When it failed,
it was present. But I checked only one instance of each, it's not enough
to draw any conclusion about the relationship between this error and the
failing test case later.
I have no idea which test case hits this error, so increase the
verbosity, in the hope we see the name of the test case running at the
time of this error.
[Why]
I still don't know what causes the transient failures in this testsuite.
The AMQP connection is closed asynchronously, therefore the next test
case is running when it finishes to close. I have no idea if it causes
troubles, but it makes the broker logs more difficult to read.
[Why]
The `test_topic_dest` test case fails from time to time in CI. I don't
know why as there are no errors logged anywhere. Let's assume it's a
timeout a bit too short.
While here, apply the same change to `test_exchange_dest`.
[Why]
`gen_tcp:close/1` simply closes the connection and doesn't wait for the
broker to handle it. This sometimes causes the next test to fail
because, in addition to that test's new connection, there is still the
previous one's process still around waiting for the broker to notice the
close.
[How]
We now wait for the connection to be closed at the end of a test case,
and wait for the connection list to have a single element when we want
to query the connnection name.
[Why]
The connection is about to be killed at the end of the test case. It's
not necessary to close it explicitly.
Moreover, on a slow environment like CI, the connection process might
have already exited when the test case tries to close it. In this case,
it fails with a `noproc` exception.
... when testing user limits
[How]
This is the same fix as the one for the vhost limits test case made in
commit 5aab965db4.
While here, fix a compiler warning about an unused variable.
[Why]
Relying on the return value of the queue deletion is fragile because the
policy is cleared asynchronously.
[How]
We now wait for the queues to reach the expected queue length, then we
delete them and ensure the length didn't change.
[Why]
Before this change, when the `idle_time_out_on_server/1` test case was runned first in the
shuffled test group, the test module was not loaded on the remote broker.
When the anonymous function was passed to meck and was executed, we got
the following crash on the broker:
crasher:
initial call: rabbit_heartbeat:'-heartbeater/2-fun-0-'/0
pid: <0.704.0>
registered_name: []
exception error: {undef,
[{#Fun<amqp_client_SUITE.14.116163631>,
[#Port<0.45>,[recv_oct]],
[]},
{rabbit_heartbeat,get_sock_stats,3,
[{file,"rabbit_heartbeat.erl"},{line,175}]},
{rabbit_heartbeat,heartbeater,3,
[{file,"rabbit_heartbeat.erl"},{line,155}]},
{proc_lib,init_p,3,
[{file,"proc_lib.erl"},{line,317}]},
{rabbit_net,getstat,[#Port<0.45>,[recv_oct]],[]}]}
This led to a failure of the test case later, when it waited for a
message from the connecrtion.
We do the same in two other test cases where this is likely to happen
too.
[How]
Loading the module first fixes the problem.
[Why]
Maven took ages to fetch dependencies at least once in CI. The testsuite
failed because it reached the time trap limit.
[How]
Increase it from 2 to 5 minutes.
[Why]
The `rabbit_consistent_hash_exchange_raft_based_metadata_store` does not
seem to be a feature flag that ever existed according to the git
history. This causes the test case to always be skipped.
[How]
Simply remove the statement that enables this ghost feature flag.
[Why]
In CI, we observe that the channel hangs sometimes.
rabbitmq_ct_client_helpers implicit connection is quite fragile in the
sense that a test case can disturb the next one in some cases.
[How]
Let's use a dedicated connection and see if it fixes the problem.
[Why]
The `stream_pub_sub_metrics` test failed at least once in CI because the
`rabbitmq_stream_consumer_max_offset_lag` was 4 instead of the expected
3 on line 815.
I couldn't reproduce the problem so far.
[How]
The test case now logs the initial value of that metric at the beginning
of the test function. Hopefully this will give us some clue for the day
it fails again.
[Why]
In CI, we observed failures where the sender runs out of credits and
don't expect that.
[How]
The `amqp_utils:send_messages/3` function already takes care of that.
Move this logic to a `send_message/2` function and use it in
`send_messages/3` and prevriously direct uses of
`amqp10_client:send_msg/2`.
[Why]
In CI, we sometimes observe two tracked connections in the return value.
I don't know yet what they are. Could it be a client that reopened its
crashed connection and because stats are updated asynchronously, we get
two tracked connections for a short period of time?
[Why]
In CI, we sometimes observe two tracked connections in the return value.
I don't know yet what they are. Could it be a client that reopened its
crashed connection and because stats are updated asynchronously, we get
two tracked connections for a short period of time?
[Why]
This doesn't replicate the common_test logs layout, but it will be good
enough to let our GitHub Actions workflow to upload the logs without
specific instructions in the workflow.
[Why]
If we use the list of reachable nodes, it includes nodes which are
currently booting. Trying to start vhost during their start can disturb
their initialization and has a great chance to fail anyway.
Add more tests for the Direct Reply-to feature in AMQP 0.9.1.
This will help the future Direct Reply-To refactoring making sure the
existing behaviour won't break.
Retry unregistering a stream from its group in case of stream
coordinator timeout/unavailability. The operation can fail during or
after a network partition, which is normally, but it is harmless to
retry it to clean up the SAC group. The operation is idempotent anyway.
Sometimes a plugin needs to list online peers
that are running, reachable, not under maintenance
and have a specific plugin enabled.
This commit introduces a few helper functions
to make such cluster member queries trivial.
This commit fixes the following test flake that occurred in CI:
```
make -C deps/rabbit ct-amqp_dotnet t=cluster_size_1:redelivery
```
After receiving the end frame, the server session proc replies with the end frame.
Usually when the test case succeeds, the server connection process receives
a DOWN for the session proc and untracks its channel number such that a
subsequent begin frame for the same channel number will create a new session
proc in the server.
In the flake however, the client receives the end, and pipelines new begin,
attach, and flow frames. These frames are received in the server connection's
mailbox before the monitor for the old session proc fires. That's why these
new frames are sent to the old session proc causing the test case to
fail.
This reveals a bug in the server.
This commit fixes this bug similarly as done in the AMQP 0.9.1 channel in
94b4a6aafd/deps/rabbit/src/rabbit_channel.erl (L1146-L1155)
Channel reuse by the client is valid and actually common, e.g. if channel-max
is 0.
These functions will be used in the child commit for a check on
the number of exchanges. We can use the projection to avoid bothering
the Khepri process with a query.
[Why]
In CI, we sometimes get a failure when we try to forget node 3. The
CLI doesn't report the nature of the error unfortunately.
I suppose it's related to the fact that node 3 is stopped and forgotten
before all three replicas were ready when the stream queue was declared.
This is just a guess though and have no proof that it is the actual
error.
[How]
We wait for the replicas after declaring the stream queue.
[Why]
Sometimes it returns `false` in CI. `meck:validate/1` can return false
in the module throws an exception. So perhaps a timing issue in CI where
the runner is usually slower than our working computers?
[Why]
Sometimes, at least in CI, it looks like the output of the CLI is
prepended with a newline, sometimes not. This breaks the check of that
output.
[How]
We just trim the output before parsing it. The parsing already takes
care of trimming internal whitespaces.
Prior to this commit, the following test case flaked:
```
make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_1:session_upgrade_v3_v5_qos1
```
The test case failed with:
```
{v5_SUITE,session_upgrade_v3_v5_qos,1112}
{test_case_failed,Received unexpected PUBLISH payload. Expected: <<"2">> Got: <<"1">>}
```
The broker logs showed:
```
2025-07-15 15:50:23.914152+00:00 [debug] <0.758.0> MQTT accepting TCP connection <0.758.0> (127.0.0.1:38594 -> 127.0.0.1:27005)
2025-07-15 15:50:23.914289+00:00 [debug] <0.758.0> Received a CONNECT, client ID: session_upgrade_v3_v5_qos, username: undefined, clean start: false, protocol version: 3, keepalive: 60, property names: []
2025-07-15 15:50:23.914403+00:00 [debug] <0.758.0> MQTT connection 127.0.0.1:38594 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2025-07-15 15:50:23.914480+00:00 [debug] <0.758.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2025-07-15 15:50:23.914641+00:00 [info] <0.758.0> Accepted MQTT connection 127.0.0.1:38594 -> 127.0.0.1:27005 for client ID session_upgrade_v3_v5_qos
2025-07-15 15:50:23.914977+00:00 [debug] <0.758.0> Received a SUBSCRIBE with subscription(s) [{mqtt_subscription,
2025-07-15 15:50:23.914977+00:00 [debug] <0.758.0> <<"session_upgrade_v3_v5_qos">>,
2025-07-15 15:50:23.914977+00:00 [debug] <0.758.0> {mqtt_subscription_opts,1,false,
2025-07-15 15:50:23.914977+00:00 [debug] <0.758.0> false,0,undefined}}]
2025-07-15 15:50:23.924503+00:00 [debug] <0.764.0> MQTT accepting TCP connection <0.764.0> (127.0.0.1:38608 -> 127.0.0.1:27005)
2025-07-15 15:50:23.924922+00:00 [debug] <0.764.0> Received a CONNECT, client ID: session_upgrade_v3_v5_qos, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: []
2025-07-15 15:50:23.925589+00:00 [error] <0.758.0> writing to MQTT socket #Port<0.63> failed: closed
2025-07-15 15:50:23.925635+00:00 [debug] <0.764.0> MQTT connection 127.0.0.1:38608 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2025-07-15 15:50:23.925670+00:00 [info] <0.758.0> MQTT connection <<"127.0.0.1:38594 -> 127.0.0.1:27005">> will terminate because peer closed TCP connection
2025-07-15 15:50:23.925727+00:00 [debug] <0.764.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2025-07-15 15:50:24.000790+00:00 [info] <0.764.0> Accepted MQTT connection 127.0.0.1:38608 -> 127.0.0.1:27005 for client ID session_upgrade_v3_v5_qos
2025-07-15 15:50:24.016553+00:00 [warning] <0.764.0> MQTT disconnecting client <<"127.0.0.1:38608 -> 127.0.0.1:27005">> with client ID 'session_upgrade_v3_v5_qos', reason: normal
```
This shows evidence that the MQTT server connection did not process the
DISCONNECT packet. The hypothesis is that the server connection did not
even process the PUBACK packet from the client. Hence, the first message
got requeued and re-delivered to the new v5 client.
This commit fixes this flake by not acking the first message.
Hence, we always expect that the first message will be redelivered to
the new v5 client.
... when testing vhost limits
[Why]
The tracking is aynchronous, thus the third MQTT connection might be
opened before the tracking is up-to-date, which the testcase doesn't
expect.
[Why]
In the `node_channel_limit` testcase, we open several channels and
verify the count of opened channels in all places but one: after the
first connection failure, when we try to open 3 channels.
Opening 3 channels in a row might not be tracked in time to reject the
third channel because the counter is updated asynchronously.
[How]
We simply wait for the counter to reach 5 before opening the third
channel.
We change all checks to use `?awaitMatch/3` in the process to be more
robust with timing issues.
[Why]
ehie flaked today since the restart took 309ms, thus above
the allowed 100ms (outside of CI, it takes single-digit ms)
[How]
Increase the allowed time but also significantly increase next_seq_id.
This test exists because in the past we had an O(n) algorithm in CQ
recovery, leading to a slow recovery of even empty queues, if they had
a very large next_seq_id. Now that this operation is O(1), a much larger
next_seq_id shouldn't affect the time it takes to run
this test, while accidentally re-introducing an O(n) algorithm should
fail this test consistently.
[Why]
It looks like `erlang_vm_dist_node_queue_size_bytes` is not always
present, even though other Erlang-specific metrics are present.
[How]
The goal is to ensure Erlang metrics are present in the output, so just
use another one that is likely to be there.
[Why]
The tests relied on `rabbit_ct_client_helpers` connection and channel
manager which doesn't seem to be robust. It causes more harm than helps
so far.
Hopefully, this will fix some test flakes in CI.
[Why]
The tests relied on `rabbit_ct_client_helpers` connection and channel
manager which doesn't seem to be robust. It causes more harm than helps
so far.
Hopefully, this will fix some test flakes in CI.
[Why]
This was the only place where a condition was checked once after a
connection close, instead of waiting for it to become true.
This caused some transient failures in CI when the connection tracking
took a bit of time to update and the check was performed before that.
... in `remove_node_when_seed_node_is_leader/1` and
`remove_node_when_seed_node_is_follower/1`.
[Why]
The check was performed after the partition so far. It was incorrect
because if a cluster change was not permitted at the time of the
partition, it would not be afterwards. Thus there was a race condition
here.
[How]
Now, the check is performed before the partition.
Thanks to this new approach, we are sure of the state of node A and
don't need the cass block near the end of the test cases.
This should fix some test flakes we see locally and in CI.
[Why]
The default checkpoint interval is 16384. Therefore with 20,000
messages published by the testcase, there is a chance a checkpoint is
created. This would hit an assertion in the testcase which expects no
checkpoints before it forces the creation of one. We see this happening
in CI. Not locally because the testcase runs fast enough.
[How]
The testcase now sends 10,000 messages. This is still a lot of messages
while staying under the default checkpoint interval.
[Why]
When `wait_for_messages_ready/3` returns, we are sure that the replicas
are in the expected state. However, the `#amqqueue{}` record is updated
in Khepri, we don't know when all Khepri store members will be
up-to-date.
It can happen that `Server0` is not up-to-date when we query that record
to get the list of replicass, leading to a test failure.
[How]
First, the check is moved to its own function is `queue_utils`.
Then, if Khepri is being used, we use a Khepri fence to ensure previous
operations were applied on the given server. This way, we get a
consistent view of the `#amqqueue{}` record and thus the list of
replicas.
Trigger a 4.2.x alpha release build / trigger_alpha_build (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 26) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 27) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 28) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Type check (1.18, 28) (push) Waiting to runDetails
Switch from ra_metrics to ra_counters
* Expose many more metrics (they are also up to date)
* Bump Seshat, Ra, Osiris, Prometheus.erl
* switch from proplists to maps
When validation fails for a policy parameter, the resulting popup can't
be read due to one extra binary encoding as well as code that escapes
HTML entites. Since the EJS template uses `<%= >` for the popup, it will
display the text as-is, and not render any HTML.
This test case tests that two links filtering from the same stream
are processed concurrently by the session if the stream contains
uncompressed sub batches.
## What?
If a receiver performs stream filtering with AMQP property filters or AMQP SQL filter
expressions, the following downsides can occur:
1. While the stream is being filtered, other links on the same
session are blocked
2. RabbitMQ sends messages late to the receiver
As an example, let's assume a receiver attaches to the start of a multi GB
stream providing a link credit of 2. Let's assume only the very first message
matches the filter. In this case, RabbitMQ scans the entire
stream without processing other links on the same session, and sends the matched
message only once the scan completed (after many seconds or even minutes).
Instead, we want other links to be processed concurrently and the
receiver might want to start processing the first matched message while
RabbitMQ continues filtering the stream.
This commit fixes these two downsides.
## How?
After a threshold of consecutively unmatched messages, the session
"pauses" filtering on that link temporarily by:
1. sending an Erlang message `resume_filtering` to itself, and
2. sending any matched messages to the receiver
Any other Erlang messages then have a chance to be processed by the
session before the filtering on that link is resumed by the
`resume_filtering` Erlang message.
Once the end of the stream is reached or link credit is exhausted, the
`credit_reply` will be returned from `rabbit_stream_queue` to
`rabbit_amqp_session`.
An alternative solution would be to use separate Erlang processes for
filtering links as they can be CPU bound and also block for disk I/O.
The deprecated feature flag `amqp_filter_set_bug` was introduced in
RabbitMQ 4.1 with phase `permitted_by_default`.
See https://github.com/rabbitmq/rabbitmq-server/pull/12415
This commit which will land in RabbitMQ 4.2 changes the phase to `denied_by_default`.
JMS:
> A string literal is enclosed in single quotes, with an included single quote represented
> by doubled single quote; for example, 'literal' and 'literal''s'.
AMQP SQL
https://docs.oasis-open.org/amqp/filtex/v1.0/csd01/filtex-v1.0-csd01.html#_Toc67929302
> A string constant is a string of arbitrary text consisting of any valid printable Unicode
> characters surrounded by single or double quotation marks. A quotation mark inside the
> string is represented by two consecutive quotation marks.
> string_constant ::= { ‘ | “ } <any> [<any>] { ‘ | “ }
Allow only upper case predefined selector literals and operator names.
The JMS spec states:
> Predefined selector literals and operator names are written here in upper case; however, they are case insensitive.
However, the AMQP SQL spec does not include such a statement.
The EBNF notation with single quotes ('AND') typically implies exact literal matching.
This commit follows the AMQP SQL's EBNF notation and therefore disallows
lower case predefined selector literals or operator names.
AMQP SQL spec:
"The left operand is of greater value than the right operand if:
... both operands are of type string or of type symbol (any combination is permitted) and the
lexicographical rank of the left operand is greater than the lexicographical rank of the right operand."
In contrast, in JMS:
"String [...] comparison is restricted to = and <>."
The spec is underspecified in what should happen if the left hand side
is a float. This commit decides that the result is undefined (unknown).
The spec is also underspecified in what should happen if either the left
hand side or the right hand side is negative. This commit decides to use
the behaviour of Erlang `rem`.
This commit fixes the following test case:
```
make -C deps/rabbit ct-amqp_dotnet t=cluster_size_1:fragmentation
```
Previously, the server sent a frame that was 1 byte too large.
This commit fixes the failing test cases
```
make -C deps/rabbit ct-msg_size_metrics t=tests:message_size
make -C deps/amqp10_client ct-system t=rabbitmq:roundtrip_large_messages
```
The second test case failed with:
```
flush {amqp10_event,
{connection,<0.193.0>,
{closed,
{framing_error,
<<"frame size (131073 bytes) > maximum frame size (131072 bytes)">>}}}}
```
This commit results in great performance improvements for AMQP 1.0.
Stream filtering via AMQP SQL Filters or AMQP Property Filters where the
broker reads messages from the stream into memory and the filter returns
false, i.e. messages are not sent to the client:
Before this commit: ~400,000 msgs/s
After this commit: ~500,000 msgs/s
There is also a ~10% increase in end-to-end throughput for normal
AMQP workloads, e.g. when sending to and receiving from a classic queue.
Prior to this commit, a lot of garbage was created leading to many minor
garbage collections.
This commit reduces the garbage being generated.
The new module amqp10_composite performs very well:
* Single tuple update setting multiple fields at once, sometimes done even in-place
without copying the tuple.
* The list of fields is passed into X registers, no need for any new
allocations.
On the serialisation side, this commit also generates less garbage by:
1. Avoiding tuple_to_list/1
2. Omitting trailing elements of the list that are null
This will also lead to fewer bytes per message sent on the wire and less
resource usage for the clients as they need to parse fewer fields.
The closing sequence must account for consumer update and metadata
update frames the broker sends when a consumer group changes and when a
stream is deleted.
Field #amqqueue.type_state can be undefined for queues declared on old
RabbitMQ versions before 3.8.0. Ensure `amqqueue:get_type_state/1`
always returns a map according to its type spec to make life of
calling code easier.
In order to clarify preference of different methods. This commit oes
not change functionality.
This highlights some inconsistences:
- If both User/Password and SslLogin are provided, in
`check_credentials` User/Password takes precedence while in
`get_vhost` SslLogin
- If SslLogin is provided (but no mapping is found) vhost from port
mapping has precedence over vhost from UserName, while in case of
no ssl it is the other way around.
This commit fixes several crashes:
1. Serialising IEEE 754-2008 decimals as well as
NaN and +-Inf for float and doubles crashed
2. Converting IEEE 754-2008 decimals as well as NaN and +-Inf
for float and dobules from amqp to amqpl crashed
The 2nd crash looks as follows:
```
exception exit: {function_clause,
[{mc_amqpl,to_091,
[<<"decimal-32">>,{as_is,116,<<124,0,0,0>>}],
[{file,"mc_amqpl.erl"},{line,747}]},
{mc_amqpl,'-convert_from/3-lc$^2/1-2-',1,
[{file,"mc_amqpl.erl"},{line,155}]},
{mc_amqpl,convert_from,3,
[{file,"mc_amqpl.erl"},{line,155}]},
{mc,convert,3,[{file,"mc.erl"},{line,358}]},
{rabbit_channel,outgoing_content,2,
[{file,"rabbit_channel.erl"},{line,2649}]},
{rabbit_channel,handle_basic_get,7,
[{file,"rabbit_channel.erl"},{line,2636}]},
{rabbit_channel,handle_cast,2,
[{file,"rabbit_channel.erl"},{line,617}]},
{gen_server2,handle_msg,2,
[{file,"gen_server2.erl"},{line,1056}]}]}
```
The 2nd crash is fixed by omitting any `{as_is, _TypeCode, _Binary}`
values during AMQP 1.0 -> AMQP 0.9.1 conversion.
This will be documented in the conversion table.
In addition to fixing these crashes, this commit adds tests that
RabbitMQ is able to store and forward IEEE 754-2008 decimals.
IEEE 754-2008 decimals can be parsed and serialsed by RabbitMQ.
However, RabbitMQ doesn't support interpreting this values. For example,
they can't be used on the headers exchange or for AMQP filter
expressions.
mixed versions tests being run between different RabbitMQ versions
with different OTP versions maybe sometimes intentionally test upgrades
with the same OTP version (eg. 3.13->4.2 with OTP26) so let's just skip
this test in such cases
[Why]
When node A becomes the leader, it still takes a few exchanges with
followers to allow cluster changes again.
Concurrently, the testcase blocks traffic between A and other nodes to
simulate a network partition. If this happens after A becomes the leader
but before cluster changes are permitted again, the testcase will never
succeed and will eventually abort with:
Case: cluster_minority_SUITE:remove_node_when_seed_node_is_leader
Reason: {error,
{{awaitMatch,
[{module,cluster_minority_SUITE},
{expression, "..."},
{pattern,"ok"},
{value,
{error,69,<<"Error:\ncluster_change_not_permitted">>}}]},
[How]
Before blocking traffic, we wait for cluster changes to be permitted
again on node A.
## What?
This commit displays effective filters of AMQP receivers in the Management UI.
There is a new column `Filters` for outgoing links.
Solves #13429
## Why?
This allows validating if the desired filters set by the receiver are
actually in place by the server.
In addition, it's convenient for a developer to check any filter values
including SQL filter expressions.
## How?
The session process stores the the formatted and effective filters in
its state.
The Management UI displays a box containing the filter name. This way
the table for the outgoing links is kept concise. Hovering with the
mouse over a box will show additionally the descriptor and the actual
filter-value/definition.
A higher priority SAC consumer was never activated when a
quiescing consumer returned or requeued it's last message.
NB: this required a new machine version: 7
Elixir 1.18 comes with a JSON package which leads to errors like this:
```
Duplicate modules: [["/home/runner/work/_temp/.setup-beam/elixir/bin/../lib/elixir/ebin/Elixir.JSON.Encoder.Float.beam",
"/home/runner/work/rabbitmq-server/rabbitmq-server/deps/json/ebin/Elixir.JSON.Encoder.Float.beam"],
```
Test case two_nodes_different_otp_version was introduced in
https://github.com/rabbitmq/rabbitmq-server/pull/14042.
In CI, the code was compiled on OTP 27. The test case then applied the
Ra commands on OTP 27 and OTP 26 at runtime. For that to work the test
transferred the compiled BEAM modules to the OTP 26 node.
Bumping OTP to 28 causes the lower version node (be it OTP 26 or OTP 27)
to error when parsing the atom table chunk of the BEAM file that was
compiled in OTP 28:
```
corrupt atom table
{error,badfile}
```
That's expected as described in https://github.com/erlang/otp/pull/8913#issue-2572291638
since https://github.com/erlang/otp/pull/8913 changes the atom table chunk
format in the BEAM files.
```
beam_lib:chunks("deps/rabbit/ebin/lqueue.beam", [atoms]).
```
will parse successfully if the file gets loaded on OTP 28 irrespective
of whether the file was compiled with OTP 27 or 28.
However, this file fails to load if it is compiled with 28 and loaded on
27.
There is the `no_long_atoms` option that we could use just for this test
case.
However, given that we have a similar test case
two_nodes_same_otp_version we skip test case two_nodes_different_otp_version
in this commit.
Really, the best solution would be to use different OTP versions in
RabbitMQ mixed version testing in addition to using different RabbitMQ
versions. This way, the test case could just RPC into the different
RabbitMQ nodes and apply the Ra commands there.
Without this change delete/3 callback for custom exchange type wasn't called and warning about invalid exchange type appeared in the logs.
What is interesting that exchange module was logged as an invalid type for previously declared exchange of that module ¯\_(ツ)_/¯.
## What?
This commit allows AMQP 1.0 clients to define SQL-like filter expressions when consuming from streams, enabling server-side message filtering.
RabbitMQ will only dispatch messages that match the provided filter expression, reducing network traffic and client-side processing overhead.
SQL filter expressions are a more powerful alternative to the [AMQP Property Filter Expressions](https://www.rabbitmq.com/blog/2024/12/13/amqp-filter-expressions) introduced in RabbitMQ 4.1.
SQL filter expressions are based on the [JMS message selector syntax](https://jakarta.ee/specifications/messaging/3.1/jakarta-messaging-spec-3.1#message-selector-syntax) and support:
* Comparison operators (`=`, `<>`, `>`, `<`, `>=`, `<=`)
* Logical operators (`AND`, `OR`, `NOT`)
* Arithmetic operators (`+`, `-`, `*`, `/`)
* Special operators (`BETWEEN`, `LIKE`, `IN`, `IS NULL`)
* Access to the properties and application-properties sections
**Examples**
Simple expression:
```sql
header.priority > 4
```
Complex expression:
```sql
order_type IN ('premium', 'express') AND
total_amount BETWEEN 100 AND 5000 AND
(customer_region LIKE 'EU-%' OR customer_region = 'US-CA') AND
properties.creation-time >= 1750772279000 AND
NOT cancelled
```
Like AMQP property filter expressions, SQL filter expressions can be
combined with Bloom filters. Combining both allows for highly customisable
expressions (SQL) and extremely fast evaluation (Bloom filter) if only a
subset of the chunks need to be read from disk.
## Why?
Compared to AMQP property filter expressions, SQL filter expressions provide the following advantage:
* High expressiveness and flexibility in defining the filter
Like for AMQP property filter expressions, the following advantages apply:
* No false positives (as is the case for Bloom filters)
* Multiple concurrent clients can attach to the same stream each consuming
only a specific subset of messages while preserving message order.
* Low network overhead as only messages that match the filter are
transferred to the client
* Likewise, lower resource usage (CPU and memory) on clients since they
don't need to deserialise messages that they are not interested in.
* If the SQL expression is simple, even the broker will save resources
because it doesn't need to serialse and send messages that the client
isn't interested in.
## How?
### JMS Message Selector Syntax vs. AMQP Extension Spec
The AMQP Filter Expressions Version 1.0 extension Working Draft 09 defines SQL Filter Expressions in Section 6.
This spec differs from the JMS message selector spec. Neither is a subset of the other. We can choose to follow either.
However, I think it makes most sense to follow the JMS spec because:
* The JMS spec is better defined
* The JMS spec is far more widespread than the AMQP Working Draft spec. (A slight variation of the AMQP Working
Draft is used by Azure Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-sql-filter)
* The JMS spec is mostly simpler (partly because matching on only simple types)
* This will allow for a single SQL parser in RabbitMQ for both AMQP clients consuming from a stream and possibly in future for JMS clients consuming from queues or topics.
<details>
<summary>AMQP extension spec vs JMS spec</summary>
AMQP
!= is synonym for <>
JMS
defines only <>
Conclusion
<> is sufficient
AMQP
Strings can be tested for “greater than”
“both operands are of type string or of type symbol (any combination is permitted) and
the lexicographical rank of the left operand is greater than the lexicographical rank of the right operand”
JMS
“String and Boolean comparison is restricted to = and <>.”
Conclusion
The JMS behaviour is sufficient.
AMQP
IN <set-expression>
set-expression can contain non-string literals
JMS:
set-expression can contain only string literals
Conclusion
The JMS behaviour is sufficient.
AMQP
EXISTS predicate to check for composite types
JMS
Only simple types
Conclusion
We want to match only for simple types, i.e. allowing matching only against values in the application-properties, properties sections and priority field of the header section.
AMQP:
Modulo operator %
Conclusion
JMS doesn't define the modulo operator. Let's start without it.
We can decide in future to add support since it can actually be useful, for example for two receivers who want to process every other message.
AMQP:
The ‘+’ operator can concatenate string and symbol values
Conclusion
Such string concatenation isn't defined in JMS. We don't need it.
AMQP:
Define NAN and INF
JMS:
“Approximate literals use the Java floating-point literal syntax.”
Examples include "7."
Conclusion
We can go with the JMS spec given that needs to be implemented anyway
for JMS support.
Scientific notations are supported in both the AMQP spec and JMS spec.
AMQP
String literals can be surrounded by single or double quotation marks
JMS
A string literal is enclosed in single quotes
Conclusion
Supporting single quotes is good enough.
AMQP
“A binary constant is a string of pairs of hexadecimal digits prefixed by ‘0x’ that are not enclosed in quotation marks”
Conclusion
JMS doesn't support binary constants. We can start without binary constants.
Matching against binary values are still supported if these binary values can be expressed as UTF-8 strings.
AMQP
Functions DATE, UTC, SUBSTRING, LOWER, UPPER, LEFT, RIGHT
Vendor specific functions
Conclusion
JMS doesn't define such functions. We can start without those functions.
AMQP
<field><array_element_reference>
<field>‘.’<composite_type_reference>
to access map and array elements
Conclusion
Same as above:
We want to match only for simple types, i.e. allowing matching only against values in the application-properties, properties sections and priority field of the header section.
AMQP
allows for delimited identifiers
JMS
Java identifier part characters
Conclusion
We can go with the Java identifiers extending the allowed characters by
`.` and `-` to reference field names such as `properties.group-id`.
JMS:
BETWEEN operator
Conclusion
The BETWEEN operator isn't supported in the AMQP spec. Let's support it as convenience since it's already available in JMS.
</details>
### Filter Name
The client provides a filter with name `sql-filter` instead of name
`jms-selector` to allow to differentiate between JMS clients and other
native AMQP 1.0 clients using SQL expressions. This way, we can also
optionally extend the SQL grammar in future.
### Identifiers
JMS message selectors allow identifiers to contain some well known JMS headers that match to well known AMQP fields, for example:
```erl
jms_header_to_amqp_field_name(<<"JMSDeliveryMode">>) -> durable;
jms_header_to_amqp_field_name(<<"JMSPriority">>) -> priority;
jms_header_to_amqp_field_name(<<"JMSMessageID">>) -> message_id;
jms_header_to_amqp_field_name(<<"JMSTimestamp">>) -> creation_time;
jms_header_to_amqp_field_name(<<"JMSCorrelationID">>) -> correlation_id;
jms_header_to_amqp_field_name(<<"JMSType">>) -> subject;
%% amqp-bindmap-jms-v1.0-wd10 § 3.2.2 JMS-defined ’JMSX’ Properties
jms_header_to_amqp_field_name(<<"JMSXUserID">>) -> user_id;
jms_header_to_amqp_field_name(<<"JMSXGroupID">>) -> group_id;
jms_header_to_amqp_field_name(<<"JMSXGroupSeq">>) -> group_sequence;
```
This commit does a similar matching for `header.` and `properties.` prefixed identifiers to field names in the AMQP property section.
The only field that is supported to filter on in the AMQP header section is `priority`, that is identifier `header.priority`.
By default, as described in the AMQP extension spec, if an identifier is not prefixed, it refers to a key in the application-properties section.
Hence, all identifiers prefixed with `header.`, and `properties.` have special meanings and MUST be avoided by applications unless they want to refer to those specific fields.
Azure Service Bus uses the `sys.` and `user.` prefixes for well known field names and arbitrary application-provided keys, respectively.
### SQL lexer, parser and evaluator
This commit implements the SQL lexer and parser in files rabbit_jms_selector_lexer.xrl and
rabbit_jms_selector_parser.yrl, respectively.
Advantages:
* Both the definitions in the lexer and the grammar in the parser are defined **declaratively**.
* In total, the entire SQL syntax and grammar is defined in only 240 lines.
* Therefore, lexer and parser are simple to maintain.
The idea of this commit is to use the same lexer and parser for native AMQP clients consumings
from streams (this commit) as for JMS clients (in the future).
All native AMQP client vs JMS client bits are then manipulated after
the Abstract Syntax Tree (AST) has been created by the parser.
For example, this commit transforms the AST specifically for native AMQP clients
by mapping `properties.` prefixed identifiers (field names) to atoms.
A JMS client's mapping from `JMS` prefixed headers can transform the AST
differently.
Likewise, this commit transforms the AST to compile a regex for complex LIKE
expressions when consuming from a stream while a future version
might not want to compile a regex when consuming from quorum queues.
Module `rabbit_jms_ast` provides such AST helper methods.
The lexer and parser are not performance critical as this work happens
upon receivers attaching to the stream.
The evaluator however is performance critical as message evaluation
happens on the hot path.
### LIKE expressions
The evaluator has been optimised to only compile a regex when necessary.
If the LIKE expression-value contains no wildcard or only a single `%`
wildcard, Erlang pattern matching is used as it's more efficient.
Since `_` can match any UTF-8 character, a regex will be compiled with
the `[unicode]` options.
### Filter errors
Any errors upon a receiver attaching to a stream causes the filter to
not become active. RabbitMQ will log a warning describing the reason and
will omit the named filter in its attach reply frame. The client lib is
responsible for detaching the link as explained in the AMQP spec:
> The receiving endpoint sets its desired filter, the sending endpoint sets the filter actually in place
(including any filters defaulted at the node). The receiving endpoint MUST check that the filter in
place meets its needs and take responsibility for detaching if it does not.
This applies to lexer and parser errors.
Errors during message evaluation will result in an unknown value.
Conditional operators on unknown are described in the JMS spec. If the
entire selector condition is unknown, the message does not match, and
will therefore not be delivered to the client.
## Clients
Support for passing the SQL expression from app to broker is provided by
the Java client in https://github.com/rabbitmq/rabbitmq-amqp-java-client/pull/216
If a stream or quorum queue has opened a file to read a consumer message
and the queue is deleted the file handle reference is lost and kept
open until the end of the channel lifetime.
The test creates network partitions and checks how the stream SAC
coordinator deals with them. It can be flaky on CI, the log statements
should help diagnose the flakiness.
On Windows the file may be in "DELETE PENDING" state following
its deletion (when the last message was acked). A subsequent
message leads us to writing to that file again but we can't
and get an {error,eacces}. In that case we wait 10ms and retry
up to 3 times.
Trigger a 4.2.x alpha release build / trigger_alpha_build (push) Waiting to runDetails
Test (make) / Build and Xref (1.17, 26) (push) Waiting to runDetails
Test (make) / Build and Xref (1.17, 27) (push) Waiting to runDetails
Test (make) / Test (1.17, 27, khepri) (push) Waiting to runDetails
Test (make) / Test (1.17, 27, mnesia) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.17, 27, khepri) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.17, 27, mnesia) (push) Waiting to runDetails
Test (make) / Type check (1.17, 27) (push) Waiting to runDetails
We've experienced lots of failures in CI:
```
GEN test/system_SUITE_data/apache-activemq-5.18.3-bin.tar.gz
make: *** [Makefile:65: test/system_SUITE_data/apache-activemq-5.18.3-bin.tar.gz] Error 28
make: Leaving directory '/home/runner/work/rabbitmq-server/rabbitmq-server/deps/amqp10_client'
Error: Process completed with exit code 2.
```
Bumping to the latest ActiveMQ Classic version may or may not help with
these failures.
Either way, we want to test against the latest ActiveMQ version. Version
5.18.3 reached end-of-life and is no longer maintained.
This commit handles edge cases in the stream SAC coordinator to make
sure it does not crash during execution. Most of these edge cases
consist in an inconsistent state, so there are very unlikely to happen.
This commit also makes sure there is no duplicate in the consumer list
of a group. Consumers are also now identified only by their connection
PID and their subscription ID, as now the timestamp they contain in
their state does not allow a field-by-field comparison.
New CLI command to trigger a rebalancing in a SAC group and activate a
consumer. This is a last resort solution if all consumers in a group
accidently end up in {connected, waiting} state.
The command re-uses an existing function, which only picks the consumer
that should be active. This means it does not try to "fix" the state
(e.g. removing a disconnected consumer because its node is definitely
gone from the cluster).
Fixes#14055
Calls to the stream SAC coordinator can fail for various reason
(e.g. a timeout because of a network partition). The stream reader does not
take into account what the SAC coordinator returns and moves on even
in case of errors. This can lead to inconsistent state for SAC groups.
This commit changes this behavior by handling unexpected errors from the
SAC coordinator and closing the connection. The client is expected to
reconnect. This is safer than risking inconsistent state.
Fixes#14040
The clean-up of a stream connection state when a stream member goes down can
remove subscriptions not affected by the member. The subscription state is
removed from the connection, but the subscription is not removed from
the SAC state (if the subscription is a SAC), because the subscription member
PID does not match the down member PID.
When the actual member of the subscription goes down, the subscription is no
longer part of the state, so the clean-up does not find the subscription
and does not remove it from the SAC state. This lets a ghost consumer in
the corresponding SAC group.
This commit makes sure only the affected subscriptions are removed from
the state when a stream member goes down.
Fixes#13961
A boolean status in the stream SAC coordinator is not enough to follow
the evolution of a consumer. For example a former active consumer that
is stepping down can go down before another consumer in the group is
activated, letting the coordinator expect an activation request that
will never arrive, leaving the group without any active consumer.
This commit introduces 3 status: active (formerly "true"), waiting
(formerly "false"), and deactivating. The coordinator will now know when
a deactivating consumer goes down and will trigger a rebalancing to
avoid a stuck group.
This commit also introduces a status related to the connectivity state
of a consumer. The possible values are: connected, disconnected, and
presumed_down. Consumers are by default connected, they can become
disconnected if the coordinator receives a down event with a
noconnection reason, meaning the node of the consumer has been
disconnected from the other nodes. Consumers can become connected again when
their node joins the other nodes again.
Disconnected consumers are still considered part of a group, as they are
expected to come back at some point. For example there is no rebalancing
in a group if the active consumer got disconnected.
The coordinator sets a timer when a disconnection occurs. When the timer
expires, corresponding disconnected consumers pass into the "presumed
down" state. At this point they are no longer considered part of their
respective group and are excluded from rebalancing decision. They are expected
to get removed from the group by the appropriate down event of a
monitor.
So the consumer status is now a tuple, e.g. {connected, active}. Note
this is an implementation detail: only the stream SAC coordinator deals with
the status of stream SAC consumers.
2 new configuration entries are introduced:
* rabbit.stream_sac_disconnected_timeout: this is the duration in ms of the
disconnected-to-forgotten timer.
* rabbit.stream_cmd_timeout: this is the timeout in ms to apply RA commands
in the coordinator. It used to be a fixed value of 30 seconds. The
default value is still the same. The setting has been introduced to
make integration tests faster.
Fixes#14070
This is simmilar to https://github.com/rabbitmq/rabbitmq-server/pull/14056.
The performance benefit is probably negligbile though since this is
called only after each batch of Ra commands.
Nevertheless, it's unnecessary to allocate a list with 3 elements and
therefore 6 words on the heap, so let's optimise it.
A stream may not have a leader temporarily for several reasons, e.g.
after it has been restarted. The stream manager may return undefined in
this case. Some client code may crash because it expects a PID or an
error, but not undefined.
This commit makes sure the leader PID is an actual Erlang PID and
returns {error, not_available} if it is not.
References #13962
[Why]
They make it more difficult to compile RabbitMQ on Windows. They were
probably useful at the time of the switch to a monorepository but I
don't see their need anymore.
Trigger a 4.2.x alpha release build / trigger_alpha_build (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 26) (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 27) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Type check (1.17, 27) (push) Has been cancelledDetails
Nightly OCI (make) / build-package-generic-unix (main, 27, 4.2.0) (push) Has been cancelledDetails
Nightly OCI (make) / build-package-generic-unix (v4.0.x, 27) (push) Has been cancelledDetails
Nightly OCI (make) / build-package-generic-unix (v4.1.x, 27) (push) Has been cancelledDetails
Nightly OCI (make) / build-and-push (main, 27) (push) Has been cancelledDetails
Nightly OCI (make) / build-and-push (v4.0.x, 27) (push) Has been cancelledDetails
Nightly OCI (make) / build-and-push (v4.1.x, 27) (push) Has been cancelledDetails
The correct format is:
```
-export(Functions).
```
ELP detected this malformed syntax.
Interestingly, prior to this commit, the functions were still exported:
```
rabbitmq_amqp_address:module_info(exports).
[{exchange,1},
{exchange,2},
{queue,1},
{module_info,0},
{module_info,1}]
```
The correct format is:
```
-export(Functions).
```
ELP detected this malformed syntax.
Interestingly, prior to this commit, the functions were still exported:
```
rabbitmq_amqp_address:module_info(exports).
[{exchange,1},
{exchange,2},
{queue,1},
{module_info,0},
{module_info,1}]
```
MQTT tests depend on a few plugins, which are just used in 1 or 2
suites each. These have caused issues in CI, triggering a bug in
rabbitmq_federation where the mirrored supervisor submits a transaction
while the cluster is being shut down. The transaction hangs and the
whole rabbitmq_mqtt job times out.
This bug has been addressed, however it is best to start just the required
plugins on each SUITE.
[Why]
Links are started by the plugins but put under the `rabbit` supervision
tree. The federation plugins supervision tree is empty unfortunately...
Links are stopped by a boot step executed by `rabbit`, as a concequence
of unregistering the plugins' parameters.
Unfortunately, links can be terminated if the channel, and implicitly
the connection stops. This happens when the `amqp_client` application
stops.
We end up with a race here:
* Because the federation plugins supervision trees are empty and the
application stop functions barely stop the pg group (which doesn't
terminate the group members), nothing waits for the links to stop.
Therefore, `rabbit` can stop `amqp_client' which is a dependency of
the federation plugins. Therefore, the links underlying channels and
connections are stopped.
* `rabbit` unregister the federation parameters, terminating the links.
The exchange links `terminate/2` function needs the channel to delete
the remote queue. But the channel and the underlying connection might
be gone.
This simply logs a `badmatch` exception:
[error] <0.884.0> Federation link could not create a disposable (one-off) channel due to an error error: {badmatch,
[error] <0.884.0> {error,
[error] <0.884.0> {noproc,
[error] <0.884.0> {gen_server,
[error] <0.884.0> call,
[error] <0.884.0> [<0.911.0>,
[error] <0.884.0> {command,
[error] <0.884.0> {open_channel,
[error] <0.884.0> none,
[error] <0.884.0> {amqp_selective_consumer,
[error] <0.884.0> []}}},
[error] <0.884.0> 130000]}}}}
[How]
The solution is to make sure links are stopped as part of the stop of
the plugins.
`rabbit_federation_pg:stop_scope/1` is expanded to stop all members of
all groups in this scope, before terminating the pg scope itself. The
new code waits for the stopped processes to exit.
We have to handle the `EXIT` signal in the link processes and change
their restart strategy in their parent supervisor from permanent to
transient. This ensures they are restarted only if they crash. This also
skips a error log message about each stopped link.
Trigger a 4.2.x alpha release build / trigger_alpha_build (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 26) (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 27) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Type check (1.17, 27) (push) Has been cancelledDetails
## What?
PR #13971 added a property test that applies the same quorum queue Raft
command on different quorum queue members on different Erlang nodes
ensuring that the state machine ends up in exaclty the same state.
The different Erlang nodes run the **same** Erlang/OTP version however.
This commit adds another property test where the different Erlang nodes
run **different** Erlang/OTP versions.
## Why?
This test allows spotting any non-determinism that could occur when
running quorum queue members in a mixed version cluster, where mixed
version means in our context different Erlang/OTP versions.
## How?
CI runs currently tests with Erlang 27.
This commit starts an Erlang 26 node in docker, specifically for the
`rabbit_fifo_prop_SUITE`.
Test case `two_nodes_different_otp_version` running Erlang 27 then transfers
a few Erlang modules (e.g. module `rabbit_fifo`) to the Erlang 26 node.
The test case then runs the Ra commands on its own node in Erlang 27 and
on the Erlang 26 node in Docker.
By default, this test case is skipped locally.
However, to run this test case locally, simply start an Erlang node as
follows:
```
erl -sname rabbit_fifo_prop@localhost
```
It's a middle ground between building on every commit, and not building
at all. We currently have a workflow to build the OCI on every PR.
However, building on every PR does not cover some use cases. For
example, providing an image to a user to preview some changes coming in
the next minor or patch. Another use case: compare main with a PR in
Kubernetes.
It's better to have a separate workflow, even at the expense of
duplication, because the "on schedule" trigger only runs on the default
branch of the repository. This "limitation" makes it complicated to
extend the current "build OCI on PRs" to also build nightly for main and
release branches.
For test case leader_locator_balanced the actual leaders elected were
nodes 1, 3, 1 because they know about machine version 6 while node 2
only knows about machine version 5.
This commit adds a property test that applies the same Ra commands in
the same order on two different Erlang nodes. The state in which both nodes end
up should be exactly the same.
Ideally, the two nodes should run different OTP versions because this
way we could test for any non-determinism across OTP versions.
However, for now, having a test with both nodes having the same OTP
verison is good enough because running this test with rabbit_fifo
machine version 5 fails while machine version 6 succeeds.
This reveales another interesting: The default "undefined" map order can
even be different using different Erlang nodes with the **same** OTP
version.
Prior to this commit map iteration order was undefined in quorum queues
and could therefore be different on different versions of Erlang/OTP.
Example:
OTP 26.2.5.3
```
Erlang/OTP 26 [erts-14.2.5.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]
Eshell V14.2.5.3 (press Ctrl+G to abort, type help(). for help)
1> maps:foreach(fun(K, _) -> io:format("~b,", [K]) end, maps:from_keys(lists:seq(1, 33), ok)).
4,25,8,1,23,10,7,9,11,12,28,24,13,3,18,29,26,22,19,2,33,21,32,20,17,30,14,5,6,27,16,31,15,ok
```
OTP 27.3.3
```
Erlang/OTP 27 [erts-15.2.6] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]
Eshell V15.2.6 (press Ctrl+G to abort, type help(). for help)
1> maps:foreach(fun(K, _) -> io:format("~b,", [K]) end, maps:from_keys(lists:seq(1, 33), ok)).
18,4,12,19,29,13,2,7,31,8,10,23,9,15,32,1,25,28,20,6,11,17,24,14,33,3,16,30,21,5,27,26,22,ok
```
This can lead to non-determinism on different members. For example, different
members could potentially return messages in a different order.
This commit introduces a new machine version fixing this bug.
[Why]
The retry logic I added in 4621fe7730
was completely wrong. If Khepri reached its own timeout of 30 seconds (as
of this writing), the mirrored supervisor would retry 50 times because
it would not check the time spent. This means it would retry for 25
minutes. Nice.
That retry would be terminated forcefully by the parent supervisor after
5 minutes if it was part of a shutdown.
[How]
This time, the code simply pass the error (timeout or something else)
down to the following `case`. It will shut the mirrored supervisor down.
This fixes very long RabbitMQ node termination (at least 5 minutes,
sometimes more) in testsuites. An example to reproduce:
gmake -C deps/rabbitmq_mqtt \
RABBITMQ_METADATA_STORE=khepri \
ct-v5 t=cluster_size_3:session_takeover_v3_v5
In this one, the third node of the cluster will take 5+ minutes to stop.
From #14014, we learned that the indentation was causing workflows to
not trigger. However, this did not seem to affect when the workflow file
itself was changed. In any case, YAML is sensible to indentation,
therefore this change is 'correct'. Removing single quotes from paths
with '*' at the end, because it is not required according to YAML and
GitHub documentation.
The path triggers now match the Selenium workflow that runs on commits
to main and release branches.
Queues are automatically declared for MQTT consumers, but they can be externally
deleted. The consumer should be disconnected in such case, because it has no way
of knowing this happened - from its perspective there are simply no messages to consume.
In RabbitMQ 3.11 the consumer was disconnected in such situation.
This behaviour changed with native MQTT, which doesn't use AMQP internally.
Test case `tcp_back_pressure_rabbitmq_internal_flow_quorum_queue` succeeds
consistently locally on macOS and fails consistently in CI since 30 May
2025.
CI also shows a test failure instance of `tcp_back_pressure_rabbitmq_internal_flow_classic_queue`, albeit much rearer.
This test case succeeds in CI when using ubuntu-22.04 but fails with ubuntu-24.04.
Even before 30 May 2025, ubuntu-24.04 was used. However the GitHub runner
version was updated from Version: 20250511.1.0 to Version: 20250527.1.0
which presumably started to cause this test to fail.
This hypothesis cannot be validated because the GitHub actions
definitions YAML file doesn't provide a means to configure this version.
File `images/ubuntu/Ubuntu2404-Readme.md` in https://github.com/actions/runner-images/compare/ubuntu24/20250511.1...ubuntu24/20250527.1 shows the diff.
The most notable changes are probably the kernel version change from Kernel Version: 6.11.0-1013-azure to Kernel Version: 6.11.0-1015-azure and some changes to file `images/ubuntu/scripts/build/configure-environment.sh`
There seem to be no RabbitMQ related changes causing this test to fail
because this test also fails with an older RabbitMQ version with the new runner
Version: 20250527.1.0.
Neither `meck` nor `inet:setopts(Socket, [{active, once}])` cause the
test failure because the test also fails with the former
`erlang:suspend_process/1` and `erlang:resume_process/1`.
The test fails due to the following timeout in the writer proc on the
server:
```
** Last message in was {'$gen_cast',
{send_command,<0.760.0>,0,
{'v1_0.transfer',
{uint,3},
{uint,2211},
{binary,<<0,0,8,162>>},
{uint,0},
true,undefined,undefined,undefined,
undefined,undefined,undefined},
<<"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx">>}}
** When Server state == #{pending => 3510,socket => #Port<0.49>,
reader => <0.755.0>,
monitored_sessions => [<0.760.0>],
pending_size => 3510}
** Reason for termination ==
** {{writer,send_failed,timeout},
[{rabbit_amqp_writer,flush,1,
[{file,"src/rabbit_amqp_writer.erl"},{line,250}]},
{rabbit_amqp_writer,handle_cast,2,
[{file,"src/rabbit_amqp_writer.erl"},{line,106}]},
{gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,2371}]},
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,2433}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,329}]}]}
```
For unknown reasons, even after the CT test case resumes consumption,
the server still times out writing to the socket.
The most important test expectation that is kept in place is that the
server won't send all the messages if the client can't receive fast
enough.
Events can be received after the boot step but before the application
is started. Creating the ETS in the supervisor solves this, as it is
started just before the event handler is installed.
Trigger a 4.2.x alpha release build / trigger_alpha_build (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 26) (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 27) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Type check (1.17, 27) (push) Has been cancelledDetails
It was expensive to delete files because we had clean up
the index and to get the messages in the file we have to
scan it.
Instead of cleaning up the index on file delete this
commit deletes from the index as soon as possible.
There are two scenarios: messages that are removed
from the current write file, and messages that are
removed from other files. In the latter case, we
can just delete the index entry on remove. For messages
in the current write file, we want to keep the entry
in case fanout is used, because we don't want to write
the fanout message multiple times if we can avoid it.
So we keep track of removes in the current write file
and do a cleanup of these entries on file roll over.
Compared to the previous implementation we will no
longer increase the ref_count of messages that are
not in the current write file, meaning we may do more
writes in fanout scenarios. But at the same time the
file delete operation is much cheaper.
Additionally, we prioritise delete calls in rabbit_msg_store_gc.
Without that change, if the compaction was lagging behind,
we could have file deletion requests queued behind many compaction
requests, leading to many unnecessary compactions of files
that could already be deleted.
Co-authored-by: Michal Kuratczyk <michal.kuratczyk@broadcom.com>
This branch redirects the client to the login page when the cookie
expires. To complete the logout process we should also clear any auth
data stored in local storage: local storage has no built-in expiration
mechanism.
To test this locally you can use `make run-broker`, set the session
timeout to one minute for quick testing:
application:set_env(rabbitmq_management, login_session_timeout, 1)
go to the management page (`http://localhost:15672/#/`), login with
default credentials and wait a minute. After this change the local
storage only contains info like `rabbitmq.vhost` and `rabbitmq.version`.
Not only when it is removed explicitly by the client. This is necessary
to make sure the consumer record is removed from the management ETS
tables (consumer_stats) and to avoid ghost consumers.
For other protocols like AMQP 091, the consumer_status ETS table is
cleaned up when a channel goes down, but there is no channel concept in
the stream protocol.
This is not consistent with other protocols or queue implementations
(which emits the event only on explicit consumer cancellation)
but is necessary to clean up stats correctly.
References #13092
Helps cleaning-up/coloring stdout for parallel targets
TODO: there are obvious races for different nodes outputs
In the next iteration I hope to implement cursor tracking for each node
As a follow-up to my GChat thread about removing default logger handler to clean CT stdout, I was looking at
injecting logger config with undefined default handler to ct_run. It is possible but breaks cth_styledout - no
nice green things whatsoever. Then I found rabbit_ct_hook which calls redirect_logger_to_ct_logs which in turn
calls logger:remove_handler(default) apparently with zero effect! To cut story short - turned out rabbit_ct_hook
must run before cth_styledout for remove_handler line to have any effect
This is mostly the same as the `messages_total` property test but checks
that the Raft indexes in `ra_indexes` are the set of the indexes checked
out by all consumers union any indexes in the `returns` queue. This is
the intended state of `ra_indexes` and failing this condition could
cause bugs that would prevent snapshotting.
The AMQP spec defines:
```
<field name="durable" type="boolean" default="false"/>
```
RabbitMQ 4.0 and 4.1 interpret the durable field as true if not set.
The idea was to favour safety over performance.
This complies with the AMQP spec because the spec allows other target or
node specific defaults for the durable field:
> If the header section is omitted the receiver MUST assume the appropriate
> default values (or the meaning implied by no value being set) for the fields
> within the header unless other target or node specific defaults have otherwise
> been set.
However, some client libraries completely omit the header section if the
app expliclity sets durable=false. This complies with the spec, but it
means that RabbitMQ cannot diffentiate between "client app forgot to set
the durable field" vs "client lib opted in for an optimisation omitting
the header section".
This is problematic with JMS message selectors where JMS apps can filter
on JMSDeliveryMode. To be able to correctly filter on JMSDeliveryMode,
RabbitMQ needs to know whether the JMS app sent the message as
PERSISTENT or NON_PERSISTENT.
Rather than relying on client libs to always send the header section
including the durable field, this commit makes RabbitMQ comply with the
default value for durable in the AMQP spec.
Some client lib maintainers accepted to send the header section, while
other maintainers refused to do so:
https://github.com/Azure/go-amqp/issues/330https://issues.apache.org/jira/browse/QPIDJMS-608
Likely the AMQP spec was designed to omit the header section when
performance is important, as is the case with durable=false. Omitting
the header section means saving a few bytes per message on the wire and
some marshalling and unmarshalling overhead on both client and server.
Therefore, it's better to push the "safe by default" behaviour from the broker
back to the client libs. Client libs should send messages as durable by
default unless the client app expliclity opts in to send messages as
non-durable. This is also what JMS does: By default JMS apps send
messages as PERSISTENT:
> The message producer's default delivery mode is PERSISTENT.
Therefore, this commit also makes the AMQP Erlang client send messages as
durable, by default.
This commit will apply to RabbitMQ 4.2.
It's arguably not a breaking change because in RabbitMQ, message durability
is actually more determined by the queue type the message is sent to rather than the
durable field of the message:
* Quroum queues and streams store messages durably (fsync or replicate)
no matter what the durable field is
* MQTT QoS 0 queues hold messages in memory no matter what the
durable field is
* Classic queues do not fsync even if the durable field is set to true
In addition, the RabbitMQ AMQP Java library introduced in RabbitMQ 4.0 sends messages with
durable=true:
53e3dd6abb/src/main/java/com/rabbitmq/client/amqp/impl/AmqpPublisher.java (L91)
The tests for selecting messages by JMSDeliveryMode relying on the
behaviour in this commit can be found on the `jms` branch.
https://github.com/erlang/otp/issues/9739
In OTP28+, splitting an empty string returns an empty list, not an empty
string (the input).
Additionally `street-address` macro was removed in OTP28 - replace with
the value it used to be.
Lastly, rabbitmq_auth_backend_oauth2 has an MQTT test, so add
rabbitmq_mqtt to TEST_DEPS
revive_local_queue_replicas -> revive_local_queue_members
can_redeliver converted from callback to capabilities key
rebalance_moduled converted from callback to capabilities key
Building from source using this command:
```
make RMQ_ERLC_OPTS= FULL=1
```
... then starting RabbitMQ via `make run-broker`, allows re-compilation
from the erl shell:
```
1> c(rabbit).
Recompiling /home/lbakken/development/rabbitmq/rabbitmq-server/deps/rabbit/src/rabbit.erl
{ok,rabbit}
```
When `+deterministic` is passed to `erlc`, the `compile` data in each
modules' information is missing the source path for the module.
Follow-up to #3442
`sets` v2 were not yet available when this module was written. Compared
to `gb_sets`, v2 `sets` are faster and more memory efficient:
> List = lists:seq(1, 50_000).
> tprof:profile(sets, from_list, [List, [{version, 2}]], #{type => call_memory}).
****** Process <0.94.0> -- 100.00% of total ***
FUNCTION CALLS WORDS PER CALL [ %]
maps:from_keys/2 1 184335 184335.00 [100.00]
184335 [ 100.0]
ok
> tprof:profile(gb_sets, from_list, [List], #{type => call_memory}).
****** Process <0.97.0> -- 100.00% of total ***
FUNCTION CALLS WORDS PER CALL [ %]
lists:rumergel/3 1 2 2.00 [ 0.00]
gb_sets:from_ordset/1 1 3 3.00 [ 0.00]
lists:reverse/2 1 100000 100000.00 [16.76]
lists:usplit_1/5 49999 100002 2.00 [16.76]
gb_sets:balance_list_1/2 65535 396605 6.05 [66.48]
596612 [100.0]
The `rabbit_mgmt_gc` gen_server performs garbage collections
periodically. When doing so it can create potentially fairly large
terms, for example by creating a set out of
`rabbit_exchange:list_names/0`. With many exchanges, for example, the
process memory usage can climb steadily especially when the management
agent is mostly idle since `rabbit_mgmt_gc` won't hit enough reductions
to cause a full-sweep GC on itself. Since the process is only active
periodically (once every 2min by default) we can hibernate it to GC the
terms it created.
This can save a medium amount of memory in situations where there are
very many pieces of metadata (exchanges, vhosts, queues, etc.). For
example on an idle single-node broker with 50k exchanges,
`rabbit_mgmt_gc` can hover around 50MB before being naturally GC'd. With
this patch the process memory usage stays consistent between `start_gc`
timer messages at around 1KB.
The `below_node_connection_limit_test` and `ready_to_serve_clients_test`
cases could possibly flake because `is_quorum_critical_single_node_test`
uses the channel manager in `rabbit_ct_client_helpers` to open a
connection. This can cause the line
true = lists:all(fun(E) -> is_pid(E) end, Connections),
to fail to match. The last connection could have been rejected if the
channel manager kept its connection open, so instead of being a pid the
element would have been `{error, not_allowed}`.
With `rabbit_ct_client_helpers:close_channels_and_connection/2` we can
reset the connection manager and force it to close its connection.
Building on push to any branch is wasteful and unnecessary, because most
of built images are never used. The workflow dispatch trigger covers the
use case to build an image from the latest commit in a branch.
The use case to validate/QA a PR is now covered by on pull request
trigger. This trigger has a caveat: PRs from forks won't produce a
docker image.
Why?
Because PRs from forks do not inject rabbitmq-server secrets. This is a
security mechanism from GitHub, to protect repository secrets.
With this trigger is possible to QA/validate PRs from other Core team
members. Technically, anyone with 'write' access to our repo to push
branches.
Returning the connection limit and active count are not really necessary
for these checks. Instead of returning them in the response to the
health check we log a warning when the connection limit is exceeded.
Test (make) / Build and Xref (1.17, 26) (push) Has been cancelledDetails
Test (make) / Build and Xref (1.17, 27) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, khepri) (push) Has been cancelledDetails
Test (make) / Test mixed clusters (1.17, 27, mnesia) (push) Has been cancelledDetails
Test (make) / Type check (1.17, 27) (push) Has been cancelledDetails
Peer Discovery AWS Integration Test / Integration Test (push) Has been cancelledDetails
[skip ci] Bump com.google.googlejavaformat:google-java-format from 1.26.0 to 1.27.0 in /deps/rabbit/test/amqp_jms_SUITE_data in the dev-deps group across 1 directory
This is useful for a load balancer, for example, to be able to avoid
sending new connections to a node which is running and has listeners
bound to TCP ports but is being drained for maintenance.
This updates the health check for protocol listeners to accept a set of
protocols, comma-separated. The check only returns 200 OK when all
requested protocols have active listeners.
This is a minor change that avoids a cluster-wide query for active
listeners. The old code called `rabbit_networking:active_listeners/0`
and then filtered the results by ones available on the local node. This
caused an RPC and concatenation of all other cluster members' listeners
and then in the next line filtered down to local nodes. Equivalently we
can use `rabbit_networking:node_listeners(node())` which dumps a local
ETS table.
This is not a very impactful change but it's nice to keep the latency of
the health-check handlers low and reduce some unnecessary cluster noise.
At CQ startup variable_queue went through each seqid from 0 to
next_seq_id looking for the first message even if there were no
messages in the queue (no segment files).
In case of a clean shutdown the value next_seq_id is stored in
recovery terms. This value can be utilized by the queue index to
provide better seqid bounds in absence of segment files.
Before this patch starting an empty classic queue with next_seq_id =
100_000_000 used to take about 26 seconds. With this patch it takes
less than 1ms.
[Why]
They were moved from `rabbit` to `rabbit_common` several years ago to
solve an dependency issue because `amqp_client` depended on the file
handle cache. This is not the case anymore.
[How]
The modules are moved back to `rabbit`.
`rabbit_common` doesn't need to depend on `os_mon` anymore. `rabbit`
already depends on it, so no changes needed here.
`include/rabbit_memory.hrl` and some test cases are moved as well to
follow the `vm_memory_monitor` module.
Consumers with a same name, consuming from the same stream should have
the same partition index. This commit adds a check to enforce this rule
and make the subscription fail if it does not comply.
Fixes#13835
Vhosts that currently don't have their own default queue type, now
inherit it from the node configuration and store it in their metadata
going forward.
The correct place for the `default_queue_type` property
is inside the `metadata` block. However, right now we'd
always export the value outside of `metadata` AND only
export it inside `metadata`, if it was not `undefined`.
This value outside of `metadata` was just misleading:
if a user exported the definitins from a fresh node,
changed `classic` to `quorum` and imported such modified
values, the DQT would still be `classic`, because RMQ looks
for the value inside `metadata`. Just to make it more confusing,
if the DQT was changed successfully one way or another, the
value outside of `metadata` would reflect that
(it always shows the correct value, but is ignored on import).
rabbit_channel may use amqp_channel as the writer.
When terminating, rabbit_channel sends a `flush` message
to its writer. If amqp_channel is in use, that led to
a `function_clause` crash.
[skip ci] Bump com.google.code.gson:gson from 2.13.0 to 2.13.1 in /deps/rabbitmq_stream_management/test/http_SUITE_data in the dev-deps group across 1 directory
[Why]
The `rabbit_khepri` module grew during the work to add Khepri support to
RabbitMQ and while Khepri was itself written. The current code is
therefore unorganized.
[How]
This commit tries to change proxy functions to be close to their Khepri
equivalent.
The module continues to set non-default options for write functions. We
also add the variants that take an option map to be consistent and not
have to deal with that in the future.
Several legacy functions were removed, either because they were no
longer called or because they were replace by a regular Khepri call.
[Why]
The `rabbit_khepri` module grew during the work to add Khepri support to
RabbitMQ and while Khepri was itself written. The current code is
therefore unorganized.
[How]
This commit tries to sort the code that manages the setup of Khepri and
the functions tha deal with the Khepri cluster. It also groups functions
which provide support for CLI commands.
It also adds documentation to several functions.
Finally, when a node joins a cluster, we stop displaying the content of
the Khepri tree.
It's a tradeoff between building the map for each incoming and outgoing
message (now that there are also outgoing interceptors) vs increased
memory usage for the MQTT proc state.
Connecting with MQTT 5.0 and client ID "xxxxxxxx", the number of words
are 201 before this commit vs 235 after this commit as determined by:
```
S = sys:get_state(MQTTConnectionPid),
erts_debug:size(S).
```
Therefore, this commit requires 34 word * 8 bytes = 272 bytes more per MQTT
connection, that is 272 MB more for 1,000,000 MQTT connections.
1. Force the config for timestamp and routing node message interceptors
to be configured with the overwrite boolean() to avoid defining
multiple default values throughout the code.
2. Add type specs
3. Extend existing test case for new MQTT client ID interceptor
4. routing node and timestamp should only set the annotation for
incoming_message_interceptors group
5. Fix `rabbitmq.conf`.
Prior to this commit there were several issue:
a.) Setting the right configuration was too user unfriendly, e.g. the user has to set
```
message_interceptor.incoming.rabbit_mqtt_message_interceptor_client_id.annotation_key = x-opt-mqtt-client-id
```
just to enable the MQTT message interceptor.
b.) The code that parses was too difficult to understand
c.) MQTT plugin was setting the env for app rabbit, which is an anti-pattern
d.) disabling a plugin (e.g. MQTT), left its message interceptors still in place
This is now all fixed, the user sets the rabbitmq.conf as follows:
```
message_interceptors.incoming.set_header_timestamp.overwrite = true
message_interceptors.incoming.set_header_routing_node.overwrite = false
mqtt.message_interceptors.incoming.set_client_id_annotation.enabled = true
```
Note that the first two lines use the same format as for RabbitMQ 4.0
for backwards compatiblity. The last line (MQTT) follows a similar
pattern.
This commit enables users to provide custom message interceptor modules,
i.e. modules to process incoming and outgoing messages. The
`rabbit_message_interceptor` behaviour defines a `intercept/4` callback,
for those modules to implement.
Co-authored-by: Péter Gömöri <gomoripeti@users.noreply.github.com>
[Why]
The intent is to have it stable and enabled by default for new
deployment in RabbitMQ 4.1.x.
To prepare for this goal, it is time to mark the feature flag as stable
to let us iron out the library and its integration into RabbitMQ.
This is not a commitment at this stage: we will revisit this near the
beginning of the release cycle and commit to it or revert to
experimental.
[Why]
We used a 4.0.x snapshot so far because we needed RabbitMQ 4.0.x to use
khepri_mnesia_migration 0.7.2.
RabbitMQ 4.0.9 was released with this update of khepri_mnesia_migration,
thus we don't need the snapshot anymore.
Now that 4.1.0 is out, 4.0.x binary releases will
no longer be available publicly.
(cherry picked from commit cf98ba5a8c)
Conflicts:
.github/workflows/release-4.0.x-alphas.yaml
(cherry picked from commit 4dc2395d63)
Key changes:
- endpoint variable to handle scraping multiple endpoints
- message size panels (new metric in 4.1)
- panels at the top of the Overview dashboard should be more up to date
(they show the latest value)
- values should be accurate if multiple endpoints are scraped
(previously, many would be doubled)
- Nodes table shows fewer volumns and shows node uptime
When debug logging is enabled, we log something at each log level
to test if logs are emitted. I don't think this is particularly useful,
but it's certainly annoying, because I constatnly need to filter
out these logs when searching if any errors happened during tests.
This auth backend behaves the same as the internal backend provided in
the core broker, but it only accepts loopback connections. External
connection attempts will receive an error.
This ensures that quorum_queues shuts down _before_
coordination where khepri run inside.
Quorum queues depend on khepri so need to be shut down first.
[Why]
The feature flag enable function is called during the initial migration
or when a node is later added to a cluster.
In this latter situation, the cluster is already formed and the Mnesia
tables were already migrated. Syncing the cluster in this specific
situation might kick another node that is currently unreachable.
[How]
If the node running the enable function is already clustered, we skip
the cluster sync.
[Why]
All callers of `khepri_adv` and `khepri_tx_adv` need updates to handle
the now uniform return type of `khepri:node_props_map()` in Khepri
0.17.0.
[How]
We don't need any compatibility code to handle "either the old return
type or the new return type" from the khepri_adv API because the
translation is done entirely in the "client side" code in Khepri -
meaning that the return value from the Ra server is the same but it is
translated differently by the functions in `khepri_adv`.
However, we need to adapt transaction functions because they may be
executed on different versions of Khepri and the behaviour of
`khepri_tx_adv` can be different. To take the possible change of return
value format, we use the new `khepri_tx:does_api_comply_with/1` to know
what to expect.
[Why]
In Khepri 0.17.0, `khepri_cluster:locally_known_members/1` and
`khepri_cluster:locally_known_node/1` were replaced with
`khepri_cluster:members/2` and `khepri_cluster:nodes/2` with `favor` set
to `low_latency` - this matches the interface for queries in Khepri.
Move leader repair earlier in tick function to ensure more
timely update of meta data store record after leader change.
Also use RPC_TIMEOUT macro for metric/stats multicalls to improve
liveness when a node is connected but partitioned / frozen.
This should address crashes like this in (found in user's logs):
```
exception error: no case clause matching
[[{connection_details,[]},
{name,<<"10.0.13.41:50497 -> 10.2.230.128:5671 (1)">>},
{node,rabbit@foobar},
{number,1},
{user,<<"...">>},
{user_who_performed_action,<<"...">>},
{vhost,<<"/">>}],
[{connection_details,[]},
{name,<<"10.0.13.41:50142 -> 10.2.230.128:5671 (1)">>},
{node,rabbit@foobar},
{number,1},
{user,<<"...">>},
{user_who_performed_action,<<"...">>},
{vhost,<<"/">>}]]
in function rabbit_federation_mgmt:format/3 (rabbit_federation_mgmt.erl, line 100)
in call from rabbit_federation_mgmt:'-status/3-lc$^0/1-0-'/4 (rabbit_federation_mgmt.erl, line 89)
in call from rabbit_federation_mgmt:'-status/4-lc$^0/1-0-'/3 (rabbit_federation_mgmt.erl, line 82)
in call from rabbit_federation_mgmt:'-status/4-lc$^0/1-0-'/3 (rabbit_federation_mgmt.erl, line 82)
in call from rabbit_federation_mgmt:status/4 (rabbit_federation_mgmt.erl, line 82)
in call from rabbit_federation_mgmt:to_json/2 (rabbit_federation_mgmt.erl, line 57)
in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1590)
in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1473)
```
## What?
This commit determines the queue topology without checking the queue type.
## Why?
This way, checking leader and replicas works the same across all queue
types without the need to introduce other rabbit_queue_type behaviour as
suggested in other PRs.
## How?
pid is the leader, nodes in queue_type_states are the members/replicas.
This commit results in an unknown stream leader during queue
declaration. However the correct leader will be returned eventually when
calling GET on the stream.
This allows restricting access to the /api/index.html and
the /cli/index.html page to authenticated users should the
user really want to. This can be enabled via advanced.config.
A connection which terminated before it was fully established
would lead to a function_clause, since metadata is not available
to really call notify_connection_closed. We can just ignore such
connections and not notify about them.
Resolves https://github.com/rabbitmq/rabbitmq-server/discussions/13670
What's Changed
* Allow force shrink to non-voter member
* ra_server_proc: Handle aux_command in all Raft states
* Increase shutdown timeout for segment writer.
* Avoid modification checks when reading sparse entries inside the Ra process
The same connection can contain several consumers belonging to a SAC
group (group key = vhost + stream + consumer name). The whole new group
must be re-evaluated to select a new active consumer after the consumers
of the down connection are removed from it.
The previous behavior would not re-evaluate the new group and could
select a consumer from the down connection, letting the group with only
inactive consumers, as the selected active consumer would never receive
the activation message from the stream SAC coordinator.
This commit fixes this problem by removing the consumers of the down
down connection from the affected groups and then performing the
appropriate operations for the groups to keep on consuming (e.g.
notifying an active consumer that it needs to step down).
References #13372
[skip ci] Bump com.google.googlejavaformat:google-java-format from 1.25.2 to 1.26.0 in /deps/rabbit/test/amqp_jms_SUITE_data in the dev-deps group across 1 directory
Ra improvements:
* Don't allow a non-voter to start elections
* Register with ra directory before initialising ra server.
* Trigger tick_timeout immediately after entering leader state.
* Set a configurable segment max size
This commit also includes a change to turn the quorum queue
become leader callback to become a noop and instead rely on
the more promptly tick_handler to handle the meta data store
update after a leader election.
This more prompt tick update means there should be a much shorter
gap between the queue metrics being deleted from the old leader
node to them being available again on the new node resulting
in smoother message count metrics.
Fix test that relied on waiting on too simplistic a property
before asserting.
Prior to this commit, when a client consumed from an unavailable quorum
queue, the following crash occurred:
```
{badmatch,{error,noproc}}
[{rabbit_quorum_queue,consume,3,[{file,\"rabbit_quorum_queue.erl\"},{line,993}]}
```
This commit fixes this bug by returning any error when registering a
quorum queue consumer to rabbit_queue_type.
This commit also refactors errors returned by
rabbit_queue_type:consume/3 to simplify and ensure seperation of
concerns.
For example prior to this commit, the channel did error
formatting specifically for consuming from streams. It's better if
the channel is unaware of what queue type it consumes from and have each
queue type implementation format their own errors.
Bump the timeout for management operations and link attachments from 20s
to 30s. We've seen timeouts in CI.
We bump the poll interval of the `?awaitMatch` macro because CI
sometimes flaked by crashing in
0e803de6dd/deps/rabbitmq_amqp_client/src/rabbitmq_amqp_client.erl (L411)
which indicates that the client lib received a response from a previous
request.
To take more frequent checkpoints for large message workload
Lower the min_checkpoint_interval substantially to allow quorum queues
better control over when checkpoints are taken.
Track bytes enqueued in the aux state and suggest a checkpoint after
every 64MB enqueued (this value is scaled according to backlog just
like the indexes condition).
This should help with more timely checkpointing when very large
messages is used.
Try evaluating byte size independently of time window
also increase max size
Delayed queuese can automatically create associated Shovels to transfer Ready messages
to the desired destination. This adds forwarded messages counter which will be used
in Management UI for better Shovel internals visibility.
(cherry picked from commit a8800b6cd75d8dc42a91f88655058f2ffa3b6ea6)
for the check introduced in #13487.
Note that encoding a regular expression pattern
with percent encoding is a pain (e.g. '.*' = '.%2a'),
so these endpoints fall back to a default pattern
value that matches all queues.
This commit fixes a bug in the Erlang AMQP 1.0 client.
Prior to this commit, to repro this bug:
1. Send more than 2^16 messages to a queue.
2. Grant more than a total of 2^16 link credit initially (on a single link
or across multiple links) on a single session without any
auto or manual link credit renewal.
The expectation is that thanks to sufficiently granted initial link-credit,
the client will receive all messages.
However, consumption stops after exactly 2^16-1 messages.
That's because the client lib was never sending a flow frame to the server.
So, after the client received all 2^16-1 messages (the initial
incoming-window set by the client), the server's remote-incoming-window
reached 0 causing the server to stop delivering messages.
The expectation is that the client lib automatically handles session
flow control without any manual involvement of the client app.
This commit implements this fix:
* We keep the server's remote-incoming window always large by default as
explained in https://www.rabbitmq.com/blog/2024/09/02/amqp-flow-control#incoming-window
* Hence, the client lib sets its incoming-window to 100,000 initially.
* The client lib tracks its incoming-window decrementing it by 1 for
every transfer it received. (This wasn't done prior to this commit.)
* Whenever this window shrinks below 50,000, the client sends a flow
frame without any link information widening its incoming-window back to 100,000.
* For test cases (maybe later for apps as well), there is a new function
`amqp10_client_session:flow/3`, which allows for a test case to do manual
session flow control. Its API is designed very similar to
`amqp10_client_session:flow_link/4` in that the test can optionally request
the lib to auto widen the session window whenever it falls below a certain threshold.
This is a squashed commit that includes the following changes by @efimov90:
* Initial-theme-fix
Added light.css
Added dark.css
Added link for light.css and dark.css with media attribute
Added switcher
* Rework-light-style
* dark theme
* Removed not needed div
* Fix folder name
* Color scheme fix
Removes color-scheme from main.css
Added color-scheme: dark to dark.css
Added color-scheme: light to light.css
* Fixed theme switch bug with sammy.js
Adapts code to works with sammy.js
* Icons update
* Reworked theme switcher
* Fix updating attributes
---------
Authored-by: Sergey Efimov <efimov90@gmail.com>
[Why]
Khepri already managed retries if needed, we can just use a timeout.
Note that the timeout was already bumped to a more appropriate 5
minutes, which also matches what we had with Mnesia. However, with 10
retries by default, it meant that this timeout at the end of `init/1`
would thus be 5 * 10 = 50 minutes.
This avoids using Mix while compiling which simplifies
a number of things and let us do further build improvements
later on.
Elixir is only enabled from within rabbitmq_cli currently.
Eunit is disabled since there are only Elixir tests.
Dialyzer will force-enable Elixir in order to process
Elixir-compiled beam files.
This commit also includes a few changes that are
related:
* The Erlang distribution will now be started for parallel-ct
* Many unnecessary PROJECT_MOD lines have been removed
* `eunit_formatters` has been removed, it provides little value
* The new `maybe_flock` Erlang.mk function is used where possible
* Build test deps when testing rabbitmq_cli (Mix won't do it anymore)
* rabbitmq_ct_helpers now use the early plugins to have Dialyzer
properly set up
It also happens from time to time that HTTP clients use the wrong port
5672. Like for TLS clients connecting to 5672, RabbitMQ now prints a
more descriptive log message.
For example
```
curl http://localhost:5672
```
will log
```
[info] <0.946.0> accepting AMQP connection [::1]:57736 -> [::1]:5672
[error] <0.946.0> closing AMQP connection <0.946.0> ([::1]:57736 -> [::1]:5672, duration: '1ms'):
[error] <0.946.0> {detected_unexpected_http_header,<<"GET / HT">>}
```
We only check here for GET and not for all other HTTP methods, since
that's the most common case.
## What?
If a TLS client app is misconfigured trying to connect to AMQP port 5672
instead to the AMQPS port 5671, this commit makes RabbitMQ log a more
descriptive error message.
```
openssl s_client -connect localhost:5672 -tls1_3
openssl s_client -connect localhost:5672 -tls1_2
```
RabbitMQ logs prior to this commit:
```
[info] <0.1073.0> accepting AMQP connection [::1]:53535 -> [::1]:5672
[error] <0.1073.0> closing AMQP connection <0.1073.0> ([::1]:53535 -> [::1]:5672, duration: '0ms'):
[error] <0.1073.0> {bad_header,<<22,3,1,0,192,1,0,0>>}
[info] <0.1080.0> accepting AMQP connection [::1]:53577 -> [::1]:5672
[error] <0.1080.0> closing AMQP connection <0.1080.0> ([::1]:53577 -> [::1]:5672, duration: '1ms'):
[error] <0.1080.0> {bad_header,<<22,3,1,0,224,1,0,0>>}
```
RabbitMQ logs after this commit:
```
[info] <0.969.0> accepting AMQP connection [::1]:53632 -> [::1]:5672
[error] <0.969.0> closing AMQP connection <0.969.0> ([::1]:53632 -> [::1]:5672, duration: '0ms'):
[error] <0.969.0> {detected_unexpected_tls_header,<<22,3,1,0,192,1,0,0>>
[info] <0.975.0> accepting AMQP connection [::1]:53638 -> [::1]:5672
[error] <0.975.0> closing AMQP connection <0.975.0> ([::1]:53638 -> [::1]:5672, duration: '1ms'):
[error] <0.975.0> {detected_unexpected_tls_header,<<22,3,1,0,224,1,0,0>>}
```
## Why?
I've seen numerous occurrences in the past few years where misconfigured TLS apps
connected to the wrong port. Therefore, RabbitMQ trying to detect a TLS client
and providing a more descriptive log message seems appropriate to me.
## How?
The first few bytes of any TLS connection are:
Record Type (1 byte):
Always 0x16 (22 in decimal) for a Handshake message.
Version (2 bytes):
This represents the highest version of TLS that the client supports. Common values:
0x0301 → TLS 1.0 (or SSL 3.1)
0x0302 → TLS 1.1
0x0303 → TLS 1.2
0x0304 → TLS 1.3
Record Length (2 bytes):
Specifies the length of the following handshake message.
Handshake Type (1 byte, usually the 6th byte overall):
Always 0x01 for ClientHello.
Prior to this change, we built the OCI for almost any change. That
doesn't make sense. For example, when there were changes to CI, it
didn't make because RabbitMQ had not changed.
CI will now build dev OCI images when there are actual code changes, or
changes to rabbit dependencies.
Erlang 27 is fully supported in main and v4.1.x. Support for Erlang 26
in v4.1 remains. It's better to "drop" erlang 26 from CI because, at the
moment, our PRs and commits to main trigger about 270 jobs. If we just
add '27' to the matrix, we would spawn ~216 more jobs, totalling around
496 jobs per PR and commit to main. That's simply too much, because it's
reaching the usage limits of Github Actions [1], namely the 256 limit of
matrix jobs.
[1]
https://docs.github.com/en/actions/administering-github-actions/usage-limits-billing-and-administration#usage-limits
Before the client authenticates, the standard
frame_max is not used. Instead, the limit is
a special constant.
This is fine for password or x.509 certificate-based
authentication but not for some JWT tokens,
which can vary in size, and take multiple
kilobytes.
8 kB specifically is the default HTTP header
length limit used by Nginx.
Sounds like this value was good enough
for a lot of Bearer headers with JWT tokens.
Closes#13541.
This commit fixes a bug in the Erlang AMQP 1.0 client.
Prior to this commit, to repro this bug:
1. Send more than 2^16 messages to a queue.
2. Grant more than a total of 2^16 link credit initially (on a single link
or across multiple links) on a single session without any
auto or manual link credit renewal.
The expectation is that thanks to sufficiently granted initial link-credit,
the client will receive all messages.
However, consumption stops after exactly 2^16-1 messages.
That's because the client lib was never sending a flow frame to the server.
So, after the client received all 2^16-1 messages (the initial
incoming-window set by the client), the server's remote-incoming-window
reached 0 causing the server to stop delivering messages.
The expectation is that the client lib automatically handles session
flow control without any manual involvement of the client app.
This commit implements this fix:
* We keep the server's remote-incoming window always large by default as
explained in https://www.rabbitmq.com/blog/2024/09/02/amqp-flow-control#incoming-window
* Hence, the client lib sets its incoming-window to 100,000 initially.
* The client lib tracks its incoming-window decrementing it by 1 for
every transfer it received. (This wasn't done prior to this commit.)
* Whenever this window shrinks below 50,000, the client sends a flow
frame without any link information widening its incoming-window back to 100,000.
* For test cases (maybe later for apps as well), there is a new function
`amqp10_client_session:flow/3`, which allows for a test case to do manual
session flow control. Its API is designed very similar to
`amqp10_client_session:flow_link/4` in that the test can optionally request
the lib to auto widen the session window whenever it falls below a certain threshold.
This is a squashed commit that includes the following changes by @efimov90:
* Initial-theme-fix
Added light.css
Added dark.css
Added link for light.css and dark.css with media attribute
Added switcher
* Rework-light-style
* dark theme
* Removed not needed div
* Fix folder name
* Color scheme fix
Removes color-scheme from main.css
Added color-scheme: dark to dark.css
Added color-scheme: light to light.css
* Fixed theme switch bug with sammy.js
Adapts code to works with sammy.js
* Icons update
* Reworked theme switcher
* Fix updating attributes
---------
Authored-by: Sergey Efimov <efimov90@gmail.com>
for the check introduced in #13487.
Note that encoding a regular expression pattern
with percent encoding is a pain (e.g. '.*' = '.%2a'),
so these endpoints fall back to a default pattern
value that matches all queues.
This avoids using Mix while compiling which simplifies
a number of things and let us do further build improvements
later on.
Elixir is only enabled from within rabbitmq_cli currently.
Eunit is disabled since there are only Elixir tests.
Dialyzer will force-enable Elixir in order to process
Elixir-compiled beam files.
This commit also includes a few changes that are
related:
* The Erlang distribution will now be started for parallel-ct
* Many unnecessary PROJECT_MOD lines have been removed
* `eunit_formatters` has been removed, it provides little value
* The new `maybe_flock` Erlang.mk function is used where possible
* Build test deps when testing rabbitmq_cli (Mix won't do it anymore)
* rabbitmq_ct_helpers now use the early plugins to have Dialyzer
properly set up
It also happens from time to time that HTTP clients use the wrong port
5672. Like for TLS clients connecting to 5672, RabbitMQ now prints a
more descriptive log message.
For example
```
curl http://localhost:5672
```
will log
```
[info] <0.946.0> accepting AMQP connection [::1]:57736 -> [::1]:5672
[error] <0.946.0> closing AMQP connection <0.946.0> ([::1]:57736 -> [::1]:5672, duration: '1ms'):
[error] <0.946.0> {detected_unexpected_http_header,<<"GET / HT">>}
```
We only check here for GET and not for all other HTTP methods, since
that's the most common case.
## What?
If a TLS client app is misconfigured trying to connect to AMQP port 5672
instead to the AMQPS port 5671, this commit makes RabbitMQ log a more
descriptive error message.
```
openssl s_client -connect localhost:5672 -tls1_3
openssl s_client -connect localhost:5672 -tls1_2
```
RabbitMQ logs prior to this commit:
```
[info] <0.1073.0> accepting AMQP connection [::1]:53535 -> [::1]:5672
[error] <0.1073.0> closing AMQP connection <0.1073.0> ([::1]:53535 -> [::1]:5672, duration: '0ms'):
[error] <0.1073.0> {bad_header,<<22,3,1,0,192,1,0,0>>}
[info] <0.1080.0> accepting AMQP connection [::1]:53577 -> [::1]:5672
[error] <0.1080.0> closing AMQP connection <0.1080.0> ([::1]:53577 -> [::1]:5672, duration: '1ms'):
[error] <0.1080.0> {bad_header,<<22,3,1,0,224,1,0,0>>}
```
RabbitMQ logs after this commit:
```
[info] <0.969.0> accepting AMQP connection [::1]:53632 -> [::1]:5672
[error] <0.969.0> closing AMQP connection <0.969.0> ([::1]:53632 -> [::1]:5672, duration: '0ms'):
[error] <0.969.0> {detected_unexpected_tls_header,<<22,3,1,0,192,1,0,0>>
[info] <0.975.0> accepting AMQP connection [::1]:53638 -> [::1]:5672
[error] <0.975.0> closing AMQP connection <0.975.0> ([::1]:53638 -> [::1]:5672, duration: '1ms'):
[error] <0.975.0> {detected_unexpected_tls_header,<<22,3,1,0,224,1,0,0>>}
```
## Why?
I've seen numerous occurrences in the past few years where misconfigured TLS apps
connected to the wrong port. Therefore, RabbitMQ trying to detect a TLS client
and providing a more descriptive log message seems appropriate to me.
## How?
The first few bytes of any TLS connection are:
Record Type (1 byte):
Always 0x16 (22 in decimal) for a Handshake message.
Version (2 bytes):
This represents the highest version of TLS that the client supports. Common values:
0x0301 → TLS 1.0 (or SSL 3.1)
0x0302 → TLS 1.1
0x0303 → TLS 1.2
0x0304 → TLS 1.3
Record Length (2 bytes):
Specifies the length of the following handshake message.
Handshake Type (1 byte, usually the 6th byte overall):
Always 0x01 for ClientHello.
Erlang 27 is fully supported in main and v4.1.x. Support for Erlang 26
in v4.1 remains. It's better to "drop" erlang 26 from CI because, at the
moment, our PRs and commits to main trigger about 270 jobs. If we just
add '27' to the matrix, we would spawn ~216 more jobs, totalling around
496 jobs per PR and commit to main. That's simply too much, because it's
reaching the usage limits of Github Actions [1], namely the 256 limit of
matrix jobs.
[1]
https://docs.github.com/en/actions/administering-github-actions/usage-limits-billing-and-administration#usage-limits
Prior to this change, we built the OCI for almost any change. That
doesn't make sense. For example, when there were changes to CI, it
didn't make because RabbitMQ had not changed.
CI will now build dev OCI images when there are actual code changes, or
changes to rabbit dependencies.
Before the client authenticates, the standard
frame_max is not used. Instead, the limit is
a special constant.
This is fine for password or x.509 certificate-based
authentication but not for some JWT tokens,
which can vary in size, and take multiple
kilobytes.
8 kB specifically is the default HTTP header
length limit used by Nginx.
Sounds like this value was good enough
for a lot of Bearer headers with JWT tokens.
Closes#13541.
[Why]
Khepri already managed retries if needed, we can just use a timeout.
Note that the timeout was already bumped to a more appropriate 5
minutes, which also matches what we had with Mnesia. However, with 10
retries by default, it meant that this timeout at the end of `init/1`
would thus be 5 * 10 = 50 minutes.
It is very hard now to distinguish different tabs. With this addition
we have titles like 'RabbitMQ - Queue vhost/name', 'RabbitMQ - Exchanges'.
To be continued...
otherwise we end up with two copies of the compiled
module on the code path some of the time.
We don't need to mix Erlang and Elixir even
more to bring in one constant that hasn't changed
since its introduction some eight years ago.
(cherry picked from commit c32b948258f226a86be91cab80448d7a536afd7d)
RMQ-1263: Add a --force option to rabbitmqctl delete_queue command.
This work was originally done by Iliia Khaprov <iliia.khaprov@broadcom.net>.
---------
Co-authored-by: Iliia Khaprov <iliia.khaprov@broadcom.net>
Co-authored-by: Michael Klishin <klishinm@vmware.com>
(cherry picked from commit d9522d3ee708250cc84443af5c3556b14f7c5ab9)
* RMQ-1263: Check if queue protected from deleted inside rabbit_amqqueue:with_delete
Delayed exchange automatically manages associated Delayed Queue. We don't want users to delete it accidentally.
If queue is indeed protected its removal can be forced by calling with
?INTERNAL_USER as ActingUser.
* RMQ-1263: Correct a type spec of amqqueue:internal_owner/1
* RMQ-1263: Add protected queues test
---------
Co-authored-by: Iliia Khaprov <iliia.khaprov@broadcom.net>
Co-authored-by: Michael Klishin <klishinm@vmware.com>
(cherry picked from commit 97f44adfad6d0d98feb1c3a47de76e72694c19e0)
* Add num_segments to Ra counters
* ra_server_proc: Fix handling of local query replies
* Remove Bazel-related files by @mkuratczyk in #520
* Replication bug fixes that could cause replication to stall
* Use infinity timeout for ra_log_ets:mem_table_please
[Why]
The CLI sometimes crashes early because it fails to configure the Erlang
distribution.
Because we use two CLI commands to watch the start of RabbitMQ, if one
of them fails, the Make recipe will exit with an error, leaving the
RabbitMQ node running.
[How]
We use a shell trap to stop the node if the shell is about to exit with
an error.
While here, we retry the `await_startup` CLI command several times
because this is the one failing the most. This is until the crash is
understood and a proper fix is committed.
For scenarios where rabbitmq needs
the certificates of an idp but the
idp has not been started yet and
hence the cert has not been generated
With this function, the idp generates
its certificates without starting
... and cache it.
[Why]
It happens at least in CI that the computed start time varies by a few
seconds. I think this comes from the Erlang time offset which might be
adjusted over time.
This affects peer discovery's sorting of RabbitMQ nodes which uses that
start time to determine the oldest node. When the start time of a node
changes, it could be considered the seed node to join by some nodes but
ignored by the other nodes, leading to troubles with cluster formation.
[Why]
It happens in CI from time to time and it was crashing the channel
process. There is always a `channel.close` method pending in the
channel mailbox.
[How]
For now, log something and ignore the DOWN message. The channel will
exit after handling the pending `channel.close` method anyway.
* Implement rabbitmq-queues leader_health_check command for quorum queues
(cherry picked from commit c26edbef33)
* Tests for rabbitmq-queues leader_health_check command
(cherry picked from commit 6cc03b0009)
* Ensure calling ParentPID in leader health check execution and
reuse and extend formatting API, with amqqueue:to_printable/2
(cherry picked from commit 76d66a1fd7)
* Extend core leader health check tests and update badrpc error handling in cli tests
(cherry picked from commit 857e2a73ca)
* Refactor leader_health_check command validators and ignore vhost arg
(cherry picked from commit 6cf9339e49)
* Update leader_health_check_command description and banner
(cherry picked from commit 96b8bced2d)
* Improve output formatting for healthy leaders and support
silent mode in rabbitmq-queues leader_health_check command
(cherry picked from commit 239a69b404)
* Support global flag to run leader health check for
all queues in all vhosts on local node
(cherry picked from commit 48ba3e161f)
* Return immediately for leader health checks on empty vhosts
(cherry picked from commit 7873737b35)
* Rename leader health check timeout refs
(cherry picked from commit b7dec89b87)
* Update banner message for global leader health check
(cherry picked from commit c7da4d5b24)
* QQ leader-health-check: check_process_limit_safety before spawning leader checks
(cherry picked from commit 17368454c5)
* Log leader health check result in broker logs (if any leaderless queues)
(cherry picked from commit 1084179a2c)
* Ensure check_passed result for leader health internal calls)
(cherry picked from commit 68739a6bd2)
* Extend CLI format output to process check_passed payload
(cherry picked from commit 5f5e9922bd)
* Format leader healthcheck result log and function exports
(cherry picked from commit ebffd7d8a4)
* Change leader_health_check command scope from queues to diagnostics
(cherry picked from commit 663fc9846e)
* Update (c) line year
(cherry picked from commit df82f12a70)
* Rename command to check_for_quorum_queues_without_an_elected_leader
and use across_all_vhosts option for global checks
(cherry picked from commit b2acbae28e)
* Use rabbit_db_queue for qq leader health check lookups
and introduce rabbit_db_queue:get_all_by_type_and_vhost/2.
Update leader health check timeout to 5s and process limit
threshold to 20% of node's process_limit.
(cherry picked from commit 7a8e166ff6)
* Update tests: quorum_queue_SUITE and rabbit_db_queue_SUITE
(cherry picked from commit 9bdb81fd79)
* Fix typo (cli test module)
(cherry picked from commit 615856853a)
* Small refactor - simpler final leader health check result return on function head match
(cherry picked from commit ea07938f3d)
* Clear dialyzer warning & fix type spec
(cherry picked from commit a45aa81bd2)
* Ignore result without strict match to avoid diayzer warning
(cherry picked from commit bb43c0b929)
* 'rabbitmq-diagnostics check_for_quorum_queues_without_an_elected_leader' documentation edits
(cherry picked from commit 845230b0b380a5f5bad4e571a759c10f5cc93b91)
* 'rabbitmq-diagnostics check_for_quorum_queues_without_an_elected_leader' output copywriting
(cherry picked from commit 235f43bad58d3a286faa0377b8778fcbe6f8705d)
* diagnostics check_for_quorum_queues_without_an_elected_leader: behave like a health check w.r.t. error reporting
(cherry picked from commit db7376797581e4716e659fad85ef484cc6f0ea15)
* check_for_quorum_queues_without_an_elected_leader: handle --quiet and --silent
plus simplify function heads.
References #13433.
(cherry picked from commit 7b392315d5e597e5171a0c8196230d92b8ea8e92)
---------
Co-authored-by: Ayanda Dube <adube14@bloomberg.net>
observer_cli (and its dependency recon) was declared as a dependency
of rabbitmq_cli and as a consequence included in all escritps. However
the major part of observer_cli runs in the broker. The cli side only
used `observer_cli:rpc_start/2` which is just an rpc call into the
target node.
By using common rpc call we can remove observer_cli and recon from the
escripts. This can be considered a minor improvement based on the
philosophy "simpler is better".
As an additional benefit auto-completing functions of the recon app
now works in `rabbitmq-diagnostics remote_shell`.
(eg. `recon:proc_c<TAB>`)
CI sometimes failed with the following error:
```
v5_SUITE:session_upgrade_v3_v5_qos failed on line 1068
Reason: {test_case_failed,Received unexpected PUBLISH payload. Expected: <<"2">> Got: <<"3">>}
```
The emqtt client auto acks by default.
Therefore, if Subv3 client was able to successfully auto ack message 2
before Subv3 disconnected, Subv5 client did not receive message 2.
This commit fixes this flake by making sure that Subv3 does not ack
message 2.
This is done by introducing a generic function that holds the common code, which then creates these two targets. The differing properties (like rsync flags) are passed in as function arguments.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
The main difference is that the "bundle" target does NOT exclude packaging and testing directories, which enables packaging and testing of the source archive.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Fix crash in close_sent since the client might receive the open frame if
it previously sent the close frame in state open_sent.
We explicitly ignore the open frame. The alternative is to add another
gen_statem state CLOSE_PIPE which might be an overkill however.
This commit also fixes a wrong comment: No sessions have begun if the
app requests the connection to be closed in state open_sent.
[Why]
This testsuite is very unstable and it is difficult to debug while it is
part of a `parallel-ct` group. It also forced us to re-run the entire
`parallel-ct` group just to retry that one testsuite.
The `buffer` socket option will be changed dynamically
based on how much data is received.
This is restricted to AMQP protocols (old and 1.0).
The algorithm is a little different than Cowboy 2.13.
The moving average is less reactive (div 8 instead of 2)
and floats are used so that using smaller lower buffer
values is possible (otherwise the rounding prevents
increasing buffer sizes). The lower buffer size was
set to 128 as a result.
Compared to the previous which was to set `buffer` to
`rcvbuf` effectively, often to 131072 on Linux for
example, the performance sees a slight improvement
in various scenarios for all message sizes using
AMQP-0.9.1 and a lower memory usage as well. But
the difference is small in the benchmarks we have
run (5% to 10%), whereas Cowboy saw a huge improvement
because its default was very small (1460).
For AMQP-1.0 this seems to be no worse but we didn't
detect a clear improvement. We saw scenarios where
small message sizes showed improvement, and large
message sizes showed a regression. But we are even
less confident with these results. David (AMQP-1.0
native developer) ran a few tests and didn't see a
regression.
The dynamic buffer code is currently identical for
old and 1.0 AMQP. But we might tweak them differently
in the future so they're left as duplicate for now.
This is because different protocols have different
behaviors and so the algorithm may need to be tweaked
differently for each protocol.
The `msg` record was used in 3.13. This commit makes 4.x understand
this record for backward compatibility, specifically for the rare case where:
1. a 3.13 node internally parsed a message from a stream via
```
Message = mc:init(mc_amqp, amqp10_framing:decode_bin(Bin), #{})
```
2. published this Message to a queue
3. RabbitMQ got upgraded to 4.x
(This commit can be reverted in some future RabbitMQ version once it's
safe to assume that these upgraded messages have been consumed.)
The changes were manually tested as described in Jira RMQ-1525.
We must consider whether the previous current file is empty
(has data written, but was already removed) when writing
large messages and opening a file specifically for the large
message. If we don't, then the file will never get deleted
as we only consider files for deletion when a message gets
removed (and there are none).
This is only an issue for large messages. Small messages
write a message than roll over to a new file, so there is
at least one valid message. Large messages close the current
file first, regardless of there being a valid message.
Prior to this commit, if a test failed, the script 'run-suites.sh' would
exit with non-zero status, stopping the exection of the job; therefore,
the steps to move the logs to the expected location won't be executed.
This commit separates the tests from the log preparation.
In certain environments, we may want to customise the docker image e.g.
to use a proxy to avoid docker hub rate limiting. The default behaviour
remains unchanged.
The `if` logic was broken because `uname -a` returns the entire uname,
including OS, Kernel version, machine type and what not. The string
always starts with the OS i.e. Linux or Darwin, therefore, the matching
for `arm*` was always false; therefore, it was always defaulting to the
`else` image, which happens to be multi-arch. However, it was using
`seleniarm`, which is a community driven effort, not the official
Selenium account.
In the official OSS image, version 123.0 is too old. The oldest
available is 127.0. This commit bumps to the latest available. We could
consider depending on version `4`. Version `4` refers to Selenium
version, whilst version 123.0/133.0 refer to the browser version.
The workflow to tests PRs is meant to run the short suite for management
UI tests. On commits, we want to run the full suite to ensure that
management UI tests are passing.
Prior to this commit, if the WebSocket client received multiple
WebSocket frames in a single Erlang message by gen_tcp, the WebSocket
client sent only the first received WebSocket frame to the application.
This commit fixes this bug by having the WebSocket client send all
WebSocket frames to the application.
The `rabbit_registry` boot step starts up the `rabbit_registry` gen
server from `rabbit_common`. This is a registry somewhat similar to
the feature flag registry - it's meant to protect an ETS table used for
looking up implementers of behaviors. The registry and its ETS table
should be available as early as possible: the step should enable
external_infrastructure rather than require it.
This target is identical to the existing "source-dist" target, except that it allows for packaging and testing of the source archive. This is done by including the packaging/ and tests/ directories in the output tarball, along with specific subdirectories that are required by tests.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
The previous behaviour was passing solely the message ID making
queue implementations such as, for example, the priority one hard
to fulfil.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
(cherry picked from commit 1f7a27c51d)
Since https://github.com/rabbitmq/rabbitmq-server/pull/13242 updated
Cowlib to v2.14.0, this commit deletes rabbit_uri as written in the
comments of rabbit_uri.erl:
```
This file is a partial copy of
https://github.com/ninenines/cowlib/blob/optimise-urldecode/src/cow_uri.erl
We use this copy because:
1. uri_string:unquote/1 is lax: It doesn't validate that characters that are
required to be percent encoded are indeed percent encoded. In RabbitMQ,
we want to enforce that proper percent encoding is done by AMQP clients.
2. uri_string:unquote/1 and cow_uri:urldecode/1 in cowlib v2.13.0 are both
slow because they allocate a new binary for the common case where no
character was percent encoded.
When a new cowlib version is released, we should make app rabbit depend on
app cowlib calling cow_uri:urldecode/1 and delete this file (rabbit_uri.erl).
```
Cowboy 2.13 contains the Websocket optimisations as well
as the ability to set the Websocket max_frame_size option
dynamically, plus plenty of other improvements.
Cowlib was added as a test dep to rabbitmq_mqtt to make
sure emqtt doesn't pull the wrong Cowlib version for Cowboy.
Only large messages delivered to multiple CQs are stored once for
multiple queues.
Non-durable queues are deprecated and will be removed, so don't even
mention them.
We don't "page out" messages anymore.
When trying to use OTP28.0-rc1, Elixir fails to compile these modules
because a module attribute cannot be a regex. It is not yet clear
whether it's something to be fixed in Elixir for OTP28 compatibility
or something that accidentally worked in the past, but either way,
using a string as an attribute is equally good and works all OTP
versions, including OTP28.0-rc1.
```
== Compilation error in file lib/rabbitmq/cli/core/command_modules.ex ==
** (ArgumentError) cannot inject attribute @commands_ns into function/macro because cannot escape #Reference<0.2201422310.1333657602.13657>. The supported values are: lists, tuples, maps, atoms, numbers, bitstrings, PIDs and remote functions in the format &Mod.fun/arity
(elixir 1.18.2) lib/kernel.ex:3729: Kernel.do_at/5
(elixir 1.18.2) expanding macro: Kernel.@/1
lib/rabbitmq/cli/core/command_modules.ex:133: RabbitMQ.CLI.Core.CommandModules.make_module_map/2
```
Since 4.0.0 (commit d45fbc3d) the shared message store writes large
messages into their own rdq files. This information can be utilised
when scanning rdq files during recovery to avoid reading in the whole
message body into memory unnecessarily.
This commit addresses the same issue that was addressed in 3.13.x by
commit baeefbec (ie. appending a large binary together from 4MB chunks
leaves a lot of garbage and memory fragmentation behind) but even more
efficiently.
Large messages which were written before 4.0.0, which don't fully fill
the rdq file, are still handled as before.
```
make -C deps/rabbit ct-rabbit_stream_queue t=cluster_size_3_parallel_1 RABBITMQ_METADATA_STORE=mnesia
```
flaked prior to this commit locally on Ubuntu with the following error after 11 runs:
```
rabbit_stream_queue_SUITE > cluster_size_3_parallel_1 > consume_from_replica
{error,
{{shutdown,
{server_initiated_close,406,
<<"PRECONDITION_FAILED - stream queue 'consume_from_replica' in vhost '/' does not have a running replica on the local node">>}},
{gen_server,call,
[<0.8365.0>,
{subscribe,
{'basic.consume',0,<<"consume_from_replica">>,
<<"ctag">>,false,false,false,false,
[{<<"x-stream-offset">>,long,0}]},
<0.8151.0>},
infinity]}}}
```
Initialising a message container from data stored in a
stream is a special case where we need to recover exchange
and routing key information from the following message annatations:
* x-exchange
* x-routing-keys
* x-cc
We do not want to do this when initialising a message container
from AMQP data just received from a publisher.
This commit introduces a new function `mc_amqp:init_from_stream/2`
that is to be used when needing a message container from a stream
message.
[Why]
During mixed-version testing, the old node might not be able to join or
rejoin a cluster if the other nodes run a newer Khepri machine version.
[How]
The old node is used as the cluster seed node and is never touched
otherwise. Other nodes are restarted or join the cluster later.
## What?
Support the `dynamic` field of sources and targets.
## Why?
1. This allows AMQP clients to dynamically create exclusive queues, which
can be useful for RPC workloads.
2. Support creation of JMS temporary queues over AMQP using the Qpid JMS
client. Exclusive queues map very nicely to JMS temporary queues
because:
> Although sessions are used to create temporary destinations, this is only
for convenience. Their scope is actually the entire connection. Their
lifetime is that of their connection and any of the connection’s sessions
are allowed to create a consumer for them.
https://jakarta.ee/specifications/messaging/3.1/jakarta-messaging-spec-3.1#creating-temporary-destinations
## How?
If the terminus contains the capability `temporary-queue` as defined in
[amqp-bindmap-jms-v1.0-wd10](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=67638)
[5.2] and as sent by Qpid JMS client,
RabbitMQ will create an exclusive queue.
(This allows a future commit to take other actions if capability
`temporary-topic` will be used, such as the additional creation of bindings.)
No matter what the desired node properties are, RabbitMQ will set the
lifetime policy delete-on-close deleting the exclusive queue when the
link which caused its creation ceases to exist. This means the exclusive
queue will be deleted if either:
* the link gets detached, or
* the session ends, or
* the connection closes
Although the AMQP JMS Mapping and Qpid JMS create only a **sending** link
with `dynamic=true`, this commit also supports **receiving** links with
`dynamic=true` for non-JMS AMQP clients.
RabbitMQ is free to choose the generated queue name. As suggested by the
AMQP spec, the generated queue name will contain the container-id and link
name unless they are very long.
Co-authored-by: Arnaud Cogoluègnes <acogoluegnes@gmail.com>
Make AMQP 1.0 connection shut down its sessions before sending the
close frame to the client similar to how the AMQP 0.9.1 connection
shuts down its channels before closing the connection.
This commit avoids concurrent deletion of exclusive queues by the session process
and the classic queue process.
This commit should also fix https://github.com/rabbitmq/rabbitmq-server/issues/2596
* Separate invalid client test from the valid one
* Apply same changes from pr #13197
* Deal with stalereferences caused by timing issues
looking up objects in the DOM
* Unlink before assertion
... are being used at the same time.
[Why]
Depending on which node clusters with which, a node running an older
version of the Khepri Ra machine may not be able to apply Ra commands
and could be stuck.
There is no real solution and this clearly an unsupported scenario. An
old node won't always be able to join a newer cluster.
[How]
In the testsuites, we skip clustering tests if we detect that multiple
Khepri Ra machine versions are being used.
The following test flaked in CI under Khepri in mixed version mode:
```
make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_3:will_delay_node_restart RABBITMQ_METADATA_STORE=khepri SECONDARY_DIST=rabbitmq_server-4.0.5 FULL=1
```
The first node took exactly 30 seconds for draining:
```
2025-02-10 15:00:09.550824+00:00 [debug] <0.1449.0> MQTT accepting TCP connection <0.1449.0> (127.0.0.1:33376 -> 127.0.0.1:27005)
2025-02-10 15:00:09.550992+00:00 [debug] <0.1449.0> Received a CONNECT, client ID: sub0, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2025-02-10 15:00:09.551134+00:00 [debug] <0.1449.0> MQTT connection 127.0.0.1:33376 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2025-02-10 15:00:09.551219+00:00 [debug] <0.1449.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2025-02-10 15:00:09.551530+00:00 [info] <0.1449.0> Accepted MQTT connection 127.0.0.1:33376 -> 127.0.0.1:27005 for client ID sub0
2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> Received a SUBSCRIBE with subscription(s) [{mqtt_subscription,<<"my/topic">>,
2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> {mqtt_subscription_opts,0,false,
2025-02-10 15:00:09.551651+00:00 [debug] <0.1449.0> false,0,undefined}}]
2025-02-10 15:00:09.556233+00:00 [debug] <0.896.0> RabbitMQ metadata store: follower leader cast - redirecting to {rabbitmq_metadata,'rmq-ct-mqtt-cluster_size_3-2-27054@localhost'}
2025-02-10 15:00:09.561518+00:00 [debug] <0.1456.0> MQTT accepting TCP connection <0.1456.0> (127.0.0.1:33390 -> 127.0.0.1:27005)
2025-02-10 15:00:09.561634+00:00 [debug] <0.1456.0> Received a CONNECT, client ID: will, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2025-02-10 15:00:09.561715+00:00 [debug] <0.1456.0> MQTT connection 127.0.0.1:33390 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2025-02-10 15:00:09.561828+00:00 [debug] <0.1456.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2025-02-10 15:00:09.562596+00:00 [info] <0.1456.0> Accepted MQTT connection 127.0.0.1:33390 -> 127.0.0.1:27005 for client ID will
2025-02-10 15:00:09.565743+00:00 [warning] <0.1460.0> This node is being put into maintenance (drain) mode
2025-02-10 15:00:09.565833+00:00 [debug] <0.1460.0> Marking the node as undergoing maintenance
2025-02-10 15:00:09.570772+00:00 [info] <0.1460.0> Marked this node as undergoing maintenance
2025-02-10 15:00:09.570904+00:00 [info] <0.1460.0> Asked to suspend 9 client connection listeners. No new client connections will be accepted until these listeners are resumed!
2025-02-10 15:00:09.572268+00:00 [warning] <0.1460.0> Suspended all listeners and will no longer accept client connections
2025-02-10 15:00:09.572317+00:00 [warning] <0.1460.0> Closed 0 local client connections
2025-02-10 15:00:09.572418+00:00 [warning] <0.1449.0> MQTT disconnecting client <<"127.0.0.1:33376 -> 127.0.0.1:27005">> with client ID 'sub0', reason: maintenance
2025-02-10 15:00:09.572414+00:00 [warning] <0.1000.0> Closed 2 local (Web) MQTT client connections
2025-02-10 15:00:09.572499+00:00 [warning] <0.1456.0> MQTT disconnecting client <<"127.0.0.1:33390 -> 127.0.0.1:27005">> with client ID 'will', reason: maintenance
2025-02-10 15:00:09.572866+00:00 [alert] <0.1000.0> Closed 0 local STOMP client connections
2025-02-10 15:00:09.577432+00:00 [debug] <0.1456.0> scheduled delayed Will Message to topic my/topic for MQTT client ID will to be sent in 10000 ms
2025-02-10 15:00:12.991328+00:00 [debug] <0.1469.0> Will reconcile virtual host processes on all cluster members...
2025-02-10 15:00:12.991443+00:00 [debug] <0.1469.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes
2025-02-10 15:00:12.992497+00:00 [debug] <0.1469.0> Done with virtual host processes reconciliation (run 3)
2025-02-10 15:00:16.511733+00:00 [debug] <0.1476.0> Will reconcile virtual host processes on all cluster members...
2025-02-10 15:00:16.511864+00:00 [debug] <0.1476.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes
2025-02-10 15:00:16.514293+00:00 [debug] <0.1476.0> Done with virtual host processes reconciliation (run 4)
2025-02-10 15:00:24.897477+00:00 [debug] <0.1479.0> Will reconcile virtual host processes on all cluster members...
2025-02-10 15:00:24.897607+00:00 [debug] <0.1479.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes
2025-02-10 15:00:24.898483+00:00 [debug] <0.1479.0> Done with virtual host processes reconciliation (run 5)
2025-02-10 15:00:24.898527+00:00 [debug] <0.1479.0> Will reschedule virtual host process reconciliation after 30 seconds
2025-02-10 15:00:32.994347+00:00 [debug] <0.1484.0> Will reconcile virtual host processes on all cluster members...
2025-02-10 15:00:32.994474+00:00 [debug] <0.1484.0> Will make sure that processes of 1 virtual hosts are running on all reachable cluster nodes
2025-02-10 15:00:32.996539+00:00 [debug] <0.1484.0> Done with virtual host processes reconciliation (run 6)
2025-02-10 15:00:32.996585+00:00 [debug] <0.1484.0> Will reschedule virtual host process reconciliation after 30 seconds
2025-02-10 15:00:39.576325+00:00 [info] <0.1460.0> Will transfer leadership of 0 quorum queues with current leader on this node
2025-02-10 15:00:39.576456+00:00 [info] <0.1460.0> Leadership transfer for quorum queues hosted on this node has been initiated
2025-02-10 15:00:39.576948+00:00 [info] <0.1460.0> Will stop local follower replicas of 0 quorum queues on this node
2025-02-10 15:00:39.576990+00:00 [info] <0.1460.0> Stopped all local replicas of quorum queues hosted on this node
2025-02-10 15:00:39.577120+00:00 [info] <0.1460.0> Will transfer leadership of metadata store with current leader on this node
2025-02-10 15:00:39.577282+00:00 [info] <0.1460.0> Khepri clustering: transferring leadership to node 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost'
2025-02-10 15:00:39.577424+00:00 [info] <0.1460.0> Khepri clustering: skipping leadership transfer, leader is already in node 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost'
2025-02-10 15:00:39.577547+00:00 [info] <0.1460.0> Leadership transfer for metadata store on this node has been done. The new leader is 'rmq-ct-mqtt-cluster_size_3-2-27054@localhost'
2025-02-10 15:00:39.577674+00:00 [info] <0.1460.0> Node is ready to be shut down for maintenance or upgrade
2025-02-10 15:00:39.595638+00:00 [notice] <0.64.0> SIGTERM received - shutting down
2025-02-10 15:00:39.595638+00:00 [notice] <0.64.0>
2025-02-10 15:00:39.595758+00:00 [debug] <0.44.0> Running rabbit_prelaunch:shutdown_func() as part of `kernel` shutdown
```
Running the same test locally revealed that [rabbit_maintenance:status_consistent_read/1](55ae918094/deps/rabbit/src/rabbit_maintenance.erl (L131))
takes exactly 30 seconds to complete.
The test case assumes a Will Delay higher than the time it takes to
drain and shut down the node. Hence, this commit increases the Will
Delay time from 10 seconds to 40 seconds.
That was not keycloak format it was an
extension to the oauth spec introuduced
a few years ago. To get a token from
keycloak using this format, a.k.a.
requesting party token, one has to specify
a different claim type called
urn:ietf:params:oauth:grant-type:uma-ticket
[Why]
Some testcases used to use node 1 as the clustering seed node. With
mixed-version testing, it could cause issues because node 1 would start
with a new version of Ra compared to node 2 and node 2 could fail to
join.
[How]
By using node 2 as the seed node, node 1 running a newer version of Ra
should be able to join because it supports talking to an older version.
Consumer count is already returned by the /channels API endpoint. Now
the consumer count column can be shown in the channels table but it is
hidden by default.
2025-02-03 18:31:55 +01:00
1149 changed files with 53364 additions and 53660 deletions
From https://docs.bazel.build/versions/master/bazel-overview.html
> Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.
## Why RabbitMQ + Bazel?
RabbitMQ, Tier1 plugins included, is a large codebase. The developer experience benefits from fast incremental compilation.
More importantly, RabbitMQ's test suite is large and takes hours if run on a single machine. Bazel allows tests to be run in parallel on a large number of remote workers if needed, and furthermore uses cached test results when branches of the codebase remain unchanged.
Bazel does not provide built in Erlang or Elixir support, nor is there an available library of bazel rules. Therefore, we have defined our own rules in https://github.com/rabbitmq/bazel-erlang. Elixir compilation is handled as a special case within this repository. To use these rules, the location of your Erlang and Elixir installations must be indicated to the build (see below).
While most of work for running tests happens in Bazel, the suite still makes use of some external tools for commands, notably gnu `make` and `openssl`. Ideally we could bring all of these tools under bazel, so that the only tool needed would be `bazel` or `bazelisk`, but that will take some time.
Create a `user.bazelrc` by making a copy of `user-template.bazelrc` and updating the paths in the first few lines.
### Run the broker
`bazel run broker`
You can set different environment variables to control some configuration aspects, like this:
```
RABBITMQ_CONFIG_FILES=/path/to/conf.d \
RABBITMQ_NODENAME=<node>@localhost \
RABBITMQ_NODE_PORT=7000 \
bazel run broker
```
This will start RabbitMQ with configs being read from the provided directory. It also will start a node with a given node name, and with all listening ports calculated from the given one - this way you can start non-conflicting rabbits even from different checkouts on a single machine.
### Running tests
Many rabbit tests spawn single or clustered rabbit nodes, and therefore it's best to run test suites sequentially on a single machine. Hence the `build --local_test_jobs=1` flag used in `.bazelrc`. Additionally, it may be reasonable to disable test sharding and stream test output when running tests locally with `--test_output=streamed` as an additional argument (to just disable sharding, but not stream output, use `--test_sharding_strategy=disabled`). Naturally that restriction does not hold if utilizing remote execution (as is the case for RabbitMQ's CI pipelines).
Erlang Common Test logs will not be placed in the logs directory when run with bazel. They can be found under `bazel-testlogs`. For instance, those of the rabbit application's backing_queue suite will be under `bazel-testlogs/deps/rabbit/backing_queue_SUITE/test.outputs/`.
### Run all tests
Note: This takes quite some time on a single machine.
`bazel test //...`
### Run tests in a 'package' and its 'subpackages'
**rabbit** is an appropriate example because it encloses the **rabbitmq_prelaunch** application.
`bazel test deps/rabbit/...`
### Run tests for a specific 'package'
`bazel test deps/rabbit_common:all`
### Run an individual common test suite
`bazel test //deps/rabbit:lazy_queue_SUITE`
## Add/update an external dependency
### from hex.pm
1. `bazel run gazelle-update-repos -- hex.pm/accept@0.3.5` to generate/update `bazel/BUILD.accept`
1. Add/update the entry in MODULE.bazel
### from github
1. `bazel run gazelle-update-repos -- --testonly github.com/extend/ct_helper@master`
1. Add/update the entry in MODULE.bazel
## Update BUILD files
`bazel run gazelle`
## Regenerate moduleindex.yaml
`bazel run :moduleindex > moduleindex.yaml`
## Additional Useful Commands
- Format all bazel files consistently (requires [buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)):
`buildifier -r .`
- Remove unused load statements from BUILD.bazel files (requires [buildozer](https://github.com/bazelbuild/buildtools/blob/master/buildozer/README.md)):
@ -7,13 +7,13 @@ Pull requests is the primary place of discussing code changes.
The process is fairly standard:
* Make sure you (or your employer/client) [signs the Contributor License Agreement](https://github.com/rabbitmq/cla) if needed (see below)
* Present your idea to the RabbitMQ core team using [GitHub Discussions](https://github.com/rabbitmq/rabbitmq-server/discussions) or [RabbitMQ community Discord server](https://rabbitmq.com/discord)
* Fork the repository or repositories you plan on contributing to
* Run `git clean -xfffd && gmake clean && gmake distclean && gmake` to build all subprojects from scratch
* Create a branch with a descriptive name
* Make your changes, run tests, ensure correct code formatting, commit with a [descriptive message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html), push to your fork
* Submit pull requests with an explanation what has been changed and **why**
* Submit a filled out and signed [Contributor Agreement](https://cla.pivotal.io/) if needed (see below)
* Be patient. We will get to your pull request eventually
Bring up a Debian Jesse instance using Vagrant, or, use the provisioning script
on a Debian Jesse server of your own. When it is done running, all necessary
package build requirements for either `apt`-based or `rpm`-based distros will
be present. See the [`README.md`](README.md#tldr) document for instructions on
building packages.
See [`rabbitmq/build-env-images`](https://github.com/rabbitmq/build-env-images) and [`rabbitmq/server-packages`](https://github.com/rabbitmq/server-packages?tab=readme-ov-file).
*Note:* as of this writing, RabbitMQ `3.7.8` is the latest version. Be sure to check out the tag appropriate for your use:
```
cd rabbitmq-server-release
# the following checks out the "next to be released" branch
# this branch and version 3.7.9 have a necessary fix for building
# on windows, see this: https://github.com/rabbitmq/rabbitmq-server-release/pull/89
git checkout v3.7.x
make deps
```
### Build Windows package
```
make UNIX_TO_DOS=unix2dos package-windows
```
See [`rabbitmq/build-env-images`](https://github.com/rabbitmq/build-env-images) and [`rabbitmq/server-packages`](https://github.com/rabbitmq/server-packages?tab=readme-ov-file).
See [`rabbitmq/server-packages`](https://github.com/rabbitmq/server-packages/).
See [`rabbitmq/build-env-images`](https://github.com/rabbitmq/build-env-images) and [`rabbitmq/server-packages`](https://github.com/rabbitmq/server-packages?tab=readme-ov-file).
Michael Davis
Arnaud Cogoluègnes
dependabot[bot]
Michael Klishin
Aitor Pérez Cedres
Jean-Sébastien Pédron
Diana Parra Corbacho
Luke Bakken
Loïc Hoguin
Michael Klishin
Péter Gömöri
David Ansari
Karl Nilsson
Michal Kuratczyk
Ben Nguyen
D Corbacho
Marcial Rosales
Sunny Katkuri
udeeksha30-netizen
Deeksha
Simon Unge
Ayanda Dube
tomyouyou
Iliia Khaprov - VMware by Broadcom
Michael Davis
Luke Bakken
LoisSotoLopez
Aaron Seo
Iliia Khaprov
Razvan Grigore
Michael Klishin
Anh Nguyen
Mirah Gary
Kartik Ganesh
Tony Lewis Hiroaki URAHAMA
David Ansari
Matteo Cafasso
GitHub
Arnaud Cogoluègnes
Marcial Rosales
Iliia Khaprov - VMware by Broadcom