Trigger a 4.3.x alpha release build / trigger_alpha_build (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 26) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 27) (push) Waiting to runDetails
Test (make) / Build and Xref (1.18, 28) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, khepri) (push) Waiting to runDetails
Test (make) / Test mixed clusters (1.18, 28, mnesia) (push) Waiting to runDetails
Test (make) / Type check (1.18, 28) (push) Waiting to runDetails
`rabbit_amqqueue:declare/7` consults the `is_enabled/0` callback of
`rabbit_queue_type` before attempting to create a queue.
`rabbit_stream_manager` emulates `rabbit_amqqueue:declare/7` rather
than calling it directly. To align with the `is_enabled/0` check in
`rabbit_amqqueue:declare/7`, `rabbit_stream_manager` needs to check
the callback during stream creation.
A quorum queue tries to repair its record in a tick handler. This can
happen during a network partition and the metadata store may itself be
unavailable, making the update likely to time out.
The default metadata store timeout is usually higher than the tick
interval, so the tick handler may be stuck during several ticks. The
record takes some time to be updated (timeout + tick interval, 30 + 5
seconds by default), significantly longer than it takes the metadata
store to trigger an election and recover.
Client applications may rely on the quorum queue topology to connect to
an appropriate node, so making the system reflect the actual topology
faster is important to them.
This commit makes the record update operations use a timeout 1-second
lower than the tick interval. The tick handler process should finish
earlier in case of metadata datastore unavailability and it should not
take more than a couple of ticks once the datastore is available to
update the record.
... instead of as part of a parallel CT group.
[Why]
This testsuite takes quite a lot of time. It affects the time to run the
parallel CT set 2 significantly, especially if it has to be restarted.
[Why]
The `*_queue_after_partition_recovery_1` testcases rely on the fact that
the queue process retries to delete its queue record in its terminate
function.
RabbitMQ 4.1.x and before don't have that. Thus, depending on the timing
of the election of the Khepri leader after a network partition, the test
might fail.
As mentioned in discussion #14426, the way that `cacerts` is handled by
cuttlefish schemas simply will not work if set.
If `cacerts` were set to a string value containing one X509 certificate,
it would eventually result in a crash because the `cacerts` ssl option
must be of [this type](https://www.erlang.org/doc/apps/ssl/ssl.html#t:client_option_cert/0):
```
{cacerts, CACerts :: [public_key:der_encoded()] | [public_key:combined_cert()]}
```
Neither of those are strings, of course.
This PR removes all use of `cacerts` in cuttlefish schemas. In addition,
it filters out `cacerts` and `certs_keys` from being JSON-encoded by an
HTTP API call to `/api/overview`. It _is_ technically possible to set
`cacerts` via `advanced.config`, so, if set, it would crash this API
call, as would `certs_keys`.
[Why]
With Mnesia, when the network partition strategy is set to
`pause_minority`, nodes on the "minority side" are stopped.
Thus, the exclusive queues that were hosted by nodes on that minority
side are lost:
* Consumers connected on these nodes are disconnected because the nodes
are stopped.
* Queue records on the majority side are deleted from the metadata
store.
This was ok with Mnesia and how this network partition handling strategy
is implemented. However, it does not work with Khepri because the nodes
on the "minority side" continue to run and serve clients. Therefore the
cluster ends up in a weird situation:
1. The "majority side" deleted the queue records.
2. When the network partition is solved, the "minority side" gets the
record deletion, but the queue processes continue to run.
This was similar for auto-delete queues.
[How]
With Khepri, we stop to delete transient queue records in general, just
because there is a node going down. Thanks to this, an exclusive or an
auto-delete queue and its consumer(s) are not affected by a network
partition: they continue to work.
However, if a node is really lost, we need to clean up dead queue
records. This was already done for durable queues with both Mnesia and
Khepri. But with Khepri, transient queue records persist in the store
like durable queue records (unlike with Mnesia).
That's why this commit changes the clean-up function,
`rabbit_amqqueue:forget_all_durable/1` into
`rabbit_amqqueue:forget_all/1` which deletes all queue records of queues
that were hosted on the given node, regardless if they are transient or
durable.
In addition to this, the queue process will spawn a temporary process
who will try to delete the underlying record indefinitely if no other
processes are waiting for a reply from the queue process. That's the
case for queues that are deleted because of an internal event (like the
exclusive/auto-delete conditions). The queue process will exit, which
will notify connections that the queue is gone.
Thanks to this, the temporary process will do its best to delete the
record in case of a network partition, whether the consumers go away
during or after that partition. That said, the node monitor drives some
failsafe code that cleans up record if the queue process was killed
before it could delete its own record.
Fixes#12949, #12597, #14527.
[Why]
So far, when there was a network partition with Mnesia, the most popular
partition handling strategies restarted RabbitMQ nodes. Therefore,
`rabbit` would execute the boot steps and one of them would notify other
members of the cluster that "this RabbitMQ node is live".
With Khepri, nodes are not restarted anymore and thus, boot steps are
not executed at the end of a network partition. As a consequence, other
members are not notified that a member is back online.
[How]
When the node monitor receives the `nodeup` message (managed by Erlang,
meaning that "a remote Erlang node just connected to this node through
Erlang distribution"), a `node_up` message is sent to all cluster
members (meaning "RabbitMQ is now running on the originating node").
Yeah, very poor naming...
This lets the RabbitMQ node monitor know when other nodes running
RabbitMQ are back online and react accordingly.
If a node is restarted, it means that another node could receive the
`node_up` message twice. The actions behind it must be idempotent.
Loïc Hoguin
Michael Davis
Arnaud Cogoluègnes
dependabot[bot]
Michael Klishin
Aitor Pérez Cedres
Jean-Sébastien Pédron
Diana Parra Corbacho
Luke Bakken
Michael Klishin