rabbitmq-server/deps/rabbit/test/amqp_system_SUITE_data/fsharp-tests/Program.fs

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

565 lines
20 KiB
Forth
Raw Normal View History

// Learn more about F# at http://fsharp.org
open System
open System.Threading
open Amqp
open Amqp.Sasl
open Amqp.Framing
open Amqp.Types
let sleep (i: int) = System.Threading.Thread.Sleep i
[<AutoOpen>]
module RabbitClient =
open RabbitMQ.Client
let consume (m: IModel) queue autoAck f =
let consumer =
{ new DefaultBasicConsumer(m) with
member x.HandleBasicDeliver(consumerTag,
deliveryTag,
redelivered,
exchange,
routingKey,
props,
body) =
f deliveryTag props body }
m.ContinuationTimeout <- (TimeSpan.FromSeconds 115.)
(* m.BasicQos(0u, 100us, false) |> ignore *)
let consumerTag = m.BasicConsume(queue, autoAck, consumer)
{ new System.IDisposable with
member __.Dispose () =
m.BasicCancel(consumerTag) }
[<AutoOpen>]
module AmqpClient =
type AmqpConnection =
{ Conn : Connection
Session: Session }
interface IDisposable with
member x.Dispose() =
try
x.Conn.Close()
x.Session.Close()
with _ -> ()
let connect uri =
let c = Address uri |> Connection
let s = Session c
{ Conn = c; Session = s }
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
let connectAnon uri =
let c = Connection(Address uri, SaslProfile.Anonymous, null, null)
let s = Session c
{ Conn = c; Session = s }
let connectAnonWithOpen uri opn =
let c = Connection(Address uri, SaslProfile.Anonymous, opn, null)
let s = Session c
{ Conn = c; Session = s }
let senderReceiver ac name address =
let s = SenderLink(ac.Session, name + "-sender" , address)
let r = ReceiverLink(ac.Session, name + "-receiver", address)
r.SetCredit(100, true)
s, r
let receive (receiver: ReceiverLink) =
let rtd = receiver.Receive()
receiver.Accept rtd
rtd
let amqpSequence xs =
let l = Amqp.Types.List();
xs |> List.iter (l.Add >> ignore)
AmqpSequence(List = l)
[<AutoOpen>]
module Test =
2023-05-18 20:25:53 +08:00
let assertEqual a b =
if a <> b then
failwith (sprintf "Expected: %A\r\nGot: %A" a b)
let assertNotNull a =
if a = null then
failwith (sprintf "Null not expected")
let assertTrue b =
if not b then
failwith (sprintf "Expected True got False!")
let sampleTypes =
["hi" :> obj
"" :> obj
"hi"B :> obj
""B :> obj
Array.create 1000 50uy :> obj
true :> obj
0y :> obj
0uy :> obj
Byte.MaxValue :> obj
0s :> obj
Int16.MaxValue :> obj
0 :> obj
Int32.MaxValue :> obj
0L :> obj
Int64.MaxValue :> obj
0us :> obj
UInt16.MaxValue :> obj
0u :> obj
UInt32.MaxValue :> obj
0ul :> obj
UInt64.MaxValue :> obj
null :> obj
"\uFFF9" :> obj
Amqp.Types.Symbol("Symbol") :> obj
DateTime.Parse("2008-11-01T19:35:00.0000000Z").ToUniversalTime() :> obj
Guid("f275ea5e-0c57-4ad7-b11a-b20c563d3b71") :> obj
]
let testOutcome uri (attach: Attach) (cond: string) =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use ac = connectAnon uri
let trySet (mre: AutoResetEvent) =
try mre.Set() |> ignore with _ -> ()
use mre = new System.Threading.AutoResetEvent(false)
let mutable errorName = null
ac.Session.add_Closed (
new ClosedCallback (fun o err -> errorName <- string err.Condition; trySet mre))
let attached = new OnAttached (
fun l attach -> errorName <- null; trySet mre)
let receiver = ReceiverLink(ac.Session, "test-receiver", attach, attached)
mre.WaitOne(1000) |> ignore
if cond = null then
receiver.Close()
assertEqual cond errorName
2023-05-18 20:25:53 +08:00
let no_routes_is_released uri =
// tests that a message sent to an exchange that resolves no routes for the
// binding key returns the Released outcome, rather than Accepted
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use ac = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let address = "/exchanges/no_routes_is_released"
2023-05-18 20:25:53 +08:00
let sender = SenderLink(ac.Session, "released-sender", address)
let trySet (mre: AutoResetEvent) =
try mre.Set() |> ignore with _ -> ()
let mutable outcome = null
use mre = new System.Threading.AutoResetEvent(false)
let msg = new Message("hi"B,
Header = Header(Ttl = 500u),
Properties = new Properties(Subject = "no_routes"))
let cb = new OutcomeCallback (fun l m o x -> outcome <- o; trySet mre)
sender.Send(msg, cb, null)
mre.WaitOne(1000) |> ignore
match outcome with
| :? Released ->
()
| _ ->
failwith (sprintf "Expected: Released\r\nGot: %A" outcome)
()
let roundtrip uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/roundtrip"
for body in sampleTypes do
let corr = "correlation"
new Message(body,
Header = Header(Ttl = 500u),
Properties = new Properties(CorrelationId = corr))
|> sender.Send
let rtd = receive receiver
assertEqual body rtd.Body
assertTrue (rtd.Header.Ttl <= 500u)
assertEqual rtd.Properties.CorrelationId corr
()
let streams uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
let name = "streams-test"
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let address = "/queues/streams"
let sender = SenderLink(c.Session, name + "-sender" , address)
//for body in sampleTypes do
let body = "hi"B :> obj
let corr = "correlation"
new Message(body,
Properties = new Properties(CorrelationId = corr))
|> sender.Send
//TODO wait for settlement
let specs = [box("first");
box("last");
box("10m");
box(0)]
for spec in specs do
printfn "testing streams spec %A" spec
let filterSet = Map()
filterSet.Add(Symbol "rabbitmq:stream-offset-spec", spec)
let source = Source(Address = address,
FilterSet = filterSet)
let attach = Attach(Source = source)
let attached = new OnAttached (fun _ _ -> ())
let receiver = ReceiverLink(c.Session, Guid.NewGuid().ToString(), attach, attached)
receiver.SetCredit(100, true)
let rtd = receiver.Receive()
assertNotNull rtd
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
assertEqual 3 rtd.MessageAnnotations.Map.Count
assertTrue (rtd.MessageAnnotations.Map.ContainsKey(Symbol "x-stream-offset"))
assertTrue (rtd.MessageAnnotations.Map.ContainsKey(Symbol "x-exchange"))
assertTrue (rtd.MessageAnnotations.Map.ContainsKey(Symbol "x-routing-key"))
assertEqual body rtd.Body
assertEqual rtd.Properties.CorrelationId corr
receiver.Close()
()
open RabbitMQ.Client
let roundtrip_to_amqp_091 uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let q = "roundtrip_to_amqp_091"
let target = "/queues/roundtrip_to_amqp_091"
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
let corr = "correlation"
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender = SenderLink(c.Session, q + "-sender" , target)
new Message("hi"B, Header = Header(),
Properties = new Properties(CorrelationId = corr))
|> sender.Send
System.Threading.Thread.Sleep 500
let cf = ConnectionFactory()
cf.Uri <- Uri uri
use c = cf.CreateConnection()
use m = c.CreateModel()
use h = new AutoResetEvent(false)
let mutable id : string = null
let con = consume m q false (fun deliveryTag props body ->
printfn "got %A" props.CorrelationId
id <- props.CorrelationId
h.Set() |> ignore
m.BasicAck(deliveryTag, false) |> ignore)
h.WaitOne() |> ignore
assertEqual id corr
()
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let default_outcome uri =
for (defOut, cond, defObj) in
["amqp:accepted:list", null, Accepted() :> Outcome
"amqp:rejected:list", null, Rejected() :> Outcome
"amqp:released:list", null, Released() :> Outcome] do
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let source = new Source(Address = "/queues/default_outcome",
DefaultOutcome = defObj)
let attach = new Attach (Source = source,
Target = Target())
testOutcome uri attach cond
let outcomes uri =
for (outcome, cond) in
["amqp:accepted:list", null
"amqp:rejected:list", null
"amqp:released:list", null
"amqp:modified:list", null
"amqp:madeup:list", "amqp:not-implemented"] do
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let source = new Source(Address = "/queues/outcomes",
Outcomes = [| Symbol outcome |])
let attach = new Attach (Source = source,
Target = Target())
testOutcome uri attach cond
let fragmentation uri =
for frameSize, size in
Fix AMQP fragmentation test in CI ``` make -C deps/rabbit ct-amqp_system t=dotnet:fragmentation ``` fails in the new make CI with: ``` amqp_system_SUITE > dotnet > fragmentation #1. {error,{{badmatch,{error,134, "Unhandled exception. Amqp.AmqpException: Invalid frame size:527, maximum frame size:512.\n at Amqp.Connection.ThrowIfClosed(String operation)\n at Amqp.Connection.AddSession(Session session)\n at Amqp.Session..ctor(Connection connection, Begin begin, OnBegin onBegin)\n at Amqp.Session..ctor(Connection connection)\n at Program.AmqpClient.connectWithOpen(String uri, Open opn) in /home/runner/work/rabbitmq-server/rabbitmq-server/deps/rabbit/test/amqp_system_SUITE_data/fsharp-tests/Program.fs:line 53\n at Program.Test.fragmentation(String uri) in /home/runner/work/rabbitmq-server/rabbitmq-server/deps/rabbit/test/amqp_system_SUITE_data/fsharp-tests/Program.fs:line 284\n at Program.main(String[] argv) in /home/runner/work/rabbitmq-server/rabbitmq-server/deps/rabbit/test/amqp_system_SUITE_data/fsharp-tests/Program.fs:line 533\n"}}, [{amqp_system_SUITE,run_dotnet_test,2, [{file,"amqp_system_SUITE.erl"}, {line,228}]}, {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]}, {test_server,run_test_case_eval1,6, [{file,"test_server.erl"},{line,1302}]}, {test_server,run_test_case_eval,9, [{file,"test_server.erl"},{line,1234}]}]}} ``` RabbitMQ includes its node name and cluster name in the open frame to the client. Running this test locally shows an open frame size of 467 bytes. The suspicion is that the node name and cluster name in CI is longer causing the open frame from RabbitMQ to the client to exceed the frame size of 512 bytes.
2024-04-26 15:36:49 +08:00
[1024u, 1024
1024u, 1100
1024u, 2048
2048u, 2048] do
let addr = Address uri
let opn = Open(ContainerId = Guid.NewGuid().ToString(),
HostName = addr.Host, ChannelMax = 256us,
MaxFrameSize = frameSize)
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnonWithOpen uri opn
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/fragmentation"
let m = new Message(String.replicate size "a")
sender.Send m
let m' = receive receiver
assertEqual (m.Body) (m'.Body)
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let message_annotations uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/message_annotations"
let ann = MessageAnnotations()
let k1 = Symbol "key1"
let k2 = Symbol "key2"
ann.[Symbol "key1"] <- "value1"
ann.[Symbol "key2"] <- "value2"
let m = new Message("testing annotations", MessageAnnotations = ann)
sender.Send m
let m' = receive receiver
assertEqual m.Body m'.Body
assertEqual (m.MessageAnnotations.Descriptor) (m'.MessageAnnotations.Descriptor)
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
// our 2 custom annotations + x-exchange + x-routing-key = 4
assertEqual 4 (m'.MessageAnnotations.Map.Count)
assertTrue (m.MessageAnnotations.[k1] = m'.MessageAnnotations.[k1])
assertTrue (m.MessageAnnotations.[k2] = m'.MessageAnnotations.[k2])
let footer uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/footer"
let footer = Footer()
let k1 = Symbol "key1"
let k2 = Symbol "key2"
footer.[Symbol "key1"] <- "value1"
footer.[Symbol "key2"] <- "value2"
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
let m = new Message("testing footer", Footer = footer)
sender.Send m
let m' = receive receiver
assertEqual m.Body m'.Body
assertEqual (m.Footer.Descriptor) (m'.Footer.Descriptor)
assertEqual 2 (m'.Footer.Map.Count)
assertTrue (m.Footer.[k1] = m'.Footer.[k1])
assertTrue (m.Footer.[k2] = m'.Footer.[k2])
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let data_types uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/data_types"
let aSeq = amqpSequence sampleTypes
(new Message(aSeq)) |> sender.Send
let rtd = receive receiver
let amqpSeq = rtd.Body :?> AmqpSequence
for a in amqpSeq.List do
List.exists ((=) a) sampleTypes |> assertTrue
let reject uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/reject"
new Message "testing reject" |> sender.Send
let m = receiver.Receive()
receiver.Reject(m)
assertEqual null (receiver.Receive(TimeSpan.FromMilliseconds 100.))
let redelivery uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/redelivery"
new Message "testing redelivery" |> sender.Send
let m = receiver.Receive()
assertTrue (m.Header.FirstAcquirer)
c.Session.Close()
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let session = Session(c.Conn)
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let receiver = ReceiverLink(session, "test-receiver", "/queues/redelivery")
let m' = receive receiver
assertEqual (m.Body :?> string) (m'.Body :?> string)
assertTrue (not m'.Header.FirstAcquirer)
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
assertEqual 1u (m'.Header.DeliveryCount)
assertEqual null (receiver.Receive(TimeSpan.FromMilliseconds 100.))
session.Close()
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let released uri =
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let sender, receiver = senderReceiver c "test" "/queues/released"
new Message "testing released" |> sender.Send
let m = receiver.Receive()
assertTrue (m.Header.FirstAcquirer)
receiver.SetCredit(0, false)
receiver.Release m
let m' = receive receiver
assertEqual (m.Body :?> string) (m'.Body :?> string)
assertTrue (not m'.Header.FirstAcquirer)
assertEqual 0u (m'.Header.DeliveryCount)
assertEqual null (receiver.Receive(TimeSpan.FromMilliseconds 100.))
c.Session.Close()
let routing uri =
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
for target, source, toProp in
[
"/queues/test", "/queues/test", ""
"/exchanges/amq.fanout", "/queues/fanout_q", ""
"/exchanges/amq.direct/direct_q", "/queues/direct_q", ""
null, "/queues/direct_q", "/exchanges/amq.direct/direct_q"
"/queues/transient_q", "/queues/transient_q", ""
"/queues/durable_q", "/queues/durable_q", ""
"/queues/quorum_q", "/queues/quorum_q", ""
"/queues/stream_q", "/queues/stream_q", ""
"/queues/autodel_q", "/queues/autodel_q", ""] do
let rnd = Random()
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use c = connectAnon uri
let sender = SenderLink(c.Session, "test-sender", target)
let receiver = ReceiverLink(c.Session, "test-receiver", source)
receiver.SetCredit(100, true)
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
use m = new Message(rnd.Next(10000),
Properties = Properties(To = toProp))
sender.Send m
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
(* printfn "%s %s %s %A" target source routingKey *)
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let m' = receiver.Receive(TimeSpan.FromMilliseconds 3000.)
receiver.Accept m'
assertTrue (m' <> null)
assertEqual (m.Body :?> int) (m'.Body :?> int)
let invalidRoutes uri =
for dest, cond in
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
["/exchanges/missing", "amqp:not-found"
"/fruit/orange", "amqp:invalid-field"] do
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
use ac = connectAnon uri
let trySet (mre: AutoResetEvent) =
try mre.Set() |> ignore with _ -> ()
let mutable errorName = null
use mre = new System.Threading.AutoResetEvent(false)
ac.Session.add_Closed (
new ClosedCallback (fun _ err -> errorName <- err.Condition; trySet mre))
let attached = new OnAttached (fun _ _ -> trySet mre)
let sender = new SenderLink(ac.Session, "test-sender",
Target(Address = dest), attached);
mre.WaitOne() |> ignore
try
let receiver = ReceiverLink(ac.Session, "test-receiver", dest)
receiver.Close()
with
| :? Amqp.AmqpException as ae ->
Support AMQP 1.0 natively ## What Similar to Native MQTT in #5895, this commits implements Native AMQP 1.0. By "native", we mean do not proxy via AMQP 0.9.1 anymore. ## Why Native AMQP 1.0 comes with the following major benefits: 1. Similar to Native MQTT, this commit provides better throughput, latency, scalability, and resource usage for AMQP 1.0. See https://blog.rabbitmq.com/posts/2023/03/native-mqtt for native MQTT improvements. See further below for some benchmarks. 2. Since AMQP 1.0 is not limited anymore by the AMQP 0.9.1 protocol, this commit allows implementing more AMQP 1.0 features in the future. Some features are already implemented in this commit (see next section). 3. Simpler, better understandable, and more maintainable code. Native AMQP 1.0 as implemented in this commit has the following major benefits compared to AMQP 0.9.1: 4. Memory and disk alarms will only stop accepting incoming TRANSFER frames. New connections can still be created to consume from RabbitMQ to empty queues. 5. Due to 4. no need anymore for separate connections for publishers and consumers as we currently recommended for AMQP 0.9.1. which potentially halves the number of physical TCP connections. 6. When a single connection sends to multiple target queues, a single slow target queue won't block the entire connection. Publisher can still send data quickly to all other target queues. 7. A publisher can request whether it wants publisher confirmation on a per-message basis. In AMQP 0.9.1 publisher confirms are configured per channel only. 8. Consumers can change their "prefetch count" dynamically which isn't possible in our AMQP 0.9.1 implementation. See #10174 9. AMQP 1.0 is an extensible protocol This commit also fixes dozens of bugs present in the AMQP 1.0 plugin in RabbitMQ 3.x - most of which cannot be backported due to the complexity and limitations of the old 3.x implementation. This commit contains breaking changes and is therefore targeted for RabbitMQ 4.0. ## Implementation details 1. Breaking change: With Native AMQP, the behaviour of ``` Convert AMQP 0.9.1 message headers to application properties for an AMQP 1.0 consumer amqp1_0.convert_amqp091_headers_to_app_props = false | true (default false) Convert AMQP 1.0 Application Properties to AMQP 0.9.1 headers amqp1_0.convert_app_props_to_amqp091_headers = false | true (default false) ``` will break because we always convert according to the message container conversions. For example, AMQP 0.9.1 x-headers will go into message-annotations instead of application properties. Also, `false` won’t be respected since we always convert the headers with message containers. 2. Remove rabbit_queue_collector rabbit_queue_collector is responsible for synchronously deleting exclusive queues. Since the AMQP 1.0 plugin never creates exclusive queues, rabbit_queue_collector doesn't need to be started in the first place. This will save 1 Erlang process per AMQP 1.0 connection. 3. 7 processes per connection + 1 process per session in this commit instead of 7 processes per connection + 15 processes per session in 3.x Supervision hierarchy got re-designed. 4. Use 1 writer process per AMQP 1.0 connection AMQP 0.9.1 uses a separate rabbit_writer Erlang process per AMQP 0.9.1 channel. Prior to this commit, AMQP 1.0 used a separate rabbit_amqp1_0_writer process per AMQP 1.0 session. Advantage of single writer proc per session (prior to this commit): * High parallelism for serialising packets if multiple sessions within a connection write heavily at the same time. This commit uses a single writer process per AMQP 1.0 connection that is shared across all AMQP 1.0 sessions. Advantages of single writer proc per connection (this commit): * Lower memory usage with hundreds of thousands of AMQP 1.0 sessions * Less TCP and IP header overhead given that the single writer process can accumulate across all sessions bytes before flushing the socket. In other words, this commit decides that a reader / writer process pair per AMQP 1.0 connection is good enough for bi-directional TRANSFER flows. Having a writer per session is too heavy. We still ensure high throughput by having separate reader, writer, and session processes. 5. Transform rabbit_amqp1_0_writer into gen_server Why: Prior to this commit, when clicking on the AMQP 1.0 writer process in observer, the process crashed. Instead of handling all these debug messages of the sys module, it's better to implement a gen_server. There is no advantage of using a special OTP process over gen_server for the AMQP 1.0 writer. gen_server also provides cleaner format status output. How: Message callbacks return a timeout of 0. After all messages in the inbox are processed, the timeout message is handled by flushing any pending bytes. 6. Remove stats timer from writer AMQP 1.0 connections haven't emitted any stats previously. 7. When there are contiguous queue confirmations in the session process mailbox, batch them. When the confirmations are sent to the publisher, a single DISPOSITION frame is sent for contiguously confirmed delivery IDs. This approach should be good enough. However it's sub optimal in scenarios where contiguous delivery IDs that need confirmations are rare, for example: * There are multiple links in the session with different sender settlement modes and sender publishes across these links interleaved. * sender settlement mode is mixed and sender publishes interleaved settled and unsettled TRANSFERs. 8. Introduce credit API v2 Why: The AMQP 0.9.1 credit extension which is to be removed in 4.0 was poorly designed since basic.credit is a synchronous call into the queue process blocking the entire AMQP 1.0 session process. How: Change the interactions between queue clients and queue server implementations: * Clients only request a credit reply if the FLOW's `echo` field is set * Include all link flow control state held by the queue process into a new credit_reply queue event: * `available` after the queue sends any deliveries * `link-credit` after the queue sends any deliveries * `drain` which allows us to combine the old queue events send_credit_reply and send_drained into a single new queue event credit_reply. * Include the consumer tag into the credit_reply queue event such that the AMQP 1.0 session process can process any credit replies asynchronously. Link flow control state `delivery-count` also moves to the queue processes. The new interactions are hidden behind feature flag credit_api_v2 to allow for rolling upgrades from 3.13 to 4.0. 9. Use serial number arithmetic in quorum queues and session process. 10. Completely bypass the rabbit_limiter module for AMQP 1.0 flow control. The goal is to eventually remove the rabbit_limiter module in 4.0 since AMQP 0.9.1 global QoS will be unsupported in 4.0. This commit lifts the AMQP 1.0 link flow control logic out of rabbit_limiter into rabbit_queue_consumers. 11. Fix credit bug for streams: AMQP 1.0 settlements shouldn't top up link credit, only FLOW frames should top up link credit. 12. Allow sender settle mode unsettled for streams since AMQP 1.0 acknowledgements to streams are no-ops (currently). 13. Fix AMQP 1.0 client bugs Auto renewing credits should not be related to settling TRANSFERs. Remove field link_credit_unsettled as it was wrong and confusing. Prior to this commit auto renewal did not work when the sender uses sender settlement mode settled. 14. Fix AMQP 1.0 client bugs The wrong outdated Link was passed to function auto_flow/2 15. Use osiris chunk iterator Only hold messages of uncompressed sub batches in memory if consumer doesn't have sufficient credits. Compressed sub batches are skipped for non Stream protocol consumers. 16. Fix incoming link flow control Always use confirms between AMQP 1.0 queue clients and queue servers. As already done internally by rabbit_fifo_client and rabbit_stream_queue, use confirms for classic queues as well. 17. Include link handle into correlation when publishing messages to target queues such that session process can correlate confirms from target queues to incoming links. 18. Only grant more credits to publishers if publisher hasn't sufficient credits anymore and there are not too many unconfirmed messages on the link. 19. Completely ignore `block` and `unblock` queue actions and RabbitMQ credit flow between classic queue process and session process. 20. Link flow control is independent between links. A client can refer to a queue or to an exchange with multiple dynamically added target queues. Multiple incoming links can also fan in to the same queue. However the link topology looks like, this commit ensures that each link is only granted more credits if that link isn't overloaded. 21. A connection or a session can send to many different queues. In AMQP 0.9.1, a single slow queue will lead to the entire channel, and then entire connection being blocked. This commit makes sure that a single slow queue from one link won't slow down sending on other links. For example, having link A sending to a local classic queue and link B sending to 5 replica quorum queue, link B will naturally grant credits slower than link A. So, despite the quorum queue being slower in confirming messages, the same AMQP 1.0 connection and session can still pump data very fast into the classic queue. 22. If cluster wide memory or disk alarm occurs. Each session sends a FLOW with incoming-window to 0 to sending client. If sending clients don’t obey, force disconnect the client. If cluster wide memory alarm clears: Each session resumes with a FLOW defaulting to initial incoming-window. 23. All operations apart of publishing TRANSFERS to RabbitMQ can continue during cluster wide alarms, specifically, attaching consumers and consuming, i.e. emptying queues. There is no need for separate AMQP 1.0 connections for publishers and consumers as recommended in our AMQP 0.9.1 implementation. 24. Flow control summary: * If queue becomes bottleneck, that’s solved by slowing down individual sending links (AMQP 1.0 link flow control). * If session becomes bottleneck (more unlikely), that’s solved by AMQP 1.0 session flow control. * If connection becomes bottleneck, it naturally won’t read fast enough from the socket causing TCP backpressure being applied. Nowhere will RabbitMQ internal credit based flow control (i.e. module credit_flow) be used on the incoming AMQP 1.0 message path. 25. Register AMQP sessions Prefer local-only pg over our custom pg_local implementation as pg is a better process group implementation than pg_local. pg_local was identified as bottleneck in tests where many MQTT clients were disconnected at once. 26. Start a local-only pg when Rabbit boots: > A scope can be kept local-only by using a scope name that is unique cluster-wide, e.g. the node name: > pg:start_link(node()). Register AMQP 1.0 connections and sessions with pg. In future we should remove pg_local and instead use the new local-only pg for all registered processes such as AMQP 0.9.1 connections and channels. 27. Requeue messages if link detached Although the spec allows to settle delivery IDs on detached links, RabbitMQ does not respect the 'closed' field of the DETACH frame and therefore handles every DETACH frame as closed. Since the link is closed, we expect every outstanding delivery to be requeued. In addition to consumer cancellation, detaching a link therefore causes in flight deliveries to be requeued. Note that this behaviour is different from merely consumer cancellation in AMQP 0.9.1: "After a consumer is cancelled there will be no future deliveries dispatched to it. Note that there can still be "in flight" deliveries dispatched previously. Cancelling a consumer will neither discard nor requeue them." [https://www.rabbitmq.com/consumers.html#unsubscribing] An AMQP receiver can first drain, and then detach to prevent "in flight" deliveries 28. Init AMQP session with BEGIN frame Similar to how there can't be an MQTT processor without a CONNECT frame, there can't be an AMQP session without a BEGIN frame. This allows having strict dialyzer types for session flow control fields (i.e. not allowing 'undefined'). 29. Move serial_number to AMQP 1.0 common lib such that it can be used by both AMQP 1.0 server and client 30. Fix AMQP client to do serial number arithmetic. 31. AMQP client: Differentiate between delivery-id and transfer-id for better understandability. 32. Fix link flow control in classic queues This commit fixes ``` java -jar target/perf-test.jar -ad false -f persistent -u cq -c 3000 -C 1000000 -y 0 ``` followed by ``` ./omq -x 0 amqp -T /queue/cq -D 1000000 --amqp-consumer-credits 2 ``` Prior to this commit, (and on RabbitMQ 3.x) the consuming would halt after around 8 - 10,000 messages. The bug was that in flight messages from classic queue process to session process were not taken into account when topping up credit to the classic queue process. Fixes #2597 The solution to this bug (and a much cleaner design anyway independent of this bug) is that queues should hold all link flow control state including the delivery-count. Hence, when credit API v2 is used the delivery-count will be held by the classic queue process, quorum queue process, and stream queue client instead of managing the delivery-count in the session. 33. The double level crediting between (a) session process and rabbit_fifo_client, and (b) rabbit_fifo_client and rabbit_fifo was removed. Therefore, instead of managing 3 separate delivery-counts (i. session, ii. rabbit_fifo_client, iii. rabbit_fifo), only 1 delivery-count is used in rabbit_fifo. This is a big simplification. 34. This commit fixes quorum queues without bumping the machine version nor introducing new rabbit_fifo commands. Whether credit API v2 is used is solely determined at link attachment time depending on whether feature flag credit_api_v2 is enabled. Even when that feature flag will be enabled later on, this link will keep using credit API v1 until detached (or the node is shut down). Eventually, after feature flag credit_api_v2 has been enabled and a subsequent rolling upgrade, all links will use credit API v2. This approach is safe and simple. The 2 alternatives to move delivery-count from the session process to the queue processes would have been: i. Explicit feature flag credit_api_v2 migration function * Can use a gen_server:call and only finish migration once all delivery-counts were migrated. Cons: * Extra new message format just for migration is required. * Risky as migration will fail if a target queue doesn’t reply. ii. Session always includes DeliveryCountSnd when crediting to the queue: Cons: * 2 delivery counts will be hold simultaneously in session proc and queue proc; could be solved by deleting the session proc’s delivery-count for credit-reply * What happens if the receiver doesn’t provide credit for a very long time? Is that a problem? 35. Support stream filtering in AMQP 1.0 (by @acogoluegnes) Use the x-stream-filter-value message annotation to carry the filter value in a published message. Use the rabbitmq:stream-filter and rabbitmq:stream-match-unfiltered filters when creating a receiver that wants to filter out messages from a stream. 36. Remove credit extension from AMQP 0.9.1 client 37. Support maintenance mode closing AMQP 1.0 connections. 38. Remove AMQP 0.9.1 client dependency from AMQP 1.0 implementation. 39. Move AMQP 1.0 plugin to the core. AMQP 1.0 is enabled by default. The old rabbitmq_amqp1_0 plugin will be kept as a no-op plugin to prevent deployment tools from failing that execute: ``` rabbitmq-plugins enable rabbitmq_amqp1_0 rabbitmq-plugins disable rabbitmq_amqp1_0 ``` 40. Breaking change: Remove CLI command `rabbitmqctl list_amqp10_connections`. Instead, list both AMQP 0.9.1 and AMQP 1.0 connections in `list_connections`: ``` rabbitmqctl list_connections protocol Listing connections ... protocol {1, 0} {0,9,1} ``` ## Benchmarks ### Throughput & Latency Setup: * Single node Ubuntu 22.04 * Erlang 26.1.1 Start RabbitMQ: ``` make run-broker PLUGINS="rabbitmq_management rabbitmq_amqp1_0" FULL=1 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3" ``` Predeclare durable classic queue cq1, durable quorum queue qq1, durable stream queue sq1. Start client: https://github.com/ssorj/quiver https://hub.docker.com/r/ssorj/quiver/tags (digest 453a2aceda64) ``` docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest bash-5.1# quiver --version quiver 0.4.0-SNAPSHOT ``` 1. Classic queue ``` quiver //host.docker.internal//amq/queue/cq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ............................................... 73.8 seconds Sender rate .......................................... 13,548 messages/s Receiver rate ........................................ 13,547 messages/s End-to-end rate ...................................... 13,547 messages/s Latencies by percentile: 0% ........ 0 ms 90.00% ........ 9 ms 25% ........ 2 ms 99.00% ....... 14 ms 50% ........ 4 ms 99.90% ....... 17 ms 100% ....... 26 ms 99.99% ....... 24 ms ``` RabbitMQ 3.x (main branch as of 30 January 2024): ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 6 73.6 2.1 3,217 1,607 0 8.0 511 4.1 163,580 16,367 2 74.1 4.1 3,217 0 0 8.0 0 6.1 229,114 32,767 3 74.1 6.1 3,217 0 0 8.0 0 8.1 261,880 16,367 2 74.1 8.1 67,874 32,296 8 8.2 7,662 10.1 294,646 16,367 2 74.1 10.1 67,874 0 0 8.2 0 12.1 360,180 32,734 3 74.1 12.1 67,874 0 0 8.2 0 14.1 392,946 16,367 3 74.1 14.1 68,604 365 0 8.2 12,147 16.1 458,480 32,734 3 74.1 16.1 68,604 0 0 8.2 0 18.1 491,246 16,367 2 74.1 18.1 68,604 0 0 8.2 0 20.1 556,780 32,767 4 74.1 20.1 68,604 0 0 8.2 0 22.1 589,546 16,375 2 74.1 22.1 68,604 0 0 8.2 0 receiver timed out 24.1 622,312 16,367 2 74.1 24.1 68,604 0 0 8.2 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/cq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-otujr23y' returned non-zero exit status 1. ``` 2. Quorum queue: ``` quiver //host.docker.internal//amq/queue/qq1 --durable --count 1m --duration 10m --body-size 12 --credit 1000 ``` This commit: ``` Count ............................................. 1,000,000 messages Duration .............................................. 101.4 seconds Sender rate ........................................... 9,867 messages/s Receiver rate ......................................... 9,868 messages/s End-to-end rate ....................................... 9,865 messages/s Latencies by percentile: 0% ....... 11 ms 90.00% ....... 23 ms 25% ....... 15 ms 99.00% ....... 28 ms 50% ....... 18 ms 99.90% ....... 33 ms 100% ....... 49 ms 99.99% ....... 47 ms ``` RabbitMQ 3.x: ``` ---------------------- Sender ----------------------- --------------------- Receiver ---------------------- -------- Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Time [s] Count [m] Rate [m/s] CPU [%] RSS [M] Lat [ms] ----------------------------------------------------- ----------------------------------------------------- -------- 2.1 130,814 65,342 9 69.9 2.1 18,430 9,206 5 7.6 1,221 4.1 163,580 16,375 5 70.2 4.1 18,867 218 0 7.6 2,168 6.1 229,114 32,767 6 70.2 6.1 18,867 0 0 7.6 0 8.1 294,648 32,734 7 70.2 8.1 18,867 0 0 7.6 0 10.1 360,182 32,734 6 70.2 10.1 18,867 0 0 7.6 0 12.1 425,716 32,767 6 70.2 12.1 18,867 0 0 7.6 0 receiver timed out 14.1 458,482 16,367 5 70.2 14.1 18,867 0 0 7.6 0 quiver: error: PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. Traceback (most recent call last): File "/usr/local/lib/quiver/python/quiver/pair.py", line 144, in run _plano.wait(receiver, check=True) File "/usr/local/lib/quiver/python/plano/main.py", line 1243, in wait raise PlanoProcessError(proc) plano.main.PlanoProcessError: Command 'quiver-arrow receive //host.docker.internal//amq/queue/qq1 --impl qpid-proton-c --duration 10m --count 1m --rate 0 --body-size 12 --credit 1000 --transaction-size 0 --timeout 10 --durable --output /tmp/quiver-b1gcup43' returned non-zero exit status 1. ``` 3. Stream: ``` quiver-arrow send //host.docker.internal//amq/queue/sq1 --durable --count 1m -d 10m --summary --verbose ``` This commit: ``` Count ............................................. 1,000,000 messages Duration ................................................ 8.7 seconds Message rate ........................................ 115,154 messages/s ``` RabbitMQ 3.x: ``` Count ............................................. 1,000,000 messages Duration ............................................... 21.2 seconds Message rate ......................................... 47,232 messages/s ``` ### Memory usage Start RabbitMQ: ``` ERL_MAX_PORTS=3000000 RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" make run-broker PLUGINS="rabbitmq_amqp1_0" FULL=1 RABBITMQ_CONFIG_FILE="rabbitmq.conf" ``` ``` /bin/cat rabbitmq.conf tcp_listen_options.sndbuf = 2048 tcp_listen_options.recbuf = 2048 vm_memory_high_watermark.relative = 0.95 vm_memory_high_watermark_paging_ratio = 0.95 loopback_users = none ``` Create 50k connections with 2 sessions per connection, i.e. 100k session in total: ```go package main import ( "context" "log" "time" "github.com/Azure/go-amqp" ) func main() { for i := 0; i < 50000; i++ { conn, err := amqp.Dial(context.TODO(), "amqp://nuc", &amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()}) if err != nil { log.Fatal("dialing AMQP server:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } _, err = conn.NewSession(context.TODO(), nil) if err != nil { log.Fatal("creating AMQP session:", err) } } log.Println("opened all connections") time.Sleep(5 * time.Hour) } ``` This commit: ``` erlang:memory(). [{total,4586376480}, {processes,4025898504}, {processes_used,4025871040}, {system,560477976}, {atom,1048841}, {atom_used,1042841}, {binary,233228608}, {code,21449982}, {ets,108560464}] erlang:system_info(process_count). 450289 ``` 7 procs per connection + 1 proc per session. (7 + 2*1) * 50,000 = 450,000 procs RabbitMQ 3.x: ``` erlang:memory(). [{total,15168232704}, {processes,14044779256}, {processes_used,14044755120}, {system,1123453448}, {atom,1057033}, {atom_used,1052587}, {binary,236381264}, {code,21790238}, {ets,391423744}] erlang:system_info(process_count). 1850309 ``` 7 procs per connection + 15 per session (7 + 2*15) * 50,000 = 1,850,000 procs 50k connections + 100k session require with this commit: 4.5 GB in RabbitMQ 3.x: 15 GB ## Future work 1. More efficient parser and serializer 2. TODO in mc_amqp: Do not store the parsed message on disk. 3. Implement both AMQP HTTP extension and AMQP management extension to allow AMQP clients to create RabbitMQ objects (queues, exchanges, ...).
2023-07-21 18:29:07 +08:00
assertEqual (Symbol cond) (ae.Error.Condition)
| _ -> failwith "invalid expection thrown"
2017-05-09 22:28:56 +08:00
let authFailure uri =
try
let u = Uri uri
let uri = sprintf "amqp://blah:blah@%s:%i" u.Host u.Port
let c = Connection(Address uri)
failwith "expected exception not received"
with
| :? Amqp.AmqpException ->
()
let accessFailureSend uri =
try
let u = Uri uri
let uri = sprintf "amqp://access_failure:boo@%s:%i" u.Host u.Port
use ac = connect uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let dest = "/queues/test"
ac.Session.add_Closed (
new ClosedCallback (fun _ err -> printfn "session err %A" err.Condition
))
let sender = new SenderLink(ac.Session, "test-sender", dest)
sender.Send(new Message "hi", TimeSpan.FromSeconds 15.)
failwith "expected exception not received"
with
| :? Amqp.AmqpException as ex ->
printfn "Exception %A" ex
()
let accessFailure uri =
try
let u = Uri uri
let uri = sprintf "amqp://access_failure:boo@%s:%i" u.Host u.Port
use ac = connect uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let dest = "/queues/test"
let receiver = ReceiverLink(ac.Session, "test-receiver", dest)
receiver.Close()
failwith "expected exception not received"
with
| :? Amqp.AmqpException as ex ->
printfn "Exception %A" ex
()
let accessFailureNotAllowed uri =
try
let u = Uri uri
let uri = sprintf "amqp://access_failure_not_allowed:boo@%s:%i" u.Host u.Port
use ac = connect uri
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
let dest = "/queues/test"
let receiver = ReceiverLink(ac.Session, "test-receiver", dest)
receiver.Close()
failwith "expected exception not received"
with
| :? Amqp.AmqpException as ex ->
printfn "Exception %A" ex
()
let (|AsLower|) (s: string) =
match s with
| null -> null
| _ -> s.ToLowerInvariant()
[<EntryPoint>]
let main argv =
match List.ofArray argv with
2017-05-09 22:28:56 +08:00
| [AsLower "auth_failure"; uri] ->
authFailure uri
0
| [AsLower "access_failure"; uri] ->
accessFailure uri
0
| [AsLower "access_failure_not_allowed"; uri] ->
accessFailureNotAllowed uri
0
| [AsLower "access_failure_send"; uri] ->
accessFailureSend uri
0
| [AsLower "roundtrip"; uri] ->
roundtrip uri
0
| [AsLower "roundtrip_to_amqp_091"; uri] ->
roundtrip_to_amqp_091 uri
0
| [AsLower "data_types"; uri] ->
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
data_types uri
0
| [AsLower "default_outcome"; uri] ->
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
default_outcome uri
0
| [AsLower "outcomes"; uri] ->
outcomes uri
0
| [AsLower "fragmentation"; uri] ->
fragmentation uri
0
| [AsLower "message_annotations"; uri] ->
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
message_annotations uri
0
| [AsLower "footer"; uri] ->
footer uri
0
| [AsLower "reject"; uri] ->
reject uri
0
| [AsLower "redelivery"; uri] ->
redelivery uri
0
Quorum queues v4 (#10637) This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from https://github.com/rabbitmq/rabbitmq-server/pull/6292#issue-1431275848 doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in https://github.com/rabbitmq/rabbitmq-server/pull/10637#discussion_r1681002352 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2024-08-08 15:48:27 +08:00
| [AsLower "released"; uri] ->
released uri
0
| [AsLower "routing"; uri] ->
routing uri
0
| [AsLower "invalid_routes"; uri] ->
invalidRoutes uri
0
| [AsLower "streams"; uri] ->
streams uri
0
2023-05-18 20:25:53 +08:00
| [AsLower "no_routes_is_released"; uri] ->
no_routes_is_released uri
0
| _ ->
printfn "test %A not found. usage: <test> <uri>" argv
1