From 96309b0fc973b5366fa89788ee3ed69385e15c6c Mon Sep 17 00:00:00 2001 From: Michael Klishin Date: Thu, 2 Nov 2023 15:06:02 -0400 Subject: [PATCH] 3.13 compat differences: document a consistency model change with Khepri --- release-notes/3.13.0.md | 53 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/release-notes/3.13.0.md b/release-notes/3.13.0.md index fbff29af83..e802bfb381 100644 --- a/release-notes/3.13.0.md +++ b/release-notes/3.13.0.md @@ -81,6 +81,59 @@ Client libraries that were compatible with RabbitMQ `3.12.x` will be compatible RabbitMQ Stream Protocol clients must be upgraded to use the stream filtering feature introduced in this release. +### Consistency Model and Schema Modification Visibility Guarantees of Khepri and Mnesia + +Khepri has an important difference from Mnesia when it comes to schema modifications such as queue +or stream declarations, or binding declarations. These changes won't be noticeable with many workloads +but can affect some, in particular, certain integration tests. + +Consider two scenarios, A and B. + +#### Scenario A + +There is only one client. The client performs the following steps: + +1. It declares a queue Q +2. It binds Q to an exchange X +3. It publishes a message M to the exchange X +4. It expects the message to be routed to queue Q +5. It consumes the message + +In this scenario, there should be no observable difference in behavior. Client's expectations +will be met. + +#### Scenario B + +There are two clients, One and Two, connected to nodes R1 and R3, and using the same virtual host. +Node R2 has no client connections. + +Client One performs the following steps: + +1. It declares a queue Q +2. It binds Q to an exchange X +3. It gets a queue declaration confirmation back +4. It notifies client 2 or client 2 implicitly finds out that it has finished the steps above (for example, in an integration test) +5. Client Two publishes a message M to X +6. Clients One and Two expect the message to be routed to Q + +In this scenario, on step three Mnesia would return when **all** cluster nodes have committed an update. +Khepri, however, will return when **a majority** of nodes, including the node handling Client One's operations, +have returned. + +This may include nodes R1 and R2 but not node R3, meaning that message M published by Client Two connected to node R3 +in the above example **is not guaranteed not be routed**. + +Once all schema changes propagate to node R3, Client Two's subsequent +publishes on node R3 **will be guaranteed** to be routed. + +This trade-off of a Raft-based system that assume that a write accepted by a majority of nodes +can be considered a succeess. + +To satisfy Client Two's expectations in scenario B Khepri could beform **consistent** (involving a majority of replicas) +queries of bindings when routing messages but that would have a **significant** impact on throughput +of certain protocols (such as MQTT) and exchange/destination types (anything that resembles a topic exchange in AMQP 0-9-1). + + ### Management Plugin and HTTP API GET /api/queues` HTTP API endpoint has dropped several rarely used metrics, resulting in 25% in traffic saving.