rabbitmq-server/deps/rabbit_common/include
David Ansari e404d10eea
Speed up fanout exchange (#14546)
* Add test case for binding args Khepri regression

This commit adds a test case for a regression/bug that occurs in Khepri.
```
make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia
```
succeeds, but
```
make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri
```
fails.

The problem is that ETS table `rabbit_khepri_index_route` cannot
differentiate between two bindings with different binding arguments, and
therefore deletes entries too early, leading to wrong routing decisions.

The solution to this bug is to include the binding arguments in the
`rabbit_khepri_index_route` projection, similar to how the binding args
are also included in the `rabbit_index_route` Mnesia table.

This bug/regression is an edge case and exists if the source exchange
type is `direct` or `fanout` and if different bindings arguments are
used by client apps. Note that such binding arguments are entirely
ignored when RabbitMQ performs routing decisions for the `direct` or
`fanout` exchange. However, there might be client apps that use binding
arguments to add some metadata to the binding, for example `app-id` or
`user` or `purpose` and might use this metadata as a form of reference
counting in deciding when to delete `auto-delete` exchanges or just for
informational/operational purposes.

* Fix regression with Khepri binding args

Fix #14533

* Speed up fanout exchange

Resolves #14531

 ## What?
Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below).
In addition to the fanout exchange, a similar speed up is achieved for the following exchange types:
* modulus hash
* random
* recent history

This applies only if Khepri is enabled.

 ## How?
Use an additional routing table (projection) whose table key is the source exchange.
Looking up the destinations happens then by an ETS table key.

Prior to this commit, CPUs were busy compiling the same match spec for every incoming message.

 ## Benchmark
1. Start RabbitMQ:
```
make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \
    RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management"
```
where `advanced.config` contains:
```
[
 {rabbitmq_management_agent, [
  {disable_metrics_collector, true}
 ]}
].
```

2. Create a queue and binding:
```
deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1
```

3. Create the load
```
java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60
```

Before this commit:
```
sending rate avg: 97394 msg/s
receiving rate avg: 97394 msg/s
```

After this commit:
```
sending rate avg: 138677 msg/s
receiving rate avg: 138677 msg/s
```

The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts:
* 13.5% before this commit
* 3.4% after this commit

 ## Downsides
Additional ETS memory usage for the new projection table.
However, the new table does not store any binding entries for the following
source exchange types:
* direct
* headers
* topic
* x-local-random

* Add exchange binding tests

Test that exchange bindings work correctly with the new projection
tables `rabbit_khepri_route_by_source` and
`rabbit_khepri_route_by_source_key`.

* Always register all projections

Khepri won’t modify a projection that is already registered (based on its name).

* Protect ets:lookup_element/4 in try catch

See https://github.com/rabbitmq/rabbitmq-server/pull/11667#issue-2401399413
for rationale.
2025-09-24 15:05:16 -04:00
..
logging.hrl rabbit_peer_discovery: Move peer discovery driving code from `rabbit_mnesia` 2023-07-07 09:42:32 +02:00
rabbit.hrl Speed up fanout exchange (#14546) 2025-09-24 15:05:16 -04:00
rabbit_core_metrics.hrl Bump (c) line year 2025-01-01 17:54:10 -05:00
rabbit_framing.hrl Increase initial AMQP 0-9-1 connection frame size limit 2025-03-15 18:22:55 -04:00
rabbit_misc.hrl Bump (c) line year 2025-01-01 17:54:10 -05:00
resource.hrl Bump (c) line year 2025-01-01 17:54:10 -05:00