rabbitmq-server/release-notes/4.2.0.md

RabbitMQ 4.2.0-beta.4

RabbitMQ 4.2.0-beta.4 is a preview of a feature release.

Breaking Changes and Compatibility Notes

Default value for AMQP 1.0 durable field.

Starting with RabbitMQ 4.2, if a sending client omits the header section, RabbitMQ assumes the durable field to be false complying with the AMQP 1.0 spec:

<field name="durable" type="boolean" default="false"/>

AMQP 1.0 apps or client libraries must set the durable field of the header section to true to mark the message as durable.

Team RabbitMQ recommends client libraries to send messages as durable by default. All AMQP 1.0 client libraries maintained by Team RabbitMQ send messages as durable by default.

Mandatory flag in Direct Reply-To

Starting with RabbitMQ 4.2, if an AMQP 0.9.1 Direct Reply-To responder (RPC server) publishes with the mandatory flag set, then amq.rabbitmq.reply-to.* is treated as a queue. Whether the requester (RPC client) is still there to consume the reply is not checked at routing time. In other words, if the responder publishes to only this queue name, then the message will be considered "routed" and RabbitMQ will therefore not send a basic.return.

Release Highlights

SQL Filter Expression for Streams

AMQP 1.0 clients can now define SQL-like filter expressions when consuming from streams, enabling server-side message filtering. RabbitMQ will only dispatch messages that match the provided filter expression, reducing network traffic and client-side processing overhead. SQL filter expressions are a more powerful alternative to the AMQP Property Filter Expressions introduced in RabbitMQ 4.1.

RabbitMQ implements a subset of AMQP Filter Expressions Version 1.0 Committee Specification Draft 01 Section 6 including support for:

• Comparison operators (=, !=, <>, >, <, >=, <=)
• Logical operators (AND, OR, NOT)
• Arithmetic operators (+, -, *, /, %)
• Special operators (LIKE, IN, IS NULL)
• UTC function
• Access to the properties and application-properties sections

Examples

Simple expression:

header.priority > 4

Complex expression:

order_type IN ('premium', 'express') AND
(customer_region LIKE 'EU-%' OR customer_region = 'US-CA') AND
UTC() < properties.absolute-expiry-time AND
NOT cancelled

To learn more, check out the new documentation guide on Stream Filtering.

Pull Request: #14184

Direct Reply-To for AMQP 1.0

RabbitMQ 4.2 adds Direct Reply-To support for AMQP 1.0, alongside the existing AMQP 0.9.1 implementation. It also works across protocols (e.g., AMQP 1.0 requester with AMQP 0.9.1 responder, or vice versa).

For more information, read our updated documentation on Direct Reply-To.

Pull Request: #14474

New Tooling for More Automated Blue-Green Deployment Migrations from 3.13.x Clusters to 4.2.x

Blue-Green Deployment migration from RabbitMQ 3.13.x to 4.2.0 is now easier to automate thanks to a new set of commands provided by rabbitmqadmin v2.

Incoming and Outgoing Message Interceptors for Native Protocols

Incoming and outgoing messages can now be intercepted on the broker. This works for AMQP 1.0, AMQP 0.9.1, and MQTT.

What the interceptor does is entirely up to its implementation, for example it can validate message metadata, add annotations, or perform arbitrary side effects. Custom interceptors can be developed and integrated via plugins.

Two new optional built-in interceptors were added to RabbitMQ:

1. Timestamps for outgoing messages
2. Setting client ID of publishing MQTT client

Detailed information can be found in the Message Interceptor documentation.

Khepri Enabled by Default for New Clusters

RabbitMQ supports two databases to store metadata such as virtual hosts, topology, runtime parameters, policies, internal users and so on: Mnesia and Khepri. That metadata store is also at the heart of clustering in RabbitMQ. As of RabbitMQ 4.2.0, Khepri is the default metadata store for new deployments.

Khepri is based on the same Raft consensus algorithm used by quorum queues and streams. The goal is to have a consistent well defined behaviour around all queries and updates of metadata across an entire cluster, especially when the cluster suffers increased latency or network issues for instance. It also comes with increased performance in several use cases, even though this was not a goal.

A new RabbitMQ 4.2.0+ node will use Khepri by default. If you upgrade an existing node or cluster, it will continue to use whatever metadata store it was using so far.

If you did not enable Khepri yet, it is recommended that you enable it:

rabbitmqctl enable_feature_flag khepri_db

Khepri will become mandatory in a future minor version. Mnesia support will be dropped in a future major version. These exact versions are to be decided.

Local Shovels

In addition to AMQP 0-9-1 and AMQP 1.0, Shovels now support a new "protocol" option called local.

These specialized shovels are internally based on AMQP 1.0 but instead of separate TCP connections, use the intra-cluster connections between cluster nodes and the internal API for consumption, publishing and AMQP 1.0 credit flow.

Such shovels can only be used for consuming and publishing within the same cluster, not across clusters, but can offer higher throughput and use fewer resources per connections than their AMQP 0-9-1 and AMQP 1.0 counterparts.

Upgrading to 4.2.0

Documentation guides on upgrades

See the Upgrading guide for documentation on upgrades and GitHub releases for release notes of individual releases.

This release series supports upgrades from 4.1.x, 4.0.xand3.13.x`.

Blue/Green Deployment-style upgrades are avaialble for migrations from RabbitMQ 3.12.x series.

New Required Feature Flags

None. The required feature flag set is the same as in 4.1.x and 4.0.x.

Mixed version cluster compatibility

RabbitMQ 4.2.0 nodes can run alongside 4.1.x and 4.0.x nodes. 4.2.x-specific features can only be made available when all nodes in the cluster upgrade to 4.2.0 or a later patch release in the new series.

While operating in mixed version mode, some aspects of the system may not behave as expected. Once all nodes are upgraded to 4.1.0, these irregularities will go away.

Mixed version clusters are a mechanism that allows rolling upgrade and are not meant to be run for extended periods of time (no more than a few hours).

Recommended Post-upgrade Procedures

This version does not require any additional post-upgrade procedures compared to other versions.

Changes Worth Mentioning

This section is incomplete and will be expanded as 4.2 approaches its release candidate stage.

Core Server

Enhancements

• In clusters with a larger number of quorum queues (say, tens of thousands), quorum queue leadership transfer is now performed gradually and not all at once.

  Previously tens of thousands of concurrent leader elections could result in timeouts and some quorum queues ending up without an elected leader.

  GitHub issue: #14401

• Schema data store (Khepri) read concurrency optimizations that can lead to low doublt digit percent throughput gains on nodes with larger numbers of cores.

  GitHub issue: #14530

• Two new rabbitmq.conf, settings log.summarize_process_state and log.error_logger_format_depth, can be used to significantly reduce the amount of queue member (replica) state logged in case of an abnormal termination.

  Limiting logging helps avoid memory allocation spikes.

  GitHub issues: #14349, #14523

• When a configured authentication or authorization backend comes from a known plugin but the plugin is not enabled, the node will now refuse to start.

  Previously the node would boot but client connections would fail because of the missing backend modules.

  GitHub issues: #13783, #14408

• Similarly to the number of queues and virtual hosts, it is now possible to configure a limit on the cluster-wide number of exchanges that applications can create:

  # Applications won't be able to declare more than 200 exchanges
  # (including the protocol-standard pre-declared ones) in the cluster
  cluster_exchange_limit = 200
  

• Routing via the fanout exchange got optimised. For messages published to the fanout exchange, end-to-end message throughput increases by up to 42%.

  GitHub issue: #14546

Bug Fixes

• default_password, ssl_options.password now can tell between a generated random password value and an encrypted value better.

  Encrypted values must be prefixed with encrypted:. All other values, including generated passwords that contain a colon (:), will be considered non-encrypted ones.

  GitHub issue: #14365

• Import of definition files that contained topic exchange permissions failed.

  GitHub issue: #14409

Stream Plugin

Bug Fixes

• Stream client connections that authenticate using a JWT token (OAuth 2) have to periodically renew their JWT tokens. Should such an update fail, the RabbitMQ Stream Protocol connection will be immediately closed.

  In addition, stream connections now verify that the newly obtained JWT token still grants access to the virtual host the client is connected to.

  GitHub issues: #14403, #14406

Dependency Changes

• ra was upgraded to 2.17.1
• khepri was upgraded to 0.17.2
• khepri_mnesia_migration was upgraded to 0.8.0
• cuttlefish was upgraded to 3.5.0

Source Code Archives

To obtain source code of the entire distribution, please download the archive named rabbitmq-server-4.2.0.tar.xz instead of the source tarball produced by GitHub.