The most important changes are [Streams](https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/), support for [Erlang 24](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/), and improved integration with our [RabbitMQ Kubernetes Operator](https://blog.rabbitmq.com/posts/2020/11/rabbitmq-kubernetes-operator-reaches-1-0/). Other changes worth mentioning are JSON logging, per-protocol and queue type message rates as Prometheus metrics and faster initial cluster formation on Kubernetes.
Find blog posts that expand on the previously mentioned topics: https://blog.rabbitmq.com/tags/v3.9.x/
## Highlights
### Streams
[Streams](http://rabbitmq.com/streams.html) are a new persistent and replicated data structure ("queue type") in RabbitMQ which models an append-only log with non-destructive consumer semantics.
They can be used as a regular AMQP 0.9.1 queue or through a new [binary protocol plugin](http://rabbitmq.com/stream.html) and associated client(s).
Streams enable messaging patterns that were not possible or practical before, for example, repeatable non-destructive consumption and mass fanouts to a very large number of consumers.
### Erlang 24 Support
[Erlang 24](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/) provides 20%-50% higher throughput for many workloads tested by the RabbitMQ core team and community members. Supporting the latest runtime version also allowed RabbitMQ to replace a few external dependencies with libraries now provided by the runtime.
Erlang 24 is now used by default in the [community RabbitMQ Docker image](https://github.com/docker-library/rabbitmq).
### Kubernetes
[RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html) automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes. It is [open source](https://github.com/rabbitmq/cluster-operator) and is developed by the RabbitMQ Core team at VMware, Inc. This Operator is not specific to 3.9 and can be used with latest 3.8.x release series.
[Messaging Topology Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html#topology-operator) makes it possible to define RabbitMQ resources (virtual hosts, users, permissions, topologies, policies, etc) as Kubernetes objects. This Operator is also [open source](https://github.com/rabbitmq/messaging-topology-operator) and can be used with latest 3.8.x release series.
### Logging in JSON
[Switch from Lager to the new Erlang Logger API for logging](https://github.com/rabbitmq/rabbitmq-server/pull/2861) introduces an option for structured logging in JSON. This is relevant for the Prometheus-based alerting feature that we have been investing in this year: [Notify me when RabbitMQ has a problem](https://blog.rabbitmq.com/posts/2021/05/alerting/).
## Obtaining Packages
RabbitMQ releases are distributed via [GitHub](https://github.com/rabbitmq/rabbitmq-server/releases), [Cloudsmith](https://cloudsmith.io/~rabbitmq/repos/), and [PackageCloud](https://packagecloud.io/rabbitmq).
### Erlang/OTP Compatibility Notes
This release [requires Erlang 23.2](https://www.rabbitmq.com/which-erlang.html) or later. Erlang 24 is recommended as it [offers 20%-50% throughput improvements](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/) for many workloads.
[Provisioning Latest Erlang Releases](https://www.rabbitmq.com/which-erlang.html#erlang-repositories) explains what package repositories and tools can be used to provision latest patch versions of Erlang 23.3.x and 24.x.
## Upgrading to 3.9
See the [Upgrading guide](https://www.rabbitmq.com/upgrade.html) for documentation on upgrades and [RabbitMQ change log](https://www.rabbitmq.com/changelog.html) for release notes of other releases.
RabbitMQ 3.9.0 nodes can run alongside `3.8.x` nodes. However, `3.8.16+` as well as `3.9.x` both require Erlang 23.2 or later (and support 24). `3.9.x`-specific features can only be made available when all nodes in the cluster upgrade to 3.9.0 or any other patch release in the series.
While operating in mixed version mode, some aspects of the system may not behave as expected. The list of known behavior changes is covered below. Once all nodes are upgraded to 3.9.0, these irregularities will go away.
Mixed version clusters are a mechanism that allows rolling upgrade and are not meant to be run for extended periods of time (no more than a few hours).
### Quorum Queue Declaration
In a mixed version cluster, declaration of new quorum queues will fail if executed on a node in the version minority. For example, in a five node cluster with two nodes running 3.8, the three 3.9 nodes are in the version majority and the remaining are in the version minority.
Existing quorum queues will operate just like before.
### Classic Queue Metrics
HTTP API responses will provide a [limited number of metrics](https://github.com/rabbitmq/rabbitmq-server/issues/3080) or metrics for a limited number of queues (for operations such as `GET /api/queues/{vhost}`) when operating in mixed version mode.
## Compatibility Notes
### Client Library Compatibility
Client libraries that were compatible with RabbitMQ `3.8.x` will be compatible with `3.9.0`.
### Upgrading to Erlang 24.x
RabbitMQ `3.8.16` and later require Erlang `23.2` and support Erlang `24` (recommended). Users of RabbitMQ versions that [do not support Erlang 23.2](https://www.rabbitmq.com/which-erlang.html) should first upgrade to the latest `3.8.x` patch release on Erlang 23.2, then upgrade to `3.9.0`, and ideally Erlang 24.
### Getting Help
Any questions about this release, upgrades or RabbitMQ in general are welcome on the [RabbitMQ mailing list](https://groups.google.com/forum/#!forum/rabbitmq-users).
* [Streams](http://rabbitmq.com/streams.html), a new durable replicated data structure that allows for repeatable (non-destructive) consumption. RabbitMQ 3.9 introduces a new binary protocol for working with streams. While streams can be used by AMQP 0-9-1 clients, in most cases it will be used via a [RabbitMQ stream binary protocol](http://rabbitmq.com/stream.html) client. The new protocol exposes all the operations specific to this queue type and is heavily optimized for them, resulting in much higher read and write throughput.
[Java](https://github.com/rabbitmq/rabbitmq-stream-java-client) and [Go](https://github.com/rabbitmq/rabbitmq-stream-go-client) clients
* Structural logging in JSON. Nodes can now log in JSON format.
* [Peer discovery](https://www.rabbitmq.com/cluster-formation.html) no longer uses randomized startup delay to ensure only one node starts as a seed one during initial (parallel) cluster formation. Instead it relies on an internal runtime locking library.
* Per-protocol and queue type message rates are now available as Prometheus metrics, as well as Streams-specific errors. This also addresses a long-standing 3.8.x counters issue for most metrics of this type, as captured in [#2783](https://github.com/rabbitmq/rabbitmq-server/issues/2783).
* Standard runtime logger is now used for logging instead of Lager, a 3rd party dependency. In modern Erlang versions, the module offers at least a comparable feature set and efficiency of Lager.
Users who relied on Lager-specific configuration, e.g. custom extensions or sinks, would have to rebuild them on top of the standard Erlang `logger` module.
* Different Raft-based features now can have different defaults and settings, with more configurability coming at a later point. For example, client ID tracking in the MQTT plugin can use settings for lower memory footprint and more frequent compaction while stream and quorum queues use settings more optimal for higher throughput on the same node.
* Continuous integration of open source RabbitMQ has switched to Bazel, GitHub Actions and [BuildBuddy](https://buildbuddy.io), resulting in much faster and incremental test runs.
* Correct an issue where the `--tags` flag was ignored if `--description` was not specified in the `rabbitmqctl add_vhost` command
### Kubernetes
#### Enhancements
* [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html) automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.
* [Messaging Topology Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html#topology-operator) makes it possible to define RabbitMQ resources (virtual hosts, users, permissions, topologies, policies, etc) as Kubernetes objects.
* Kubernetes [peer discovery](https://www.rabbitmq.com/cluster-formation.html) mechanism no longer uses randomized delays to avoid the inherent race condition during [initial cluster formation](https://www.rabbitmq.com/cluster-formation.html#initial-formation-race-condition). Instead it relies on an internal distributed locking mechanism available in modern Erlang releases.
While randomized startup delay was reasonably effective at ensuring only one cluster was formed even during parallel node startup, it by definitely assumes that with some low probability more than one node can still start as seed nodes, and be joined by different groups of cluster peers. The locking mechanism does not have this downside. Now that RabbitMQ requires Erlang 23.2, the internal locking library becomes an option for peer discovery.
`cluster_formation.randomized_startup_delay_range.min` and `cluster_formation.randomized_startup_delay_range.max` configuration keys will no longer have any effect.
* [AWS peer discovery mechanism](https://www.rabbitmq.com/cluster-formation.html#peer-discovery-aws) now supports [Instance Metadata Service v2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html). In case it is not available, requests will fall back to the original metadata service endpoint.
The v2 endpoint offers substantial security improvements and is one of the [AWS best practices](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-standards-fsbp-controls.html#ec2-8-remediation).
