rabbitmq-server/release-notes/3.8.4.md

## RabbitMQ 3.8.4

RabbitMQ `3.8.4` is a maintenance release. It focuses on bug fixes and usability
improvements. etcd [peer discovery](https://www.rabbitmq.com/cluster-formation.html) plugin includes potentially breaking changes which are covered in detail below.

This is the first release to be [fully compatible](https://groups.google.com/forum/#!topic/rabbitmq-users/wlPIWz3UYHQ) with [Erlang 23](http://blog.erlang.org/OTP-23-Highlights/).

### Erlang/OTP Compatibility Notes

This release [**requires Erlang/OTP 21.3**](https://www.rabbitmq.com/which-erlang.html) or later.
`22.3` or `23.0` releases are recommended.

This is the first release to officially support Erlang 23. Compared to `3.8.3`, which
generally works as expected on Erlang 23, [Prometheus monitoring](https://www.rabbitmq.com/prometheus.html) plugin had to
be adapted to support a memory allocator metric format that has changed compared to
earlier Erlang releases.

There is one **remaining known Erlang 23 incompatibility**: `rabbitmq-diagnostics observer` **will fail** on Erlang 23.
This will be addressed in a future release.
The root cause is the same memory allocator metric change.

[Provisioning Latest Erlang Releases](https://www.rabbitmq.com/which-erlang.html#erlang-repositories) explains
what package repositories and tools can be used to provision latest patch versions of Erlang `22.3.x`.


## Upgrade and Compatibility Notes

### Raft Segment File Format Change

In earlier `3.8.x` versions, Raft log segment files were unintentionally limited to 4 GiB,
which can be problematic for environments that use quorum queues and medimum to large message
payloads.

This releases removes the limitation by changing the segment file on-disk format.
While **no action is required** for upgrading to `3.8.4`, operators should keep in mind
that segment files produced by `3.8.4` nodes won't be readable by earlier versions.
In other words, this release **does not support downgrading** to earlier `3.8.x` versions.

### Custom Prefixless Node Name

Environments that override node name using `NODENAME` in `rabbitmq-env.conf` may have to
also define `RABBITMQ_NODENAME` in the same file for CLI tools to pick the value up:

``` ini
# how node name is typically overridden in rabbitmq-env.conf[.bat] files
NODENAME=rabbit@custom.hostname
# for CLI tools in 3.8.4, otherwise not necessary
RABBITMQ_NODENAME=rabbit@custom.hostname
```

This behaviour change compared to earlier `3.8.x` version was not intentional and was
[corrected for `3.8.5`](https://github.com/rabbitmq/rabbitmq-cli/pull/422).

Alternatively, it is possible to provide target node name using a command line flag:

``` shell
rabbitmq-diagnostics -n rabbit@custom.hostname status
```

### Upgrade Doc Guides and Change Log

See [3.8.0 release notes](https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.0) upgrade and compatibility notes first if upgrading from an earlier release.

See the [Upgrading guide](https://www.rabbitmq.com/upgrade.html) for general documentation on upgrades and [RabbitMQ change log](https://www.rabbitmq.com/changelog.html) for release notes of other releases.

### Upgrading to Erlang 21.x or Later Versions

When upgrading to this release from `3.7.6` or an older version, extra care has to be taken.

Since CLI tools from RabbitMQ releases older than 3.7.7 will fail on Erlang 21 or later, RabbitMQ **must be upgraded at the same time as Erlang**.
Alternatively the node can be upgraded to `3.7.18` first, then Erlang 22.x or 23.x, then RabbitMQ to most recent
3.8.x release.

### Getting Help

Any questions about this release, upgrades or RabbitMQ in general are welcome on the [RabbitMQ mailing list](https://groups.google.com/forum/#!forum/rabbitmq-users).



## Changes Worth Mentioning

### Core Server

#### Bug Fixes

 * A node recovering a large disk backlog did not throttle its recovery activity, potentially
   using large amounts of memory at peak during recovery.

   GitHub issue: [rabbitmq/rabbitmq-server#2254](https://github.com/rabbitmq/rabbitmq-server/issues/2254)

 * Raft log segment files were unintentionally limited to 4 GiB in size, which could result
   in failing log flush operations in environments where medium and large messages were published to
   quorum queues.

   This problem is addressed by a change to the on disk format of Raft log segment files.
   The change is irreversible, meaning that **there is no downgrade path** from `3.8.4` to earlier releases.

   GitHub issue: [rabbitmq/ra#176](https://github.com/rabbitmq/ra/pull/176)

 * A reached delivery limit on a quorum queue could result in an exception for polling (`basic.get`) consumers.

   GitHub issue: [rabbitmq/rabbitmq-server#2330](https://github.com/rabbitmq/rabbitmq-server/pull/2330)

 * A publish to a quorum queue that happened to not have an elected leader failed with an exception.
   Such operations are generally expected to be retried by channels.

   GitHub issue: [rabbitmq/rabbitmq-server#2295](https://github.com/rabbitmq/rabbitmq-server/pull/2295)

 * Separate authN and authZ backends did not behave as expected when `rabbitmq_auth_backend_cache` was enabled.

   GitHub issue: [rabbitmq/rabbitmq-server#2322](https://github.com/rabbitmq/rabbitmq-server/issues/2322)

 * Handle a condition where epmd connection was closed during node registration.

   GitHub issue: [rabbitmq/rabbitmq-server#2332](https://github.com/rabbitmq/rabbitmq-server/pull/2332)

 * Registering a quorum queue consumer did not result in a `consumer.created` event like it would
   for classic queues.

   GitHub issue: [rabbitmq/rabbitmq-server#2341](https://github.com/rabbitmq/rabbitmq-server/issues/2341)

#### Enhancements

 * Reduced peak memory use of quorum queues that go idle (not under constant load).

   GitHub issue: [rabbitmq/rabbitmq-server#2272](https://github.com/rabbitmq/rabbitmq-server/pull/2272)

 * Change default configuration values for quorum queues to improve publisher back pressure when under heavy load.

   GitHub issue: [rabbitmq/rabbitmq-server#2334](https://github.com/rabbitmq/rabbitmq-server/pull/2334)

 * Definition import is now partially concurrent, improving CPU utilisation and reducing the amount of time
   taken to import a large definition file.

   GitHub issue: [rabbitmq/rabbitmq-server#2224](https://github.com/rabbitmq/rabbitmq-server/issues/2224)

 * Improved configuration merging for cases when multiple files are used, e.g. `rabbitmq.conf` plus
   `advanced.config` or even multiple `.conf` files (the latter is an experimental feature).

   GitHub issue: [rabbitmq/rabbitmq-server#2277](https://github.com/rabbitmq/rabbitmq-server/pull/2277)

 * More peer discovery errors are retried.

   GitHub issue: [rabbitmq/rabbitmq-server#1627](https://github.com/rabbitmq/rabbitmq-server/issues/1627)

 * Allow product name, version and MOTD to be overwritten via environment variables.

   GitHub issue: [rabbitmq/rabbitmq-server#2247](https://github.com/rabbitmq/rabbitmq-server/pull/2247)

 * Improve display of product name and version.

   GitHub issue: [rabbitmq/rabbitmq-server#2326](https://github.com/rabbitmq/rabbitmq-server/pull/2326)

 * Make RabbitMQ a regular Erlang/OTP application.

   GitHub issue: [rabbitmq/rabbitmq-server#2180](https://github.com/rabbitmq/rabbitmq-server/issues/2180)


### CLI Tools

#### Enhancements

 * New `rabbitmq-upgrade` commands: `await_online_quorum_plus_one` and `await_online_synchronized_mirror`.
   The commands will wait for a safe quorum of nodes (quorum queues) or a synchronised leader replica (classic mirrored queues)
   to be or come online within a period of time.

   ``` sh
   # Exits with a non-zero exit code if there is not a quorum + 1 of nodes online for all
   # quorum queues within 120 seconds (default timeout)
   rabbitmq-upgrade await_online_quorum_plus_one
   # Will wait for a quorum + 1 of nodes to be online for all quorum queues for 0 seconds...

   # The timeout can be adjusted via the -t flag:
   rabbitmq-upgrade await_online_quorum_plus_one -t 60
   # Will wait for a quorum + 1 of nodes to be online for all quorum queues for 60 seconds...

   # There is an equivalent command for classic mirrored queues
   rabbitmq-upgrade await_online_synchronized_mirror
   ```

   GitHub issue: [rabbitmq/rabbitmq-cli#409](https://github.com/rabbitmq/rabbitmq-cli/issues/409)

 * New `rabbitmq-diagnostics` command: `os_env`

   GitHub issue: [rabbitmq/rabbitmq-cli#412](https://github.com/rabbitmq/rabbitmq-cli/pull/412)


### Prometheus Plugin

#### Enhancements

 * Compatibility with Erlang 23 memory allocator metric format.

   GitHub issue: [rabbitmq/rabbitmq-prometheus#42](https://github.com/rabbitmq/rabbitmq-prometheus/issues/42)

 * Node color labeling in Grafana dashboards now works out of the box with one more common naming scheme.

   GitHub issue: [rabbitmq/rabbitmq-prometheus#33](https://github.com/rabbitmq/rabbitmq-prometheus/issues/33)

   Contributed by @Aakcht.


### Federation Plugin

#### Bug Fixes

 * Credential obfuscation now uses the same [shared secret](https://www.rabbitmq.com/clustering.html#erlang-cookie)
   on all nodes to make sure that a link running on node A can be restarted by a command against node B.
   Previously each node used its own generated secret for obfuscation, which means certain operations
   on links originated on other nodes were not possible.

   GitHub issue: [rabbitmq/rabbitmq-server#2351](https://github.com/rabbitmq/rabbitmq-server/pull/2351)

#### Enhancements

 * Internal resources cleanup is now more selective on abnormal link termination. A new upstream parameter, `resource-cleanup-mode`,
   can be set to `"never"` used to prevent internal queues from being deleted by exchange federation links on termination.
   Use this when data safety is more important than protection from resource leaks. It then will be up to the operator to make
   sure such resources are periodically deleted or use a suitable [TTL](https://www.rabbitmq.com/ttl.html).

   GitHub issue: [rabbitmq/rabbitmq-federation#105](https://github.com/rabbitmq/rabbitmq-federation/issues/105)

 * SNI is now enabled by default for TLS-enabled connections.

   Contributed by Carl Hörberg (CloudAMQP).

   GitHub issue: [rabbitmq/rabbitmq-erlang-client#132](https://github.com/rabbitmq/rabbitmq-erlang-client/pull/132)


### Shovel Plugin

#### Bug Fixes

 * Credential obfuscation now uses the same [shared secret](https://www.rabbitmq.com/clustering.html#erlang-cookie)
   on all nodes to make sure that a shovel running on node A can be restarted by a command against node B.
   Previously each node used its own generated secret for obfuscation, which means certain operations
   on shovels originated on other nodes were not possible.

   GitHub issue: [rabbitmq/rabbitmq-server#2351](https://github.com/rabbitmq/rabbitmq-server/pull/2351)

 * Shovels are now correctly removed even when deletion commands are executed against a different node
   (not the one on which the target Shovel has been created).

   GitHub issue: [rabbitmq/rabbitmq-shovel#68](https://github.com/rabbitmq/rabbitmq-shovel/issues/68)

 * More negative AMQP 1.0 endpoint connection responses are handled.

   Contributed by Renaud @bougueil Mariana.

   GitHub issue: [rabbitmq/rabbitmq-amqp1.0-client#48](https://github.com/rabbitmq/rabbitmq-amqp1.0-client/pull/48)

 * Messages forwarded from an AMQP 0-9-1 source to an AMQP 1.0 destination now propagates more
   message properties.

   GitHub issue: [rabbitmq/rabbitmq-shovel#70](https://github.com/rabbitmq/rabbitmq-shovel/issues/70)

 * Pre-3.7 `delete-after` Shovel property is now considered as well as the new `src-delete-after` property.

   GitHub issue: [rabbitmq/rabbitmq-shovel#72](https://github.com/rabbitmq/rabbitmq-shovel/issues/72)

#### Enhancements

 * Revisited [Shovel documentation guides](https://www.rabbitmq.com/shovel.html).

 * SNI is now enabled by default for TLS-enabled connections.

   Contributed by Carl Hörberg (CloudAMQP).

   GitHub issue: [rabbitmq/rabbitmq-erlang-client#132](https://github.com/rabbitmq/rabbitmq-erlang-client/pull/132)


### Management Plugin

#### Enhancements

 * More frequent garbage collection by stats DB cache processes in environments where HTTP API queries produced
   large data sets.

   GitHub issue: [rabbitmq/rabbitmq-management#788](https://github.com/rabbitmq/rabbitmq-management/pull/788)

 * `rabbitmqadmin` CLI switches now take precedence over config file values. This is more typical
   for command line tools.

   GitHub issue: [rabbitmq/rabbitmq-management#804](https://github.com/rabbitmq/rabbitmq-management/issues/804)

 * `rabbitmqadmin` binding deletion now defaults to the same property key value as binding creation.

   Contributed by Ivan Kaloyanov.

   GitHub issue: [rabbitmq/rabbitmq-management#779](https://github.com/rabbitmq/rabbitmq-management/pull/779)

#### Bug Fixes

 * TLS settings of this plugin now supersede those of RabbitMQ core listeners. In more practical terms,
   it is now possible to use a separate set of certificate/private key pairs for management UI and
   the HTTP API.

   GitHub issue: [rabbitmq/rabbitmq-management#800](https://github.com/rabbitmq/rabbitmq-management/issues/800)

 * Queue page failed to render with a `TypeError` when stats were disabled.

   GitHub issue: [rabbitmq/rabbitmq-management#787](https://github.com/rabbitmq/rabbitmq-management/issues/787)

 * `rabbitmqadmin import` mangled non-ASCII characters.

   GitHub issue: [rabbitmq/rabbitmq-management#785](https://github.com/rabbitmq/rabbitmq-management/issues/785)

 * `rabbitmqadmin`'s pretty JSON format was not compatible with Python 3.

   GitHub issue: [rabbitmq/rabbitmq-management#797](https://github.com/rabbitmq/rabbitmq-management/pull/797)


### etcd Peer Discovery Plugin

This release effectively replaces the etcd-based peer discovery plugin with a new one which
[uses etcd v3 API](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/22) and a different API client.
etcd v2 API has been effectively deprecated and requires extra effort to enable with the latest etcd release,
so this was **treated as a medium severity bug** and ships in a patch release.

There are several new features that come with this new client. Existing already formed clusters won't have to
change anything in the short term as [peer discovery only runs during cluster formation](https://www.rabbitmq.com/cluster-formation.html#peer-discovery-how-does-it-work).
All configuration keys of the original plugin are still accepted by the plugin, and most will have
the expected or semantically similar effect.

When provisioning new clusters, etcd 3.4+ will now be required.

#### Enhancements

 * The plugin now uses an etcd v3 API client and v3 gRPC-based API. This includes using a forward-compatible API version,
   natively-provided endpoint for locking instead of reinventing one, and other improvements listed below.
   It also means that **etcd 3.4+** is now required by the plugin.

   Unless specified otherwise, all other configuration settings of the original plugin are still supported.

   `rabbitmq-autocluster`-era environment variable-based configuration of the plugin has been removed. Use the config
   file-based one instead.

   `cluster_formation.etcd.scheme` configuration key won't fail validation but will no longer have any effect.
   `cluster_formation.etcd.key_prefix` will be used but only after the hardcoded `/rabbitmq/discovery`
   and `/rabbitmq/locks` prefixes. Having a predictable key prefix is more important in this version of the etcd API.

   GitHub issue: [rabbitmq/rabbitmq-peer-discovery-etcd#22](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/22)

 * Multiple etcd endpoings now can be specified as connection candidates. Previously used endpoint configuration keys,
   `cluster_formation.etcd.host` and `cluster_formation.etcd.port`, are still supported and added to the final list of
   endpoints.

   GitHub issue: [rabbitmq/rabbitmq-peer-discovery-etcd#7](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/7)

 * Support for TLS-enabled connections to etcd.

   GitHub issue: [rabbitmq/rabbitmq-peer-discovery-etcd#24](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/24)

 * Support for [etcd authentication](https://etcd.io/docs/v3.4.0/op-guide/authentication/).

   GitHub issue: [rabbitmq/rabbitmq-peer-discovery-etcd#6](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/6)


### Consul Peer Discovery Plugin

#### Bug Fixes

 * In case `CONSUL_PORT` is set to a blank string, the plugin will fall back to the default port value
   instead of failing with an exception. Note that the use of environment variables over config file
   keys is **highly discouraged** and will be removed in a future version.

   GitHub issue: [rabbitmq/rabbitmq-peer-discovery-consul#41](https://github.com/rabbitmq/rabbitmq-peer-discovery-consul/issues/41)


### LDAP Plugin

#### Enhancements

 * Setting `` to `0` will force a new connection to be opened for every operation. This helps work around
   obscure issues likely caused by per-connection caching in some LDAP implementations.
   Users of this setting are expected to also use the [rabbitmq-auth-backend-cache](https://github.com/rabbitmq/rabbitmq-auth-backend-cache) plugin.

   GitHub issue: [rabbitmq/rabbitmq-auth-backend-ldap#120](https://github.com/rabbitmq/rabbitmq-auth-backend-ldap/pull/120)


### OAuth 2.0 Plugin

#### Enhancements

 * When token key ID is missing in configuration, a warning will be logged to
   make troubleshooting easier.

   GitHub issue: [rabbitmq/rabbitmq-auth-backend-oauth2#44](https://github.com/rabbitmq/rabbitmq-auth-backend-oauth2/issues/44)


### Trust Store Plugin

#### Bug Fixes

 * Some debug messages were unintentionally logged as errors.

   Contributed by @mgrafl.

   GitHub issue: [rabbitmq/rabbitmq-trust-store#75](https://github.com/rabbitmq/rabbitmq-trust-store/issues/75)


## Dependency Upgrades

 * New dependency: [eetcd](https://github.com/zhongwencool/eetcd/)
 * ra was upgraded [from v1.0.8 to v1.1.2](https://github.com/rabbitmq/ra/compare/v1.0.8...v1.1.2)
 * `credentials_obfuscation` was upgraded [from v1.0.0 to v2.0.0](https://github.com/rabbitmq/credentials-obfuscation/compare/v1.0.0...v2.0.0)
 * Prometheus client was upgraded [from 4.5.0 to 4.6.0](https://github.com/deadtrickster/prometheus.erl/compare/v4.5.0...v4.6.0)


## Source code archives

**Warning**: The source code archive provided by GitHub only contains the source of the broker, not the plugins or the client libraries.
Please download the archive named `rabbitmq-server-3.8.4.tar.xz`.