History

Loïc Hoguin c5d150a7ef Use Erlang.mk's native Elixir support for CLI This avoids using Mix while compiling which simplifies a number of things and let us do further build improvements later on. Elixir is only enabled from within rabbitmq_cli currently. Eunit is disabled since there are only Elixir tests. Dialyzer will force-enable Elixir in order to process Elixir-compiled beam files. This commit also includes a few changes that are related: * The Erlang distribution will now be started for parallel-ct * Many unnecessary PROJECT_MOD lines have been removed * `eunit_formatters` has been removed, it provides little value * The new `maybe_flock` Erlang.mk function is used where possible * Build test deps when testing rabbitmq_cli (Mix won't do it anymore) * rabbitmq_ct_helpers now use the early plugins to have Dialyzer properly set up		2025-03-18 10:02:49 +01:00
..
examples	Update branch names and links	2022-09-29 13:58:56 -07:00
priv/schema	k8s peer discovery v2 (#13050 )	2025-02-04 17:07:27 +01:00
src	k8s peer discovery v2 (#13050 )	2025-02-04 17:07:27 +01:00
test	k8s peer discovery v2 (#13050 )	2025-02-04 17:07:27 +01:00
.gitignore	Cleanup .gitignore files for the monorepo	2024-06-28 12:00:52 +02:00
CODE_OF_CONDUCT.md	Replace files with symlinks	2022-04-15 06:04:29 -07:00
CONTRIBUTING.md	Replace files with symlinks	2022-04-15 06:04:29 -07:00
LICENSE	Replace @rabbitmq.com addresses with rabbitmq-core@groups.vmware.com	2023-06-20 15:40:13 +04:00
LICENSE-MPL-RabbitMQ	…
Makefile	Use Erlang.mk's native Elixir support for CLI	2025-03-18 10:02:49 +01:00
README.md	k8s peer discovery v2 (#13050 )	2025-02-04 17:07:27 +01:00

README.md

RabbitMQ Peer Discovery Kubernetes

Overview

This is an implementation of RabbitMQ peer discovery interface for Kubernetes.

On Kubernetes, RabbitMQ should be deployed as a StatefulSet. Each Pod in a StatefulSet has a name made up of the StatefulSet name and an ordinal index. The ordinal index values almost always start with 0, although this is configurable.

This plugin only allows the node with the lowest ordinal index (generally the pod with the -0 suffix) to form a new cluster. This node is referred to as the seed node.

All other nodes will join the seed node, or will forever keep try to join it, if they can't.

In the most common scenario, this means that:

the pod with -0 suffix will start immediately, effectively forming a new single-node cluster
any other pod will join the pod with -0 suffix and synchronize the cluster metadata with it

Configuration

In most cases, no configuration should be necessary beyond enabling this plugin.

If you use the RabbitMQ Cluster Operator or the Bitnami Helm Chart, this plugin is enabled by default, so you don't have to do anything.

Advanced Configuration

If you use a different ordinal start value in your StatefulSet, you have to configure this plugin to use it:

cluster_formation.k8s.ordinal_start = N

where N matches the .spec.ordinals.start value of the StatefulSet.

If the plugin doesn't work for any reason (a very unusual Kubernetes configuration or issues with hostname resolution) and you have to force RabbitMQ to use a different seed node than it would automatically, you can do this:

cluster_formation.k8s.seed_node = rabbit@seed-node-hostname

If cluster_formation.k8s.seed_node is configured, this plugin will just use this value as the seed node. If you do this, please open a GitHub issue and explain why the plugin didn't work for you, so we can improve it.

Historical Notes

This implementation (version 2) of the plugin was introduced in RabbitMQ 4.1 and has little to do with the original design, which was included in RabbitMQ from version 3.7.0 until 4.0. Nevertheless, backwards compatibility is maintained, by simply ignoring the configuration options of the original implementation.

The original implementation of this plugin performed peer discovery by querying the Kubernetes API for the list of endpoints serving as the backends of a Kubernetes Service. However, this approach had a few issues:

The query was not necessary, given that the Pod names are predictable
The query could fail or require configuration to work (eg. TLS settings had to be adjusted in some environments)
To perform the query, access to the Kubernetes API was required (unnecessary privileges)
In some environments, the plugin was prone to race conditions and could form more than 1 cluster

The new design solves all those problems:

It doesn't query the Kubernetes API at all
Only one node is allowed to form a cluster

Supported RabbitMQ Versions

Version 2 was first included in RabbitMQ 4.1.

Version 1 of this plugin (which queried the Kubernetes API) was included from RabbitMQ 3.7.0 until 4.0.

Documentation

See RabbitMQ Cluster Formation guide for an overview of the peer discovery subsystem, general and Kubernetes-specific configurable values and troubleshooting tips.