Since feature flag `message_containers` introduced in 3.13.0 is required in 4.0,
we can also require all other feature flags introduced in or before 3.13.0
and remove their compatibility code for 4.0:
* restart_streams
* stream_sac_coordinator_unblock_group
* stream_filtering
* stream_update_config_command
Since Erlang/OTP 26:
```
OTP-18445
Application(s):
erts, stdlib
It is no longer necessary to enable a feature in the runtime system in order to load modules that are using it.
It is sufficient to enable the feature in the compiler when compiling it.
That means that to use feature maybe_expr in Erlang/OTP 26, it is sufficient to enable it during compilation.
In Erlang/OTP 27, feature maybe_expr will be enabled by default, but it will be possible to disable it.
```
This version of rules_erlang adds coverage support
Bazel has sort of standardized on lcov for coverage, so that is what
we use.
Example:
1. `bazel coverage //deps/rabbit:eunit -t-`
2. `genhtml --output genhtml "$(bazel info
output_path)/_coverage/_coverage_report.dat"`
3. `open genhtml/index.html`
Multiple tests can be run with results aggregated, i.e. `bazel
coverage //deps/rabbit:all -t-`
Running coverage with RBE has a lot of caveats,
https://bazel.build/configure/coverage#remote-execution, so the above
commands won't work as is with RBE.
Bazel build files are now maintained primarily with `bazel run
gazelle`. This will analyze and merge changes into the build files as
necessitated by certain code changes (e.g. the introduction of new
modules).
In some cases there hints to gazelle in the build files, such as `#
gazelle:erlang...` or `# keep` comments. xref checks on plugins that
depend on the cli are a good example.
This is the latest commit in the series, it fixes (almost) all the
problems with missing and circular dependencies for typing.
The only 2 unsolved problems are:
- `lg` dependency for `rabbit` - the problem is that it's the only
dependency that contains NIF. And there is no way to make dialyzer
ignore it - looks like unknown check is not suppressable by dialyzer
directives. In the future making `lg` a proper dependency can be a
good thing anyway.
- some missing elixir function in `rabbitmq_cli` (CSV, JSON and
logging related).
- `eetcd` dependency for `rabbitmq_peer_discovery_etcd` - this one
uses sub-directories in `src/`, which confuses dialyzer (or our bazel
machinery is not able to properly handle it). I've tried the latest
rules_erlang which flattens directory for .beam files, but it wasn't
enough for dialyzer - it wasn't able to find core erlang files. This
is a niche plugin and an unusual dependency, so probably not worth
investigating further.
RabbitMQ 3.12 requires feature flag `feature_flags_v2` which got
introduced in 3.11.0 (see
https://github.com/rabbitmq/rabbitmq-server/pull/6810).
Therefore, we can mark all feature flags that got introduced in 3.11.0
or before 3.11.0 as required because users will have to upgrade to
3.11.x first, before upgrading to 3.12.x
The advantage of marking these feature flags as required is that we can
start deleting any compatibliy code for these feature flags, similarly
as done in https://github.com/rabbitmq/rabbitmq-server/issues/5215
This list shows when a given feature flag was first introduced:
```
classic_mirrored_queue_version 3.11.0
stream_single_active_consumer 3.11.0
direct_exchange_routing_v2 3.11.0
listener_records_in_ets 3.11.0
tracking_records_in_ets 3.11.0
empty_basic_get_metric 3.8.10
drop_unroutable_metric 3.8.10
```
In this commit, we also force all required feature flags in Erlang
application `rabbit` to be enabled in mixed version cluster testing
and delete any tests that were about a feature flag starting as disabled.
Furthermore, this commit already deletes the callback (migration) functions
given they do not run anymore in 3.12.x.
All other clean up (i.e. branching depending on whether a feature flag
is enabled) will be done in separate commits.
Prior to this commit, 1 MQTT publisher publishing to 1 Million target
classic queues requires around 680 MB of process memory.
After this commit, it requires around 290 MB of process memory.
This commit requires feature flag classic_queue_type_delivery_support
and introduces a new one called no_queue_name_in_classic_queue_client.
Instead of storing the binary queue name 4 times, this commit now stores
it only 1 time.
The monitor_registry is removed since only classic queue clients monitor
their classic queue server processes.
The classic queue client does not store the queue name anymore. Instead
the queue name is included in messages handled by the classic queue
client.
Storing the queue name in the record ctx was unnecessary.
More potential future memory optimisations:
* When routing to destination queues, looking up the queue record,
delivering to queue: Use streaming / batching instead of fetching all
at once
* Only fetch ETS columns that are necessary instead of whole queue
records
* Do not hold the same vhost binary in memory many times. Instead,
maintain a mapping.
* Remove unnecessary tuple fields.
This allows us to stop ignorning undefined callback warnings
When mix compiles rabbitmqctl, it produces a 'consolidated' directory
alongside the 'ebin' dir. Some of the modules in consolidated are
intended to be used instead of those provided by elixir. We now handle
the conflicts properly in the bazel build.
Directory name for tools in runfiles is a bit unpredictable. Maybe
there is a better way, but at least this should cover all the cases
that I've observed.
One unexpected consequence is that this directory name can contains `~`,
and it's not being properly escaped in `ct:pal` calls (causing
badarg's for `io:format/4`.
When starting the broker/cluster with `-c dbg`, warnings are no longer treated as errors.
This allows starting a broker/cluster to test a WIP change (eg with some
unused functions).
It was set to the empty string before to make sure no feature flags were
enabled out-of-the-box when starting a new RabbitMQ node.
With 3.11.0, several of them will become required and their
corresponding compatibility and migration code goes away. Therefore
older nodes must have them enabled in order to cluster successfully.
This change is only useful in 3.11.x. It must not be backported to older
release branches.
When the fetch of the secondary umbrella was moved into bzlmod, this
changed its path at the time of test execution. Tests will now fail if
a secondary umbrella is specified for bazel, but does not exist. The
path is also corrected.
Also rework elixir dependency handling, so we no longer rely on mix to
fetch the rabbitmq_cli deps
Also:
- Specify ra version with a commit rather than a branch
- Fixup compilation options for erlang 23
- Add missing ra reference in MODULE.bazel
- Add missing flag in oci.yaml
- Reduce bazel rbe jobs to try to save memory
- Use bazel built erlang for erlang git master tests
- Use the same cache for all the workflows but windows
- Avoid using `mix local.hex --force` in elixir rules
- Fetching seems blocked in CI, and this should reduce hex api usage in
all builds, which is always nice
- Remove xref and dialyze tags since rules_erlang 3 includes them in
the defaults
Before this commit, on Ubuntu, even though locale and env variable was
set correctly:
```
$ locale
LANG=C.UTF-8
```
```
$ env | grep LANG
LANG=C.UTF-8
```
bazel tests were failing because that env variable does not get set in
the linux-sandbox:
```
bazel test //deps/rabbit:queue_parallel_SUITE
...
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8.
Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name messages messages_ready messages_unacknowledged
stream_queue_publish 1 1 0
a_queue_with_extra_bcc 3 3 0
queue_without_extra_bcc 1 1 0
queue_with_extra_bcc 1 1 0
extra.bcc 3 3 0
my_extra_bcc 1 1 0
Exit code: 0 (pid <0.1987.0>)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
rabbit_ct_broker_helpers:rabbitmqctl_list failed on line 1098
Reason: {badmatch,[<<"warning: the VM is running with "...>>,<<"Time...}
```
This commits sets UTF-8 in the Bazel test environment.
rabbitmq_cli uses some private rules_erlang apis that have changed in
the upcoming release
Additionally:
- Avoid including both standard and test versions of amqp_client in
integration test suites
- Eliminate most of the compilation order hints (explicit first_srcs)
in the bazel build
- Fix an include statement - in bazel, an app is not available to
itself as a library at compilation time
bazel-erlang has been renamed rules_erlang. v2 is a substantial
refactor that brings Windows support. While this alone isn't enough to
run all rabbitmq-server suites on windows, one can at least now start
the broker (bazel run broker) and run the tests that do not start a
background broker process
David Ansari
Rin Kuryloski
Jean-Sébastien Pédron
Rin Kuryloski
Alexey Lebedeff
Michal Kuratczyk
Jean-Sébastien Pédron
Alexey Lebedeff