Commit Graph

50928 Commits

Author SHA1 Message Date
Michael Klishin dd1d2ee9a4
Merge pull request #5125 from rabbitmq/microsecond
Use microsecond abbreviation 'us' instead of 'µs'
2022-06-30 20:46:20 +04:00
David Ansari ad27021630 Use microsecond abbreviation 'us' instead of 'µs'
'us' is used when Unicode is not available.

Prior to this commit:
```
$ kubectl logs r1-server-0 -c rabbitmq | ag time
2022-06-30 13:37:35.253927+00:00 [debug] <0.336.0> wal: recovered 00000003.wal time taken 0ms
2022-06-30 13:37:35.262592+00:00 [debug] <0.349.0> wal: recovered 00000003.wal time taken 0ms
2022-06-30 13:37:35.489016+00:00 [debug] <0.352.0> Feature flags: time to find supported feature flags: 76468 �s
2022-06-30 13:37:35.495193+00:00 [debug] <0.352.0> Feature flags: time to regen registry: 6032 �s
2022-06-30 13:37:35.500574+00:00 [debug] <0.361.0> Feature flags: time to find supported feature flags: 937 �s
2022-06-30 13:37:35.500603+00:00 [debug] <26705.398.0> Feature flags: time to find supported feature flags: 891 �s
2022-06-30 13:37:35.507998+00:00 [debug] <26705.398.0> Feature flags: time to regen registry: 7199 �s
2022-06-30 13:37:35.509092+00:00 [debug] <0.361.0> Feature flags: time to regen registry: 8396 �s
```
2022-06-30 13:54:25 +00:00
David Ansari 5ebbb75c01 Fix log format in consistent hash exchange
Before this commit:
```
Consistent hashing exchange: removing binding from exchange '"exchange 'e1' in vhost '/'"' to destination '"queue 'qq2' in vhost '/'"' with routing key '2'
```

The single quotes do not make sense.
2022-06-30 13:19:11 +00:00
Michael Klishin b48dd093d6 rabbit_binding:list_between/2: consider all route tables 2022-06-30 16:21:32 +04:00
Michael Klishin e8af9881f9 Introduce rabbit_binding:list_between/2
and has_any_between/2

(cherry picked from commit 088b7a63a9)
2022-06-30 15:34:00 +04:00
Michael Klishin 1cfa2aadd4
Merge pull request #5121 from rabbitmq/chx-node-restart
Make adding bindings idempotent
2022-06-30 15:33:18 +04:00
David Ansari ba22b3307f Add some function and type specs 2022-06-30 09:45:46 +00:00
David Ansari 878f369b7a Make adding bindings idempotent
First binding wins.
Duplicate bindings, i.e. bindings with the same source exchange and
same destination queue / exchange but possibly different routing key
(weight) are ignored from now on by the consistent hash exchange.

This applies only to bindings being added.
For bindings being deleted, any duplicate binding (independent of its
routing key) will delete all buckets for the given source and
destination. (This is to ensure that buckets for a given source and
destination can be deleted for when upgrading from a version prior
to this commit. This was also the behaviour prior to this commit,
so nothing changes in that regard.)

Note that duplicate bindings continue to be created in RabbitMQ.
(They are only ignored by the consistent hash exchange.)

Adding a binding will perform linear search in the bucket map.
This is already stated in the README:
"These two operations use linear algorithms to update the ring."

The linear search when adding a binding could be optimised by
adding another Mnesia table field which will require a new migration and
feature flag. Hence, such an optimization is left out in this commit.

Fixes #3386.
2022-06-30 09:24:02 +00:00
Jean-Sébastien Pédron df080b89cf
rabbit_feature_flags: Fix a stale comment about experimental feature flags 2022-06-28 16:41:02 +02:00
Jean-Sébastien Pédron a48a4a09c4
Merge pull request #3940 from rabbitmq/add-feature-flags-controller
Add a feature flags controller
2022-06-28 13:33:59 +02:00
Jean-Sébastien Pédron 0fd051234e
Git: Don't ignore `rabbitmq_ct_helpers` and `rabbitmq_ct_client_helpers` in deps 2022-06-28 11:08:29 +02:00
Jean-Sébastien Pédron da807547f5
rabbit_core_ff: Remove no-op migration functions
There is no need to add a migration function which does nothing.
Ignoring the third argument and always returning `ok` is incorrect
anyway.
2022-06-28 10:13:19 +02:00
Jean-Sébastien Pédron cefb6aaef5
cth_log_redirect_any_domains: common_test hook to handle domain-specific log messages
common_test installs its own logger handler, which is great.
Unfortunately, this logger handler drops all messages having a domain,
except when the domain is among the domains used by Erlang itself.

In RabbitMQ, we use logger domains to categorize messages. Therefore
those messages are dropped by the common_test's logger handler.

This commit introduces another logger handler which sits on top of the
common_test one and makes sure messages with a domain are logged as
well.
2022-06-28 10:13:19 +02:00
Jean-Sébastien Pédron bcb8733880
rabbit_feature_flags: Add a feature flags controller process
This gen_statem-based process is responsible for handling concurrency
when feature flags are enabled and synchronized when a cluster is
expanded.

This clarifies and stabilizes the behavior of the feature flag subsystem
w.r.t. situations where e.g. a feature flag migration function takes
time to update data and a new node joins a cluster and synchronizes its
feature flag states with the cluster. There was a chance that the
feature flag was marked as enabled on the joining node, even though the
migration function didn't take care of that node.

With this new feature flags controller, enabling or synchronizing
feature flags blocks and delays any concurrent operations which try to
modify feature flags states too.

This change also clarifies where and when the migration function is
called: it is called at least once on each node who knows the feature
flag and when the state goes from "disabled" to "enabled" on that node.

Note that even if the feature flag is being enabled on a subset of the
nodes (because other nodes already have it enabled), it is marked as
"state_changing" everywhere during the migration. This is to prevent
that a node where it is enabled assumes it is enabled on all nodes who
know the feature flag.

There is a new feature as well: just after a feature flag is enabled,
the migration function is called a second time for any post-enable
actions. The feature flag is marked as enabled between these "enable"
and "post-enable" steps. The success or failure of this "post-enable"
run does not affect the state of the feature flag (i.e. it is ignored).

A new migration function API is introduced to allow more advanced
things. The new API is:

    my_migration_function(
      #ffcommand{name = ...,
                 props = ...,
		 command = enable | post_enable,
		 extra = #{...}})

The record is defined in `include/feature_flags.hrl`. Here is the
meaning of each field:

* `name` and `props` are the equivalent of the `FeatureName` and
  `FeatureProps` arguments of the previous migration function API.

* `command` is basically the same as the previous `Arg` arguments.

* `extra` is map containing context-specific information. For instance, it
  contains the list of nodes where the feature flag state changes.

This whole new behavior is behind a new feature flag called
`feature_flags_v2`. If a feature flag uses the new migration function
API, `feature_flags_v2` will be automatically enabled.

If many feature flags are enabled at once (like when a fresh RabbitMQ
node is started for the first time), `feature_flags_v2` will be enabled
first if it is in the list.
2022-06-28 10:13:19 +02:00
Michael Klishin f95d2ba4b4
Merge pull request #5113 from rabbitmq/unique-extra-bcc
Do not route to duplicate extra BCC destinations
2022-06-28 00:01:39 +04:00
David Ansari 0ec9566e95 Do not route to duplicate extra BCC destinations 2022-06-27 17:17:20 +00:00
Michael Klishin 2d12f8bb86
Merge pull request #5107 from Ayanda-D/handle-qq-timeouts
Handle quorum queue timeout errors on rare 'basic.get' operations
2022-06-27 08:42:57 +04:00
Ayanda Dube e6dce57c8c Handle quorum queue timeout errors from rabbit_fifo_client:dequeue/3 on 'basic.get' 2022-06-26 22:37:33 +01:00
Michael Klishin adb6e0468d
Merge pull request #5078 from rabbitmq/test-logs-enhancements
Allow `bazel run test-logs` to work when logs are compressed
2022-06-22 12:15:56 +04:00
Michael Klishin bebe9654a0
Merge pull request #5085 from rabbitmq/amqp-stream-consumer-file-handle-leak-fix 2022-06-21 23:17:23 +04:00
Karl Nilsson 8635ded7fe Close osiris log when cancelling AMQP stream consumer
Else we'd end up leaking file handles.
2022-06-21 16:22:13 +01:00
Philip Kuryloski 8facde8f8d Allow `bazel run test-logs` to work when logs are compressed
As happens if the spawn_strategy is not set to local

Also makes the implementaion of related commands more consistent
2022-06-20 18:18:22 +02:00
Jean-Sébastien Pédron 2e3ba4c1d7
unit_config_value_encryption_SUITE: Fix log message + add stacktrace
The format string started with "~s" but there was no corresponding
argument. I just removed the "~s".

While here, the log message now contains the stacktrace too.
2022-06-20 13:37:20 +02:00
Jean-Sébastien Pédron 3eb6374e4e
rabbitmq_cli: Take "make" command from $MAKE env. variable
This unbreaks the build of rabbitmq_cli on platforms where GNU Make is
installed under another name than `make`. This is the case on Mac OSX
and *BSD for instance where GNU Make is available as `gmake`.
2022-06-20 11:36:02 +02:00
Rin Kuryloski 3bbd28a50a
Merge pull request #5058 from rabbitmq/add-missing-module-deps
Add missing bazel deps
2022-06-18 17:49:20 +02:00
Philip Kuryloski 4b870a5b98 Adjustments to :package-generic-unix for rules_pkg 0.5.1 2022-06-18 14:12:28 +02:00
Michael Klishin cb78795fe8
Merge pull request #5068 from rabbitmq/bump-rbe
Adopt latest rabbitmq-server-buildenv:linux-rbe for RBE
2022-06-18 12:04:38 +04:00
GitHub 71e93fc201 Use latest rbe-erlang-platform/linux-rbe
for remote build execution (RBE) with BuildBuddy
2022-06-18 03:29:53 +00:00
Philip Kuryloski 9b73834cdd Match the rules_pkg version across WORKSPACE.bazel and MODULE.bazel 2022-06-18 00:26:10 +02:00
Philip Kuryloski 7c76498dca Add minimal conflict resolution for the rbe module extension
This is a very simple strategy - multiple declarations of the same
repository for rbe are allowed, only if identical
2022-06-17 09:16:14 +00:00
Philip Kuryloski 4afd473af3 Additional fixes for use of rabbitmq-server as a bazel module
- add more missing deps
- add an extension that allows @rbe to be fetched via bzlmod
2022-06-17 09:16:14 +00:00
Philip Kuryloski 136630841c Add missing bazel deps 2022-06-17 09:16:14 +00:00
Rin Kuryloski cb9950dc29
Merge pull request #5059 from rabbitmq/more-oci-flake-mitigations
Again try to make Dockerfile more cacheable
2022-06-17 11:14:12 +02:00
Philip Kuryloski 178a7fa539 Again try to make Dockerfile more cacheable 2022-06-17 10:58:43 +02:00
Rin Kuryloski 843911919d
Merge pull request #5051 from rabbitmq/nobzlmod-build-fixes
Fixup the bazel build when used without bzlmod
2022-06-15 11:51:58 +02:00
Philip Kuryloski 2fd155e55d Fixup the bazel build when used without bzlmod
As currently we do not use bzlmod when building on Windows
2022-06-15 11:18:41 +02:00
Rin Kuryloski 295f6e2afb
Merge pull request #5044 from rabbitmq/improve-oci-cache-hits
Attempt to improve the cache hit rate for the oci.yaml workflow
2022-06-15 09:24:04 +02:00
Michael Klishin fcd45342c8
Merge pull request #5045 from rabbitmq/fixup-local-build 2022-06-14 20:41:38 +04:00
Philip Kuryloski 09fa85e570 Remove extra tar options not supported on macos
They would have improved the reproducibility of the :source_archive
rule, but are not essential. Fixes the macos breakage introduced by
5027.
2022-06-14 17:53:49 +02:00
Philip Kuryloski 0dcf896677 Attempt to improve the cache hit rate for the oci.yaml workflow 2022-06-14 15:30:55 +02:00
Rin Kuryloski e80022428f
Merge pull request #5027 from rabbitmq/bazel-source-archive
Add a rule for building a source archive
2022-06-14 15:02:01 +02:00
Michael Klishin 8b41ea0960
Merge pull request #5036 from rabbitmq/improve-stream-log-msg
Add action to stream coordinator warning
2022-06-14 17:01:07 +04:00
Michael Klishin ceec38505e
Merge pull request #5037 from rabbitmq/stream-log-level-connection-closed 2022-06-14 16:38:10 +04:00
David Ansari b2e821e772 Change normal connection close log level from warning to debug
This line gets logged when the client closes the connection to the
stream port before it authenticates successfully.

Some external load balancers for example connect to the stream port to
do health checks without sending any stream protocol frame.

This commits prevents the RabbitMQ log from being polluted.
2022-06-14 14:12:33 +02:00
Karl Nilsson dc35e6dc8b Add action to stream coordinator warning 2022-06-14 10:13:50 +01:00
Philip Kuryloski 34db1a3c38 Add a rule for building a source archive
that includes the external dependencies
2022-06-14 10:44:09 +02:00
Rin Kuryloski 4fd20ad376
Merge pull request #5025 from rabbitmq/fix-rabbitmqctl-missing-libs
Fixup rabbitmqctl build for missing formatters & otp 25
2022-06-13 16:09:05 +02:00
Philip Kuryloski 247a5ba91e Fixup an echo statement in oci.yaml 2022-06-13 12:43:35 +02:00
Philip Kuryloski 1633c00638 Use rules_erlang 3.2.0 2022-06-13 12:13:54 +02:00
Philip Kuryloski f52efbb909 Fixup rabbitmqctl build for missing formatters & otp 25 2022-06-13 12:08:57 +02:00