Commit Graph

59086 Commits

Author SHA1 Message Date
Diana Parra Corbacho e9a365b20e tests: clustering_prop_SUITE force stats on every wait 2024-11-14 15:02:47 +01:00
Diana Parra Corbacho 067a54aa40 tests: clustering_SUITE wait for metrics 2024-11-14 15:02:47 +01:00
David Ansari de804d1fa7 Support publishing AMQP 1.0 to Event Exchange
## What?

Prior to this commit, the `rabbitmq_event_exchange` internally published
always AMQP 0.9.1 messages to the `amq.rabbitmq.event` topic exchange.
This commit allows users to configure the plugin to publish AMQP 1.0
messages instead.

 ## Why?

Prior to this commit, when an AMQP 1.0 client consumed events,
event properties that are lists were omitted. For example property
`client_properties` of event `connection.created` or property
`arguments` of event `queue.created` were omitted because of the following sequence:
1. The event exchange plugins listens for all kind of internal events.
2. The event exchange plugin re-publishes all events as AMQP 0.9.1 message to the event exchange.
3. Later, when an AMQP 1.0 client consumes this message, the broker must translate the message from AMQP 0.9.1 to AMQP 1.0.
4. This translation follows the rules outlined in https://www.rabbitmq.com/docs/conversions#amqpl-amqp
5. Specifically, in this table the row before the last one describes the rule we're hitting here. It says that if the AMQP 0.9.1
header value is not an `x-` prefixed header and its value is an array or table, then this header is not converted.
That's because AMQP 1.0 application-properties must be simple types as mandated in https://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#type-application-properties

 ## How?

The user can configure the plugin as follows to have the plugin
internally publish AMQP 1.0 messages:
```
event_exchange.protocol = amqp_1_0
```

To support complex types such as lists, the plugin sets all event
properties as AMQP 1.0 message-annotations. The plugin prefixes all message
annotation keys with `x-opt-` to comply with the AMQP 1.0 spec.

 ## Alternative Design

An alternative design would have been to format all event properties
e.g. as JSON within the message body. However, this breaks routing on
specific event property values via a headers exchange.

 ## Documentation
https://github.com/rabbitmq/rabbitmq-website/pull/2129
2024-11-14 12:52:09 +01:00
Anh Nguyen dc9311a561 Update Erlang Distribution dashboard panel and instance filtering
- Modified metric expression and legend format in State of distribution links
- Changed panel type from 'flant-statusmap-panel' to 'status-history' for Process state
2024-11-14 11:04:07 +07:00
Michael Klishin 27952938f8
Merge pull request #12712 from rabbitmq/gh_12608
QQ: reduce memory use when dropping many messages at once.
2024-11-13 11:42:20 -05:00
Anh Nguyen b9dc0ea3b4 Add instance filtering to Erlang BEAM Grafana dashboard metrics
- Updated metric expressions to include instance filtering with {instance=\"$node\"}
  for the following metrics:
  - erlang_vm_statistics_run_queues_length
  - erlang_vm_statistics_dirty_io_run_queue_length
  - erlang_vm_statistics_dirty_cpu_run_queue_length
- Added 'DS_PROMETHEUS' as a templated data source variable
2024-11-13 20:20:02 +07:00
Karl Nilsson bfa293ab3b QQ: reduce memory use when dropping many messages at once.
As may happen when a max_length configuration change is made
when there are many messages on the queue.
2024-11-13 09:07:40 +00:00
Michael Klishin c78bc8a9c3
4.1: Avoid an exception when an AMQP 0-9-1-originating message with expiration set is converted for an MQTT consumer (#12710)
* MQTT: avoid an exception

when an AMQP 0-9-1 publisher publishes a message
that has expiration set.

Stack trace was contributed in #12707 by @rdsilio.

* mc_mqtt_SUITE test for #12707 #12710

* MQTT protocol_interop_SUITE: new test for #12710 #12707

* Simplify tests

---------

Co-authored-by: David Ansari <david.ansari@gmx.de>
2024-11-13 09:20:43 +01:00
Michael Klishin 6c16b4dd64
4.0.1 release notes: clarify that Blue/Green deployments are an option for 3.13.x clusters with Khepri enabled 2024-11-12 19:26:21 -05:00
Michael Klishin 62a4c0a922
Continuous build workflow: trigger when rabbitmq-components.mk changes 2024-11-12 11:46:18 -05:00
Michael Klishin e09b02fa46
Merge pull request #12704 from rabbitmq/osiris-v1.8.4
Osiris v1.8.4
2024-11-12 11:26:48 -05:00
Karl Nilsson 5f4715979c Osiris v1.8.4
This osiris release contains a bug fix that would cause an osiris
member to crash during recovery if certain unexpected files
were present in the log directory. (.e.g ".nfsXXXXXXXXXXXX") type
files used by the NFS file system when in use files are deleted.
2024-11-12 10:20:38 +00:00
Michael Klishin aba62b9d12
Mention node_tags #12702 in rabbitmq.conf.example 2024-11-11 22:56:47 -05:00
Michael Klishin 859fd03d0d
Merge pull request #12702 from rabbitmq/su_aws/node_tags
Make it possible to set some node-specific metadata using tags
2024-11-11 18:34:26 -05:00
Michael Klishin 94c8f01699
rabbitmq-diagnostics status: handle output of 3.13.x and previously released 4.0.x nodes
In a mixed cluster environment,
'rabbitmq-diagnostics status' can hit a node
that does not return any node tags.

Be more defensive and handle such cases
by simply displaying "(none)" for such
values.
2024-11-11 17:41:33 -05:00
Simon Unge 3d35416635 Node tags local to broker, add to /api/overview output and ctl status command 2024-11-11 20:49:21 +00:00
Jean-Sébastien Pédron 638e3a4b08
rabbit_amqqueue: Add `is_feature_used` callback to `transient_nonexcl_queues` depr. feature
[Why]
Without this callback, the deprecated features subsystem can't report if
the feature is used or not.

This reduces the usefulness of the HTTP API endpoint or the CLI command
that help verify if a cluster is using deprecated features.

[How]
The callback counts transient non-exclusive queues and return `true` if
there are one or more of them.

References #12619.
2024-11-11 15:57:52 +01:00
Jean-Sébastien Pédron ddaea6facb
CLI: Finish `check_if_any_deprecated_features_are_used` implementation
[Why]
The previous implementation bypassed the deprecated features subsystem.
It only cared about classic mirrored queues and called some
queue-related code directly to determine if this specific feature was
used.

[How]
The command code is simplified by calling the deprecated subsystem to
list used deprecated features instead.

References #12619.
2024-11-11 10:13:25 +01:00
Michael Klishin 38bc831ad9
Alpha workflows: drop values that are no longer used 2024-11-11 01:13:32 -05:00
Michael Klishin d224d01cca
Alpha workflows: use a short SHA here 2024-11-11 00:57:02 -05:00
Michael Klishin df7fac06d8
Alpha workflows: fixes 2024-11-11 00:33:14 -05:00
Michael Klishin 07b694f1e3
4.1.x alpha workflows: syntax correction 2024-11-11 00:27:54 -05:00
Michael Klishin 89ce948e5b
Alpha workflows: generate and set prerelease_identifier (again) 2024-11-11 00:26:37 -05:00
Michael Klishin 10a7706e83
release-4.1.x-alphas workflow: trigger a run 2024-11-10 23:04:41 -05:00
Michael Klishin eaaacbeb99
release-4.0.x-alphas workflow: sync with its 4.1.x counterpart 2024-11-10 22:57:02 -05:00
Michael Klishin 4d961fefcc
4.1.x alpha release workflow: rely on github.event.repository.pushed_at 2024-11-10 22:52:53 -05:00
Michael Klishin 440de4673e
4.1.x alpha release workflow: WIP 2024-11-10 22:50:51 -05:00
Michael Klishin 6d8ea93fc5
Trigger a new 4.1.x alpha build workflow 2024-11-10 22:18:30 -05:00
Michael Klishin 4f4ab35790
Trigger a new 4.1.x alpha build workflow 2024-11-10 21:51:36 -05:00
Michael Klishin 0810e2832d
Trigger a new 4.1.x alpha build workflow 2024-11-10 21:48:21 -05:00
Michael Klishin aff0bc814a
4.1.x alpha workflow: try release-kit/unix-timestamp@v1 2024-11-10 21:44:20 -05:00
Michael Klishin e988b7e660
Trigger a new 4.1.x alpha build workflow 2024-11-10 21:26:45 -05:00
Michael Klishin 2acc51ddc5
Trigger a new 4.1.x alpha build workflow 2024-11-10 20:52:13 -05:00
Michael Klishin 074f38d16f
Trigger a new 4.1.x alpha build workflow 2024-11-10 20:29:53 -05:00
Michael Klishin b43a7263f5
List cluster_tags in rabbitmq.conf.example #12552 #12659 #12699 2024-11-10 20:26:24 -05:00
Michael Klishin f5801be6db
Merge pull request #12659 from rabbitmq/su_aws/cluster_tag
Make it possible to set some cluster metadata besides the name using tags
2024-11-10 19:17:53 -05:00
Michael Klishin 7c66fba0c3
Make it possible to clear cluster_tags via rabbitmq.conf 2024-11-10 14:38:34 -05:00
Michael Klishin 9e649aefc0
We no longer use 'maybe' in this module 2024-11-10 14:35:14 -05:00
Michael Klishin e5d805ea6d
Cluster tags: set unconditionally
Otherwise once set, it would not be possible
to change them by updating rabbitmq.conf
2024-11-10 14:30:51 -05:00
Michael Klishin 6b614fc879
rabbitmq.conf.example: add management.http.max_body_size 2024-11-09 18:02:16 -05:00
Michael Klishin 961e5c5a21
Undo the Bazel-related change from #12696
(cherry picked from commit a66c926985)
2024-11-09 17:47:06 -05:00
Michael Klishin 673826425a
Merge pull request #12696 from rabbitmq/mk-http-api-lower-body-length-limit-for-binding-creation
HTTP API: reduce body size limit for the endpoint used to bind queues/streams/exchanges
2024-11-09 17:13:03 -05:00
Michael Klishin 3dc5c463a4
Pass Dialyzer 2024-11-09 16:53:45 -05:00
Michael Klishin b0abf88aa8
rabbit_mgmt_util: minor refactoring 2024-11-09 16:38:48 -05:00
Michael Klishin fb300d2a4b
HTTP API: limit default body size for binding creation
It does not need to use the "worst case scenario"
default HTTP request body size limit that
is primarily necessary because definition imports
can be large (MiBs in size, for example).

Since exchange, queue names and routing key
have limits of 255 bytes and optional arguments
can practically be expected to be short, we
can lower the limit to < 10 KiB.
2024-11-09 16:16:35 -05:00
Simon Unge eeea517da5 Store tags in global parameters 2024-11-08 21:41:49 +00:00
Simon Unge f5ef64ad06 Add cluster tag config that is exposed via HTTP /api/overview and CTL cluster_status 2024-11-08 21:05:02 +00:00
Michael Klishin 8b554474a6
Simplify 4.1.x and 4.0.x alpha release workflows
timestamping is now performed in rabbitmq/server-packages
2024-11-08 11:22:06 -05:00
David Ansari d309ec9afd
Merge pull request #12694 from rabbitmq/mqtt_flake_will_delay_session_takeover
Fix flake will_delay_session_takeover
2024-11-08 16:26:32 +01:00
David Ansari ae423721ad Fix flake will_delay_session_takeover
Prior to this commit, the following flake occurred in CI for
```
make -C deps/rabbitmq_mqtt ct-v5 t=cluster_size_1:will_delay_session_takeover
```

```
=== Location: [{v5_SUITE,will_delay_session_takeover,1473},
              {test_server,ts_tc,1793},
              {test_server,run_test_case_eval1,1302},
              {test_server,run_test_case_eval,1234}]
=== === Reason: {test_case_failed,"Received unexpected PUBLISH payload. Expected: <<\"will-3a\">> Got: <<\"will-4a\">>"}
```

The RabbitMQ logs for this single node test show:
```
2024-11-04 14:43:35.039196+00:00 [debug] <0.1334.0> MQTT accepting TCP connection <0.1334.0> (127.0.0.1:42576 -> 127.0.0.1:27005)
2024-11-04 14:43:35.039336+00:00 [debug] <0.1334.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: []
2024-11-04 14:43:35.039438+00:00 [debug] <0.1334.0> MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.039537+00:00 [debug] <0.1334.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.039729+00:00 [info] <0.1334.0> Accepted MQTT connection 127.0.0.1:42576 -> 127.0.0.1:27005 for client ID c3
2024-11-04 14:43:35.040297+00:00 [debug] <0.1337.0> MQTT accepting TCP connection <0.1337.0> (127.0.0.1:42580 -> 127.0.0.1:27005)
2024-11-04 14:43:35.040442+00:00 [debug] <0.1337.0> Received a CONNECT, client ID: c4, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: []
2024-11-04 14:43:35.040534+00:00 [debug] <0.1337.0> MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.040597+00:00 [debug] <0.1337.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.040793+00:00 [info] <0.1337.0> Accepted MQTT connection 127.0.0.1:42580 -> 127.0.0.1:27005 for client ID c4
2024-11-04 14:43:35.041463+00:00 [debug] <0.1340.0> MQTT accepting TCP connection <0.1340.0> (127.0.0.1:42596 -> 127.0.0.1:27005)
2024-11-04 14:43:35.041715+00:00 [debug] <0.1340.0> Received a CONNECT, client ID: c1, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2024-11-04 14:43:35.041806+00:00 [debug] <0.1340.0> MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.041881+00:00 [debug] <0.1340.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.041982+00:00 [warning] <0.1328.0> MQTT disconnecting client <<"127.0.0.1:42560 -> 127.0.0.1:27005">> with duplicate id 'c1'
2024-11-04 14:43:35.042062+00:00 [info] <0.1340.0> Accepted MQTT connection 127.0.0.1:42596 -> 127.0.0.1:27005 for client ID c1
2024-11-04 14:43:35.045624+00:00 [debug] <0.1345.0> MQTT accepting TCP connection <0.1345.0> (127.0.0.1:42602 -> 127.0.0.1:27005)
2024-11-04 14:43:35.045781+00:00 [debug] <0.1345.0> Received a CONNECT, client ID: c2, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2024-11-04 14:43:35.045874+00:00 [debug] <0.1345.0> MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.045943+00:00 [debug] <0.1345.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.046032+00:00 [warning] <0.1331.0> MQTT disconnecting client <<"127.0.0.1:42566 -> 127.0.0.1:27005">> with duplicate id 'c2'
2024-11-04 14:43:35.046281+00:00 [info] <0.1345.0> Accepted MQTT connection 127.0.0.1:42602 -> 127.0.0.1:27005 for client ID c2
2024-11-04 14:43:35.047063+00:00 [debug] <0.1350.0> MQTT accepting TCP connection <0.1350.0> (127.0.0.1:42614 -> 127.0.0.1:27005)
2024-11-04 14:43:35.047702+00:00 [debug] <0.1350.0> Received a CONNECT, client ID: c3, username: undefined, clean start: true, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2024-11-04 14:43:35.047910+00:00 [debug] <0.1350.0> MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.048467+00:00 [debug] <0.1350.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.049701+00:00 [info] <0.1350.0> Accepted MQTT connection 127.0.0.1:42614 -> 127.0.0.1:27005 for client ID c3
2024-11-04 14:43:35.050907+00:00 [warning] <0.1334.0> MQTT disconnecting client <<"127.0.0.1:42576 -> 127.0.0.1:27005">> with duplicate id 'c3'
2024-11-04 14:43:35.051248+00:00 [debug] <0.1353.0> MQTT accepting TCP connection <0.1353.0> (127.0.0.1:42626 -> 127.0.0.1:27005)
2024-11-04 14:43:35.051395+00:00 [debug] <0.1353.0> Received a CONNECT, client ID: c4, username: undefined, clean start: false, protocol version: 5, keepalive: 60, property names: ['Session-Expiry-Interval']
2024-11-04 14:43:35.051519+00:00 [debug] <0.1353.0> MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 picked vhost using plugin_configuration_or_default_vhost
2024-11-04 14:43:35.051590+00:00 [debug] <0.1353.0> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
2024-11-04 14:43:35.051871+00:00 [info] <0.1353.0> Accepted MQTT connection 127.0.0.1:42626 -> 127.0.0.1:27005 for client ID c4
2024-11-04 14:43:35.051960+00:00 [warning] <0.1337.0> MQTT disconnecting client <<"127.0.0.1:42580 -> 127.0.0.1:27005">> with duplicate id 'c4'
2024-11-04 14:43:35.052689+00:00 [debug] <0.1337.0> sent Will Message to topic my/topic for MQTT client ID c4
2024-11-04 14:43:35.054119+00:00 [debug] <0.1334.0> sent Will Message to topic my/topic for MQTT client ID c3
```

We see nicely how RabbitMQ sends the will message for both c3 and c4.
However, the order in which RabbitMQ sends is not guaranteed.
Hence, we adapt the test expectation to not depend on the order of Will
messages being received.
2024-11-08 16:12:52 +01:00