root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 6 Packages Projects Releases Wiki Activity
rabbitmq-server/deps/rabbit/Makefile

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

397 lines
16 KiB
Makefile
Raw Normal View History

Makefile: Use `erlang.mk`
2015-08-10 17:50:41 +08:00
PROJECT = rabbit
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
PROJECT_DESCRIPTION = RabbitMQ
PROJECT_MOD = rabbit
PROJECT_REGISTERED = rabbit_amqqueue_sup \
rabbit_direct_client_sup \
rabbit_log \
rabbit_node_monitor \
rabbit_router
using ?= operator
2009-09-22 21:59:30 +08:00
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
define PROJECT_ENV
[
{tcp_listeners, [5672]},
{num_tcp_acceptors, 10},
{ssl_listeners, []},
Bump rabbit.num_ssl_acceptors default to 10 Previously the value was limited to 1 due to a bug in Ranch 1.0: it didn't depend on the ssl app, which means Ranch/ssl shut down order was not guaranteed. This led to scary but entirely harmless exception traces in the log, one per TLS acceptor. The issue is no longer present in Ranch 1.3.x and 1.4.0. Per discussion with @essen.
2017-10-29 21:19:30 +08:00
{num_ssl_acceptors, 10},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{ssl_options, []},
Set the default vm_memory_high_watermark to 0.6 (#12161) The default of 0.4 was very conservative even when it was set years ago. Since then: - we moved to CQv2, which have much more predictable memory usage than (non-lazy) CQv1 used to - we removed CQ mirroring which caused large sudden memory spikes in some situations - we removed the option to store message payload in memory in quorum queues For the past two years or so, we've been running all our internal tests and benchmarks using the value of 0.8 with no OOMkills at all (note: we do this on Kubernetes where the Cluster Operators overrides the available memory levaing some additional headroom, but effectively we are still using more than 0.6 of memory).
2024-08-29 18:10:49 +08:00
{vm_memory_high_watermark, 0.6},
Default mem calc strategy should be "rss" This is due to a scenario in which the Erlang VM allocator stats report a huge increase in memory consumption which is only reflected in VSS increase, not RSS PT #152081051
2017-10-19 05:33:39 +08:00
{vm_memory_calculation_strategy, rss},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{disk_free_limit, 50000000}, %% 50MB
{backing_queue_module, rabbit_variable_queue},
%% 0 ("no limit") would make a better default, but that
%% breaks the QPid Java client
{frame_max, 131072},
Change channel_max default to 2048 Closes #1593.
2018-05-09 23:06:20 +08:00
%% see rabbitmq-server#1593
Use 2047 Since channel 0 exists on every connection for negotiation and error communication. 655365 = (1 << 16) - 1, so 2047 = (1 << 11) - 1.
2018-05-10 01:39:19 +08:00
{channel_max, 2047},
Support handle-max ## What? 1. Support `handle-max` field in the AMQP 1.0 `begin` frame 2. Add a new setting `link_max_per_session` which defaults to 256. 3. Rename `session_max` to `session_max_per_connection` ## Why? 1. Operators might want to limit the number of links per session. A similar setting `consumer_max_per_channel` exists for AMQP 0.9.1. 2. We should use RabbitMQ 4.0 as an opportunity to set a sensible default as to how many links can be active on a given session simultaneously. The session code does iterate over every link in some scenarios (e.g. queue was deleted). At some point, it's better to just open 2nd session instead of attaching hundreds or thousands of links to a single session. A default `link_max_per_session` of 256 should be more than enough given that `session_max_per_connection` is 64. So, the defaults allow `256 * 64 = 16,384` links to be active on an AMQP 1.0 connection. (Operators might want to lower both defaults.) 3. The name is clearer given that we might introduce `session_max_per_node` in the future since `channel_max_per_node` exists for AMQP 0.9.1. ### Additional Context > Link handles MAY be reused once a link is closed for both send and receive. > To make it easier to monitor AMQP link attach frames, it is RECOMMENDED that > implementations always assign the lowest available handle to this field.
2024-09-06 16:12:24 +08:00
{session_max_per_connection, 64},
{link_max_per_session, 256},
See #7593. Use connection_max to stop connections in rabbitmq
2023-03-29 08:07:57 +08:00
{ranch_connection_max, infinity},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{heartbeat, 60},
{msg_store_file_size_limit, 16777216},
Message store settings we already have use the "msg_store" prefix
2020-06-04 01:36:17 +08:00
{msg_store_shutdown_timeout, 600000},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{fhc_write_buffering, true},
{fhc_read_buffering, false},
{queue_index_max_journal_entries, 32768},
{queue_index_embed_msgs_below, 4096},
{default_user, <<"guest">>},
{default_pass, <<"guest">>},
{default_user_tags, [administrator]},
{default_vhost, <<"/">>},
{default_permissions, [<<".*">>, <<".*">>, <<".*">>]},
{loopback_users, [<<"guest">>]},
{password_hashing_module, rabbit_password_hashing_sha256},
{server_properties, []},
{collect_statistics, none},
{collect_statistics_interval, 5000},
{mnesia_table_loading_retry_timeout, 30000},
{mnesia_table_loading_retry_limit, 10},
Add SASL mechanism ANONYMOUS ## 1. Introduce new SASL mechanism ANONYMOUS ### What? Introduce a new `rabbit_auth_mechanism` implementation for SASL mechanism ANONYMOUS called `rabbit_auth_mechanism_anonymous`. ### Why? As described in AMQP section 5.3.3.1, ANONYMOUS should be used when the client doesn't need to authenticate. Introducing a new `rabbit_auth_mechanism` consolidates and simplifies how anonymous logins work across all RabbitMQ protocols that support SASL. This commit therefore allows AMQP 0.9.1, AMQP 1.0, stream clients to connect out of the box to RabbitMQ without providing any username or password. Today's AMQP 0.9.1 and stream protocol client libs hard code RabbitMQ default credentials `guest:guest` for example done in: * https://github.com/rabbitmq/rabbitmq-java-client/blob/0215e85643a9ae0800822869be0200024e2ab569/src/main/java/com/rabbitmq/client/ConnectionFactory.java#L58-L61 * https://github.com/rabbitmq/amqp091-go/blob/ddb7a2f0685689063e6d709b8e417dbf9d09469c/uri.go#L31-L32 Hard coding RabbitMQ specific default credentials in dozens of different client libraries is an anti-pattern in my opinion. Furthermore, there are various AMQP 1.0 and MQTT client libraries which we do not control or maintain and which still should work out of the box when a user is getting started with RabbitMQ (that is without providing `guest:guest` credentials). ### How? The old RabbitMQ 3.13 AMQP 1.0 plugin `default_user` [configuration](https://github.com/rabbitmq/rabbitmq-server/blob/146b4862d8e570b344c99c37d91246760e218b18/deps/rabbitmq_amqp1_0/Makefile#L6) is replaced with the following two new `rabbit` configurations: ``` {anonymous_login_user, <<"guest">>}, {anonymous_login_pass, <<"guest">>}, ``` We call it `anonymous_login_user` because this user will be used for anonymous logins. The subsequent commit uses the same setting for anonymous logins in MQTT. Hence, this user is orthogonal to the protocol used when the client connects. Setting `anonymous_login_pass` could have been left out. This commit decides to include it because our documentation has so far recommended: > It is highly recommended to pre-configure a new user with a generated username and password or delete the guest user > or at least change its password to reasonably secure generated value that won't be known to the public. By having the new module `rabbit_auth_mechanism_anonymous` internally authenticate with `anonymous_login_pass` instead of blindly allowing access without any password, we protect operators that relied on the sentence: > or at least change its password to reasonably secure generated value that won't be known to the public To ease the getting started experience, since RabbitMQ already deploys a guest user with full access to the default virtual host `/`, this commit also allows SASL mechanism ANONYMOUS in `rabbit` setting `auth_mechanisms`. In production, operators should disable SASL mechanism ANONYMOUS by setting `anonymous_login_user` to `none` (or by removing ANONYMOUS from the `auth_mechanisms` setting. This will be documented separately. Even if operators forget or don't read the docs, this new ANONYMOUS mechanism won't do any harm because it relies on the default user name `guest` and password `guest`, which is recommended against in production, and who by default can only connect from the local host. ## 2. Require SASL security layer in AMQP 1.0 ### What? An AMQP 1.0 client must use the SASL security layer. ### Why? This is in line with the mandatory usage of SASL in AMQP 0.9.1 and RabbitMQ stream protocol. Since (presumably) any AMQP 1.0 client knows how to authenticate with a username and password using SASL mechanism PLAIN, any AMQP 1.0 client also (presumably) implements the trivial SASL mechanism ANONYMOUS. Skipping SASL is not recommended in production anyway. By requiring SASL, configuration for operators becomes easier. Following the principle of least surprise, when an an operator configures `auth_mechanisms` to exclude `ANONYMOUS`, anonymous logins will be prohibited in SASL and also by disallowing skipping the SASL layer. ### How? This commit implements AMQP 1.0 figure 2.13. A follow-up commit needs to be pushed to `v3.13.x` which will use SASL mechanism `anon` instead of `none` in the Erlang AMQP 1.0 client such that AMQP 1.0 shovels running on 3.13 can connect to 4.0 RabbitMQ nodes.
2024-08-14 18:19:17 +08:00
%% The identity to act as for anonymous logins.
{anonymous_login_user, <<"guest">>},
{anonymous_login_pass, <<"guest">>},
%% "The server mechanisms are ordered in decreasing level of preference."
%% AMQP §5.3.3.1
{auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'ANONYMOUS']},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{auth_backends, [rabbit_auth_backend_internal]},
{delegate_count, 16},
{trace_vhosts, []},
{ssl_cert_login_from, distinguished_name},
{ssl_handshake_timeout, 5000},
{ssl_allow_poodle_attack, false},
{handshake_timeout, 10000},
{reverse_dns_lookups, false},
{cluster_partition_handling, ignore},
{cluster_keepalive_interval, 10000},
Make autoheal state transition timeout configurable In continuation to #2208.
2020-01-12 00:23:30 +08:00
{autoheal_state_transition_timeout, 60000},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
{tcp_listen_options, [{backlog, 128},
{nodelay, true},
{linger, {true, 0}},
{exit_on_close, false}
]},
{ssl_apps, [asn1, crypto, public_key, ssl]},
%% see rabbitmq-server#227 and related tickets.
%% msg_store_credit_disc_bound only takes effect when
%% messages are persisted to the message store. If messages
%% are embedded on the queue index, then modifying this
%% setting has no effect because credit_flow is not used when
%% writing to the queue index. See the setting
%% queue_index_embed_msgs_below above.
Bump msg_store_io_batch_size according to #1098 Avoids a validation warning and if we increase initial message store disk bound credit to 4000, all values become proportionally increased (compared to < 3.6.7 defaults).
2017-02-18 03:31:29 +08:00
{msg_store_credit_disc_bound, {4000, 800}},
{msg_store_io_batch_size, 4096},
Mention #1098
2017-02-08 16:56:05 +08:00
%% see rabbitmq-server#143,
%% rabbitmq-server#949, rabbitmq-server#1098
Disable background GC by default; bump credit flow defaults Quite a bit of evidence of different kinds of workloads suggest that background GC does not contribute meaningfully to reducing node RAM usage on most of them. It does, however, on each run produce a massive spike in VM and CPU context switches, causing latency spikes. Some users report that running without background GC cuts their latency to a half or even a third. Multi-hour stress tests that max out all node CPU cores suggest that our current credit_flow default can still be bumped safely, reducing the amount of time publishers are throttled and making publisher latency and node's observed message throughput more predictable.
2017-02-05 19:41:40 +08:00
{credit_flow_default_credit, {400, 200}},
Changed or overridden default configuration values related to quorum queues Specifically: - changed default quorum_commands_soft_limit from 256 to 32 - override Ra wal_max_batch_size to 4096
2020-05-01 18:43:42 +08:00
{quorum_commands_soft_limit, 32},
Set default cluster size to 3
2021-01-28 17:46:28 +08:00
{quorum_cluster_size, 3},
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
%% see rabbitmq-server#248
%% and rabbitmq-server#667
{channel_operation_timeout, 15000},
Use the new URLs of the `www.rabbitmq.com` website They changed with the switch to Docusaurus. This avoids a redirect and gives cleaner search results.
2024-09-23 17:34:54 +08:00
%% See https://www.rabbitmq.com/docs/consumers#acknowledgement-timeout
Bump consumer acknowledgement timeout default to 30 minutes Apparently 15 minutes is not enough for some. 1 hour seems to be unreasonably long to our team, though. References #2990, #3032
2021-05-12 01:25:29 +08:00
%% 30 minutes
{consumer_timeout, 1800000},
Merge branch 'stable'
2016-12-07 22:21:58 +08:00
%% used by rabbit_peer_discovery_classic_config
{cluster_nodes, {[], disc}},
Makefile: Remove cipher/hash/iterations default values from default app. env. The code already queries `rabbit_pbe`.
2019-04-19 22:06:07 +08:00
{config_entry_decoder, [{passphrase, undefined}]},
Disable background GC by default; bump credit flow defaults Quite a bit of evidence of different kinds of workloads suggest that background GC does not contribute meaningfully to reducing node RAM usage on most of them. It does, however, on each run produce a massive spike in VM and CPU context switches, causing latency spikes. Some users report that running without background GC cuts their latency to a half or even a third. Multi-hour stress tests that max out all node CPU cores suggest that our current credit_flow default can still be bumped safely, reducing the amount of time publishers are throttled and making publisher latency and node's observed message throughput more predictable.
2017-02-05 19:41:40 +08:00
{background_gc_enabled, false},
Fix Makefile
2017-02-07 18:25:20 +08:00
{background_gc_target_interval, 60000},
Handle concurrent vhost creation and deletion better In the case of a vhost creation and immediate deletion those operations can end up executing concurrently. This changes a number of things to be more resilient: * Default vhost restart strategy changes to "continue" (previosly "ignore"). It's a tough default to pick but we believe the damage of a single vhost terminating the entire node that may have a lot of users is greater than the need to set up an alternative strategy for environments where only one vhost is used. Note that the event of message store or vhost supervision tree termination should be very rare. Per discussion with @kjnilsson. * For vhost [message store] recovery, we now log "no_such_vhost" errors in a sensible way instead of a bunch of process terminations. * Max restart intensity settings for rabbit_vhost_sup_wrapper are changed to accommodate integration test suites that rapidly set up temporary vhost and delete them. In most failure message store scenarios the new value is still low enough to lead to vhost supervision tree termination when recovery is impossible. There are also drive by doc changes. Fixes #1280.
2017-07-01 11:39:03 +08:00
%% rabbitmq-server#589
Add vhost_restart_strategy to default app config.
2017-05-08 19:58:01 +08:00
{proxy_protocol, false},
Naming
2017-04-12 18:28:25 +08:00
{disk_monitor_failure_retries, 10},
Merge branch 'master' into rabbitmq-server-1146-full Conflicts: Makefile
2017-05-08 23:01:10 +08:00
{disk_monitor_failure_retry_interval, 120000},
Handle concurrent vhost creation and deletion better In the case of a vhost creation and immediate deletion those operations can end up executing concurrently. This changes a number of things to be more resilient: * Default vhost restart strategy changes to "continue" (previosly "ignore"). It's a tough default to pick but we believe the damage of a single vhost terminating the entire node that may have a lot of users is greater than the need to set up an alternative strategy for environments where only one vhost is used. Note that the event of message store or vhost supervision tree termination should be very rare. Per discussion with @kjnilsson. * For vhost [message store] recovery, we now log "no_such_vhost" errors in a sensible way instead of a bunch of process terminations. * Max restart intensity settings for rabbit_vhost_sup_wrapper are changed to accommodate integration test suites that rapidly set up temporary vhost and delete them. In most failure message store scenarios the new value is still low enough to lead to vhost supervision tree termination when recovery is impossible. There are also drive by doc changes. Fixes #1280.
2017-07-01 11:39:03 +08:00
%% either "stop_node" or "continue".
Wording
2017-07-06 20:20:18 +08:00
%% by default we choose to not terminate the entire node if one
%% vhost had to shut down, see server#1158 and server#1280
Configurable default consumer prefetch for new channels rabbitmq-server#1367 [#151216783]
2017-10-27 17:30:29 +08:00
{vhost_restart_strategy, continue},
Explain
2017-10-28 06:41:30 +08:00
%% {global, prefetch count}
Quorum queues (#1706) * Test queue.declare method with quorum type [#154472130] * Cosmetics [#154472130] * Start quorum queue Includes ra as a rabbit dependency [#154472152] * Update info and list operations to use quorum queues Basic implementation. Might need an update when more functionality is added to the quorum queues. [#154472152] * Stop quorum queue [#154472158] * Restart quorum queue [#154472164] * Introduce UId in ra config to support newer version of ra Improved ra stop [#154472158] * Put data inside VHost specific subdirs [#154472164] * Include ra in rabbit deps to support stop_app/start_app command [#154472164] * Stop quorum queues in `rabbit_amqqueue:stop/1` [#154472158] * Revert creation of fifo ets table inside rabbit Now supported by ra [#154472158] * Filter quorum queues [#154472158] * Test restart node with quorum queues [#154472164] * Publish to quorum queues [#154472174] * Use `ra:restart_node/1` [#154472164] * Wait for stats to be published when querying quorum queues [#154472174] * Test publish and queue length after restart [#154472174] * Consume messages from quorum queues with basic.get [#154472211] * Autoack messages from quorum queues on basic.get [#154472211] * Fix no_ack meaning no_ack = true is equivalent to autoack [#154472211] * Use data_dir as provided in the config If we modify the data_dir, ra is not able to delete the data when a queue is deleted [#154472158] * Remove unused code/variables [#154472158] * Subscribe to a quorum queue Supports auto-ack [#154472215] * Ack messages consumed from quorum queues [#154472221] * Nack messages consumed from quorum queues [#154804608] * Use delivery tag as consumer tag for basic.get in quorum queues [#154472221] * Support for publisher confirms in quorum queues [#154472198] * Integrate with ra_fifo_client * Clear queue state on queue.delete [#154472158] * Fix quorum nack [#154804608] * Test redelivery after nack [#154804608] * Nack without requeueing [#154472225] * Test multiple acks [#154804208] * Test multiple nacks [#154804314] * Configure dead letter exchange with queue declare [#155076661] * Use a per-vhost process to handle dead-lettering Needs to hold state for quorum queues [#155401802] * Implement dead-lettering on nack'ed messages [#154804620] * Use queue name as a resource on message delivery Fixes a previously introduced bug [#154804608] * Handle ra events on dead letter process [#155401802] * Pass empty queue states to queue delete Queue deletion on vhost deletion calls directly to rabbit_amqqueue. Queue states are not available, but we can provide an empty map as in deletion the states are only needed for cleanup. * Generate quorum queue stats and events Consumer delete events are still pending, as depend on basic.cancel (not implemented yet), ra terminating or ra detecting channel down [#154472241] * Ensure quorum mapping entries are available before metric emission [#154472241] * Configure data_dir, uses new RABBITMQ_QUORUM_BASE env var [#154472152] * Use untracked enqueues when sending wihtout channel Updated several other calls missed during the quorum implementation * Revert "Configure data_dir, uses new RABBITMQ_QUORUM_BASE env var" This reverts commit f2261212410affecb238fcbd1fb451381aee4036. * Configure data_dir, uses new RABBITMQ_QUORUM_DIR based on mnesia dir [#154472152] * Fix get_quorum_state * Fix calculation of quorum pids * Move all quorum queues code to its own module [#154472241] * Return an error when declaring a quorum queue with an incompatible argument [#154521696] * Cleanup of quorum queue state after queue delete Also fixes some existing problems where the state wasn't properly stored [#155458625] * Revert Revert "Declare a quorum queue using the queue.declare method" * Remove duplicated state info [#154472241] * Start/stop multi-node quorum queue [#154472231] [#154472236] * Restart nodes in a multi-node quorum cluster [#154472238] * Test restart and leadership takeover on multiple nodes [#154472238] * Wait for leader down after deleting a quorum cluster It ensures an smooth delete-declare sequence without race conditions. The test included here detected the situation before the fix. [#154472236] * Populate quorum_mapping from mnesia when not available Ensures that leader nodes that don't have direct requests can get the mapping ra name -> queue name * Cosmetics * Do not emit core metrics if queue has just been deleted * Use rabbit_mnesia:is_process_alive Fixes bug introduced by cac9583e1bb2705be7f06c2ab7f416a75d11c875 [#154472231] * Only try to report stats if quorum process is alive * Implement cancel consumer callback Deletes metrics and sends consumer deleted event * Remove unnecessary trigger election call ra:restart_node has already been called during the recovery * Apply cancellation callback on node hosting the channel * Cosmetics * Read new fifo metrics which store directly total, ready and unack * Implement basic.cancel for quorum queues * Store leader in amqqueue record, report all in stats [#154472407] * Declare quorum queue in mnesia before starting the ra cluster Record needs to be stored first to update the leader on ra effects * Revert * Purge quorum queues [#154472182] * Improve use of untracked_enqueue Choose the persisted leader id instead of just using the id of the leader at point of creation. * Store quorum leader in the pid field of amqqueue record Same as mirrored queues, no real need for an additional field * Improve recovery When a ra node has never been started on a rabbit node ensure it doesn't fail but instead rebuilds the config and starts the node as a new node. Also fix issue when a quorum queue is declared when one of it's rabbit nodes are unavailable. [#157054606] * Cleanup core metrics after leader change [#157054473] * Return an error on sync_queue on quorum queues [#154472334] * Return an error on cancel_sync_queue on quorum queues [#154472337] * Fix basic_cancel and basic_consume return values Ensure the quorum queue state is always returned by these functions. * Restore arity of amqqeueu delete and purge functions. This avoids some breaking changes in the cli. * Fix bug returning consumers. * remove rogue debug log * Integrate ingress flow control with quorum queues [#157000583] * Configure commands soft limit [#157000583] * Support quorum pids on rabbit_mnesia:is_process_alive * Publish consumers metric for quorum queues * Whitelist quorum directory in is_virgin_node Allow the quorum directoy to exist without affecting the status of the Rabbit node. * Delete queue_metrics on leader change. Also run the become_leader handler in a separate process to avoid blocking. [#157424225] * Report cluster status in quorum queue infos. New per node status command. Related to [#157146500] * Remove quorum_mapping table As we can store the full queue name resource as the cluster id of the ra_fifo_client state we can avoid needed the quorum_mapping table. * Fix xref issue * Provide quorum members information in stats [#157146500] * fix unused variable * quorum queue multiple declare handling Extend rabbit_amqqueue:internal_declare/2 to indicate if the queue record was created or exisiting. From this we can then provide a code path that should handle concurrent queue declares of the same quorum queue. * Return an error when declaring exclusive/auto-delete quorum queue [#157472160] * Restore lost changes from 79c9bd201e1eac006a42bd162e7c86df96496629 * recover another part of commit * fixup cherry pick * Ra io/file metrics handler and stats publishing [#157193081] * Revert "Ra io/file metrics handler and stats publishing" This reverts commit 05d15c786540322583fc655709825db215b70952. * Do not issue confirms on node down for quorum queues. Only a ra_event should be used to issue positive confirms for a quorum queue. * Ra stats publishing [#157193081] * Pick consumer utilisation from ra data [#155402726] * Handle error when deleting a quorum queue and all nodes are already down This is in fact a successful deletion as all raft nodes are already 'stopped' [#158656366] * Return an error when declaring non-durable quorum queues [#158656454] * Rename dirty_query to committed_query * Delete stats on leader node [#158661152] * Give full list of nodes to fifo client * Handle timeout in quorum basic_get * Fix unused variable error * Handle timeout in basic get [#158656366] * Force GC after purge [#158789389] * Increase `ra:delete_cluster` timeout to 120s * Revert "Force GC after purge" This reverts commit 5c98bf22994eb39004760799d3a2c5041d16e9d4. * Add quorum member command [#157481599] * Delete quorum member command [#157481599] * Implement basic.recover for quorum queues [#157597411] * Change concumer utilisation to use the new ra_fifo table and api. * Set max quorum queue size limit Defaults to 7, can be configured per queue on queue.declare Nodes are selected randomly from the list of nodes, but the one that is executing the queue.declare command [#159338081] * remove potentially unrelated changes to rabbit_networking * Move ra_fifo to rabbit Copied ra_fifo to rabbit and renamed it rabbit_fifo. [#159338031] * rabbit_fifo tidy up * rabbit_fifo tidy up * rabbit_fifo: customer -> consumer rename * Move ra_fifo tests [#159338031] * Tweak quorum_queue defaults * quorum_queue test reliability * Optimise quorum_queue test suite. By only starting a rabbit cluster per group rather than test. [#160612638] * Renamings in line with ra API changes * rabbit_fifo fixes * Update with ra API changes Ra has consolidated and simplified it's api. These changes update to confirm to that. * Update rabbit_fifo with latest ra changes * Clean up out of date comment * Return map of states * Add test case for basic.get on an empty queue Before the previous patch, any subsequent basic.get would crash as the map of states had been replaced by a single state. * Clarify use of deliver tags on record_sent * Clean up queues after testcase * Remove erlang monitor of quorum queues in rabbit_channel The eol event can be used instead * Use macros to make clearer distinctions between quorum/classic queues Cosmetic only * Erase queue stats on 'eol' event * Update to follow Ra's cluster_id -> cluster_name rename. * Rename qourum-cluster-size To quorum-initial-group-size * Issue confirms on quorum queue eol Also avoid creating quorum queue session state on queue operation methods. * Only classic queues should be notified on channel down * Quorum queues do not support global qos Exit with protocol error of a basic.consume for a quorum queue is issued on a channel with global qos enabled. * unused variable name * Refactoring Strictly enfornce that channels do not monitor quorum queues. * Refactor foreach_per_queue in the channel. To make it call classic and quorum queues the same way. [#161314899] * rename function * Query classic and quorum queues separately during recovery as they should not be marked as stopped during failed vhost recovery. * Remove force_event_refresh function As the only user of this function, the management API no longer requires it. * fix errors * Remove created_at from amqqueue record [#161343680] * rabbit_fifo: support AMQP 1.0 consumer credit This change implements an alternative consumer credit mechanism similar to AMQP 1.0 link credit where the credit (prefetch) isn't automatically topped up as deliveries are settled and instead needs to be manually increased using a credit command. This is to be integrated with the AMQP 1.0 plugin. [#161256187] * Add basic.credit support for quorum queues. Added support for AMQP 1.0 transfer flow control. [#161256187] * Make quorum queue recover idempotent So that if a vhost crashes and runs the recover steps it doesn't fail because ra servers are still running. [#161343651] * Add tests for vhost deletion To ensure quorum queues are cleaned up on vhost removal. Also fix xref issue. [#161343673] * remove unused clause * always return latest value of queue * Add rabbitmq-queues scripts. Remove ra config from .bat scripts. * Return error if trying to get quorum status of a classic queue.
2018-10-29 17:47:29 +08:00
{default_consumer_prefetch, {false, 0}},
Report version series maintenance status via CLI as part of 'rabbitmq-diagnostics status'
2022-08-28 01:49:35 +08:00
%% interval at which the channel can perform periodic actions
Move consumer timeout tests to own SUITE Also handle case where client does not support consumer cancellation and rename the queue_cleanup timer to a generic "tick" timer for channels to perform periodic activities. [#164212469]
2019-04-08 23:12:26 +08:00
{channel_tick_interval, 60000},
Reduce default maximum message size to 16MB
2024-06-14 15:18:46 +08:00
%% Default max message size is 16 MB
{max_message_size, 16777216},
Bump default GC threshold to 1GB In our testing we observe no meaningfully different peak memory (carrier size) limit and 18 to 65% lower tail latencies with small messages, and teen % lower with 50 MB messages. Throughput is single digit % higher. CPU usage was not meaningfully different on a four core host. References rabbitmq/rabbitmq-common#343.
2019-12-20 13:02:51 +08:00
%% Socket writer will run GC every 1 GB of outgoing data
Introduce per-user channel tracking Make 'tracking_execution_timeout' configurable Add per_user_connection_channel_tracking_SUITE
2020-06-10 17:10:48 +08:00
{writer_gc_threshold, 1000000000},
%% interval at which connection/channel tracking executes post operations
Stream Queue This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
2020-09-29 18:43:24 +08:00
{tracking_execution_timeout, 15000},
Add auth attemp metrics
2020-08-26 22:42:40 +08:00
{stream_messages_soft_limit, 256},
Periodically reconcile virtual host processes for up to 10 times. When a cluster is formed from scratch and a virtual host is declared in the process (can be via definitions, or plugins, or any other way), that virtual host's process tree will be started on all reachable cluster nodes at that moment in time. However, this can be just a subset of all nodes expected to join the cluster. The most effective solution is to run this reconciliation process on a timer for up to 5 minutes by default. This matches how long some other parts of RabbitMQ (3.x) expect cluster formation to take, at most. Per discussion with @dcorbacho @mkuratczyk.
2024-05-24 01:42:51 +08:00
{track_auth_attempt_source, false},
{credentials_obfuscation_fallback_secret, <<"nocookie">>},
{dead_letter_worker_consumer_prefetch, 32},
{dead_letter_worker_publisher_confirm_timeout, 180000},
Introduce a way to disable virtual host reconciliation Certain test suites need virtual hosts to be down for an extended period of time, and this feature gets in a way ;)
2024-06-07 02:47:56 +08:00
{vhost_process_reconciliation_run_interval, 30},
%% for testing
Add a rabbit.license_line default so that products that build on top could adjust what's printed in the standard banner. References #12390
2024-10-02 08:23:38 +08:00
{vhost_process_reconciliation_enabled, true},
{license_line, "Licensed under the MPL 2.0. Website: https://rabbitmq.com"}
Move from .app.src to Makefile variables This is the recommended way with Erlang.mk. By default, the version is computed with git-describe(1) or inherited from rabbitmq-server-release when creating the source archive. One can override the version from the command line by setting the `PROJECT_VERSION` variable. One nice bonus is that `make run-broker` will present a meaningful version by default, and one can set the version with `PROJECT_VERSION`. [#130992027]
2016-12-06 19:18:02 +08:00
]
endef
rabbitmq_prelaunch: Adapt Makefiles and Bazel files after the move
2023-12-08 20:00:46 +08:00
LOCAL_DEPS = sasl os_mon inets compiler public_key crypto ssl syntax_tools xmerl
Stream Coordinator refactor
2021-01-14 01:12:14 +08:00
Remove looking_glass It has largely been superseded by `perf`. It is no longer generally useful. It can always be added to BUILD_DEPS for the rare cases it is needed, or installed locally and pointed to by setting its path to ERL_LIBS.
2024-06-26 15:56:46 +08:00
BUILD_DEPS = rabbitmq_cli
Fix make dialyze for deps/rabbit
2024-04-16 20:07:13 +08:00
DEPS = ranch rabbit_common amqp10_common rabbitmq_prelaunch ra sysmon_handler stdout_formatter recon redbug observer_cli osiris syslog systemd seshat khepri khepri_mnesia_migration cuttlefish gen_batch_server
Enable AMQP 1.0 clients to manage topologies ## What? * Allow AMQP 1.0 clients to dynamically create and delete RabbitMQ topologies (exchanges, queues, bindings). * Provide an Erlang AMQP 1.0 client that manages topologies. ## Why? Today, RabbitMQ topologies can be created via: * [Management HTTP API](https://www.rabbitmq.com/docs/management#http-api) (including Management UI and [messaging-topology-operator](https://github.com/rabbitmq/messaging-topology-operator)) * [Definition Import](https://www.rabbitmq.com/docs/definitions#import) * AMQP 0.9.1 clients Up to RabbitMQ 3.13 the RabbitMQ AMQP 1.0 plugin auto creates queues and bindings depending on the terminus [address format](https://github.com/rabbitmq/rabbitmq-server/tree/v3.13.x/deps/rabbitmq_amqp1_0#routing-and-addressing). Such implicit creation of topologies is limiting and obscure. For some address formats, queues will be created, but not deleted. Some of RabbitMQ's success is due to its flexible routing topologies that AMQP 0.9.1 clients can create and delete dynamically. This commit allows dynamic management of topologies for AMQP 1.0 clients. This commit builds on top of Native AMQP 1.0 (PR #9022) and will be available in RabbitMQ 4.0. ## How? This commits adds the following management operations for AMQP 1.0 clients: * declare queue * delete queue * purge queue * bind queue to exchange * unbind queue from exchange * declare exchange * delete exchange * bind exchange to exchange * unbind exchange from exchange Hence, at least the AMQP 0.9.1 management operations are supported for AMQP 1.0 clients. In addition the operation * get queue is provided which - similar to `declare queue` - returns queue information including the current leader and replicas. This allows clients to publish or consume locally on the node that hosts the queue. Compared to AMQP 0.9.1 whose commands and command fields are fixed, the new AMQP Management API is extensible: New operations and new fields can easily be added in the future. There are different design options how management operations could be supported for AMQP 1.0 clients: 1. Use a special exchange type as done in https://github.com/rabbitmq/rabbitmq-management-exchange This has the advantage that any protocol client (e.g. also STOMP clients) could dynamically manage topologies. However, a special exchange type is the wrong abstraction. 2. Clients could send "special" messages with special headers that the broker interprets. This commit decided for a variation of the 2nd option using a more standardized way by re-using a subest of the following latest AMQP 1.0 extension specifications: * [AMQP Request-Response Messaging with Link Pairing Version 1.0 - Committee Specification 01](https://docs.oasis-open.org/amqp/linkpair/v1.0/cs01/linkpair-v1.0-cs01.html) (February 2021) * [HTTP Semantics and Content over AMQP Version 1.0 - Working Draft 06](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=65571) (July 2019) * [AMQP Management Version 1.0 - Working Draft 16](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=65575) (July 2019) An important goal is to keep the interaction between AMQP 1.0 client and RabbitMQ simple to increase usage, development and adoptability of future RabbitMQ AMQP 1.0 client library wrappers. The AMQP 1.0 client has to create a link pair to the special `/management` node. This allows the client to send and receive from the management node. Similar to AMQP 0.9.1, there is no need for a reply queue since the reply will be sent directly to the client. Requests and responses are modelled via HTTP, but sent via AMQP using the `HTTP Semantics and Content over AMQP` extension (henceforth `HTTP over AMQP` extension). This commit tries to follow the `HTTP over AMQP` extension as much as possible but deviates where this draft spec doesn't make sense. The projected mode §4.1 is used as opposed to tunneled mode §4.2. A named relay `/management` is used (§6.3) where the message field `to` is the URL. Deviations are * §3.1 mandates that URIs are not encoded in an AMQP message. However, we percent encode URIs in the AMQP message. Otherwise there is for example no way to distinguish a `/` in a queue name from the URI path separator `/`. * §4.1.4 mandates a data section. This commit uses an amqp-value section as it's a better fit given that the content is AMQP encoded data. Using an HTTP API allows for a common well understood interface and future extensibility. Instead of re-using the current RabbitMQ HTTP API, this commit uses a new HTTP API (let's call it v2) which could be used as a future API for plain HTTP clients. ### HTTP API v1 The current HTTP API (let's call it v1) is **not** used since v1 comes with a couple of weaknesses: 1. Deep level of nesting becomes confusing and difficult to manage. Examples of deep nesting in v1: ``` /api/bindings/vhost/e/source/e/destination/props /api/bindings/vhost/e/exchange/q/queue/props ``` 2. Redundant endpoints returning the same resources v1 has 9 endpoints to list binding(s): ``` /api/exchanges/vhost/name/bindings/source /api/exchanges/vhost/name/bindings/destination /api/queues/vhost/name/bindings /api/bindings /api/bindings/vhost /api/bindings/vhost/e/exchange/q/queue /api/bindings/vhost/e/exchange/q/queue/props /api/bindings/vhost/e/source/e/destination /api/bindings/vhost/e/source/e/destination/props ``` 3. Verbs in path names Path names should be nouns instead. v1 contains verbs: ``` /api/queues/vhost/name/get /api/exchanges/vhost/name/publish ``` ### AMQP Management extension Only few aspects of the AMQP Management extension are used. The central idea of the AMQP management spec is **dynamic discovery** such that broker independent AMQP 1.0 clients can discover objects, types, operations, and HTTP endpoints of specific brokers. In fact, clients are only conformant if: > All request addresses are dynamically discovered starting from the discovery document. > A requesting container MUST NOT use fixed assumptions about the addressing structure of the management API. While this is a nice and powerful idea, no AMQP 1.0 client and no AMQP 1.0 server implement the latest AMQP 1.0 management spec from 2019, partly presumably due to its complexity. Therefore, the idea of such dynamic discovery has failed to be implemented in practice. The AMQP management spec mandates that the management endpoint returns a discovery document containing broker specific collections, types, configuration, and operations including their endpoints. The API endpoints of the AMQP management spec are therefore all designed around dynamic discovery. For example, to create either a queue or an exchange, the client has to ``` POST /$management/entities ``` which shows that the entities collection acts as a generic factory, see section 2.2. The server will then create the resource and reply with a location header containing a URI pointing to the resource. For RabbitMQ, we don’t need such a generic factory to create queues or exchanges. To list bindings for a queue Q1, the spec suggests ``` GET /$management/Queues/Q1/$management/entities ``` which again shows the generic entities endpoint as well as a `$management` endpoint under Q1 to allow a queue to return a discovery document. For RabbitMQ, we don’t need such generic endpoints and discovery documents. Given we aim for our own thin RabbitMQ AMQP 1.0 client wrapper libraries which expose the RabbitMQ model to the developer, we can directly use fixed HTTP endpoint assumptions in our RabbitMQ specific libraries. This is by far simpler than using the dynamic endpoints of the management spec. Simplicity leads to higher adoption and enables more developers to write RabbitMQ AMQP 1.0 client library wrappers. The AMQP Management extension also suffers from deep level of nesting in paths Examples: ``` /$management/Queues/Q1/$management/entities /$management/Queues/Q1/Bindings/Binding1 ``` as well as verbs in path names: Section 7.1.4 suggests using verbs in path names, for example “purge”, due to the dynamic operations discovery document. ### HTTP API v2 This commit introduces a new HTTP API v2 following best practices. It could serve as a future API for plain HTTP clients. This commit and RabbitMQ 4.0 will only implement a minimal set of HTTP API v2 endpoints and only for HTTP over AMQP. In other words, the existing HTTP API v1 Cowboy handlers will continue to be used for all plain HTTP requests in RabbitMQ 4.0 and will remain untouched for RabbitMQ 4.0. Over time, after 4.0 shipped, we could ship a pure HTTP API implementation for HTTP API v2. Hence, the new HTTP API v2 endpoints for HTTP over AMQP should be designed such that they can be re-used in the future for a pure HTTP implementation. The minimal set of endpoints for RabbitMQ 4.0 are: `` GET / PUT / DELETE /vhosts/:vhost/queues/:queue ``` read, create, delete a queue ``` DELETE /vhosts/:vhost/queues/:queue/messages ``` purges a queue ``` GET / DELETE /vhosts/:vhost/bindings/:binding ``` read, delete bindings where `:binding` is a binding ID of the following path segment: ``` src=e1;dstq=q2;key=my-key;args= ``` Binding arguments `args` has an empty value by default, i.e. there are no binding arguments. If the binding includes binding arguments, `args` will be an Erlang portable term hash provided by the server similar to what’s provided in HTTP API v1 today. Alternatively, we could use an arguments scheme of: ``` args=k1,utf8,v1&k2,uint,3 ``` However, such a scheme leads to long URIs when there are many binding arguments. Note that it’s perfectly fine for URI producing applications to include URI reserved characters `=` / `;` / `,` / `$` in a path segment. To create a binding, the client therefore needs to POST to a bindings factory URI: ``` POST /vhosts/:vhost/bindings ``` To list all bindings between a source exchange e1 and destination exchange e2 with binding key k1: ``` GET /vhosts/:vhost/bindings?src=e1&dste=e2&key=k1 ``` This endpoint will be called by the RabbitMQ AMQP 1.0 client library to unbind a binding with non-empty binding arguments to get the binding ID before invoking a ``` DELETE /vhosts/:vhost/bindings/:binding ``` In future, after RabbitMQ 4.0 shipped, new API endpoints could be added. The following is up for discussion and is only meant to show the clean and simple design of HTTP API v2. Bindings endpoint can be queried as follows: to list all bindings for a given source exchange e1: ``` GET /vhosts/:vhost/bindings?src=e1 ``` to list all bindings for a given destination queue q1: ``` GET /vhosts/:vhost/bindings?dstq=q1 ``` to list all bindings between a source exchange e1 and destination queue q1: ``` GET /vhosts/:vhost/bindings?src=e1&dstq=q1 ``` multiple bindings between source exchange e1 and destination queue q1 could be deleted at once as follows: ``` DELETE /vhosts/:vhost/bindings?src=e1&dstq=q1 ``` GET could be supported globally across all vhosts: ``` /exchanges /queues /bindings ``` Publish a message: ``` POST /vhosts/:vhost/queues/:queue/messages ``` Consume or peek a message (depending on query parameters): ``` GET /vhosts/:vhost/queues/:queue/messages ``` Note that the AMQP 1.0 client omits the `/vhost/:vhost` path prefix. Since an AMQP connection belongs to a single vhost, there is no need to additionally include the vhost in every HTTP request. Pros of HTTP API v2: 1. Low level of nesting Queues, exchanges, bindings are top level entities directly under vhosts. Although the HTTP API doesn’t have to reflect how resources are stored in the database, v2 does nicely reflect the Khepri tree structure. 2. Nouns instead of verbs HTTP API v2 is very simple to read and understand as shown by ``` POST /vhosts/:vhost/queues/:queue/messages to post messages, i.e. publish to a queue. GET /vhosts/:vhost/queues/:queue/messages to get messages, i.e. consume or peek from a queue. DELETE /vhosts/:vhost/queues/:queue/messages to delete messages, i.e. purge a queue. ``` A separate new HTTP API v2 allows us to ship only handlers for HTTP over AMQP for RabbitMQ 4.0 and therefore move faster while still keeping the option on the table to re-use the new v2 API for pure HTTP in the future. In contrast, re-using the HTTP API v1 for HTTP over AMQP is possible, but dirty because separate handlers (HTTP over AMQP and pure HTTP) replying differently will be needed for the same v1 endpoints.
2024-02-08 01:26:13 +08:00
TEST_DEPS = rabbitmq_ct_helpers rabbitmq_ct_client_helpers meck proper amqp_client rabbitmq_amqp_client rabbitmq_amqp1_0
Use Lager to log RabbitMQ messages By default, RabbitMQ now logs messages to a single file ($RABBITMQ_LOGS). The $RABBITMQ_SASL_LOGS variable is unused. To configure how and which messages are logged, it's recommended to do it from rabbitmq.config, not from the environment variable. The old `log_levels` parameter is unsupported and categories are replaced by Lager extra sinks. If you had in your rabbitmq.config: {rabbit, [ {log_levels, [{connection, info}]} ]} You can now configure Lager like this: {lager, [ {extra_sinks, [ {rabbit_connection_lager_event, [ {handlers, [{lager_forwarder_backend, [lager_event, info]}]} ]} ]} ]} rabbitmq-build.mk from rabbitmq-common is included in the top-level Makefile. It sets the appropriate compiler options to enable Lager's lager_transform parse_transform module. rabbit_log calls are now converted by this parse_transform to direct calls to lager:log(). To keep backward compatibility with other plugins, the rabbit_log module still implements all the <level>() functions. Compared to the parse_transformed calls, the main difference is the logged message does not carry the file:line metadata. Fixes #94.
2015-12-05 01:07:00 +08:00
Remove looking_glass It has largely been superseded by `perf`. It is no longer generally useful. It can always be added to BUILD_DEPS for the rare cases it is needed, or installed locally and pointed to by setting its path to ERL_LIBS.
2024-06-26 15:56:46 +08:00
PLT_APPS += mnesia runtime_tools
Fix unknown type specs
2020-11-02 18:40:24 +08:00
Switch from Lager to the new Erlang Logger API for logging The configuration remains the same for the end-user. The only exception is the log root directory: it is now set through the `log_root` application env. variable in `rabbit`. People using the Cuttlefish-based configuration file are not affected by this exception. The main change is how the logging facility is configured. It now happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is removed. The supported outputs remain the same: the console, text files, the `amq.rabbitmq.log` exchange and syslog. The message text format slightly changed: the timestamp is more precise (now to the microsecond) and the level can be abbreviated to always be 4-character long to align all messages and improve readability. Here is an example: 2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE == 2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> 2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5 2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com The example above also shows that multiline messages are supported and each line is prepended with the same prefix (the timestamp, the level and the Erlang process PID). JSON is also supported as a message format and now for any outputs. Indeed, it is possible to use it with e.g. syslog or the exchange. Here is an example of a JSON-formatted message sent to syslog: Mar 3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}} For quick testing, the values accepted by the `$RABBITMQ_LOGS` environment variables were extended: * `-` still means stdout * `-stderr` means stderr * `syslog:` means syslog on localhost * `exchange:` means logging to `amq.rabbitmq.log` `$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in addition to the existing `+color` one). With that modifier, messages are formatted as JSON intead of plain text. The `rabbitmqctl rotate_logs` command is deprecated. The reason is Logger does not expose a function to force log rotation. However, it will detect when a file was rotated by an external tool. From a developer point of view, the old `rabbit_log*` API remains supported, though it is now deprecated. It is implemented as regular modules: there is no `parse_transform` involved anymore. In the code, it is recommended to use the new Logger macros. For instance, `?LOG_INFO(Format, Args)`. If possible, messages should be augmented with some metadata. For instance (note the map after the message): ?LOG_NOTICE("Logging: switching to configured handler(s); following " "messages may not be visible in this log output", #{domain => ?RMQLOG_DOMAIN_PRELAUNCH}), Domains in Erlang Logger parlance are the way to categorize messages. Some predefined domains, matching previous categories, are currently defined in `rabbit_common/include/logging.hrl` or headers in the relevant plugins for plugin-specific categories. At this point, very few messages have been converted from the old `rabbit_log*` API to the new macros. It can be done gradually when working on a particular module or logging. The Erlang builtin console/file handler, `logger_std_h`, has been forked because it lacks date-based file rotation. The configuration of date-based rotation is identical to Lager. Once the dust has settled for this feature, the goal is to submit it upstream for inclusion in Erlang. The forked module is calld `rabbit_logger_std_h` and is based `logger_std_h` in Erlang 23.0.
2021-01-13 00:55:27 +08:00
dep_syslog = git https://github.com/schlagert/syslog 4.0.0
Ra 2.7.0 This includes a new ra:key_metrics/1 API that is more available than parsing the output of sys:get_status/1. the rabbit_quorum_queue:status/1 function has been ported to use this API instead as well as now inludes a few new fields.
2023-09-07 00:09:12 +08:00
Use EXTRA_SOURCES for `*_usage.erl`
2015-08-25 22:53:21 +08:00
define usage_xml_to_erl
$(subst __,_,$(patsubst $(DOCS_DIR)/rabbitmq%.1.xml, src/rabbit_%_usage.erl, $(subst -,_,$(1))))
endef
DOCS_DIR = docs
docs: Convert manpages from DocBook to mdoc(7) Writing those manpages directly in the target markup language allows to use its full expressiveness, unlike the generated output from DocBook. To create their HTML versions, we use mandoc(1), plus a small awk script to adapt the generated HTML to our needs. I tried groff(1) and man2html(1) but the former transforms nearly everything to simple <p> and the later tries to reproduce the rendering of a manpage in a terminal. Manpages in section 1 are moved to section 8 because they are commands to manage a system service, not general purpose commands. This commits requires a change in the website to integrate the new generated HTML manpages. [#143563295]
2017-04-25 16:50:00 +08:00
MANPAGES = $(wildcard $(DOCS_DIR)/*.[0-9])
WEB_MANPAGES = $(patsubst %,%.html,$(MANPAGES))
Generate manpages in markdown in addition to HTML [Why] This file format can be used directly by Docusaurus. The HTML version could work too, but generated files require patching to work correctly.
2023-12-18 19:16:04 +08:00
MD_MANPAGES = $(patsubst %,%.md,$(MANPAGES))
In development environment, check for ENABLED_PLUGINS_DIR for rabbitmqctl too
2016-11-07 20:03:18 +08:00
make: Move rabbitmq-early-test.mk to rabbitmq-early-plugin.mk No real need to have two files, especially since it contains only a few variable definitions. Plan is to only keep separate files for larger features such as dist or run.
2024-07-25 18:15:50 +08:00
DEP_EARLY_PLUGINS = rabbit_common/mk/rabbitmq-early-plugin.mk
make: Restrict Erlang.mk plugin inclusion This has no real impact on performance[1] but should make it clear which application can run the broker and/or publish to Hex.pm. In particular, applications that we can't run the broker from will now give up early if we try to. Note that while the broker can't normally run from the amqp_client application's directory, it can run from tests and some of the tests start the broker. [1] on my machine
2024-07-25 20:15:31 +08:00
DEP_PLUGINS = rabbit_common/mk/rabbitmq-plugin.mk
Update erlang.mk and disable `edoc` plugin
2015-08-26 00:34:52 +08:00
Remove duplicate rabbitmq-components.mk and erlang.mk files Also adjust the references in rabbitmq-components.mk to account for post monorepo locations
2021-03-22 17:38:17 +08:00
include ../../rabbitmq-components.mk
include ../../erlang.mk
complain about missing env vars in the right place especially before trying to use them
2010-06-09 19:41:46 +08:00
Add target to run bats tests to Makefile The `bats` target will clone and install the bats command in a manner similar to `kerl` or `hex.pm`, i.e. to ERLANG_MK_TMP [#150452491] Signed-off-by: Luke Bakken <lbakken@pivotal.io>
2017-08-23 01:18:05 +08:00
ifeq ($(strip $(BATS)),)
BATS := $(ERLANG_MK_TMP)/bats/bin/bats
endif
BATS_GIT ?= https://github.com/sstephenson/bats
BATS_COMMIT ?= v0.4.0
$(BATS):
$(verbose) mkdir -p $(ERLANG_MK_TMP)
$(gen_verbose) git clone --depth 1 --branch=$(BATS_COMMIT) $(BATS_GIT) $(ERLANG_MK_TMP)/bats
.PHONY: bats
bats: $(BATS)
$(verbose) $(BATS) $(TEST_DIR)
tests:: bats
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
SLOW_CT_SUITES := backing_queue \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
channel_interceptor \
cluster \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
cluster_rename \
clustering_management \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
config_schema \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
confirms_rejects \
consumer_timeout \
crashing_queues \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
dynamic_ha \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
dynamic_qq \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
eager_sync \
Makefile: Add `feature_flags_SUITE` to the slow testsuites
2019-03-01 22:59:02 +08:00
feature_flags \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
health_check \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
many_node_ha \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
metrics \
msg_store \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
partitions \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
per_user_connection_tracking \
per_vhost_connection_limit \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
per_vhost_connection_limit_partitions \
Makefile: Add verbose testsuites to slow testsuites This is only so we don't reach the log size limit on Travis CI.
2019-01-08 22:01:18 +08:00
per_vhost_msg_store \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
per_vhost_queue_limit \
policy \
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
priority_queue \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
priority_queue_recovery \
publisher_confirms_parallel \
queue_parallel \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
quorum_queue \
rabbit_core_metrics_gc \
Makefile: Add verbose testsuites to slow testsuites This is only so we don't reach the log size limit on Travis CI.
2019-01-08 22:01:18 +08:00
rabbit_fifo_prop \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
rabbitmq_queues_cli_integration \
rabbitmqctl_integration \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
simple_ha \
sync_detection \
Makefile: Expand the list of slow testsuites Those are all the testsuites which take more than 5 minutes in Concourse.
2020-01-22 22:50:24 +08:00
unit_inbroker_non_parallel \
Makefile: Update the list of slow testsuites This should help reduce the load in Travis CI or when doing quick testing locally.
2019-01-08 21:19:18 +08:00
unit_inbroker_parallel \
vhost
test: A few minor style fixes References #1136 [#136613953]
2017-03-09 16:55:15 +08:00
FAST_CT_SUITES := $(filter-out $(sort $(SLOW_CT_SUITES)),$(CT_SUITES))
Use fast and slow targets from new Makefile plugin References #1136 [#136613953]
2017-03-07 19:41:32 +08:00
make: Remove rabbitmq-tests.mk Everything in this file seems to be dead code except ct-slow/ct-fast, which have been replaced by their equivalent in the rabbit Makefile.
2024-07-05 19:23:05 +08:00
ct-fast:
$(MAKE) ct CT_SUITES='$(FAST_CT_SUITES)'
ct-slow:
$(MAKE) ct CT_SUITES='$(SLOW_CT_SUITES)'
Use fast and slow targets from new Makefile plugin References #1136 [#136613953]
2017-03-07 19:41:32 +08:00
rabbit tests: Redirect logs to ct always Doing it on a per test suite basis leads to issues if multiple suites try to configure it, and there's no cleanup performed anyway.
2024-08-23 23:01:47 +08:00
CT_OPTS += -ct_hooks rabbit_ct_hook []
make & make CI: Small cleanups
2024-09-25 18:24:08 +08:00
# Parallel CT.
#
# @todo We must ensure that the CT_OPTS also apply to ct-master
# @todo We should probably refactor ct_master.erl to have node init in a separate .erl
make: Initial work on using ct_master to run tests Because `ct_master` is yet another Erlang node, and it is used to run multiple CT nodes, meaning it is in a cluster of CT nodes, the tests that change the net_ticktime could not work properly anymore. This is because net_ticktime must be the same value across the cluster. The same value had to be set for all tests in order to solve this. This is why it was changed to 5s across the board. The lower net_ticktime was used in most places to speed up tests that must deal with cluster failures, so that value is good enough for these cases. One test in amqp_client was using the net_ticktime to test the behavior of the direct connection timeout with varying net_ticktime configurations. The test now mocks the `net_kernel:get_net_ticktime()` function to achieve the same result.
2024-08-02 23:29:45 +08:00
define ct_master.erl
StartOpts = #{
host => "localhost",
connection => standard_io,
args => ["-hidden"]
},
{ok, Pid1, _} = peer:start(StartOpts#{name => "rabbit_shard1"}),
{ok, Pid2, _} = peer:start(StartOpts#{name => "rabbit_shard2"}),
{ok, Pid3, _} = peer:start(StartOpts#{name => "rabbit_shard3"}),
{ok, Pid4, _} = peer:start(StartOpts#{name => "rabbit_shard4"}),
peer:call(Pid1, net_kernel, set_net_ticktime, [5]),
peer:call(Pid2, net_kernel, set_net_ticktime, [5]),
peer:call(Pid3, net_kernel, set_net_ticktime, [5]),
peer:call(Pid4, net_kernel, set_net_ticktime, [5]),
peer:call(Pid1, persistent_term, put, [rabbit_ct_tcp_port_base, 23000]),
peer:call(Pid2, persistent_term, put, [rabbit_ct_tcp_port_base, 25000]),
peer:call(Pid3, persistent_term, put, [rabbit_ct_tcp_port_base, 27000]),
peer:call(Pid4, persistent_term, put, [rabbit_ct_tcp_port_base, 29000]),
Make CI: Have ct_master return the test results Instead of having a CT hook just to know whether our tests failed.
2024-10-15 18:19:08 +08:00
[{[_], {ok, Results}}] = ct_master_fork:run("$1"),
make: Initial work on using ct_master to run tests Because `ct_master` is yet another Erlang node, and it is used to run multiple CT nodes, meaning it is in a cluster of CT nodes, the tests that change the net_ticktime could not work properly anymore. This is because net_ticktime must be the same value across the cluster. The same value had to be set for all tests in order to solve this. This is why it was changed to 5s across the board. The lower net_ticktime was used in most places to speed up tests that must deal with cluster failures, so that value is good enough for these cases. One test in amqp_client was using the net_ticktime to test the behavior of the direct connection timeout with varying net_ticktime configurations. The test now mocks the `net_kernel:get_net_ticktime()` function to achieve the same result.
2024-08-02 23:29:45 +08:00
peer:stop(Pid4),
peer:stop(Pid3),
peer:stop(Pid2),
peer:stop(Pid1),
Make CI: Have ct_master return the test results Instead of having a CT hook just to know whether our tests failed.
2024-10-15 18:19:08 +08:00
lists:foldl(fun
({_, {_, 0, {_, 0}}}, Err) -> Err + 1;
(What, Peer) -> halt(Peer)
end, 1, Results),
halt(0)
make: Initial work on using ct_master to run tests Because `ct_master` is yet another Erlang node, and it is used to run multiple CT nodes, meaning it is in a cluster of CT nodes, the tests that change the net_ticktime could not work properly anymore. This is because net_ticktime must be the same value across the cluster. The same value had to be set for all tests in order to solve this. This is why it was changed to 5s across the board. The lower net_ticktime was used in most places to speed up tests that must deal with cluster failures, so that value is good enough for these cases. One test in amqp_client was using the net_ticktime to test the behavior of the direct connection timeout with varying net_ticktime configurations. The test now mocks the `net_kernel:get_net_ticktime()` function to achieve the same result.
2024-08-02 23:29:45 +08:00
endef
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
PARALLEL_CT_SET_1_A = amqp_client unit_cluster_formation_locking_mocks unit_cluster_formation_sort_nodes unit_collections unit_config_value_encryption unit_connection_tracking
Support AMQP filter expressions (#12415) * Support AMQP filter expressions ## What? This PR implements the following property filter expressions for AMQP clients consuming from streams as defined in [AMQP Filter Expressions Version 1.0 Working Draft 09](https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=66227): * properties filters [section 4.2.4] * application-properties filters [section 4.2.5] String prefix and suffix matching is also supported. This PR also fixes a bug where RabbitMQ would accept wrong filters. Specifically, prior to this PR the values of the filter-set's map were allowed to be symbols. However, "every value MUST be either null or of a described type which provides the archetype filter." ## Why? This feature adds the ability to RabbitMQ to have multiple concurrent clients each consuming only a subset of messages while maintaining message order. This feature also reduces network traffic between RabbitMQ and clients by only dispatching those messages that the clients are actually interested in. Note that AMQP filter expressions are more fine grained than the [bloom filter based stream filtering](https://www.rabbitmq.com/blog/2023/10/16/stream-filtering) because * they do not suffer false positives * the unit of filtering is per-message instead of per-chunk * matching can be performed on **multiple** values in the properties and application-properties sections * prefix and suffix matching on the actual values is supported. Both, AMQP filter expressions and bloom filters can be used together. ## How? If a filter isn't valid, RabbitMQ ignores the filter. RabbitMQ only replies with filters it actually supports and validated successfully to comply with: "The receiving endpoint sets its desired filter, the sending endpoint [RabbitMQ] sets the filter actually in place (including any filters defaulted at the node)." * Delete streams test case The test suite constructed a wrong filter-set. Specifically the value of the filter-set didn't use a described type as mandated by the spec. Using https://azure.github.io/amqpnetlite/api/Amqp.Types.DescribedValue.html throws errors that the descriptor can't be encoded. Given that this code path is already tests via the amqp_filtex_SUITE, this F# test gets therefore deleted. * Re-introduce the AMQP filter-set bug Since clients might rely on the wrong filter-set value type, we support the bug behind a deprecated feature flag and gradually remove support this bug. * Revert "Delete streams test case" This reverts commit c95cfeaef74160894050ae51a563bf839384d2d7.
2024-10-07 23:12:26 +08:00
PARALLEL_CT_SET_1_B = amqp_address amqp_auth amqp_credit_api_v2 amqp_filtex amqp_system signal_handling single_active_consumer unit_access_control_authn_authz_context_propagation unit_access_control_credential_validation unit_amqp091_content_framing unit_amqp091_server_properties unit_app_management
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
PARALLEL_CT_SET_1_C = amqp_proxy_protocol amqpl_consumer_ack amqpl_direct_reply_to backing_queue bindings rabbit_db_maintenance rabbit_db_msup rabbit_db_policy rabbit_db_queue rabbit_db_topic_exchange rabbit_direct_reply_to_prop cluster_limit cluster_minority term_to_binary_compat_prop topic_permission transactions unicode unit_access_control
PARALLEL_CT_SET_1_D = amqqueue_backward_compatibility channel_interceptor channel_operation_timeout classic_queue classic_queue_prop config_schema peer_discovery_dns peer_discovery_tmp_hidden_node per_node_limit per_user_connection_channel_limit
PARALLEL_CT_SET_2_A = cluster confirms_rejects consumer_timeout rabbit_access_control rabbit_confirms rabbit_core_metrics_gc rabbit_cuttlefish rabbit_db_binding rabbit_db_exchange
PARALLEL_CT_SET_2_B = clustering_recovery crashing_queues deprecated_features direct_exchange_routing_v2 disconnect_detected_during_alarm exchanges unit_gen_server2
PARALLEL_CT_SET_2_C = disk_monitor dynamic_qq unit_disk_monitor unit_file_handle_cache unit_log_management unit_operator_policy
PARALLEL_CT_SET_2_D = queue_length_limits queue_parallel quorum_queue_member_reconciliation rabbit_fifo rabbit_fifo_dlx rabbit_stream_coordinator
PARALLEL_CT_SET_3_A = definition_import per_user_connection_channel_limit_partitions per_vhost_connection_limit_partitions policy priority_queue_recovery rabbit_fifo_prop rabbit_fifo_v0 rabbit_stream_sac_coordinator unit_credit_flow unit_queue_consumers unit_queue_location unit_quorum_queue
PARALLEL_CT_SET_3_B = cluster_upgrade list_consumers_sanity_check list_queues_online_and_offline logging lqueue maintenance_mode rabbit_fifo_q
PARALLEL_CT_SET_3_C = cli_forget_cluster_node feature_flags_v2 mc_unit message_containers_deaths_v2 message_size_limit metadata_store_migration
PARALLEL_CT_SET_3_D = metadata_store_phase1 metrics mirrored_supervisor msg_store peer_discovery_classic_config proxy_protocol runtime_parameters unit_stats_and_metrics unit_supervisor2 unit_vm_memory_monitor
PARALLEL_CT_SET_4_A = clustering_events rabbit_local_random_exchange rabbit_message_interceptor rabbitmq_4_0_deprecations unit_pg_local unit_plugin_directories unit_plugin_versioning unit_policy_validators unit_priority_queue
PARALLEL_CT_SET_4_B = per_user_connection_tracking per_vhost_connection_limit rabbit_fifo_dlx_integration rabbit_fifo_int
make: Add new CT suites and clarify check on CT_SUITES
2024-09-25 19:24:36 +08:00
PARALLEL_CT_SET_4_C = msg_size_metrics unit_msg_size_metrics per_vhost_msg_store per_vhost_queue_limit priority_queue upgrade_preparation vhost
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
PARALLEL_CT_SET_4_D = per_user_connection_channel_tracking product_info publisher_confirms_parallel queue_type rabbitmq_queues_cli_integration rabbitmqctl_integration rabbitmqctl_shutdown routing
PARALLEL_CT_SET_1 = $(sort $(PARALLEL_CT_SET_1_A) $(PARALLEL_CT_SET_1_B) $(PARALLEL_CT_SET_1_C) $(PARALLEL_CT_SET_1_D))
PARALLEL_CT_SET_2 = $(sort $(PARALLEL_CT_SET_2_A) $(PARALLEL_CT_SET_2_B) $(PARALLEL_CT_SET_2_C) $(PARALLEL_CT_SET_2_D))
PARALLEL_CT_SET_3 = $(sort $(PARALLEL_CT_SET_3_A) $(PARALLEL_CT_SET_3_B) $(PARALLEL_CT_SET_3_C) $(PARALLEL_CT_SET_3_D))
PARALLEL_CT_SET_4 = $(sort $(PARALLEL_CT_SET_4_A) $(PARALLEL_CT_SET_4_B) $(PARALLEL_CT_SET_4_C) $(PARALLEL_CT_SET_4_D))
SEQUENTIAL_CT_SUITES = clustering_management dead_lettering feature_flags metadata_store_clustering quorum_queue rabbit_stream_queue
PARALLEL_CT_SUITES = $(PARALLEL_CT_SET_1) $(PARALLEL_CT_SET_2) $(PARALLEL_CT_SET_3) $(PARALLEL_CT_SET_4)
ifneq ($(filter-out $(SEQUENTIAL_CT_SUITES) $(PARALLEL_CT_SUITES),$(CT_SUITES)),)
make: Add new CT suites and clarify check on CT_SUITES
2024-09-25 19:24:36 +08:00
$(error Some test suites in CT_SUITES but not configured for CI: $(filter-out $(SEQUENTIAL_CT_SUITES) $(PARALLEL_CT_SUITES),$(CT_SUITES)))
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
endif
define tpl_parallel_ct_test_spec
make: Set CT_LOGS_DIR to top-level logs/ directory All CT logs will now be under <toplevel>/logs. An improved test workflow would be to always keep the logs/all_runs.html page open in the browser and refresh it whenever tests are run in any of the rabbit applications.
2024-09-24 19:40:54 +08:00
{logdir, "$(CT_LOGS_DIR)"}.
{logdir, master, "$(CT_LOGS_DIR)"}.
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
{create_priv_dir, all_nodes, auto_per_run}.
Make CI: Have ct_master return the test results Instead of having a CT hook just to know whether our tests failed.
2024-10-15 18:19:08 +08:00
{auto_compile, false}.
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
{node, shard1, 'rabbit_shard1@localhost'}.
{node, shard2, 'rabbit_shard2@localhost'}.
{node, shard3, 'rabbit_shard3@localhost'}.
{node, shard4, 'rabbit_shard4@localhost'}.
{define, 'Set1', [$(call comma_list,$(addsuffix _SUITE,$1))]}.
{define, 'Set2', [$(call comma_list,$(addsuffix _SUITE,$2))]}.
{define, 'Set3', [$(call comma_list,$(addsuffix _SUITE,$3))]}.
{define, 'Set4', [$(call comma_list,$(addsuffix _SUITE,$4))]}.
{suites, shard1, "test/", 'Set1'}.
{suites, shard2, "test/", 'Set2'}.
{suites, shard3, "test/", 'Set3'}.
{suites, shard4, "test/", 'Set4'}.
endef
define parallel_ct_set_target
tpl_parallel_ct_test_spec_set_$1 = $$(call tpl_parallel_ct_test_spec,$(PARALLEL_CT_SET_$(1)_A),$(PARALLEL_CT_SET_$(1)_B),$(PARALLEL_CT_SET_$(1)_C),$(PARALLEL_CT_SET_$(1)_D))
parallel-ct-set-$(1): test-build
$(verbose) mkdir -p $(CT_LOGS_DIR)
$(verbose) $$(call core_render,tpl_parallel_ct_test_spec_set_$(1),ct.set-$(1).spec)
$$(call erlang,$$(call ct_master.erl,ct.set-$(1).spec),-sname parallel_ct_$(PROJECT)@localhost -hidden -kernel net_ticktime 5)
endef
$(foreach set,1 2 3 4,$(eval $(call parallel_ct_set_target,$(set))))
make & make CI: Small cleanups
2024-09-25 18:24:08 +08:00
# @todo Generate ct.test.spec from Makefile variables instead of hardcoded for ct-master
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
parallel-ct: test-build
make: Initial work on using ct_master to run tests Because `ct_master` is yet another Erlang node, and it is used to run multiple CT nodes, meaning it is in a cluster of CT nodes, the tests that change the net_ticktime could not work properly anymore. This is because net_ticktime must be the same value across the cluster. The same value had to be set for all tests in order to solve this. This is why it was changed to 5s across the board. The lower net_ticktime was used in most places to speed up tests that must deal with cluster failures, so that value is good enough for these cases. One test in amqp_client was using the net_ticktime to test the behavior of the direct connection timeout with varying net_ticktime configurations. The test now mocks the `net_kernel:get_net_ticktime()` function to achieve the same result.
2024-08-02 23:29:45 +08:00
$(verbose) mkdir -p $(CT_LOGS_DIR)
Use Make in CI This is a proof of concept that mostly works but is missing some tests, such as rabbitmq_mqtt or rabbitmq_cli. It also doesn't apply to mixed version testing yet.
2024-09-02 20:55:02 +08:00
$(call erlang,$(call ct_master.erl,ct.test.spec),-sname parallel_ct_$(PROJECT)@localhost -hidden -kernel net_ticktime 5)
Use fast and slow targets from new Makefile plugin References #1136 [#136613953]
2017-03-07 19:41:32 +08:00
Use rabbit_common .ez archive instead of deps/
2015-08-27 23:12:32 +08:00
# --------------------------------------------------------------------
# Compilation.
# --------------------------------------------------------------------
`rabbitmq_common` is now `rabbit_common`
2015-08-25 23:46:35 +08:00
RMQ_ERLC_OPTS += -I $(DEPS_DIR)/rabbit_common/include
Deprecated features: New module to manage deprecated features (!) This introduces a way to declare deprecated features in the code, not only in our communication. The new module allows to disallow the use of a deprecated feature and/or warn the user when he relies on such a feature. [Why] Currently, we only tell people about deprecated features through blog posts and the mailing-list. This might be insufficiant for our users that a feature they use will be removed in a future version: * They may not read our blog or mailing-list * They may not understand that they use such a deprecated feature * They might wait for the big removal before they plan testing * They might not take it seriously enough The idea behind this patch is to increase the chance that users notice that they are using something which is about to be dropped from RabbitMQ. Anopther benefit is that they should be able to test how RabbitMQ will behave in the future before the actual removal. This should allow them to test and plan changes. [How] When a feature is deprecated in other large projects (such as FreeBSD where I took the idea from), it goes through a lifecycle: 1. The feature is still available, but users get a warning somehow when they use it. They can disable it to test. 2. The feature is still available, but disabled out-of-the-box. Users can re-enable it (and get a warning). 3. The feature is disconnected from the build. Therefore, the code behind it is still there, but users have to recompile the thing to be able to use it. 4. The feature is removed from the source code. Users have to adapt or they can't upgrade anymore. The solution in this patch offers the same lifecycle. A deprecated feature will be in one of these deprecation phases: 1. `permitted_by_default`: The feature is available. Users get a warning if they use it. They can disable it from the configuration. 2. `denied_by_default`: The feature is available but disabled by default. Users get an error if they use it and RabbitMQ behaves like the feature is removed. They can re-enable is from the configuration and get a warning. 3. `disconnected`: The feature is present in the source code, but is disabled and can't be re-enabled without recompiling RabbitMQ. Users get the same behavior as if the code was removed. 4. `removed`: The feature's code is gone. The whole thing is based on the feature flags subsystem, but it has the following differences with other feature flags: * The semantic is reversed: the feature flag behind a deprecated feature is disabled when the deprecated feature is permitted, or enabled when the deprecated feature is denied. * The feature flag behind a deprecated feature is enabled out-of-the-box (meaning the deprecated feature is denied): * if the deprecation phase is `permitted_by_default` and the configuration denies the deprecated feature * if the deprecation phase is `denied_by_default` and the configuration doesn't permit the deprecated feature * if the deprecation phase is `disconnected` or `removed` * Feature flags behind deprecated feature don't appear in feature flags listings. Otherwise, deprecated features' feature flags are managed like other feature flags, in particular inside clusters. To declare a deprecated feature: -rabbit_deprecated_feature( {my_deprecated_feature, #{deprecation_phase => permitted_by_default, msgs => #{when_permitted => "This feature will be removed in RabbitMQ X.0"}, }}). Then, to check the state of a deprecated feature in the code: case rabbit_deprecated_features:is_permitted(my_deprecated_feature) of true -> %% The deprecated feature is still permitted. ok; false -> %% The deprecated feature is gone or should be considered %% unavailable. error end. Warnings and errors are logged automatically. A message is generated automatically, but it is possible to define a message in the deprecated feature flag declaration like in the example above. Here is an example of a logged warning that was generated automatically: Feature `my_deprecated_feature` is deprecated. By default, this feature can still be used for now. Its use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined. To continue using this feature when it is not permitted by default, set the following parameter in your configuration: "deprecated_features.permit.my_deprecated_feature = true" To test RabbitMQ as if the feature was removed, set this in your configuration: "deprecated_features.permit.my_deprecated_feature = false" To override the default state of `permitted_by_default` and `denied_by_default` deprecation phases, users can set the following configuration: # In rabbitmq.conf: deprecated_features.permit.my_deprecated_feature = true # or false The actual behavior protected by a deprecated feature check is out of scope for this subsystem. It is the repsonsibility of each deprecated feature code to determine what to do when the deprecated feature is denied. V1: Deprecated feature states are initially computed during the initialization of the registry, based on their deprecation phase and possibly the configuration. They don't go through the `enable/1` code at all. V2: Manage deprecated feature states as any other non-required feature flags. This allows to execute an `is_feature_used()` callback to determine if a deprecated feature can be denied. This also allows to prevent the RabbitMQ node from starting if it continues to use a deprecated feature. V3: Manage deprecated feature states from the registry initialization again. This is required because we need to know very early if some of them are denied, so that an upgrade to a version of RabbitMQ where a deprecated feature is disconnected or removed can be performed. To still prevent the start of a RabbitMQ node when a denied deprecated feature is actively used, we run the `is_feature_used()` callback of all denied deprecated features as part of the `sync_cluster()` task. This task is executed as part of a feature flag refresh executed when RabbitMQ starts or when plugins are enabled. So even though a deprecated feature is marked as denied in the registry early in the boot process, we will still abort the start of a RabbitMQ node if the feature is used. V4: Support context-dependent warnings. It is now possible to set a specific message when deprecated feature is permitted, when it is denied and when it is removed. Generic per-context messages are still generated. V5: Improve default warning messages, thanks to @pstack2021. V6: Rename the configuration variable from `permit_deprecated_features.*` to `deprecated_features.permit.*`. As @michaelklishin said, we tend to use shorter top-level names.
2023-02-23 00:26:52 +08:00
EDOC_OPTS += {preprocess,true},{includes,["."]}
Correct logic so that a 'make' with no explicit goals still behaves correctly
2010-03-23 19:23:55 +08:00
Makefile: Use `erlang.mk`
2015-08-10 17:50:41 +08:00
ifdef INSTRUMENT_FOR_QC
Makefile: Set TEST_ERLC_OPTS
2015-08-14 18:21:42 +08:00
RMQ_ERLC_OPTS += -DINSTR_MOD=gm_qc
Doc: Fix all errors reported by edoc With this, `gmake edoc` returns successfully and HTML docs are correctly generated.
2021-04-09 19:13:47 +08:00
EDOC_OPTS += ,{macros,[{'INSTR_MOD',gm_qc}]}
Correct logic so that a 'make' with no explicit goals still behaves correctly
2010-03-23 19:23:55 +08:00
else
Makefile: Set TEST_ERLC_OPTS
2015-08-14 18:21:42 +08:00
RMQ_ERLC_OPTS += -DINSTR_MOD=gm
Doc: Fix all errors reported by edoc With this, `gmake edoc` returns successfully and HTML docs are correctly generated.
2021-04-09 19:13:47 +08:00
EDOC_OPTS += ,{macros,[{'INSTR_MOD',gm}]}
Correct logic so that a 'make' with no explicit goals still behaves correctly
2010-03-23 19:23:55 +08:00
endif
Makefile: Use `erlang.mk`
2015-08-10 17:50:41 +08:00
ifdef CREDIT_FLOW_TRACING
Makefile: Set TEST_ERLC_OPTS
2015-08-14 18:21:42 +08:00
RMQ_ERLC_OPTS += -DCREDIT_FLOW_TRACING=true
Makefile: Use `erlang.mk`
2015-08-10 17:50:41 +08:00
endif
Rename TRACE_SUP2 to TRACE_SUPERVISOR2 and add to Makefile
2022-08-29 05:41:54 +08:00
ifdef TRACE_SUPERVISOR2
RMQ_ERLC_OPTS += -DTRACE_SUPERVISOR2=true
endif
Makefile: Connect manpages to the build
2015-08-14 00:48:53 +08:00
# --------------------------------------------------------------------
Makefile: Sort content a bit
2015-08-14 01:01:27 +08:00
# Documentation.
Makefile: Connect manpages to the build
2015-08-14 00:48:53 +08:00
# --------------------------------------------------------------------
Do not compress manpages, the packaging system handles that
2015-10-07 00:48:45 +08:00
.PHONY: manpages web-manpages distclean-manpages
docs:: manpages web-manpages
manpages: $(MANPAGES)
@:
Generate manpages in markdown in addition to HTML [Why] This file format can be used directly by Docusaurus. The HTML version could work too, but generated files require patching to work correctly.
2023-12-18 19:16:04 +08:00
web-manpages: $(WEB_MANPAGES) $(MD_MANPAGES)
Do not compress manpages, the packaging system handles that
2015-10-07 00:48:45 +08:00
@:
Makefile: Connect manpages to the build
2015-08-14 00:48:53 +08:00
docs: Convert manpages from DocBook to mdoc(7) Writing those manpages directly in the target markup language allows to use its full expressiveness, unlike the generated output from DocBook. To create their HTML versions, we use mandoc(1), plus a small awk script to adapt the generated HTML to our needs. I tried groff(1) and man2html(1) but the former transforms nearly everything to simple <p> and the later tries to reproduce the rendering of a manpage in a terminal. Manpages in section 1 are moved to section 8 because they are commands to manage a system service, not general purpose commands. This commits requires a change in the website to integrate the new generated HTML manpages. [#143563295]
2017-04-25 16:50:00 +08:00
# We use mandoc(1) to convert manpages to HTML plus an awk script which
# does:
# 1. remove tables at the top and the bottom (they recall the
# manpage name, section and date)
# 2. "downgrade" headers by one level (eg. h1 -> h2)
# 3. annotate .Dl lines with more CSS classes
%.html: %
$(gen_verbose) mandoc -T html -O 'fragment,man=%N.%S.html' "$<" | \
awk '\
/^<table class="head">$$/ { remove_table=1; next; } \
/^<table class="foot">$$/ { remove_table=1; next; } \
/^<\/table>$$/ { if (remove_table) { remove_table=0; next; } } \
{ if (!remove_table) { \
line=$$0; \
gsub(/<h2/, "<h3", line); \
gsub(/<\/h2>/, "</h3>", line); \
gsub(/<h1/, "<h2", line); \
gsub(/<\/h1>/, "</h2>", line); \
Adapt man page styling for recent website CSS changes
2019-01-31 10:08:57 +08:00
gsub(/class="D1"/, "class=\"D1 lang-bash\"", line); \
gsub(/class="Bd Bd-indent"/, "class=\"Bd Bd-indent lang-bash\"", line); \
Modify html docs to not use smart quotes Fixes rabbitmqr/rabbitmq-website#502
2019-09-05 06:53:01 +08:00
gsub(/&#[xX]201[cCdD];/, "\\&quot;", line); \
rabbit/Makefile: Delete `.html` from local URL in manpages
2024-09-23 19:24:54 +08:00
gsub(/\.html/, "", line); \
docs: Convert manpages from DocBook to mdoc(7) Writing those manpages directly in the target markup language allows to use its full expressiveness, unlike the generated output from DocBook. To create their HTML versions, we use mandoc(1), plus a small awk script to adapt the generated HTML to our needs. I tried groff(1) and man2html(1) but the former transforms nearly everything to simple <p> and the later tries to reproduce the rendering of a manpage in a terminal. Manpages in section 1 are moved to section 8 because they are commands to manage a system service, not general purpose commands. This commits requires a change in the website to integrate the new generated HTML manpages. [#143563295]
2017-04-25 16:50:00 +08:00
print line; \
} } \
' > "$@"
Generate manpages in markdown in addition to HTML [Why] This file format can be used directly by Docusaurus. The HTML version could work too, but generated files require patching to work correctly.
2023-12-18 19:16:04 +08:00
%.md: %
$(gen_verbose) mandoc -T markdown -O 'fragment,man=%N.%S.md' "$<" | \
sed -E -e 's/\{/\&lcub;/g' \
> "$@"
Makefile: Connect manpages to the build
2015-08-14 00:48:53 +08:00
distclean:: distclean-manpages
distclean-manpages::
Generate manpages in markdown in addition to HTML [Why] This file format can be used directly by Docusaurus. The HTML version could work too, but generated files require patching to work correctly.
2023-12-18 19:16:04 +08:00
$(gen_verbose) rm -f $(WEB_MANPAGES) $(MD_MANPAGES)