root
/
rabbitmq-server
mirror of https://github.com/rabbitmq/rabbitmq-server.git
1
0
Fork 0
Code Issues Actions 1 Packages Projects Releases Wiki Activity
rabbitmq-server/deps/rabbit/test/queue_parallel_SUITE.erl

769 lines
32 KiB
Erlang
Raw Normal View History

Switch to Mozilla Public License 2.0 (MPL 2.0)
2020-07-10 21:31:17 +08:00
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
%%
Bump (c) year
2021-01-22 14:00:14 +08:00
%% Copyright (c) 2011-2021 VMware, Inc. or its affiliates. All rights reserved.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
%%
%%
-module(queue_parallel_SUITE).
-include_lib("common_test/include/ct.hrl").
-include_lib("kernel/include/file.hrl").
-include_lib("amqp_client/include/amqp_client.hrl").
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).
-define(TIMEOUT, 30000).
-import(quorum_queue_utils, [wait_for_messages/2]).
all() ->
[
{group, parallel_tests}
].
groups() ->
AllTests = [publish,
consume,
consume_first_empty,
consume_from_empty_queue,
consume_and_autoack,
subscribe,
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe_consumers,
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_with_autoack,
consume_and_ack,
consume_and_multiple_ack,
subscribe_and_ack,
subscribe_and_multiple_ack,
subscribe_and_requeue_multiple_nack,
subscribe_and_nack,
subscribe_and_requeue_nack,
subscribe_and_multiple_nack,
consume_and_requeue_nack,
consume_and_nack,
consume_and_requeue_multiple_nack,
consume_and_multiple_nack,
basic_cancel,
purge,
basic_recover,
delete_immediately_by_resource
],
[
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
{parallel_tests, [], [
Compaction: more logging, a test
2019-05-10 05:00:47 +08:00
{classic_queue, [parallel], AllTests ++ [delete_immediately_by_pid_succeeds,
trigger_message_store_compaction]},
{mirrored_queue, [parallel], AllTests ++ [delete_immediately_by_pid_succeeds,
trigger_message_store_compaction]},
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
{quorum_queue, [parallel], AllTests ++ [
delete_immediately_by_pid_fails,
extra_bcc_option
]},
Use in memory limit for quorum queue tests All should work as normal, only memory usage it's affected [#164735591]
2019-03-27 18:37:58 +08:00
{quorum_queue_in_memory_limit, [parallel], AllTests ++ [delete_immediately_by_pid_fails]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{quorum_queue_in_memory_bytes, [parallel], AllTests ++ [delete_immediately_by_pid_fails]},
{stream_queue, [parallel], [publish,
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
subscribe,
extra_bcc_option]}
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
]}
].
suite() ->
[
{timetrap, {minutes, 3}}
].
%% -------------------------------------------------------------------
%% Testsuite setup/teardown.
%% -------------------------------------------------------------------
init_per_suite(Config) ->
rabbit_ct_helpers:log_environment(),
rabbit_ct_helpers:run_setup_steps(Config).
end_per_suite(Config) ->
rabbit_ct_helpers:run_teardown_steps(Config).
init_per_group(classic_queue, Config) ->
rabbit_ct_helpers:set_config(
Config,
[{queue_args, [{<<"x-queue-type">>, longstr, <<"classic">>}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, []},
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
{queue_durable, true}]);
init_per_group(quorum_queue, Config) ->
Use the new feature flags helper in `rabbit_ct_broker_helpers` It simplifies the test of feature flags support. While here, update the `queue_parallel` testsuite to use it.
2019-02-13 22:17:03 +08:00
case rabbit_ct_broker_helpers:enable_feature_flag(Config, quorum_queue) of
ok ->
rabbit_ct_helpers:set_config(
Config,
[{queue_args, [{<<"x-queue-type">>, longstr, <<"quorum">>}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, []},
Use the new feature flags helper in `rabbit_ct_broker_helpers` It simplifies the test of feature flags support. While here, update the `queue_parallel` testsuite to use it.
2019-02-13 22:17:03 +08:00
{queue_durable, true}]);
Skip ->
Skip
end;
Use in memory limit for quorum queue tests All should work as normal, only memory usage it's affected [#164735591]
2019-03-27 18:37:58 +08:00
init_per_group(quorum_queue_in_memory_limit, Config) ->
case rabbit_ct_broker_helpers:enable_feature_flag(Config, quorum_queue) of
ok ->
rabbit_ct_helpers:set_config(
Config,
[{queue_args, [{<<"x-queue-type">>, longstr, <<"quorum">>},
{<<"x-max-in-memory-length">>, long, 1}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, []},
Use in memory limit for quorum queue tests All should work as normal, only memory usage it's affected [#164735591]
2019-03-27 18:37:58 +08:00
{queue_durable, true}]);
Skip ->
Skip
end;
init_per_group(quorum_queue_in_memory_bytes, Config) ->
case rabbit_ct_broker_helpers:enable_feature_flag(Config, quorum_queue) of
ok ->
rabbit_ct_helpers:set_config(
Config,
[{queue_args, [{<<"x-queue-type">>, longstr, <<"quorum">>},
{<<"x-max-in-memory-bytes">>, long, 1}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, []},
Use in memory limit for quorum queue tests All should work as normal, only memory usage it's affected [#164735591]
2019-03-27 18:37:58 +08:00
{queue_durable, true}]);
Skip ->
Skip
end;
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
init_per_group(mirrored_queue, Config) ->
rabbit_ct_broker_helpers:set_ha_policy(Config, 0, <<"^max_length.*queue">>,
<<"all">>, [{<<"ha-sync-mode">>, <<"automatic">>}]),
Config1 = rabbit_ct_helpers:set_config(
Config, [{is_mirrored, true},
{queue_args, [{<<"x-queue-type">>, longstr, <<"classic">>}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, []},
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
{queue_durable, true}]),
rabbit_ct_helpers:run_steps(Config1, []);
Stream Queue This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
2020-09-29 18:43:24 +08:00
init_per_group(stream_queue, Config) ->
case rabbit_ct_broker_helpers:enable_feature_flag(Config, stream_queue) of
ok ->
rabbit_ct_helpers:set_config(
Config,
[{queue_args, [{<<"x-queue-type">>, longstr, <<"stream">>}]},
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
{consumer_args, [{<<"x-stream-offset">>, long, 0}]},
Stream Queue This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
2020-09-29 18:43:24 +08:00
{queue_durable, true}]);
Skip ->
Skip
end;
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
init_per_group(Group, Config0) ->
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
case lists:member({group, Group}, all()) of
true ->
Switch to 3 node clusters instead of 2 As this works better with mixed Ra machine versions
2020-07-15 17:49:34 +08:00
ClusterSize = 3,
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
Config = rabbit_ct_helpers:merge_app_env(
Move consumer timeout tests to own SUITE Also handle case where client does not support consumer cancellation and rename the queue_cleanup timer to a generic "tick" timer for channels to perform periodic activities. [#164212469]
2019-04-08 23:12:26 +08:00
Config0, {rabbit, [{channel_tick_interval, 1000},
Stream Queue This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
2020-09-29 18:43:24 +08:00
{quorum_tick_interval, 1000},
{stream_tick_interval, 1000}]}),
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
Config1 = rabbit_ct_helpers:set_config(
Config, [ {rmq_nodename_suffix, Group},
{rmq_nodes_count, ClusterSize}
]),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
rabbit_ct_helpers:run_steps(Config1,
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
rabbit_ct_broker_helpers:setup_steps() ++
rabbit_ct_client_helpers:setup_steps());
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
false ->
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
rabbit_ct_helpers:run_steps(Config0, [])
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
end.
end_per_group(Group, Config) ->
case lists:member({group, Group}, all()) of
true ->
rabbit_ct_helpers:run_steps(Config,
rabbit_ct_client_helpers:teardown_steps() ++
rabbit_ct_broker_helpers:teardown_steps());
false ->
Config
end.
init_per_testcase(Testcase, Config) ->
Group = proplists:get_value(name, ?config(tc_group_properties, Config)),
Q = rabbit_data_coercion:to_binary(io_lib:format("~p_~p", [Group, Testcase])),
Q2 = rabbit_data_coercion:to_binary(io_lib:format("~p_~p_2", [Group, Testcase])),
Config1 = rabbit_ct_helpers:set_config(Config, [{queue_name, Q},
{queue_name_2, Q2}]),
rabbit_ct_helpers:testcase_started(Config1, Testcase).
end_per_testcase(Testcase, Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
amqp_channel:call(Ch, #'queue.delete'{queue = ?config(queue_name, Config)}),
amqp_channel:call(Ch, #'queue.delete'{queue = ?config(queue_name_2, Config)}),
rabbit_ct_helpers:testcase_finished(Config, Testcase).
publish(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]).
consume(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]).
consume_first_empty(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
consume_empty(Ch, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
consume(Ch, QName, true, [<<"msg1">>]),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
rabbit_ct_client_helpers:close_channel(Ch).
consume_from_empty_queue(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
consume_empty(Ch, QName).
consume_and_autoack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
consume(Ch, QName, true, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]).
subscribe(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
%% Let's set consumer prefetch so it works with stream queues
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
?assertMatch(#'basic.qos_ok'{},
amqp_channel:call(Ch, #'basic.qos'{global = false,
prefetch_count = 10})),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive_basic_deliver(false),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
rabbit_ct_client_helpers:close_channel(Ch),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]).
subscribe_consumers(Config) ->
[Server | _] = rabbit_ct_broker_helpers:get_node_configs(Config, nodename),
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
CArgs = ?config(consumer_args, Config),
?assertMatch(#'basic.qos_ok'{},
amqp_channel:call(Ch, #'basic.qos'{global = false,
prefetch_count = 10})),
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
%% validate we can retrieve the consumers
Consumers = rpc:call(Server, rabbit_amqqueue, consumers_all, [<<"/">>]),
[Consumer] = lists:filter(fun(Props) ->
Resource = proplists:get_value(queue_name, Props),
QName == Resource#resource.name
end, Consumers),
?assert(is_pid(proplists:get_value(channel_pid, Consumer))),
?assert(is_binary(proplists:get_value(consumer_tag, Consumer))),
?assertEqual(true, proplists:get_value(ack_required, Consumer)),
?assertEqual(10, proplists:get_value(prefetch_count, Consumer)),
?assertEqual([], proplists:get_value(arguments, Consumer)),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
rabbit_ct_client_helpers:close_channel(Ch).
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_with_autoack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>]),
wait_for_messages(Config, [[QName, <<"2">>, <<"2">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, true, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive_basic_deliver(false),
receive_basic_deliver(false),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]).
consume_and_ack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
[DeliveryTag] = consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
consume_and_multiple_ack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
[_, _, DeliveryTag] = consume(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag,
multiple = true}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_ack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive
{#'basic.deliver'{delivery_tag = DeliveryTag}, _} ->
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_multiple_ack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive_basic_deliver(false),
receive_basic_deliver(false),
receive
{#'basic.deliver'{delivery_tag = DeliveryTag}, _} ->
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag,
multiple = true}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
Compaction: more logging, a test
2019-05-10 05:00:47 +08:00
trigger_message_store_compaction(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
N = 12000,
[publish(Ch, QName, [binary:copy(<<"a">>, 5000)]) || _ <- lists:seq(1, N)],
wait_for_messages(Config, [[QName, <<"12000">>, <<"12000">>, <<"0">>]]),
AllDTags = rabbit_ct_client_helpers:consume_without_acknowledging(Ch, QName, N),
ToAck = lists:filter(fun (I) -> I > 500 andalso I < 11200 end, AllDTags),
[amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = Tag,
multiple = false}) || Tag <- ToAck],
%% give compaction a moment to start in and finish
timer:sleep(5000),
amqp_channel:cast(Ch, #'queue.purge'{queue = QName}),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_requeue_multiple_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive_basic_deliver(false),
receive_basic_deliver(false),
receive
{#'basic.deliver'{delivery_tag = DeliveryTag,
redelivered = false}, _} ->
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = true,
requeue = true}),
receive_basic_deliver(true),
receive_basic_deliver(true),
receive
{#'basic.deliver'{delivery_tag = DeliveryTag1,
redelivered = true}, _} ->
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag1,
multiple = true}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
end
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
consume_and_requeue_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>]),
wait_for_messages(Config, [[QName, <<"2">>, <<"2">>, <<"0">>]]),
[DeliveryTag] = consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"2">>, <<"1">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = false,
requeue = true}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"2">>, <<"2">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
consume_and_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
[DeliveryTag] = consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = false,
requeue = false}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
consume_and_requeue_multiple_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
[_, _, DeliveryTag] = consume(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = true,
requeue = true}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
consume_and_multiple_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
[_, _, DeliveryTag] = consume(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = true,
requeue = false}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_requeue_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive
{#'basic.deliver'{delivery_tag = DeliveryTag,
redelivered = false}, _} ->
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = false,
requeue = true}),
receive
{#'basic.deliver'{delivery_tag = DeliveryTag1,
redelivered = true}, _} ->
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag1}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
end
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive
{#'basic.deliver'{delivery_tag = DeliveryTag,
redelivered = false}, _} ->
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = false,
requeue = false}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
subscribe_and_multiple_nack(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive_basic_deliver(false),
receive_basic_deliver(false),
receive
{#'basic.deliver'{delivery_tag = DeliveryTag,
redelivered = false}, _} ->
wait_for_messages(Config, [[QName, <<"3">>, <<"0">>, <<"3">>]]),
amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = DeliveryTag,
multiple = true,
requeue = false}),
wait_for_messages(Config, [[QName, <<"0">>, <<"0">>, <<"0">>]])
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
%% TODO test with single active
basic_cancel(Config) ->
[Server | _] = rabbit_ct_broker_helpers:get_node_configs(Config, nodename),
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
CArgs = ?config(consumer_args, Config),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
CTag = atom_to_binary(?FUNCTION_NAME, utf8),
Stream Queue This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
2020-09-29 18:43:24 +08:00
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, QName, false, CTag, CArgs),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
receive
{#'basic.deliver'{delivery_tag = DeliveryTag}, _} ->
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
amqp_channel:call(Ch, #'basic.cancel'{consumer_tag = CTag}),
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
Consumers = rpc:call(Server, rabbit_amqqueue, consumers_all, [<<"/">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
?assertEqual([], lists:filter(fun(Props) ->
Resource = proplists:get_value(queue_name, Props),
QName == Resource#resource.name
end, Consumers)),
publish(Ch, QName, [<<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"2">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.ack'{delivery_tag = DeliveryTag}),
wait_for_messages(Config, [[QName, <<"2">>, <<"2">>, <<"0">>]])
after 5000 ->
exit(basic_deliver_timeout)
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
end,
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
purge(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>]),
wait_for_messages(Config, [[QName, <<"2">>, <<"2">>, <<"0">>]]),
[_] = consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"2">>, <<"1">>, <<"1">>]]),
{'queue.purge_ok', 1} = amqp_channel:call(Ch, #'queue.purge'{queue = QName}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
basic_recover(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
publish(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
[_] = consume(Ch, QName, [<<"msg1">>]),
wait_for_messages(Config, [[QName, <<"1">>, <<"0">>, <<"1">>]]),
amqp_channel:cast(Ch, #'basic.recover'{requeue = true}),
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
wait_for_messages(Config, [[QName, <<"1">>, <<"1">>, <<"0">>]]),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
delete_immediately_by_pid_fails(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
Args = ?config(queue_args, Config),
Durable = ?config(queue_durable, Config),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
Cmd = ["eval", "{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<\"/\">>, queue, <<\"" ++ binary_to_list(QName) ++ "\">>)), Pid = rabbit_amqqueue:pid_of(Q), rabbit_amqqueue:delete_immediately([Pid])."],
{ok, Msg} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, Cmd),
?assertEqual(match, re:run(Msg, ".*error.*", [{capture, none}])),
?assertEqual({'queue.declare_ok', QName, 0, 0},
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = Durable,
passive = true,
auto_delete = false,
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
arguments = Args})),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
delete_immediately_by_pid_succeeds(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
Args = ?config(queue_args, Config),
Durable = ?config(queue_durable, Config),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
Cmd = ["eval", "{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<\"/\">>, queue, <<\"" ++ binary_to_list(QName) ++ "\">>)), Pid = rabbit_amqqueue:pid_of(Q), rabbit_amqqueue:delete_immediately([Pid])."],
{ok, Msg} = rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, Cmd),
?assertEqual(match, re:run(Msg, ".*ok.*", [{capture, none}])),
?assertExit(
{{shutdown, {server_initiated_close, 404, _}}, _},
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = Durable,
passive = true,
auto_delete = false,
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
arguments = Args})),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
delete_immediately_by_resource(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
Args = ?config(queue_args, Config),
Durable = ?config(queue_durable, Config),
QName = ?config(queue_name, Config),
declare_queue(Ch, Config, QName),
Cmd = ["eval", "rabbit_amqqueue:delete_immediately_by_resource([rabbit_misc:r(<<\"/\">>, queue, <<\"" ++ binary_to_list(QName) ++ "\">>)])."],
?assertEqual({ok, "ok\n"}, rabbit_ct_broker_helpers:rabbitmqctl(Config, 0, Cmd)),
?assertExit(
{{shutdown, {server_initiated_close, 404, _}}, _},
amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = Durable,
passive = true,
auto_delete = false,
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
arguments = Args})),
rabbit_ct_client_helpers:close_channel(Ch),
ok.
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
extra_bcc_option(Config) ->
{_, Ch} = rabbit_ct_client_helpers:open_connection_and_channel(Config, 0),
QName = <<"a_queue_with_extra_bcc">>,
delete_queue(Ch, QName),
declare_queue(Ch, Config, QName),
ExtraBCC = <<"extra.bcc">>,
delete_queue(Ch, ExtraBCC),
declare_bcc_queue(Ch, ExtraBCC),
set_queue_options(Config, QName, #{
extra_bcc => ExtraBCC
}),
publish(Ch, QName, [<<"msg1">>, <<"msg2">>, <<"msg3">>]),
wait_for_messages(Config, [[QName, <<"3">>, <<"3">>, <<"0">>]]),
wait_for_messages(Config, [[ExtraBCC, <<"3">>, <<"3">>, <<"0">>]]),
delete_queue(Ch, QName),
delete_queue(Ch, ExtraBCC).
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
%%%%%%%%%%%%%%%%%%%%%%%%
%% Test helpers
%%%%%%%%%%%%%%%%%%%%%%%%
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
declare_queue(Ch, Config, QName) ->
Args = ?config(queue_args, Config),
Durable = ?config(queue_durable, Config),
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = QName,
arguments = Args,
durable = Durable}).
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
declare_bcc_queue(Ch, QName) ->
#'queue.declare_ok'{} = amqp_channel:call(Ch, #'queue.declare'{queue = QName,
durable = true}).
delete_queue(Ch, QName) ->
#'queue.delete_ok'{} = amqp_channel:call(Ch, #'queue.delete'{queue = QName}).
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
publish(Ch, QName, Payloads) ->
[amqp_channel:call(Ch, #'basic.publish'{routing_key = QName}, #amqp_msg{payload = Payload})
|| Payload <- Payloads].
consume(Ch, QName, Payloads) ->
consume(Ch, QName, false, Payloads).
consume(Ch, QName, NoAck, Payloads) ->
[begin
{#'basic.get_ok'{delivery_tag = DTag}, #amqp_msg{payload = Payload}} =
amqp_channel:call(Ch, #'basic.get'{queue = QName,
no_ack = NoAck}),
DTag
end || Payload <- Payloads].
consume_empty(Ch, QName) ->
?assertMatch(#'basic.get_empty'{},
amqp_channel:call(Ch, #'basic.get'{queue = QName})).
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, Queue, NoAck, CArgs) ->
subscribe(Ch, Queue, NoAck, <<"ctag">>, CArgs).
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
subscribe(Ch, Queue, NoAck, Ctag, CArgs) ->
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
amqp_channel:subscribe(Ch, #'basic.consume'{queue = Queue,
no_ack = NoAck,
Run some stream tests in parallel They might seem a bit pointless, as most tests here don't apply to streams, but discovered a bug on parallel declares. The coordinator might return an error (unavailable) when we try to start it in parallel, as ra detects some nodes up when it thinks it should be fully down and shuts down the cluster.
2020-10-22 18:12:05 +08:00
consumer_tag = Ctag,
arguments = CArgs},
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
self()),
receive
Refactor channel state To put static fields into a nested record to avoid some copying on update. Also exit the channel when a basic.get times out. [#164212469]
2019-04-06 00:43:32 +08:00
#'basic.consume_ok'{consumer_tag = Ctag} ->
Move some quorum queue tests to a generic queue type suite. Additional tests for basic.cancel which must NOT return unacked messages to the queue as implemented by quorum queues.
2019-02-06 21:15:07 +08:00
ok
end.
receive_basic_deliver(Redelivered) ->
receive
{#'basic.deliver'{redelivered = R}, _} when R == Redelivered ->
ok
end.
Implement consumer channel timeouts Such that if a consumer has a message awaiting ack for longer than this timeout the consumer will either be cancelled (if the client supports it) or the channel will be closed to ensure the message does not get stuck permanently.
2019-04-03 21:53:01 +08:00
flush(T) ->
receive X ->
ct:pal("flushed ~w", [X]),
flush(T)
after T ->
ok
end.
Make it possible for queues to have extra BCC targets specified as options This introduces a backup mechanism that can be controlled by plugins via policies. Benchmarks suggest the cost of this change on Erlang 24 is well under 1%. With a stream target, it is less than routing to one extra queue of the same type (e.g. a quorum queue).
2021-05-19 02:42:00 +08:00
set_queue_options(Config, QName, Options) ->
rabbit_ct_broker_helpers:rpc(Config, 0, ?MODULE, set_queue_options1, [QName, Options]).
set_queue_options1(QName, Options) ->
rabbit_misc:execute_mnesia_transaction(fun() ->
rabbit_amqqueue:update(rabbit_misc:r(<<"/">>, queue, QName),
fun(Q) ->
amqqueue:set_options(Q, Options)
end)
end).