Commit Graph

1149 Commits

Author SHA1 Message Date
Philip Kuryloski 388654c542
Add a partial Bazel build (#2938)
Adds WORKSPACE.bazel, BUILD.bazel & *.bzl files for partial build & test with Bazel. Introduces a build-time dependency on https://github.com/rabbitmq/bazel-erlang
2021-03-29 11:01:43 +02:00
Philip Kuryloski 09e85d2e3d
Merge pull request #2935 from rabbitmq/rabbitmq-queue-int-tests
Fix integration tests to wait until ra cluster is ready
2021-03-26 17:28:06 +01:00
dcorbacho a1caff2a86 Fix integration tests to wait until ra cluster is ready
Publish/confirm before grow/shrink members is enough
2021-03-26 17:04:50 +01:00
Philip Kuryloski 1ead01081a Increase startup delay range in peer_discovery_classic_config_SUITE
I suspect the second ra system for coordination requires a bit more
time in boot, as this seems to flake more often since the merge
2021-03-26 14:11:36 +01:00
Philip Kuryloski 3c0c0901b1 Restore retry in peer_discovery_classic_config_SUITE
It was accidentally left commented out
2021-03-25 20:05:36 +01:00
Philip Kuryloski c313f36b57 Fix Makefile for feature_flags_SUITE_data/my_plugin
It was not updated for the rabbitmq-components.mk consolidation
2021-03-25 19:43:48 +01:00
Philip Kuryloski 008e47ef3c Fixup the behavior of rabbit_mnesia:is_virgin_node/0
Given the addition of the Coord ra system (and additional files on disk)
2021-03-25 10:49:17 +01:00
kjnilsson 8d8b67bb34 fix rabbit_fifo_int_SUITE 2021-03-24 14:17:34 +00:00
Michael Klishin 8eac876bc8
Use "quorum_queues" for QQ Ra system
"quorum" and "coordination" are not very distinctive
2021-03-22 21:44:19 +03:00
kjnilsson 75cea78415
fixes 2021-03-22 21:44:19 +03:00
kjnilsson f6f02a5d2d
ra systems wip 2021-03-22 21:44:15 +03:00
Philip Kuryloski a63f169fcb Remove duplicate rabbitmq-components.mk and erlang.mk files
Also adjust the references in rabbitmq-components.mk to account for
post monorepo locations
2021-03-22 15:40:19 +01:00
Michael Klishin 373285093e
Merge pull request #2899 from rabbitmq/parallel-stream-suite
Run most stream tests in parallel
2021-03-19 22:21:18 +03:00
Jean-Sébastien Pédron 9fd2d68e7a
rabbit_prelaunch_logging: $RABBITMQ_LOGS doesn't override log level
... if it is set in the configuration file.

Here is an example of that use case:
* The official Docker image sets RABBITMQ_LOGS=- in the environment
* A user of that image adds a configuration file with:
      log.console.level = debug

The initial implementation, introduced in rabbitmq/rabbitmq-server#2861,
considered that if the output is overriden in the environment (through
$RABBITMQ_LOGS), any output configuration in the configuration file is
ignored.

The problem is that the output-specific configuration could also set the
log level which is not changed by $RABBITMQ_LOGS. This patch fixes that
by keeping the log level from the configuration (if it is set obviously)
even if the output is overridden in the environment.
2021-03-19 15:43:28 +01:00
dcorbacho 9b3b5d48ec Run most stream tests in parallel
The test suite isn't faster, I guess some contention on the coordinator,
but is finding some bugs.
2021-03-17 21:32:42 +01:00
kjnilsson cbf0107605 Stream coordinator bug fix
Fix issue where a deleted replica could be restarted if the leader went
down whilst the replica was still running it's start phase.
2021-03-17 13:54:28 +00:00
kjnilsson 9d83e0c5d9 Add logging to config decryption test
To possibly get a bit more information on failure reasons on GH Actions.
2021-03-16 16:28:41 +00:00
kjnilsson 3a26cf8654 Stream coordinator: handle commands for unknown streams
To avoid crashing.
2021-03-12 15:04:40 +00:00
kjnilsson 1709208105 Throw resource error when no local stream member
As well as some additional tests
2021-03-12 15:04:40 +00:00
dcorbacho e19aca8075 Use right map fields to compute streams info 2021-03-12 15:04:40 +00:00
kjnilsson 7fa3f6b6e1 Stream Coordinator: primitive backoff
Sleep for 5s after a failure due to a node being down before reporting
back to stream coordinator (which will immediately retry).

stream coordinator: correct command type spec

tidy up

fix rabbit_fifo_prop tests

stream coord: add function for member state query
2021-03-12 15:03:47 +00:00
kjnilsson bb3e0a7674 Move stream coordinator unit tests into ct suite 2021-03-12 15:03:10 +00:00
kjnilsson 9fb2e6d2dd Stream Coordinator refactor 2021-03-12 15:03:08 +00:00
Jean-Sébastien Pédron cdcf602749
Switch from Lager to the new Erlang Logger API for logging
The configuration remains the same for the end-user. The only exception
is the log root directory: it is now set through the `log_root`
application env. variable in `rabbit`. People using the Cuttlefish-based
configuration file are not affected by this exception.

The main change is how the logging facility is configured. It now
happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is
removed.

The supported outputs remain the same: the console, text files, the
`amq.rabbitmq.log` exchange and syslog.

The message text format slightly changed: the timestamp is more precise
(now to the microsecond) and the level can be abbreviated to always be
4-character long to align all messages and improve readability. Here is
an example:

    2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE ==
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com

The example above also shows that multiline messages are supported and
each line is prepended with the same prefix (the timestamp, the level
and the Erlang process PID).

JSON is also supported as a message format and now for any outputs.
Indeed, it is possible to use it with e.g. syslog or the exchange. Here
is an example of a JSON-formatted message sent to syslog:

    Mar  3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}}

For quick testing, the values accepted by the `$RABBITMQ_LOGS`
environment variables were extended:
  * `-` still means stdout
  * `-stderr` means stderr
  * `syslog:` means syslog on localhost
  * `exchange:` means logging to `amq.rabbitmq.log`

`$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in
addition to the existing `+color` one). With that modifier, messages are
formatted as JSON intead of plain text.

The `rabbitmqctl rotate_logs` command is deprecated. The reason is
Logger does not expose a function to force log rotation. However, it
will detect when a file was rotated by an external tool.

From a developer point of view, the old `rabbit_log*` API remains
supported, though it is now deprecated. It is implemented as regular
modules: there is no `parse_transform` involved anymore.

In the code, it is recommended to use the new Logger macros. For
instance, `?LOG_INFO(Format, Args)`. If possible, messages should be
augmented with some metadata. For instance (note the map after the
message):

    ?LOG_NOTICE("Logging: switching to configured handler(s); following "
                "messages may not be visible in this log output",
                #{domain => ?RMQLOG_DOMAIN_PRELAUNCH}),

Domains in Erlang Logger parlance are the way to categorize messages.
Some predefined domains, matching previous categories, are currently
defined in `rabbit_common/include/logging.hrl` or headers in the
relevant plugins for plugin-specific categories.

At this point, very few messages have been converted from the old
`rabbit_log*` API to the new macros. It can be done gradually when
working on a particular module or logging.

The Erlang builtin console/file handler, `logger_std_h`, has been forked
because it lacks date-based file rotation. The configuration of
date-based rotation is identical to Lager. Once the dust has settled for
this feature, the goal is to submit it upstream for inclusion in Erlang.
The forked module is calld `rabbit_logger_std_h` and is based
`logger_std_h` in Erlang 23.0.
2021-03-11 15:17:36 +01:00
Michael Klishin d77609bba4
Merge pull request #2846 from rabbitmq/cleanup-rabbit-fifo-usage
Clean up rabbit_fifo_usage table on queue.delete
2021-03-03 18:33:33 +03:00
Michael Klishin a2f98f25e9
Merge pull request #2804 from rabbitmq/rabbitmq-server-2756
Add federation support for quorum queues
2021-02-25 19:10:15 +03:00
dcorbacho a147cc4877 Clean up rabbit_fifo_usage table on queue.delete 2021-02-25 16:57:43 +01:00
Michael Klishin cd1a271499
As of Lager 3.8.2, Lager has a log_root default
so override it unconditionally.
2021-02-25 00:43:02 +03:00
dcorbacho 699cd1ab29 Add federation support for quorum queues 2021-02-18 17:15:47 +01:00
Carl Hörberg 413bfe7b37 Disable Erlang busy wait by default
By disabling Erlang busy wait threshold CPU usage with 5000 idle connection
drops from 110% to 14%. Throughput does not seem to be affected at all,
if any thing it actually goes up a bit when you have 5000 idle connections
(because less CPU cycles are wasted polling idle connections).

rabbitmq-perf-test-2.13.0/bin/runjava com.rabbitmq.perf.PerfTest -s 8000 -z 15

With default erlang busy wait threshold:
id: test-115706-497, sending rate avg: 39589 msg/s
id: test-115706-497, receiving rate avg: 39570 msg/s

With busy wait disabled:
id: test-115807-719, sending rate avg: 40340 msg/s
id: test-115807-719, receiving rate avg: 40301 msg/s

rabbitmq-diagnostics runtime_thread_stats output while running the
PerfTest:

with default busy wait threshold:

Stats per type:
         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
           aux    0.01%    0.00%    0.00%    0.00%    0.00%    0.00%   99.98%
dirty_cpu_sche    0.00%    0.00%    0.00%    0.03%    0.05%    0.00%   99.92%
dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.01%    0.00%   99.99%
          poll    0.00%    0.67%    0.00%    0.00%    0.00%    0.00%   99.33%
     scheduler    0.69%    0.18%   28.41%    5.49%    9.50%    7.43%   48.29%

without busy wait threshold:

Stats per type:
         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
           aux    0.01%    0.00%    0.00%    0.00%    0.01%    0.00%   99.98%
dirty_cpu_sche    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
          poll    0.00%    0.77%    0.00%    0.00%    0.00%    0.00%   99.23%
     scheduler    0.70%    0.14%   28.29%    5.41%    0.86%    7.22%   57.38%
2021-02-10 12:35:12 +01:00
Michael Klishin ad20bfbc40
Use new crypto API cipher name here
References rabbitmq/credentials-obfuscation#10
2021-02-09 11:22:48 +03:00
Michael Klishin 0939cec51a
Exclude aes_ige256 in one more test suite 2021-02-08 11:21:16 +03:00
Michael Klishin c7b9c39352
Don't perform CMQ leadership transfer when entering maintenance mode
The time this operation can take in clusters with a lot of classic
mirrored queue (say, 10s or 100s of thousands) be prohibitive for
upgrades.

Upgrades that use a health check to ensure that there are in-sync
replicas before entering maintenance mode, in which case
the transfer is not really necessary.

All of the above is more obvious with the recent changes in #2749.
2021-01-27 19:11:26 +03:00
Michael Klishin 52479099ec
Bump (c) year 2021-01-22 09:00:14 +03:00
kjnilsson f2418cfe4c Fix crash bug in QQ state conversion
When there are consumers in the service queue.
2021-01-20 14:19:33 +00:00
kjnilsson 2f0dba45d8 Stream: Channel resend on leader change
Detect when a new stream leader is elected and make stream_queues
re-send any unconfirmed, pending messages to ensure they did not get
lost during the leader change. This is done using the osiris
deduplication feature to ensure the resend does not create duplicates of
messages in the stream.
2021-01-13 12:09:44 +00:00
dcorbacho 9ef9dde6ce Apply retention policy in all osiris members 2021-01-12 12:18:13 +00:00
dcorbacho e5a2eaaa0d Update retention when only stream retention policy has changed
In any other case, the worker needs to be restarted
2021-01-12 12:18:13 +00:00
Michal Kuratczyk 6a81589c11 Expose `bypass_pem_cache` through rabbitmq.conf
Bypassing PEM cache may speed up TLS handshakes in some cases as described
here:
https://blog.heroku.com/how-we-sped-up-sni-tls-handshakes-by-5x
2020-12-17 16:53:14 +01:00
Michael Klishin 4ea9ce1c0b
Clarify what version will be the first to use this format 2020-12-09 12:48:56 +03:00
Michael Klishin e4c37db689
Support importing users with arrays of tags
as opposed to a comma-separated binary.

Part of #2667.
2020-12-08 18:22:56 +03:00
kjnilsson 6fdb7d29ec Handle errors in crashing_queues_SUITE
As the connection may crash during the previous declaration and a caught
error would be returned in amqp_connection:open_channel/1 that wasn't
handled previously. Exactly how things fail in this test is most likely
very timing dependent and may vary.

Also fixes mqtt test where the process that set up a mock auth ETS table
was transient when an rpc timeout was introduced
2020-12-03 13:56:09 +00:00
Luke Bakken ccf624211a
Add test that fails prior to the change for #2668 2020-12-02 12:33:02 -08:00
Arnaud Cogoluègnes ffd66027af
Merge pull request #2506 from rabbitmq/stream-timestamp-offset
Support timestamp offsets for stream consumers
2020-11-27 14:49:38 +01:00
Arnaud Cogoluègnes 43cfb45a74
Convert AMQP 091 timestamp to millisecond
For start offset in stream queue.
2020-11-27 14:47:36 +01:00
kjnilsson ea7c9e9b61 QQ: Emit release cursor for empty basic gets
Else an application that polled an empty quorum queue frequntly using basic.get
would never result in a snapshot being taken and results in unlimited
log growth.
2020-11-19 15:59:51 +00:00
dcorbacho f23a51261d Merge remote-tracking branch 'origin/master' into stream-timestamp-offset 2020-11-18 14:27:41 +00:00
kjnilsson d88b623c18 Use correct credit mode x-credit
When the x-credit consumer arg is defined Quorum Queues should use use
credit mode `credited` and not `simple_prefetch`.
2020-11-16 10:45:10 +01:00
Philip Kuryloski a1fe3ab061 Change repo "root" to deps/rabbit
rabbit must not be the monorepo root application, as other applications depend on it
2020-11-13 14:34:42 +01:00