Commit Graph

2225 Commits

Author SHA1 Message Date
Rin Kuryloski bef2756b5f Fixes for various dialyzer warnings 2022-12-02 09:14:08 +01:00
Rin Kuryloski 57d7f7f8b5 Fix dialyzer warnings in rabbit_common 2022-12-02 09:13:56 +01:00
Rin Kuryloski 344e4553ba Fix rabbit_misc:hexify/1 when called with a list 2022-12-02 09:13:47 +01:00
Jean-Sébastien Pédron ae06f79c10
rabbit: Introduce `rabbit:data_dir/0`
This function returns the data directory where all subsystems should
store their files.

Historically, this was the Mnesia directory. But semantically, this
should be the reverse: RabbitMQ owns the data directory and Mnesia is
configured to put its files there too.

`rabbit_mnesia:dir/0` now calls `rabbit:data_dir/0`.

Other subsystems will be modified in a follow-up commit to call
`rabbit:data_dir/0` instead of `rabbit_mnesia:dir/0`.
2022-11-30 14:41:28 +01:00
Jean-Sébastien Pédron 99b14fd0fa
rabbit_env: Rename `mnesia_*dir` to `data_*dir`
The location and name of this directory remains the same for
compatibility reasons. Therefore, it sill contains "mnesia" in its name.
However, semantically, we want this directory to be unrelated to Mnesia.

In the end, many subsystems write files and directories there, including
Mnesia, all Ra systems and in the future, Khepri.
2022-11-30 13:00:49 +01:00
Jean-Sébastien Pédron e0b4bce4f4
rabbit_env: Rename `data_dir` to `home_dir`
This value is used internally by `rabbit_env` and usually not read by
RabbitMQ otherwise.

This patch prepares the rename of `mnesia_dir` to `data_dir`, in order
to not semantically rely on Mnesia configuration or use to locate data,
whether it is stored in Mnesia or not.
2022-11-30 13:00:45 +01:00
Michal Kuratczyk 05ec5e488f
Remove unnecessary call to priority_queue:len/1
Seems like we are not using it anywhere in our code base.
It's unlikely that it's used somewhere else and even if it is,
the API is backwards compatible - we just pass 0, as if the priority_queue
was empty.
2022-11-24 13:02:43 +01:00
Michael Klishin 8ad1c8be5c
Bump credentials_obfuscation to 3.2.0 2022-11-08 03:35:33 +04:00
Michael Klishin fb2b219522
Merge pull request #6126 from rabbitmq/partially-revert-delegate-opt-3865
Partially revert delegate optimisation
2022-10-17 21:05:58 +04:00
Karl Nilsson 8804434a01 Partially revert delegate optimisation
That was done in PR #3865.

The changes introduced in #3865 can cause message arrival ordering guarantees
between two logical erlang process (sending messages via delegate) to
be violated as a message sent to a single destination can overtake a prior
message sent as part of a fan-out. This is due to the fact that the fan-out
take a different route via the delegate process than the direct delivery that
bypasses it.

This commit only reverses it for the `invoke_no_result/2|3` API and leaves the
optimisation in for the synchronous `invoke/` API. This means that the message
send ordering you expect between erlang processes still can be violated when
mixing invoke and invoke_no_result invocations. As far as I can see there are
no places where the code relies on this and there are uses of invoke (mgmt db)
that very well could benefit from avoiding the additional copying.
2022-10-17 15:15:22 +01:00
Michael Klishin 08d804786c Handle a rare case with multiple dist ports
See rabbitmq/rabbitmq-server#6097 for background
2022-10-17 15:37:33 +04:00
Luke Bakken 7fe159edef
Yolo-replace format strings
Replaces `~s` and `~p` with their unicode-friendly counterparts.

```
git ls-files *.erl | xargs sed -i.ORIG -e s/~s>/~ts/g -e s/~p>/~tp/g
```
2022-10-10 10:32:03 +04:00
Jean-Sébastien Pédron 4b132daaba
Remove upgrade-specific log file
This category should be unused with the decommissioning of the old
upgrade subsystem (in favor of the feature flags subsystem). It means:
1. The upgrade log file will not be created by default anymore.
2. The `$RABBITMQ_UPGRADE_LOG` environment variable is now unsupported.

The configuration variables remain to avoid breaking an existing and
working configuration.
2022-10-06 21:28:50 +02:00
Luke Bakken 74cddc29ab
Update from `master` to `main`
Fixes #5980

Reference rabbitmq/rabbitmq-metronome#16
2022-10-03 12:32:04 -07:00
Michael Klishin 69b06d30f1
Merge pull request #4522 from rabbitmq/loic-cq-dont-reduce-memory-usage
CQ: Merge lazy/default behavior into a unified mode
2022-10-01 20:11:41 +04:00
Luke Bakken 553aab51c9
Update branch names and links
Fix publish of libs to hex.pm

@lhoguin noticed that the hex packages for the amqp_client, amqp10_client and related project do not currently work with erlang.mk. This PR fixes this issue.

Tested using this project: https://github.com/lukebakken/amqp-clients-test.git
2022-09-29 13:58:56 -07:00
Michal Kuratczyk 2855278034
Migrate from supervisor2 to supervisor 2022-09-27 13:53:06 +02:00
Loïc Hoguin 3683ab9a6e
CQ: Use v2 sets instead of gb_sets for confirms
For the following flags I see an improvement of
30k/s to 34k/s on my machine:

-x 1 -y 1 -A 1000 -q 1000 -c 1000 -s 1000 -f persistent
-u cqv2 --queue-args=x-queue-version=2
2022-09-27 12:00:08 +02:00
Luke Bakken 3c1c7ada58
Log errors from unicode:characters_to_list 2022-09-24 11:37:29 -07:00
Luke Bakken 755ac7176b
Follow-up to #5486
Discovered by @dumbbell

Ensure externally read strings are saved as utf-8 encoded binaries. This
is necessary since `cmd.exe` on Windows uses ISO-8859-1 encoding and
directories can have latin1 characters, like `RabbitMQ Sérvér`.

The `é` is represented by decimal `233` in the ISO-8859-1 encoding. The
unicode code point is the same decimal value, `233`, so you will see
this in the charlist data. However, when encoded using utf-8, this
becomes the two-byte sequence `C3 A9` (hexidecimal).

When reading strings from env variables and configuration, they will be
unicode charlists, with each list item representing a unicode code
point. All of Erlang string functions can handle strings in this form.
Once these strings are written to ETS or Mnesia, they will be converted
to utf-8 encoded binaries. Prior to these changes just
`list_to_binary/1` was used.

Fix xref error

re:replace requires an iodata, which is not a list of unicode code points

Correctly parse unicode vhost tags

Fix many format strings to account for utf8 input. Try again to fix unicode vhost tags

More format string fixes, try to get the CONFIG_FILE var correct

Be sure to use the `unicode` option for re:replace when necessary

More unicode format strings, add unicode option to re:split

More format strings updated

Change ~s to ~ts for vhost format strings

Change ~s to ~ts for more vhost format strings

Change ~s to ~ts for more vhost format strings

Add unicode format chars to disk monitor

Quote the directory on unix

Finally figure out the correct way to pass unicode to the port
2022-09-24 11:19:59 -07:00
David Ansari d4a25f9092 Use maps:get/3
instead of maps:find/2.
This creates less garbage.
2022-09-09 10:52:38 +00:00
David Ansari b6952540a3 Remove rabbit_misc:atom_to_binary/1
Nowadays, we have erlang:atom_to_binary/1.
2022-09-09 10:52:38 +00:00
David Ansari b953b0f10e Stop sending stats to rabbit_event
Stop sending connection_stats from protocol readers to rabbit_event.
Stop sending queue_stats from queues to rabbit_event.
Sending these stats every 5 seconds to the event manager process is
superfluous because noone handles these events.

They seem to be a relict from before rabbit_core_metrics ETS tables got
introduced in 2016.

Delete test head_message_timestamp_statistics because it tests that
head_message_timestamp is set correctly in queue_stats events
although queue_stats events are used nowhere.
The functionality of head_message_timestamp itself is still tested in
deps/rabbit/test/priority_queue_SUITE.erl and
deps/rabbit/test/temp/head_message_timestamp_tests.py
2022-09-09 10:52:38 +00:00
Michael Klishin ab5eeb745c
Merge pull request #5659 from Ayanda-D/allow-ignoring-supervisor2-error-reports
Controllable way to ignore supervisor2 error reports
2022-08-29 02:13:57 +04:00
Ayanda Dube 55526755ec Ensure clean shutdown of rabbit_event manager in supervisor2_SUITE to avoid noisy test terminations 2022-08-28 22:45:52 +01:00
Ayanda Dube f10f81bf90 Rename TRACE_SUP2 to TRACE_SUPERVISOR2 and add to Makefile 2022-08-28 22:41:54 +01:00
Ayanda Dube daf1a52e99 Inlcude test_event_handler in rabbit_common supervisor2_SUITE bazel 2022-08-28 22:37:34 +01:00
Ayanda Dube 80f2116935 Use local include to reference rabbit.hrl in test_event_handler 2022-08-28 19:52:44 +01:00
Michael Klishin cb7d840700 Basic unit tests for rabbit_date_time 2022-08-28 14:43:35 +04:00
Michael Klishin 2153acf8e1 Introduce a concept of release series EOL date
into the code as well as application environment default. This way
this value can be checked and every branch can use a value that makes
sense to it.
2022-08-27 17:33:35 +04:00
Ayanda Dube bd70cb1b9c Use SupPID provided in ?TRACE_SUPERVISOR2_ERROR_LOG macro through
SupName.
2022-08-26 19:43:56 +01:00
Ayanda Dube be6c188d64 Implement tests for controlling supervisor2 error reports.
Tests behaviour controlled at compile time through, macro.
2022-08-26 18:35:47 +01:00
Ayanda Dube fa508aca99 Add a test_event_handler for rabbit_common event based tests 2022-08-26 18:30:37 +01:00
Ayanda Dube 2f78330d35 Control supervisor2 error reports with application variable to be set
in e.g. the `advanced.config` file, or manually in runtime.

This also adds tracing through use of `rabbit_event`, controllable by
use of compile time flag, e.g. TRACE_SUP2.
2022-08-26 18:27:01 +01:00
Johan Rhodin 70c2fb78dc
Fix typo 2022-08-23 16:02:48 -05:00
Péter Gömöri e979b2fad6 Fix a typo in a comment in rabbit_env 2022-08-23 00:54:35 +02:00
Michal Kuratczyk b6006aa207 Remove an old check
It returns true in all supported OTP version.
2022-08-19 17:00:48 +02:00
Michael Klishin b378483a1e
Merge pull request #5486 from rabbitmq/rabbitmq-server-5471-2
Run rabbitmq-env-conf.bat on win32 taking spaces into account
2022-08-16 23:43:09 +04:00
Luke Bakken 032afd8dbc
Fix `badmatch` by taking edge case into account
A couple users reported `badmatch` crashes due to scenarios where
`inet:peername/1` does not return the expected value, most likely due to
the port closing between the time they are listed and when
`inet:peername/1` is called.

Fixes #5496

Discussion in #5490
2022-08-16 09:26:15 -07:00
Rin Kuryloski 575c5f9975 Remove all of the .travis.yml files
since we no longer use them
2022-08-16 09:46:31 +02:00
Luke Bakken fae769458c
Ensure CONF_ENV_FILE is run correctly on win32
Fixes #5471
2022-08-11 10:40:41 -07:00
Michael Klishin 0639f3ecd2
rabbit_json:decode/2: raise a two-tuple error 2022-07-30 04:26:13 +04:00
Michael Klishin 2bd585dd64
rabbit_json:decode/1: use erlang:error/1 like the JSX version did 2022-07-29 10:34:52 +04:00
Michael Klishin bad6e394b2
Swap JSX for Thoas in more places
and adapt rabbit_json for Thoas in the process.

Pair: @pjk25
2022-07-29 10:34:51 +04:00
Michael Klishin 9c99f76579
Replace JSX with Thoas for JSON operations
Thoas is more efficient both in terms of encoding
time and peak memory footprint.

In the process we have discovered an issue:
https://github.com/lpil/thoas/issues/15

Pair: @pjk25
2022-07-29 10:34:47 +04:00
Gabriele Santomaggio 72d522a8ce
change timeout to 5 seconds
Signed-off-by: Gabriele Santomaggio <G.santomaggio@gmail.com>
2022-07-26 14:48:06 +02:00
dcorbacho 40b05357d0 Move mirrored_supervisor to rabbit application
where it belongs
2022-07-21 12:08:35 +02:00
Rin Kuryloski e80c17c56b Be less noisy when sysctl cannot be used for vm_memory_monitor
This avoids printing the full stacktrace when the error comes from the
sysctl invocation, the error message itself is sufficient

In practice, when testing with bazel with macos, sysctl is blocked by
the sandbox, so logging the stacktrace is rather noisy for tests
2022-07-18 12:43:39 +00:00
Luke Bakken f56c84489f
Continuously run powershell to get disk free space
Much faster than starting it every time.

Fixes #4140

Take noeol into account
2022-07-13 07:42:39 -07:00
Jean-Sébastien Pédron bcb8733880
rabbit_feature_flags: Add a feature flags controller process
This gen_statem-based process is responsible for handling concurrency
when feature flags are enabled and synchronized when a cluster is
expanded.

This clarifies and stabilizes the behavior of the feature flag subsystem
w.r.t. situations where e.g. a feature flag migration function takes
time to update data and a new node joins a cluster and synchronizes its
feature flag states with the cluster. There was a chance that the
feature flag was marked as enabled on the joining node, even though the
migration function didn't take care of that node.

With this new feature flags controller, enabling or synchronizing
feature flags blocks and delays any concurrent operations which try to
modify feature flags states too.

This change also clarifies where and when the migration function is
called: it is called at least once on each node who knows the feature
flag and when the state goes from "disabled" to "enabled" on that node.

Note that even if the feature flag is being enabled on a subset of the
nodes (because other nodes already have it enabled), it is marked as
"state_changing" everywhere during the migration. This is to prevent
that a node where it is enabled assumes it is enabled on all nodes who
know the feature flag.

There is a new feature as well: just after a feature flag is enabled,
the migration function is called a second time for any post-enable
actions. The feature flag is marked as enabled between these "enable"
and "post-enable" steps. The success or failure of this "post-enable"
run does not affect the state of the feature flag (i.e. it is ignored).

A new migration function API is introduced to allow more advanced
things. The new API is:

    my_migration_function(
      #ffcommand{name = ...,
                 props = ...,
		 command = enable | post_enable,
		 extra = #{...}})

The record is defined in `include/feature_flags.hrl`. Here is the
meaning of each field:

* `name` and `props` are the equivalent of the `FeatureName` and
  `FeatureProps` arguments of the previous migration function API.

* `command` is basically the same as the previous `Arg` arguments.

* `extra` is map containing context-specific information. For instance, it
  contains the list of nodes where the feature flag state changes.

This whole new behavior is behind a new feature flag called
`feature_flags_v2`. If a feature flag uses the new migration function
API, `feature_flags_v2` will be automatically enabled.

If many feature flags are enabled at once (like when a fresh RabbitMQ
node is started for the first time), `feature_flags_v2` will be enabled
first if it is in the list.
2022-06-28 10:13:19 +02:00
Michael Klishin b8718c736e
Merge pull request #5020 from rabbitmq/stream-max-segment-size-limit
Limit stream max segment size to 3 GB
2022-06-10 18:53:15 +04:00
Michael Klishin aa8b0dc778
Merge pull request #4985 from rabbitmq/rabbitmq-server-4981
Fix cluster links statistic
2022-06-10 13:50:03 +04:00
Arnaud Cogoluègnes e44b65957d
Limit stream max segment size to 3 GB
Values too large can overflow the stream position field
in the index (32 bit int).
2022-06-10 11:45:57 +02:00
Philip Kuryloski 15a79466b1 Use the new xref2 macro from rules_erlang
That adopts the modern erlang.mk xref behaviour
2022-06-09 23:18:28 +02:00
Luke Bakken 86a509df80
Fix cluster links statistic
Use the `sys_dist` ets table to get distribution port information.

Fixes #4981

Get cluster links stats for TLS dist

Use code from prometheus.erl to get dist links info
2022-06-09 07:20:45 -07:00
Philip Kuryloski 327f075d57 Make rabbitmq-server work with rules_erlang 3
Also rework elixir dependency handling, so we no longer rely on mix to
fetch the rabbitmq_cli deps

Also:

- Specify ra version with a commit rather than a branch
- Fixup compilation options for erlang 23
- Add missing ra reference in MODULE.bazel
- Add missing flag in oci.yaml
- Reduce bazel rbe jobs to try to save memory
- Use bazel built erlang for erlang git master tests
- Use the same cache for all the workflows but windows
- Avoid using `mix local.hex --force` in elixir rules
  - Fetching seems blocked in CI, and this should reduce hex api usage in
    all builds, which is always nice
- Remove xref and dialyze tags since rules_erlang 3 includes them in
  the defaults
2022-06-08 14:04:53 +02:00
Luke Bakken b172bca19b
Add rabbit_consult module
This will be used to fix rabbitmq/osiris#78

If a RabbitMQ `advanced.config` file contains the following:

```
{customize_hostname_check, [
    {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
]}
```

...`file:consult/1` will fail because it does not evaluate terms in the
file.

The code in `rabbit_consult` was copied from this OTP module:

https://github.com/erlang/otp/blob/master/lib/ssl/src/ssl_dist_sup.erl

...and then modified for our use.

Add Bazel suite

Use the same license as Erlang/OTP, add link to source cc @dumbbell

Add test and ensure value returned matches file:consult/1

Add test data file

Ensure that Funs are converted to binaries before jsx:encode is called

Add a check that customize_hostname_check can be JSON encoded

Ensure that customize_hostname_check and match_fun are filtered out from listener data
2022-06-05 06:13:49 -07:00
Michael Klishin 5777dc9d44
Merge pull request #4955 from rabbitmq/switch-from-python-to-python3 2022-06-02 13:37:15 +04:00
Philip Kuryloski b24ab46a1a Switch the default python binary for codegen scripts to python3 2022-06-02 10:58:42 +02:00
Loïc Hoguin dc70cbf281
Update Erlang.mk and switch to new xref code 2022-05-31 13:51:12 +02:00
David Ansari 2d14403dad Reduce expiry limit from 100 to 10 years 2022-05-18 12:11:57 +00:00
David Ansari de4eeb678e Set maximum expiration
When applications accidentally set an unreasonable high value for
the message TTL expiration field, e.g. 6779303336614035452,
before this commit quorum queue and classic queue processes crashed:

```
2022-05-17 13:35:26.488670+00:00 [notice] <0.1000.0> queue 'test' in vhost '/': candidate -> leader in term: 2 machine version: 2
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>   crasher:
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>     initial call: ra_server_proc:init/1
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>     pid: <0.1000.0>
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>     registered_name: '%2F_test'
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>     exception error: bad argument
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>       in function  erlang:start_timer/4
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>          called as erlang:start_timer(6779303336614035351,<0.1000.0>,
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>                                       {timeout,expire_msgs},
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>                                       [])
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>          *** argument 1: exceeds the maximum supported time value
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>       in call from gen_statem:loop_timeouts_start/16 (gen_statem.erl, line 2108)
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>     ancestors: [<0.999.0>,ra_server_sup_sup,<0.250.0>,ra_systems_sup,ra_sup,
2022-05-17 13:35:26.489492+00:00 [error] <0.1000.0>                   <0.186.0>]
```

In this commit, we disallow expiry fields higher than 100 years.
This causes the channel to be closed which is better than crashing the
queue process.

This new validation applies to message TTLs and queue expiry.

From the docs of erlang:start_timer:
"The absolute point in time, the timer is set to expire on, must be in the interval
[erlang:convert_time_unit(erlang:system_info(start_time), native, millisecond),
 erlang:convert_time_unit(erlang:system_info(end_time), native, millisecond)].
If a relative time is specified, the Time value is not allowed to be negative.

end_time:
The last Erlang monotonic time in native time unit that can be represented
internally in the current Erlang runtime system instance.
The time between the start time and the end time is at least a quarter of a millennium."
2022-05-18 11:01:17 +00:00
Michael Klishin 949f7e37db
Bump credentials_obfuscation to 3.1.0 2022-05-16 23:54:32 +04:00
Michael Klishin 5cf3a43523
Revert "Make master depend on credentials_obfuscation main branch"
This reverts commit 62819f3bcc.

This commit has served its QA purpose.

Will either update Bazel dependency to use git or pin to
a new release instead.
2022-05-14 02:58:07 +04:00
Michael Klishin 62819f3bcc
Make master depend on credentials_obfuscation main branch 2022-05-14 01:42:09 +04:00
David Ansari 84ff7e6dea Increase routing throughput for direct exchange 2022-05-11 15:25:15 +00:00
Philip Kuryloski 2e62cae69e Use rabbitmqctl await_startup instead if erl eval
when starting background nodes for tests
2022-05-04 13:33:03 +02:00
Michael Klishin a66ac8c347
Bump credentials_obfuscation to 3.0.0 2022-05-02 11:57:55 +04:00
Michael Klishin 1d9ef20adb
Drop -Wrace_conditions from Dialyzer opts in more places
for Erlang 25 compatibility
2022-04-30 01:49:09 +04:00
Michael Klishin 7cb439cda6
Merge pull request #4665 from rabbitmq/rabbitmq-server-4588-follow-up
OAuth 2: expand all scope aliases provided
2022-04-27 22:04:32 +04:00
Michael Klishin ca290f1116
OAuth 2: expand all scope aliases provided
Per discussion with @MarcialRosales.

In follow-up to #4588.
2022-04-27 21:21:40 +04:00
Philip Kuryloski a4da3bb332 Add a MODULE.bazel file and enable bzlmod 2022-04-27 16:07:52 +02:00
Michael Klishin f69fe93136
Correct a double quote introduced in #4603 2022-04-20 16:06:36 +04:00
Michael Klishin 7c47d0925a
Revert "Correct a double quote introduced in #4603"
This reverts commit 6a44e0e2ef.

That wiped a lot of files unintentionally
2022-04-20 16:05:56 +04:00
Michael Klishin 6a44e0e2ef
Correct a double quote introduced in #4603 2022-04-20 16:01:29 +04:00
Michael Klishin cfb1d44920
Correct GitHub repo links in Hex metadata 2022-04-20 11:14:12 +04:00
Luke Bakken fb5068a80e
Begin work migrating to native hex publishing
Use hex for credentials_obfuscation dep
2022-04-19 13:21:52 -07:00
Luke Bakken dba25f6462
Replace files with symlinks
This prevents duplicated and out-of-date instructions.
2022-04-15 06:04:29 -07:00
Philip Kuryloski 2dd9bde891 Bring over PROJECT_APP_EXTRA_KEYS values from make to bazel 2022-04-07 17:39:33 +02:00
Philip Kuryloski a22234f6eb Updates for rules_erlang 2.5.0
rabbitmq_cli uses some private rules_erlang apis that have changed in
the upcoming release

Additionally:
- Avoid including both standard and test versions of amqp_client in
integration test suites
- Eliminate most of the compilation order hints (explicit first_srcs)
in the bazel build
- Fix an include statement - in bazel, an app is not available to
itself as a library at compilation time
2022-04-07 14:54:37 +02:00
Loïc Hoguin 499e0b9197
Remove the CQv1 disabled stats from management/Prometheus 2022-04-05 12:37:54 +02:00
Loïc Hoguin 81c0fa7cc1
CQv1: Disable expensive stats killing v1 performance
In particular:

 - io_file_handle_open_attempt
 - queue_index_journal_write

Neither have proven to be very useful in recent years
and with the move to FHC-less and journal-less v2 index
they will slowly become irrelevant. This should be a
good compromise until we can switch to v2 permanently
or rework the stats module to use counters.
2022-04-05 12:37:49 +02:00
Luke Bakken 9799fc6674
Add necessary items for hex-publish to work for amqp10_client 2022-03-29 10:25:45 -07:00
Michael Klishin c5c730c371
Merge pull request #4324 from felixhuettner/fhcs_write_concurrency
Enable write_concurrency for file handle cache stats
2022-03-22 12:20:08 +04:00
Michael Klishin c38a3d697d
Bump (c) year 2022-03-21 01:21:56 +04:00
Felix Huettner fab514f643 Enable write_concurrency for file handle cache stats
During most of the time the file_handle_cache_stats ets table is
used for writing only.
By enabeling `write_concurrency` on the table we allow different values
to be written concurrently without taking a global lock.

There the only codepath reading from the ets table is run on the
`collect_statistics_interval` interval and reads the whole table.
So we can assume we are not blocking any large amount of concurrent reads.
2022-03-14 14:58:36 +01:00
Karl Nilsson 4b193608bb
Merge pull request #3121 from rabbitmq/quorum-queues-v2
QQ: introduce new machine version (2)
2022-03-08 16:19:04 +00:00
Michael Klishin 8ff862f0f6
Merge pull request #4233 from ansd/nxdomain
Wait between retries if nxdomain
2022-03-04 22:05:41 +06:00
David Ansari 1c17773c91 Add at-least once dead-lettering for quorum queues
and message TTL
2022-02-28 16:27:56 +01:00
Philip Kuryloski d8201726ae Ignore dialyzer warnings for most apps 2022-02-21 09:19:56 +01:00
Michael Klishin 127f39a090
Squash a compilation warning 2022-01-28 23:01:19 +03:00
Michael Klishin 057e033431
Introduce definition hashing during import
as an opt-in feature. The goal is to avoid re-importing the definition
from the definition file/directory/source if we know the content
has not changed. Since this feature won't be appropriate for
every environment (sometimes unconditional reimporting is expected),
the feature is opt-in.

This is still a WIP.
2022-01-28 22:52:00 +03:00
Philip Kuryloski efcd881658 Use rules_erlang v2
bazel-erlang has been renamed rules_erlang. v2 is a substantial
refactor that brings Windows support. While this alone isn't enough to
run all rabbitmq-server suites on windows, one can at least now start
the broker (bazel run broker) and run the tests that do not start a
background broker process
2022-01-18 13:43:46 +01:00
Luke Bakken 95a60fc3be
Replace one use of filelib:is_regular/1
This specific case is called multiple times by the Prometheus plugin. It eventually calls `file:read_file_info/1` which leaks on Windows

See #3936
2022-01-11 09:08:34 -08:00
Michael Klishin d2e0fae1aa
Bump (c) year in node startup banner 2022-01-05 09:03:46 +04:00
Luke Bakken 7f0285834e
Fix all uses of file:read_file/1
This is to address another memory leak on win32 reported here:

https://groups.google.com/g/rabbitmq-users/c/UE-wxXerJl8

"RabbitMQ constant memory increase (binary_alloc) in idle state"

The root cause is the Prometheus plugin making repeated calls to `rabbit_misc:otp_version/0` which then calls `file:read_file/1` and leaks memory on win32.

See https://github.com/erlang/otp/issues/5527 for the report to the Erlang team.

Turn `badmatch` into actual error
2022-01-03 11:33:36 -08:00
Michael Klishin 394ee4880a
delegate: documentation edits 2021-12-26 03:19:33 +03:00
tomyouyou 40db6563b8
Optimisation for 'delegate'
This is copied from https://github.com/rabbitmq/rabbitmq-common/pull/349


If a message is sent to only one queue(in most application scenarios), passing through the 'delegate' is meaningless. Otherwise, it increases the delay of the message and the possibility of 'delegate' congestion.

Here are some test data:
node1: Pentium(R) Dual-Core CPU E5300 @ 2.60GHz
node2: Pentium(R) Dual-Core CPU E5300 @ 2.60GHz

Join node1 and node2 to a cluster. Create 100 queues on node2, and start 100 consumers to receive messages from these queues.
Start 100 publishers on node1 to send messages to the queues of node2. Each publisher will send 10k messages at the rate of 100/s(10k/s theoretically in total), and all the messages for all publishers is 1 million.

Before optimisation:
{1,[{msg_time,812312(=<1ms),177922(=<5ms),9507(=<50ms),221(=<500ms),38(=<1000ms),0,0,0,0,1061,1069,0,0}]}

After optimisation:
{1,[{msg_time,902854(=< 1ms),93993(=<5ms),3038(=<50ms),96(=<500ms),19(=<1000ms),0,0,0,0,1049,1060,0,0}]}

Additional information:

Time counted here is the stay time of a message in the cluster, that is, Time(leaving from node2 at) - Time(reaching node1 at).
"812312(=<1ms)" is the number of messages with time consumption less than or equal to 1ms.
Overall, the optimisation is effective.
2021-12-07 17:58:42 +08:00
Sagi Sarussi eff5c063cb
Grammar 2021-11-10 15:31:06 +02:00
Arnaud Cogoluègnes 75b2a53f52
Merge pull request #3503 from rabbitmq/super-stream-cli
Add functions to create/delete super stream in manager
2021-10-14 11:01:58 +02:00
Alexey Lebedeff 23be565a38 Allow having multiple `make run-broker` simulatneously
It was automatically happening for e.g. `make start-cluster`.

But some plugins were not covered by default generated config, and
running rabbit from 2 different worktrees was a bit complicated.
2021-10-13 10:52:02 +02:00
Arnaud Cogoluègnes 6b9589bae4
Handle stream arguments in add_super_stream command
max-age, leader-locator, etc.
2021-10-11 16:50:03 +02:00
Michael Klishin 798be7dcaf
Peer discovery AWS, K8S: more Dialyzer fixes 2021-10-07 03:42:44 +03:00
Michal Kuratczyk 1a2126676d Update CLA URL 2021-09-23 14:29:36 +02:00
Michael Klishin 5fb118e8ef
Merge pull request #3409 from rabbitmq/lh-increase-queue-shutdown-timeout
Increase classic queue shutdown timeout
2021-09-21 16:46:21 +03:00
Philip Kuryloski bcd565f2cb Use the +deterministic compiler flag when building with Make 2021-09-17 09:34:50 +02:00
Loïc Hoguin 09c8cd4f98
Increase classic queue shutdown timeout
A value that is too low will prevent the index from shutting
down in time when there are many queues. This leads to the
process being killed and on the next RabbitMQ restart a
(potentially very long) dirty recovery is needed.

The value of 10 minutes was chosen to mirror the shutdown
timeout of the message store. Since both queues and message
store need to have shut down gracefully in order to have
a clean restart it makes sense to use the same value.

Related: c40c2628a9
2021-09-13 10:59:30 +02:00
Philip Kuryloski 5fd9d1f638 Use Ranch 2.1.0 in bazel build
Matches 063d32626d
2021-09-10 14:46:36 +02:00
Loïc Hoguin 0a2bde4ab1
Use the new function ranch_proxy_header:to_connection_info/1 2021-09-09 13:15:08 +02:00
Paweł Chmielowski 6af41527f1 Fix some edge cases in handling ssl information from PROXY protocol
When we fail to parse name of cipher suite from PROXY protocol
just say that no ssl is used, instead of trying to fill that
with data from connection between proxy and our server.
2021-09-06 19:36:42 +02:00
Jean-Sébastien Pédron 689c56cb04
Logging: Add `single_line` flag support to $RABBITMQ_LOG
A user could already enable single-line logging (the `single_line`
option of `logger_formatter` or RabbitMQ internal formatters) from the
configuration file. For example:

    log.console.formatter.single_line = on

With this patch, the option can be enabled from the `$RABBITMQ_LOG`
environment variable as well:

    make run-broker RABBITMQ_LOG=+single_line
2021-09-01 09:31:54 +02:00
Jean-Sébastien Pédron 33c6cbf239
rabbit_env: Add $RABBITMQ_DEFAULT_{USER,PASS,VHOST} and $RABBITMQ_ERLANG_COOKIE vars
Those environment variables are unset by default. The default values are
set in the `rabbit` application environment and can be configured in the
configuration file. However, the environment variables will take
precedence over them respectively if they are set.
2021-08-11 15:50:36 +02:00
Alexey Lebedeff 87532c4af4 Make test/development related makefile vars lazy
They were trying to run `hostname` and `which`, which produced a bunch
of error messages in a hermetic build environment.

And performance of those `shell` calls is not very important, as they
are caled just a few times during script runtime anyway (there is a
hack to make these lazy, but evaluating only once - but it's hardly
worth it).
2021-08-06 17:32:04 +02:00
Michael Klishin c84115f102
Warn when lock could not be acquired successfully 2021-08-05 11:31:35 +03:00
Michael Klishin 22add3c8ca
Acquire locks when starting mirrored supervisor children
Unlike pg2, pg in Erlang 24 is eventually consistent. So this
reintroduces some of the same kind of locking mirrored_supervisor
used to rely on implicitly via pg2.

Per discussion with @lhoguin.

Closes #3260.

References #3132, #3154.
2021-08-05 00:39:16 +03:00
Michael Klishin 9a0f4b17cd
More mirrored supervisor logging 2021-08-04 16:58:44 +03:00
Michael Klishin 4f3d5e5e13
Mirrored supervisor: more logging at debug level 2021-08-04 15:56:45 +03:00
Philip Kuryloski c9e016d545 Remove duplicate compilation options in rabbit_common 2021-08-03 14:31:05 +02:00
Michael Klishin 0617419e75
This *is* a Unicode character, U+060E 2021-07-28 19:04:16 +03:00
Ilya Khaprov 854f26ac18
Do not crash on badarg when env file has an 'unusual' unicode character
closes #3212
2021-07-28 16:39:17 +02:00
Jean-Sébastien Pédron ac7df333b6
Revert "rabbitmq-dist.mk: Install CLI scripts as part of the build"
This has the unfortunate side effect of causing a rebuild of all
applications every time. I need to figure out another place to build and
install the CLI during build time (instead of as part of the dist
target).

This reverts commit 4322cca66e.
2021-07-23 10:46:56 +02:00
Philip Kuryloski ae955d44c6 Avoid building the cli scripts in rabbitmq_stream_common
This isn't an issue in the monorepo, but causes problems externally
projects using rabbitmq-components.mk
2021-07-23 10:40:20 +02:00
Jean-Sébastien Pédron 80b428d5d3
rabbit_runtime: Find erl(1) in the ERTS bin directory
erl(1) might not be in the directory pointed by init's root argument.
This is the case with Erlang releases built with relx for instance.
2021-07-22 10:21:37 +02:00
Jean-Sébastien Pédron 4322cca66e
rabbitmq-dist.mk: Install CLI scripts as part of the build
... instead of the `dist` target. This way, the Erlang release can use
them.
2021-07-22 10:21:33 +02:00
Michael Klishin 29bb9c5b0c
Merge pull request #3175 from processone/proxy_protocol_tls_info
Extract TLS informations that are delivered in PROXY protocol frame
2021-07-13 15:08:40 +03:00
Philip Kuryloski 8f9de08de7 Also assert no missing suites for all other deps 2021-07-12 18:05:55 +02:00
Paweł Chmielowski d5daf7598b Extract TLS informations that are delivered in PROXY protocol frame 2021-07-05 13:29:59 +02:00
Michael Klishin 65ccf7ce42
Mirrored supervisor: make it easier to keep track of group membership changes
in the logs.

Referenes #3148.
2021-07-05 14:02:37 +03:00
Michael Klishin 4f01d450e0
Make sure not to wait for epmd-starter subprocess forever
Per discussion in #3162
2021-06-30 15:20:09 +03:00
Michael Klishin 62e7785a48
Canonicalize a few more paths in the startup banner
References #3149
2021-06-28 13:48:13 +03:00
Pavel Heimlich 020b22f9ea fix hostname(1) calls on Solaris 2021-06-17 13:23:05 +02:00
Philip Kuryloski f9d114ad75 dialyze rabbit 2021-05-28 13:27:21 +02:00
Philip Kuryloski fe2e2cfe78 Use dialyzer on rabbit_common
Only dialyze errors will cause a build failure, similar to Erlang.mk
2021-05-28 13:03:55 +02:00
Michael Klishin 6acee761e0
Handle cases where virtual host config file does not yet exist
This happens during queue migration from a pre-3.7 version.

References #2954.
2021-05-27 17:38:53 +03:00
Philip Kuryloski e02132ee0f Fix rabbitmq-components.mk path when publishing to hex pm 2021-05-25 10:29:51 +02:00
Philip Kuryloski 98e71c45d8 Perform xref checks on many tier-1 plugins 2021-05-21 12:03:22 +02:00
Arnaud Cogoluègnes c42930acb0
Set stream plugin default port to 5552 2021-05-19 15:38:52 +02:00
Philip Kuryloski 9053d96ca0 Fix call to deprecated function
https://erlang.org/doc/general_info/deprecations.html#otp-22
2021-05-19 13:38:17 +02:00
Michael Klishin fc7501c309
Convert policy event keys to atoms
so that they are easier to match on
2021-04-28 12:57:06 +03:00
Michael Klishin 60a9a89940
Allow for more ASN.1 string (sub)types 2021-04-20 01:19:38 +03:00
Michael Klishin 73eb9b56ec
Decode other name using 'OTP-PUB-KEY':decode/2
and assume it is a string-like value ("directory string")
because other values would not make much sense in the
username extraction context.

References #2983.
2021-04-20 01:16:29 +03:00
Michael Klishin e15f2b46f3
SAN of type otherName: strip two leading characters
instead of specific ones since they will vary with the payload
(one of them likely indicates UTF string length).

This is still not perfect because we limit the maximum
allowed length but it works fine with identifiers up to 100
characters long, which should be good enough for this
best effort handling of an abscure SAN type.

References ##2983.
2021-04-20 00:20:44 +03:00
Michael Klishin 1587501733
Pass Dialyzer
x.509 certificate extensions do not really have types in
OTP's public_key, so our hands are tied.
2021-04-19 13:56:39 +03:00
Michael Klishin 81f3005b95
rabbit_common: pass Dialyzer 2021-04-19 13:41:29 +03:00
Jean-Sébastien Pédron 6936f2d8a9 rabbit_env: Fix Bourne shell concatenated literal parsing
The parser didn't handle literals of the form:
    'single-quoted'unquoted'single-quoted-again'"or-even-double-quoted"

In particular, the unquoted parsing assumed that nothing else could
follow it. The testsuite is extended with the issue reporter's case.

While here, improve escaped characters handling. They used to be not
parsed specifically at all.

Fixes #2969.
2021-04-19 12:32:53 +02:00
Michael Klishin 21a6b1ba05
x.509 certificate auth mechanism: support (some) SANS of type otherName
Note that the type by definition contains arbitrary values. According
to the OTP types, they are triplets that represent effectively
a key/value pair. So we assume the pair is a string that needs a bit
massaging, namely stripping the UTF encoding prefix OTP AnotherName
decoder leaves in.

Kudos to @Thibi2000 for providing an example value.

Closes #2983.
2021-04-19 02:33:26 +03:00
Jean-Sébastien Pédron 3d1db4b676 rabbit_env: Fix arguments to the ?LOG_DEBUG format string
Format string arguments must be in a list.
2021-04-13 14:20:19 +02:00
Michael Klishin 7f98bc3d1c
Add more VM memory monitor tests, pass Dialyzer
(cherry picked from commit 57ec1f8768)
2021-04-11 11:36:30 +03:00
Michael Klishin 19b104a22f
Update vm_memory_high_watermark/0 type 2021-04-11 10:43:38 +03:00
Michael Klishin 30cbbba167
High VM watermark: support {relative, N} values set via advanced.config
for usability. It is not any different from when a float value
is used and only exists as a counterpart to '{absolute, N}'.

Also nothing changes for rabbitmq.conf users as that format performs
validation and correct value translation.

See #2694, #2965 for background.
2021-04-11 10:28:35 +03:00
kjnilsson b576242952 Increase rabbit_stream_queue_SUITE timetrap
And set the default of make start-cluster to 3 nodes.
2021-04-06 15:50:22 +01:00
Carl Hörberg aac99700e7 get destination address from PROXY protocol 2021-03-30 12:58:49 +02:00
Philip Kuryloski 768a87980d Remove extra spaces in the RABBITMQ_PLUGINS_DIR during tests 2021-03-29 16:59:53 +02:00
Philip Kuryloski 0e7b537af7 Fix self-referencing recursive make variable
Introduced in 388654c542
2021-03-29 14:18:17 +02:00
Philip Kuryloski 388654c542
Add a partial Bazel build (#2938)
Adds WORKSPACE.bazel, BUILD.bazel & *.bzl files for partial build & test with Bazel. Introduces a build-time dependency on https://github.com/rabbitmq/bazel-erlang
2021-03-29 11:01:43 +02:00
Philip Kuryloski b62c5499ce Reinstate the `prepare-dist` tasks for rabbitmq components
The consolidation of `rabbitmq-components.mk` broke the previous
method by which rabbit components were detected. Now we check
$(RABBITMQ_COMPONENTS) directly.
2021-03-23 17:42:23 +01:00
Philip Kuryloski a63f169fcb Remove duplicate rabbitmq-components.mk and erlang.mk files
Also adjust the references in rabbitmq-components.mk to account for
post monorepo locations
2021-03-22 15:40:19 +01:00
David Ansari dc589856f2 Wait between retries if nxdomain
In kind version 0.10.0, when creating a 5-node RabbitMQ cluster
with the new parallel PodManagementPolicy, we observed that some
pods were restarted. Their logs included:

```
10:10:03.794 [error]
10:10:03.804 [error] BOOT FAILED
10:10:03.805 [error] ===========
BOOT FAILED
10:10:03.805 [error] ERROR: epmd error for host r1-server-0.r1-nodes.rabbitmq-system: nxdomain (non-existing domain)
10:10:03.805 [error]
===========
ERROR: epmd error for host r1-server-0.r1-nodes.rabbitmq-system: nxdomain (non-existing domain)
10:10:04.806 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"r1-server-0.r1-nodes.rabbitmq-system",nxdomain} in context start_error
10:10:04.806 [error] CRASH REPORT Process <0.152.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"r1-server-0.r1-nodes.rabbitmq-system",nxdomain}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 138
```

Eventually, after some pods restarted up to 2 times, all pods were running and ready.

In kind, we observed that during the first couple of seconds, nslookup was failing as well for that domain
with nxdomain.
It took up to 30 seconds until nslookup succeeded.

With this commit, pods don't need to be restarted when creating a fresh
RabbitMQ cluster.
2021-03-18 18:21:02 +01:00
kjnilsson 52f745dcde Update rabbitmq-components.mk
use v1.x branch of ra
2021-03-18 15:14:40 +00:00
Loïc Hoguin d5e3bdd623
Add ADDITIONAL_PLUGINS variable
This allows including additional applications or third party
plugins when creating a release, running the broker locally,
or just building from the top-level Makefile.

To include Looking Glass in a release, for example:

$ make package-generic-unix ADDITIONAL_PLUGINS="looking_glass"

A Docker image can then be built using this release and will
contain Looking Glass:

$ make docker-image

Beware macOS users! Applications such as Looking Glass include
NIFs. NIFs must be compiled in the right environment. If you
are building a Docker image then make sure to build the NIF
on Linux! In the two steps above, this corresponds to Step 1.

To run the broker with Looking Glass available:

$ make run-broker ADDITIONAL_PLUGINS="looking_glass"

This commit also moves Looking Glass dependency information
into rabbitmq-components.mk so it is available at all times.
2021-03-12 12:29:28 +01:00
Michael Klishin 97ff62d3b2
Drop trailing newlines from logged messages where possible
Lager strips trailing newline characters but OTP logger with the default
formatter adds a newline at the end. To avoid unintentional multi-line log
messages we have to revisit most messages logged.

Some log entries are intentionally multiline, others
are printed to stdout directly: newlines are required there
for sensible formatting.
2021-03-11 15:17:37 +01:00
Jean-Sébastien Pédron cdcf602749
Switch from Lager to the new Erlang Logger API for logging
The configuration remains the same for the end-user. The only exception
is the log root directory: it is now set through the `log_root`
application env. variable in `rabbit`. People using the Cuttlefish-based
configuration file are not affected by this exception.

The main change is how the logging facility is configured. It now
happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is
removed.

The supported outputs remain the same: the console, text files, the
`amq.rabbitmq.log` exchange and syslog.

The message text format slightly changed: the timestamp is more precise
(now to the microsecond) and the level can be abbreviated to always be
4-character long to align all messages and improve readability. Here is
an example:

    2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE ==
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com

The example above also shows that multiline messages are supported and
each line is prepended with the same prefix (the timestamp, the level
and the Erlang process PID).

JSON is also supported as a message format and now for any outputs.
Indeed, it is possible to use it with e.g. syslog or the exchange. Here
is an example of a JSON-formatted message sent to syslog:

    Mar  3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}}

For quick testing, the values accepted by the `$RABBITMQ_LOGS`
environment variables were extended:
  * `-` still means stdout
  * `-stderr` means stderr
  * `syslog:` means syslog on localhost
  * `exchange:` means logging to `amq.rabbitmq.log`

`$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in
addition to the existing `+color` one). With that modifier, messages are
formatted as JSON intead of plain text.

The `rabbitmqctl rotate_logs` command is deprecated. The reason is
Logger does not expose a function to force log rotation. However, it
will detect when a file was rotated by an external tool.

From a developer point of view, the old `rabbit_log*` API remains
supported, though it is now deprecated. It is implemented as regular
modules: there is no `parse_transform` involved anymore.

In the code, it is recommended to use the new Logger macros. For
instance, `?LOG_INFO(Format, Args)`. If possible, messages should be
augmented with some metadata. For instance (note the map after the
message):

    ?LOG_NOTICE("Logging: switching to configured handler(s); following "
                "messages may not be visible in this log output",
                #{domain => ?RMQLOG_DOMAIN_PRELAUNCH}),

Domains in Erlang Logger parlance are the way to categorize messages.
Some predefined domains, matching previous categories, are currently
defined in `rabbit_common/include/logging.hrl` or headers in the
relevant plugins for plugin-specific categories.

At this point, very few messages have been converted from the old
`rabbit_log*` API to the new macros. It can be done gradually when
working on a particular module or logging.

The Erlang builtin console/file handler, `logger_std_h`, has been forked
because it lacks date-based file rotation. The configuration of
date-based rotation is identical to Lager. Once the dust has settled for
this feature, the goal is to submit it upstream for inclusion in Erlang.
The forked module is calld `rabbit_logger_std_h` and is based
`logger_std_h` in Erlang 23.0.
2021-03-11 15:17:36 +01:00
Gerhard Lazu 5ac24dd2be
Make start-cluster work when run in the top-level
Subsequent nodes fail to start since ports are already in use. This
makes it possible to start multiple nodes locally with all plugins
enabled.

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-03-10 13:15:44 +00:00
dcorbacho 61f7b2a723 Update to ranch 2.0 2021-03-08 23:11:05 +01:00
Michael Klishin fd86959429
Revert "Make rabbit_log:log/2 return 'ok' unconditionally"
This reverts commit 01c4ca2aa9.
2021-03-04 13:13:00 +03:00
Michael Klishin 01c4ca2aa9
Make rabbit_log:log/2 return 'ok' unconditionally
instead of relying on a Lager transform-driven return value.

This should avoid erlang/otp#4576 on Erlang 24.

Per discussion with @lhoguin.
2021-03-04 13:09:29 +03:00
Michael Klishin 8920f05a15
Revert "Squash one more Erlang 24 warning"
This reverts commit fb4f88e7dd.

See erlang/otp#4576
2021-03-04 06:10:23 +03:00
Michael Klishin fb4f88e7dd
Squash one more Erlang 24 warning 2021-03-04 05:38:38 +03:00
Michael Klishin 3a169cc9df
Drive-by: squash an Erlang 24-specific warning 2021-03-04 04:41:20 +03:00
Michael Klishin b6c4831e75
Bump Lager to 3.9.1 2021-03-04 04:36:39 +03:00
Michael Klishin ad47eba2fa
pg2 => pg for OTP 24 compatibility
there is still one failing queue federation test.
2021-03-04 04:06:40 +03:00
Loïc Hoguin 66ac1bf5e9
Bump observer_cli to 1.6.1
More responsive when the system is overloaded with file calls.
2021-03-01 21:55:27 +03:00
Arnaud Cogoluègnes f53c5dc998
Use port 5551 for streams when starting cluster 2021-03-01 21:55:24 +03:00
Michael Klishin 8fe3df9343
Upgrade Lager to 3.9.0 for OTP 24 compatibility
`lager_util:expand_path/1` use changes are
due to erlang-lager/lager#540
2021-02-26 00:52:15 +03:00
Jean-Sébastien Pédron 561d2e660d
rabbit_env: Use consistent style 2021-02-25 11:13:23 +01:00
Jean-Sébastien Pédron a45e87e57b
mk/rabbitmq-run.mk: Compute rabbitmq_stream TCP port based on $(RABBITMQ_NODE_PORT)
We already do this for other TCP ports RabbitMQ listens on. This fixes
the use of `gmake start-cluster` with the rabbitmq_stream plugin
enabled.
2021-02-25 10:47:50 +01:00
Michael Klishin 98b46b2ffb
rabbit_env: pass Dialyzer again 2021-02-25 06:22:36 +03:00
Michael Klishin 7e93cc8d3b
rabbit_env: remove one more direct call to os:list_env_vars/0 2021-02-25 06:13:41 +03:00
Michael Klishin 26b676f021
Try ignore_xref 2021-02-25 06:11:18 +03:00
Michael Klishin cdd0c1662c
Pass xref on OTP 22 and 23 2021-02-25 06:08:16 +03:00
Michael Klishin 752f16abf7
os:list_env_vars/0 was renamed to os:env/0 in OTP 24 2021-02-25 01:10:40 +03:00
Michael Klishin 00b7a84191
Limit direct reply-to identifier length growth
as node names grow.

Prior to this change, direct reply-to consumer channels
were encoded using term_to_binary/1, which means the result
would grow together with node name (since node name
is one of the components of an Erlang pid type).

This means that with long enough hostnames, reply-to
identifiers could overflow the 255 character limit of
message property field type, longstr.

With this change, the encoded value uses a hash of the node name
and then locates the actual node name from a map of
hashes to current cluster members.

In addition, instead of generating non-predictable "secure"
GUIDs the feature now generates "regular" predictable GUIDs
which compensates some of the additional PID pre- and post-processing
outlined above.
2021-02-24 18:21:26 +03:00
Michael Klishin f73e851f9c
Bump observer_cli to 1.6.0 2021-02-24 12:53:55 +03:00
Michael Klishin a5098b28a7
Bump Lager to 3.8.2 for OTP 24 compatibility 2021-02-24 12:53:30 +03:00
Michael Klishin 88bf1ba754
Don't filter out lz4 and looking_glass from dependency list
Now that dependencies are packaged as directories and not .ez
files, the fact that both LG and LZ4 are NIFs is no longer
an issue. And having it as regular dependencies simplifies
REPL-driven profiling.

Per discussion with @dumbbell.
2021-02-16 17:03:36 +03:00
Michael Klishin b7f01c102b
Use an up-to-date Looking Glass repo name 2021-02-16 16:21:49 +03:00
Michael Klishin caddfd3791
Skip aes_ige256 in config value encryption/decryption tests
it is not supported by some libcrypto versions, including CI.

See [1][2].

1. https://github.com/rabbitmq/credentials-obfuscation/pull/10/files
2. https://bugs.erlang.org/browse/ERL-1478
2021-02-07 23:05:27 +03:00
Michael Klishin e5def7f612
Use credentials_obfuscation master in RabbitMQ master 2021-02-07 20:58:07 +03:00
Michael Klishin b11a79cccf
Bump (c) year in header files 2021-02-04 07:04:58 +03:00
Michael Klishin 394d36ab76
Use a record here
accessing record/tuple elements by index is
increases the risk of code breakage when
the record changes.
2021-02-01 20:19:12 +03:00
Arnaud Cogoluègnes b921ac11a8
Merge pull request #2712 from rabbitmq/rabbitmq-stream-prometheus
Add stream prometheus plugin
2021-01-27 16:46:37 +01:00
Michael Klishin 52479099ec
Bump (c) year 2021-01-22 09:00:14 +03:00
Michael Klishin f9e2996b51
Refactor #2733 a bit more 2021-01-21 15:09:20 +03:00
Michael Klishin 9176944c95
Use slightly more idiomatic Erlang 2021-01-21 11:56:30 +03:00
tomyouyou ae173f11bb
'get_memory_use(ratio)' will have an exception when MemoryLimit is 'infinity' 2021-01-21 16:14:56 +08:00
Arnaud Cogoluègnes b5315c0166
Merge branch 'master' into rabbitmq-stream-prometheus 2021-01-18 11:26:06 +01:00
Michael Klishin a087621257
Handle a 'noport' clause here
Follow-up to #2722
2021-01-16 18:56:13 +05:00
Michael Klishin 075ee09996
Naming, wording 2021-01-16 01:45:12 +05:00
Gabriele Santomaggio 19d28f260a Add log 2021-01-15 13:04:24 +01:00
Gabriele Santomaggio 9ec5b4bf86 Add spec 2021-01-15 11:17:43 +01:00
Gabriele Santomaggio ce5550b974 Add retry for epdm name (hostname)
fixes https://github.com/rabbitmq/rabbitmq-server/issues/2718
2021-01-15 09:09:19 +01:00
Arnaud Cogoluègnes bf72683eb2
Add stream prometheus plugin 2021-01-11 16:49:56 +01:00
Michael Klishin ccfd6627eb
Remove Ranch dependency from rabbit_common
It is not actually used there and may complicate dependency resolution
for RabbitMQ Erlang client users.

Per discussion with @dumbbell
2021-01-07 17:53:47 +03:00
Arnaud Cogoluègnes cbd3c8dfdd
Merge branch 'master' into rabbitmq-stream-management 2021-01-04 09:50:47 +01:00
Michael Klishin e8cb671633
Bump credentials_obfuscation version 2020-12-21 03:42:42 +03:00
Arnaud Cogoluègnes 41160210ab
Start stream consumer REST endpoint 2020-12-01 18:23:22 +01:00
Arnaud Cogoluègnes 23d7e8114c
Introduce stream management plugin 2020-11-19 14:48:25 +01:00
Jean-Sébastien Pédron 778e8dad5c
rabbit_common: Remove the rabbitmq-github-actions Erlang.mk plugin
This is unused after the switch to the "monorepository".
2020-11-17 15:29:05 +01:00
Jean-Sébastien Pédron 47686ee1f0
Remove unused .github directories
They were valid until the switch to the "monorepository" when everything
was merged into a single Git repository.
2020-11-17 13:33:16 +01:00
Arnaud Cogoluègnes 722dda6eaf Update rabbitmq-components.mk to include stream plugin 2020-11-03 14:26:45 +01:00
Luke Bakken fd73767051 Revert "Handle potential undefined value returned from process dictionary"
This reverts commit 04627ef2ef.
2020-11-02 10:02:46 -08:00
Michael Klishin e6972966bb Part of rabbitmq/rabbitmq-auth-mechanism-ssl#12 2020-10-30 18:46:09 +03:00
Michael Klishin 0284c222e5 Merge branch 'master' into update-cowboy-dep 2020-10-21 09:44:22 +03:00
Michael Klishin 4dfffd312f Remove a (now) unused delegate 2020-10-20 21:14:42 +03:00
Luke Bakken f0c1d17e3a Remove code to guess thread pool size
Part of rabbitmq/rabbitmq-server#2473
2020-10-20 10:23:54 -07:00
Loïc Hoguin 44be0b1114 Update Cowboy to 2.8.0
This will enable stacktraces in the logs again.
2020-10-20 17:07:45 +02:00
Luke Bakken 04627ef2ef Handle potential undefined value returned from process dictionary
CI failures such as this suggest that file handle values can be deleted
mid-operation:

https://github.com/rabbitmq/rabbitmq-server/actions/runs/304216635
2020-10-15 05:49:16 -07:00
Michael Klishin 45158f49ac Merge pull request #409 from rabbitmq/auth-attempt-metrics
Add auth attempt metrics
2020-10-14 23:56:02 +03:00
Jean-Sébastien Pédron 6d1c7dc6d7 Travis CI: Test against Erlang 22.3+23.0 and Elixir 1.10 2020-10-13 11:39:43 +02:00
Jean-Sébastien Pédron b9defc1bff rabbit_env: Fix parsing of $CONF_ENV_FILE output
The `set` command in the implementation of `/bin/sh` included in the
official RabbitMQ Docker image returns multi-line variable values
differently than the tested Bourne shell implementation (GNU Bash, dash
and FreeBSD sh).

I don't know what implementation is used by that Docker image, but here
is the output of `set`, for a variable set to "\n'test'":

    TEST_VAR='
    '"'"'test'"'"

The problem was reported in the following discussion:
https://github.com/rabbitmq/rabbitmq-server/discussions/2458

While here, add a small testcase to check a couple outputs.
2020-10-07 15:33:49 +02:00
kjnilsson db59327963 Move rabbit_mics:confirm_to_sender/3 to server
As it is only used by the server and does not need to be a shared
function.
2020-10-07 13:05:07 +01:00
Nicolas Caille 6f3681210e remove TLSv1.3 from BAD_SSL_PROTOCOL_VERSIONS 2020-10-06 21:27:53 +02:00
Jean-Sébastien Pédron 3a65b3c624 rabbitmq-github-actions.mk: Use Elixir 1.10.4
This is the minimum requirement now, because that's the oldest version
we test against.
2020-10-05 15:46:07 +02:00
Jean-Sébastien Pédron 8bd3bede55 rabbitmq-github-actions.mk: Test with Erlang 23.1
... instead of 23.0.

Erlang 23.1 is the version the Concourse pipelines use. We expect the
Concourse pipelines and the GitHub Actions workflow to be on the same
page.
2020-10-05 15:44:49 +02:00
Jean-Sébastien Pédron e5127eaea3 rabbit_log: Restore alphabetical order in the defintion of category() 2020-10-05 14:21:52 +02:00
kjnilsson e0a210cc67 Dialyzer fixes 2020-10-02 10:22:10 +01:00
kjnilsson b87898185b Take stream queue dir into account
For rabbit_env suite.
2020-09-30 14:41:59 +01:00
Arnaud Cogoluègnes 749faa85fa Add stream plugin port in start-cluster target 2020-09-30 14:41:59 +01:00
kjnilsson 28b6b812b1 Add osiris logger sink 2020-09-30 14:41:59 +01:00
kjnilsson bbd0cc8198 Add osiris logging shim 2020-09-30 14:41:59 +01:00
kjnilsson f7dcd081fa change return message format 2020-09-30 14:41:59 +01:00
dcorbacho 0cec212896 Stream queue data dir 2020-09-30 14:41:59 +01:00
Michael Klishin 5a215e0f03 Merge branch 'master' into lrb-update-supervisor 2020-09-30 11:22:15 +03:00
Michael Klishin 8f7ecf363f Migrate a supervisor2 suite by @lukebakken from rabbitmq-server
References rabbitmq/rabbitmq-server#2457.
2020-09-30 11:00:32 +03:00
Luke Bakken 39ec461331 Fix transient type handling 2020-09-29 13:49:34 -07:00
Luke Bakken a0bd637558 Fix function ordering to restart correctly. Reduce log spam. 2020-09-29 11:52:35 -07:00
Luke Bakken 6ed6573cef Update mix_task_archive_deps version
https://github.com/rabbitmq/mix_task_archive_deps/releases/tag/0.5.0

See this as well:

https://github.com/rabbitmq/mix_task_archive_deps/pull/2

https://groups.google.com/d/topic/rabbitmq-users/0d8TBs_qmGE/discussion
2020-09-28 15:36:02 -07:00
Michael Klishin 2c14e1dd3f Merge branch 'master' into lrb-update-supervisor 2020-09-26 10:00:57 +03:00
Michael Klishin 160742ea8a rpc module calls can return a {badrpc, term()} 2020-09-24 23:15:32 +03:00
dcorbacho 9c56b0cffc Add get_auth_attempts_by_source api 2020-09-23 15:52:50 +01:00
Luke Bakken c1d59f600c Remove tabs 2020-09-22 12:13:44 -07:00
Luke Bakken 111124bc05 Several bug fixes found when running tests
Validate delay value
2020-09-22 12:08:32 -07:00
dcorbacho 8bb1fbdf3d Merge remote-tracking branch 'origin/master' into auth-attempt-metrics 2020-09-22 17:11:08 +01:00
dcorbacho 0d09fb0a85 Split auth attempt metrics into global and detailed
GC collection can then be done by deleting all entries on the ETS table
and total counters per protocol can be kept without individually scanning
all entries
2020-09-22 16:47:10 +01:00
dcorbacho 4c6251d3fb Include protocol in auth metrics 2020-09-22 16:46:52 +01:00
Luke Bakken db7dfdf0cf Remove old supervisor2 module 2020-09-21 15:15:11 -07:00
Luke Bakken 0a87aeace5 Begin migration of code from supervisor2 to the 22.3.4.10 version
Moved a lot of code from old supervisor to new one

All compilation errors fixed

Use the macro, Luke
2020-09-21 15:14:49 -07:00
Philip Kuryloski c1a4790c7b Erlang 21.3 -> 22.3 for `make github-actions`
This has no direct effect until GitHub Actions workflows are
regenerated using make
2020-09-02 10:21:20 +02:00
Luke Bakken 4c9d67bb9d Do not change the behavior of rpc_call 2020-09-02 03:42:58 +03:00
Luke Bakken bf199aa36c Set RPC_TIMEOUT to 15 seconds 2020-09-02 03:42:58 +03:00