The `set` command in the implementation of `/bin/sh` included in the
official RabbitMQ Docker image returns multi-line variable values
differently than the tested Bourne shell implementation (GNU Bash, dash
and FreeBSD sh).
I don't know what implementation is used by that Docker image, but here
is the output of `set`, for a variable set to "\n'test'":
TEST_VAR='
'"'"'test'"'"
The problem was reported in the following discussion:
https://github.com/rabbitmq/rabbitmq-server/discussions/2458
While here, add a small testcase to check a couple outputs.
... instead of 23.0.
Erlang 23.1 is the version the Concourse pipelines use. We expect the
Concourse pipelines and the GitHub Actions workflow to be on the same
page.
GC collection can then be done by deleting all entries on the ETS table
and total counters per protocol can be kept without individually scanning
all entries
net_adm:name/1 returns a new value, 'noport', in Erlang 24. This value
being absent in the function spec in previous versions of Erlang, we get
a warning from Dialyzer until we start to the yet-to-be-release Erlang
24 in CI. Therefore we disable this specific warning.
... instead of .ez archives.
The benefits for doing this:
* We can use native code, as is the case for lz4 and zstd bindings in
the Tanzu RabbitMQ version for instance. Indeed, Erlang does not
support loading native code (NIF or port drivers) from .ez archives.
* We can remove custom code to handle .ez archives. We have special
cases in Erlang.mk plugins as well as the `rabbit_plugins` module, in
particular the code to extract .ez archives (even though Erlang knows
how to use them directly).
* Prevent hard to debug situations when the .ez archive name does not
match the top-level directory inside the archive. In this case, Erlang
says it can't load the application but doesn't tell much more.
* Debugging and "hot-patching" plugins become easier: we just have to
copy the recompiled .beam file in place of the existing one. There
is no need to unpack the plugin, replace the file and recreate the
archive.
* Release packages can be smaller. gzip, bzip2 and xz, common
compression algorithm for Unix packages, give much better result if
they compress the .beam files directly instead of "compressing" zip
files (the .ez archives are plain zip archives). For instance, the
generic-unix package goes from 15 MiB when using .ez archives to just
12 MiB when using directory.
I would also like to experiment with Erlang releases in the future.
Using directories for Erlang applications instead of .ez archives is
mandatory for this to work according to my latest tests.
Of course, this change doesn't break support for .ez archives (and we
will keep support for this). End users can still download third-party
plugins as .ez archives and drop them in the plugins directory.
On Windows, the current working directory is also searched, which can
lead to problems. Instead, use `init:get_argument(root)` to get the root
of the Erlang release, then we know `bin/erl` will always be present.
In addition to the `rabbitmq-components.mk` existence check, we now
verfy that the directory is named `deps`.
This is to increase the chance that, if we find a
`rabbitmq-componentS.mk` file in the upper directories, this project is
indeed inside a DEPS_DIR.
For instance, in our GitHub Actions workflows, when we prepared the
secondary umbrellas for mixed-version testing, it happened that the
secondary umbrellas were under a clone of rabbitmq-server. Therefore
the first (and only) condition was met and the Makefile erroneously
considered it was inside a DEPS_DIR. As a consequence, dependencies of
the umbrellas were fetched in the wrong place.
net_adm:name/1 returns a new value, 'noport', in Erlang 24. This value
being absent in the function spec in previous versions of Erlang, we get
a warning from Dialyzer until we start to the yet-to-be-release Erlang
24 in CI. Therefore we disable this specific warning.
... instead of .ez archives.
The benefits for doing this:
* We can use native code, as is the case for lz4 and zstd bindings in
the Tanzu RabbitMQ version for instance. Indeed, Erlang does not
support loading native code (NIF or port drivers) from .ez archives.
* We can remove custom code to handle .ez archives. We have special
cases in Erlang.mk plugins as well as the `rabbit_plugins` module, in
particular the code to extract .ez archives (even though Erlang knows
how to use them directly).
* Prevent hard to debug situations when the .ez archive name does not
match the top-level directory inside the archive. In this case, Erlang
says it can't load the application but doesn't tell much more.
* Debugging and "hot-patching" plugins become easier: we just have to
copy the recompiled .beam file in place of the existing one. There
is no need to unpack the plugin, replace the file and recreate the
archive.
* Release packages can be smaller. gzip, bzip2 and xz, common
compression algorithm for Unix packages, give much better result if
they compress the .beam files directly instead of "compressing" zip
files (the .ez archives are plain zip archives). For instance, the
generic-unix package goes from 15 MiB when using .ez archives to just
12 MiB when using directory.
I would also like to experiment with Erlang releases in the future.
Using directories for Erlang applications instead of .ez archives is
mandatory for this to work according to my latest tests.
Of course, this change doesn't break support for .ez archives (and we
will keep support for this). End users can still download third-party
plugins as .ez archives and drop them in the plugins directory.
On Windows, the current working directory is also searched, which can
lead to problems. Instead, use `init:get_argument(root)` to get the root
of the Erlang release, then we know `bin/erl` will always be present.
In addition to the `rabbitmq-components.mk` existence check, we now
verfy that the directory is named `deps`.
This is to increase the chance that, if we find a
`rabbitmq-componentS.mk` file in the upper directories, this project is
indeed inside a DEPS_DIR.
For instance, in our GitHub Actions workflows, when we prepared the
secondary umbrellas for mixed-version testing, it happened that the
secondary umbrellas were under a clone of rabbitmq-server. Therefore
the first (and only) condition was met and the Makefile erroneously
considered it was inside a DEPS_DIR. As a consequence, dependencies of
the umbrellas were fetched in the wrong place.
and add a VMware copyright notice.
We did not mean to make this code Incompatible with Secondary Licenses
as defined in [1].
1. https://www.mozilla.org/en-US/MPL/2.0/FAQ/
When we source the $CONF_ENV_FILE script, we set a few variables which
this script expects. Those variables are given without their prefix. For
instance, $MNESIA_BASE.
The $CONF_ENV_FILE script can set $RABBITMQ_MNESIA_BASE. Unfortunately
before this patch, the variable would be ignored, in favor of the
default value which was passed to the script ($MNESIA_BASE).
The reason is that variables set by the script are handled in the
alphabetical order. Thus $MNESIA_BASE is handled first, then
$RABBITMQ_MNESIA_BASE.
Because the code didn't give any precedence, the first variable set
would "win". This explains why users who set $RABBITMQ_MNESIA_BASE in
$CONF_ENV_FILE, but using RabbitMQ 3.8.4+ (which introduced
`rabbit_env`), unexpectedly had their node use the default Mnesia base
directory.
The patch is rather simple: when we check if a variable is already set,
we give precedence to the $RABBITMQ_* prefixed variables. Therefore, if
the $CONF_ENV_FILE script sets $RABBITMQ_MNESIA_BASE, this value will be
used, regardless of the value of $MNESIA_BASE.
This didn't happen with variables set in the environment (i.e. the
environment of rabbitmq-server(8)) because the prefixed variables
already had precedence.
Fixesrabbitmq/rabbitmq-common#401.
This allows RabbitMQ to configure `rabbit_log` as a Logger handler.
See a related commit in rabbit_prelaunch_early_logging in
rabbitmq-server, where `rabbit_log` is being configured as a Logger
handler. The commit message explains the reason behind this.
The default timeout of 30 seconds was not sufficient to allow graceful shutdown of a message store with millions of persistent messages. Rather than increase the timeout in general, introduce a new macro with a default of 600 seconds
... instead of the cache action.
The cache action is quite unstable (failing to download the cached
files). In this commit, we try to use the artefacts instead. At this
point, we don't know if it is more reliable, but we'll see with time.
As an added bonus, we can download the archives passed between jobs for
inspection if we need.
Otherwise, for instance, running Dialyzer in the Erlang client fails with the
following error if it was cloned directly (i.e. outside of the Umbrella):
dialyzer: Bad directory for -pa: .../amqp_client/deps/rabbitmq_cli/_build/dev/lib/rabbitmqctl/ebin
... and their value.
Both prefixed and non-prefixed variables are returned by this function.
While here, fix a conflict between $RABBITMQ_HOME and $HOME in
var_is_used/1: the latter shouldn't be considered as used.
When we generate the workflows, we pick the latest tag of each release
branch. That list of tags is used to clone secondary umbrellas in the
workflows and run the testsuites against each of them.
When generating workflows for `master`, we take the latest tag of each
release branch.
When generating workflows for a release branch, we take the latest tag
of each older release branch, plus the first tag of the same release
branch.
Some examples:
* `master` is tested with 3.8.3 and 3.7.25
* `v3.8.x` is tested with 3.8.0 and 3.7.25
We need a monotonically increasing number for the version used by the
Concourse S3 resource. A Git commit hash does not work because they do
not have this property.
The main entry point is `make github-actions` which generates the
workflows.
Currently, it handles workflows to test the project with different
versions of Erlang.
It generates a file called `$(PROJECT)-rabbitmq-deps.mk` which has a
dependency definition line of the form expected by Erlang.mk, for each
RabbitMQ component the project depends on.
Therefore the line indicates:
* `git` as the fetch method
* the repository URL
* the Git commit hash the dependency is on
Here is an example for rabbitmq-server:
dep_rabbit_common := git https://github.com/rabbitmq/rabbitmq-common.git d9ccd8d9cdd58310901f318fed676aff59be5afb
dep_rabbitmq_cli := git https://github.com/rabbitmq/rabbitmq-cli.git f6eaae292d27da4ded92b7c1b51a8ddcfefa69c2
dep_rabbitmq_codegen := git https://github.com/rabbitmq/rabbitmq-codegen.git 65da2e86bd65c6b6ecd48478ab092721696bc709
The double-quoting was requited in the flock(1)/lockf(1) blocks because
of the use of `sh -c`. However it's incorrect in the `else` block.
Follow-up to commit 3f32a36e50.
The CLI has a high startup time. To speed up the
`start-background-broker` and `stop-node` recipes, two CLI calls are
replaced by two more basic commands which achieve the same goal.
The problem with the previous approach was that the `$(wildcard ...)`
directives might be evaluated too early: `deps/rabbit` might not be
available yet.
Moving the computation to the body of the recipe fixes the problem
because dependencies are available at this point.
In other words, if instead of cloning the Umbrella, one cloned
rabbitmq-server directly, the `install-cli-scripts` recipe would fail to
copy the scripts because it assumed `rabbit` was under `$(DEPS_DIR)`.
Now expected places are checked and an error is emitted if the recipe
can't find the right one.
dispatch_sync sits inbetween the behavior of submit and submit_async,
blocking the caller until a worker begins the task, as opposed
to not blocking at all, or blocking util the task has finished.
This is useful when you want to throttle submissions to the pool
from a single process, such that all workers are busy, but there
exists no backlog of work for the pool.
On Darwin, the default tar fails with unkown --transform flag.
FAILS: bsdtar 2.8.3 - libarchive 2.8.3
SUCCEEDS: tar (GNU tar) 1.32
re https://github.com/rabbitmq/rabbitmq-common/pull/364
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Michael Klishin
Arnaud Cogoluègnes
Jean-Sébastien Pédron
Luke Bakken
Michael Klishin
Loïc Hoguin
Jean-Sébastien Pédron
kjnilsson
Nicolas Caille
dcorbacho
Philip Kuryloski
Luke Bakken
Michael Klishin
Anupama Singh
Ayanda-D
Gerhard Lazu
Philip Kuryloski