Adds WORKSPACE.bazel, BUILD.bazel & *.bzl files for partial build & test with Bazel. Introduces a build-time dependency on https://github.com/rabbitmq/bazel-erlang
This allows including additional applications or third party
plugins when creating a release, running the broker locally,
or just building from the top-level Makefile.
To include Looking Glass in a release, for example:
$ make package-generic-unix ADDITIONAL_PLUGINS="looking_glass"
A Docker image can then be built using this release and will
contain Looking Glass:
$ make docker-image
Beware macOS users! Applications such as Looking Glass include
NIFs. NIFs must be compiled in the right environment. If you
are building a Docker image then make sure to build the NIF
on Linux! In the two steps above, this corresponds to Step 1.
To run the broker with Looking Glass available:
$ make run-broker ADDITIONAL_PLUGINS="looking_glass"
This commit also moves Looking Glass dependency information
into rabbitmq-components.mk so it is available at all times.
Lager strips trailing newline characters but OTP logger with the default
formatter adds a newline at the end. To avoid unintentional multi-line log
messages we have to revisit most messages logged.
Some log entries are intentionally multiline, others
are printed to stdout directly: newlines are required there
for sensible formatting.
The configuration remains the same for the end-user. The only exception
is the log root directory: it is now set through the `log_root`
application env. variable in `rabbit`. People using the Cuttlefish-based
configuration file are not affected by this exception.
The main change is how the logging facility is configured. It now
happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is
removed.
The supported outputs remain the same: the console, text files, the
`amq.rabbitmq.log` exchange and syslog.
The message text format slightly changed: the timestamp is more precise
(now to the microsecond) and the level can be abbreviated to always be
4-character long to align all messages and improve readability. Here is
an example:
2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE ==
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5
2021-03-03 10:22:30.377860+01:00 [info] <0.229.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
The example above also shows that multiline messages are supported and
each line is prepended with the same prefix (the timestamp, the level
and the Erlang process PID).
JSON is also supported as a message format and now for any outputs.
Indeed, it is possible to use it with e.g. syslog or the exchange. Here
is an example of a JSON-formatted message sent to syslog:
Mar 3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}}
For quick testing, the values accepted by the `$RABBITMQ_LOGS`
environment variables were extended:
* `-` still means stdout
* `-stderr` means stderr
* `syslog:` means syslog on localhost
* `exchange:` means logging to `amq.rabbitmq.log`
`$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in
addition to the existing `+color` one). With that modifier, messages are
formatted as JSON intead of plain text.
The `rabbitmqctl rotate_logs` command is deprecated. The reason is
Logger does not expose a function to force log rotation. However, it
will detect when a file was rotated by an external tool.
From a developer point of view, the old `rabbit_log*` API remains
supported, though it is now deprecated. It is implemented as regular
modules: there is no `parse_transform` involved anymore.
In the code, it is recommended to use the new Logger macros. For
instance, `?LOG_INFO(Format, Args)`. If possible, messages should be
augmented with some metadata. For instance (note the map after the
message):
?LOG_NOTICE("Logging: switching to configured handler(s); following "
"messages may not be visible in this log output",
#{domain => ?RMQLOG_DOMAIN_PRELAUNCH}),
Domains in Erlang Logger parlance are the way to categorize messages.
Some predefined domains, matching previous categories, are currently
defined in `rabbit_common/include/logging.hrl` or headers in the
relevant plugins for plugin-specific categories.
At this point, very few messages have been converted from the old
`rabbit_log*` API to the new macros. It can be done gradually when
working on a particular module or logging.
The Erlang builtin console/file handler, `logger_std_h`, has been forked
because it lacks date-based file rotation. The configuration of
date-based rotation is identical to Lager. Once the dust has settled for
this feature, the goal is to submit it upstream for inclusion in Erlang.
The forked module is calld `rabbit_logger_std_h` and is based
`logger_std_h` in Erlang 23.0.
Decorators are only executed if the plugin is enabled (registered in
rabbit_queue_decorator), thus initialising the links through `startup`.
There is no obvious way for a fed link to start if the plugin isn't running,
so this check can be removed.
`rabbit_federation_queue_link:go()` and the exchange equivalent are asynchronous (cast).
They are luckily executed after the links are started by the decorators, and probably
most of the time they find the links up and ready to `go`. However, this might not always
be the case. The retry introduced on the previous commit that is triggered by the link itself
once if finds that federation is down, guarantees that the `go` will be handled by the link
process. Thus, the calls to rabbit_federation_queue_link:go() and rabbit_federation_exchange_link:go()
can be removed from the app.
For the rare case where federated exchange experience rapid
binding changes. As of rabbitmq/rabbitmq-federation#97,
such environments have a race condition between binding
and message propagation.
The queue decorator is called even when no links are present, which caused the
pg2 group to be created. At the same time, deleting a link wasn't deleting
the queue group. Closes#2497
very rare for amqp_connection:open_channel/2 to
timeout and result in the same fate of direct
connection and channel leaks. However, it can't
be ruled out in an unstable TCP network. This
commit prevents possibility of process leaks
on such failed channel establishment cases.
on failed upstream connection attempts due to AMQP client
timeouts. On long running nodes, these have potential to
eventually take down a node on reaching or exceeding the
the process limit.
Asserting on links across all virtual hosts makes no sense
for some tests, and creates additional shared state and
makes these assertions significantly more timing-sensitive.
Per discussion with @lukebakken.
This makes it possible to federate messages between more than two
virtual hosts in a single cluster. Previously cycle detection would
drop messages that have traversed a "hop" (federation link)
in the same cluster.
Pair: @lukebakken.
when obfuscating an upstream.
credentials_obfuscation:encrypt/1 requires a binary and
in some rare cases we've seen that the password value can
be passed in as a string.
and add a VMware copyright notice.
We did not mean to make this code Incompatible with Secondary Licenses
as defined in [1].
1. https://www.mozilla.org/en-US/MPL/2.0/FAQ/
Timeouts were not handled gracefully which could disrupt
the link for no good reason. Such one-off checks are
meant to be best effort and not performed particularly
frequently. Their failures should result in visible
log messages but not link restarts.
While at it, improve logging in related code paths.
Per discussion with @kjnilsson.
Set it to 'never' to make sure the link never removes its internal queue.
It's highly advisable to then specify queue expiration (TTL) to avoid
resource leaks.
The queue is durable and might contain valuable data.
On the other hand, when federation link is shutting down
because policies have changed or the plugin is being disabled,
it makes sense to delete the resources it was using.
References #105.
Consumer tag is currently generated from the upstream name.
Consumer tag serves to identify a downsream from the upstream queue,
while upsream name identifies upstream from the downstream node.
It would be more convenient to use different values.
Addresses #66
This would make it easier to distinguish federation links from other
consumers and from each other (if configured with different names)
Adresses #66 in a way
Philip Kuryloski
kjnilsson
Loïc Hoguin
Michael Klishin
Jean-Sébastien Pédron
dcorbacho
Michael Klishin
Arnaud Cogoluègnes
Ayanda-D
Jean-Sébastien Pédron
Luke Bakken
Luke Bakken
Gerhard Lazu
Michael Klishin
Grigory Starinkin
Diana Corbacho
Daniil Fedotov
Diego Herrera Celedón