Commit Graph

508 Commits

Author SHA1 Message Date
Iliia Khaprov 19f122fea4 Prometheus core metrics collector: Do not render any sampels that are NaN or undefined
close #8740
2023-07-08 18:45:03 +02:00
Simon Unge 2a2af36b9c rename fix 2023-06-23 14:53:57 -07:00
Simon Unge 8b3ca4c972 See #8605. Add authentcation support to prometheus. 2023-06-23 13:54:45 -07:00
Michael Klishin 55442aa914 Replace @rabbitmq.com addresses with rabbitmq-core@groups.vmware.com
Don't ask why we have to do it. Because reasons!
2023-06-20 15:40:13 +04:00
Rin Kuryloski eb94a58bc9 Add a workflow to compare the bazel/erlang.mk output
To catch any drift between the builds
2023-05-15 13:54:14 +02:00
Michael Klishin 59fe5dc01b
Prometheus: handle scenarios when no listener is configured
Start a plain TCP one with all defaults.
2023-05-06 00:19:58 +04:00
Chunyi Lyu 4ddb0c2038 Support TLS-only listener for Prometheus
- tcp listener can be turned off by setting
'prometheus.tcp.listener = none'
- config schema follows web_mqtt and web_stomp
2023-05-05 15:44:53 +01:00
Rin Kuryloski a944439fba Replace globs in bazel with explicit lists of files
As this is preferred in rules_erlang 3.9.14
2023-04-25 17:29:12 +02:00
Rin Kuryloski 854d01d9a5 Restore the original -include_lib statements from before #6466
since this broke erlang_ls

requires rules_erlang 3.9.13
2023-04-20 12:40:45 +02:00
Rin Kuryloski 8de8f59d47 Use gazelle generated bazel files
Bazel build files are now maintained primarily with `bazel run
gazelle`. This will analyze and merge changes into the build files as
necessitated by certain code changes (e.g. the introduction of new
modules).

In some cases there hints to gazelle in the build files, such as `#
gazelle:erlang...` or `# keep` comments. xref checks on plugins that
depend on the cli are a good example.
2023-04-17 18:13:18 +02:00
Rin Kuryloski 8a7eee6a86 Ignore warnings when building plt files for dependencies
As we don't generally care if a dependency has warnings, only the
target
2023-04-17 10:09:24 +02:00
Rin Kuryloski 609171ec70 Rename the tanzu cli scope to vmware
And update other references to commercial editions
2023-02-16 13:49:54 +01:00
Ilia Kurenkov 051419e46b
Fix descriptions of auth metrics. 2023-02-11 23:36:21 +01:00
Alexey Lebedeff c7da0da8b8 Cleanup dialyzer calls
- Use the same base .plt everywhere, so there is no need to list
standard apps everywhere
- Fix typespecs: some typos and the use of not-exported types
2023-02-06 17:05:30 +01:00
Rin Kuryloski 5ef8923462 Avoid the need to pass package name to rabbitmq_integration_suite 2023-01-18 15:25:27 +01:00
Rin Kuryloski a317b30807 Use improved assert_suites2 macro from rules_erlang 3.9.0 2023-01-18 15:07:06 +01:00
Michael Klishin ba7b44df8a
Merge pull request #6879 from rabbitmq/dialyzer-warnings-rabbitmq-prometheus
Fix all dialyzer warnings in rabbitmq_prometheus
2023-01-13 12:21:15 -06:00
Alexey Lebedeff cd92258346 Fix all dialyzer warnings in rabbitmq_prometheus 2023-01-13 15:52:26 +01:00
Michal Kuratczyk 510415f8b9
Update prometheus.erl to 4.10.0
Since 4.10.0 was released specifically to address an issue we
encountered in RabbitMQ integration with prometheus.erl, new test was
added to validate this functionality in the future.
2023-01-13 10:24:41 +01:00
Michael Klishin 8e8def801c
Wording 2023-01-03 14:19:58 -05:00
Ilia Kurenkov 98a5e34e90 DRY: Link docs for `/metrics/detailed` endpoint to website. 2023-01-03 13:45:45 +01:00
Michael Klishin ec4f1dba7d
(c) year bump: 2022 => 2023 2023-01-01 23:17:36 -05:00
Luke Bakken 7fe159edef
Yolo-replace format strings
Replaces `~s` and `~p` with their unicode-friendly counterparts.

```
git ls-files *.erl | xargs sed -i.ORIG -e s/~s>/~ts/g -e s/~p>/~tp/g
```
2022-10-10 10:32:03 +04:00
Loïc Hoguin 73dd0acf01
rabbit_prometheus_http_SUITE: Update tests for new CQs
CQs without consumers will have only one message in memory.
2022-09-27 12:00:10 +02:00
Michael Klishin 96b6e6c368
Merge pull request #5463 from rabbitmq/global-metrics-values
Move message rate metrics from aggregated to global counters
2022-09-12 20:25:08 +04:00
Iliia Khaprov - VMware e15d12d767
Merge pull request #5449 from rabbitmq/grafana-9-support
Update RabbitMQ Dashboards to support latest Grafana versions
2022-08-24 21:25:56 +02:00
David Ansari c3cccf4963
Rename run_queues_length_total to run_queues_length
It's a gauge, not a counter.
@deadtrickster fixed the bug in d0feb0df58
See #4380
2022-08-24 18:04:14 +02:00
Péter Gömöri bf00ee4cfc Fix a typo in a comment in prometheus_rabbitmq_core_metrics_collector 2022-08-23 00:54:35 +02:00
Connor Rogers 6ee0a318e8
Move message rate metrics from channel/queue aggregation to global counters 2022-08-08 16:19:01 +01:00
Connor Rogers c88326ef23
Add README.md for creating/updating dashboards 2022-08-05 17:41:47 +01:00
Connor Rogers e35fd65ff3
Fix overview graphs in Grafana 9
'-1' is no longer accepted as of Grafana 9, and causes a console error when rendering
2022-08-05 17:09:34 +01:00
Connor Rogers 9ac6862e06
Fix dist link graph
Both directions of the link were showing as one entry instead of two.

This is beacuse of https://github.com/flant/grafana-statusmap/issues/277
2022-08-05 16:51:21 +01:00
Connor Rogers 42f30ba7c3
Set time series to show all series in tooltip 2022-08-05 16:14:29 +01:00
Connor Rogers 40767cdae4
Take dashboard definitions straight from exported Grafana for simplicity 2022-08-05 16:09:30 +01:00
Connor Rogers 4d28eef0f8
Migrate from deprecated panels in Grafana 2022-08-05 15:46:27 +01:00
Connor Rogers 8e404ecd04
Update to supported Grafana and Prometheus versions 2022-08-05 12:52:26 +01:00
Jean-Sébastien Pédron 6e9ee4d0da
Remove test code which depended on the `quorum_queue` feature flags
These checks are now irrelevant as the feature flag is required.
2022-08-01 12:41:30 +02:00
Iliia Khaprov 360db38db0 Add process_start_time_seconds metrics. See #4539 2022-07-13 08:35:33 +02:00
Philip Kuryloski 15a79466b1 Use the new xref2 macro from rules_erlang
That adopts the modern erlang.mk xref behaviour
2022-06-09 23:18:28 +02:00
Philip Kuryloski 327f075d57 Make rabbitmq-server work with rules_erlang 3
Also rework elixir dependency handling, so we no longer rely on mix to
fetch the rabbitmq_cli deps

Also:

- Specify ra version with a commit rather than a branch
- Fixup compilation options for erlang 23
- Add missing ra reference in MODULE.bazel
- Add missing flag in oci.yaml
- Reduce bazel rbe jobs to try to save memory
- Use bazel built erlang for erlang git master tests
- Use the same cache for all the workflows but windows
- Avoid using `mix local.hex --force` in elixir rules
  - Fetching seems blocked in CI, and this should reduce hex api usage in
    all builds, which is always nice
- Remove xref and dialyze tags since rules_erlang 3 includes them in
  the defaults
2022-06-08 14:04:53 +02:00
Loïc Hoguin dc70cbf281
Update Erlang.mk and switch to new xref code 2022-05-31 13:51:12 +02:00
Michael Klishin 018b04a1ea
Wording 2022-04-26 23:08:00 +04:00
Péter Gömöri 35b21797ae Expose head_message_timestamp via Prometheus plugin as well
It is already exposed via rabbitmqctl and the API. It is also exposed by
old or unofficial prometheus plugins and other monitoring
integrations (DataDog).
2022-04-26 15:42:57 +02:00
Michael Klishin 7c47d0925a
Revert "Correct a double quote introduced in #4603"
This reverts commit 6a44e0e2ef.

That wiped a lot of files unintentionally
2022-04-20 16:05:56 +04:00
Michael Klishin 6a44e0e2ef
Correct a double quote introduced in #4603 2022-04-20 16:01:29 +04:00
Luke Bakken dba25f6462
Replace files with symlinks
This prevents duplicated and out-of-date instructions.
2022-04-15 06:04:29 -07:00
Loïc Hoguin 499e0b9197
Remove the CQv1 disabled stats from management/Prometheus 2022-04-05 12:37:54 +02:00
Michael Klishin c38a3d697d
Bump (c) year 2022-03-21 01:21:56 +04:00
David Ansari a3905da47c Add note about missed Prometheus counter updates
Currently, the quorum queue state machine updates counters via mod_call effects
which are not guaranteed to be executed.

They are updated via mod_call effects such that only the leader
increments the counter (and not the followers).

In certain failure scenarios when dead-lettering lots of messages
at the same time, these mod_call effects might not be executed.

Hence, one shouldn't rely that counters for dead lettered messages
and dead lettered confirmed messages match up 100% even though all
dead-lettered messages were confirmed eventually.
2022-02-28 16:28:09 +01:00
David Ansari 8c286cc680 Add Prometheus metrics for dead-lettered messages
> curl -s localhost:15692/metrics | grep rabbitmq_global_messages_dead_lettered
\# TYPE rabbitmq_global_messages_dead_lettered_delivery_limit_total counter
\# HELP rabbitmq_global_messages_dead_lettered_delivery_limit_total Total number of messages dead-lettered due to delivery-limit exceeded
rabbitmq_global_messages_dead_lettered_delivery_limit_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_least_once"} 0
rabbitmq_global_messages_dead_lettered_delivery_limit_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_delivery_limit_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="disabled"} 0
\# TYPE rabbitmq_global_messages_dead_lettered_expired_total counter
\# HELP rabbitmq_global_messages_dead_lettered_expired_total Total number of messages dead-lettered due to message TTL exceeded
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_classic_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_classic_queue",dead_letter_strategy="disabled"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_least_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="disabled"} 0
\# TYPE rabbitmq_global_messages_dead_lettered_rejected_total counter
\# HELP rabbitmq_global_messages_dead_lettered_rejected_total Total number of messages dead-lettered due to basic.reject or basic.nack
rabbitmq_global_messages_dead_lettered_rejected_total{queue_type="rabbit_classic_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_rejected_total{queue_type="rabbit_classic_queue",dead_letter_strategy="disabled"} 0
rabbitmq_global_messages_dead_lettered_rejected_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_least_once"} 0
rabbitmq_global_messages_dead_lettered_rejected_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_rejected_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="disabled"} 0
\# TYPE rabbitmq_global_messages_dead_lettered_confirmed_total counter
\# HELP rabbitmq_global_messages_dead_lettered_confirmed_total Total number of messages dead-lettered and confirmed by target queues
rabbitmq_global_messages_dead_lettered_confirmed_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_least_once"} 0
\# TYPE rabbitmq_global_messages_dead_lettered_maxlen_total counter
\# HELP rabbitmq_global_messages_dead_lettered_maxlen_total Total number of messages dead-lettered due to overflow drop-head or reject-publish-dlx
rabbitmq_global_messages_dead_lettered_maxlen_total{queue_type="rabbit_classic_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_maxlen_total{queue_type="rabbit_classic_queue",dead_letter_strategy="disabled"} 0
rabbitmq_global_messages_dead_lettered_maxlen_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_maxlen_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="disabled"} 0

A few notes:
* dead_letter_strategy 'disabled' means either user did not configure
  dead-letter-exchange or configured dead-letter-exchange does not
  exist.
* Only time series that make sense get output.
  Example 1: Combination of 'at_least_once' and 'maxlen' will always be 0.
  Hence, we omit that time series.
  Example 2: 'confirmed' makes only sense with quorum queues and
  'at_least_once'.
  Example 3: 'delivery_limit' makes only sense with quorum queues.
* Users get to know *why* messages were dead-lettered.
* Before this commit, there was no possibilities for users to alert
  based on messages being dropped from the head of the queue when
  overflow=drop-head.
* Users can now easily create alerts:
  Example 1: Message gets silently dropped (i.e.
  dead_letter_strategy='disabled') instead of actually dead-lettered.
  Example 2: Detect dead-letter topology misconfigurations.
  Example 3: Messages expire
  Example 4: Messages overflow
  Example 5: Messages requeued too often
* Stream queues by definition do not dead-letter.
2022-02-28 16:28:02 +01:00
Philip Kuryloski 226e00fcd2 Tighten up dialyzer usage
now that rules_erlang no longer cascades up dialyzer warnings from deps
2022-02-24 11:18:41 +01:00
Philip Kuryloski d8201726ae Ignore dialyzer warnings for most apps 2022-02-21 09:19:56 +01:00
Philip Kuryloski efcd881658 Use rules_erlang v2
bazel-erlang has been renamed rules_erlang. v2 is a substantial
refactor that brings Windows support. While this alone isn't enough to
run all rabbitmq-server suites on windows, one can at least now start
the broker (bazel run broker) and run the tests that do not start a
background broker process
2022-01-18 13:43:46 +01:00
Alexey Lebedeff 7676ed9685 Use `rabbitmq_cluster_` prefix for cluster-wide metrics 2021-11-24 16:49:43 +01:00
Michael Klishin 38d64a54b1
Wording 2021-11-24 14:19:57 +03:00
Michael Klishin a1c0cd3785
Wording 2021-11-24 14:02:10 +03:00
Alexey Lebedeff 6e3012aaf9 Add optional metrics for vhost and exchange count
These can make sense in some scenarios, e.g. when vhost/exchanges are
+created using self-service automation
2021-11-24 11:00:41 +01:00
Luke Bakken bd2858c208
Compile the regex 2021-11-22 08:30:17 -08:00
dcorbacho a7c9b66653 Use own key to exclude queues 2021-11-16 16:53:17 +01:00
dcorbacho 242cb539b3 Exclude queues from aggregated metrics in prometheus collector
Uses same exclusion pattern as the management agent
2021-11-16 10:23:39 +01:00
Alexey Lebedeff 8598c51579 Pre-render prometheus labels
This makes per-object metrics twice as fast.

Depends on https://github.com/deadtrickster/prometheus.erl/pull/137
2021-11-09 13:04:39 +01:00
Alexey Lebedeff b9ebfb8980 Fix ssl port handling in prometheus plugin
All ssl options were stored in the same proplist, and the code was
then trying to determine whether an option actually belongs to ranch
ssl options or not.

Some keys landed in the wrong place, like it did happen in #2975 -
different ports were mentioned in listener config (default at
top-level, and non-default in `ssl_opts`). Then `ranch` and
`rabbitmq_web_dispatch` were treating this differently.

This change just moves all ranch ssl opts into proper place using
schema, removing any need for guessing in code.

The only downside is that advanced config compatibility is broken.
2021-10-20 14:55:33 +02:00
Michael Klishin 3826a0df25
Compile #3561 2021-10-13 01:27:16 +03:00
Michael Klishin 670f240537
Compile #3561 2021-10-12 20:17:51 +03:00
Johannes Würbach 84de860b4c
feat(prom): expose cluster id in identity 2021-10-12 15:43:46 +02:00
Alexey Lebedeff 989a299720 Emit identity info in prometheus /metrics/detailed endpoint
This is needed to make filtering metrics on a cluster name possible.
2021-09-28 19:35:02 +02:00
Alexey Lebedeff 5501d07b8b Use rabbitmq_ct_helpers to allocate prometheus port
This test always used standard 15692 before, which were causing
conflicts with e.g. local `make run-broker`.
2021-09-22 15:23:35 +02:00
Alexey Lebedeff 4bb2262140 Allow selective querying for prometheus plugin 2021-09-20 14:59:17 +02:00
Michael Klishin 47b20e8f7c
Prometheus: alarm-related metric naming 2021-08-17 20:58:24 +03:00
Ilya Khaprov 9fed915192
Add alarms prometheus collector.
close #2653
2021-08-16 20:32:29 +02:00
Gerhard Lazu 62d82e1660
Break down metrics by node in all RabbitMQ-Stream pie charts
Otherwise we won't be able to see which nodes are running "hot"

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-08-11 13:39:30 +01:00
David Ansari 4b774db5c1 Use same threshold color for "Errors since boot" 2021-08-02 17:05:17 +02:00
David Ansari c99ee6961e Use same colorMode in all RabbitMQ-Stream panels
Co-authored-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-08-02 13:33:00 +02:00
David Ansari ea18c31288 Make RabbitMQ-Stream dashboard work via ConfigMap
Before this commit, importing the dashboard via ConfigMap as seen in
1eb1dc618e
didn't work because DS_PROMETHEUS variable was undefined in Grafana.

Related to https://github.com/rabbitmq/rabbitmq-server/pull/3250

Co-authored-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-08-02 13:12:48 +02:00
Gerhard Lazu 65afbb931b
Ensure RabbitMQ-Stream dashboard works correctly after import
This breaks the docker-compose integration, but we need to move away
from it anyways, the whole dev flow needs revisiting after our focus on
K8s.

$__rate_interval does not work with irate, dropping it in favour of 60s,
same as all other dashboards.

This is a follow-up to https://github.com/rabbitmq/rabbitmq-server/pull/3250

Thanks @ansd for mentioning about the post-import issues.

It was uploaded as https://grafana.com/api/dashboards/14798/revisions/3/download

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-07-30 13:53:02 +01:00
Gerhard Lazu 35a6369327
Restart stream-perf-test on-failure
This handles the scenario where rmq2 is not available, and
stream-perf-test exits with a non-zero exit code. Good spot @ansd!

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-07-30 11:25:36 +01:00
David Ansari 47d572908d Convert string to integer for ulimits.nofile
Before this commit:

> make overview metrics
services.rmq1.ulimits.nofile.hard must be a integer
make: *** [Makefile:68: overview] Error 15

Accoring to the docs
https://docs.docker.com/compose/compose-file/compose-file-v3/#ulimits
this must be an integer.
2021-07-30 09:46:38 +02:00
Gerhard Lazu 6f5c4118ea
Publish RabbitMQ-Stream dashboard to grafana.com
Removed the Dockerfile and slimmed down the Makefile, all of this is now
handled by https://github.com/rabbitmq/rabbitmq-server/blob/master/.github/workflows/oci.yaml
cc @Zerpet @pjk25

More details here (including the steps used to publish to grafana.com):
https://github.com/rabbitmq/release-engineering/issues/11#issuecomment-887627938

I don't want to hold up this PR, will invest in automating the
steps described in the previous link another time. Time to 🚀

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-07-29 19:34:05 +01:00
Philip Kuryloski b26289cb47 Adjust rabbitmq_prometheus test suite timeouts in bazel 2021-07-22 11:00:14 +02:00
Gerhard Lazu 66ef8adfc8
Fix accept dependency in rabbitmq_prometheus
It's a runtime dependency, not a build dependency.

This is a fix and should be backported to v3.9.x, after rc.2 and just
before the final release. Would you disagree @dumbbell?

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-07-21 13:38:54 +01:00
Philip Kuryloski 8f9de08de7 Also assert no missing suites for all other deps 2021-07-12 18:05:55 +02:00
dcorbacho b636ad2565 Rename protocol error counters to _total 2021-06-30 12:46:41 +02:00
dcorbacho c9305d948a
Use number of publishing channels as global publishers in amqp091 2021-06-29 08:10:42 +01:00
Philip Kuryloski 8c7e7e0656 Revert "Default all `rabbitmq_integration_suite` to flaky in bazel"
This reverts commit 70cb8147b2.
2021-06-23 20:53:14 +02:00
Gerhard Lazu c7971252cd
Global counters per protocol + protocol AND queue_type
This way we can show how many messages were received via a certain
protocol (stream is the second real protocol besides the default amqp091
one), as well as by queue type, which is something that many asked for a
really long time.

The most important aspect is that we can also see them by protocol AND
queue_type, which becomes very important for Streams, which have
different rules from regular queues (e.g. for example, consuming
messages is non-destructive, and deep queue backlogs - think billions of
messages - are normal). Alerting and consumer scaling due to deep
backlogs will now work correctly, as we can distinguish between regular
queues & streams.

This has gone through a few cycles, with @mkuratczyk & @dcorbacho
covering most of the ground. @dcorbacho had most of this in
https://github.com/rabbitmq/rabbitmq-server/pull/3045, but the main
branch went through a few changes in the meantime. Rather than resolving
all the conflicts, and then making the necessary changes, we (@gerhard +
@kjnilsson) took all learnings and started re-applying a lot of the
existing code from #3045. We are confident in this approach and would
like to see it through. We continued working on this with @dumbbell, and
the most important changes are captured in
https://github.com/rabbitmq/seshat/pull/1.

We expose these global counters in rabbitmq_prometheus via a new
collector. We don't want to keep modifying the existing collector, which
grew really complex in parts, especially since we introduced
aggregation, but start with a new namespace, `rabbitmq_global_`, and
continue building on top of it. The idea is to build in parallel, and
slowly transition to the new metrics, because semantically the changes
are too big since streams, and we have been discussing protocol-specific
metrics with @kjnilsson, which makes me think that this approach is
least disruptive and... simple.

While at this, we removed redundant empty return value handling in the
channel. The function called no longer returns this.

Also removed all DONE / TODO & other comments - we'll handle them when
the time comes, no need to leave TODO reminders.

Pairs @kjnilsson @dcorbacho @dumbbell
(this is multiple commits squashed into one)

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-06-22 14:14:21 +01:00
Philip Kuryloski 70cb8147b2 Default all `rabbitmq_integration_suite` to flaky in bazel
Most tests that can start rabbitmq nodes have some chance of
flaking. Rather than chase individual flakes for now, this commit
changes the default (though it can still be overriden, as is the case
for config_scheme_SUITE in many places, since I have yet to see that
particular suite flake).
2021-06-21 16:10:38 +02:00
Philip Kuryloski 30f9a95b9f Add dialyze for remaning tier-1 plugins 2021-06-01 10:19:10 +02:00
Philip Kuryloski a3dbdecb8c Mark //deps/rabbitmq_prometheus:rabbit_prometheus_http_SUITE flaky 2021-05-21 18:32:20 +02:00
Philip Kuryloski 98e71c45d8 Perform xref checks on many tier-1 plugins 2021-05-21 12:03:22 +02:00
Philip Kuryloski e6df6615e1 Futher bazel file refactoring and deduplication 2021-05-11 16:15:33 +02:00
Philip Kuryloski b39cd342f2 buildifier formatting 2021-05-05 14:20:38 +02:00
Philip Kuryloski d61aa69039 Add rabbitmq_prometheus to bazel 2021-05-05 11:43:03 +02:00
Gerhard Lazu 1e5708b0c5
Fix Grafana dashboards when importing from URL
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-03-22 19:27:13 +00:00
Philip Kuryloski a63f169fcb Remove duplicate rabbitmq-components.mk and erlang.mk files
Also adjust the references in rabbitmq-components.mk to account for
post monorepo locations
2021-03-22 15:40:19 +01:00
kjnilsson 52f745dcde Update rabbitmq-components.mk
use v1.x branch of ra
2021-03-18 15:14:40 +00:00
Loïc Hoguin d5e3bdd623
Add ADDITIONAL_PLUGINS variable
This allows including additional applications or third party
plugins when creating a release, running the broker locally,
or just building from the top-level Makefile.

To include Looking Glass in a release, for example:

$ make package-generic-unix ADDITIONAL_PLUGINS="looking_glass"

A Docker image can then be built using this release and will
contain Looking Glass:

$ make docker-image

Beware macOS users! Applications such as Looking Glass include
NIFs. NIFs must be compiled in the right environment. If you
are building a Docker image then make sure to build the NIF
on Linux! In the two steps above, this corresponds to Step 1.

To run the broker with Looking Glass available:

$ make run-broker ADDITIONAL_PLUGINS="looking_glass"

This commit also moves Looking Glass dependency information
into rabbitmq-components.mk so it is available at all times.
2021-03-12 12:29:28 +01:00
Jean-Sébastien Pédron cdcf602749
Switch from Lager to the new Erlang Logger API for logging
The configuration remains the same for the end-user. The only exception
is the log root directory: it is now set through the `log_root`
application env. variable in `rabbit`. People using the Cuttlefish-based
configuration file are not affected by this exception.

The main change is how the logging facility is configured. It now
happens in `rabbit_prelaunch_logging`. The `rabbit_lager` module is
removed.

The supported outputs remain the same: the console, text files, the
`amq.rabbitmq.log` exchange and syslog.

The message text format slightly changed: the timestamp is more precise
(now to the microsecond) and the level can be abbreviated to always be
4-character long to align all messages and improve readability. Here is
an example:

    2021-03-03 10:22:30.377392+01:00 [dbug] <0.229.0> == Prelaunch DONE ==
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Starting RabbitMQ 3.8.10+115.g071f3fb on Erlang 23.2.5
    2021-03-03 10:22:30.377860+01:00 [info] <0.229.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com

The example above also shows that multiline messages are supported and
each line is prepended with the same prefix (the timestamp, the level
and the Erlang process PID).

JSON is also supported as a message format and now for any outputs.
Indeed, it is possible to use it with e.g. syslog or the exchange. Here
is an example of a JSON-formatted message sent to syslog:

    Mar  3 11:23:06 localhost rabbitmq-server[27908] <0.229.0> - {"time":"2021-03-03T11:23:06.998466+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","meta":{"domain":"rabbitmq.prelaunch","file":"src/rabbit_prelaunch_logging.erl","gl":"<0.228.0>","line":311,"mfa":["rabbit_prelaunch_logging","configure_logger",1],"pid":"<0.229.0>"}}

For quick testing, the values accepted by the `$RABBITMQ_LOGS`
environment variables were extended:
  * `-` still means stdout
  * `-stderr` means stderr
  * `syslog:` means syslog on localhost
  * `exchange:` means logging to `amq.rabbitmq.log`

`$RABBITMQ_LOG` was also extended. It now accepts a `+json` modifier (in
addition to the existing `+color` one). With that modifier, messages are
formatted as JSON intead of plain text.

The `rabbitmqctl rotate_logs` command is deprecated. The reason is
Logger does not expose a function to force log rotation. However, it
will detect when a file was rotated by an external tool.

From a developer point of view, the old `rabbit_log*` API remains
supported, though it is now deprecated. It is implemented as regular
modules: there is no `parse_transform` involved anymore.

In the code, it is recommended to use the new Logger macros. For
instance, `?LOG_INFO(Format, Args)`. If possible, messages should be
augmented with some metadata. For instance (note the map after the
message):

    ?LOG_NOTICE("Logging: switching to configured handler(s); following "
                "messages may not be visible in this log output",
                #{domain => ?RMQLOG_DOMAIN_PRELAUNCH}),

Domains in Erlang Logger parlance are the way to categorize messages.
Some predefined domains, matching previous categories, are currently
defined in `rabbit_common/include/logging.hrl` or headers in the
relevant plugins for plugin-specific categories.

At this point, very few messages have been converted from the old
`rabbit_log*` API to the new macros. It can be done gradually when
working on a particular module or logging.

The Erlang builtin console/file handler, `logger_std_h`, has been forked
because it lacks date-based file rotation. The configuration of
date-based rotation is identical to Lager. Once the dust has settled for
this feature, the goal is to submit it upstream for inclusion in Erlang.
The forked module is calld `rabbit_logger_std_h` and is based
`logger_std_h` in Erlang 23.0.
2021-03-11 15:17:36 +01:00
Michael Klishin f6e8320fc9
Merge branch 'otp-24-ranch' 2021-03-10 07:37:51 +03:00
dcorbacho 61f7b2a723 Update to ranch 2.0 2021-03-08 23:11:05 +01:00
Gerhard Lazu c18ad7a5b6
Fix colors for node names that include digits in Grafana dashboards
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-03-08 13:19:14 +00:00