Commit Graph

101 Commits

Author SHA1 Message Date
Gerhard Lazu 0945511e7f Capture learnings from ERL-959 into Erlang Distribution Grafana dashboard
It explains the correlation between inet packets & TCP packets, and why
the inet packet size varies when TLS is used for inter-node
communication.

[finishes 166419953]
2019-06-03 18:04:07 +01:00
Gerhard Lazu 90b5653dc1 Improve Memory available panel title & description
Thanks @essen!
2019-06-03 18:02:45 +01:00
Gerhard Lazu ebd4ffc67f Modify the default distribution link buffer
It makes a big difference for stable throughput. See screenshots from
https://bugs.erlang.org/browse/ERL-959

We need to test this in a real network - I'm thinking GCP -, outside of
Docker. The results will inform whether we should change the default -
which is 1436 bytes.

[#166419953]
2019-06-03 18:00:26 +01:00
Gerhard Lazu e7fa4a2753 Bump Erlang/OTP to 22.0.2 2019-06-03 17:59:57 +01:00
Gerhard Lazu 4df7ccf666 Default docker-compose file target to up & tail logs 2019-05-30 15:42:21 +01:00
Gerhard Lazu a6219d19d3 Add make target to clean all containers & volumes in Docker 2019-05-30 15:31:55 +01:00
Gerhard Lazu cd0e804180 Use consistent naming for the Erlang Distribution cluster
[#165818813]
2019-05-30 15:31:52 +01:00
Gerhard Lazu 112254ed96 Enable filtering Erlang Distribution metrics in Grafana by cluster
[finishes #165818813]
2019-05-30 13:58:56 +01:00
Gerhard Lazu 9ddb7e6cd6 Enable filtering RabbitMQ Overview metrics in Grafana by cluster
[#165818813]
2019-05-30 13:57:12 +01:00
Gerhard Lazu 6eeeb6b0b4 Set up separate clusters for Overview & Distribution metrics
This will allow to simulate multiple clusters feeding metrics into
Prometheus & Grafana.

[#165818813]
2019-05-30 13:56:17 +01:00
Gerhard Lazu 931e07455e Extract metrics services into a separate docker-compose file
Add cadvisor & node-exporter & Docker metrics.
Inspired by https://github.com/stefanprodan/dockprom
There are no Grafana dashboards for these metrics yet. The dockprom ones
don't show any panels in Grafana 6.

[#165818813]
2019-05-30 13:53:02 +01:00
Gerhard Lazu 7b632674c9 Update Distribution & Overview dashboard tags 2019-05-30 09:37:24 +01:00
Gerhard Lazu 6e614301d0 Bump Grafana & Prometheus images to latest stable 2019-05-30 09:36:52 +01:00
Gerhard Lazu 21aa9e9e68 Awlays install Grafana plugins
Even though this slows down Grafana container startup, we need to ensure
that this plugin is present, otherwise the panels that track process
state won't work. This will be slow the first time the plugin is
downloaded, and slightly faster on subsequent runs.

[#166004512]
2019-05-30 09:35:12 +01:00
Gerhard Lazu 06bae443f6 Put more pressure on the distribution
This forces the tls_connection and tls_sender processes to be in
runnable & running states more often. waiting is the ideal state.

[#166004512]
2019-05-30 09:30:21 +01:00
Gerhard Lazu f06502ff5a Improve RabbitMQ Overview Grafana dashboard
* pin nodes to specific colours
* add message-related single-stats
* reshuffle rows
  * node metrics are most useful
  * queue, channel & connection churn are least useful
2019-05-29 18:21:25 +01:00
Gerhard Lazu 2645082738 Finish Erlang Distribution Grafana dashboard
Includes Erlang node to colour pinning

Adds a few make targets to help with docker-compose repetitive commands
& Grafana dashboard updates.

Split Overview & Distribution Docker deployments

re deadtrickster/prometheus.erl#92

[finishes #166004512]
2019-05-29 18:19:09 +01:00
Gerhard Lazu c4f0105bad Fix invalid docker-compose
An empty environment is not allowed
2019-05-20 22:21:06 +01:00
Gerhard Lazu 4e81af4cfc Pin RabbitMQ nodes to colours in all Grafana panesl
Regex is greedy, need to look into non-greedy matching, especially for
Erlang Distribution metrics.

[#166004512]
2019-05-20 22:19:43 +01:00
Gerhard Lazu 6a6877cc8b Do not install Grafana plugins all the time
Even if the plugins are stored on a volume, updating them can be slow.
2019-05-20 22:19:02 +01:00
Gerhard Lazu 204f170e7e Bump Grafana, Prometheus & PerfTest versions to latest 2019-05-20 22:18:32 +01:00
Gerhard Lazu d01f96a9af Bump Erlang/OTP to 22.0.1 in Docker image
Picking up the latest 3.8.0 alpha while at it.

It's time to start stress-testing OTP 22 re docker-library/rabbitmq#336

[#166037004]
2019-05-20 21:43:01 +01:00
Gerhard Lazu d1460d5b44 Stress Erlang Distribution metrics on OTP 21
We (+@essen) have answered a bunch of questions (see the story) and
improved the metrics + dashboard in the process. Added some improvements
to the RabbitMQ Overview metrics as well.

[#166004104]
2019-05-20 21:41:27 +01:00
Gerhard Lazu 1f333ebed6 Display the number of Erlang Distribution links
[#166004104]
2019-05-20 10:12:22 +01:00
Jean-Sébastien Pédron 31fd5d3bcf Update rabbitmq-components.mk 2019-05-17 15:20:25 +02:00
Michael Klishin fe6a8fe8f0 Update rabbitmq-components.mk 2019-05-17 08:37:05 +03:00
Michael Klishin dd76a4e244 Update rabbitmq-components.mk 2019-05-17 00:45:49 +03:00
Gerhard Lazu 691af35dfb Mirror slow-consumer-persistent queue to all 3 nodes
This puts load on the distribution and makes the Erlang-Distribution
dashboard show an interesting behaviour in TCP sockets. @dcorbacho
thinks so too.

re deadtrickster/prometheus.erl#92

[#166004512]
2019-05-15 18:49:58 +01:00
Gerhard Lazu f9ce43677b Review metrics with @dcorbacho
[accepts #165831668]
2019-05-15 17:06:42 +01:00
Gerhard Lazu ebde2ff663 Default Erlang Distribution Grafana dashboard to 10 minutes
It's the same as RabbitMQ Overview
2019-05-15 17:05:07 +01:00
Gerhard Lazu f5681155a8 Bump Erlang/OTP version to 21.3.8.1 2019-05-15 14:26:02 +01:00
Gerhard Lazu ff1da6aa86 Fix premature prometheus dep version bump
This version was used to QA deadtrickster/prometheus.erl#92 locally,
didn't intend to push it upstream. Thanks for the spot @dumbbell!
2019-05-15 10:06:07 +01:00
Gerhard Lazu 7652799e05 Add Grafana dashboard for Erlang Distribution
Just the first version, imperfect in many ways, but better than nothing.

[#166004512]
2019-05-14 16:17:04 +01:00
Michael Klishin 37ec1d50c2 Update rabbitmq-components.mk 2019-05-14 11:50:59 +03:00
Michael Klishin 0bd741f760 Update rabbitmq-components.mk 2019-05-13 16:45:49 +03:00
Gerhard Lazu 6272f30724 Missed get empty Grafana panel
[#165831668]
2019-05-09 17:38:34 +01:00
Gerhard Lazu c596efb58e Review all metrics to ETS mappings
Clarify descriptions, improve metric names, fix typos etc. Follow-up to
deadtrickster/prometheus_rabbitmq_exporter#75.

Helpful metric descriptions

* https://www.rabbitmq.com/monitoring.html
* https://docs.signalfx.com/en/latest/integrations/integrations-reference/integrations.rabbitmq.html
* https://github.com/rabbitmq/rabbitmq-common/blob/master/include/rabbit_core_metrics.hrl
* https://github.com/rabbitmq/rabbitmq-common/blob/master/src/rabbit_core_metrics.erl

Thanks for the pair-up @michaelklishin!

[finishes #165831668]
2019-05-09 17:25:17 +01:00
Gerhard Lazu b731233ee6 Fix top-level app reference in README config example 2019-05-09 13:22:00 +01:00
Gerhard Lazu 8186847754 Increase msg redelivered / s thresholds to 20 & 100 2019-05-07 16:29:57 +01:00
Gerhard Lazu 982b1c798d Force connection, channel & queue churn 2019-05-07 16:25:49 +01:00
Gerhard Lazu 5b7a2edc62 Add redelivered thresholds & make unroutable panels red
Warn at 2 redelivered msg/s & critical at 10 redelivered msg/s
2019-05-07 15:40:37 +01:00
Gerhard Lazu ceffc587a1 Add consumer that nacks messages every 0.5s
re rabbitmq/rabbitmq-perf-test#204
2019-05-07 15:40:31 +01:00
Gerhard Lazu d78cb66435 Fix basic.get metrics on Grafana dashboard
Use 1m instead of $__interval for rates that track metrics with slow
rate of change. Using $__interval will miss changes.

Stop rounding, it skews values.

All `basic.get` metrics are bad. The 0 threshold and the red colour for
all lines is hopefully enought to convey this.

re rabbitmq/rabbitmq-perf-test#203

[finishes #165852775]
2019-05-07 13:03:41 +01:00
Gerhard Lazu e506502bf9 Bump Erlang to latest stable, 21.3.8
Add make target to quickly find the latest OTP version & SHA256
checksum.
2019-05-07 12:55:52 +01:00
Gerhard Lazu ce89fdbcd1 Use latest PerfTest, with consumer rate fix
rabbitmq/rabbitmq-perf-test#202
2019-05-02 17:58:39 +01:00
Gerhard Lazu 9d6708ebab Use full descriptions in titles
Otherwise it's really hard to know what we are looking at when expanding
panels.

Also, pin to colours. Otherwise, rabbit@rabbitmq1 metrics in one panel
will appear yellow, and green in another panel. This is a one-off
which doesn't scale, should be automated in some way. Grafana doesn't
support pinning colors to labels 🤔
2019-05-02 17:33:47 +01:00
Gerhard Lazu d44c0edf35 Fix rate interval for / s metrics
This explains why we want rate() instead of irate() and a 1m interval
for smaller changes & $__interval for higher ones:
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusRateVsIrate

[finishes #164374397]
2019-05-02 14:43:09 +01:00
Gerhard Lazu 250136e4a7 Expose queue created metric, fix queue deleted metric
[#164374397]
2019-05-02 12:17:14 +01:00
Gerhard Lazu 7206cd31f1 Expose Erlang distribution port 2019-05-01 18:18:16 +01:00
Gerhard Lazu 3ec84b3bbd Add object churn panels
queue_created is not currently exported via rabbitmq_prometheus.

[#164374397]
2019-05-01 18:16:49 +01:00