diff --git a/docs/sources/administration/configuration.md b/docs/sources/administration/configuration.md index 27ebf32c7e6..896b8935c08 100644 --- a/docs/sources/administration/configuration.md +++ b/docs/sources/administration/configuration.md @@ -1529,6 +1529,30 @@ For example: allowed_origins = "https://*.example.com" ``` +### ha_engine + +> **Note**: Available in Grafana v8.1 and later versions. + +**Experimental** + +The high availability (HA) engine name for Grafana Live. By default, it's not set. The only possible value is "redis". + +For more information, refer to [Configure Grafana Live HA setup]({{< relref "../live/live-ha-setup.md" >}}). + +### ha_engine_address + +> **Note**: Available in Grafana v8.1 and later versions. + +**Experimental** + +Address string of selected the high availability (HA) Live engine. For Redis, it's a `host:port` string. Example: + +```ini +[live] +ha_engine = redis +ha_engine_address = 127.0.0.1:6379 +``` +
## [plugin.grafana-image-renderer] diff --git a/docs/sources/live/live-ha-setup.md b/docs/sources/live/live-ha-setup.md index 1569b1d0dcc..1a3024198ca 100644 --- a/docs/sources/live/live-ha-setup.md +++ b/docs/sources/live/live-ha-setup.md @@ -7,10 +7,52 @@ weight = 130 # Configure Grafana Live HA setup -Live features in Grafana v8.0 are designed to work with a single Grafana server instance only. We will add the option for HA configuration in future Grafana releases to eliminate the current limitations. +By default, Grafana Live uses in-memory data structures and in-memory PUB/SUB hub for handling subscriptions. -Currently, if you have several Grafana server instances behind a load balancer, you may come across the following limitations: +In a high availability Grafana setup involving several Grafana server instances behind a load balancer, you can find the following limitations: - Built-in features like dashboard change notifications will only be broadcasted to users connected to the same Grafana server process instance. - Streaming from Telegraf will deliver data only to clients connected to the same instance which received Telegraf data, active stream cache is not shared between different Grafana instances. - A separate unidirectional stream between Grafana and backend data source may be opened on different Grafana servers for the same channel. + +To bypass these limitations, Grafana v8.1 has an experimental Live HA engine that requires Redis to work. + +## Configure Redis Live engine + +When the Redis engine is configured, Grafana Live keeps its state in Redis and uses Redis PUB/SUB functionality to deliver messages to all subscribers throughout all Grafana server nodes. + +Here is an example configuration: + +``` +[live] +ha_engine = redis +ha_engine_address = 127.0.0.1:6379 +``` + +After running: + +- All built-in real-time notifications like dashboard changes are delivered to all Grafana server instances and broadcasted to all subscribers. +- Streaming from Telegraf delivers messages to all subscribers. +- A separate unidirectional stream between Grafana and backend data source opens on different Grafana servers. Publishing data to a channel delivers messages to instance subscribers, as a result, publications from different instances on different machines do not produce duplicate data on panels. + +At the moment we only support single Redis node. + +> **Note:** It's possible to use Redis Sentinel and Haproxy to achieve a highly available Redis setup. Redis nodes should be managed by [Redis Sentinel](https://redis.io/topics/sentinel) to achieve automatic failover. Haproxy configuration example: +> ``` +> listen redis +> server redis-01 127.0.0.1:6380 check port 6380 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions +> server redis-02 127.0.0.1:6381 check port 6381 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 backup +> bind *:6379 +> mode tcp +> option tcpka +> option tcplog +> option tcp-check +> tcp-check send PING\r\n +> tcp-check expect string +PONG +> tcp-check send info\ replication\r\n +> tcp-check expect string role:master +> tcp-check send QUIT\r\n +> tcp-check expect string +OK +> balance roundrobin +> ``` +> Next, point Grafana Live to Haproxy address:port.