Add snippets for 'Create log alert rules with Grafana Alerting' learning journey (#109059)

This commit is contained in:
Jack Baldry 2025-08-01 16:57:18 +01:00 committed by GitHub
parent 4287734114
commit 0faa03edbe
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 32 additions and 2 deletions

View File

@ -67,6 +67,8 @@ By default, notification messages include a link to the dashboard panel. Additio
## Create alert rules from panels
{{< shared id="create-alert-from-panel" >}}
To streamline alert creation, you can create an alert rule directly from a panel.
1. Navigate to a dashboard in the **Dashboards** section.
@ -77,6 +79,8 @@ To streamline alert creation, you can create an alert rule directly from a panel
- Sets the alert rule query using the panel query.
1. Complete the alert rule configuration and click **Save rule** to initiate the alert rule.
{{< /shared >}}
You can then [view the alert state on the panel](ref:view-alert-state-on-panels).
By default, notification messages include a link to the dashboard panel. Additionally, you can [enable displaying panel screenshots in notifications](ref:images-in-notifications).

View File

@ -21,6 +21,8 @@ This section provides a set of guides and examples of best practices for Grafana
Designing and configuring an alert management set up that works takes time. Here are some additional tips on how to create an effective alert management set up:
{{< shared id="alert-planning-fundamentals" >}}
**Which are the key metrics for your business that you want to monitor and alert on?**
- Find events that are important to know about and not so trivial or frequent that recipients ignore them.
@ -44,3 +46,5 @@ Designing and configuring an alert management set up that works takes time. Here
- Avoid noisy, unnecessary alerts by using silences, mute timings, or pausing alert rule evaluation.
- Continually tune your alert rules to review effectiveness. Remove alert rules to avoid duplication or ineffective alerts.
- Continually review your thresholds and evaluation rules.
{{< /shared >}}

View File

@ -81,30 +81,40 @@ Alert instances are routed for [notifications](ref:notifications) in two scenari
## Evaluation group
Every alert rule and recording rule is assigned to an evaluation group.
{{< shared id="evaluation-group-basics" >}}
Each evaluation group contains an **evaluation interval** that determines how frequently the rule is checked. For instance, the evaluation may occur every `10s`, `30s`, `1m`, `10m`, etc.
Every alert rule and recording rule is assigned to an evaluation group. Each evaluation group contains an **evaluation interval** that determines how frequently the rule is checked. For instance, the evaluation may occur every `10s`, `30s`, `1m`, `10m`, etc.
{{< /shared >}}
Rules can be evaluated concurrently or sequentially. For details, see [How rules are evaluated within a group](ref:evaluation-within-a-group).
## Pending period
{{< shared id="pending-period-basics" >}}
You can set a **Pending period** to prevent unnecessary notifications caused by temporary issues.
When the alert condition is met, the alert instance enters the **Pending** state. It remains in this state until the condition has been continuously true for the entire **Pending period**.
This ensures the condition breach is stable before the alert transitions to the **Alerting** state and routed for notification.
{{< /shared >}}
- **Normal** -> **Pending** -> **Alerting**<sup>\*</sup>
You can also set the **Pending period** to zero to skip the **Pending** state entirely and transition to **Alerting** immediately.
## Keep firing for
{{< shared id="keep-firing-for" >}}
You can set a **Keep firing for** period to avoid repeated firing-resolving-firing notifications caused by flapping conditions.
When the alert condition is no longer met during the **Alerting** state, the alert instance enters the **Recovering** state.
{{< /shared >}}
- **Alerting****Recovering****Normal (Resolved)**<sup>\*</sup>
- After the **Keep firing for** period elapses, the alert transitions to the **Normal** state and is marked as **Resolved**.
- If the alert condition is met again, the alert transitions back to the **Alerting** state, and no new notifications are sent.

View File

@ -62,10 +62,14 @@ Labels and annotations add additional information about an alert using key/value
## Labels
{{< shared id="labels-basics" >}}
**Labels** are unique identifiers of an [alert instance](ref:alert-instances). You can use them for searching, silencing, and routing notifications.
Examples of labels are `server=server1` or `team=backend`. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.
{{< /shared >}}
For example, an alert instance might have the label set `{alertname="High CPU usage",server="server1"}` while another alert instance might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alert instances because although their `alertname` labels are the same, their `server` labels are different.
{{< figure alt="Image shows an example of an alert instance and the labels used on the alert instance." src="/static/img/docs/alerting/unified/multi-dimensional-alert.png" >}}
@ -134,6 +138,8 @@ If multiple label keys are sanitized to the same value, the duplicates have a sh
## Annotations
{{< shared id="annotations-basics" >}}
Annotations add additional information to alert instances, helping responders identify and address potential issues.
Create clear and self-explanatory annotations so that first responders can investigate without needing deeper knowledge of the alert setup.
@ -145,6 +151,8 @@ Annotations are displayed in Grafana and are included by default in notification
- `runbook_url`: The runbook page to guide operators managing a potential incident.
- `__dashboardUid__` and `__panelId__`: [Link the alert to a dashboard and panel](ref:link-alert-rules-to-panels) to facilitate alert investigation.
{{< /shared >}}
For example, you can edit the annotation `summary` to explain why the alert was triggered:
```

View File

@ -83,10 +83,14 @@ Start defining your [contact points](ref:contact-points) to specify how to recei
### Contact points
{{< shared id="contact-points-fundamentals" >}}
[Contact points](ref:contact-points) contain the configuration for sending alert notifications, specifying destinations like email, Slack, IRM, webhooks, and their notification messages.
A contact point is a list of integrations, each sending a message to a specific destination.
{{< /shared >}}
By default, notification messages include common alert details, such as the number of alerts, alert names, labels, annotations, and other alert information. You can also customize notification messages and use notification templates.
First, create the contact point and test the notifications. Then, configure the alert rule to send its notifications to either a contact point or through Notification Policies.