grafana/docs/sources/alerting/fundamentals/_index.md

151 lines
8.1 KiB
Markdown
Raw Permalink Normal View History

---
aliases:
Alerting docs: restructure `Introduction` (#84248) * Rename `Data sources` title * Relocate and rename `Introduction/Notification templates` * Rename `alert-rules/alert-instances` to `alert-rules/multi-dimensional-alerts` * Move `fundamentals/high-availability` to `setup/enable-ha` * Fix 404 high-availability alerting link on Setup HA Grafana docs * Move alert manager/contact poitns/notification templates within Notifications * Remove `Alerting on numeric data` * Restructure Introduction v2 * Continue Intro restructuring * Update docs/sources/alerting/fundamentals/alert-rules/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Complete contact point TODO * Alias: alertManager * Aliases `annotation-label` + content changes * Aliases to `templating-labels-annotations` * Aliases to `queries-conditions` * Rename `rule-evaluation.md` file * Aliases: `contact points` * Aliases to `message-templating` * Aliases to `alert-rules` * Update links to new URL slugs * Remove duplicated alias * Remove trailing slash for external heading links * Remove trailing slash in heading links to other grafana pages * Change URL directory slug `fundamentals/notifications` * rename title `Configure High Availability` * Content changes * Update docs/sources/alerting/fundamentals/alert-rules/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-alert-state-history/index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-high-availability/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-alert-state-history/index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-high-availability/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-high-availability/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/set-up/configure-high-availability/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/fundamentals/alert-rules/_index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Fix broken link reference * Fix `queries-and-conditions` * Fix `alert-rule-evaluation` ref link * Fix aliases + inline doc comments * Fix broken link --------- Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
2024-03-14 23:58:18 +08:00
- ./metrics/ # /docs/grafana/<GRAFANA_VERSION>/alerting/metrics/
- ./unified-alerting/fundamentals/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/fundamentals/
canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/
description: Learn about the fundamentals of Grafana Alerting as well as the key features it offers
Explicitly set all front matter labels in the source files (#71548) * Set every page to have defaults of 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration pages to have of 'Cloud', 'Enterprise', and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/enterprise-licensing pages to have 'Enterprise' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/organization-management pages to have 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/provisioning pages to have 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/recorded-queries pages to have labels cloud,enterprise * Set administration/roles-and-permissions/access-control pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/stats-and-license pages to have labels cloud,enterprise * Set alerting pages to have labels cloud,enterprise,oss * Set breaking-changes pages to have labels cloud,enterprise,oss * Set dashboards pages to have labels cloud,enterprise,oss * Set datasources pages to have labels cloud,enterprise,oss * Set explore pages to have labels cloud,enterprise,oss * Set fundamentals pages to have labels cloud,enterprise,oss * Set introduction/grafana-cloud pages to have labels cloud Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix introduction pages products Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set panels-visualizations pages to have labels cloud,enterprise,oss * Set release-notes pages to have labels cloud,enterprise,oss * Set search pages to have labels cloud,enterprise,oss * Set setup-grafana/configure-security/audit-grafana pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/configure-authentication pages to have labels cloud,enterprise,oss * Set setup-grafana/configure-security/configure-authentication/enhanced-ldap pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-authentication/saml pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-database-encryption/encrypt-secrets-using-hashicorp-key-vault pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-request-security pages to have labels cloud,enterprise,oss Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/configure-team-sync pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/export-logs pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set troubleshooting pages to have labels cloud,enterprise,oss * Set whatsnew pages to have labels cloud,enterprise,oss * Apply updated labels from review Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> Co-authored-by: Isabel <76437239+imatwawana@users.noreply.github.com> --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> Co-authored-by: Isabel <76437239+imatwawana@users.noreply.github.com>
2023-07-18 16:10:12 +08:00
labels:
products:
- cloud
- enterprise
- oss
menuTitle: Introduction
title: Introduction to Grafana Alerting
weight: 100
refs:
notifications:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/
notification-policies:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
silences:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
alert-rules:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/
contact-points:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/
mute-timings:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings/
external-alertmanagers:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager/
alert-rule-evaluation:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/rule-evaluation/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/rule-evaluation/
group-alert-notifications:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/
multi-dimensional-alerts-example:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/best-practices/multi-dimensional-alerts/
tutorials:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/tutorials/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/best-practices/tutorials/
---
Docs: Cleanup alerting documentation, part 1 (#40737) * First commit. * Adding shared content. * More changes. * More changes * Updated few more topics, fixed broken relrefs. * Checking in changes. * Some more topics scrubbed. * Minor update. * Few more changes. * Index pages are finally somewhat sorted. Added relevant information and new topics. * Updated Alert grouping. * Last bunch of changes for today. * Updated folder names, relrefs, and some topic weights. * Fixed typo in L37, notifications topic. * Fixed another typo. * Run prettier. * Fixed remaining broken relrefs. * Minor reorg, added link to basics some overview topic. * Some more re-org of the basics section. * Some more changes. * More changes. * Update docs/sources/shared/alerts/grafana-managed-alerts.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/_index.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/_index.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/opt-in.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/notification-policies.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alert-groups.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/_index.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Ran prettier and applied suggestion from code review. * Update docs/sources/alerting/unified-alerting/message-templating/_index.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Change from code review. Also fixed typo "bos" in playlist topic. * Ran prettier to fix formatting issues. * Update docs/sources/alerting/unified-alerting/alerting-rules/edit-cortex-loki-namespace-group.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/alertmanager.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/alertmanager.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/evaluate-grafana-alerts.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * More changes from code review. * Replaced drop down with drop-down * Fix broken relrefs * Update docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-rule.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/rule-list.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Few more. * Couple more. Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com>
2021-10-28 04:57:54 +08:00
# Introduction to Grafana Alerting
Docs: Cleanup alerting documentation, part 1 (#40737) * First commit. * Adding shared content. * More changes. * More changes * Updated few more topics, fixed broken relrefs. * Checking in changes. * Some more topics scrubbed. * Minor update. * Few more changes. * Index pages are finally somewhat sorted. Added relevant information and new topics. * Updated Alert grouping. * Last bunch of changes for today. * Updated folder names, relrefs, and some topic weights. * Fixed typo in L37, notifications topic. * Fixed another typo. * Run prettier. * Fixed remaining broken relrefs. * Minor reorg, added link to basics some overview topic. * Some more re-org of the basics section. * Some more changes. * More changes. * Update docs/sources/shared/alerts/grafana-managed-alerts.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/_index.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/_index.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/opt-in.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/notification-policies.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alert-groups.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/_index.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Ran prettier and applied suggestion from code review. * Update docs/sources/alerting/unified-alerting/message-templating/_index.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Change from code review. Also fixed typo "bos" in playlist topic. * Ran prettier to fix formatting issues. * Update docs/sources/alerting/unified-alerting/alerting-rules/edit-cortex-loki-namespace-group.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/alertmanager.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/alertmanager.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/basics/evaluate-grafana-alerts.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/contact-points.md Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> * More changes from code review. * Replaced drop down with drop-down * Fix broken relrefs * Update docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-rule.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Update docs/sources/alerting/unified-alerting/alerting-rules/rule-list.md Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com> * Few more. * Couple more. Co-authored-by: Fiona Artiaga <89225282+GrafanaWriter@users.noreply.github.com> Co-authored-by: Eve Meelan <81647476+Eve832@users.noreply.github.com>
2021-10-28 04:57:54 +08:00
Grafana Alerting lets you define alert rules across multiple data sources and manage notifications with flexible routing.
Built on the Prometheus alerting model, it integrates with the Grafana stack to provide a scalable and effective alerting setup across a wide range of environments.
{{< admonition type="tip" >}}
For a hands-on introduction, refer to [Getting started with Grafana Alerting tutorials](ref:tutorials).
{{< /admonition >}}
## How it works at a glance
{{< figure src="/media/docs/alerting/alerting-configure-notifications-v2.png" max-width="750px" alt="How Grafana Alerting works" >}}
1. Grafana Alerting periodically evaluates alert rules by executing their data source queries and checking their conditions.
1. Each alert rule can produce multiple alert instances—one per time series or dimension.
1. If a condition is breached, an alert instance fires.
1. Firing (and resolved) alert instances are sent for notifications, either directly to a contact point or through notification policies for more flexibility.
## Fundamentals
The following concepts are key to your understanding of how Grafana Alerting works.
### Alert rules
An [alert rule](ref:alert-rules) consists of one or more queries and expressions that select the data you want to measure. It also contains a condition, which is the threshold that an alert rule must meet or exceed to fire.
In the alert rule, choose the contact point or notification policies to determine how to receive the alert notifications.
### Alert rule evaluation
[Alert rules are frequently evaluated](ref:alert-rule-evaluation) and the state of their alert instances is updated accordingly. Only alert instances that are in a firing or resolved state are sent in notifications.
### Alert instances
Each alert rule can produce multiple alert instances (also known as alerts) - one alert instance for each time series or dimension. This is allows you to observe multiple resources in a single expression.
```promql
sum by(cpu) (
rate(node_cpu_seconds_total{mode!="idle"}[1m])
)
```
A rule using the PromQL expression above creates as many alert instances as the amount of CPUs after the first evaluation, enabling a single rule to report the status of each CPU.
{{< figure src="/static/img/docs/alerting/unified/multi-dimensional-alert.png" alt="Multiple alert instances from a single alert rule" >}}
_For a demo, see the [multi-dimensional alerts example](ref:multi-dimensional-alerts-example)._
### Contact points
[Contact points](ref:contact-points) determine the notification message and where notifications are sent. For example, you might have a contact point that sends notifications to an email address, to Slack, to an incident management system (IRM) such as Grafana IRM or PagerDuty, or to a webhook.
### Notification messages
By default, notification messages include alert details, such as the number of alerts, their status, and annotations to help responders address alert issues. Notification messages can also be customized.
In the alert rule, you can choose a contact point to receive the alert notifications or use notification policies instead.
### Notification policies
[Notification policies](ref:notification-policies) are an advanced option for handling alert notifications by distinct scopes, such as by team or service—ideal for managing large alerting systems.
Notification policies routes alerts to contact points via label matching. They are defined in a tree structure, where the root of the notification policy tree is the **Default notification policy**, which ensures all alert instances are handled.
{{< figure src="/media/docs/alerting/notification-routing.png" max-width="750px" alt="A diagram displaying how the notification policy tree routes alerts" caption="Routing firing alert instances through notification policies" >}}
<br/>
Each notification policy decides where to send the alert (contact point) and when to send the notification (timing options).
### Notification grouping
To reduce alert noise, Grafana Alerting [groups related firing alerts into a single notification](ref:group-alert-notifications) by default. You can customize this behavior in the alert rule or notification policy settings.
### Silences and mute timings
[Silences](ref:silences) and [mute timings](ref:mute-timings) allow you to pause notifications without interrupting alert rule evaluation. Use a silence to pause notifications on a one-time basis, such as during a maintenance window; and use mute timings to pause notifications at regular intervals, such as evenings and weekends.