This tutorial is a continuation of the [Get started with Grafana Alerting - Grouping notifications](http://www.grafana.com/tutorials/alerting-get-started-pt3/) tutorial.
> Note: Some of the templating features in Grafana Alerting discussed in this tutorial are currently available in Grafana Cloud but have not yet been released to the Open Source (OSS) version.
- Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting - Templating](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt4/). It's a fully configured environment with all the dependencies already installed.
1. Change to the directory where you cloned the repository:
<!-- INTERACTIVE exec START -->
```
cd tutorial-environment
```
<!-- INTERACTIVE exec END -->
1. Run the Grafana stack:
<!-- INTERACTIVE ignore START -->
```
docker compose up -d
```
<!-- INTERACTIVE ignore END -->
{{<docs/ignore>}}
<!-- INTERACTIVE exec START -->
```bash
docker-compose up -d
```
<!-- INTERACTIVE exec END -->
{{< /docs/ignore >}}
The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.
<!-- INTERACTIVE ignore START -->
{{<admonitiontype="note">}}
If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
{{</admonition>}}
<!-- INTERACTIVE ignore END -->
{{<docs/ignore>}}
NOTE:
If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
{{< /docs/ignore >}}
<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->
## How templating works
In Grafana, you can use [templates](https://grafana.com/docs/grafana/latest/alerting/fundamentals/templates/) to dynamically pull in specific data about the alert rule. This results in more flexible and informative alert notification messages. You can template either alert rule labels and annotations, or notification templates. Both use the Go template language.
[Labels and annotations](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/) are key fields where templates are applied. One of the main advantages of using templating in annotations is the ability to incorporate dynamic data from queries, allowing alerts to reflect real-time information relevant to the triggered condition. By using templating in annotations, you can customize the content of each alert instance, such as including instance names and metric values, so the notification becomes more informative.
### Notification templates
The real power of templating lies in how it helps you format notifications with dynamic alert data. [Notification templates](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/) let you pull in details from annotations to create clear and consistent messages. They also make it simple to reuse the same format across different contact points, saving time and effort.
Notification templates allow you to customize how information is presented in each notification. For example, you can use templates to organize and format details about firing or resolved alerts, making it easier for recipients to understand the status of each alert at a glance—all within a single notification.
This particular notification template pulls in summary and description annotations for each alert instance and organizes them into separate sections, such as "firing" and "resolved." This way, instead of getting a long list of individual alert notifications, users can receive one well-structured message with all the relevant details grouped together.
This approach is helpful when you want to reduce notification noise, especially in situations where multiple instances of an alert are firing at the same time (e.g., high CPU usage across several instances). You can leverage templates to create a unified, easy-to-read notification that includes all the pertinent details.
<!-- INTERACTIVE page step2.md END -->
<!-- INTERACTIVE page step3.md START -->
## Step 1: Template labels and annotations
Now that we've introduced how templating works, let’s move on to the next step. We guide you through creating an alert rule with a summary and description annotation. In doing so, we incorporate CPU usage and instance names, which we later use in our notification template.
### Create an alert rule
1. Sign in to Grafana:
- **Grafana Cloud** users: Log in via Grafana Cloud.
- **OSS users**: Go to [http://localhost:3000](http://localhost:3000).
1. Create an alert rule that includes a summary and description annotation:
- Enter an **alert rule name**. Name it `High CPU usage`
1.**Define query an alert condition** section:
- Select TestData data source from the drop-down menu.
[TestData](https://grafana.com/docs/grafana/latest/datasources/testdata/) is included in the demo environment. If you’re working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.
- From **Scenario** select **CSV Content**.
- Copy in the following CSV data:
```
region,cpu-usage,service,instance
us-west,88,web-server-1,server-01
us-west,81,web-server-1,server-02
us-east,79,web-server-2,server-03
us-east,52,web-server-2,server-04
```
This dataset simulates a data source returning multiple time series, with each time series generating a separate alert instance.
- Keep Last as the value for the reducer function (`WHEN`), and `IS ABOVE 75` as the threshold value, representing CPU usage above 75% .This is the value above which the alert rule should trigger.
- Choose an **Evaluation interval** (how often the alert will be evaluated). Choose `1m`.
- Set the **pending period** to 0s (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
Select who should receive a notification when an alert rule fires.
- Select a **Contact point**. If you don’t have any contact points, click _View or create contact points_.
1.**Configure notification message** section:
In this step, you’ll configure the **summary** and **description** annotations to make your alert notifications informative and easy to understand. These annotations use templates to dynamically include key information about the alert.
- **Summary** annotation: Enter the following code as the value for the annotation.:
```go
{{- "\n" -}}
Instance: {{ index $labels "instance" }}
{{- "\t" -}} Usage: {{ index $values "A"}}%{{- "\n" -}}
```
This template automatically adds the instance name (from the [$labels](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#labels) data) and its current CPU usage (from [$values["A"]](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#values)) into the alert summary. `\t`: Adds a tab space between the instance name and the value. And, `\n`: Inserts a new line after the value.
Output example:
```
server-01 88
```
This output helps you quickly see which instance is affected and its usage level.
1. Optional: Add a description to help the on-call engineer to better understand what the alert rule is about. Eg. This alert monitors CPU usage across instances and triggers if any instance exceeds a usage threshold of 75%.
1. Click **Save rule and exit**.
Now that we’ve configured an alert rule with dynamic templates for the **summary** annotation, the next step is to customize the alert notifications themselves. While the default notification message includes the summary annotation and works well, it can often be too verbose.
{{<figuresrc="/media/docs/alerting/templated-annotation-alert.png"max-width="1200px"caption="Default email alert notification with templated annotation">}}
To make our alert notifications more concise and tailored to our needs, we’ll create a custom **notification template** that references the summary annotation we just set up. Notification templates are especially useful because they can be reused across multiple contact points, ensuring consistent alert messages.
<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->
## Step 2: Template notifications
In this step, we use a built-in notification template to format alert notifications in a clear and organized way. Notification templates allow us to customize the structure of alert messages, making them easier to read and more relevant.
Without a notification template, the alert messages would include the default Grafana formatting (`default.message`, see image above).
1. From the **Add example** dropdown menu, choose `Print firing and resolved alerts`.
This template prints out alert instances into two sections: **firing alerts** and **resolved alerts**, and includes only the key details for each. In addition, it adds our summary and description annotations.
```
{{- /* Example displaying firing and resolved alerts separately in the notification. */ -}}
{{- /* Edit the template name and template content as needed. */ -}}
{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
{{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
{{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
Summary: {{.Annotations.summary}}
Status: {{ .Status }}
Description: {{.Annotations.description}}
{{ end -}}
```
{{<docs/ignore>}}
Note: Your notification template name (`{{define "<NAME>"}}`) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups.
{{< /docs/ignore >}}
<!-- INTERACTIVE ignore START -->
{{<admonitiontype="note">}}
Your notification template name (`{{define "<NAME>"}}`) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups.
{{</admonition>}}
<!-- INTERACTIVE ignore END -->
Here’s a breakdown of the template:
-`{{ define "custom.firing_and_resolved_alerts" -}}` section: Displays the number of resolved alerts and their summaries, using the `alert.summary_and_description` template to include the summary, status, and description for each alert.
-`.Alerts.Firing` section: Similarly lists the number of firing alert instances and their details.
-`alert.summary_and_description`: This sub-template pulls the summary annotation you configured earlier.
In the **Preview** area, you can see a sample of how the notification would look. Since we’ve already created our alert rule, you can take it a step further by previewing how an actual alert instance from your rule would appear in the notification.
1. Click **Edit payload**.
1. Click **Use existing alert instance**.
You should see our alert rule listed on the left.
1. Click the alert rule.
1. Select an instance.
1. Click **Add alert data to payload**.
The alert instance is added to the bottom of the preview.
{{<figuresrc="/media/docs/alerting/alert-instance-preview-in-template.png"max-width="1200px"caption="Preview of an alert instance in a notification template">}}
1. Click **Save**.
With the notification template ready, the next step is to apply it to your contact point to see it in action.
- Under **Select notification template**, search `custom.firing_and_resolved_alerts`.
- Click **Save**.
1. Save your contact point.
<!-- INTERACTIVE page step5.md END -->
<!-- INTERACTIVE page step6.md START -->
### Receiving notifications
Now that the template has been applied to the contact point, you should receive notifications in the specified contact point.
Note: you might need to pause the alert rule evaluation and resume it to trigger the notification.
{{<figuresrc="/media/docs/alerting/templated-notification-cpu.png"max-width="1200px"caption="Templated email notification for CPU and memory usage">}}
In the screen capture, you can see how the notification template groups the alert instances into two sections: **firing alerts** and **resolved alerts**. Each section includes only the key details for each alert, ensuring the message remains concise and focused. Additionally, the summary and description annotations we created earlier are included, providing affected instance and CPU usage.
<!-- INTERACTIVE page step6.md END -->
<!-- INTERACTIVE page finish.md START -->
## Conclusion
In this tutorial, we learned how to use templating in Grafana Alerting to create dynamic and actionable alert notifications. We explored how to configure alert rules with annotations, design custom notification templates, and apply them to contact points to enhance the clarity and efficiency of alert messages. By organizing alert instances into concise notifications, you can reduce noise and ensure on-call engineers quickly understand and address critical issues.
To deepen your understanding of Grafana’s templating, explore the following resources:
- **Overview of the functions and operators used in templates**:
- [**Notification template reference**](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/reference/): Lists the data available for use in notification templates and explores specific functions.
- [**Alert rule template reference**](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/): Covers the specifics of creating dynamic labels and annotations for alert rules using elements such as variables and functions.
In [Get started with Grafana Alerting: Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) you learn how to dynamically route alerts and link them to dashboards.
In [Get started with Grafana Alerting: Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) you learn how to dynamically route alerts and link them to dashboards.
