Loki stores your logs and only indexes labels for each log stream. Using Loki with Grafana Alerting is a powerful way to keep track of what's happening in your environment. You can create metric alerts based on content in your log lines to notify your team. Even better, you can add label data from the log message directly into your alert notification.
In this tutorial, you'll:
- Create a conditional alert using Loki.
- Create a custom alert message template.
- Configure an email notification that includes part of the log message.
## Before you begin
- Ensure you’re on Grafana 8 or later with [Grafana Alerting](https://grafana.com/docs/grafana/latest/alerting/set-up/migrating-alerts/) enabled.
- Ensure you’ve [configured a Loki datasource](https://grafana.com/docs/grafana/latest/datasources/loki/#configure-the-data-source) in Grafana.
- If you already have logs to work with, you can skip the optional sections and go straight to [create an alert](#create-an-alert).
- If you want to use a log-generating sample script to create the logs demonstrated in this tutorial, refer to the optional steps:
- [Use promtail and log-generating script](#optional-use-promtail-and-a-python-script-to-create-sample-logs-and-send-them-to-loki)
- [Use docker with promtail and the log-generating script](#optional-running-the-tutorial-using-grafana-loki-and-promtail-with-docker-compose)
## Create an alert
In these steps you'll create an alert and define an expression to evaluate. These examples use a classic condition.
1. Choose **Grafana Managed Alert** to create an alert that uses expressions.
1. Select your Loki datasource from the drop-down.
1. Enter the alert query in the query editor, switch to **code** mode in the top right corner of the editor to paste the query below:
```
sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != `status=200` | pattern `<_> <message> duration<_>` [10m]))
```
This query will count the number of log lines with a status code that is not 200 (OK), then sum the result set by message type using an **instant query** and the time interval indicated in brackets. It uses the logql pattern parser to add a new label called `message` that contains the level, method, url, and status from the log line.
You can use the **explain query** toggle button for a full explanation of the query syntax. The optional log-generating script creates a sample log line similar to the one below:
{{% admonition type="note" %}}If you're using your own logs, modify the logql query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
{{% / admonition %}}
1. Update the default expressions to match the values shown in the tables below:
**Box B - reduce expression**
| | |
| -------- | ------ |
| Function | Sum |
| Input | A |
| Mode | Strict |
**Box C - threshold expression**
| | |
| ---------------- | --------------------------|
| Input | B |
| Expression value | Is above 5 |
| Alert condition |This is the alert condition|
1. Expand **Options** and select **Instant** as the query type.
1. Click **preview** to see a preview of the query result and alert evaluation.
1. Expression B shows a table of labels and values returned. The message label captured the message string from the log line
and the value shows the number of times that string occurred during the evaluation interval.
- Choose a folder or use **+add new** to add a new folder for this alert.
- Select an existing evaluation group from the drop-down or create a new one if this is your first alert.
- Set the **for** value to **0s** so the alert will fire instantly.
- Leave Configure no data and error handling No data handling on the default values.
1. Add an annotation that refers to labels and values from the query result in your alert notification.
- Choose **+Add new** in the drop down and type the annotation name **AlertValues** into the blank box.
- In the blank `text` box paste `{{ $labels.message }} has returned an error status {{$values.B}} times.`
1. Click the **Save and exit** button at the top of the alert definition page.
### Create a Loki managed alert
[Loki managed alerts](https://grafana.com/docs/loki/latest/rules/#alerting-and-recording-rules) are stored and evaluated by Loki. They use LogQL for their expressions.
1. Choose Mimir or Loki managed alert to create an alert using Loki.
1. Select your Loki data source from the drop-down.
1. The optional script will output a sample log line similar to this:
1. Enter the alert query below if you’re using the sample logs or modify it for your own file path and condition.
```
sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != `status=200` | pattern `<_> <message> duration<_>` [5m])) > 5
```
This query will search the interval period and count the number of log lines with a status code that is not 200 (OK), then sum the result set by message type. It uses the logql pattern parser to add a new label called `message` that captured the level, method, url, and status from the log line.
For loki alerts, the interval needs to be specified in brackets instead of a variable and the alert threshold is added to the query. For this example, the interval is 5m and the alert will fire if there are more than 5 non-200 status messages.
1. Click **preview alert** to see a preview of the labels and value. Hover over the **i** icon under the info column to see the query values.
1. Add an annotation that refers to labels and values from the query result in your alert notification.
- Choose **+Add new** in the drop down and type the annotation name **AlertValues** into the blank box.
- In the blank `text` box, paste the following:
```
{{ $labels.message }} has returned an error status {{$values.B}} times
```
1. Click **Save rule and exit** at the top of the alert screen.
## Create a message template
1.**Add an alert message template** and reference the annotation from your alert.
- In Alerting under the Contact points tab:
- Choose **Grafana** to use the built-in alertmanager
- Click **+Add template**
- Name the template `mynotification`
- Add the snippet below to your alert template in the **Content** field. Notice that you will reference the annotation from your alert by name `(.Annotations.AlertValues)` to insert the annotation string into the alert notification:
```
{{ define "myalert" }}
[{{.Status}}] {{ .Labels.alertname }}
{{ .Annotations.AlertValues }}
{{ end }}
{{ define "mymessage" }}
{{ if gt (len .Alerts.Firing) 0 }}
{{ len .Alerts.Firing }} firing:
{{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
{{ len .Alerts.Resolved }} resolved:
{{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
{{ end }}
{{ end }}
```
- There are two sections to the notification template:
1. The `myalert` template creates a single alert notification based on a specific alert.
1. The `mymessage` template will find all of the grouped alerts that are firing and send them in a single notification.
- Save the template.
1. Add the template to your contact point
1. Navigate to **Alerts > Contact point** and edit the email contact point. If you're using Grafana Cloud, SMTP is already enabled. Otherwise, for local installations you'll need to [configure SMTP](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#smtp).
1. Add an email address in the to field for the recipient.
1. Expand Optional Email Settings and refer to the template by adding this to the body field:
```
{{ template "mynotification" . }}
```
**Tada! You're finished!** Grafana will email an alert with a message that looks similar to the one below. The format varies slightly depending on which type of alert you created - Loki or Grafana managed. The contents should be the same:
## Optional: Use promtail with a sample log-generating script
This optional step uses a python script to generate the sample logs used in this tutorial to create alerts.
1. [Install promtail](https://grafana.com/docs/loki/latest/clients/promtail/installation/) on your local machine and configure it to send logs to your Loki instance.
1. Install Python3 on your local machine if needed.
1. Copy the python script below and paste it into a new file on your local machine.
