Alert Rules and Notifications

Grafana ships with built-in support for a number of data sources including CloudWatch, Prometheus, Grafana datasource for Prometheus Alertmanager, just to mention a few. Prometheus is an open-source systems monitoring and alerting toolkit that collects and stores its metrics as time-series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Each data source has a specific Query Editor that is customized for the features and capabilities that the particular data source exposes. The query language and capabilities of each data source are obviously very different. You can combine data from multiple data sources onto a single Dashboard, but each Panel is tied to a specific data source that belongs to a particular Organization.

Grafana dashboard alerts have two main components:

Alert rule - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.
Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.

You can perform the following actions for alerts:

Create an alert rule
View existing alert rules and their current state
Testing alert rules and troubleshooting
Alert Notification Channels

Create Alerts

Grafana alerting allows you to attach rules to your dashboard panels. When you save the dashboard, Grafana extracts the alert rules into a separate alert rule storage and schedules them for evaluation.

In the Alert tab of the graph panel you can configure how often the alert rule should be evaluated and the conditions that need to be met for the alert to change state and trigger its notifications.

Adding an Alert

Alerts are added and configured in the Alert Tab of any dashboard graph panel, letting you build and visualize an alert using existing queries.

To persist your alert rule changes remember to save the dashboard.

Follow these steps to add or edit an alert rule:

Navigate to the panel you want to add or edit an alert rule for, click the title, and then click Edit.
On the Alert tab, click Create Alert. If an alert already exists for this panel, then you can just edit the fields on the Alert tab.
Fill out the fields. Descriptions are listed below in Alert rule fields.
When you have finished writing your rule, click Save in the upper right corner to save alert rule and the dashboard.
(Optional but recommended) Click Test rule to make sure the rule returns the results you expect.

Note: To delete an alert, scroll to the bottom of the alert and then click Delete.

View existing alert rules

Grafana stores individual alert rules in the panels where they are defined, but you can also view a list of all existing alert rules and their current state.

In the Grafana sidebar, hover your cursor over the Alerting (bell) icon and then click Alert Rules.

All configured alert rules are listed, along with their current state.

You can do several things while viewing alerts:

Filter alerts by name - Type an alert name in the Search alerts field.
Filter alerts by state - In States, select which alert states you want to see. All others will be hidden.
Pause or resume an alert - Click the Pause or Play icon next to the alert to pause or resume evaluation. See Pause an alert rule for more information.
Access alert rule settings - Click the alert name or the Edit alert rule (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don’t know which panel it is defined in.

Pause an alert rule

Pausing the evaluation of an alert rule can sometimes be useful. For example, during a maintenance window, pausing alert rules can avoid triggering a flood of alerts.

In the Grafana sidebar, hover your cursor over the Alerting (bell) icon and then click Alert Rules. All configured alert rules are listed, along with their current state.
Find your alert in the list, and click the Pause icon on the right. The Pause icon turns into a Play icon.
Click the Play icon to resume evaluation of your alert.

Troubleshoot Alerts

If alerts are not behaving as you expect, here are some steps you can take to troubleshoot and figure out what is going wrong.

The first level of troubleshooting you can do is click Test Rule. You will get results back that you can expand to the point where you can see the raw data that was returned from your query.

To do this, click on Edit Alert Rule,

Then click on Test Rule as shown here:

Testing Rule:

Alert Notification Channels

When an alert changes state, it sends out notifications. Each alert rule can have multiple notifications. In order to add a notification to an alert rule, you first need to add and configure a notification channel (can be email, PagerDuty, or other integration).

This is done from the Notification channels page.

Add a notification channel

In the Grafana sidebar, hover your cursor over the Alerting (bell) icon and then click Notification channels.
Click New channel.
Fill out the fields or select the options described below.

New notification channel fields

Name - Enter a name for this channel. It will be displayed when users add notifications to alert rules.
Type - Select the channel type. Refer to the List of supported notifiers for details.
Default (send on all alerts) - When selected, this option sends a notification on this channel for all alert rules.
Include Image - See Enable images in notifications for details.
Disable Resolve Message - When selected, this option disables the resolve message [OK] that is sent when the alerting state returns to false.
Send reminders - When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders should be sent using number of seconds (s), minutes (m) or hours (h), for example 30s, 3m, 5m or 1h.

Important: Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.

Back to top