Create an alert definition

When you define an alert, you specify what conditions trigger the alert and what notifications it sends.

Types of alerts

You can create alerts that are triggered by metric or attribute values, log files, or anomaly events. The type of alert you create determines how you define the condition or conditions that trigger the alert.

• Metric alerts are triggered when a metric or attribute value crosses a threshold, is not being reported, or meets another condition that you specify. You can define two types of metric alerts:

  • Entity Metric alerts monitor the specified metric or attribute values on one or more entities. For example, you can be alerted when the response time for a website is above a threshold, or when the state of a network device is Critical or Down.

    You can also create a parent-child relationship alert. This alert is triggered on a parent entity when the specified metric or attribute value of a child entity exceeds a threshold. For example, you can be alerted when the temperature of the hardware sensor on a network device is too high.

  • Metric Group alerts are triggered when the aggregated value of a metric crosses a threshold. These alerts are not associated with specific entities or an entity type. Use Metric Group alerts to monitor metrics that are obtained through an integration but are not associated with an entity. You can also create Metric Group alerts if you want know when the aggregated value of a metric crosses a threshold (for example, high latency or high request volume), but you do not want an alert for each individual issue.

• Log alerts monitor incoming log messages, including syslog messages, SNMP traps, audit logs, and Windows event logs. The condition that triggers the alert can be defined based on a combination of factors, including the source IP address, text included in the message, and the severity level.

• Event notifications are triggered when a metric value on one of the specified entities is significantly outside its normal range.

Create a metric or attribute value alert

To create an alert that is triggered based on the value of one or more metrics and attributes, complete the following tasks.

Task 1: Specify the alert details

1. Open the Active Alerts page (Alerts > Active Alerts) or the Alert Settings page (Alerts > Alert Settings).

2. In the upper-right corner, click Create Alert.

   The Create Alert wizard opens.

3. Complete the Details page:

   1. Enter a name to identify the alert and, optionally, enter a description of what the alert does.

   2. (Optional) To help users resolve issues quickly, enter a runbook URL. This URL opens a document (for example, a wiki page) that describes what actions should be taken when the alert is triggered. This link is included in the alert notification.

   3. Specify the alert severity.

   4. If you do not want the alert to be enabled immediately, click the slider to disable it. You can enable it later.

   5. Click Next to open the Conditions page.

Task 2: Specify the condition type and scope

1. Under Condition type, click Metric condition.

2. Under Alert on, select one of the following:

   • To set a threshold on a metric or attribute associated with one or more entities (an entity metric alert), select Entity. Then continue with step 3.

   • To set a threshold on a metric without selecting entities (a metric group alert), select Metric group. Then continue with Task 3.

3. Under Select a scope, select the type of entity. Then specify which entities you want to alert on.

   If you are creating alerting on a parent entity based on the value of a child entity (a relationship alert), select the entity type of the parent relationship.

   To select	Do this
   All entities of the specified type	Under Scope by, click All.
   Specific entities	Under Scope by, click Selected <entityType>. Under Select <entityType>, click the drop-down menu and then click each entity you want to select.
   All entities whose name contains a certain string	Under Scope by, click Contains text. Under Contains text, enter the string that the entity name must include. The case must match. The alert applies to all entities of the specified type whose name includes that string. After the alert is created, it automatically applies to any new entities with that string in the name.
   All entities whose name does not contain a certain string	Under Scope by, click Does not contain text. Under Does not contain text, enter a string that the entity name must not include. The case must match. The alert applies to all entities of the specified type whose name does not include that string. After the alert is created, it automatically applies to any new entities without that string in the name.
   Entities based on the value of one or more metrics or attributes	Under Scope by, click Search query. Click in the Search query field to display the available metrics and attributes. Click a metric or attribute, and then click a value. For example, if the entity type is Website, select features and then rum to select all website entities whose metrics are collected by a RUM script. Repeat as needed to specify multiple metrics or attributes. If you include multiple values for the same metric or attribute, an entity is included if it has either value. For example, if you select healthScore.category:bad and healthScore.category:unknown, the alert includes all entities of the selected type with a health score of bad or unknown. If you include multiple metrics or attributes, an entity is included if it has all metric or attribute values. For example, if you select healthScore.category:bad and features:rum, the alert includes website entities with a health score of bad and whose metrics are collected by a RUM script.

Task 3: Define one or more conditions

Define one or more conditions that trigger the alert. You can define a condition based on:

• The value of a metric
• The value of an attribute
• The value of a metric or an attribute on a child entity

An alert can include multiple types of conditions.

Define a condition based on a metric value

1. In the drop-down menu at the top of the Condition box, select Metric.

2. Under Metric, select the metric whose value determines if this condition is met. Enter part of the metric name to filter the list. For metric descriptions, see Metrics for SolarWinds Observability entities.

   • If you selected an entity type, only the metrics available for that entity type are listed. When you select a metric, a graph shows the value of the metric for up to five of the selected entities. To show or hide an entity, click the entity name in the legend below the graph. To display information about different entities, click Select from Scope and choose the entities.

   • If you selected Metric Group, all metrics are available. When you select a metric, a graph shows the aggregated values for that metric.

3. (Optional) Use an attribute associated with the selected metric to further narrow the scope.

   1. Under Included (for a value that must be present) or Excluded (for a value that cannot be present), click Add.

   2. Select an attribute associated with the selected metric.

   3. Select an operator, and then specify the values that must be present or that cannot be present for the condition to be met:

      • If you select contains, enter a string that is present in the values you want to select.

        For example, if the entity type is Network Device and the metric attribute is sw.collector.Nodes.Display.Name, enter NX to limit the alert to network devices whose display name includes NX.

      • If you select equal to, click one or more values in the third drop-down menu to select those values.

   For example, let's say you are creating an alert to monitor the response time of websites whose metrics are collected by a probe. You want this alert to be triggered only when the probe is sent from the region of Europe. Under Included, click Add. Then select probe.region as the attribute, equal to as the operator, and Europe as the value.

4. Under Trigger when metric is, define the threshold for the metric. Based on your specifications, the alert is triggered when the metric value is above or below a certain level, or when it is not being reported.

5. Under During last, choose the time frame used for evaluating the alert condition, and specify how the values collected during that time frame are aggregated.

   For example, if the time frame is the past hour and the aggregation method is average, the alert would be triggered if the average of the values collected during the past hour crossed the condition threshold.

   After you define a condition, a red dotted line on the graph shows whether any of the entities or the aggregated metric value would trigger the alert during the selected period. In the example below, the Support website would have triggered the alert when the average response time went above 1600 ms.

   

Define a condition based on an attribute value

To see many of the attributes for an entity type, look at the filters available for that entity type in the Entity Explorer. Hover over the to see the source name for that attribute from the collected data.

1. In the drop-down menu at the top of the Condition box, select Attribute.

2. Under Attribute, select the attribute whose value determines if this condition is met.

3. Select an operator, and then specify the values that trigger this alert.

   If you choose the in operator, you can enter multiple values, separated by commas. Enclose strings in quotation marks. For example:

   "Critical", "Unreachable", "Down"

   For all other operators, enter a single value. Do not enclose strings in quotation marks. For example:

   Down

   The selected attribute determines which operators are available. (For example, "higher than" is not available for an attribute whose values are strings.)

   If the attribute value is this data type	These operators are available
   Boolean	equal to not equal to Enter the value as true or false.
   Float Integer	equal to not equal to higher than lower than higher than or equal to lower than or equal to in
   String	equal to not equal to in

Define a condition based on the value of a metric or an attribute on a child entity

1. In the drop-down menu at the top of the Condition box, select Relationship.

   This option is available only if, under Select a scope, you selected an entity type that has child entities (for example, Network Device).

2. Under Relationship, select the child entity type that triggers the alert.

3. Under Scope by, specify which child entities you want to trigger the alert.

   To select	Do this
   All child entities of the specified type	Under Scope by, click All.
   Specific child entities	Under Scope by, click <entityType>. Under Select <entityType>, click the drop-down menu and then click each entity you want to select.
   All entities whose name contains a certain string	Under Scope by, click Contains text. Under Contains text, enter the string that the entity name must include. The case must match. The alert applies to all entities of the specified type whose name includes that string. After the alert is created, it automatically applies to any new entities with that string in the name.
   All entities whose name does not contain a certain string	Under Scope by, click Does not contain text. Under Does not contain text, enter a string that the entity name must not include. The case must match. The alert applies to all entities of the specified type whose name does not include that string. After the alert is created, it automatically applies to any new entities without that string in the name.
   Entities based on the value of one or more metrics or attributes	Under Scope by, click Search query. Click in the Search query field to display the available metrics and attributes. Click a metric or attribute, and then click a value. For example, if the entity type is Website, select features and then rum to select all website entities whose metrics are collected by a RUM script. Repeat as needed to specify multiple metrics or attributes.

4. Under Trigger when count is, specify the number of child entities that must meet the condition to trigger the alert.

   For example, say a network device provides a service to a large number of end users. You don't want an alert every time one interface is down, but you do want an alert if more than 10 are down. Select higher than, and enter 10.

5. Define the condition that must exist on the child entity to trigger the alert. Depending on the type of triggering condition, see:

   • Define a condition based on a metric value
   • Define a condition based on an attribute value

Add another condition

To add another condition, do one of the following:

• Click Add New Condition. Then click All conditions are true (AND) or At least one condition is true (OR).

• To add a condition that is similar to the first one and must also be true, click the vertical ellipsis and then click Duplicate.

  A condition with the same definition is added, connected by the AND operator. Edit the condition as needed.

Task 4: (Optional) Trigger separate alerts based on a property value

Under Group alert data for selected attribute, select a property if you want to trigger a separate alert for each value of that property. The property value is included in the alert notification. In some situations, this additional information can help you troubleshoot the issue that triggered the alert.

For example, let's say you create an alert to monitor the response time for a website, and you select the property probe.city. If the issue occurs in multiple cities, a separate alert is triggered for each city. If both the probe in London and the probe in Tokyo detect a high response time, two alerts are triggered.

The drop-down menu lists the properties of the metric that is selected in the alert condition. If the alert has multiple conditions with different metrics, the drop-down menu lists only properties that are common to all the metrics. If the metrics have no common properties, the drop-down menu is empty.

Task 5: Specify the notifications

Click Next to open the Actions tab, and define one or more actions to be performed when this alert is triggered:

1. Click Add.

2. Select the service that will send the notification or perform an action.

   An action can be, for example, a message sent to a Slack channel, a PagerDuty alert, or a Webhook request.

3. Select one or more configurations to define the action to be performed. To define a new configuration, click Add a New Configuration and enter the required information. (See Notification Services settings for more information about what to enter.)

   To see the details of an existing action or notification, click Settings > Notification Services, and select a service.

4. To send notifications or perform actions using more than one service, click Add. Then select the service and configurations.

5. If you want to notify recipients when the alert condition no longer exists, select Send an additional notification when the Alert is cleared.

6. Click Next.

Task 6: Review

1. On the Summary page, review the alert definition and make changes if necessary.

2. Click Create.

Create a log alert

You can create a log alert from the Active Alerts or Alert Settings page, like other alert types. You can also perform a search in the Logs Explorer, and then create an alert that automatically populates the condition with the current search criteria.

1. Do one of the following to open the alert creation wizard:

   • Start from the Logs Explorer, and use the current search criteria as the condition:

     1. In the left pane, click Logs to open the Logs page.

     2. Enter search criteria in the search box.

     3. Click the Create alert for this search query icon .

        The Create Quick Alert dialog opens.

   • Start from an alerts page:

     1. Open the Active Alerts page (Alerts > Active Alerts) or the Alert Settings page (Alerts > Alert Settings).

     2. In the upper-right corner, click Create Alert.

        The Create Alert wizard opens.

2. Complete the Details page:

   1. Enter a name to identify the alert and, optionally, enter a description of what the alert does.

   2. (Optional) To help users resolve issues quickly, enter a runbook URL. This URL opens a document (for example, a wiki page) that describes what actions should be taken when the alert is triggered. This link is included in the alert notification.

   3. Specify the alert severity.

   4. If you do not want the alert to be enabled immediately, click the slider to disable it. You can enable it later.

   5. Click Next to open the Conditions page.

3. If you do not want the alert to be enabled immediately, click the slider to disable it. You can enable it later from the Alert Settings page.

4. Click Next.

   The Conditions page opens. If you opened the wizard from the Logs Explorer, the condition defaults to the search query you entered there.

5. Under Trigger on following query, add the condition to search for in logs, or verify the condition sent from the Logs Explorer.

   For more information about syntax, see Logs Explorer.

6. Under Trigger when the number of logs is, specify the following, and then click Next:

   • The threshold, or the number of logs that must meet the condition to trigger the alert.

   • The time period to search logs for the specified condition.

   For example, if the threshold is higher than 1 during the last 30 minutes, the alert is triggered if two or more log files meet the condition in a 30-minute time period.

7. Click Next to open the Actions tab, and define one or more actions to be performed when this alert is triggered:

   1. Click Add.

   2. Select the service that will send the notification or perform an action.

      An action can be, for example, a message sent to a Slack channel, a PagerDuty alert, or a Webhook request.

   3. Select one or more configurations to define the action to be performed. To define a new configuration, click Add a New Configuration and enter the required information. (See Notification Services settings for more information about what to enter.)

      To see the details of an existing action or notification, click Settings > Notification Services, and select a service.

   4. To send notifications or perform actions using more than one service, click Add. Then select the service and configurations.

   5. If you want to notify recipients when the alert condition no longer exists, select Send an additional notification when the Alert is cleared.

   6. Click Next.

8. On the Summary page, review the alert definition, and then click Create.

Alert on anomaly events

1. Open the Active Alerts page (Alerts > Active Alerts) or the Alert Settings page (Alerts > Alert Settings).

2. In the upper-right corner, click Create Alert.

   The Create Alert wizard opens.

3. Complete the Details page:

   1. Enter a name to identify the alert and, optionally, enter a description of what the alert does.

   2. (Optional) To help users resolve issues quickly, enter a runbook URL. This URL opens a document (for example, a wiki page) that describes what actions should be taken when the alert is triggered. This link is included in the alert notification.

   3. Specify the alert severity.

   4. If you do not want the alert to be enabled immediately, click the slider to disable it. You can enable it later.

   5. Click Next to open the Conditions page.

4. Under Condition type, click Event condition.

5. Under Event type, select Anomaly.

6. Under Select a scope, select the type of entity. Then specify which entities of the selected type you want to alert on.

   To select	Do this
   All entities of the specified type	Under Scope by, click Selected <entityType>. Under Select <entityType>, leave the default value of <entityType>.
   Specific entities	Under Scope by, click Selected <entityType>. Under Select <entityType>, click the drop-down menu and then click each entity you want to select.
   All entities whose name contains a certain string	Under Scope by, click Contains text. Under Contains text, enter the string that the entity name must include. The case must match. The alert applies to all entities of the specified type whose name includes that string. After the alert is created, it automatically applies to any new entities with that string in the name.
   All entities whose name does not contain a certain string	Under Scope by, click Does not contain text. Under Does not contain text, enter a string that the entity name must not include. The case must match. The alert applies to all entities of the specified type whose name does not include that string. After the alert is created, it automatically applies to any new entities without that string in the name.

7. Select the metric to alert on.

   The alert will be triggered when this metric value is an anomaly. The graph shows the number of anomalies that occurred during the selected time period.

   

8. Click Next to open the Actions tab, and define one or more actions to be performed when this alert is triggered:

   1. Click Add.

   2. Select the service that will send the notification or perform an action.

      An action can be, for example, a message sent to a Slack channel, a PagerDuty alert, or a Webhook request.

   3. Select one or more configurations to define the action to be performed. To define a new configuration, click Add a New Configuration and enter the required information. (See Notification Services settings for more information about what to enter.)

      To see the details of an existing action or notification, click Settings > Notification Services, and select a service.

   4. To send notifications or perform actions using more than one service, click Add. Then select the service and configurations.

   5. If you want to notify recipients when the alert condition no longer exists, select Send an additional notification when the Alert is cleared.

   6. Click Next.

9. On the Summary page, review the alert definition, and then click Create.

Alert on Kubernetes events

You can create a Kubernetes event alert from the Active Alerts or Alert Settings page, like other alert types. You can also create a quick alert from the Events tab of the Kubernetes cluster details view of the Entity Explorer.

1. Do one of the following to open the alert creation wizard:

   • Start from the Entity Explorer:

     1. In the left pane, click Explore to open the Entity Explorer.

     2. From the entity type drop-down menu, select Kubernetes Clusters.

     3. Click the Kubernetes cluster name to open the Entity Details page, and click the Events tab.

     4. Hover over the event you want to create an alert for, click the vertical ellipsis () on the right, and select Create alert.

        The Create Quick Alert dialog opens.

   • Start from an alerts page:

     1. Open the Active Alerts page (Alerts > Active Alerts) or the Alert Settings page (Alerts > Alert Settings).

     2. In the upper-right corner, click Create Alert.

        The Create Alert wizard opens.

2. Complete the Details page:

   1. Enter a name to identify the alert and, optionally, enter a description of what the alert does.

   2. (Optional) To help users resolve issues quickly, enter a runbook URL. This URL opens a document (for example, a wiki page) that describes what actions should be taken when the alert is triggered. This link is included in the alert notification.

   3. Specify the alert severity.

   4. If you do not want the alert to be enabled immediately, click the slider to disable it. You can enable it later.

   5. Click Next to open the Conditions page.

3. Under Condition type, click Event condition.

4. Under Event type, select Kubernetes events.

5. Under Select a scope, select the type of entity. Then specify which entities of the selected type you want to alert on.

   To select	Do this
   All entities of the specified type	Under Scope by, click Selected <entityType>. Under Select <entityType>, leave the default value of <entityType>.
   Specific entities	Under Scope by, click Selected <entityType>. Under Select <entityType>, click the drop-down menu and then click each entity you want to select.
   All entities whose name contains a certain string	Under Scope by, click Contains text. Under Contains text, enter the string that the entity name must include. The case must match. The alert applies to all entities of the specified type whose name includes that string. After the alert is created, it automatically applies to any new entities with that string in the name.
   All entities whose name does not contain a certain string	Under Scope by, click Does not contain text. Under Does not contain text, enter a string that the entity name must not include. The case must match. The alert applies to all entities of the specified type whose name does not include that string. After the alert is created, it automatically applies to any new entities without that string in the name.

   If you opened the wizard from the Entity Explorer, the condition defaults to the search query you entered there.

6. Under Trigger on following query, add the tags to search for in Kubernetes events, or verify the tags sent from the Entity Explorer.

7. Under Trigger when the number of events is, specify the following, and then click Next:

   • The threshold, or the number of events that must meet the condition to trigger the alert.

   • The time period to search events for the specified condition.

   For example, if the threshold is higher than 1 during the last 30 minutes, the alert is triggered if two or more Kubernetes events meet the condition in a 30-minute time period.

8. Click Next to open the Actions tab, and define one or more actions to be performed when this alert is triggered:

   1. Click Add.

   2. Select the service that will send the notification or perform an action.

      An action can be, for example, a message sent to a Slack channel, a PagerDuty alert, or a Webhook request.

   3. Select one or more configurations to define the action to be performed. To define a new configuration, click Add a New Configuration and enter the required information. (See Notification Services settings for more information about what to enter.)

      To see the details of an existing action or notification, click Settings > Notification Services, and select a service.

   4. To send notifications or perform actions using more than one service, click Add. Then select the service and configurations.

   5. If you want to notify recipients when the alert condition no longer exists, select Send an additional notification when the Alert is cleared.

   6. Click Next.

9. On the Summary page, review the alert definition, and then click Create.

Create an alert based on an existing alert

To quickly create a new alert that is similar to an existing alert, you can duplicate the existing alert definition and then make changes.

1. In the left pane, click Alerts > Alert Settings.

   The Alert Settings page lists all configured alert definitions.

2. Locate the alert you want to duplicate.

3. Hover over the table row, and click the vertical ellipsis () in the far-right column. Then click Duplicate.

   The Create Alert Wizard opens. The default name is the original alert name with - Copy appended to it. All other settings are the same as the original alert.

4. Enter a different name, and update other settings as needed. Depending on the type of alert, see one of the previous sections for more information.

Create an alert based on a template

To facilitate creating alerts for the most common use cases, SolarWinds Observability lets you use pre-filled alert templates configured with various frequently used conditions. The templates can be fine-tuned and adapted to your specific needs.

1. In the left pane, click Alerts > Alert Settings.

2. At the top of the page, click the Templates tab.

3. Select the template that best suits your needs, hover over the table row, and click the vertical ellipsis () in the far-right column. Then click Create Alert.

   The Create Alert Wizard opens.

4. Enter a different name, and update other settings as needed. Depending on the type of alert, see one of the previous sections for more information.

Relevant templates are also suggested within onboarding wizards or in vertical ellipsis menus for specific entities.