Notification Services settings

Alert notification methods are versatile. SolarWinds Observability supports several methods for sending notifications.

SolarWinds Observability allows you to establish multiple notification services and configure alerts to terminate at different services. Alert notification services are shared across your account so you can use notification services other users have set up.

Click Settings > Notification Services to manage existing configurations or add new configurations.

Email notifications

An email is sent to any users configured to receive a notification on a triggered event. If the alert is configured to check for events every minute, an email is sent every minute while the alert is triggered.

Set up email notifications

1. Click Settings > Notification Services > Email.

2. Click Create configuration.

3. Enter the name and description of the configuration.

4. Select email recipients from the list or manually enter the email addresses.

5. Click Notification test to verify the email is sent. If not, fix any issues.

6. Click Create.

Microsoft Teams

SolarWinds Observability provides Microsoft Teams integration for sending static alerts to a Teams channel. SolarWinds Observability can also insert data from the alert into the message.

Set up a Teams integration

1. Click Settings > Notification Services > Microsoft Teams.

2. Click Create configuration.

3. Enter the name and description of the configuration.

4. Add an Incoming Webhook to a Teams Channel:

   1. Follow Microsoft's instructions on how to Create an Incoming Webhook.

   2. In SolarWinds Observability, enter the Webhook URL in the field Microsoft Teams Webhook URL.

5. Click Notification test to verify the notification is sent. If not, fix any issues.

6. Click Create.

OpsGenie

SolarWinds Observability can send alerts via email or using a webhook. Either method can be used to forward SolarWinds Observability alerts to OpsGenie.

Set up an OpsGenie integration

1. Create an OpsGenie account if you haven’t done so already.

2. Navigate to the Integrations tab and click Add new integration. You are presented with a new API key. Copy the API key string.

3. In SolarWinds Observability, click Settings > Notification Services > OpsGenie.

4. Click Create configuration.

5. Enter the name and description of the configuration.

6. Select the API Hostname.

7. Enter the API key you copied from OpsGenie into the field provided.

8. Use the Recipients and Teams fields to specify who should be notified by email for the alert.

9. Enter any possible tags.

10. Click Notification test to verify the notification is sent. If not, fix any issues.

11. Click Create.

PagerDuty

SolarWinds Observability provides PagerDuty integration. PagerDuty is a third-party service that aggregates monitoring tools into a single point of ingestion. If you already have a PagerDuty account, integration with SolarWinds Observability helps keep your notifications in one place.

Set up a PagerDuty integration

1. PagerDuty Support provides instructions for setting up alerts and integrations through PagerDuty. Follow the instructions on the PagerDuty Support page. When presented with the option to select Integrations, select SolarWinds Observability. When creation of the new service with integration is complete, PagerDuty provides you with an integration key. Copy the key.

2. In SolarWinds Observability, click Settings > Notification Services > PagerDuty.

3. Click Create configuration.

4. Enter the name and description of the configuration.

5. In the Routing Key field, enter the integration key you copied in step 1.

6. In the Summary field, provide a summary of the issue that is triggering the alert.

7. In the Dedup Key field, enter the deduplication key for correlating trigger conditions.

ServiceNow

ServiceNow integration with SolarWinds Observability creates new incidents based on SolarWinds Observability alerts. To integrate, set up an OAuth JSON Web Token (JWT) endpoint in ServiceNow, and use the web token and instance name to configure a new notification service in SolarWinds Observability. See Create an OAuth JWT API endpoint for external clients for detailed instructions on how to set up a JWT endpoint in ServiceNow.

Set up a ServiceNow integration

1. In your ServiceNow instance, navigate to System OAuth > Application Registry > Create an OAuth JWT API endpoint for external clients. Complete the required fields in the form.

   Set an access token lifespan value large enough to persist for the duration of this integration.

2. Find your ServiceNow access token and instance name for this integration.

   1. To identify your ServiceNow instance name, find the URL used to log in to ServiceNow. The instance name can be found in the subdomain of the URL. For example, if you use https://swo123.service-now.com/ to log in to ServiceNow, swo123 would be the instance name.

   2. Open the OAuth JWT API endpoint you just created and toggle the password visibility of the Client Secret to on.

   3. Run the following curl command, replacing:

      • YourClientID with the Client ID found in the new JWT endpoint's details.
      • YourClientSecret with the Client Secret found in the new JWT endpoint's details.
      • YourUsername and YourPassword with the user name and password you use to log in to ServiceNow.
      • YourInstanceName with the instance name for your ServiceNow instance.

      curl -d"grant_type=password&client_id=YourClientID&client_secret=YourClientSecret&username=YourUsername&password=YourPassword" https://YourInstanceName.service-now.com/oauth_token.do

   4. Locate the access token in the curl response. In the following example curl response, KynMY2H0uwWkRc8g8YLXjnQxWbH5_wbnSiLsnaOoKw61GZkkV0ytZP74uF7hJyjfsWfaaFijqQzq2kcABNJxNA is the access token:

      {
          "access_token": "KynMY2H0uwWkRc8g8YLXjnQxWbH5_wbnSiLsnaOoKw61GZkkV0ytZP74uF7hJyjfsWfaaFijqQzq2kcABNJxNA",
          "refresh_token": "rXlb1sHjxMhmt4cBFJw0jr1XllVlOClQgjL9445NVmPZ8ZrbVt10sqRc0htNNFc64MchAwjMrjDOMRy6C4y97A",
          "scope": "useraccount",
          "token_type": "Bearer",
          "expires_in": 17999999
      }

3. In SolarWinds Observability, navigate to Settings > Notifications Services > ServiceNow.

4. Click Create Configuration.

5. Enter a name and description for the ServiceNow notification service.

6. In the App Token field, enter the access token identified in step 2.

7. In the Instance field, enter your ServiceNow instance name identified in step 2.

8. Click Notification Test to send a test incident to ServiceNow and verify the configuration is correct. If not, fix any issues.

9. Click Create.

Slack

SolarWinds Observability provides Slack integration for sending static alerts to a Slack channel. SolarWinds Observability can also insert data from the alert into the message.

To set up a Slack integration, click Notification Services > Slack > Create configuration. A pop-up informs you that you will be connected to Slack to complete the configuration.

SolarWinds Service Desk

SolarWinds Service Desk integration with SolarWinds Observability creates new incidents based on SolarWinds Observability alerts. To integrate, you must set up a web token in SolarWinds Service Desk and use that token to configure a new notification service. See Token authentication for API integration for instructions on how to set up token authentication in SolarWinds Service Desk.

Set up a SolarWinds Service Desk integration

1. Follow the SolarWinds Service Desk instructions to Set up token authentication.

2. In SolarWinds Observability, navigate to Settings > Notifications Services > SolarWinds Service Desk.

3. Click Create Configuration.

   The Create SolarWinds Service Desk Configuration dialog opens.

4. Provide a name and description.

5. Select the host region.

6. Under App Token, paste the token copied from SolarWinds Service Desk.

7. Click Notification Test to verify that the notification is sent successfully. If not, fix any issues.

8. Click Create.

Webhook

Use webhook configurations to integrate SolarWinds Observability alerts with your existing notification services.

Set up a webhook configuration

1. Click Settings > Notification Services > Webhook.

2. Click Create Configuration.

3. Enter a name to identify the configuration and, optionally, provide a description.

4. Select the method:

   • GET webhook requests are simple. Information about the triggered alert is appended to the webhook URL as a query string.

   • POST webhook requests can include more details. Information about the triggered alert is included in the body.

5. Enter the URL to which the webhook requests will be sent.

6. If you selected POST as the method:

   1. Under Content Type, select the format of the request. By default, the request is sent as a JSON object.

   2. Do one of the following to specify what information the body includes:

      • To send the default body content, leave the HTTP POST Body field blank.

        The type of alert determines what content is included in the associated webhook alert notification.

      • To customize the body content, enter the text and variables that you want to include in the HTTP POST Body field. See Customize the POST body content below.

7. If you want to include authentication:

   1. Select Advanced Settings.

   2. Specify Basic or Token authentication.

   3. Enter the credentials (for basic authentication) or the header name and value (for token authentication).

8. Click Notification Test to send a test webhook request and verify the configuration is correct. If not, fix any issues.

9. Click Create.

Zapier

SolarWinds Observability provides an integration with Zapier. Zapier is a third-party service enabling you to integrate your web applications and automate workflows. To use Zapier as a notification service, you need to have a Zapier account. Before you create a new configuration in SolarWinds Observability, log in to your Zapier account and create a new Zap, select Webhooks by Zapier as the Trigger app, and select Catch Hook as the Trigger.

Customize the POST body content

To customize the body of the notification for any notification type, you can enter static text and the parameters in the following sections, which represent alert configuration details, entity details, and other variable values. The following examples are in JSON, but you can also enter XML.

When you enter content:

• To ensure that the last element of an array skips the comma, include the {^last}}, {{/last}} tag after the last item in the array. For example:

  \"entities\": [
    {{#entities}}
      {
        \"name\": \"{{name}}\", 
        \"type\": \"{{type}}\", 
        \"triggeredAt\": \"{{triggeredAt}}}\", 
        \"activeDuration\": \"{{activeDuration}}
      }
      {{^last}}, {{/last}}
    {{/entities}}
  ],

• Any special characters are encoded in the parsed output. For example:

  "detailUrl": "https://example.com/abc/123/xyz"

  becomes:

  "detailUrl": "https://example.com/abc/123/xyz"

  To avoid this, use triple curly braces instead of double braces. For example, change {{detailedUrl}} to {{{detailedUrl}}}. For more information, see the Handlebars language guide.

• If the value for a parameter is not available, that area in the notification is blank. To prevent it from being blank, you can specify a default value, such as "None", to be used instead. For example:

  {{^numberOfAdditionalEntities}}
  None
  {{/numberOfAdditionalEntities}}}

• You must know the type of alert that you are creating the webhook notification for. If you include a parameter that is not valid for the associated alert definition, the parameter is ignored.

Common parameters

The following parameters are available for all types of alerts.

• {{type}}: The type of alert
• {{name}}: The name of the alert definition
• {{severity}}: The severity of the alert (for example, Warning or Critical)
• {{priority}}: The priority of the alert
• {{detailUrl}}: A link to the alert definition overview
• {{condition}}: The condition that triggered the alert

Parameters for stand-alone metric condition alerts

Use the following parameters in notifications for metric alerts.

• {{#affectedEvaluations}}
  • {{name}}: The name of the measured or monitored metric
  • {{value}}: The metric value

JSON example

{
  "name": "{{name}}",
  "condition": "{{condition}}",
  "priority": "{{priority}}",
  "severity": "{{severity}}",
  "timestamp": "{{timestamp}}",
  "detailUrl": "{{detailUrl}}",
  "affectedEvaluations": [
    {{#affectedEvaluations}}
    {
      "values": [
        {{#values}}
        {
          "name": "{{name}}",
          "value": "{{value}}"
        }
        {{^last}},{{/last}}
        {{/values}}
      ]
    }
    {{^last}},{{/last}}
    {{/affectedEvaluations}}
  ]
}

Parameters for entity metric condition alerts

Use the following parameters in notifications for metric alerts.

• {{entities}}
  • {{type}}: The entity type
  • {{displayName}}: The display name of the entity
  • {{entityId}}: The ID of the entity
  • {{timestamp}}: The time when the alert was triggered
• {{#metrics}}
  • {{name}}: The name of the measured or monitored metric
  • {{value}}: The metric value
  • {{threshold}}: The threshold at which the alert is triggered

JSON example

{
  "name": "{{name}}",
  "severity": "{{severity}}",
  "priority": "{{priority}}",
  "detailUrl": "{{detailUrl}}",
  "condition": "{{condition}}",
  "entities":[
    {{#entities}}
        {
            "type": "{{type}}",
            "displayName": "{{displayName}}",
            "entityId": "{{entityId}}",
            "timestamp": "{{timestamp}}",
            "tags":[
                {{#tags}}
                    {
                      "key": "{{key}}",
                      "value": "{{value}}"
                    }
                    {{^last}},{{/last}}
                {{/tags}}
            ],
            "values":[
                {{#values}}
                    {
                        "name": "{{name}}",
                        "value": "{{value}}"
                    }
                    {{^last}},{{/last}}
                {{/values}}
            ],
           "relationship": [
               {{#relationship}}
                   {
                       "name": "{{name}}",
                       "size": "{{size}}",
                       "additionalEntities": "{{additionalEntities}}",
                      "childValues": [
                          {{#childValues}}
                              {
                                  "name": "{{name}}",
                                  "values":[
                                      {{#values}}
                                          {
                                              "name": "{{name}}",
                                              "value": "{{value}}"
                                          }
                                          {{^last}},{{/last}}
                                      {{/values}}
                                  ]
                              }
                              {{^last}},{{/last}}
                          {{/childValues}}
                      ]
                  }
                  {{^last}},{{/last}}
              {{/relationship}}
           ]
       {{^last}},{{/last}}
    {{/entities}}
]
}

Parameters for log query alerts

Use the following parameters in notifications for log query alerts.

• {{recordCount}}: The number of logs that match the defined query
• {{threshold}}: The threshold at which the alert is triggered
• {{timespanStart}}: The beginning of the time range for searching logs
• {{timespanEnd}}: The end of the time range for searching logs
• {{query}}: The query for searching logs

JSON example

{
  "name": "{{name}}",
  "severity": "{{severity}}",
  "priority": "{{priority}}",
  "detailUrl": "{{detailUrl}}",
  "recordCount": "{{recordCount}}",
  "threshold": "{{threshold}}",
  "timespanStart": "{{timespanStart}}",
  "timespanEnd": "{{timespanEnd}}",
  "query": "{{query}}",
  "logViewerUrl": "{{logViewerUrl}}"
  "events":[
        {{#events}}
        {
        "message": {{message}}
        }
        {{^last}},{{/last}}
        {{/events}}
  ]
}

Parameters for anomaly event alerts

Use the following parameters in notifications for anomaly event alerts.

• {{entities}}
  • {{type}}: The entity type
  • {{name}}: The display name of the entity
  • {{entityId}}: The ID of the entity
  • {{timestamp}}: The time when the alert was triggered
  • {{#metrics}}
    • {{name}}: The name of the measured or monitored metric
    • {{value}}: The difference between the observed value and the normal range
    • {{direction}}: A word indicating whether the observed value is above or below the normal range
    • {{range}}: The range of values that define the normal state

JSON example

{
    "name": "{{name}}",
    "severity": "{{severity}}",
    "priority": "{{priority}}",
    "detailUrl": "{{detailUrl}}",
    "numberOfAdditionalEntities": "{{numberOfAdditionalEntities}}",
    "name": "{{name}}",
    "entities": [
        {{#entities}}
        {
            "entityId": "{{entityId}}",
            "metrics": [
                {{#metrics}}
                {
                    "direction": "{{direction}}",
                    "name": "{{name}}",
                    "range": "{{range}}",
                    "value": "{{value}}"
                }
                {{^last}},{{/last}}
                {{/metrics}}
            ],
            "name": "{{name}}",
            "timestamp": "{{timestamp}}",
            "type": "{{type}}"
        }
        {{^last}},{{/last}}
        {{/entities}}
    ],
}

Parameters for Kubernetes event alerts

Use the following parameters in notifications for Kubernetes event alerts.

• {{entities}}
  • {{type}}: The entity type
  • {{name}}: The display name of the entity
  • {{entityId}}: The ID of the entity
  • {{timestamp}}: The time when the alert was triggered
  • {{recordCount}}: The number of logs that match the defined query
  • {{timespanStart}}: The beginning of the time range for searching logs
  • {{timespanEnd}}: The end of the time range for searching logs
  • {{query}}: The query for searching Kubernetes logs
  • {{#events}}
    • {{message}}: The log message of the event

JSON example

{
    "name": "{{name}}",
    "severity": "{{severity}}",
    "priority": "{{priority}}",
    "detailUrl": "{{detailUrl}}",
    "timespanStart": "{{timespanStart}}",
    "timespanEnd": "{{timespanEnd}}",
    "threshold": "{{threshold}}",
    "recordCount": "{{recordCount}}",
    "query": "{{query}}",
    "entities": [
        {{#entities}}
        {
            "entityId": "{{entityId}}",
            "name": "{{name}}"
        }
        {{^last}},{{/last}}
        {{/entities}}
    ],
    "events": [
        {{#events}}
        {
            "message": "{{message}}",
        }
        {{^last}},{{/last}}
        {{/events}}
    ]
}