Troubleshoot SEM rules and email responses

This section provides troubleshooting steps to try if your SEM rules that are not firing as expected or if your rules are not sending the expected notifications.

General rule troubleshooting

If you created a rule that generates unexpected results, verify the following to track down the root cause:

  1. On the SEM toolbar, click Monitor, and then check for the requisite events.

    For example, if your rule is based on the NewGroupMember event, locate a requisite event in the All Events or default Change Management filter.

  2. If you cannot view the requisite events, troubleshoot your devices and connectors to move the events into SEM.

  3. Check for an InternalRuleFired event in the SolarWinds Events filter.

    If you see an InternalRuleFired event for your rule, go to the next step.

  4. If you do not see an InternalRuleFired event for your rule, verify that:

    • The rule is enabled.
    • The Correlation Time or Response Window in your rule was not modified.
    • You did not click Activate Rules after saving your rule.
    • The time on your device is not more than five minutes off from the time on your SEM appliance.

  1. If you see an InternalRuleFired event for your rule but SEM does not respond to the rule as expected, check the following:

    • Send Email Message
      Verify you configured and started the Email Active Response connector on the SEM Manager. Additionally, verify you associated an email address for your selected SEM user as your email account.
    • Agent-based Actions
      Verify you installed the SEM Agent on a computer that will respond to SEM.
    • Block IP
      If using the Block IP active response, verify that you configured the active response connector for the targeted firewall that will respond to this action. The active response connector is separate from the data-gathering connector.

The rule fires but you do not receive an email

Problem statement: You see the expected InternalRuleFired alerts in the default SolarWinds Alerts and Rule Activity filters in the SEM console, but you are not getting the expected email notification.

To resolve this issue:

  1. Verify that the ExtraneousInfo field of the InternalRuleFired alert shows the associated email action in Email [recipient] format.

  2. If this action is not present, add the Send Email Message action to the rule.

  3. Verify that the intended recipient has an email address associated with his SEM user account:

    1. Open the SEM console. See Log in to the SEM web console for steps.

    2. On the SEM toolbar, navigate to Build > Users.
    3. Click the SEM user account associated with the intended recipient.

  4. If the Contact Information box is blank in the User Information pane, edit the user to add an email address.

    If you cannot add an email address to an Active Directory user, create a separate user, add the email to that user account, and then select that user in the email template.

  5. Verify that the Email Active Response connector is configured on your SEM Manager.
    1. On the SEM toolbar, navigate to Manage > Appliances.

    2. Next to your SEM Manager, click , and then select Connectors.

    3. In the Connector Configuration window, select the Configured check box.

  6. If Email Active Response is not in the list, clear the Configured check box configure the missing connector.

The rule does not fire and expected alerts do not display

Problem statement: You cannot see the expected InternalRuleFired alerts in the default SolarWinds Alerts or Rule Activity filters in the SEM console or the alerts needed to fire your rule anywhere in your SEM console.

To determine if the requisite alerts are in your SEM console, create a filter or nDepth search that matches the correlations in your rule.

If the alerts are not present, complete the following procedure:

  1. Review the network devices sending syslog data to the SEM and validate the configurations on that network device to send data. Verify that one of your devices is logging the events you want to capture.

    For example:

    • Remote logging devices, such as firewalls and web filters, should be logging your web traffic events
    • Domain controllers and end-user computers should be logging domain-level and local authentication and change management events

      If you have multiple domain controllers, they will not all replicate every domain event. Each server only logs the events they execute.

    • Other servers, such as database servers and web servers, should be logging events associated with their particular functions.
  2. Verify that the SEM is receiving data.

    Verify that the SEM icons display a syslog or Agent connection. Syslog device IPs display with the icon in the Manage > Nodes grid. Agent host names and IP addresses appear in the Manage > Nodes list with the icon.

    Next, verify that the syslog facility or Agent is receiving data. If a network syslog device is sending syslog data to the SEM, you can view the SEM syslog files for that data.

    1. Open the CMC command line. See Log in to the SEM CMC command line interface for steps.
    2. Type appliance, and enter the checklogs command.

      You can also open a PuTTY session on port 32022 as a cmc user.

    3. View the syslog that was chosen by the network device. All the data received in this area is UDP traffic received on port 514.

  3. If your device is not in the Nodes list, configure your computers by installing a SEM Agent or configure other devices (such as firewalls) to log to your SEM VM or appliance. After your device is in the list, continue to the next step.

  4. If your device is in the Nodes list, configure the appropriate connectors:

    1. Open the SEM console. See Log in to the SEM web console for steps.

    2. On the SEM toolbar, navigate to Manage > Appliances.
    3. Next to the Agent or SEM Manager, click , and then select Connectors.

      Use the Search box at the top of the Refine Results pane to locate the appropriate connectors.

    4. Configure the syslog connector according to your needs.

    5. On the SEM toolbar, navigate to Manage > Nodes.

    6. Next to the Agent, click .

    7. Configure the Agent connector as required.

Alerts display but the rule does not fire

Problem statement: You see the alerts required to fire your rule in the SEM console, but your rule still doesn't fire.

To resolve this issue:

  1. Verify that all your rules are activated in all open SEM consoles:

    1. Open the SEM console. See Log in to the SEM web console for steps.

    2. On the SEM toolbar, navigate to Build > Rules.
    3. Click Activate Rules.

      All rule changes you impsemented in your SEM Manager are synchronized.

    4. Repeat these steps for all open SEM consoles in your environment.

  2. Compare the InsertionTime and DetectionTime values in the alerts you expected to fire your rule.

    If the time is off by more than five minutes, verify and correct the time settings on your SEM VM or appliance, and any remote logging devices as necessary.

  3. If your rules will not fire, restart the Manager service on your SEM VM/appliance.

    In general, consider doing this once every six months:

    1. Open the CMC command line. See Log in to the SEM CMC command line interface for steps.
    2. At the cmc> prompt, enter manager and press Enter.

    3. At the cmc::manager> prompt, type restart and press Enter.

    4. To confirm your entry, press Enter.

      Restarting the SEM Manager service disconnects the Manager for a few seconds. No data is lost during this process.

    5. To leave the CMC interface, enter exit, and then press enter twice.

The rule fires but the email is blank

Problem statement: You receive an email notification for the alert, but the fields in the custom email template are blank.

To resolve this issue:

  1. Open the SEM console. See Log in to the SEM web console for steps.

  2. On the SEM toolbar, navigate to Build > Rules.

  3. Locate your rule in the Rules grid.

  4. Next to your targeted rule, click , and then select Edit. Notice that the files in the Action box are blank.

  5. Copy the event assigned to this rule.

    This is the string before the dot in the Correlation box.

  6. Click Events, and then enter the event in the search field.

  7. Drag the event fields required for your rule into the Actions box.

  8. To close the Rule Creation window, click Save.

  9. Click Activate Rules.

View and modify the time on your SEM appliance

  1. Open the CMC command line. See Log in to the SEM CMC command line interface for steps.

  2. At the cmc> prompt, enter appliance.

  3. At the cmc::appliance> prompt, enter dateconfig.

  4. Press Enter through all the prompts to view the current date and time settings on your SEM appliance.

    By default, SEM receives a time synchronization from the VM host computer. Without the synchronization, the SEM time is not correct and the rules may not trigger when required.

  5. Disable the time sync on the VM host computer and enable SEM to receive time information from an NTP server.
    1. At the cmc::appliance> prompt, enter ntpconfig and press Enter.

    2. Press Enter to start the configuration script.

    3. Enter the IP addresses of your NTP servers separated by spaces.

    4. Enter y and press Enter to verify your entry.

  6. To leave the CMC interface, enter exit, and then press Enter twice.

The rule is not triggered when it should be

Check your rule logic and timestamps. The SEM VM host layer may need to be configured for NTP. By default, rules will not fire when incoming data drifts more than five minutes from the SEM VM clock.

  1. Open the CMC command line. See Log in to the SEM CMC command line interface for steps.
  2. To enter the appliance menu, type appliance.

  3. Enter the dateconfig command, and confirm the date and time. You can change the time with this command, but when the vSphere/Hyper-V time sync pushes the time to SEM, this will change.