Troubleshoot API pollers in SAM

This topic describes issues you may encounter when using API Pollers. For more tips, check the Success Center and the SolarWinds online IT community, THWACK.

Here are some general tips to resolve API poller issues:

• Review API poller requirements.
• Check that the URL for the external API is accessible.
• Verify credentials included in API requests.
• Confirm that Solarwinds.Orion.ApiPoller.Service.exe is active in Task Manager.
• Upgrade to the latest version of SAM to leverage the latest improvements for this feature.
• Examine API poller logs.
• Review API documentation to verify requirements, such as a scope that defines endpoints. For example, API pollers based on the Microsoft 365 Mobile Device Management template use the following scope: https://manage.office.com/.default.
  

Issue: An "Out of API Poller metrics" message appears after sending an API request.

Each monitored metric counts against your SAM license. The way licenses are consumed depends on your SAM license model. For details, see API poller licensing

Issue: Error messages appear when you save a new API poller.

Review the following logs located at ProgramData\SolarWinds\Logs\Orion\ where polling occurs — either on the Main Polling Engine or an Additional Polling Engine (APE).

• ApolloWebApi.log
• OrionWeb.Log

See also Work with API poller logs.

Issue: An API Poller send request returns errors.

When you add an API poller, the Send request is routed to a Website or Additional Website. Make sure the endpoint is accessible from the URL.

In your web browser, open the developer tools (also called DevTools) and examine the results of the send request on the Network tab. If errors occur in a header, try removing the header.

If an external API is unstable, you may be able to drill down into the response data to find exception messages.

If a "Remote certificate is invalid according to the validation procedure" message appears in an API poller response, check SSL certificate verification settings.

Issue: API poller monitoring stops after upgrading SAM

Periodically, SolarWinds updates templates to support modifications made to external APIs maintained by third-party vendors (for example, Microsoft Azure), as announced in the SAM release notes or on THWACK. You may need to recreate existing pollers based on templates that were updated to restart polling.

Issue: A message similar to the following example appears: "Bad Request: Cannot resolve macro value for SUBSCRIPTION_ID. It is not defined or has no value."

This message indicates that a macro is not configured correctly. See Use macros and custom properties to populate API requests in SAM.

Issue: A "Bad request: Unexpected character following indexer" message appears

Periodically, SolarWinds updates API poller templates, as announced in SAM release notes or on THWACK. This message appears if an API poller is based on an outdated template. Recreate the API poller based on the latest template.

Issue: An API poller stopped when the node moved to a different polling engine.

API pollers are not updated when you change a node's polling engine. Edit the API poller and save it again.

Issue: API poller responses are returned but metrics are not updated in the SolarWinds Platform Web Console.

Verify that responses use valid JSON code.

The API Poller feature does not currently support XML output.

Issue: An API poller cannot maintain the API connection.

Run the following PowerShell script on the server that hosts the polling engine:

$uri = "Request-URL"
$method = "Get | Post"
$headers = - the same as User defined @{ 'userId' = 'UserIDValue' 'token' = 'TokenValue' }

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.

If firewall rules block remote connections, API pollers cannot connect.

If a request uses basic authentication, adjust the script as follows:

1. Create a string like username:password
2. Use a site like Base64 encode to encode the string. (© 2019 base64encode.net, available at Base64encode.net, link obtained on September 16, 2019)
3. Define $headers as @{ 'Authorization' = ' Basic ENCODEDString' }

4. Execute the command. Here is an example:

   Invoke-RestMethod -Uri $uri -Method Post -Headers $headers
   

5. Confirm that the return message is in JSON format.

If this method returns an error, check the client network configuration.

Issue: An API poller times out repeatedly.

By default, API poller requests timeout in 60 seconds. You can change that value in Advanced Configuration settings.

1. Log in to the SolarWinds Platform server as an administrator.
2. Use the SolarWinds Platform Service Manager to stop services.
3. To access Advanced Configuration Settings, copy the following text: /Admin/AdvancedConfiguration/Global.aspx.

4. Paste text into your browser address bar, after /Orion, as shown in this example, and press Enter.

   <your production server>/Orion/Admin/AdvancedConfiguration/Global.aspx

5. On the Global tab, scroll to RequestTimeout, adjust the default value (60), and then click Save.
6. Restart services.

Issue: API polling occurs too frequently

Polling occurs every two minutes, by default. You can change that interval on the Advanced Configuration Settings page in the SolarWinds Platform Web Console, accessible by following the steps above.

Note that the Advanced Configuration Settings page includes two tabs: Global and Server-specific:

• <your production server>/Orion/Admin/AdvancedConfiguration/Global.aspx
• <your production server>/Orion/Admin/AdvancedConfiguration/ServerSpecific.aspx

Issue: API response is "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel."

This response appears if the default SSL certificate verification option was disabled for an API poller.

Issue: When assigning an API poller template to a node, the following message appears: "API poller cannot be created. Could not load template."

Check your internet connection.

Issue: Pollers based on Microsoft 365 and Azure API Poller templates don't work. InvalidAuthenticationToken error codes appear and this message: Access token validation failure. Invalid audience.

Navigate to the API poller and click Configure to check API Settings.

• Verify that OAuth 2.0 is selected as the Authorization type.
• Make sure credentials include a scope to define endpoints. For example, pollers based on the Microsoft 365 Mobile Device Management API poller template require the following scope:
  https://manage.office.com/.default.
• Check permissions.
  For example, the Microsoft Azure Active Directory API poller template requires credentials with these Microsoft Graph permissions:
  • Directory.Read.All
  • Group.Read.All
  • SecurityEvents.Read.All

To determine requirements, see the SAM API Poller Template Guide.

Issue: "Internal server error" messages appear.

In Task Manager, determine if the Solarwinds.Orion.ApiPoller.Service.exe process is running.

If it isn't running, restart the SolarWinds Platform Module Engine service:

1. Create a separate backup of the following folder: C:\Windows\Temp\.net\SolarWinds.Credentials.Orion.WebApi

2. Use the SolarWinds Platform Service Manager to stop the service.

3. Remove data from the C:\Windows\Temp\.net\SolarWinds.Credentials.Orion.WebApi folder.

4. Start the SolarWinds Platform Module Engine service.

Issue: A "No connection could be made because the target machine actively refused it" message appears.

Make sure required ports are enabled. See API poller requirements.

Issue: The API Poller page appears blank after upgrading SAM in a Windows Server 2012 R2 environment.

In Windows Server 2012 R2 environments running SAM 2020.2.4 or earlier, ports for the API Poller feature (49157) and credentials (49156) may conflict with ports used by other SolarWinds Platform modules. The API Poller page doesn't load data and errors similar to the following examples occur in API poller log files:

[Warning] Unable to bind to "https://localhost:49157" on the "IPv4 loopback" interface: '"An attempt was made to access a socket in a way forbidden by its access permissions."'
[Fatal] Unable to start Kestrel.

Upgrade to SAM 2020.2.6 or later. Alternatively, update ports in related files, as described next.

• Change the API poller port from 49157 to 38010 in the following files:
  • Update the Url property in $(Orion.InstallDirectory)\SolarWinds.Orion.ApiPoller\SolarWinds.Orion.ApiPoller.Service\appsettings.json , where Orion.WebDirectory is usually C:\Program Files (x86)\SolarWinds\Orion.
  • Update the SolarWindsApiPollerWebApiUrl property in $(Orion.InstallDirectory)\SolarWinds.Orion.ApiPoller\SolarWinds.Orion.ApiPoller.BusinessLayer.dll.config.
  • Update the SolarWindsApiPollerWebApiUrl property in $(Orion.WebDirectory)\SolarWinds\api2\bin\SolarWinds.Orion.ApiPoller.WebApi.dll.config, where Orion.WebDirectory is usually C:\Inetpub\SolarWinds.

• Change the SolarWinds.Credentials port from 49156 to 38008 in the following files:

  • Update the Url property in$(Orion.InstallDirectory)\SolarWinds.Credentials\SolarWinds.Credentials\appsettings.json
  • Update the SolarWindsCredentialsEndpoint property in$(Orion.InstallDirectory)\Orion\SolarWinds.Orion.ApiPoller\SolarWinds.Orion.ApiPoller.Service\appsettings.json
  • Update the SolarWindsCredentialsWebApiUrl property in $(Orion.WebDirectory)\api2\bin\SolarWinds.Orion.Credentials.WebApi.dll.config

Issue: Cannot poll API metrics for HP ProLiant servers

The HP Integrated Lights-Out (iLO) REST API is not supported due to a known HP issue, even if this workaround is applied (©2021 HP Enterprise Development LP, available at support.hpe.com).

Work with API poller logs

To review logs, search for ApiPoller or polled URLs in files stored on the SolarWinds Platform server or the Additional Polling Engine (APE) to which an API poller is assigned, including:

• ProgramData\SolarWinds\Logs\Cortex
• ProgramData\SolarWinds\Logs\Cortex\Plugins\SolarWinds.CortexPlugin.Orion.ApiPoller.log
• ProgramData\SolarWinds\Logs\Orion\ApolloWebApi.log
• ProgramData\SolarWinds\Logs\Orion\OrionWeb.Log
• ProgramData\SolarWinds\Logs\SolarWinds.Credentials\SolarWinds.Credentials\SolarWinds.Credentials.log
• ProgramData\SolarWinds.Orion.ApiPoller\SolarWinds.Orion.ApiPoller.Service\ SolarWinds.Orion.ApiPoller.Service.log

To collect API poller diagnostics for SolarWinds Support:

1. Increase logging to Debug.

   1. Navigate to the default service location:

      C:\Program Files (x86)\SolarWinds\Orion\SolarWinds.Orion.ApiPoller\SolarWinds.Orion.ApiPoller.Service

   2. Use a text editor to add the following JSON code to the appsettings.json file:

      "Serilog": {
"MinimumLevel": "Debug"
},
      

2. Use the SolarWinds Platform Service Manager to restart the SolarWinds Platform Module Engine.
3. Reproduce the issue several times.
4. Collect the latest diagnostics and provide them to Support.