Documentation forAppOptics

Troubleshooting Windows

If you run into any issue installing the SolarWinds Snap Agent or getting metrics reported, please check the troubleshooting techniques below.

Restart the agent

The agent supports standard service control commands including start, stop and restart.

For example, using a command prompt launched as Administrator, you can restart the agent by running net commands:

net stop swisnapd
net start swisnapd

For example, using a command prompt launched as Administrator, you can restart the agent by running scripts provided with installation:

restart_solarwinds_snap_agent_service

For example, using a command prompt launched as Administrator, you can stop and start the agent by running scripts provided with installation:

start_solarwinds_snap_agent_service
stop_solarwinds_snap_agent_service

View the agent logs

The agent log files are located under C:\ProgramData\SolarWinds\Snap\log. There should be the following files:

  • solarwinds_swisnap.log – diagnostic messages from the process monitoring the agent service
  • Install_<date>.log – diagnostic messages during the agent install
  • swisnapd.log – diagnostic messages from the agent as it runs

By default only messages at or above the warning level are reported in swisnapd.log. To increase logging verbosity:

  • Set the log level to debug in the agent config file

  • Restart the agent

  • Check for new messages in the log file, for example:

    notepad C:\ProgramData\SolarWinds\Snap\log\swisnapd.log

Check the loaded plugins

The SolarWinds Snap Agent includes the swisnap command line tool to interact with the snap daemon on which our agent is based. Out of the box, our agent will automatically load two plugins that enable collecting system metrics (aosystem) and publishing to AppOptics (publisher-appoptics). To check that they are loaded, you can run swisnap plugin list and confirm they are listed as loaded under the STATUS column:

swisnap plugin list

swisnap plugin list

NAME                    TYPE         API          VERSION    RUNTIME    SIGNED     STATUS     LOADED TIME
publisher-appoptics     publisher    PluginsV1    48.0.0                false      loaded     Mon, 18 May 2020 17:44:47 UTC
aosystem                collector    PluginsV2    58.0.0     local      false      running    Mon, 18 May 2020 17:44:48 UTC
processes               collector    PluginsV2    11.0.0     local      false      running    Mon, 18 May 2020 17:44:48 UTC
publisher-appoptics     publisher    PluginsV2    48.0.0     local      false      running    Mon, 18 May 2020 17:44:48 UTC
publisher-processes     publisher    PluginsV2    48.0.0     local      false      running    Mon, 18 May 2020 17:44:48 UTC

Check the state of tasks

Similar to checking loaded plugins, you can use the swisnap command line tool to check the state of tasks, which define the metrics collection and publishing jobs run by the agent. Out of the box, our agent will automatically define a task to report system metrics continuously every minute.

  • Use swisnap task list to get a list:

    swisnap task list
    +----------------------------------------+-----------+-------------+--------------+-----------+--------+------------+
|            TASK ID / PLUGIN            |   TYPE    | LAST ACTION |   DURATION   | PROCESSED | STATUS | LAST ERROR |
+----------------------------------------+-----------+-------------+--------------+-----------+--------+------------+
| 1a037d34-00-task-aosystem.yaml         |           |             |              |           |        |            |
|   1a037d34-ff0b-423c-a70f-dd2d13e286a2 |           |             |              |           |        |            |
|     aosystem                           | collector | load        | 0s           |         0 |        | N/A        |
|     publisher-appoptics                | publisher | load        | 0s           |         0 |        | N/A        |
| bf0d3744-00-task-processes.yaml        |           |             |              |           |        |            |
|   bf0d3744-fe49-457c-b7ae-b7283319cc39 |           |             |              |           |        |            |
|     processes                          | collector | collect     | 3.005011279s |        28 | +      | N/A        |
|     publisher-processes                | publisher | publish     | 1.453608565s |        28 | +      | N/A        |
| c61f29c7-00-task-aosystem-warmup.yaml  |           |             |              |           |        |            |
|   c61f29c7-b096-4033-8f7e-26f78effd171 |           |             |              |           |        |            |
|     aosystem                           | collector | collect     | 1.687354ms   |         1 | ++++++ | N/A        |
|     publisher-appoptics                | publisher | publish     | 438.179707ms |         1 | ++++++ | N/A        |
+----------------------------------------+-----------+-------------+--------------+-----------+--------+------------+

  • The above output shows a running task. To further confirm that it is the one reporting system metrics, you can either use the swisnap task details <task id> command to print to console the task details, or use the swisnap task watch <task id> command which logs to console the metrics being gathered at each task interval. An example of the watch command:

    swisnap task watch 53c0afb1-1e47-471f-b1c0-af69207842eb
    [2020-05-18T17:47:51.667573043Z] /processes/snap-plugin-publisher-appoptics/cpu = 0
[2020-05-18T17:47:51.667581169Z] /processes/snap-plugin-publisher-appoptics/memory = 1.0773205757141113
[2020-05-18T17:47:51.66758757Z] /processes/snap-plugin-publisher-appoptics/count = 1
...
(ctl-c to quit)

Run the plugin directly

If you're experiencing issues with a specific integration and the agent logs are not providing much help, you can also run the binary for the plugin independently of the swisnap service to attempt a collection. This could reveal errors or permission issues that are being obscured.

For plugins compatible with V1 plugin API:

C:\Program Files\SolarWinds\Snap\bin\<plugin_binary> --config '<config to use in JSON format>'

For example to run snap-plugin-collector-aoiss with cmd.exe:

"C:\Program Files\SolarWinds\Snap\bin\snap-plugin-collector-aoiis.exe" --config "{}"
Runtime Details:
  PluginName: iis, Version: 6
  RPC Type: gRPC, RPC Version: 1
  Operating system: windows
  Architecture: amd64
  Go version: go1.11.13
printRuntimeDetails took 0s

Config Policy:
NAMESPACE    KEY         TYPE       REQUIRED    DEFAULT     MINIMUM     MAXIMUM
iis          url         string     true
iis          token       string     true
iis          username    string     false
iis          password    string     false
iis          domain      string     false
printConfigPolicy took 3.9747ms

showDiagnostics took 4.9761ms

For plugins compatible with V2 plugin API:

C:\Program Files\SolarWinds\Snap\bin\<plugin_binary> --debug-mode --plugin-config '<config to use in JSON format>'

For example to run snap-plugin-collector-bridge with cmd.exe:

"C:\Program Files\SolarWinds\Snap\bin\snap-plugin-collector-bridge.exe" --debug-mode --plugin-config "{\"redis\": {\"servers\": [\"tcp://localhost:6379\"]}}"
Gathered metrics (length=0):

Agent not reporting metrics: "context deadline exceeded" error

Occasionally you may run into a situation where the SolarWinds Snap Agent will not connect to AppOptics or report metrics.

If you see a similar error to the following in C:\ProgramData\SolarWinds\Snap\log\swisnapd.log:

ERRO[2018-06-12T09:12:08Z] time="2018-06-12T09:12:08Z" level=error msg="Error getting cpu" err="unable to get cpu. Error: context deadline exceeded" _module=plugin-exec io=stderr plugin=publisher-appoptics

This is caused by a WMI Performance Counter timeout. To increase the timeout, change the value of host_check_timeout setting in publisher-appoptics and publisher-processes configuration files. C:\ProgramData\SolarWinds\Snap\plugins.d\publisher-appoptics.yaml:

v1:
  publisher:
    publisher-appoptics:
      all:
        endpoint:
          token: "c6260446e584d8afcafcafcafcafcafcafc112233441128e68ef4ea3f2b289d50"

          ## URL to the AppOptics REST API endpoint for collecting measurements data
          # url: "https://api.appoptics.com/v1/measurements"
          ...
          ## host_check_timeout allows to configure timeout for querying host operating system for identification informations,
          ## like interfaces, CPUs, etc. By default it is set to 5s.
          host_check_timeout: "5s"
...
v2:
  publisher:
    publisher-appoptics:
      all:
        endpoint:
          token: "c6260446e584d8afcafcafcafcafcafcafc112233441128e68ef4ea3f2b289d50"

          ## URL to the AppOptics REST API endpoint for collecting measurements data
          # url: "https://api.appoptics.com/v1/measurements"
          ...
          ## host_check_timeout allows to configure timeout for querying host operating system for identification informations,
          ## like interfaces, CPUs, etc. By default it is set to 5s.
          host_check_timeout: "5s"

C:\ProgramData\SolarWinds\Snap\plugins.d\publisher-processes.yaml:

v2:
  publisher:
    publisher-processes:
      all:
        endpoint:
          token: "c6260446e584d8afcafcafcafcafcafcafc112233441128e68ef4ea3f2b289d50"

          ## URL to the AppOptics REST API endpoint for collecting measurements data
          # url: "https://api.appoptics.com/v1/measurements"
          ...
          ## host_check_timeout allows to configure timeout for querying host operating system for identification informations,
          ## like interfaces, CPUs, etc. By default it is set to 5s.
          host_check_timeout: "5s"

If the agent will still be unable to collect host identification, you can try the following steps to resolve this issue:

https://support.microsoft.com/en-us/help/2554336/how-to-manually-rebuild-performance-counters-for-windows-server-2008-6

Navigation Notice: When the APM Integrated Experience is enabled, AppOptics shares a common navigation and enhanced feature set with other integrated experience products. How you navigate AppOptics and access its features may vary from these instructions.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.