Documentation forAppOptics

Troubleshooting (legacy agent)

The following content pertains to troubleshooting for the legacy AppOptics NET Agent.

AppOptics agents are no long receiving updates. The new SolarWinds Observability libraries can send APM data in AppOptics and are regularly updated with new features and improvements. If you are still relying on the AppOptics agents and your components are supported by the new libraries, consider transitioning to the SolarWinds Observability libraries for your APM needs. For more information about the benefits of migrating to the SolarWinds Observability libraries. Alternatively, you can use SolarWinds Observability as your primary APM solution.

If you have already transitioned to the new SolarWinds Observability NET Library, see the SolarWinds NET Library documentation for troubleshooting information.

SolarWinds Observability libraries are not compatible with AppOptics agents. Do not use a mix of SolarWinds Observability libraries and AppOptics agents to instrument applications that are part of a distributed trace.

If you are not receiving traces and metrics in your AppOptics dashboard these are some options to troubleshoot.

Diagnostic Tool

How do I run the diagnostic tool in Windows?

A connectivity diagnostic tool is available with the .NET Agent. To run it in Windows, open a command prompt and run the Core, the Framework, or the Azure App Service version of the executable located in the agent install directory:

  • .NET Framework – C:\Program Files\AppOptics\APM\dotnet> dotnet_diagnostic_tool.exe

  • .NET Core – C:\Program Files\AppOptics\APM\dotnet> dotnet_diagnostic_tool_core.exe

  • Azure App Service – D:\home\SiteExtensions\AppOptics.Agent.AzureAppServices.Extension\AppOptics\<dotnet-runtime>\dotnet_diagnostic_tool_core.exe where <dotnet-runtime> is one of the supported .NET Core Runtime installed on machine (e.g. net5.0 for .NET 5, netcoreapp3.1 for .NET Core 3.1)

The optional parameters are:

  • /ServiceKey : Service key to be used for the diagnostic tool. By default it uses the current service key configured configured globally for the host. It does not pick up any service keys defined at the <iisSites>, <applicationPools> or <applications> levels.
  • /Verbose : Enable printing to console of verbose level log messages from the diagnostic run.

For example, to verify connection using a specific service key:

C:\Program Files\AppOptics\APM\dotnet> dotnet_diagnostic_tool.exe /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service

Look for the message Has connection succeeded? YES which indicates that the connection has succeeded.

If there are any errors, review the log files C:\Program Files\AppOptics\APM\dotnet\dotnet_diagnostic_tool.log and C:\Windows\Temp\AppOptics\DotNetAgentLog_dotnet_diagnostic_tool*.

How do I run the diagnostic tool in Linux?

A connectivity diagnostic tool is available with the .NET Agent. To run it in Linux, in a shell session run the executable <dotnet-runtime>/dotnet_diagnostic_tool_core located under the agent install directory, where <dotnet-runtime> is one of the supported .NET Core Runtime installed on machine (e.g. net5.0 for .NET 5, netcoreapp3.1 for .NET Core 3.1).

The example below assumes you've already set the agent install directory environment variable APPOPTICS_HOME_APM_DOTNET:

cd ${APPOPTICS_HOME_APM_DOTNET}/<dotnet-runtime> 
./dotnet_diagnostic_tool_core 

The optional parameters are:

  • /ServiceKey : Service key to be used for the diagnostic tool. By default it uses the current service key configured configured globally for the host.
  • /Verbose : Enable printing to console of verbose level log messages from the diagnostic run.

For example, to verify connection using a specific service key:

dotnet_diagnostic_tool_core/dotnet_diagnostic_tool_core /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service

Look for the message Has connection succeeded? YES which indicates that the connection has succeeded.

If there are any errors, review the log files /var/tmp/appoptics/DotNetAgentLog_dotnet_diagnostic_tool_core*.

What are the diagnostic tool's exit codes?

The diagnostic tool provides exit code as below:

Exit Status Code Name Description
0 OK Diagnostics successful
101 UNKNOWN_ERROR Unexpected error. See error message for details
102 INVALID_FORMAT_ARGUMENT Argument passed to the diagnostic tool is not in valid format. For example the service key is too short. See error message for details
103 INVALID_ARGUMENTS Argument passed to the diagnostic tool is incorrect. For example using argument name not recognized by the diagnostic. See error message for details
104 INVALID_SERVICE_KEY Service key is rejected by AppOptics server
105 CONNECTION_FAILURE Failed to connect to the AppOptics server within the timeout limit of 5 seconds.
106 TRY_LATER AppOptics server rejected the diagnostic request with status TRY_LATER
107 LIMIT_EXCEEDED AppOptics server rejected the diagnostic request with status LIMIT_EXCEEDED

Script to gather diagnostic information to send to Support

If you need to contact Support, you can run a script to collect the required information. The script gathers all .NET Library log files, .NET Library registry entries, and IIS log files. It archives this information in a ZIP file. Send this ZIP file and the script log file to Support.

How do I run the script in Windows?

  1. Start a PowerShell session as an administrator.

  2. To gather diagnostic information, run the following commands:

    cd %PROGRAMFILES%\AppOptics\APM\dotnet
    .\agent-diagnostic-logs.ps1 -IncludeIISLogs

To learn more about the options included with the script, run the following command:

.\agent-diagnostic-logs.ps1 -Help

How do I run the script on the Azure platform?

On the Azure platform, if you are using AppOptics.Agent.AzureAppServices.Extension (an Azure site extension), complete the following steps to use this script.

  1. Open a web console command prompt.

  2. Run the following commands:

    cd D:\home\SiteExtensions\AppOptics.Agent.AzureAppServices.Extension\AppOptics\
    powershell -File "agent-diagnostic-logs.ps1"

    The ZIP file and script log file are created in the following location: 

    D:\home\SiteExtensions\AppOptics.Agent.AzureAppServices.Extension\AppOptics

How do I run the script in a Linux-based OS?

If the package dotnet-agent-package.tgz was extracted to a location other than ${HOME}/appoptics, change the path in the following commands.

  1. Open a shell.

  2. Run the following commands:

    cd ${HOME}/appoptics
    ./agent-diagnostic-logs.sh

To learn more about the options included with the script, run the following command:

./agent-diagnostic-logs.sh --help

.NET Agent (Windows, Linux, and SDK)

The following are troubleshooting steps for various .NET agent questions. See .NET Core SDK for additional SDK-specific questions.

Are requests being sent to the application?

In order for traces and metric data to be sent to your AppOptics dashboard, your application must be receiving requests.

  • For an IIS Hosted application, run the iisreset command again and ensure that the application is receiving requests.
  • For a Non-IIS application, restart the application and ensure that it is receiving requests.

Is the AppOptics .NET Agent or SDK able to connect to the AppOptics server?

Where can agent logs be found?

For information about the location of the AppOptics .NET Agent log files:

Do you have another profiler installed?

  • Only one APM agent can be used at a time. An APM agent using the CLR profiler interface can be identified by it's CLSID configured in the COR_PROFILER environment variable.
  • Search the application server environment variables and the registry (regedit.exe) for COR_PROFILER. There should only be one unique CLSID specified for the COR_PROFILER environment variable (note: the COR_PROFILER environment variable may be in multiple locations).
  • The AppOptics .NET Agent CLSID is COR_PROFILER={824293AD-22E2-4DAA-BC28-166C140543BE}.

Was the AppOptics Profiler loaded successfully?

  • To verify that the AppOptics Profiler component of the .NET Agent checked to see if it should be loaded, open Event Viewer and check under Windows Logs > Application for events from source AppOpticsProfiler and the following message:

    Message: .NET Runtime checking if AppOptics Profiler should be loaded. Profiler CLSID: '{824293AD-22E2-4DAA-BC28-166C140543BE}'.

    If the AppOptics Profiler was loaded successfully there should be another event immediately after the one above, from source .NET Runtime and the following message:

    .NET Runtime version <version number> - The profiler was loaded successfully. Profiler CLSID: '{824293AD-22E2-4DAA-BC28-166C140543BE}'. Process ID (decimal): <process id>. Message ID: [0x2507].
  • Check for any error events reported under Windows > Application from source of either .NET Runtime or AppOpticsProfiler.
  • If the AppOptics Profiler was not loaded successfully, the log level for the Event Viewer can be set to a higher level with the agent configuration property EventViewerLogLevel. Additional information will be logged to the Event Viewer by the AppOptics Profiler.

Has the agent been initialized?

  • One way to check if the agent has been initialized, is to look at the response headers returned. The agent will add the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the agent probably wasn’t loaded correctly.
  • Review the other troubleshooting options to try to resolve the issue.

Are requests being sampled for tracing?

  • Requests may not be sampled due to a low sample rate or connection issues.
  • The X-Trace response header or the SDK Method Trace.GetCurrentTraceId() can be used to obtain the current X-Trace:
    • The X-Trace ends in 00 if a request isn’t traced and ends in 01 if the request is being traced.
    • Some requests should have a X-Trace header returned that ends in 01. If all X-Trace headers being returned end in 00 the server may not be able to connect to the AppOptics collector.
  • Review the AppOptic .NET Agent logs to see if there are any connection issues logged.

Are there warnings reported by the agent?

  • Review the agent logs.
  • Sometimes configuration errors such as invalid formatting in the config file can lead to instrumentation problems.

How to increase log verbosity?

  • Increase the logging level verbosity to Debug to get more detailed information added to the agent logs. Use the agent configuration property LogLevel to change the logging level.
  • Run the iisreset command or restart the .NET Core application to have the new level take effect.

Is the service key configured correctly?

  • The service key is set the following ways. depending on the configuration:
    • added to the agent.config file located in the directory Program Files\AppOptics\APM\dotnet.
    • added to the agent.config file is included in the AppOptics.Instrumentation NuGet package
    • using the environment variable APPOPTICS_SERVICE_KEY.
  • The first part of the service key (before the colon :) should match one of the AppOptics API tokens listed in the AppOptics dashboard under Organization Settings > API Tokens.
  • The format of the API token recently changed. The new token format which is 71 characters long, may not work with previous versions of the agent (before v3.7.2). Try updating to the latest version of the agent which supports the new format.

Was an IIS site previously reporting data and then stopped reporting data?

  • Changes made to a server after the .NET agent is installed may affect the .NET agent. These changes include updates to IIS configuration or registry settings.

  • To see if changes have been made to the IIS configuration, see Is the AppOpticsInstrumentation HttpModule being loaded?. This question is only applicable to IIS sites using a classic mode application pool.

  • To determine if changes were made to the registry that affect the .NET agent, search for COR_ENABLE_PROFILING=1 in the registry. This value should be part of the data for the Environment value in the WSSVC key.

  • Instructions for finding a registry key:

    1. Open the Registry Editor (open a command prompt, type in regedit.exe and press Enter).
    2. From the menu, select Edit → Find.
    3. Enter the string to search (COR_ENABLE_PROFILING=1) and ensure Data is selected.
    4. Click Find Next.

    Alternatively, use the REG QUERY command in the command prompt, below is an example of the query and expected Environment value in the search results:

    reg query hklm\system\ /s /f COR_ENABLE_PROFILING=1 
    Environment  REG_MULTI_SZ  COR_ENABLE_PROFILING=1\0COR_PROFILER={824293AD-22E2-4DAA-BC28-166C140543BE}\0APPOPTICS_HOME_APM_DOTNET=C:\Program Files\AppOptics\APM\dotnet\ 
  • If COR_ENABLE_PROFILING=1 cannot be found in the registry the .NET agent will not instrument IIS sites. Reinstall the .NET agent to re-enable instrumentation for IIS sites and disable the process that is removing the registry entries.

  • If the process that is removing the registry entries cannot be disabled, reinstall the .NET agent and select to Instrument both IIS and NON-IIS .NET applications.

Is the AppOpticsInstrumentation HttpModule being loaded?

Starting with agent version 3.10.0, the AppOpticsInstrumentation HttpModule is only required for IIS sites using a classic mode application pool.

  • The installer for the AppOptics .NET Agent installs the AppOpticsInstrumentation module in IIS. The AppOpticsInstrumentation HttpModule is required for the instrumentation of IIS sites using a classic mode application pool. The application configuration for an IIS website could be configured to clear all HttpModules for a site and only enable specific HttpModules. If this is the case, the instrumentation of the IIS site will not work as the AppOpticsInstrumentation HttpModule would not be loaded.

    Example configuration in the web.config file which clears all HttpModules:

    <system.web>
      <httpModules>
        <clear/>
      </httpModules>
    </system.web>
  • If the IIS site being instrumented clears all HttpModules, the configuration will need to be updated to add the AppOpticsInstrumentation HttpModule. The version of the module installed by the AppOptics .NET agent installer must match the configuration added to the web.config file.

    Example configuration for adding the AppOpticsInstrumentation HttpModule:

    <system.web>
      <httpModules>
        <add type= "AppOptics.Agent.HttpInstrumentationModule, AppOptics.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" name="AppOpticsInstrumentation" />
      </httpModules>
    </system.web>
  • The .NET agent installs the AppOpticsInstrumentation HttpModule by adding the module to the configuration files used by IIS. A web deployment process may reset the IIS configuration files, thus disabling the AppOpticsInstrumentation HttpModule. To determine if the AppOpticsInstrumentation HttpModule has been removed you need to know the .NET CLR version of the application pool and if it's running in 32bit or 64bit mode:

    1. Right-click on the application pool name and select Advanced Settings…
    2. Look at these settings: .NET CLR Version - either v2.0 or v4.0 and Enable 32-bit Applications - either True or False
    3. Location of config file:
      • 32 bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework\v2.0.<build number of version installed>\CONFIG\web.config
      • 64 bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework64\v2.0.<build number of version installed>\CONFIG\web.config
      • 32 bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework\v4.0.<build number of version installed>\CONFIG\web.config
      • 64 bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework64\v4.0.<build number of version installed>\CONFIG\web.config
    4. In the web.config file, the element: system.web/httpModules should include an element for the AppOpticsInstrumentation HttpModule:
    5. <system.web>
        ...
        <httpModules>
          ....
          <add name="AppOpticsInstrumentation" type="AppOptics.Agent.HttpInstrumentationModule, AppOptics.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
        </httpModules>
      </system.web>

How can you configure a service key for an application pool?

  • Information on how to configure a different service key for an application pool can be found here.

Are metrics not being displayed for a non-IIS application?

  • Ensure the Instrument NON-IIS .NET Applications component was selected during the agent installation.
  • For non-IIS applications additional configuration needs to be added to the agent.config file located in the directory Program Files\AppOptics\APM\dotnet. The Instrumenting Non-IIS Applications section describes the configuration that needs to be added to the agent.config file.
  • The supported auto-instrumented non-IIS applications include standalone WCF services. For other types of non-IIS applications the custom instrumentation SDK would need to be used.

Are custom instrumentation spans or other spans missing from traces?

  • If a method is trivial enough, the CLR might decide to inline the method, which is optimization that replaces a method call with the method body. If that happens, the method doesn't exist anymore and so AppOptics can't instrument it.

  • To prevent this, inlining can be disabled either for an individual method, as shown below, or for all apps via the inlining config parameter.

    [MethodImpl(MethodImplOptions.NoInlining)]
    void YourMethod()
    { 
      // do something 
    }

Does the .NET Agent need to be upgraded?

  • Release notes for the .NET agent can be found here.
  • The current installed version of the .NET agent can be found when viewing Program and Features in the Control Panel. The shortcut for opening the Program and Features window is appwiz.cpl. The name of the agent is AppOptics .NET Agent <version number>.
  • Upgrade instructions for the .NET agent can be found here.

Is the agent configuration file valid?

You can run the Diagnostic Tool which checks the configuration XML validity. An example of the diagnostic output when the XML is invalid due to missing comment terminator:

C:\Program Files\AppOptics\APM\dotnet>dotnet_diagnostic_tool.exe
...
Trace: OboeInit.LoadSettings(): Invoked ('C:\Program Files\AppOptics\APM\dotnet\agent.config').
Error: OboeInit.LoadSettings(): An error has occured while reading and/or parsing the instrumentation settings. An XML comment cannot contain '--', and '-' cannot be the last character. Line 27, position 5.

Are your .NET Core/5+ IIS websites missing expected spans or code profiles and is the service entry span IIS?

For .NET Core/5+ IIS websites the service entry span should be aspnet-core. If this is not the case check the application pool setting: .NET CLR Version for the .NET Core/5+ IIS website. If the .NET CLR Version is set to v2.0 or v4.0 rather than No Managed Code this will cause the .NET Framework instrumentation to check if it should be loaded for the IIS website. The .NET Framework instrumentation can be disabled for an application pool by using the iisInstrumentation attribute of the applicationPool element in the agent.config file. See Configuration (legacy agent) for more information.

.NET Core SDK

The following are additional troubleshooting steps for .NET SDK-only questions.

The troubleshooting questions below assume the AppOptics.Instrumentation NuGet package has been added to your application. As well, either the AppOptics middleware has been added to your ASP.NET Core application or SDK methods have been added to your application or both.

Has the agent been initialized?

  • If the AppOptics middleware is being used, one way to check if the agent has been initialized, is to look at the response headers returned. The agent will add the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the agent probably wasn’t loaded correctly.
  • Review the other troubleshooting options to try to resolve the issue.

Are requests being sampled for tracing?

  • Requests may not be sampled due to a low sample rate or connection issues.
  • Review the AppOptic .NET Agent logs to see if there are any connection issues logged.
  • The AppOptics middleware will add the X-Trace header to the response or the SDK Method Trace.GetCurrentTraceId() can be used to obtain the current X-Trace. The X-Trace ends in 00 if a request isn’t traced and ends in 01 if the request is being traced.
  • Some requests should have an X-Trace that ends in 01. If all X-Trace’s end in 00 the server may not be able to connect to the AppOptics collector.

How to increase log verbosity?

  • Increase the logging level verbosity to Debug to get more detailed information added to the agent logs. Use the agent configuration property LogLevel to change the logging level.
  • Restart the .NET Core application to have the new level take effect.

Is the service key configured correctly?

  • The service key is added to the agent.config file. The agent.config file is included in the AppOptics.Instrumentation NuGet package. Alternatively, the service key can be set using the environment variable APPOPTICS_SERVICE_KEY.
  • The first part of the service key (before the colon :) should match one of the AppOptic API tokens listed in the AppOptics dashboard under Organization Settings > API Tokens.
  • The format of the API token recently changed. The new token format which is 71 characters long, may not work with previous versions of the agent (before v3.7.2). Try updating to the latest version of the agent which supports the new format.

Does the .NET core site need to support characters such as &, *, <, >, : in the path?

  • If the .NET core site requires characters such as as &, *, <, >, : in the path and the .NET Agent is also installed on the server this may cause the .NET core site to return a 400 response code. The .NET agent installs an HttpModule which causes sites running in managed code application pools to go through the ASP.NET request validation. By default the ASP.NET request validation rejects certain characters in the path and a 400 response code is returned.

  • One of the following options can be used to resolve this issue:

    • Option 1: In IIS Manager, open the Edit Application Pool window and set the .NET CLR version to No Managed Code. See Step 6 in this link for more information.
    • Option 2: Uninstall the .NET agent if only .NET core sites need to be instrumented on the server.
    • Option 3: Add the following to the .NET core site's web.config to stop the validation of the characters in the path:
      <system.web>
        <httpRuntime requestPathInvalidCharacters="" />
      </system.web>

Can the Linux SDK find and load the native shared library?

  • On Linux platforms, the .NET Core Runtime looks for the oboe_<major>_<minor>_<patch>.so shared library under the <published-dotnet-core-app>/runtimes/linux-x64/native/ or <published-dotnet-core-app>/runtimes/linux-x86/native/ directory depending on if the application is 64 bit or 32 bit. This is the default behaviour for .NET Core Runtime when searching for native shared libraries.

    For a 64 bit application the directory structure should be:

    <published-dotnet-core-app>
      |--- <dotnet-core-app>
      |--- agent.config
      |--- AppOptics.Instrumentation.dll
      |--- AppOptics.Instrumentation.Internal.dll
      |--- runtimes
           |--- linux-x64
                |---native
                |---libe_sqlite3.so
                |---libuv.so
                |---oboe_<major>_<minor>_<patch>.so
      ...

    For a 32 bit application the directory structure should be:

    <published-dotnet-core-app>
      |--- <dotnet-core-app>
      |--- agent.config
      |--- AppOptics.Instrumentation.dll
      |--- AppOptics.Instrumentation.Internal.dll
      |--- runtimes
           |--- linux-x86
                |---native
                    |---libe_sqlite3.so
                    |---libuv.so
                    |---oboe_<major>_<minor>_<patch>.so
      ...

    If the oboe_<major>_<minor>_<patch>.so shared library is moved to a different location or the directory structure doesn’t follow .NET conventions for native shared libraries, then the LD_LIBRARY_PATH environment variable can be used to update the search library path, as described in the ld man page.

    export LD_LIBRARY_PATH=<path-to-oboe_<major>_<minor>_<patch>.so>
  • You can identify loading issues for the shared library by:

    Using the strace tool. Execute strace dotnet and look for lstat, stat, and open functions in the output to see which paths are searched for the shared libraries.

    strace dotnet <dotnet-core-app> > strace.output.log 2>&1

    Using the LD_DEBUG environment variable. Set it to libs for the dotnet process to capture the search path for shared libraries, as described in the ld man page.

    LD_DEBUG=libs dotnet <dotnet-core-app> > lddebug.output.log 2>&1

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.