Documentation forSolarWinds Observability

Troubleshoot the .NET Library

If you are not receiving traces and metrics in the Traces Explorer or Metrics Explorer, here are some options for troubleshooting. See Guided troubleshooting questions to step through the process of identifying your .NET troubleshooting issue(s) and potential solution(s).

Diagnostic tool

How do I run the diagnostic tool in Windows?

A connectivity diagnostic tool is available with the .NET Library. To run it in Windows, use the command line to run the Core, the Framework, or the Azure App Service version of the executable located in the library installation directory:

.NET Framework %PROGRAMFILES%\SolarWinds\APM\dotnet> dotnet_diagnostic_tool.exe
.NET Core %PROGRAMFILES%\SolarWinds\APM\dotnet> dotnet_diagnostic_tool_core.exe
Azure App Service

D:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWindsAPM\dotnet-runtime\dotnet_diagnostic_tool_core.exe

where dotnet-runtime is a supported .NET Core Runtime installed on the machine (for example, net5.0 for .NET 5, netcoreapp3.1 for .NET Core 3.1)

The optional parameters are:

/ServiceKey The Service key to be used for the diagnostic tool. By default, it uses the current ServiceKey configured globally for the host. It does not pick up any Service keys defined at the <iisSites>, <applicationPools>, or <applications> levels.
/Verbose Enables printing to console of verbose level log messages from the diagnostic run.
/Timeout Timeout for testing the connection in milliseconds. If the connection doesn’t succeed before the timeout is reached the connection check will timeout.  

For example, in Windows (where %PROGRAMFILES% is in the standard C:\Program Files\ directory) to verify connection to the my-service service using a specific Service key and a timeout of 5000ms (5 seconds):

C:\Program Files\SolarWinds\APM\dotnet> dotnet_diagnostic_tool.exe /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service /Timeout 5000

Look for the message Has connection succeeded? YES, which indicates that the connection succeeded.

If there are any errors, review the log files %PROGRAMFILES%\SolarWinds\APM\dotnet\dotnet_diagnostic_tool.log and C:\Windows\Temp\SolarWinds\DotNetAgentLog_dotnet_diagnostic_tool*.

How do I run the diagnostic tool in Linux?

A connectivity diagnostic tool is available with the .NET Library. To run it in Linux, in a shell session run the executable <dotnet-runtime>/dotnet_diagnostic_tool_core located under the library installation directory, where <dotnet-runtime> is a supported .NET Core Runtime installed on the machine (for example, net5.0 for .NET 5, netcoreapp3.1 for .NET Core 3.1).

The following example assumes you have already set the library installation directory environment variable SW_APM_HOME_DOTNET:

cd ${SW_APM_HOME_DOTNET}/<dotnet-runtime> 
./dotnet_diagnostic_tool_core 

The optional parameters are:

/ServiceKey The Service key to be used for the diagnostic tool. By default, it uses the current ServiceKey configured globally for the host.
/Verbose Enables printing to the console of verbose level log messages from the diagnostic run.
/Timeout Timeout for testing the connection in milliseconds. If the connection doesn’t succeed before the timeout is reached the connection check will timeout.  

For example, to verify connection to the my-service service using a specific Service key and a timeout of 5000ms (5 seconds):

dotnet_diagnostic_tool_core/dotnet_diagnostic_tool_core /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service /Timeout 5000

Look for the message Has connection succeeded? YES, which indicates that the connection succeeded.

If there are any errors, review the log files /var/tmp/solarwinds/DotNetAgentLog_dotnet_diagnostic_tool_core*.

What are the diagnostic tool's exit codes?

The diagnostic tool provides the following exit codes:

Exit status code Name Description
0 OK Diagnostics successful.
101 UNKNOWN_ERROR Unexpected error. See the error message for details.
102 INVALID_FORMAT_ARGUMENT Argument passed to the diagnostic tool is not in a valid format. For example, the Service key is too short. See the error message for details.
103 INVALID_ARGUMENTS Argument passed to the diagnostic tool is incorrect. For example, it uses an argument name not recognized by the diagnostic tool. See the error message for details
104 INVALID_SERVICE_KEY The Service key is rejected by the SolarWinds APM collector.
105 CONNECTION_FAILURE Failed to connect to the SolarWinds APM collector within the timeout limit of 5 seconds.
106 TRY_LATER The SolarWinds APM collector rejected the diagnostic request with the status TRY_LATER.
107 LIMIT_EXCEEDED The SolarWinds APM collector rejected the diagnostic request with the status LIMIT_EXCEEDED.
108 ERROR_INVALID_APM_COLLECTOR The format of the ingestion endpoint is invalid.

.NET Library (Windows, Linux, and SDK)

The following are troubleshooting steps for various .NET Library questions. See .NET Core SDK for additional SDK-specific questions.

Are requests being sent to the application?

In order for traces and metric data to be sent to the Traces Explorer and Metrics Explorer, your application must be receiving requests.

  • For an IIS Hosted application, run the iisreset command again and ensure that the application is receiving requests.
  • For a Non-IIS application, restart the application and ensure that it is receiving requests.

Is the SolarWinds Observability .NET Library or SDK able to connect to the SolarWinds APM collector?

Where can library logs be found?

For information about the location of the SolarWinds Observability .NET Library log files, see:

You can also run a script to gather diagnostic information, including all .NET Library log files, .NET Library registry entries, and IIS log files.

Do you have another profiler installed?

Only one APM library can be used at a time. An APM library using the CLR profiler interface can be identified by its CLSID configured in the COR_PROFILER environment variable:

  1. Search the application server environment variables and the registry (regedit.exe) for COR_PROFILER. There should be only one unique CLSID specified for the COR_PROFILER environment variable.

    The COR_PROFILER environment variable might be in multiple locations.

  2. The SolarWinds Observability .NET Library CLSID is COR_PROFILER={AAAA7777-11FE-4F94-8EDA-312C0EDF4141}.

Did the SolarWinds Observability Profiler load successfully?

  1. Verify that the SolarWinds Observability Profiler component of the .NET Library performed a test to see if it should be loaded. Open the Event Viewer and look under Windows Logs > Application for events from source solarwinds_apm_profiler and the following message:

    Message: .NET Runtime checking if SolarWinds Observability Profiler should be loaded. Profiler CLSID: '{AAAA7777-11FE-4F94-8EDA-312C0EDF4141}'.
  2. Verify that the SolarWinds Observability Profiler loaded successfully. Look for another event immediately after the one above, from source .NET Runtime with the following message:

    .NET Runtime version <version number> - The profiler was loaded successfully. Profiler CLSID: '{AAAA7777-11FE-4F94-8EDA-312C0EDF4141}'. Process ID (decimal): <process id>. Message ID: [0x2507].
  3. Check for any error events reported under Windows > Application from the source of either .NET Runtime or solarwinds_apm_profiler.

If the SolarWinds Observability Profiler was not loaded successfully, set the log level for the Event Viewer to a higher level with the library configuration property EventViewerLogLevel. Additional information will be logged to the Event Viewer by the SolarWinds Observability Profiler.

Has the library been initialized?

Look at the response headers returned. The library adds the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the library probably wasn’t loaded correctly.

Review the other troubleshooting options to resolve the issue.

Are requests being sampled for tracing?

Requests might not be sampled due to a low sample rate or connection issues. The X-Trace response header or the SDK Method Trace.GetCurrentTraceId() can be used to obtain the current trace context:

  • The trace context ends in 00 if a request isn’t traced. It ends in 01 if the request is being traced.

  • If tracing is enabled (the default configuration), there should be some requests with a trace context ending in 01. If all returned traced contexts end in 00, the server might not be able to connect to the SolarWinds APM collector.

Review the .NET Library logs to see if there are any connection issues logged.

Are warnings being reported by the library?

Review the .NET Library logs. Sometimes configuration errors such as invalid formatting in the solarwinds_apm.config config file can lead to instrumentation problems.

How do I get more information in the logs?

Increase the logging level verbosity to Debug to get more detailed information added to the library logs. Use the library configuration property LogLevel to change the logging level.

Run the iisreset command or restart the .NET Core application to have the new level take effect.

Is the Service key configured correctly?

The Service key is set the following ways, depending on the configuration:

  • By adding it to the solarwinds_apm.config file in the %PROGRAMFILES% directory (typically C:\Program Files\): %PROGRAMFILES%\SolarWinds\APM\dotnet

  • By adding it to the solarwinds_apm.config file included in the SolarWindsAPM.Agent and SolarWindsAPM.Agent.AzureAppServices.Extension NuGet packages

  • By using the environment variable SW_APM_SERVICE_KEY

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when adding the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

Did an IIS site that was previously reporting data stop reporting data?

Changes made to a server after the .NET Library is installed can affect the .NET Library. These changes include updates to the IIS configuration or registry settings.

If your IIS site uses a classic mode application pool, see if changes were made to the IIS configuration. See Is the SolarWindsAPMInstrumentation HttpModule being loaded?

To determine if changes were made to the registry that affect the .NET Library, search for COR_ENABLE_PROFILING=1 in the registry. This value should be part of the data for the Environment value in the WSSVC key. To find the registry key:

  1. Open the Registry Editor (enter regedit.exe in a command prompt).
  2. From the menu, select Edit > Find.
  3. Enter the string to search (COR_ENABLE_PROFILING=1) and ensure Data is selected.
  4. Click Find Next.

If COR_ENABLE_PROFILING=1 cannot be found in the registry, the .NET Library will not instrument IIS sites. Reinstall the .NET Library to re-enable instrumentation for IIS sites and disable the process that is removing the registry entries.

If another process is removing your registry entries and that process cannot be disabled, reinstall the .NET Library and select Instrument both IIS and NON-IIS .NET applications.

Is the SolarWindsAPMInstrumentation HttpModule being loaded?

The SolarWindsAPMHttpModule is required only for IIS sites using a classic mode application pool.

Is the web.config clearing all HttpModules for the site?

The installer for the SolarWinds Observability .NET Library installs the SolarWindsAPMInstrumentation module in IIS. The SolarWindsAPMInstrumentation HttpModule is required for the instrumentation of IIS sites using a classic mode application pool. The application configuration for an IIS website could be configured to clear all HttpModules for a site and enable only specific HttpModules. If this is the case, the instrumentation of the IIS site will not work because the SolarWindsAPMInstrumentation HttpModule would not be loaded.

The following is an example of a configuration in the web.config file that clears all HttpModules:

<system.web>
  <httpModules>
    <clear/>
  </httpModules>
</system.web>

To resolve this, update the configuration to add the SolarWindsAPMInstrumentation HttpModule. The version of the module installed by the SolarWinds Observability .NET Library installer must match the configuration added to the web.config file.

The following is an example of a configuration for adding the SolarWindsAPMInstrumentation HttpModule:

<system.web>
  <httpModules>
    <add name="SolarWindsAPMInstrumentation" type="SolarWinds.Apm.Agent.HttpInstrumentationModule, SolarWinds.Apm.Agent, Version=5.0.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
  </httpModules>
</system.web>

Is a web deployment process resetting IIS config files?

The SolarWindsAPMHttpModule is required only for IIS sites using a classic mode application pool.

The .NET Library installs the SolarWindsAPMHttpModule by adding the module to the config files used by IIS. A web deployment process might reset the IIS config files, thus disabling the SolarWindsAPMHttpModule. To determine if the SolarWindsAPMHttpModule has been removed, identify the .NET CLR version of the application pool and determine if it is running in 32-bit or 64-bit mode:

  1. Right-click the application pool name and select Advanced Settings.

  2. Look at these settings: .NET CLR Version - either v2.0 or v4.0 and Enable 32-bit Applications - either True or False

  3. Location of config file:

    • 32-bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework\v2.0.<build number of version installed>\CONFIG\web.config
    • 64-bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework64\v2.0.<build number of version installed>\CONFIG\web.config
    • 32-bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework\v4.0.<build number of version installed>\CONFIG\web.config
    • 64-bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework64\v4.0.<build number of version installed>\CONFIG\web.config

In the web.config file, the element: system.web/httpModules should include an element for the SolarWindsAPMHttpModule:

<system.web>
  ...
  <httpModules>
    ....
    <add name="SolarWindsAPMInstrumentation" type="SolarWindsAPM.Agent.HttpInstrumentationModule, SolarWindsAPM.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
  </httpModules>
</system.web>

If the element is not included, uninstall the library and install the library again.

How can you configure a Service key for an application pool?

Information on how to configure a different Service key for an application pool can be found here.

Are metrics not being displayed for a non-IIS application?

Ensure the Instrument NON-IIS .NET Applications component was selected during the library installation.

For non-IIS applications, additional configuration needs to be added to the solarwinds_apm.config file in the %PROGRAMFILES% directory (typically C:\Program Files\) located in: %PROGRAMFILES%\SolarWinds\APM\dotnet. See Install for stand-alone, non-IIS applications for the configuration that needs to be added to the solarwinds_apm.config file.

The supported auto-instrumented non-IIS applications include standalone WCF services. For other types of non-IIS applications, use the Instrumentation SDK.

Are custom instrumentation spans or other spans missing from traces?

If a method is trivial enough, the CLR might decide to inline the method, which is optimization that replaces a method call with the method body. If that happens, the method doesn't exist anymore and SolarWinds Observability can't instrument it. To prevent this, inlining can be disabled either for an individual method, as shown below, or for all apps via the Inlining parameter.

[MethodImpl(MethodImplOptions.NoInlining)]
void YourMethod()
{ 
  // do something 
}

Is the library config file valid?

You can run the Diagnostic tool, which checks the configuration XML validity. An example of the diagnostic output when the XML is invalid due to missing comment terminator:

C:\Program Files\SolarWinds\APM\dotnet>dotnet_diagnostic_tool.exe
...
Trace: OboeInit.LoadSettings(): Invoked ('C:\Program Files\SolarWinds\APM\dotnet\solarwinds_apm.config').
Error: OboeInit.LoadSettings(): An error has occured while reading and/or parsing the instrumentation settings. An XML comment cannot contain '--', and '-' cannot be the last character. Line 27, position 5.

Are your .NET Core/5+ IIS websites missing expected spans or code profiles and is the service entry span IIS?

For .NET Core/5+ IIS websites, the service entry span should be aspnet-core. If this is not the case, check the application pool setting: .NET CLR Version for the .NET Core/5+ IIS website. If the .NET CLR Version is set to v2.0 or v4.0 rather than No Managed Code, this will cause the .NET Framework instrumentation to check if it should be loaded for the IIS website.

The .NET Framework instrumentation can be disabled for an application pool by using the iisInstrumentation attribute of the applicationPool element in the solarwinds_apm.config file. See Configure the .NET Library for more information.

.NET Core SDK

The following are additional troubleshooting steps for .NET SDK-only questions.

The troubleshooting questions below assume the SolarWindsAPM.Agent NuGet package has been added to your application, and either the SolarWinds Observability middleware was added to your ASP.NET Core application, SDK methods were added to your application, or both the middleware and SDK methods were added to your application.

Has the library been initialized?

If the SolarWinds Observability middleware is being used, you can determine if the library has been initialized by looking at the response headers returned. The library adds the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the library probably wasn’t loaded correctly.

Review the other troubleshooting options to try to resolve the issue.

Are requests being sampled for tracing?

Requests might not be sampled due to a low sample rate or connection issues. Review the NET library logs to see if there are any connection issues logged.

To troubleshoot connection issues, look at trace context value. The SolarWinds Observability middleware adds the X-Trace header to the response, and the SDK Method Trace.GetCurrentTraceId() returns the trace context. The trace context ends in 00 if a request isn’t traced. It ends in 01 if the request is being traced.

If tracing is enabled (the default configuration), there should be some trace contexts that end in 01. If all trace contexts end in 00, this might indicate that the server is not able to connect to the SolarWinds APM collector.

How to increase log verbosity?

Increase the logging level verbosity to Debug to get more detailed information added to the library logs. Use the library configuration property LogLevel to change the logging level.

Restart the .NET Core application to have the new level take effect.

Is the Service key configured correctly?

The Service key is set in one of the following ways, depending on the configuration:

  • The solarwinds_apm.config file included in the SolarWindsAPM.Agent and SolarWindsAPM.Agent.AzureAppServices.Extension NuGet packages.

  • The environment variable SW_APM_SERVICE_KEY.

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when adding the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

Does the .NET core site need to support characters such as &, *, <, >, : in the path?

If the .NET core site requires characters such as &, *, <, >, : in the path and the .NET Library is installed on the same server, this might cause the .NET core site to return a 400 response code. The .NET Library installs an HttpModule which uses ASP.NET request validation for sites running in managed code application pools. By default, the ASP.NET request validation rejects certain characters in the path and returns a 400 response code.

One of the following options can be used to resolve this issue:

Can the Linux SDK find and load the native shared library?

On Linux platforms, the .NET Core Runtime looks for the oboe_<major>_<minor>_<patch>.so shared library under the <published-dotnet-core-app>/runtimes/linux-x64/native/ directory, where <published-dotnet-core-app> is the location of the app. This is the default behavior for .NET Core Runtime when searching for native shared libraries.

The directory structure should be:

<published-dotnet-core-app>
  |--- <dotnet-core-app>
  |--- solarwinds_apm.config
  |--- SolarWindsAPM.Instrumentation.dll
  |--- SolarWindsAPM.Instrumentation.Internal.dll
  |--- runtimes
       |--- linux-x64
            |---native
                |---libe_sqlite3.so
                |---libuv.so
                |---oboe_<major>_<minor>_<patch>.so
  ...

If the oboe_<major>_<minor>_<patch>.so shared library is moved to a different location or the directory structure doesn’t follow .NET conventions for native shared libraries, then the LD_LIBRARY_PATH environment variable can be used to update the search library path, as described in the LD.SO(8) page in the Linux Programmer's Manual.

export LD_LIBRARY_PATH=<path-to-oboe>/oboe_<major>_<minor>_<patch>.so 

You can identify loading issues for the shared library by:

  • Using the strace tool, as described in the Linux man pages site. Execute strace dotnet and look for lstat, stat, and open functions in the output to see which paths are searched for the shared libraries.

    strace dotnet <dotnet-core-app> > strace.output.log 2>&1
  • Using the LD_DEBUG environment variable. Set it to libs for the dotnet process to capture the search path for shared libraries, as described in the LD.SO(8) page.

    LD_DEBUG=libs dotnet <dotnet-core-app> > lddebug.output.log 2>&1

Script to gather diagnostic information to send to Support

If you need to contact Support, you can run a script to collect the required information. The script gathers all .NET Library log files, .NET Library registry entries, and IIS log files. It archives this information in a ZIP file. Send this ZIP file and the script log file to Support.

How do I run the script in Windows?

  1. Start a PowerShell session as an administrator.

  2. To gather diagnostic information, run the following commands:

    cd "%PROGRAMFILES%\SolarWinds\APM\dotnet"
    .\agent-diagnostic-logs.ps1

To learn more about the options included with the script, run the following command:

.\agent-diagnostic-logs.ps1 -Help

How do I run the script on the Azure platform?

On the Azure platform, if you are using SolarWindsAPM.Agent.AzureAppServices.Extension (an Azure site extension), complete the following steps to use this script.

  1. Open a web console command prompt.

  2. Run the following commands:

    cd d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds
    powershell -File "agent-diagnostic-logs.ps1"

    The ZIP file and script log file are created in the following location: 

    d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds

How do I run the script in a Linux-based OS?

If the package solarwinds-apm-dotnet-package.tgz was extracted to a location other than ${HOME}/solarwinds, change the path in the following commands.

  1. Open a shell.

  2. Run the following commands:

    cd ${HOME}/solarwinds
    ./agent-diagnostic-logs.sh

To learn more about the options included with the script, run the following command:

./agent-diagnostic-logs.sh --help

Guided troubleshooting questions

For each troubleshooting question, select the options below it that best describe your situation to view suggested solutions.

Can the IIS website or .NET process start?

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.