Troubleshoot the .NET Library
If you are not receiving traces and metrics in the Traces Explorer or Metrics Explorer, here are some options for troubleshooting. See Guided troubleshooting questions to step through the process of identifying your .NET troubleshooting issue(s) and potential solution(s).
Guided troubleshooting questions
For each troubleshooting question, select the options below it that best describe your situation to view suggested solutions. If you need further help after following this guide, please run the script to gather diagnostic information and provide the logs to support.
Can the IIS website or .NET process start?
Are X-Trace headers added to the response header of the website?
Is the X-Trace header a sampled header (ends in 01)?
Try a few requests to check if X-Trace headers start ending in 01.
Tracing requests should be working, and data should appear in the Metrics Explorer or Traces Explorer.
If traces still aren’t appearing in the Traces Explorer:
- Check if broken traces being created.
- Check the library logs to see if any errors are being reported.
Make sure the library can connect to the SolarWinds APM collector.
Use the diagnostic tool to check the library connection with the collector.
Check to make sure the API token used for the Service key is valid.
The diagnostic tool will return an Invalid Service key response code if the API token is invalid.
Look at the format of the solarwinds_apm.config
file to make sure it's valid.
If the XML in the solarwinds_apm.config
file is invalid, the diagnostic tool will display the message [solarwinds_apm.config]: ServiceKey setting not found in solarwinds_apm.config file
.
The error message OboeInit.LoadSettings(): An error has occurred while reading and/or parsing the instrumentation settings, <additional details about the error in the configuration file>
will also show in one of the following locations:
- In the logs for the diagnostic tool (
SolarWindsAPM_DotNetAgentLog_dotnet_diagnostic_tool_<YYYY-MM-DD>.log
) - In the logs for the instrumented site or application (an example log file name for an IIS site is
SolarWindsAPM_DotNetAgentLog_LMW3SVC<site_id>ROOT-<YYYY-MM-DD>.log
)
Is a proxy used?
If a proxy is used, ensure the proxy configuration is in the solarwinds_apm.config
file.
Make sure outgoing connections are not blocked.
See Firewall or access control requirements for a list of URLs and ports to open.
Make sure the library logs contain a successful Initialization message.
Example:
Level=INFO Message=OboeWrapper.Init(): Invoked (layerName=IIS kvps=109 result=True processID=27520 serviceName=test-service). (Scope=HttpRuntimeInstrumenter)
There is probably an issue with the library being loaded into the instrumented process.
Does the Event Viewer contain a message indicating if the profiler was loaded into the process or not?
Verify the library is installed on a supported platform.
See the supported components for the .NET Library.
For classic ASP (e.g. .asp
URLs) applications, enable SolarWindsAPMHttpModule
.
To support classic ASP, the SolarWindsAPMHttpModule
must be enabled. See SolarWindsAPMHttpModule to monitor Classic ASP applications.
Look at the library logs to see if any errors have been logged.
See Where can library logs be found?
IIS should have been stopped before installing the library.
If IIS wasn’t stopped before running the installation, stop IIS, uninstall the library, and reinstall the library. Then start IIS.
Check if another profiler was installed and loaded instead of the SolarWinds Observability APM library profiler.
Do not use other APM libraries with the SolarWinds Observability .NET Agent.
For IIS hosted .NET 6+ applications, check the application pool settings.
The application pool setting .NET CLR Version for the .NET 6+ IIS website should be set to No Managed Code. See Are your .NET 6+ IIS websites missing expected spans or code profiles and is the service entry span IIS?
Verify the library is installed on a supported platform.
See the supported components for the .NET Library.
Check if another profiler was installed and loaded instead of the SolarWinds Observability APM library profiler.
Do not use other APM libraries with the SolarWinds Observability APM library.
Look at the format of the solarwinds_apm.config
file to make sure it's valid.
If the XML in the solarwinds_apm.config
file is invalid, the diagnostic tool will display the message [solarwinds_apm.config]: ServiceKey setting not found in solarwinds_apm.config file
.
The error message OboeInit.LoadSettings(): An error has occurred while reading and/or parsing the instrumentation settings, <additional details about the error in the configuration file>
will also show in one of the following locations:
- In the logs for the diagnostic tool (
SolarWindsAPM_DotNetAgentLog_dotnet_diagnostic_tool_<YYYY-MM-DD>.log
) - In the logs for the instrumented site or application (an example log file name for an IIS site is
SolarWindsAPM_DotNetAgentLog_LMW3SVC<site_id>ROOT-<YYYY-MM-DD>.log
)
For NON-IIS applications, check if the process name is set up correctly.
Make sure the NON-IIS application process name is configured in the solarwinds_apm.config
file under the applications
section. See application.
Check to see if the application received requests.
For traces and metric data to be sent to SolarWinds Observability, your application must receive requests.
Make sure the installation of the library was successful.
To find problems with the installation:
- Check the installation directory for deployed files.
- Review the installer logs to see if there were any errors with the installation.
- Verify the user that installed the library had Administrator access.
Make sure the profiler environment variables were set up correctly.
Use the Microsoft Process Explorer to confirm the profiler environment variables are set on the instrumented process.
Agent Installer:
.NET Framework Env Variables:
COR_ENABLE_PROFILING = 1
COR_PROFILER = {AAAA7777-11FE-4F94-8EDA-312C0EDF4141}
.NET 6+ Env Variables:
CORECLR_ENABLE_PROFILING = 1
CORECLR_PROFILER = {AAAA7777-11FE-4F94-8EDA-312C0EDF4141}
Look at the format of the solarwinds_apm.config
file to make sure it's valid.
If the XML in the solarwinds_apm.config
file is invalid, the diagnostic tool will display the message [solarwinds_apm.config]: ServiceKey setting not found in solarwinds_apm.config file
.
The error message OboeInit.LoadSettings(): An error has occurred while reading and/or parsing the instrumentation settings, <additional details about the error in the configuration file>
will also show in one of the following locations:
- In the logs for the diagnostic tool (
SolarWindsAPM_DotNetAgentLog_dotnet_diagnostic_tool_<YYYY-MM-DD>.log
) - In the logs for the instrumented site or application (an example log file name for an IIS site is
SolarWindsAPM_DotNetAgentLog_LMW3SVC<site_id>ROOT-<YYYY-MM-DD>.log
)
For websites, IIS should have been stopped before installing the library.
If IIS wasn’t stopped before running the installation, stop IIS, uninstall the library, and reinstall the library. Then start IIS.
For websites, iisreset
should have been run after installing the library.
If it was not run, run iisreset
in the command prompt.
For NON-IIS applications, was NON-IIS selected in the installer?
Check if the process name is set up correctly.
Make sure the NON-IIS application process name is configured in the solarwinds_apm.config
file under the applications
section. See application.
Run the installer again and either select Instrumented NON-IIS .NET Framework Applications or select Instrument NON-IIS .NET 6+ Applications, depending on the version of .NET used by your application.
Was the server restarted after installing the library?
Try restarting the server.
Verify the library is installed on a supported platform.
See the supported components for the .NET Library.
Look at the Event Viewer to see if there's an event indicating what the error is.
If an error states the SolarWindsAPMInstrumentation Http Module
cannot be loaded, stop IIS, uninstall the library, and reinstall the library. Then start IIS.
Connection script
The connection script attempts to establish a secure connection to the SolarWinds APM collector. If the trusted CA certificate required for the connection is not currently in the certificate store, this script should trigger the Windows Automatic Root Certificates Update component, which ensures subsequent connections succeed.
The following explanation of the Automatic Root Certificate Update component comes from Microsoft documentation for Automatic Root Certificates Update Configuration.
The Automatic Root Certificates Update component is designed to automatically check the list of trusted authorities on the Microsoft Windows Update Web site. Specifically, there is a list of trusted root certification authorities (CAs) stored on the local computer.
When an application is presented with a certificate issued by a CA, it will check the local copy of the trusted root CA list. If the certificate is not in the list, the Automatic Root Certificates Update component will contact the Microsoft Windows Update Web site to see if an update is available.
If the CA has been added to the Microsoft list of trusted CAs, its certificate will automatically be added to the trusted certificate store on the computer.
The Windows Automatic Root Certificates Update component must be enabled for the connection script to work. Make sure this component is not disabled by a group policy. See How to enable the automatic root certificates update.
How do I run the connection script in Windows?
-
Start a PowerShell session as an administrator.
-
To establish a secure connection to collector, run the following commands:
cd "%PROGRAMFILES%\SolarWinds\APM\dotnet" .\establish-collector-connection.ps1 -Collector apm.collector.xx-yy.cloud.solarwinds.com
To learn more about the options included with the script, run the following command:
.\establish-collector-connection.ps1 -Help
How do I run the script in an Azure App Service instance?
If you are using SolarWindsAPM.Agent.AzureAppServices.Extension
(an Azure site extension), complete the following steps to run the script.
-
Open a web console command prompt.
-
Run the following commands:
cd d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds powershell -File "establish-collector-connection.ps1" -Collector apm.collector.xx-yy.cloud.solarwinds.com
The script log file is created in the following location:
d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds
Diagnostic tool
How do I run the diagnostic tool in Windows?
A connectivity diagnostic tool is available with the .NET Library. To run it in Windows, use the command line to run the Core, the Framework, or the Azure App Service version of the executable located in the library installation directory:
.NET Framework | %PROGRAMFILES%\SolarWinds\APM\dotnet> dotnet_diagnostic_tool.exe
|
.NET Core | %PROGRAMFILES%\SolarWinds\APM\dotnet> dotnet_diagnostic_tool_core.exe
|
Azure App Service |
where |
The optional parameters are:
/ServiceKey
|
The Service key to be used for the diagnostic tool. By default, it uses the current ServiceKey configured globally for the host. It does not pick up any Service keys defined at the <iisSites> , <applicationPools> , or <applications> levels. |
|
/Verbose
|
Enables printing to console of verbose level log messages from the diagnostic run. | |
/Timeout
|
Timeout for testing the connection in milliseconds. If the connection doesn’t succeed before the timeout is reached the connection check will timeout. |
For example, in Windows (where %PROGRAMFILES%
is in the standard C:\Program Files\
directory) to verify connection to the my-service
service using a specific Service key and a timeout of 5000ms (5 seconds):
C:\Program Files\SolarWinds\APM\dotnet> dotnet_diagnostic_tool.exe /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service /Timeout 5000
Look for the message Has connection succeeded? YES
, which indicates that the connection succeeded.
If there are any errors, review the log files %PROGRAMFILES%\SolarWinds\APM\dotnet\dotnet_diagnostic_tool.log
and C:\Windows\Temp\SolarWinds\DotNetAgentLog_dotnet_diagnostic_tool*
.
How do I run the diagnostic tool in Linux?
A connectivity diagnostic tool is available with the .NET Library. To run it in Linux, in a shell session run the executable <dotnet-runtime>/dotnet_diagnostic_tool_core
located under the library installation directory, where <dotnet-runtime>
is a supported .NET Runtime installed on the machine (for example, net6.0
for .NET 6).
The following example assumes you have already set the library installation directory environment variable SW_APM_HOME_DOTNET
:
cd ${SW_APM_HOME_DOTNET}/<dotnet-runtime>
./dotnet_diagnostic_tool_core
The optional parameters are:
/ServiceKey
|
The Service key to be used for the diagnostic tool. By default, it uses the current ServiceKey configured globally for the host. | |
/Verbose
|
Enables printing to the console of verbose level log messages from the diagnostic run. | |
/Timeout
|
Timeout for testing the connection in milliseconds. If the connection doesn’t succeed before the timeout is reached the connection check will timeout. |
For example, to verify connection to the my-service
service using a specific Service key and a timeout of 5000ms (5 seconds):
dotnet_diagnostic_tool_core/dotnet_diagnostic_tool_core /ServiceKey 0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service /Timeout 5000
Look for the message Has connection succeeded? YES
, which indicates that the connection succeeded.
If there are any errors, review the log files /var/tmp/solarwinds/DotNetAgentLog_dotnet_diagnostic_tool_core*
.
What are the diagnostic tool's exit codes?
The diagnostic tool provides the following exit codes:
Exit status code | Name | Description |
---|---|---|
0
|
OK
|
Diagnostics successful. |
101
|
UNKNOWN_ERROR
|
Unexpected error. See the error message for details. |
102
|
INVALID_FORMAT_ARGUMENT
|
Argument passed to the diagnostic tool is not in a valid format. For example, the Service key is too short. See the error message for details. |
103
|
INVALID_ARGUMENTS
|
Argument passed to the diagnostic tool is incorrect. For example, it uses an argument name not recognized by the diagnostic tool. See the error message for details |
104
|
INVALID_SERVICE_KEY
|
The Service key is rejected by the SolarWinds APM collector. |
105
|
CONNECTION_FAILURE
|
Failed to connect to the SolarWinds APM collector within the timeout limit of 5 seconds. |
106
|
TRY_LATER
|
The SolarWinds APM collector rejected the diagnostic request with the status TRY_LATER . |
107
|
LIMIT_EXCEEDED
|
The SolarWinds APM collector rejected the diagnostic request with the status LIMIT_EXCEEDED . |
108
|
ERROR_INVALID_APM_COLLECTOR
|
The format of the ingestion endpoint is invalid. |
Script to gather diagnostic information to send to Support
If you need to contact Support, you can run a script to collect the required information. The script gathers all .NET Library log files, .NET Library registry entries, and IIS log files. It archives this information in a ZIP file. Send this ZIP file and the script log file to Support.
How do I run the script in Windows?
-
Start a PowerShell session as an administrator.
-
To gather diagnostic information, run the following commands:
cd "%PROGRAMFILES%\SolarWinds\APM\dotnet" .\agent-diagnostic-logs.ps1
To learn more about the options included with the script, run the following command:
.\agent-diagnostic-logs.ps1 -Help
How do I run the script on the Azure App Service?
If you are using SolarWindsAPM.Agent.AzureAppServices.Extension
(an Azure site extension), complete the following steps to run the script.
-
Open a web console command prompt.
-
Run the following commands:
cd d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds powershell -File "agent-diagnostic-logs.ps1"
The ZIP file and script log file are created in the following location:
d:\home\SiteExtensions\SolarWindsAPM.Agent.AzureAppServices.Extension\SolarWinds
How do I run the script in a Linux-based OS?
If the package solarwinds-apm-dotnet-package.tgz
was extracted to a location other than ${HOME}/solarwinds
, change the path in the following commands.
-
Open a shell.
-
Run the following commands:
cd ${HOME}/solarwinds ./agent-diagnostic-logs.sh
To learn more about the options included with the script, run the following command:
./agent-diagnostic-logs.sh --help
.NET Library (Windows, Linux, and SDK)
The following are troubleshooting steps for various .NET Library questions. See .NET Core SDK for additional SDK-specific questions.
Are requests being sent to the application?
In order for traces and metric data to be sent to the Traces Explorer and Metrics Explorer, your application must be receiving requests.
- For an IIS Hosted application, run the
iisreset
command again and ensure that the application is receiving requests. - For a Non-IIS application, restart the application and ensure that it is receiving requests.
Is the SolarWinds Observability .NET Library or SDK able to connect to the SolarWinds APM collector?
- Look for indications of connection issues in the logs. See Where can library logs be found?
- If your server is behind a firewall, you might not be able to connect to the SolarWinds APM collector. See Firewall or access control requirements.
- Run the diagnostic tool to ensure that your application is able to connect to the SolarWinds APM collector.
- Run the connection script to ensure that your application is able to connect to the SolarWinds APM collector.
Where can library logs be found?
For information about the location of the SolarWinds Observability .NET Library log files, see:
- Windows Log file locations
- Linux Log file location
You can also run a script to gather diagnostic information, including all .NET Library log files, .NET Library registry entries, and IIS log files.
Do you have another profiler installed?
If you have the AppOptics .NET agent installed, it must be uninstalled before installing the SolarWinds Observability APM Library. See Migrate the .NET Library from AppOptics to SolarWinds Observability for more information on upgrading from the AppOptics .NET agent to the SolarWinds Observability APM Library.
Only one APM library can be used at a time. An APM library using the CLR profiler interface can
be identified by its CLSID configured in the COR_PROFILER
environment variable:
-
Search the application server environment variables and the registry (
regedit.exe
) forCOR_PROFILER
. There should be only one unique CLSID specified for theCOR_PROFILER
environment variable.The
COR_PROFILER
environment variable might be in multiple locations. - The SolarWinds Observability .NET Library CLSID is
COR_PROFILER={AAAA7777-11FE-4F94-8EDA-312C0EDF4141}
.
Did the SolarWinds Observability Profiler load successfully?
-
Verify that the SolarWinds Observability Profiler component of the .NET Library performed a test to see if it should be loaded. Open the Event Viewer and look under Windows Logs > Application for events from source
solarwinds_apm_profiler
and the following message:Message: .NET Runtime checking if SolarWinds Observability Profiler should be loaded. Profiler CLSID: '{AAAA7777-11FE-4F94-8EDA-312C0EDF4141}'.
-
Verify that the SolarWinds Observability Profiler loaded successfully. Look for another event immediately after the one above, from source
.NET Runtime
with the following message:.NET Runtime version <version number> - The profiler was loaded successfully. Profiler CLSID: '{AAAA7777-11FE-4F94-8EDA-312C0EDF4141}'. Process ID (decimal): <process id>. Message ID: [0x2507].
- Check for any error events reported under Windows > Application from the source of either
.NET Runtime
orsolarwinds_apm_profiler
.
If the SolarWinds Observability Profiler was not loaded successfully, set the log level for the Event Viewer to
a higher level with the library configuration property EventViewerLogLevel
.
Additional information will be logged to the Event Viewer by the SolarWinds Observability Profiler.
Has the library been initialized?
Look at the response headers returned. The library
adds the header X-Trace
to the response. If the response doesn’t contain the header X-Trace
, then
the library probably wasn’t loaded correctly.
Review the other troubleshooting options to resolve the issue.
Are requests being sampled for tracing?
Requests might not be sampled due to a low sample rate or connection issues. The X-Trace
response header or the SDK Method Trace.GetCurrentTraceId()
can be used to obtain the current trace context:
-
The trace context ends in
00
if a request isn’t traced. It ends in01
if the request is being traced. -
If tracing is enabled (the default configuration), there should be some requests with a trace context ending in
01
. If all returned traced contexts end in00
, the server might not be able to connect to the SolarWinds APM collector.
Review the .NET Library logs to see if there are any connection issues logged.
Are warnings being reported by the library?
Review the .NET Library logs. Sometimes configuration errors such as invalid formatting in the solarwinds_apm.config
config file can lead to instrumentation problems.
How do I get more information in the logs?
Increase the logging level verbosity to Debug
to get more detailed information added to the library logs. Use the
library configuration property LogLevel
to change the logging level.
Run the iisreset
command or restart the .NET Core application to have the new level take effect.
Is the Service key configured correctly?
The Service key is set the following ways, depending on the configuration:
-
By adding it to the
solarwinds_apm.config
file in the%PROGRAMFILES%
directory (typicallyC:\Program Files\
):%PROGRAMFILES%\SolarWinds\APM\dotnet
-
By adding it to the
solarwinds_apm.config
file included in theSolarWindsAPM.Agent
andSolarWindsAPM.Agent.AzureAppServices.Extension
NuGet packages -
By using the environment variable
SW_APM_SERVICE_KEY
The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when you add the new service.
It YourApiToken:YourServiceName
. Replace YourApiToken
with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName
with your chosen name for this service. See API Tokens.
The service name is also called the entity's service ID in SolarWinds Observability. The service name can also be seen in the Overview tab in the service entity's Entity Explorer details view. Editing the display name of the service entity in SolarWinds Observability does not affect the Service key.
Did an IIS site that was previously reporting data stop reporting data?
Changes made to a server after the .NET Library is installed can affect the .NET Library. These changes include updates to the IIS configuration or registry settings.
If your IIS site uses a classic mode application pool, see if changes were made to the IIS configuration. See Is the SolarWindsAPMInstrumentation HttpModule being loaded?
To determine if changes were made to the registry that affect the .NET Library, search for COR_ENABLE_PROFILING=1
in the registry. This value should be part of the data for the Environment
value in the WSSVC
key. To find the registry key:
- Open the Registry Editor (enter
regedit.exe
in a command prompt). - From the menu, select Edit > Find.
- Enter the string to search (
COR_ENABLE_PROFILING=1
) and ensure Data is selected. - Click Find Next.
If COR_ENABLE_PROFILING=1
cannot be found in the registry, the .NET Library will not instrument IIS sites. Reinstall the .NET Library to re-enable instrumentation
for IIS sites and disable the process that is removing the registry entries.
If another process is removing your registry entries and that process cannot be disabled, reinstall the .NET Library and select Instrument both IIS and NON-IIS .NET applications.
Is the SolarWindsAPMInstrumentation HttpModule being loaded?
The SolarWindsAPMHttpModule is required only for IIS sites using a classic mode application pool.
Is the web.config
clearing all HttpModules for the site?
The installer for the SolarWinds Observability .NET Library installs the SolarWindsAPMInstrumentation module in IIS. The SolarWindsAPMInstrumentation
HttpModule is required for the instrumentation of IIS sites using a classic mode application pool. The application configuration for an IIS website could be
configured to clear all HttpModules
for a site and enable only specific HttpModules
. If this is the case, the instrumentation
of the IIS site will not work because the SolarWindsAPMInstrumentation HttpModule would not be loaded.
The following is an example of a configuration in the web.config
file that clears all HttpModules
:
<system.web>
<httpModules>
<clear/>
</httpModules>
</system.web>
To resolve this, update the configuration to add the SolarWindsAPMInstrumentation
HttpModule. The version of the module installed by the SolarWinds Observability .NET Library installer must match the configuration added
to the web.config
file.
The following is an example of a configuration for adding the SolarWindsAPMInstrumentation HttpModule:
<system.web>
<httpModules>
<add name="SolarWindsAPMInstrumentation" type="SolarWinds.Apm.Agent.HttpInstrumentationModule, SolarWinds.Apm.Agent, Version=5.0.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
</httpModules>
</system.web>
Is a web deployment process resetting IIS config files?
The SolarWindsAPMHttpModule is required only for IIS sites using a classic mode application pool.
The .NET Library installs the SolarWindsAPMHttpModule by adding the module to the config files used by IIS. A web deployment process might reset the IIS config files, thus disabling the SolarWindsAPMHttpModule. To determine if the SolarWindsAPMHttpModule has been removed, identify the .NET CLR version of the application pool and determine if it is running in 32-bit or 64-bit mode:
-
Right-click the application pool name and select Advanced Settings.
-
Look at these settings: .NET CLR Version - either
v2.0
orv4.0
and Enable 32-bit Applications - eitherTrue
orFalse
-
Location of config file:
- 32-bit - .NET CLR v2.0 app:
C:\Windows\Microsoft.NET\Framework\v2.0.<build number of version installed>\CONFIG\web.config
- 64-bit - .NET CLR v2.0 app:
C:\Windows\Microsoft.NET\Framework64\v2.0.<build number of version installed>\CONFIG\web.config
- 32-bit - .NET CLR v4.0 app:
C:\Windows\Microsoft.NET\Framework\v4.0.<build number of version installed>\CONFIG\web.config
- 64-bit - .NET CLR v4.0 app:
C:\Windows\Microsoft.NET\Framework64\v4.0.<build number of version installed>\CONFIG\web.config
- 32-bit - .NET CLR v2.0 app:
In the web.config
file, the element: system.web/httpModules
should include an element for the SolarWindsAPMHttpModule:
<system.web>
...
<httpModules>
....
<add name="SolarWindsAPMInstrumentation" type="SolarWindsAPM.Agent.HttpInstrumentationModule, SolarWindsAPM.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
</httpModules>
</system.web>
If the element is not included, uninstall the library and install the library again.
How can you configure a Service key for an application pool?
Information on how to configure a different Service key for an application pool can be found here.
Are metrics not being displayed for a non-IIS application?
For .NET Framework applications, ensure the Instrument NON-IIS .NET Framework Applications
component was selected during the library installation. For .NET 6+ applications, ensure the Instrument NON-IIS .NET Applications
component was selected during the library installation.
For non-IIS applications, additional configuration needs to be added to the solarwinds_apm.config
file in the %PROGRAMFILES%
directory (typically C:\Program Files\
) located in: %PROGRAMFILES%\SolarWinds\APM\dotnet
. See Installation for stand-alone, non-IIS applications for the configuration that needs to be added to the solarwinds_apm.config
file.
The supported auto-instrumented non-IIS applications include standalone WCF services. For other types of non-IIS applications, use the Instrumentation SDK.
Are custom instrumentation spans or other spans missing from traces?
If a method is trivial enough, the CLR might decide to inline the method, which is optimization that replaces a method call with the method body. If that happens, the method doesn't exist anymore and SolarWinds Observability can't instrument it. To prevent this, inlining can be disabled either for an individual method, as shown below, or for all apps via the Inlining parameter.
[MethodImpl(MethodImplOptions.NoInlining)]
void YourMethod()
{
// do something
}
Is the library config file valid?
You can run the diagnostic tool, which checks the configuration XML validity. An example of the diagnostic output when the XML is invalid due to missing comment terminator:
C:\Program Files\SolarWinds\APM\dotnet>dotnet_diagnostic_tool.exe
...
Trace: OboeInit.LoadSettings(): Invoked ('C:\Program Files\SolarWinds\APM\dotnet\solarwinds_apm.config').
Error: OboeInit.LoadSettings(): An error has occured while reading and/or parsing the instrumentation settings. An XML comment cannot contain '--', and '-' cannot be the last character. Line 27, position 5.
Are your .NET 6+ IIS websites missing expected spans or code profiles and is the service entry span IIS?
For .NET 6+ IIS websites, the service entry span should be aspnet-core
. If this is not the case, check the application pool setting: .NET CLR Version
for the .NET 6+ IIS website. If the .NET CLR Version
is set to v2.0
or v4.0
rather than No Managed Code
, this will cause the .NET Framework instrumentation to check if it should be loaded for the IIS website.
The .NET Framework instrumentation can be disabled for an application pool by using the iisInstrumentation
attribute of the applicationPool
element in the solarwinds_apm.config
file. See Configure the .NET Library for more information.
Does your .NET Framework application get a verification exception after you install the SWO .NET APM Library?
The automatic instrumentation of .NET Framework applications may cause a System.Security.VerificationException
because the instrumentation may impact the verification done by the JIT compiler at runtime. The message for the exception may be Operation could destabilize the runtime
. The verification done by the JIT compiler that causes the exception can be disabled using the SecurityTransparencyInstrumentation
configuration setting. See Configure the .NET Library to learn how to set this setting.
.NET Core SDK
The following are additional troubleshooting steps for .NET SDK-only questions.
The troubleshooting questions below assume the SolarWindsAPM.Agent NuGet package has been added to your application, and either the SolarWinds Observability middleware was added to your ASP.NET Core application, SDK methods were added to your application, or both the middleware and SDK methods were added to your application.
Has the library been initialized?
If the SolarWinds Observability middleware is being used, you can determine if the library has been initialized by looking
at the response headers returned. The library adds the header X-Trace
to the response. If the response
doesn’t contain the header X-Trace
, then the library probably wasn’t loaded correctly.
Review the other troubleshooting options to try to resolve the issue.
Are requests being sampled for tracing?
Requests might not be sampled due to a low sample rate or connection issues. Review the NET library logs to see if there are any connection issues logged.
To troubleshoot connection issues, look at trace context value. The SolarWinds Observability middleware adds the X-Trace
header to the response, and the SDK Method Trace.GetCurrentTraceId()
returns the trace context. The trace context ends in 00
if a request isn’t traced. It ends
in 01
if the request is being traced.
If tracing is enabled (the default configuration), there should be some trace contexts that end in 01
. If all trace contexts end in 00
, this might indicate that the server
is not able to connect to the SolarWinds APM collector.
How to increase log verbosity?
Increase the logging level verbosity to Debug
to get more detailed information added to the library logs. Use the
library configuration property LogLevel
to change the logging level.
Restart the .NET Core application to have the new level take effect.
Is the Service key configured correctly?
The Service key is set in one of the following ways, depending on the configuration:
-
The
solarwinds_apm.config
file included in theSolarWindsAPM.Agent
andSolarWindsAPM.Agent.AzureAppServices.Extension
NuGet packages. -
The environment variable
SW_APM_SERVICE_KEY
.
The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when you add the new service.
It YourApiToken:YourServiceName
. Replace YourApiToken
with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName
with your chosen name for this service. See API Tokens.
The service name is also called the entity's service ID in SolarWinds Observability. The service name can also be seen in the Overview tab in the service entity's Entity Explorer details view. Editing the display name of the service entity in SolarWinds Observability does not affect the Service key.
Does the .NET core site need to support characters such as &, *, <, >, : in the path?
If the .NET core site requires characters such as &, *, <, >, :
in the path and the .NET Library is installed on the same server, this might cause the .NET core site to return a 400
response code. The .NET Library installs
an HttpModule
which uses ASP.NET request validation for sites running in managed code application pools.
By default, the ASP.NET request validation rejects certain characters in the path and returns a 400
response code.
One of the following options can be used to resolve this issue:
-
Option 1: In IIS Manager, open the Edit Application Pool window and set the .NET CLR version to
No Managed Code
. For detailed instructions, see Step 6 in Advanced configuration of the ASP.NET Core Module and IIS: Install Web Deploy when publishing with Visual Studio from the Microsoft ASP.NET Core documentation. -
Option 2: If only .NET 6+ sites need to be instrumented on the server, select Instrument .NET Core and .NET 5.0+ applications during the .NET Library installation.
-
Option 3: Add the following to the .NET core site's
web.config
to stop the validation of the characters in the path:<system.web> <httpRuntime requestPathInvalidCharacters="" /> </system.web>
Can the Linux SDK find and load the native shared library?
On Linux platforms, the .NET Core Runtime looks for the oboe_<major>_<minor>_<patch>.so
shared library under the <published-dotnet-core-app>/runtimes/linux-x64/native/
directory, where <published-dotnet-core-app>
is the location of the app. This is the default
behavior for .NET Core Runtime when searching for native shared libraries.
The directory structure should be:
<published-dotnet-core-app>
|--- <dotnet-core-app>
|--- solarwinds_apm.config
|--- AppOptics.Instrumentation.dll
|--- AppOptics.Instrumentation.Internal.dll
|--- runtimes
|--- linux-x64
|---native
|---libe_sqlite3.so
|---libuv.so
|---oboe_<major>_<minor>_<patch>.so
...
If the oboe_<major>_<minor>_<patch>.so
shared library is moved to a different location or the directory structure doesn’t follow .NET conventions for native shared libraries, then the LD_LIBRARY_PATH
environment variable can be used to update the search library path, as described in the LD.SO(8) page in the Linux Programmer's Manual.
export LD_LIBRARY_PATH=<path-to-oboe>/oboe_<major>_<minor>_<patch>.so
You can identify loading issues for the shared library by:
-
Using the strace tool, as described in the Linux man pages site. Execute
strace dotnet
and look forlstat
,stat
, andopen
functions in the output to see which paths are searched for the shared libraries.strace dotnet <dotnet-core-app> > strace.output.log 2>&1
-
Using the
LD_DEBUG
environment variable. Set it tolibs
for the dotnet process to capture the search path for shared libraries, as described in the LD.SO(8) page.LD_DEBUG=libs dotnet <dotnet-core-app> > lddebug.output.log 2>&1
What if installer does not proceed due to unmet requirements?
Starting with version 5.3.2, the installer will stop if it detects some of the requirements to successfully install and enable the .NET Library are not met. These include if IIS is not stopped, or another CLR profiler is enabled.
If you must deploy the library despite not meeting these requirements, the Command Line Install can be used with the /VERYSILENT
, /NOCLOSEAPPLICATIONS
, and /NOCONNECTIVITYCHECK
options to install the library. In this case, a host restart is typically needed for the instrumentation to be enabled.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.