Documentation forAppOptics

Troubleshooting

If you are not receiving traces and metrics in your AppOptics dashboard these are some options to troubleshoot.

.NET Framework Agent

Are requests being sent to the application?

  • Run the iisreset command again and ensure that your application is receiving requests. In order for traces and metric data to be sent to your AppOptics dashboard your application must be receiving requests.

Is the AppOptics .NET Agent able to connect to the AppOptics server?

Where can agent logs be found?

Do you have another profiler installed?

  • Only one APM agent can be used at a time. An APM agent using the CLR profiler interface can be identified by it's CLSID configured in the COR_PROFILER environment variable.
  • Search the application server environment variables and the registry (regedit.exe) for COR_PROFILER. There should only be one unique CLSID specified for the COR_PROFILER environment variable (note: the COR_PROFILER environment variable may be in multiple locations).
  • The AppOptics .NET Agent CLSID is COR_PROFILER={824293AD-22E2-4DAA-BC28-166C140543BE}.

Was the AppOptics Profiler loaded successfully?

  • To verify that the AppOptics Profiler component of the .NET Agent checked to see if it should be loaded, open Event Viewer and check under Windows Logs > Application for events from source AppOpticsProfiler and the following message:

    Copy
    Message: .NET Runtime checking if AppOptics Profiler should be loaded. Profiler CLSID: '{824293AD-22E2-4DAA-BC28-166C140543BE}'.

    If the AppOptics Profiler was loaded successfully there should be another event immediately after the one above, from source .NET Runtime and the following message:

    Copy
    .NET Runtime version <version number> - The profiler was loaded successfully. Profiler CLSID: '{824293AD-22E2-4DAA-BC28-166C140543BE}'. Process ID (decimal): <process id>. Message ID: [0x2507].
  • Check for any error events reported under Windows > Application from source of either .NET Runtime or AppOpticsProfiler.
  • If the AppOptics Profiler was not loaded successfully, the log level for the Event Viewer can be set to a higher level with the agent configuration property EventViewerLogLevel. Additional information will be logged to the Event Viewer by the AppOptics Profiler.

Has the agent been initialized?

  • One way to check if the agent has been initialized, is to look at the response headers returned. The agent will add the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the agent probably wasn’t loaded correctly.
  • Review the other troubleshooting options to try to resolve the issue.

Are requests being sampled for tracing?

  • Requests may not be sampled due to a low sample rate or connection issues. The X-Trace response header ends in 00 if a request isn’t traced and ends in 01 if the request is being traced.
  • Some requests should have a X-Trace header returned that ends in 01. If all X-Trace headers being returned end in 00 the server may not be able to connect to the AppOptics collector.
  • Review the AppOptic .NET Agent logs to see if there are any connection issues logged.

Are there warnings reported by the agent?

  • Review the agent logs.
  • Sometimes configuration errors such as invalid formatting in the config file can lead to instrumentation problems.

How to increase log verbosity?

  • Increase the logging level verbosity to Debug to get more detailed information added to the agent logs. Use the agent configuration property LogLevel to change the logging level.
  • Run the iisreset command to have the new level take effect.

Is the service key configured correctly?

  • The service key is added to the agent.config file located in the directory Program Files\AppOptics\APM\dotnet. The first part of the service key (before the colon :) should match one of the AppOptics API tokens listed in the AppOptics dashboard under Organization Settings > API Tokens.
  • The format of the API token recently changed. The new token format which is 71 characters long, may not work with previous versions of the agent (before v3.7.2). Try updating to the latest version of the agent which supports the new format.

Was an IIS site previously reporting data and has now stopped reporting data?

  • Changes made to a server after the .NET agent is installed may affect the .NET agent. These changes include updates to IIS configuration or registry settings.

  • For checking if changes have been made to IIS configuration please see the Is the AppOpticsInstrumentation HttpModule being loaded? question below. This question is only applicable to IIS sites using a classic mode application pool.

  • To determine if changes were made to the registry that affect the .NET agent, search for APP_TYPE=TYPE_IIS in the registry. This value should be part of the data for the Environment value in the WSSVC key.

  • Instructions for finding a registry key:

    1. Open the Registry Editor (open a command prompt, type in regedit.exe and press Enter)
    2. From the menu, select Edit → Find.
    3. Enter the string to search (APP_TYPE=TYPE_IIS) and ensure Data is selected.
    4. Click the Find Next button.

    Alternately, use the REG QUERY command in the command prompt, below is an example of the query and expected Environment value in the search results:

    Copy
    reg query hklm\system\ /s /f app_type=type_iis
    Environment  REG_MULTI_SZ  COR_ENABLE_PROFILING=1\0APP_TYPE=TYPE_IIS\0COR_PROFILER={824293AD-22E2-4DAA-BC28-166C140543BE}\0APPOPTICS_HOME_APM_DOTNET=C:\Program Files\AppOptics\APM\dotnet\
  • If APP_TYPE=TYPE_IIS cannot be found in the registry the .NET agent will not instrument IIS sites. Please reinstall the .NET agent to re-enable instrumentation for IIS sites and disable the process that is removing the registry entries.

  • If the process that is removing the registry entries cannot be disabled, follow these steps:

    1. Reinstall the .NET agent and select to Instrument both IIS and NON-IIS .NET applications.
    2. Add the following system environment variable: APP_TYPE=TYPE_IIS.

Is the AppOpticsInstrumentation HttpModule being loaded?

Starting with agent version 3.10.0, the AppOpticsInstrumentation HttpModule is only required for IIS sites using a classic mode application pool.

  • The installer for the AppOptics .NET Agent installs the AppOpticsInstrumentation module in IIS. The AppOpticsInstrumentation HttpModule is required for the instrumentation of IIS sites using a classic mode application pool. The application configuration for an IIS website could be configured to clear all HttpModules for a site and only enable specific HttpModules. If this is the case, the instrumentation of the IIS site will not work as the AppOpticsInstrumentation HttpModule would not be loaded.

    Example configuration in the web.config file which clears all HttpModules:

    Copy
    <system.web>
      <httpModules>
        <clear/>
      </httpModules>
    </system.web>
  • If the IIS site being instrumented clears all HttpModules, the configuration will need to be updated to add the AppOpticsInstrumentation HttpModule. The version of the module installed by the AppOptics .NET agent installer must match the configuration added to the web.config file.

    Example configuration for adding the AppOpticsInstrumentation HttpModule:

    Copy
    <system.web>
      <httpModules>
        <add type= "AppOptics.Agent.HttpInstrumentationModule, AppOptics.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" name="AppOpticsInstrumentation" />
      </httpModules>
    </system.web>
  • The .NET agent installs the AppOpticsInstrumentation HttpModule by adding the module to the configuration files used by IIS. A web deployment process may reset the IIS configuration files, thus disabling the AppOpticsInstrumentation HttpModule. To determine if the AppOpticsInstrumentation HttpModule has been removed you need to know the .NET CLR version of the application pool and if it's running in 32bit or 64bit mode:

    1. Right-click on the application pool name and select Advanced Settings…
    2. Look at these settings: .NET CLR Version - either v2.0 or v4.0 and Enable 32-bit Applications - either True or False
    3. Location of config file:
      • 32 bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework\v2.0.<build number of version installed>\CONFIG\web.config
      • 64 bit - .NET CLR v2.0 app: C:\Windows\Microsoft.NET\Framework64\v2.0.<build number of version installed>\CONFIG\web.config
      • 32 bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework\v4.0.<build number of version installed>\CONFIG\web.config
      • 64 bit - .NET CLR v4.0 app: C:\Windows\Microsoft.NET\Framework64\v4.0.<build number of version installed>\CONFIG\web.config
    4. In the web.config file, the element: system.web/httpModules should include an element for the AppOpticsInstrumentation HttpModule:
    5. Copy
      <system.web>
        ...
        <httpModules>
          ....
          <add name="AppOpticsInstrumentation" type="AppOptics.Agent.HttpInstrumentationModule, AppOptics.Agent, Version=3.10.0.0, Culture=neutral, PublicKeyToken=9195cde59f6d12e5" />
        </httpModules>
      </system.web>

How can you configure a service key for an application pool?

  • Information on how to configure a different service key for an application pool can be found here.

Are metrics not being displayed for a non-IIS application?

  • Ensure the Instrument NON-IIS .NET Applications component was selected during the agent installation.
  • For non-IIS applications additional configuration needs to be added to the agent.config file located in the directory Program Files\AppOptics\APM\dotnet. The Instrumenting Non-IIS Applications section describes the configuration that needs to be added to the agent.config file.
  • The supported auto-instrumented non-IIS applications include standalone WCF services. For other types of non-IIS applications the custom instrumentation SDK would need to be used.

Are custom instrumentation spans or other spans missing from traces?

  • If a method is trivial enough, the CLR might decide to inline the method, which is optimization that replaces a method call with the method body. If that happens, the method doesn't exist anymore and so AppOptics can't instrument it.

  • To prevent this, inlining can be disabled either for an individual method, as shown below, or for all apps via the inlining config parameter.

    Copy
    [MethodImpl(MethodImplOptions.NoInlining)]
    void YourMethod()
    { 
      // do something 
    }

Does the .NET Agent need to be upgraded?

  • Release notes for the .NET agent can be found here.
  • The current installed version of the .NET agent can be found when viewing Program and Features in the Control Panel. The shortcut for opening the Program and Features window is appwiz.cpl. The name of the agent is AppOptics .NET Agent <version number>.
  • Upgrade instructions for the .NET agent can be found here.

.NET Core SDK

The troubleshooting questions below assume the AppOptics.Instrumentation nuget package has been added to your application. As well, either the AppOptics middleware has been added to your ASP.NET Core application or SDK methods have been added to your application or both.

Are requests being sent to the application?

  • Restart your .NET Core application and ensure that your application is receiving requests. In order for traces and metric data to be sent to your AppOptics dashboard your application must be receiving requests.
  • For ASP.NET Core applications, in order to have the automatic starting of traces and metric reporting your application needs to add the AppOptics middleware.

Is the AppOptics .NET Core SDK able to connect to the AppOptics server?

Where can agent logs be found?

Has the agent been initialized?

  • If the AppOptics middleware is being used, one way to check if the agent has been initialized, is to look at the response headers returned. The agent will add the header X-Trace to the response. If the response doesn’t contain the header X-Trace, then the agent probably wasn’t loaded correctly.
  • Review the other troubleshooting options to try to resolve the issue.

Are requests being sampled for tracing?

  • Requests may not be sampled due to a low sample rate or connection issues.
  • Review the AppOptic .NET Agent logs to see if there are any connection issues logged.
  • The AppOptics middleware will add the X-Trace header to the response or the SDK Method Trace.GetCurrentTraceId() can be used to obtain the current X-Trace. The X-Trace ends in 00 if a request isn’t traced and ends in 01 if the request is being traced.
  • Some requests should have an X-Trace that ends in 01. If all X-Trace’s end in 00 the server may not be able to connect to the AppOptics collector.

How to increase log verbosity?

  • Increase the logging level verbosity to Debug to get more detailed information added to the agent logs. Use the agent configuration property LogLevel to change the logging level.
  • Restart the .NET Core application to have the new level take effect.

Is the service key configured correctly?

  • The service key is added to the agent.config file. The agent.config file is included in the AppOptics.Instrumentation nuget package. Alternatively, the service key can be set using the environment variable APPOPTICS_SERVICE_KEY.
  • The first part of the service key (before the colon :) should match one of the AppOptic API tokens listed in the AppOptics dashboard under Organization Settings > API Tokens.
  • The format of the API token recently changed. The new token format which is 71 characters long, may not work with previous versions of the agent (before v3.7.2). Try updating to the latest version of the agent which supports the new format.

Does the .NET core site need to support characters such as &, *, <, >, : in the path?

  • If the .NET core site requires characters such as as &, *, <, >, : in the path and the .NET Framework agent is also installed on the server this may cause the .NET core site to return a 400 response code. The .NET Framework agent installs an HttpModule which causes sites running in managed code application pools to go through the ASP.NET request validation. By default the ASP.NET request validation rejects certain characters in the path and a 400 response code is returned.

  • One of the following options can be used to resolve this issue:

    • Option 1: In IIS Manager, open the Edit Application Pool window and set the .NET CLR version to No Managed Code. See Step 6 in this link for more information.
    • Option 2: Uninstall the .NET Framework agent if only .NET core sites need to be instrumented on the server.
    • Option 3: Add the following to the .NET core site's web.config to stop the validation of the characters in the path:
    Copy
    <system.web>
      <httpRuntime requestPathInvalidCharacters="" />
    </system.web>

Can the Linux SDK find and load the native shared library?

  • On Linux platforms, the .NET Core Runtime looks for the oboe_<major>_<minor>_<patch>.so shared library under the <published-dotnet-core-app>/runtimes/linux-x64/native/ or <published-dotnet-core-app>/runtimes/linux-x86/native/ directory depending on if the application is 64 bit or 32 bit. This is the default behaviour for .NET Core Runtime when searching for native shared libraries.

    For a 64 bit application the directory structure should be:

    Copy
    <published-dotnet-core-app>
      |--- <dotnet-core-app>
      |--- agent.config
      |--- AppOptics.Instrumentation.dll
      |--- AppOptics.Instrumentation.Internal.dll
      |--- runtimes
           |--- linux-x64
                |---native
                |---libe_sqlite3.so
                |---libuv.so
                |---oboe_<major>_<minor>_<patch>.so
      ...

    For a 32 bit application the directory structure should be:

    Copy
    <published-dotnet-core-app>
      |--- <dotnet-core-app>
      |--- agent.config
      |--- AppOptics.Instrumentation.dll
      |--- AppOptics.Instrumentation.Internal.dll
      |--- runtimes
           |--- linux-x86
                |---native
                    |---libe_sqlite3.so
                    |---libuv.so
                    |---oboe_<major>_<minor>_<patch>.so
      ...

    If the oboe_<major>_<minor>_<patch>.so shared library is moved to a different location or the directory structure doesn’t follow .NET conventions for native shared libraries, then the LD_LIBRARY_PATH environment variable can be used to update the search library path, as described in the ld man page.

    Copy
    $ export LD_LIBRARY_PATH=<path-to-oboe_<major>_<minor>_<patch>.so>
  • You can identify loading issues for the shared library by:

    Using the strace tool. Execute strace dotnet and look for lstat, stat, and open functions in the output to see which paths are searched for the shared libraries.

    Copy
    $ strace dotnet <dotnet-core-app> > strace.output.log 2>&1

    Using the LD_DEBUG environment variable. Set it to libs for the dotnet process to capture the search path for shared libraries, as described in the ld man page.

    Copy
    $ LD_DEBUG=libs dotnet <dotnet-core-app> > lddebug.output.log 2>&1

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.