Troubleshoot the Python Library

Have you checked the log for errors?

By default, the logs are sent to stderr. Common errors in the logs are that the Service key is undefined (the environment variable SW_APM_SERVICE_KEY is missing) and the Python Library was not installed with the native library bindings (which should happen automatically on most platforms).

Is your application is instrumented?

To verify whether your application is correctly instrumented, look for X-Trace headers. For example, use curl to view the X-Trace header.

curl -i http://the-url-of-your-website
HTTP/1.1 200 OK
...
X-Trace: 00-B5CEC860E76D82425C8048BBF5469BE9BC580A62-8B4A5611C2CA3A56-01

Is the instrumentation installed and enabled?

You will find the HTTP header X-Trace from the HTTP response if the instrumentation is installed and enabled.

Is the request traced?

The last two characters of X-Trace header indicates whether the request is traced. It is traced if the last two characters are 01, and it is not traced if the last two characters are 00. Not all requests are expected to be traced. In high-traffic environments, automatic sampling is used.

Are requests traced but not appearing in SolarWinds Observability SaaS?

If the last two characters are 01 but traces cannot be seen in the Traces Explorer, verify the Service key configuration.

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when you add the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

The service name is also called the entity's service ID in SolarWinds Observability SaaS. The service name can also be seen in the Overview tab in the service entity's Entity Explorer details view. Editing the display name of the service entity in SolarWinds Observability SaaS does not affect the Service key.

Have you enabled verbose logging?

Enable verbose logging by assigning the SW_APM_DEBUG_LEVEL environment variable to 6.

export SW_APM_DEBUG_LEVEL=6

See logging level for more information.

Have you configured logging to file?

If the SW_APM_LOG_FILEPATH environment variable has been set, logs will be written to the file at that path instead of to stderr.

See logging output filepath for more information.

Is the library connected to the SolarWinds APM collector?

Review the Python Library log messages to see if any connection issues are logged.

If your server is behind a firewall, you may not be able to connect to the SolarWinds APM collector. See Firewall or access control requirements.

Is the Service key configured correctly?

The Service key is set by defining the system environment variable SW_APM_SERVICE_KEY.

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when you add the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

The service name is also called the entity's service ID in SolarWinds Observability SaaS. The service name can also be seen in the Overview tab in the service entity's Entity Explorer details view. Editing the display name of the service entity in SolarWinds Observability SaaS does not affect the Service key.

Are there duplicate spans in traces?

A trace in SolarWinds Observability SaaS may contain duplicate spans due to existing dependencies between Python libraries and each dependent library having OpenTelemetry instrumentation libraries available. For example, an RPC request can involve the requests and urllib3 libraries, which are both instrumented by default. Similarly, a database call can involve the sqlalchemy and asyncpg libraries, which are both instrumented by default.

If you decide that you only want one dependent library to record spans, you can set the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable as described above. To instrument only requests, set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=urllib3.

Do you have more than one APM extension installed?

Do not run other APM libraries alongside the SolarWinds Observability Python Library. Remove other APM libraries from your Python application before using the SolarWinds Observability Python Library.

To remove the AppOptics Agent from your application, see Migrate the Python Library from AppOptics to SolarWinds Observability SaaS.

Are there package version conflicts at application package installation?

You could have a package version conflict if you installed additional OpenTelemetry-based libraries to customize the behavior (see Customize OpenTelemetry propagators and Customize OpenTelemetry traces exporters) of the installed Python Library solarwinds-apm. If the version number requirements are too strict, there may be a pkg_resources.VersionConflict error.

Consider using approximate versioning. For example, in the package configuration for my_custom_propagator, use opentelemetry-api ~= 1.11.1 instead of opentelemetry-api == 1.11.1.

Are spans for database queries showing entire statements?

OpenTelemetry instrumentation libraries export database query spans with the attribute db.statement. There is a security risk if a query is prepared using a hard-coded expression or a plain, string-interpolated expression. If the query is prepared this way, db.statement contains all values, including entity IDs.

To avoid this risk, build queries using a database interface library such as dbapi, asyncpg, pymongo, or sqlalchemy. These libraries make queries with parameterized expressions and use %s for parameters in db.statement.

Is there an instrumentation library-specific issue?

psycopg2

OpenTelemetry instrumentation of psycopg2 will not work if only the binary is installed ( psycopg2-binary). Install the full version of psycopg2 with the command pip install psycopg2.

sqlite3

Previous versions of OpenTelemetry instrumentation sqlite3 break application-database connections and display the error sqlite3.ProgrammingError: Base Connection.__init__ not called. This issue was fixed by the OpenTelemetry community and included in Python Library version 0.8.0. See Upgrade the Python Library

If you cannot upgrade to Python Library 0.8.0, either disable sqlite3 instrumentation with the OTEL_PYTHON DISABLED_INSTRUMENTATIONS environment variable, or use a different framework (PostgreSQL, MariaDB, or MySQL).

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.