Documentation forSolarWinds Observability

Troubleshooting the Python Library

Check the logs

By default the logs are sent to stderr. Common errors in the logs include if the Service key is undefined (the environment variable SW_APM_SERVICE_KEY is missing) and if the Python Library was not installed with the native library bindings built (which should happen automatically on most platforms).

Check if your application is instrumented

To verify whether your application is correctly instrumented, look for X-Trace headers. For example, use curl to view the X-Trace header.

curl -i http://the-url-of-your-website
HTTP/1.1 200 OK
...
X-Trace: 00-B5CEC860E76D82425C8048BBF5469BE9BC580A62-8B4A5611C2CA3A56-01

Is the instrumentation installed and enabled

You will find the HTTP header X-Trace from the HTTP response if the instrumentation is installed and enabled.

Is the request traced

The last two characters of X-Trace header indicates whether the request is traced: it is traced if the last two characters are 01, and it is not traced if the last two characters are 00. Not all requests are expected to be traced; in high-traffic environments, automatic sampling is used.

Are requests traced but not appearing in SolarWinds Observability

If the last two characters are 01 but traces cannot be seen in the Traces Explorer, verify the Service key configuration.

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when adding the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

Enabling verbose logging

Enable verbose logging by define the SW_APM_DEBUG_LEVEL environment variable to 6.

Do not enable verbose logging in a production environment, it will flood your system with verbose logs.

export SW_APM_DEBUG_LEVEL=6

See logging level for more information.

Is the library connected to the SolarWinds APM collector?

Review the Python Library log messages to see if there are any connection issues logged.

If your server is behind a firewall, you may not be able to connect to the SolarWinds APM collector. See Firewall or access control requirements.

Is the Service key configured correctly?

The Service key is set by defining the system environment variable SW_APM_SERVICE_KEY.

The Service key is used to identify your account and the service being instrumented. It is shown in the Add Data dialog when adding the new service.

It is a required configuration and should be in the form of YourApiToken:YourServiceName. Replace YourApiToken with the SolarWinds Observability API token (ingestion type) generated for this service, and replace YourServiceName with your chosen name for this service. See API Tokens.

Are there duplicate spans in traces?

A trace in SolarWinds Observability may contain duplicate spans due to existing dependencies between Python libraries and each dependent library having OpenTelemetry instrumentation libraries available. For example, both Python requests and urllib3 libraries can be instrumented, while requests a dependency on urllib3. Similarly, sqlalchemy and asyncpg are both instrumented.

If you decide that you only want one dependent library to record spans, you can set the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable as described above. To only instrument requests, set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=urllib3.

Are there package version conflicts at application package installation?

You could have a package version conflict if you installed additional OpenTelemetry-based libraries to customize the behavior (see Customize OpenTelemetry Propagators and Customize OpenTelemetry Exporters) of the installed Python Library solarwinds-apm. If the version number requirements are too strict, there may be a pkg_resources.VersionConflict error.

Consider using approximate versioning. For example, in the package configuration for my_custom_propagator, use opentelemetry-api ~= 1.11.1 instead of opentelemetry-api == 1.11.1.

Are spans for database queries showing entire statements?

OpenTelemetry instrumentation libraries will export database query spans with the attribute db.statement. There is a security risk if a query is prepared using a hard-coded expression or a plain, string-interpolated expression. If the query is prepared this way, db.statement contains all values, including entity IDs.

To avoid this risk, build queries using a database interface library such as dbapi, asyncpg, pymongo, or sqlalchemy. These libraries make queries with parameterized expressions and will use %s for parameters in db.statement.

Instrumentation library-specific issues

psycopg2

OpenTelemetry instrumentation of psycopg2 will not work if only the binary is installed ( psycopg2-binary). Install the full version of psycopg2 with the command pip install psycopg2.

sqlite3

OpenTelemetry instrumentation sqlite3 currently breaks application-database connections and displays the error sqlite3.ProgrammingError: Base Connection.__init__ not called. This is a known issue to the OpenTelemetry community and is in the process of being fixed. For now, either disable sqlite3 instrumentation with the OTEL_PYTHON DISABLED_INSTRUMENTATIONS environment variable, or use a different framework (PostgreSQL, MariaDB, or MySQL).

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.