Troubleshoot the Python Library

Have you checked the log for errors?

By default, the logs are sent to stderr. Common errors in the logs are that the is undefined (the environment variable is missing) and the Python Library was not installed with the native library bindings (which should happen automatically on most platforms).

Is your application is instrumented?

To verify whether your application is correctly instrumented, look for X-Trace headers. For example, use curl to view the X-Trace header.

curl -i http://the-url-of-your-website
HTTP/1.1 200 OK
...
X-Trace: 00-B5CEC860E76D82425C8048BBF5469BE9BC580A62-8B4A5611C2CA3A56-01

Is the instrumentation installed and enabled?

You will find the HTTP header X-Trace from the HTTP response if the instrumentation is installed and enabled.

Is the request traced?

The last two characters of X-Trace header indicates whether the request is traced. It is traced if the last two characters are 01, and it is not traced if the last two characters are 00. Not all requests are expected to be traced. In high-traffic environments, automatic sampling is used.

Are requests traced but not appearing in ?

If the last two characters are 01 but traces cannot be seen in the , verify the configuration.

Is your application run using a pre-fork server?

Some Python web server frameworks pre-create multiple child processes/workers before handling client requests. Auto-instrumentation of these pre-fork servers can result in issues with sampling settings and metrics export, which rely on threads. Some examples are Flask or Django apps run using Gunicorn or Uvicorn, with more than one --workers configured.

To work around this, we recommend following OpenTelemetry's programmatic auto-instrumentation guide. Create a wrapper Python script for programmatic auto-instrumentation:

from opentelemetry.instrumentation.auto_instrumentation import initialize
initialize()

from your_app import app

Then, run your app without the prefix command:

command_to_run_your_wrapper

Have you enabled verbose logging?

Enable verbose logging by assigning the environment variable to 6.

Do not enable verbose logging in a production environment. Doing so will flood your system with verbose logs.

export =6

See logging level for more information.

Have you configured logging to file?

If the environment variable has been set, logs will be written to the file at that path instead of to stderr.

See logging output filepath for more information.

Is the library connected to the ?

Review the Python Library log messages to see if any connection issues are logged.

If your server is behind a firewall, you may not be able to connect to the . See Firewall or access control requirements.

Is the configured correctly?

The [%=NH-services.gen-ServiceKey%] is set by defining the system environment variable .

Are there duplicate spans in traces?

A trace in may contain duplicate spans due to existing dependencies between Python libraries and each dependent library having OpenTelemetry instrumentation libraries available. For example, an RPC request can involve the requests and urllib3 libraries, which are both instrumented by default. Similarly, a database call can involve the sqlalchemy and asyncpg libraries, which are both instrumented by default.

If you decide that you only want one dependent library to record spans, you can set the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable as described above. To instrument only requests, set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=urllib3.

Do you have more than one APM extension installed?

Do not run other APM libraries alongside the Python Library. Remove other APM libraries from your Python application before using the Python Library.

To remove the AppOptics Agent from your application, see Migrate the Python Library from AppOptics to [%=Standard.ProductLong%].

Are there package version conflicts at application package installation?

You could have a package version conflict if you installed additional OpenTelemetry-based libraries to customize the behavior (see Customize OpenTelemetry Propagators and Customize OpenTelemetry Exporters) of the installed Python Library solarwinds-apm. If the version number requirements are too strict, there may be a pkg_resources.VersionConflict error.

Consider using approximate versioning. For example, in the package configuration for my_custom_propagator, use opentelemetry-api ~= 1.11.1 instead of opentelemetry-api == 1.11.1.

Are spans for database queries showing entire statements?

OpenTelemetry instrumentation libraries export database query spans with the attribute db.statement. There is a security risk if a query is prepared using a hard-coded expression or a plain, string-interpolated expression. If the query is prepared this way, db.statement contains all values, including entity IDs.

To avoid this risk, build queries using a database interface library such as dbapi, asyncpg, pymongo, or sqlalchemy. These libraries make queries with parameterized expressions and use %s for parameters in db.statement.

Is there an instrumentation library-specific issue?

psycopg2

OpenTelemetry instrumentation of psycopg2 will not work if only the binary is installed ( psycopg2-binary). Install the full version of psycopg2 with the command pip install psycopg2.

sqlite3

Previous versions of OpenTelemetry instrumentation sqlite3 break application-database connections and display the error sqlite3.ProgrammingError: Base Connection.__init__ not called. This issue was fixed by the OpenTelemetry community and included in Python Library version 0.8.0. See Upgrade the Python Library

If you cannot upgrade to Python Library 0.8.0, either disable sqlite3 instrumentation with the OTEL_PYTHON DISABLED_INSTRUMENTATIONS environment variable, or use a different framework (PostgreSQL, MariaDB, or MySQL).