Documentation forSolarWinds Observability SaaS

Trace context in queries

Correlating application queries with database performance

Databases are typically an integral part of a distributed system and critical to performance. When an application is instrumented with the APM library, instrumentation collects client-side information about database queries made by the application such as the database type, the query sent to the server, and the response time. Instrumentation can optionally insert contextual values (called trace context) into the query which makes it possible on the database server's side to identify the source of a query (for example, queries from a particular transaction).

Propagating the trace context in queries also enables deeper insight when the database server is monitored with DBO and APM is integrated with DBO. The client-side observations about queries in APM can be correlated to the server-side observations about database performance in DBO.

Trace context

When a request enters an instrumented application, the APM libraries create a unique trace context and maintain that context through the lifetime of the transaction. The trace context is included when the library exports performance metrics and traces, and the context can be propagated to your application logs (if enabled).

Propagating the context in queries is achieved by inserting a SQL comment either before or after the detected SQL statement, based on the sqlcommenter approach. The comment can have one or more key/value fields. Multiple fields are separated by the comma (,) character.

On the database server, the inserted comment can typically be seen in server logs. This is an example from the MySQL general query logs that shows the Python Library has propagated trace context including the unique trace identifier (traceparent) along with application and database client framework information:

2023-06-19T22:44:34.817229Z 10 Query SELECT city.id, city.name, city.countrycode, city.district, city.population FROM city WHERE city.id = 1818 
/*controller='my_cool_view_function',db_driver='mysqldb',db_framework='sqlalchemy%3A0.39b0',framework='flask%3A2.2.5',route='/manual/',traceparent='00-a9914fe1718b2470dd29185b4d46746e-40a575bfd839229a-01'*/

The traceparent field is in the form /*traceparent='00-<TraceID>-<SpanID>-<Trace Flags>'*/. The sections below describe Trace ID, Span ID and Trace Flags in more detail.

Trace ID

The Trace ID is a hex string that represents the associated transaction trace. This Trace ID can also be seen in the Transaction Details page of the Traces Explorer. An example Trace ID is: abcdef1234567890abcdef1234567890.

Span ID

The Span ID is a hex string representing the associated span. This Span ID can be seen in the Span panel of the Transaction Details page in the Traces Explorer. An example Span ID is: 1234abcd1234abcd.

Trace Flags

Trace Flags indicates whether the trace was sampled. 01 means there was a corresponding sampled trace, and 00 means no trace was sampled.

Enable trace context in queries

Trace context in queries is disabled by default. Note that this feature is currently supported only for SQL statements, with the following limitations:

  • Server-side prepared statements: The APM libraries can only access the statement during the prepare operation, which means if trace context is inserted, subsequent executions of the statement will likely carry the wrong trace context. Thus, prepared statements are excluded where possible or otherwise noted in the library-specific pages below.

  • Stored procedures: The APM libraries cannot feasibly insert the trace context into stored procedure call operations.

  • Multiple statements sent in a single call to the database server: The trace context will only be inserted at the beginning of the first statement.

See the links below for information about enabling this feature and the support matrix for each library: