Integrate APM with DBO
The SolarWinds Observability SaaS APM and DBO modules can be integrated provide a better understanding of how the database server impacts the performance of services. With this integration, you can easily navigate between the two modules and see data correlation.
Support matrix
APM and DBO Integration is available client libraries on the APM instrumentation side. Click a library name to see the database types and client libraries or frameworks:
.NET
Trace context is inserted only for statements from a transaction sampled for detailed tracing.
Trace context is not inserted into server-side prepared statements and stored procedures.
| Database type | Client library/framework |
|---|---|
| MySQL | ADO.NET data providers MySql.Data.MySqlClient and MySqlConnector |
Go
| Database type | Client library/framework | Notes |
|---|---|---|
| MySQL, PostgreSQL | GitHub - XSAM/otelsql: OpenTelemetry instrumentation for database/sql | SolarWinds does not recommend enabling this feature if your application uses server-side prepared statements, because the inserted context will likely be incorrect for executions of the statement. |
Java
Node.js
| Database type |
Client library/framework |
Notes |
|---|---|---|
| MySQL | mysql2 | Trace context is not inserted into server-side prepared statements. |
Python
| Database type |
Client library/framework |
Notes |
|---|---|---|
| MySQL | SQLAlchemy | |
| PostgreSQL | psycop
psycopg2 |
SolarWinds does not recommend enabling this feature if your application uses server-side prepared statements, because the inserted context will likely be incorrect for executions of the statement. |
| SQLAlchemy | Should only be enabled for integration of APM with DBO if trace context in queries is not enabled for psycopg/psycopg2. Otherwise, multiple comments are inserted. |
Setup
Complete the following tasks to enable trace context propagation by the APM Agent, and to enable trace context processing by the DBO Agent.
Enable trace context propagation by the APM Agent
On the APM side, you must enable trace context propagation to database queries. The APM instrumentation library injects trace context as a comment to queries that are executed by the database engine:
/*traceparent='00-e65e156d1acf2ddcf5930d3554783925-1c65fa749e2de1ae-01'*/
Context injection configuration varies depending on the instrumented service implementation language. Follow the links in Trace context in queries to enable this feature for your service.
Enable trace context processing by the DBO Agent
The following configuration is required on the DBO Agent side. You can configure integration in the global configuration file (which affects all DBO Agents running on the system) or an agent-specific configuration file (which affects one DBO Agent), The agent configuration file overrides the global database configuration file. For more information about DBO configuration, see Database configuration files.
-
Open either the DBO global or agent-specific configuration file in a text editor:
/etc/solarwinds/dbo-global.conf(global)/etc/solarwinds/<agent-name>.conf(agent-specific)
-
Set the following property in the configuration file:
{ "tag-delimiters": "=," } -
Set the following property only for MySQL off-host deployment if
"perf-schema-method": "events_statements_history_long"is not used:{ "enable-apm-dbo-trace": "true", }
APM data in the context of DBO queries
After a successful configuration, you will see APM data in the context of the DBO module.
Direct links from query samples to traces
After integration, the Samples tab shows links between traced requests and the query samples captured by DBO, showing the context in which each query was executed.
List of traced executions of the selected query
After integration, the DBO Query details page includes the Traces tab, which lists traced executions of the selected query. You can filter the list using the smart search feature and easily drill down to traces associated with poor query performance.
DBO data in the context of APM
After successful configuration, you will see DBO data in the context of trace details.
Links from trace details to query performance details in the DBO module
You can drill down from queries traced by the APM module to performance details in the DBO module. You can navigate either to the query in DBO, or to the exact query sample if it is available for the selected trace.
Relationships in the entities model
After integration, the entities model is aware of the relation between a service and a database instance.
Integration troubleshooting
Verifying that trace context is correctly injected into queries
If APM is correctly configured, you should see trace context injected as a comment in sample details on the DBO side. You can filter queries by the keyword traceparent to see only query samples with trace context injected.
Verifying that DBO spans are correctly published by the DBO Agent
To verify that the DBO Agent publishes server side spans for an operation, navigate from the query sample to the trace details view. Search for DBO events on the event graph view. If an event is available, it means that the DBO Agent correctly published the span that represents the server side of a query execution.
I see mapping only for a limited number of queries
After a successful configuration, it's possible that only frequently executed queries get links from trace details to DBO. To get links, you must wait until the query is sampled by the DBO module. This could take a few hours if the query is rarely executed.
Not all query samples have links to traces
Links might be not available in the following situations:
-
The trace has not been processed yet. Keep in mind that it may take a few minutes before the trace is persisted.
-
The trace is not sampled. If x-trace ends with
-00it means, that trace is not sampled and will not be stored in the system. For example:/*traceparent='00-2d06b0c9c906815dd146fb24f3a4678a-d48cf7904f9a1b62-00'*/ -
Trace context is not injected as a comment to a query because integration is not configured correctly on the APM Instrumentation side.