Python trace context in logs

Adding trace context to application logs provides the ability to correlate the log messages from a traced transaction, and if sampled, the log messages to the transaction trace detail. Trace context in logs is disabled by default. Trace context can be included automatically using the Python Library configuration or inserted using the SDK. See the following sections:

About trace context in logs

The Python Library can automatically add the trace context to logs generated by the Python logging module and other logging modules based on this standard module. In particular, it will add:

• A string of the form [trace_id=<Trace ID> span_id=<Span ID> trace_flags=<Trace Flags> resource.service.name=<Service Name>] to the beginning of each logged message.

  Example

  [trace_id=4442bce141ca9101bddcbec545d8e826 span_id=53033a49320fe226 trace_flags=01 resource.service.name=my-cool-service-name]

• Custom entries in the LogRecord dictionary of the form 'otelTraceId': '<Trace ID>', 'otelSpanId': '<Span ID>', 'otelTraceSampled': True, 'otelServiceName': '<Service Name>'.

  Example

  'otelTraceId': '4442bce141ca9101bddcbec545d8e826', 'otelSpanId': '53033a49320fe226', 'otelTraceSampled': True, 'otelServiceName': 'my-cool-service'

Insert trace context automatically

Automatic insertion works for any logger that uses the Python logging module. To enable the feature in the Python Library configuration using the OTEL_PYTHON_LOG_CORRELATION system environment variable, see Trace context in logs. When a message is logged, the logging facility creates a LogRecord instance containing all information related to the event being logged. The Python Library automatically adds four new OpenTelemetry attributes to the created LogRecord instance. The new attributes contain trace information in the format 'otelTraceId': '<Trace ID>', 'otelSpanId': '<Span ID>', 'otelTraceSampled': True, 'otelServiceName': '<Service Name>'.

In addition to the extension of the LogRecord instance, the library also inserts the trace_id, span_id, trace_flags, and resource.service.name into the formatted message when it is emitted by a handler. This insertion happens automatically. You do not need to modify the handler's formatter. The trace information is injected at the beginning of the actual message to be logged.

This automatic injection is only performed if the message is passed through a formatter native to the logging module (for example, logging.Formatter).

Trace context can be added to structured logs if the structured logs are created using logging facilities that extend the native Python logging module. An example of a compatible logging facility is python-json-logging. To add trace context to those structured logs, use the four attributes added to the LogRecord instance.

Sample automatic insertion usage

import logging

logging.warning('Warning to be logged.')

Results

The sample automatic insertion usage above would produce a log message similar to this one.

[trace_id=5d22a781de417d5b74107dee0355be95 span_id=e6534888a6046cb7 trace_flags=01 resource.service.name=my-cool-service-name] - Warning to be logged.

Customize log format with automatic insertion

To use a custom log format with the SolarWinds Observability SaaS Python Library, configure the OTEL_PYTHON_LOG_FORMAT system environment variable.

Note that instrumentation when OTEL_PYTHON_LOG_CORRELATION is enabled makes the first call to logging.basicConfig. Subsequent calls have no effect on the global logger.

Use the SDK to insert trace context

If your logging setup is not supported for automatic insertion or you want finer control of trace context in logs, you can use the OpenTelemetry API. The API allows you to insert trace context into any kind of log. You do not need to set up OTEL_PYTHON_LOG_CORRELATION to use the OTel API.

Sample SDK usage

import logging
from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME

logger = logging.getLogger(__name__)
service_name = trace.get_tracer_provider().resource.attributes[SERVICE_NAME]
ctx = trace.get_current_span().get_span_context()
logger.warning(
    "service_name=%s, trace_id=%s, span_id=%s, trace_flags=%02d",
    service_name,
    ctx.trace_id,
    ctx.span_id,
    ctx.trace_flags,
  )

Results

The sample SDK above would produce a log message similar to this one.

service_name: my-cool-service-name, trace_id: 333869841001235999982427187735164987618, span_id: 13032681704019136744, trace_flags: 01

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.

Use the SDK to insert trace context in AWS Lambda CloudWatch logs

If your setup is in AWS Lambda, trace context can be inserted into CloudWatch logs using the OpenTelemetry API. Do not enable OTEL_PYTHON_LOG_CORRELATION in AWS Lambda since it results in log duplication.

Sample SDK usage

import logging
from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME

logger = logging.getLogger(__name__)
service_name = trace.get_tracer_provider().resource.attributes[SERVICE_NAME]

def lambda_handler(event, context):
    ctx = trace.get_current_span().get_span_context()
    logger.warning(
        "service_name=%s, trace_id=%s, span_id=%s, trace_flags=%02d",
        service_name,
        ctx.trace_id,
        ctx.span_id,
        ctx.trace_flags,
    )
    return {
        "statusCode": 200,
    }