Application logs are often critical in troubleshooting, but in a busy system it can be hard to disentangle interleaved log messages to see those for a particular transaction. One approach is to assign each transaction a unique ID that is included in the log message; the challenge then becomes how to propagate this ID through concurrency mechanisms such as threaded or asynchronous processing and across distributed services.
Fortunately, this is exactly what our agents do. When a request enters an instrumented application, the agent creates a unique trace context (aka Trace ID) and maintains it through the lifetime of the transaction in order to report performance metrics, and if sampled, the transaction trace. And now it's possible to automatically add this Trace ID to your application logs.
The trace context inserted into application logs takes the form of a Trace ID that is 40 hex characters followed by a dash and one character denoting if the request is sampled (1) or not (0). Some examples:
5CAE5A1CB07D4273ED5197B4DB004D20AD7F698A-1– sampled (has a corresponding trace)
BB7386517A8A9216FF160D69F1EBC5DC7CF7509C-0– not sampled (no corresponding trace)
0000000000000000000000000000000000000000-0– no valid trace context, for example the current code flow is not traced
Agents will automatically insert the Trace ID with a tag or attribute name
ao.traceId; please see the agent-specific pages
under Enabling for details on the inserted value under different configuration options and logging libraries.
Transaction traces are also identified by this Trace ID in the Trace Details view. Have you ever noticed when pulling up a trace that the URL ends with a hex string?
With trace context in logs enabled, you can search application logs on this value (in the above example
72321948795199A0421FE3D949E98A1597740C26) to easily find log messages corresponding to the trace.
Conversely, if you see a Trace ID ending in
-1 in the log message, there will be a corresponding trace that you can look
up in the dashboard – the easiest way is to copy the Trace ID excluding
-1 from the log message, pull up any trace in the
Trace Details view and replace its Trace ID in the URL with the copied value, then reload the browser.
Trace context in logs is disabled by default. See the links below on how to enable this feature in each agent, supported logging libraries and SDK examples:
- .NET (agent version 3.8.0 and up)
- Go (agent version 1.7.0 and up)
- Java (agent version 6.10.0 and up)
- Node.js (agent version 6.1.0 and up)
- PHP (agent version 3.6.0 and up)
- Python (agent version 4.1.0 and up)
- Ruby (agent version 4.7.0 and up)
Navigation Notice: When the APM Integrated Experience is enabled, AppOptics shares a common navigation and enhanced feature set with other integrated experience products. How you navigate AppOptics and access its features may vary from these instructions.