Trigger Trace

The following content pertains to trigger trace and trace options for the legacy AppOptics agents.

A common use case for AppOptics APM is looking at a trace in the Trace Details view to understand the performance and behavior of a request. However, trying to find this for a particular request can be challenging since not all requests are traced in detail; to minimize impact on the instrumented application, agents sample just a subset of requests for capturing detailed data.

This is where trace options and trigger trace comes in. Trace options is a custom HTTP header X-Trace-Options that can be set on a request to carry additional information to the agents, one such option being trigger-trace which we’ll call a trigger trace request.

When a trigger trace request is received, the agents will treat it as selected for sampling. It will still be constrained by our rate limiting mechanism and other configuration settings but in practice this means you can, within usage limits, send a request and expect it to result in a trace detail. A basic example using cURL:

# send a trigger trace request
			curl -I -H 'x-trace-options: trigger-trace' localhost:8080/examples/

# response headers show trace was triggered
HTTP/1.1 200
X-Trace: 2BE3C6C60854AD302B2151B60D9FA2534E91B45F45A3C1FCAE91F5A44101
X-Trace-Options-Response: trigger-trace=ok

As shown above, a trigger trace interaction involves the "input" request header X-Trace-Options and its corresponding "output" response header X-Trace-Options-Response. The following sections describe these headers in more detail and walk through an example of how to find the triggered trace in the dashboard.

X-Trace-Options

The trace options header is expected to contain only the ASCII character set. Options must be separated by the semicolon (;) character and thus the option itself must not contain this character. The following options are supported:

trigger-trace

A flag that if present attempts to trigger tracing on this request; any resulting trace will have a TriggeredTrace: true KV set on the root span. Example:

x-trace-options: trigger-trace

custom-*

One or more key-value pairs of extra information that is added to any resulting trace as a KV on the root span. A pair must be separated by the equal sign (=) where the key portion must start with the lowercase string custom- and not contain any whitespace. Each accepted pair is added as a KV with casing preserved. Example:

x-trace-options: custom-key-1=value1;custom-KeyTwo=ValueTwo123

would become the following root span KVs:

• custom-key-1: value1
• custom-KeyTwo: ValueTwo123

X-Trace-Options-Response

Agents that support trace options will inject the response header X-Trace-Options-Response to communicate status including the reason a trigger trace request was not traced and any ignored options. The response status fields are key-value pairs separated by the equal sign (=) and multiple status fields are separated by semicolon (;). The possible response status fields are:

trigger-trace

The value ok confirms that the trace was triggered. A "non-ok" value gives the reason otherwise, and can include:

• rate-exceeded – triggered traces are limited to a rate of roughly one every ten seconds. This status indicates the request has exceeded the limit and will not be traced.
• not-requested – indicates that the agent did not recognize a trigger-trace option in the trace options request header. In this case the request will go through regular sampling and either be selected for tracing or not.
• settings-not-available – the agent needs to fetch settings from the AppOptics server (the collector) periodically; if it isn’t able to, requests will not be traced. If you see this message consistently, please check that the agent is using a valid service key and can connect properly.
• tracing-disabled – tracing can be disabled on the service (.NET example) or for the requested transaction (Java example). Trigger trace does not override this configuration setting, so the request will still not be traced nor get metrics reported.
• trigger-tracing-disabled – the trigger-trace option can be disabled via configuration, you can find detais via the agent-specific links below. In this case the request will not be traced.

Only one trigger trace status will be set in the response, the most likely are:

x-trace-options-response: trigger-trace=ok

x-trace-options-response: trigger-trace=rate-exceeded

ignored

A comma-separated list of any unknown option or leftover data from parsing the trace options request header, which are ignored by the agents. Example:

x-trace-options-response: ignored=a-key,CUSTOM-KEY,bad data

Full Example

We'll use cURL again in this example, but you can try things out with any HTTP client or browser plugin that allows setting and reading the request and response headers. This time we'll send a trigger trace request and also set a custom KV on the resulting trace:

curl -I -H 'x-trace-options: trigger-trace;custom-UseCase=happy-path' localhost:8080/examples/

HTTP/1.1 200
X-Trace: 2BBDD121A9C32C1A0BA97AE8545B66BF32A45E4F084811FDEC25CCAE1B01
X-Trace-Options-Response: trigger-trace=ok

The response headers tell us that all of the trace options were processed (nothing was ignored) and that a trace was triggered. As described in the FAQ on sampling, the X-Trace response header ending in 01 means the request was traced; additionally, this header value embeds the Trace ID which uniquely identifies a trace and allows us to easily pull it up in the dashboard! The URL in the Trace Details view conveniently accepts any "service name" portion in the path and handles the full X-Trace value gracefully. For example, when you load a trace in the dashboard the corresponding URL might look like:

https://my.appoptics.com/apm/999999/services/javatest/traces/BDD121A9C32C1A0BA97AE8545B66BF32A45E4F08

but if you pasted in the following URL it loads the same trace:

https://my.appoptics.com/apm/999999/services/lookup/traces/2BBDD121A9C32C1A0BA97AE8545B66BF32A45E4F084811FDEC25CCAE1B01

So to view the triggered trace, simply copy the X-Trace response header value, pull up any trace in the Trace Details view and replace its Trace ID in the URL with the copied value, then reload the browser. Pulling up our triggered trace, we can see in the Raw Data view it has both TriggeredTrace: true and the requested custom KV custom-UseCase: happy-path:

Enabling

Trigger trace is enabled by default, note that in most agents it is only supported for auto-instrumented application/HTTP servers and excludes gRPC. See the links below for agent-specifc configuration and other details:

• .NET (agent version 3.9.0 and up)
• Go (agent version 1.9.0 and up) – supports SDK-started tracing and gRPC
• Java (agent version 6.14.0 and up)
• Node.js (agent version 6.7.0 and up)
• PHP (agent version 3.8.0 and up)
• Ruby (agent version 4.9.0 and up)

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.