A common use case for AppOptics APM is looking at a trace in the Trace Details view to understand the performance and behavior of a request. However, trying to find this for a particular request can be challenging since not all requests are traced in detail; to minimize impact on the instrumented application, agents sample just a subset of requests for capturing detailed data.
This is where trace options and trigger trace comes in. Trace options is a custom HTTP header
X-Trace-Options that can be set on a request to carry additional information to the agents, one such option being
trigger-trace which we’ll call a trigger trace request.
When a trigger trace request is received, the agents will treat it as selected for sampling. It will still be constrained by our rate limiting mechanism and other configuration settings but in practice this means you can, within usage limits, send a request and expect it to result in a trace detail. A basic example using cURL:
# send a trigger trace request curl -I -H 'x-trace-options: trigger-trace' localhost:8080/examples/
# response headers show trace was triggered HTTP/1.1 200 X-Trace: 2BE3C6C60854AD302B2151B60D9FA2534E91B45F45A3C1FCAE91F5A44101 X-Trace-Options-Response: trigger-trace=ok
As shown above, a trigger trace interaction involves the "input" request header
X-Trace-Options and its corresponding "output" response header
X-Trace-Options-Response. The following sections describe these headers in more detail and walk through an example of how to find the triggered trace in the dashboard.
The trace options header is expected to contain only the ASCII character set. Options must be separated by the semicolon (
;) character and thus the option itself must not contain this character. The following options are supported:
A flag that if present attempts to trigger tracing on this request; any resulting trace will have a
TriggeredTrace: true KV set on the root span. Example:
One or more key-value pairs of extra information that is added to any resulting trace as a KV on the root span. A pair must be separated by the equal sign (
=) where the key portion must start with the lowercase string
custom- and not contain any whitespace. Each accepted pair is added as a KV with casing preserved. Example:
would become the following root span KVs:
Agents that support trace options will inject the response header
X-Trace-Options-Response to communicate status including the reason a trigger trace request was not traced and any ignored options. The response status fields are key-value pairs separated by the equal sign (
=) and multiple status fields are separated by semicolon (
;). The possible response status fields are:
ok confirms that the trace was triggered. A "non-ok" value gives the reason otherwise, and can include:
rate-exceeded– triggered traces are limited to a rate of roughly one every ten seconds. This status indicates the request has exceeded the limit and will not be traced.
not-requested– indicates that the agent did not recognize a
trigger-traceoption in the trace options request header. In this case the request will go through regular sampling and either be selected for tracing or not.
settings-not-available– the agent needs to fetch settings from the AppOptics server (the collector) periodically; if it isn’t able to, requests will not be traced. If you see this message consistently, please check that the agent is using a valid service key and can connect properly.
tracing-disabled– tracing can be disabled on the service (.NET example) or for the requested transaction (Java example). Trigger trace does not override this configuration setting, so the request will still not be traced nor get metrics reported.
trigger-traceoption can be disabled via configuration, you can find detais via the agent-specific links below. In this case the request will not be traced.
Only one trigger trace status will be set in the response, the most likely are:
A comma-separated list of any unknown option or leftover data from parsing the trace options request header, which are ignored by the agents. Example:
x-trace-options-response: ignored=a-key,CUSTOM-KEY,bad data
We'll use cURL again in this example, but you can try things out with any HTTP client or browser plugin that allows setting and reading the request and response headers. This time we'll send a trigger trace request and also set a custom KV on the resulting trace:
curl -I -H 'x-trace-options: trigger-trace;custom-UseCase=happy-path' localhost:8080/examples/
HTTP/1.1 200 X-Trace: 2BBDD121A9C32C1A0BA97AE8545B66BF32A45E4F084811FDEC25CCAE1B01 X-Trace-Options-Response: trigger-trace=ok
The response headers tell us that all of the trace options were processed (nothing was
ignored) and that a trace was triggered. As described in the FAQ on sampling, the
X-Trace response header ending in
01 means the request was traced; additionally, this header value embeds the Trace ID which uniquely identifies a trace and allows us to easily pull it up in the dashboard! The URL in the Trace Details view conveniently accepts any "service name" portion in the path and handles the full
X-Trace value gracefully. For example, when you load a trace in the dashboard the corresponding URL might look like:
but if you pasted in the following URL it loads the same trace:
So to view the triggered trace, simply copy the
X-Trace response header value, pull up any trace in the Trace Details view and replace its Trace ID in the URL with the copied value, then reload the browser. Pulling up our triggered trace, we can see in the Raw Data view it has both
TriggeredTrace: true and the requested custom KV
Trigger trace is enabled by default, note that in most agents it is only supported for auto-instrumented application/HTTP servers and excludes gRPC. See the links below for agent-specifc configuration and other details:
- .NET (agent version 3.9.0 and up)
- Go (agent version 1.9.0 and up) – supports SDK-started tracing and gRPC
- Java (agent version 6.14.0 and up)
- Node.js (agent version 6.7.0 and up)
- PHP (agent version 3.8.0 and up)
- Ruby (agent version 4.9.0 and up)
Navigation Notice: When the APM Integrated Experience is enabled, AppOptics shares a common navigation and enhanced feature set with other integrated experience products. How you navigate AppOptics and access its features may vary from these instructions.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.