Documentation forSolarWinds Observability

Trigger trace

The Traces Explorer is useful to understand the performance and behavior of a request. However, since APM libraries sample a subset of requests to minimize impact on the instrumented application, finding all details for a specific request may be a challenge. Trace options and trigger trace help ensure data you need is collected and sent to SolarWinds Observability.

Trace options is a custom HTTP header (X-Trace-Options) that is set on a request to carry additional information to the libraries. One of the options you can set is a trigger trace request (trigger-trace). When a trigger trace request is received, the libraries select the trace for sampling. Traces are still constrained by rate limiting mechanism and other configuration settings, but, in general, if you send a trigger trace request, the result will be available in the Traces Explorer, within usage limits.

A basic example sending a trigger trace request using cURL is below.

# send a trigger trace request
curl -I -H 'x-trace-options: trigger-trace' localhost:8080/examples/

The trigger trace interaction involves sending a request with the X-Trace-Options input header and getting a response that shows whether the trigger trace request was handled. 

# response headers show trace was triggered
HTTP/1.1 200 OK 
X-Trace: 00-e2accbe02b0587a4f4662b381f6fabd4-226d73eadfe12c73-01 
X-Trace-Options-Response: trigger-trace=ok

The X-Trace response header contains the trace context for this response in W3C trace context format. The fields in the header are delimited by a dash (-) and the second field is the Trace ID. In the example above, the Trace ID is e2accbe02b0587a4f4662b381f6fabd4.

The following sections describe these headers in more detail and walk through an example of how to find the triggered trace in the dashboard.

X-Trace-Options

The trace options header is expected to contain only the ASCII character set. Options must be separated by the semicolon (;) character so the option itself must not contain a semicolon. The following options are supported:

trigger-trace

A flag that, if present, attempts to trigger tracing on this request; any resulting trace will have a TriggeredTrace: true KV set on the root span. Example:

x-trace-options: trigger-trace

custom-*

One or more key-value pairs of extra information that is added to any resulting trace as a KV on the root span. A pair must be separated by the equal sign (=), where the key portion must start with the lowercase string custom- and not contain any whitespace. Each accepted pair is added as a KV with casing preserved. Example:

x-trace-options: custom-key-1=value1;custom-KeyTwo=ValueTwo123

would become the following root span KVs:

  • custom-key-1: value1
  • custom-KeyTwo: ValueTwo123

X-Trace-Options-Response

Libraries that support trace options will inject the response header X-Trace-Options-Response to communicate status including the reason a trigger trace request was not traced and any ignored options. The response status fields are key-value pairs separated by the equal sign (=) and multiple status fields are separated by semicolon (;). The possible response status fields are:

trigger-trace

The value ok confirms that the trace was triggered. A "non-ok" value gives the reason otherwise, and can include:

  • rate-exceeded – triggered traces are limited to a rate of roughly one every ten seconds. This status indicates the request has exceeded the limit and will not be traced.
  • not-requested – indicates that the library did not recognize a trigger-trace option in the trace options request header. In this case, the request will go through regular sampling and either be selected for tracing or not.
  • settings-not-available – the library needs to fetch settings from the SolarWinds APM collector periodically; if it isn’t able to, requests will not be traced. If you see this message consistently, please check that the library is using a valid service key and can connect properly.
  • tracing-disabled – tracing can be disabled on the service (.NET example) or for the requested transaction (Java example). Trigger trace does not override this configuration setting, so the request will still not be traced nor get metrics reported.
  • trigger-tracing-disabled – the trigger-trace option can be disabled via configuration, you can find details via the library-specific links below. In this case the request will not be traced.

Only one trigger trace status will be set in the response. The most likely statuses are:

x-trace-options-response: trigger-trace=ok

x-trace-options-response: trigger-trace=rate-exceeded

ignored

A comma-separated list of any unknown option or leftover data from parsing the trace options request header, which are ignored by the libraries. Example:

x-trace-options-response: ignored=a-key,CUSTOM-KEY,bad data

Full Example

You can send an example request with a trigger trace request using cURL or with an HTTP client or browser plugin that allows you to set and read request and response headers. The following example uses cURL to send a trigger trace request and set a custom key-value pair on the resulting trace.

curl -I -H 'x-trace-options: trigger-trace;custom-UseCase=happy-path' localhost:8080/examples/

The resulting response headers are below.

HTTP/1.1 200
X-Trace: 00-e2accbe02b0587a4f4662b381f6fabd4-226d73eadfe12c73-01
X-Trace-Options-Response: trigger-trace=ok

The response headers indicate all the trace options were processed (nothing was ignored) and that a trace was triggered. The X-Trace response header ending in 01 means the request was traced; additionally, this header value embeds the Trace ID. The Trace ID uniquely identifies a trace and allows you to search for it in the Traces Explorer. When you load a trace in the Traces Explorer, the corresponding URL might look like:

https://my.xx-yy.cloud.solarwinds.com/1234567890/traces/E2ACCBE02B0587A4F4662B381F6FABD4/67C8AA9ADFF3006D/details

The portion of the URL in red, after the traces folder, is the Trace ID. Both upper and lowercase characters are accepted. If you load the following URLs in the browser, they will load the same trace: 

https://my.xx-yy.cloud.solarwinds.com/1234567890/traces/e2accbe02b0587a4f4662b381f6fabd4/details

or

https://my.xx-yy.cloud.solarwinds.com/1234567890/traces/E2ACCBE02B0587A4F4662B381F6FABD4/details

To view the triggered trace, copy the Trace ID portion of the X-Trace response header value, then load any trace in the Trace Details view. Replace the Trace ID in the URL with the copied value from the response header and load the modified URL in the browser.

Review the Raw Data view for the triggered trace to see both TriggeredTrace: true and the requested custom key-value pair custom-UseCase: happy-path:

Enabling

Trigger trace is enabled by default. In most libraries, it is only supported for auto-instrumented applications or HTTP servers and excludes gRPC. See the following topics for library-specific configuration and other details:

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.