A trace is the path of a single request through an application. SolarWinds Observability gathers traces continuously from observed services and mines them for data. Traces provide information about what happens as a request moves across multiple hosts and services. They can help you understand upstream and downstream service dependencies, assess performance at every step in the request, and troubleshoot issues such as errors or high latency.
The Traces Explorer provides a high-level overview of your traces, as well as detailed insights into each trace.
Open the Traces Explorer
To access the Traces Explorer, click Analyze in the left pane, and then click Traces. The Traces page displays high-level information about traces that occurred during the default time period.
The Traces Explorer can also be accessed from the details view of other entities. It displays information about traces associated with the selected entity.
- From the Service details view, click the Traces tab.
- From the Service details view, click the Exceptions tab. Then click a number in the Occurrences column to view trace information associated with that group of exceptions.
- From the Service details view, click the Databases tab. Then click a number in the Occurrences column to view trace information associated with that group of query operations.
- From the Service details view, click the Caches tab. Then click a number in the Occurrences column to view trace information associated with that group of cache operations.
- From the Service details view, click the Remote Services tab. Then click a number in the Occurrences column to view trace information associated with that group of remote services.
- From the Website details view, click in the Traces tab.
By default, the Traces Explorer displays Traced Requests, or traces of requests to a service. Select an option from the drop-down menu at the top of the page to see traces of database queries, remote service calls, cache calls, or exceptions.
The type of traces you select determines which table columns are shown and what filters are available.
Specify the traces you want to see
To narrow the list of traces, you can select the time period, apply filters, or select an area on the heatmap.
Change the time period
The Traces Explorer displays traces collected during the specified time period. To change the time period, click the drop-down menu in the upper-right corner and choose how much historical data to display.
Apply filters to further specify which traces to display. Click to open the Filters pane, and select values for one or more filters.
The Transaction filter, which shows only requests from a certain type of transaction, is available for all trace types.
Other filters are available based on the selected trace type. For example, if you select Traced Queries, you could filter the list based on database type, database host, query operation, or the table being queried.
Hover over the information icon next to a filter to display the tag name related to that filtration option. These tag names can be used in the Search bar and the Metrics Explorer. .
Select boxes on the heatmap
Click a box on the heatmap to select it, or hold the cursor down and drag the mouse pointer to select a group of boxes. The table below the heatmap lists only the traces that the selected boxes represent.
Click the heatmap anywhere outside the selected area to deselect the boxes and display information about all traces represented on the heatmap.
The trace heatmap provides information about the traces that are currently selected for display (based on time period, filters, or other specifications):
The x-axis indicates when a trace ran. Each unit represents a time period (for example, from 2022-09-21 02:41:47 to 2022-09-21 02:42:16),
The y-axis indicates the duration of a trace. Each unit represents a time range (for example, between 6.7 ms and 7 ms).
Each box represents the traces that ran during the time period and whose duration was within the time range.
To identify traces with the highest latency, look for boxes near the top of the heatmap.
The intensity of the box color indicates the number of traces the box represents.
Exception traces are shades of red. All other trace types are shades of blue.
Hover over a box on the heatmap to see details about the number of traces it represents and the duration of those traces.
The table below the heatmap lists each trace represented on the heatmap. The information shown about each trace includes the service and transaction the trace was a part of, the date and time of the trace, and additional details depending on what type of trace is selected.
Click the name of any entity listed in the table (for example, service, transaction, website, or host) to display information about that entity in the Entity Explorer.
Display trace details
In the table below the heatmap, click in the Details column to view detailed information about a trace and the spans that make up that trace. A span represents a specific operation within a trace.
The details view includes information about any errors that occurred during the trace, any queries executed during the trace, and any log events associated with the trace. It also includes a graphical representation of the spans within the trace. You can display detailed information about each span. Use the span information to understand the request flow and to identify where any errors or performance issues occurred. Span details include the duration of the span (both as a time and as a percentage of the total trace duration), which can help you prioritize troubleshooting activities.
Select a service
By default, the details window displays information about all services. If a trace spans multiple services, you can view the details of a trace within a specific service. Select the service from the drop-down menu at the top of the details view.
The chart at the top of the page shows the spans within the trace. Each trace starts with a root span, which represents the entire operation. A trace can also include one or more child spans that describe sub-operations.
The chart contains two sections:
The overview section at the top provides a simplified picture of the spans. It includes time indicators and, if errors occurred, symbols to indicate when they occurred.
The main section shows the hierarchy of the spans. The length of the rectangle that represents a span indicates the span's duration. Its position on the graph represents the span's execution order. If span B is executed by span A (it has a "child of" relationship), it is shown below span A.
If span B is executed after span A (it has a "follows from" relationship), it is shown to the right of span A.
Position your cursor in the overview (top) section on one side of the area you want to magnify.
The cursor becomes a plus sign.
Drag the cursor to one side to define the area you want to magnify.
In the main (bottom) section, the selected area is expanded to the width of the chart.
To zoom out again, click the overview section anywhere outside the selected area.
To view information about a span, click the rectangle that represents that span in the main section of the span chart. The Inspector Panel displays detailed information, including any backtraces, queries, errors, or log events associated with the span.
If there are other spans before or after that span on the span layer, you can click the right or left arrow in the upper-right corner of the panel to display information about the span before or after the current span.
Display details about a span layer
To view information about a span layer, click the name of the span layer in the Trace Breakdown table. Or, if the Inspector Panel is currently showing information about a span, click the arrow in the upper-left corner of the panel to display information about the span layer.
The Inspector Panel displays high-level information about all the spans in the span layer.