Traces Explorer

A trace is the path of a single request through an application. SolarWinds Observability SaaS gathers traces continuously from observed services and mines them for data. Traces provide information about what happens as a request moves across multiple hosts and services. They can help you understand upstream and downstream service dependencies, assess performance at every step in the request, and troubleshoot issues such as errors or high latency.

The Traces Explorer provides a high-level overview of your traces, as well as detailed insights into each trace.

Open the Traces Explorer

To access the Traces Explorer, click Analyze in the left pane, and then click Traces. The Traces page displays high-level information about traces that occurred during the default time period.

The Traces Explorer can also be accessed from the details view of other entities. It displays information about traces associated with the selected entity.

• From the Service details view, click the Traces tab.
• From the Service details view, click the Exceptions tab. Then click a number in the Occurrences column to view trace information associated with that group of exceptions.
• From the Service details view, click the Databases tab. Then click a number in the Occurrences column to view trace information associated with that group of query operations.
• From the Service details view, click the Caches tab. Then click a number in the Occurrences column to view trace information associated with that group of cache operations.
• From the Service details view, click the Remote Services tab. Then click a number in the Occurrences column to view trace information associated with that group of remote services.
• From the Website entity details view, click in the Traces tab.

Specify the type of traces to view

By default, the Traces Explorer displays Traced Requests, or traces of requests to a service. Select an option from the drop-down menu at the top of the page to see traces of database queries, remote service calls, cache calls, or exceptions.

The type of traces you select determines which table columns are shown and what filters are available.

Change the time period

The Traces Explorer displays data collected during the specified time period. To change the time period, click the drop-down menu in the upper-right corner and choose how much historical data to display. You can choose to view all recent data for standard lengths of time, or to view data between two dates.

To show data from a custom time period, choose Custom. In the calendar that appears, click the starting date of your time period and click the ending date. The time period's start and end times default to the current time. To change a start or end time, click the clock next to the time you want to change and click the desired time.

Locate specific traces

To locate specific traces, you can apply filters or search by tag.

Filter traces

Apply filters to further specify which traces to display.

The Transaction filter, which shows only requests from a certain type of transaction, is available for all trace types.

The Status filter narrows down traces by Error, Unset, and explicit OK request statuses, and is available for all trace types, except for Traced Exceptions.

The Percentile slider filters response time durations of traces within the selected percentile range during the time period. Percentiles show where a particular response time falls compared to the average of all response time durations. You can use this slider to set the minimum and maximum percentiles to filter out outliers from the Traces List.

Other filters are available based on the selected trace type. For example, if you select Traced Queries, you could filter the list based on database type, database host, query operation, or the table being queried.

Apply filters

1. If the Filters pane is closed, click the filter icon to open it.

   The Filters pane lists the filters available for the selected entity type.

2. Select one or more values for each filter you want to apply.

   The Traces Explorer lists only the entities that meet the filter criteria. The Search box displays the tag and value(s) that correspond to each filter you applied.

   

Clear filters

To remove individual filters, clear the values you want to remove. To remove all filters, click Clear All at the bottom of the Filters pane.

Display the associated tag name

Hover over the information icon next to a filter to display the tag name related to that filtration option. These tag names can be used in the Search bar and the Metrics Explorer.

Search for traces

Each filter option corresponds to a tag. However, some tags available in the Search box are not available in the Filters pane.

1. Click inside the Search box.

   A drop-down menu shows the tags available for the selected trace type.

2. Click the tag you want to filter by.

   The drop-down menu shows all values for the selected tag that are associated with observed traces.

3. Click a value to select it, and then press Enter to perform the search.

   The Traces Explorer lists only traces with the specified tag value.

   The Search box displays your selections in the format tag:value, and the corresponding filters are selected in the Filters pane.

   To further refine the list, you can add other tags. If you include multiple criteria, entities must meet all criteria.

Saved searches

If you regularly use the same combination of search criteria (applied filters or tags), you can save the search to quickly run it again.

Save a search

1. Apply filters or enter search criteria to define the search you want to save.

2. Click the folder icon on the right side of the Search box, and click Save Search.

   The Create Saved Search dialog opens.

3. Enter a name to identify the search.

4. If you want the saved search to be available only to you, select Private.

5. Click Save.

Run a saved search

1. Click the folder icon on the right side of the Search box.

   The drop-down menu lists the saved searches you have access to.

2. Click the name of the search you want to run.

Select boxes on the heatmap to show only those traces

Click a box on the heatmap to select it, or hold the cursor down and drag the mouse pointer to select a group of boxes. The table below the heatmap lists only the traces that the selected boxes represent.

Click the heatmap anywhere outside the selected area to deselect the boxes and display information about all traces represented on the heatmap.

About the trace heatmap

The trace heatmap provides information about the traces that are currently selected for display (based on time period, filters, or other specifications):

• The x-axis indicates when a trace ran. Each unit represents a time period (for example, from 2022-09-21 02:41:47 to 2022-09-21 02:42:16),

• The y-axis indicates the duration of a trace. Each unit represents a time range (for example, between 6.7 ms and 7 ms).

• Each box represents the traces that ran during the time period and whose duration was within the time range.

  To identify traces with the highest latency, look for boxes near the top of the heatmap.

• The intensity of the box color indicates the number of traces the box represents. There are nine color variations. In Light theme, darker boxes represent higher numbers of traces. In Dark theme, lighter boxes represent higher numbers of traces.

  Exception traces are shades of red. All other trace types are shades of blue.

Hover over a box on the heatmap to see details about the number of traces it represents and the duration of those traces.

The table below the heatmap lists each trace represented on the heatmap. The information shown about each trace includes the service and transaction the trace was a part of, the date and time of the trace, and additional details depending on what type of trace is selected.

Click the name of any entity listed in the table (for example, service, transaction, website, or host) to display information about that entity in the Entity Explorer.

Display trace details

In the table below the heatmap, click in the Details column to view detailed information about a trace and the spans that make up that trace. A span represents a specific operation within a trace.

The details view includes information about any errors that occurred during the trace, any queries executed during the trace, and any log events associated with the trace. It also includes a graphical representation of the spans within the trace. You can display detailed information about each span. Use the span information to understand the request flow and to identify where any errors or performance issues occurred. Span details include the duration of the span (both as a time and as a percentage of the total trace duration), which can help you prioritize troubleshooting activities.

Select a service

By default, the details window displays information about all services. If a trace spans multiple services, you can view the details of a trace within a specific service. Select the service from the drop-down menu at the top of the details view.

About the span chart

The chart at the top of the page shows the spans within the trace. Each trace starts with a root span, which represents the entire operation. A trace can also include one or more child spans that describe sub-operations.

The chart contains two sections:

• The overview section at the top provides a simplified picture of the spans. It includes time indicators and, if errors occurred, symbols to indicate when they occurred.

• The main section shows the hierarchy of the spans. The length of the rectangle that represents a span indicates the span's duration. Its position on the graph represents the span's execution order. If span B is executed by span A (it has a "child of" relationship), it is shown below span A.

  

  If span B is executed after span A (it has a "follows from" relationship), it is shown to the right of span A.

  

Magnify part of the main section of the span chart

1. Position your cursor in the overview (top) section on one side of the area you want to magnify.

   The cursor becomes a plus sign.

   

2. Drag the cursor to one side to define the area you want to magnify.

   In the main (bottom) section, the selected area is expanded to the width of the chart.

   

3. To zoom out again, click the overview section anywhere outside the selected area.

Display details about a span

To view information about a span, click the rectangle that represents that span in the main section of the span chart. The Inspector Panel displays detailed information, including any backtraces, queries, errors, or log events associated with the span.

If there are other spans before or after that span on the span layer, you can click the right or left arrow in the upper-right corner of the panel to display information about the span before or after the current span.

Display details about a span layer

To view information about a span layer, click the name of the span layer in the Trace Breakdown table. Or, if the Inspector Panel is currently showing information about a span, click the arrow in the upper-left corner of the panel to display information about the span layer.

The Inspector Panel displays high-level information about all the spans in the span layer.