Documentation forSolarWinds Observability

Configure custom instrumentation

Custom instrumentation can be used to get extra application-specific visibility in SolarWinds Observability. In an APM-instrumented application, the language SDK methods provided by the library are a simple way to get additional visibility. The SDK methods can be used to log more details as spans on a traced request, and to report custom timer/counter metrics.

Custom spans for distributed tracing

Traces are formed of spans tracking service execution and data logged along the way. The APM libraries log spans for key application and language features out of the box, including most common frameworks and libraries.

However, it's quite simple to add custom span instrumentation in your favorite language, for instance:

import solarwinds-apm
def my_possibly_slow_method(...):

Primer: spans

All library SDK calls, whether exposed as annotation, decorator, or function, essentially create events that are the building blocks of a span, which in turn build a trace.

An event can be represented as a set of key-value pairs. This is displayed in the raw data when you click on a span in the Traces Explorer. Certain key-value pairs, such as those keyed by Host*, are automatically added during trace processing and should not be set in the SDK call.

Details: events, spans, and traces

An event is a set of key-value pairs that records a point in the execution flow of a request through the application. There are four types of events as defined by the Label key-value pair:

  • entry and exit are used to begin and end a span
  • info is used to log more information into a particular span
  • error is used to log errors/exceptions

A pair of entry and exit events form a span, which captures timing and other important data about a particular operation in the application.

A collection of linked spans form the trace, which shows the entire flow and performance of a request (aka transaction) as it traverses the components and distributed systems in the application.

Spans are usually linked automatically by the library's context tracking. There could be some edge cases where explicit context propagation is necessary by using SDK calls. Refer to each language library SDK for details.

Each trace has a root span which is not linked to any parent span, and represents both the entire transaction and its entry point into the application.

The SDK annotations, decorators, and higher level span-specific functions take care of creating the underlying events to form a span. However, when using lower level SDK calls that create a single event, you must ensure that the calls are made in the proper order and that they set the expected key-value pairs to generate the matching entry and exit required for a span. The rules are as follows:

  • The first SDK call marking entry into a new span must set Label to entry and Layer to a descriptive span name.
  • The final SDK call marking exit from the span must set Label to exit and Layer to the same span name as the entry event.
  • A span can have any number of intervening events (most commonly info and error) between the entry and exit. SolarWinds recommends that you explicitly set Layer to the same span name as the entry event; if unset, Layer defaults to the name of the current span.
  • A span can have nested sub-spans.

Special span types

Trace processing interprets the key-value pairs reported on a span, using the set of rules specified below, so that certain spans receive special treatment. Spans that do not fit these rules will still be processed but categorized as generic spans.

The key-value pairs can be set in any of the entry, info, or exit events of the span.

The required key-value pairs must be set for a span to be recognized as a particular type, while optional key-value pairs provide additional information that enhance SolarWinds Observability trace processing and analytics.


Cache represents the operations against key-value stores such as Memcached and Redis.

Key Name Value Type Required Description
Flavor string yes The cache type memcache, redis, etc.
KVOp string yes The type of operation get, set, etc.
RemoteHost string yes (if no RemoteHostPool) The hostname or IP of the remote host connected to for this operation.
RemoteHostPool array yes (if no RemoteHost) A list of the hostnames or IPs that are part of the pool being used.
Spec string yes Must be cache.
KVHit boolean no Whether the get operation was a hit/miss.
KVHitCount integer no For multiget operations, the number of hits from requested keys.
KVKey string no Key name being operated on. This should be a single value. Omit if there are multiple keys.
KVKeyCount integer no For multiget operations, the number of keys requested.

Controller action

MVC controller and action information can be associated to the transaction by adding these key-value pairs to the root span of the trace as one of the following:

  • An individual event with Label info and the required key-value pairs.
  • The required key-value pairs bundled into the entry or exit event of the root span.
Key Name Value Type Required Description
Action string yes Name of the action, truncated at 40 UTF-8 characters.
Controller string yes Name of the controller, truncated at 40 UTF-8 characters.


Operations against a variety of SQL and NoSQL databases are represented by the database span.

Key Name Value Type Required Description
Flavor string yes The type of database being accessed, for example, mongodb, oracle, postgresql.
Query string yes The query text. If longer than 2048 bytes it should be truncated.
RemoteHost string yes (if no RemoteHostPool) The hostname or IP of the remote host connected to for this operation.
RemoteHostPool array yes (if no RemoteHost) A list of the hostnames or IPs that are part of the pool being used.
Spec string yes Must be query.
Database string no The name of the database being used.
QueryTruncated boolean no Whether the Query has been truncated.
QueryArgs array no An array that contains the query parameters set for prepared statements. This should be a list-style array if possible, but can be a dictionary/map-style array if necessary based on argument parameterization.


Error/exception is not strictly a span, since it is a point-in-time occurrence that should be reported as either:

  • an individual event with Label error and the required key-value pairs, or
  • for compactness, the required key-value pairs bundled into the entry or exit event of the span in which the error occurred
Key Name Value Type Required Description
Label string yes (if outside of a span) Must be error.
ErrorClass string yes The type of error.
ErrorMsg string yes A more specific error message.
Layer string no Name of the span at the point of error/exception.

Remote service

This represents a call to a remote service via RPC/HTTP transport.

Key Name Value Type Required Description
IsService boolean yes Must be set to true.
RemoteURL string yes The requested URI.
Spec string yes Must be rsc.
HTTPMethod string no The HTTP method used such as GET, POST, etc.
HTTPStatus string no The HTTP status code returned.

Server/application entry point

This is the start, or root span, of each trace and represents the point where a transaction first enters the service. In a distributed architecture the trace may cross multiple services and thus have multiple entry point spans; this is possible because a unique trace context is created for the transaction upon entry into the first service. Whenever execution flow leaves the current process, for example via an RPC call, the trace context must be propagated in the call (in many cases as an X-Trace header), and the entry point of the downstream service must look for and continue an incoming trace context.

The APM libraries automatically instrument most HTTP-based entry and exit points, so you should seldom need to use the SDK to create the span types described below. If you do need to use the SDK to start a trace, note that there are specific calls provided by each language library such as .NET's Trace.StartTrace() and Trace.ContinueTrace() that should be used in order to propagate any incoming Trace ID.

Note also that each language library SDK provides a way to set a custom transaction name on the trace, which is automatically added to the root span as the TransactionName key-value pairs.

Web server

This represents an HTTP server entry point into the application.

Key Name Value Type Required Description
HTTP-Host string yes The domain name of the server, and (optionally) the port number on which the server is listening., e.g. Host header of the request,
HTTPMethod string yes The HTTP request method GET, POST, etc.
Spec string yes Must be ws.
Status integer yes The HTTP response status code, e.g. 200, 404, etc.
URL string yes The URL portion after domain name and port, typically the path to the resource and any query parameters, e.g. value of the REQUEST_URI server variable.
ClientIP string no Client IP address, e.g. value of the REMOTE_ADDR server variable.

Custom timers/counters for metric reporting and alerting

Custom metrics allow timing and counters/gauges for use in dashboarding and alerting:

long startTime = System.currentTimeMillis();

// some work here...

long duration = System.currentTimeMillis() - startTime;
Metrics.summaryMetric("my-work-duration", duration, tags);
Metrics.incrementMetric("my-work-count", tags);

Language SDKs

The APM library SDK provides support specifically tailored to the conventions of each language:

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.