Configure custom instrumentation
Custom instrumentation can be used to get extra application-specific visibility in SolarWinds Observability. In an APM-instrumented application, the language SDK methods provided by the library are a simple way to get additional visibility. The SDK methods can be used to log more details as spans on a traced request, and to report custom timer/counter metrics.
Custom spans for distributed tracing
Traces are formed of spans tracking service execution and data logged along the way. The APM libraries log spans for key application and language features out of the box, including most common frameworks and libraries.
However, custom span instrumentation can be added in the applicable language. For example:
import solarwinds-apm
@solarwinds-apm.log_method('slow_thing')
def my_possibly_slow_method(...):
...Primer: Spans
All library SDK calls, whether exposed as annotation, decorator, or function, essentially create events that are the building blocks of a span, which, in turn, build a trace.
An event can be represented as a set of key-value pairs. This is displayed in the raw data when you click on a span in the Traces Explorer. Certain key-value pairs, such as those keyed by Host*, are automatically added during trace processing and should not be set in the SDK call.
Details: Events, spans, and traces
An event is a set of key-value pairs that records a point in the execution flow of a request through the application. There are four types of events as defined by the Label key-value pair:
entryandexitare used to begin and end a spaninfois used to log more information into a particular spanerroris used to log errors/exceptions
A pair of entry and exit events form a span, which captures timing and other important data about a specific operation in the application.
A collection of linked spans form the trace, which shows the entire flow and performance of a request (aka transaction) as it traverses the components and distributed systems in the application.
Spans are usually linked automatically by the library's context tracking. There could be some edge cases where explicit context propagation is necessary by using SDK calls. Refer to each language library SDK for details.
Each trace has a root span, which is not linked to any parent span, and represents both the entire transaction and its entry point into the application.
The SDK annotations, decorators, and higher-level, span-specific functions create the underlying events to form a span. However, when using lower level SDK calls that create a single event, you must ensure that the calls are made in the proper order and that they set the expected key-value pairs to generate the matching entry and exit required for a span. The rules are as follows:
- The first SDK call marking entry into a new span must set Label to
entryand Layer to a descriptive span name. - The final SDK call marking exit from the span must set Label to
exitand Layer to the same span name as the entry event. - A span can have any number of intervening events (most commonly
infoanderror) between the entry and exit. SolarWinds recommends that you explicitly set Layer to the same span name as the entry event; if unset, Layer defaults to the name of the current span. - A span can have nested sub-spans.
Special span types
Trace processing interprets the key-value pairs reported on a span, using the set of rules specified below, so that certain spans receive special treatment. Spans that do not fit these rules will still be processed but categorized as generic spans.
The key-value pairs can be set in any of the entry, info, or exit events of the span.
The required key-value pairs must be set for a span to be recognized as a particular type, while optional key-value pairs provide additional information that enhance SolarWinds Observability trace processing and analytics.
Cache
Cache represents the operations against key-value stores such as Memcached and Redis.
| Key name | Value type | Required | Description |
|---|---|---|---|
| Flavor | string | yes | The cache type memcache, redis, etc. |
| KVOp | string | yes | The type of operation get, set, etc. |
| RemoteHost | string | yes (if no RemoteHostPool) | The hostname or IP of the remote host connected to for this operation. |
| RemoteHostPool | array | yes (if no RemoteHost) | A list of the hostnames or IPs that are part of the pool being used. |
| Spec | string | yes | Must be cache. |
| KVHit | boolean | no | Whether the get operation was a hit/miss. |
| KVHitCount | integer | no | For multiget operations, the number of hits from requested keys. |
| KVKey | string | no | Key name being operated on. This should be a single value. Omit if there are multiple keys. |
| KVKeyCount | integer | no | For multiget operations, the number of keys requested. |
Controller action
MVC controller and action information can be associated to the transaction by adding these key-value pairs to the root span of the trace as one of the following:
- An individual event with Label
infoand the required key-value pairs. - The required key-value pairs bundled into the entry or exit event of the root span.
| Key name | Value type | Required | Description |
|---|---|---|---|
| Action | string | yes | Name of the action, truncated at 40 UTF-8 characters. |
| Controller | string | yes | Name of the controller, truncated at 40 UTF-8 characters. |
Database
Operations against a variety of SQL and NoSQL databases are represented by the database span.
| Key name | Value type | Required | Description |
|---|---|---|---|
| Flavor | string | yes | The type of database being accessed, for example, mongodb, oracle, postgresql. |
| Query | string | yes | The query text. If longer than 2048 bytes it should be truncated. |
| RemoteHost | string | yes (if no RemoteHostPool) | The hostname or IP of the remote host connected to for this operation. |
| RemoteHostPool | array | yes (if no RemoteHost) | A list of the hostnames or IPs that are part of the pool being used. |
| Spec | string | yes | Must be query. |
| Database | string | no | The name of the database being used. |
| QueryTruncated | boolean | no | Whether the Query has been truncated. |
| QueryArgs | array | no | An array that contains the query parameters set for prepared statements. This should be a list-style array, if possible, but can be a dictionary/map-style array based on argument parameterization, if necessary. |
Error
Error/exception is not strictly a span, since it is a point-in-time occurrence that should be reported as either:
- An individual event with Label
errorand the required key-value pairs, or - For compactness, the required key-value pairs bundled into the entry or exit event of the span in which the error occurred
| Key name | Value type | Required | Description |
|---|---|---|---|
| Label | string | yes (if outside of a span) | Must be error. |
| ErrorClass | string | yes | The type of error. |
| ErrorMsg | string | yes | A more specific error message. |
| Layer | string | no | Name of the span at the point of error/exception. |
Remote service
This represents a call to a remote service via RPC/HTTP transport.
| Key name | Value type | Required | Description |
|---|---|---|---|
| IsService | boolean | yes | Must be set to true. |
| RemoteURL | string | yes | The requested URI. |
| Spec | string | yes | Must be rsc. |
| HTTPMethod | string | no | The HTTP method used such as GET, POST, etc. |
| HTTPStatus | string | no | The HTTP status code returned. |
Server/application entry point
This is the start, or root span, of each trace and represents the point where a transaction first enters the service. In a distributed architecture, the trace may cross multiple services and, therefore, may have multiple entry point spans; this is possible because a unique trace context is created for the transaction upon entry into the first service. Whenever execution flow leaves the current process, for example via an RPC call, the trace context must be propagated in the call (in many cases as an X-Trace header), and the entry point of the downstream service must look for and continue an incoming trace context.
The APM libraries automatically instrument most HTTP-based entry and exit points, so you should seldom need to use the SDK to create the span types described below. If you do need to use the SDK to start a trace, note that there are specific calls provided by each language library such as .NET's Trace.StartTrace() and Trace.ContinueTrace() that should be used in order to propagate any incoming Trace ID.
Note also that each language library SDK provides a way to set a custom transaction name on the trace, which is automatically added to the root span as the TransactionName key-value pairs.
Web server
This represents an HTTP server entry point into the application.
| Key name | Value type | Required | Description |
|---|---|---|---|
HTTP-Host
|
string | yes | The domain name of the server, and (optionally) the port number on which the server is listening., e.g. Host header of the request, www.solarwinds.com. |
HTTPMethod
|
string | yes | The HTTP request method GET, POST, etc. |
Spec
|
string | yes | Must be ws. |
Status
|
integer | yes | The HTTP response status code, e.g. 200, 404, etc. |
URL
|
string | yes | The URL portion after domain name and port, typically the path to the resource and any query parameters, e.g. value of the REQUEST_URI server variable. |
ClientIP
|
string | no | Client IP address, e.g. value of the REMOTE_ADDR server variable. |
Custom timers/counters for metric reporting and alerting
Custom metrics allow timing and counters/gauges for use in dashboarding and alerting:
long startTime = System.currentTimeMillis();
// some work here...
long duration = System.currentTimeMillis() - startTime;
Metrics.summaryMetric("my-work-duration", duration, tags);
Metrics.incrementMetric("my-work-count", tags);Language SDKs
The APM library SDK provides support specifically tailored to the conventions of each language:
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.