Documentation forAppOptics

Configuration (legacy agent)

The following content pertains to configuration for the legacy AppOptics Java Agent.

AppOptics agents are no long receiving updates. The new SolarWinds Observability libraries are regularly updated with new features and improvements. If you are still relying on the AppOptics agents and your components are supported by the new libraries, consider migrating to SolarWinds Observability.

If you have already transitioned to the new SolarWinds Observability Java Library, see the SolarWinds Java Library documentation for configuration information.

SolarWinds Observability libraries are not compatible with AppOptics agents. Do not use a mix of SolarWinds Observability libraries and AppOptics agents to instrument applications that are part of a distributed trace.

How to set configuration

All configuration options can be set as properties in the javaagent.json file. By default, this file should be placed in the same folder as the Java Agent jar appoptics-agent.jar.

Certain configurations can also be overridden via:

  • javaagent option in Java command. For example

    -javaagent:/usr/local/appoptics/appoptics-agent.jar=config=/path/to/file

  • System environment variable

If the same configuration is set in more than one of the above places (for example, service key in both System Environment Variable APPOPTICS_SERVICE_KEY and Property agent.serviceKey in javaagent.json), then the setting is applied with this precedence:

System Environment Variable > javaagent option > Property in "javaagent.json"

If the javaagent.json file is absent, Java Agent will just use default values for all configuration options. It is recommended to place all configuration in javaagent.json, the json file is also available at https://files.appoptics.com/java/latest/javaagent.json

Common Configuration Tasks

Complete list of configuration options

The following is the complete list of instrumentation configuration options:

Service Key

Service key used to identify your account and the service being instrumented. It should be in the form of <api token>:<service name>

  • Config File Property: agent.serviceKey
  • javaagent option: service_key
  • System Environment Variable: APPOPTICS_SERVICE_KEY
  • Required: Yes
  • Example: "agent.serviceKey":"0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service" (json) or javaagent:/usr/local/appoptics/appoptics-agent.jar=service_key=0123456789abcde0123456789abcde0123456789abcde0123456789abcde1234:my-service (jvm args)

Hide Parameters

By default URL query args are reported for both incoming URLs and outgoing calls to remote services. Set agent.hideParams and instrumentation will stop collecting and reporting query args from URLs.

  • Config File Property: agent.hideParams
  • Required: No
  • Possible values: Value is an array of module names. Valid module names are:
    • ALL: hide URL query args for all instrumented modules
    • SERVLET: http servlets/filters
    • NETTY: netty server
    • APACHE_HTTP: apache http client
    • WEB_SERVICE: soap/rest clients
    • JBOSS: jboss ejb client/server
    • URL_CONNECTION: HttpURLConnection client
    • AKKA_ACTOR: Akka actor Messages
    • SPRAY_CLIENT: Spray client
    • UNDERTOW: Undertow server
    • AKKA_HTTP: Akka http client/server
    • JETTY_HTTP_CLIENT: Jetty http client
  • Default: Disabled. Do not hide query parameters.
  • Example: "agent.hideParams":["SERVLET", "APACHE_HTTP"]

Logging Level

Simple configuration for seting the logging level. For more advanced options such as streaming logs to a separate file, please refer to Detailed Logging Configuration

  • Config File Property: agent.logging
  • javaagent option: logging
  • Required: Yes
  • Possible values : Valid log levels in order of descending severity are ‘fatal', 'error', ‘warn', ‘info', ‘debug', and ‘trace', and ‘off'.
  • Default: info
  • Example:"agent.logging":"info"

Detailed Logging Configuration

By default our agent only prints logs to the stderr and stdout streams which most application servers redirect to their own log files. Agent logs can also be redirected to a separate log file using configuration, below are the detailed logging configuration options for our agent:

  • level logging level, same as the string value configuration in Logging Level but in json node format. This is a mandatory parameter and by default is "info".

  • stdout either with value "enabled" or "disabled". This determines whether agent info level (or below) logs will be printed to stdout stream or not. This is an optional parameter and by default is "enabled". Take note that this is only available in the json config file and can only be "disabled" if file node is defined.

  • stderr either with value "enabled" or "disabled". This determines whether agent warning level (or above) logs will be printed to stderr stream or not. This is an optional parameter and by default is "enabled". Take note that this is only available in the json config file and can only be "disabled" if file node is defined.

  • file contains various parameters (see the child nodes below) for logging to a separate file. This is an optional parameter and by default is not defined, hence no logs are written to a separate file.

    • location the location of the log file. If a relative path is provided, it is relative to the same folder as the agent jar. Note that folders in the path need to already exist and please ensure the application user has sufficient permission to write to the location. This is a mandatory parameter if file node is defined.
    • maxSize the max size (in megabytes) of the current log file before rolling to a backup. This is an optional parameter and by default is 10MB.
    • maxBackup the max number of backup files. If this limit is reached, then the oldest backup file will be discarded next time. This is an optional parameter and by default is 5.

  • Example:

    • javaagent.json

      "agent.logging": {
  "level": "info",
  "stderr": "enabled",
  "stdout": "enabled",
  "file": {
    "location": "appoptics.log",
    "maxSize": 10,
    "maxBackup": 5
  }
}
    • javaagent option javaagent:/usr/local/appoptics/appoptics-agent.jar=log_file=appoptics.log
    • System environment variable APPOPTICS_JAVA_LOG_FILE: export APPOPTICS_JAVA_LOG_FILE=appoptics.log (Linux) or $env:APPOPTICS_JAVA_LOG_FILE="appoptics.log" (Windows PowerShell)

Trace Context in Logs

Trace context as an String identifier can be added to logs to correlate log entries to traces. See Trace Context in Logs on how this feature is supported in the Java agent.

There are 2 configurable options:

  • Automatically create a Trace ID attribtue in the MDC/Thread Context of the logging framework with key ao.traceId, such a value can then be referenced in the log layout configuration. This feature is supported for Log4j 1.2, Log4j2 and Logback/Slf4j and controlled by config scope mdc
  • Automatically insert Trace ID into log messages with structured layout. This feature is supported for Log4j2 structured layouts such as JSON and YAML and is controlled by config scope autoInsert

Take note that if the code flow is not traced (for example, not a servlet) then there is no valid context and no action will be taken by the agent.

Both of the above options are disabled by default and can be enabled by agent.logging.traceId.

  • Config File Property: agent.logging.traceId with 2 scopes mdc and autoInsert. See example for details.
  • Required: No
  • Possible values:
    • disabled: no action for the scope
    • enabled: always perform the Trace ID action for the scope
    • sampledOnly: only perform the Trace ID action if the related request is sampled for detailed tracing
  • Default: Not present. All Trace ID actions are disabled for logs
  • Example:
"agent.logging.traceId": {
  "mdc": "sampledOnly",
  "autoInsert": "sampledOnly"
}

Config File location

Specify the location location of the Java Agent configuration file.

  • javaagent option: config
  • Required: No
  • Default: /javaagent.json
  • Example: javaagent:/usr/local/appoptics/appoptics-agent.jar=config=<config file custom location>

Disable Java Agent

Completely disable the Java APM Agent via configuration.

  • Config File Property: agent.enabled
  • System Environment Variable: APPOPTICS_AGENT_ENABLED
  • Required: No
  • Default: Not present. Agent is enabled
  • Example: "agent.enabled":false (json), export APPOPTICS_AGENT_ENABLED=false (Linux) or $env:APPOPTICS_AGENT_ENABLED="false" (Windows PowerShell)

Tracing Mode (Disable tracing)

Change the tracing mode to disable tracing for this service. Requests will no longer be monitored to create any traces or metrics.

However, X-Trace context will still be injected in the http request/response headers and JMX metrics will still be reported.

To completely disable the agent, either remove the agent or use the Disable Java Agent config.

  • Config File Property: agent.tracingMode
  • javaagent option: tracing_mode
  • Required: No
  • Possible values:
    • disabled: never continue existing traces or start new ones, and never report metrics for requests.
    • enabled: create/continue traces subject to sampling and rate limits, and always report metrics for requests. This is the default tracing mode.
  • Default: Not present. Tracing is enabled
  • Example: "agent.tracingMode":"disabled"

Tracing Mode per Transaction (Disable tracing on particular requests)

Change the tracing mode on particular requests based on the matching criteria. Matching requests will either have tracing enabled or disabled depending on the tracing property. Requests with "tracing" : "disabled" will no longer be monitored to create any traces or metrics, while "tracing" : "enabled" can be used to achieve white-listing of requests when tracing is disabled at the service level.

  • Config File Property: agent.transactionSettings
  • Config value should be an array of json entry and each entry should contain:
    • A matcher (required) - regular expression or list of extensions, where the key is either:

      • regex: value should be a java regular expression. Since the pattern is defined in json, double backslashes are needed for java regex escapes, one for java string so it retains a backslash in order to escape whatever follows (for example .) in the regex pattern.

        For web (http or https) requests, the value is matched against the full request URL, including the protocol. For example, (http|https)://www\\.importantdomain\\.com.* which would match all requests to www.importantdomain.com.

        For non-web requests, the value is matched against the resource name of the corresponding framework:

        • RabbitMQ Consumer : Exchange Name
        • Kafka Consumer : Topic
        • JMS Consumer : Queue name
        • Quartz Job : Job key (which is <Job Group>.<Job Name>)
        • Spring-batch Job : Job name
        • Spring-scheduling: <Job class>.<Job method>
      • extensions: value should be a list of extension (such as css, jpg, etc.)
    • tracing (required) - value should either be enabled or disabled

    • type (deprecated; required for version 6.19.1 or older) - value should be url, when used in version 6.20.0 or above, such property will be ignored

    • If a request matches more than one entry in the array, then the first matching entry will be taken
  • Required: No
  • Default: Not present. No transaction specific tracing mode
  • Example: 
    "agent.transactionSettings": [
  {
    "extensions" : ["png", "jpg"],
    "tracing" : "disabled"
  },
  {
    "regex" : ".*static.*",
    "tracing" : "disabled"
  }
]

Override default entry span name

Override the default span name, which is the application server, e.g. tomcat, jetty, jboss, etc, detected at start up.

  • Config File Property: agent.layer
  • javaagent option: layer
  • Required: No
  • Example: "agent.layer":"MyLayer" (json) or javaagent:/usr/local/appoptics/appoptics-agent.jar=layer=MyLayer (jvm args)

Instrument all JDBC drivers

If ‘false' the agent will instrument on sql elements of only the most popular packages, e.g., mysql, postgres, oracle, DB2. If ‘true' it will instrument sql elements on all packages that extend java.sql classes.

  • Config File Property: agent.jdbcInstAll
  • javaagent option: jdbc_inst_all
  • Required: Yes
  • Default: false
  • Example: "agent.jdbcInstAll":true (json) or javaagent:/usr/local/appoptics/appoptics-agent.jar=jdbc_inst_all=true (jvm args)
  • Notes: Setting this parameter to ‘true', could introduce noise into your trace data.

Sanitize SQL query

Sanitize the SQL query reported. Take note that for value that is greater than 0 (ie sanization enabled), then query parameters in PreparedStatement will NOT be reported

  • Config File Property: agent.sqlSanitize
  • Required: No
  • Possible values:
    • 0: disable sanitizing
    • 1: remove literals auto-detecting literal quotes, disable SQL parameters reporting
    • 2: remove literals dropping doubled quoted values, disable SQL parameters reporting
    • 4: remove literals keeping double quoted values, disable SQL parameter reporting
  • Default: 1
  • Example: "agent.sqlSanitize":1

Sanitize MongoDB query

Sanitize the MongoDB query reported, values of the query would be replaced by ?. This feature is enabled by default.

  • Config File Property: agent.mongoSanitize
  • javaagent option: mongo_sanitize
  • System Environment Variable : APPOPTICS_MONGO_SANITIZE
  • Required: No
  • Default: true
  • Example:

    • javaagent.json:

      “agent.mongoSanitize” : false

    • javaagent option:

      javaagent:/usr/local/appoptics/appoptics-agent.jar=mongo_sanitize=false

    • System environment variable:

      export APPOPTICS_MONGO_SANITIZE=false (Linux) or $env:APPOPTICS_MONGO_SANITIZE="false” (Windows PowerShell)

Enable/Disable Kafka trace context propagation

By default we disable Kafka trace context propagation by v2 message headers as it is known to have issues with older Kafka clients (v 0.10 or older).

You can use this config option to enable correlation of trace context between producer and consumer if both of them are running v 0.11 or newer.

  • Config File Property: agent.kafkaPropagation
  • javaagent option: kafka_propagation
  • System Environment Variable : APPOPTICS_KAFKA_PROPAGATION
  • Required: No
  • Default: false
  • Example:

    • javaagent.json:

      “agent.kafkaPropagation” : true

    • javaagent option:

      javaagent:/usr/local/appoptics/appoptics-agent.jar=kafka_propagation=true

    • System environment variable:

      export APPOPTICS_KAFKA_PROPAGATION=true (Linux) or $env:APPOPTICS_KAFKA_PROPAGATION="true” (Windows PowerShell)

Enable/Disable backtraces

An optional parameter to explicitly control backtrace reporting.

This parameter is not defined by default, which the agent will report backtraces on a pre-defined set of modules - more common and less noisy modules (for example JDBC query).

However, extra backtraces can be enabled by setting this to ALL - take note that this might adversely impact performance as backtrace reporting is usually costly.

To disable all backtraces, set this parameter to have empty array as value. For details, please refer to the examples below.

  • Config File Property: agent.backtraceModules
  • Required: No
  • Example:
    • parameter not defined - enable backtrace reporting on default modules
    • "agent.backtraceModules":[] - empty array, disable all backtrace reporting
    • "agent.backtraceModules":["ALL"] - enable backtrace reporting on all modules, this might impose high overhead especially for cache instrumentation
    • "agent.backtraceModules":["JDBC", "APACHE_HTTP"] - enable backtrace reporting for JDBC and Apache Http Client only

Set Hostname Alias

An optional logical/readable hostname that can be used to easily identify the host.

  • Config File Property: agent.hostnameAlias
  • javaagent option: hostname_alias
  • Required: No
  • Default: Not present
  • Example: "agent.hostnameAlias":"my-service-host" (json) or javaagent:/usr/local/appoptics/appoptics-agent.jar=hostname_alias=my-service-host (jvm args)

Prepend Domain to Transaction Name

An optional boolean parameter to prepend domain to transaction name if set to true

  • Config File Property: transaction.prependDomain
  • Required: No
  • Default: Not present
  • Example: "transaction.prependDomain":true

Enable/Disable hbase scanner next tracing

Instrument Hbase ‘scanner_next'. When the agent.hbaseScannerNext flag is not present or set to ‘false', Hbase ‘scanner_next. operations will not be traced.

  • Config File Property: agent.hbaseScannerNext
  • Required: No
  • Default: Not present
  • Example: "agent.hbaseScannerNext":false

Akka Actor tracing

A list of akka actor path names to be instrumented. You may use complete names or a regular expression in java pattern format. The list also applies to the path names of routers; if a router path name matches, all of its child actors will be monitored.

  • Config File Property: agent.akkaActors
  • Required: No
  • Default: Not present; no akka actor interactions reported.

  • Example: In this example, the actor with suffix path my-app/user/master, any actors with the path prefix my-app/user/worker-, and any child actors spawned by the router with suffix path name my-app/user/router will be instrumented.

    "agent.akkaActors":[".*/my-app/user/master", ".*/my-app/user/worker-.*", ".*/my-app/user/router"]

We recommend against using wildcards with generic scope, such as .*, which matches all actors. There could be a large number of system actors running in the background, and capturing interactions on all of them might impose noticeable overhead.

JMX Metrics

Our agent by default reports several core JVM health metrics via JMX MBeans. However the list of JMX MBeans can be customized for each java application to export more specific metrics.

There are 3 configuration properties for JMX:

  • monitor.jmx.enable takes a boolean value to enable or disable agent JMX reporting.

  • monitor.jmx.scopes takes an array of JMX MBean/attribute entries to locate metrics. Each entry consists of an object name and an attribute list in the form of <Object Name>:[<Attribute 1>, <Attribute2>...].

    Object Name

    This refers to the pattern for JMX MBean look-up. A single object name can match multiple MBeans. The following example matches all MBeans exposed by domain "Catalina" with type "Manager":

    "Catalina:type=Manager,*"

    More information on Object Name can be found in Java Management Extensions (JMX) - Best Practices - Oracle.

    Attribute List

    Attributes of each JMX MBean with matching Object Name will be filtered by this list and only the matching attributes will be reported. For example, this reports attribute "HeapMemoryUsage" and "NonHeapMemoryUsage" from the Memory MBean:

    "java.lang:type=Memory":["HeapMemoryUsage", "NonHeapMemoryUsage"]

    Wildcard can be used to report all attributes, however this is NOT recommended due to possibly high number of metric entries.

    A high value might impose a higher cost if wildcards are used. The use of wildcards is not recommended.

    "java.lang:type=Memory":"*"

  • monitor.jmx.maxEntry takes an integer value as a cap of JMX metrics reported per cycle. This is optional and default is 100 entries.

Enable/Disable JMX Metrics

  • Config File Property: monitor.jmx.enable
  • Required: Yes
  • Default: true
  • Example: "monitor.jmx.enable":true

JMX Metrics collection scope

  • Config File Property: monitor.jmx.scopes
  • Required: Yes
  • Default:

{
  "java.lang:type=MemoryPool,":["Usage"],
  "java.lang:type=Memory":["HeapMemoryUsage","NonHeapMemoryUsage"],
  "java.lang:type=GarbageCollector,":["CollectionTime"],
  "java.lang:type=Threading":["ThreadCount"],
  "java.lang:type=OperatingSystem":["ProcessCpuTime","AvailableProcessors","ProcessCpuLoad"],
  "java.lang:type=Runtime,*":["Uptime"]
}

Enable/Disable Code Profiling

Take note that our current code profiling is only applicable to Servlet processing and SDK initiated traces.

There are several configuration properties for profiling:

  • enabled takes a boolean value to enable or disable code profiling, default is "false". Can be set in the javaagent.json config file or via environment variable.
  • interval interval in milliseconds which determines how often the profiler obtains method execution information from the JVM. Shorter interval means more frequent sampling which could lead to extra overhead. Default interval is 20ms and the minimum is 10ms. Can be set in the javaagent.json config file or via environment variable.
  • excludePackages a list of java packages which the code profiler excludes from its data reporting. Classes with package names starting with any entry in the list will be omitted. By default various java basic method calls are excluded. In order to show all operations (ie. no exclusion), use an empty array as value.

Code Profiling Config

  • Config File Property: profiler
  • System Environment Variable: APPOPTICS_PROFILER_ENABLED, APPOPTICS_PROFILER_INTERVAL
  • Required: No
  • Default: Code profiling disabled

Example:

  • javaagent.json

    "profiler":
{
  "enabled" : true,
  "excludePackages" : ["java", "javax", "com.sun", "sun", "sunw"],
  "interval" : 20
}

  • System environment variable APPOPTICS_PROFILER_ENABLED : export APPOPTICS_PROFILER_ENABLED=true (Linux) or $env:APPOPTICS_PROFILER_ENABLED="true" (Windows PowerShell)

  • System enviromnent variable APPOPTICS_PROFILER_INTERVAL : export APPOPTICS_PROFILER_INTERVAL=100 (Linux) or $env:APPOPTICS_PROFILER_INTERVAL="100" (Windows PowerShell)

Proxy

Traffic from the agent to the AppOptics backend (aka the collector) can be configured to go through a proxy server using this option.

This feature requires Java 8u252 or newer.

The config value should either be http://<proxyHost>:<proxyPort> for a proxy server that does not require authentication, or http://<username>:<password>@<proxyHost>:<proxyPort> for a proxy server that requires Basic authentication.

Take note that while HTTP is the only type of connection supported between the agent and proxy (hence only http:// is allowed in the schema of the config value), the agent traffic to AppOptics is still encrypted using SSL/TLS.

It is encouraged to define this agent proxy config even if the process level jvm arguments https.proxyHost and https.proxyPort are defined, as those arguments would only work for proxy server that requires no authentication.

Agent traffic cannot be authenticated by java.net.Authenticator#setDefault. For proxy server that requires authentication, this agent proxy config has to be used.

  • Config File Property: agent.proxy
  • javaagent option: proxy
  • System Environment Variable: APPOPTICS_PROXY
  • Required: No
  • Default: Not present; no proxy
  • Example: "agent.proxy":"http://my-proxy-user:password123@my-proxy-server:8080" (json) or javaagent:/usr/local/appoptics/appoptics-agent.jar=proxy=http://my-proxy-user:password123@my-proxy-server:8080 (jvm args)

Disable/Set timeout on EC2 instance check

Disable or adjust the timeout (in millisecond) on EC2 instance check carried out at agent startup. This can be used to increase the timeout for apps running on AWS EC2 or OpenStack instances, if the agent check consistently fails due to timeout. It is safe to set this value to 0 for apps that don't run on EC2/OpenStack.

Valid range is from 0 to 3000(inclusive). Value 0 disables the check. Value outside of the range will be adjusted to the closest valid value.

By default the check is enabled with value 1000 (1 second timeout)

  • Config File Property: agent.ec2MetadataTimeout
  • System Environment Variable: APPOPTICS_EC2_METADATA_TIMEOUT
  • Required: No
  • Default: 1000
  • Example: "agent.ec2MetadataTimeout":0

Enable/Disable Trigger Trace

By default the Trigger Trace feature is enabled, you might disable it by setting this config property.

  • Config File Property: agent.triggerTrace
  • System Environment Variable: APPOPTICS_TRIGGER_TRACE
  • Required: No
  • Default: Not present; Trigger Trace feature is enabled
  • Example: "agent.triggerTrace":"disabled"

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.