Documentation forAppOptics

Elasticsearch

Overview

This plugin collects metrics from the Elasticsearch API. The plugin handles internally calls to the following endpoints:

The oldest supported Elasticsearch version is 2.4 (last released in late 2016).

This plugin is currently available for x86_64 Linux and Windows platforms.

Setup

The elasticsearch plugin is included with the SolarWinds Snap Agent by default, please follow the directions below to enable it for a agent instance.

Prerequisites

This plugin requires that Elasticsearch server is running and its API is available from the host where the plugin runs. You might want to customize values in the ‘Network' section in elasticsearch configuration file (typically /etc/elasticsearch/elasticsearch.yml):

Copy 
...
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.0.100
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
...

Configuration

The agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent, but requires you to provide the correct settings for your Elasticsearch server. To enable the plugin:

1. Make a copy of the elasticsearch example configuration file /opt/SolarWinds/Snap/etc/plugins.d/elasticsearch.yaml.example, renaming it to /opt/SolarWinds/Snap/etc/plugins.d/elasticsearch.yaml:

Copy 
$ sudo cp /opt/SolarWinds/Snap/etc/plugins.d/elasticsearch.yaml.example /opt/SolarWinds/Snap/etc/plugins.d/elasticsearch.yaml
  1. Update /opt/SolarWinds/Snap/etc/plugins.d/elasticsearch.yaml configuration file with settings specific to your Elasticsearch server for example:
Copy 
  collector:
    elasticsearch:
      all:
        server: "127.0.0.1"
        port: 9200
        timeout: 5s
load:
  plugin: snap-plugin-collector-aoelasticsearch
  task: task-aoelasticsearch.yaml

The setting server is optional and defaults to localhost. It should be set to the valid Elasticsearch API server IP or FQDN. Accepted formats are:

  • IP

  • IP:Port

  • scheme://IP

  • scheme://IP:Port

  • scheme://credentials@IP

  • scheme://credentials@IP:Port

    The setting port is optional and defaults to 9200. It should match with the port Elasticsearch API is listening on.

    The setting timeout is optional and defaults to 5s. It is a timeout for connecting Elasticsearch API.

  1. Restart the agent:
Copy 
$ sudo service swisnapd restart
  1. Enable the Elasticsearch plugin in the AppOptics UI

On the Integrations Page you will see the Elasticsearch plugin available if the previous steps were successful. If you do not see the plugin, see Troubleshooting Linux.

Select the Elasticsearch plugin to open the configuration menu in the UI, and enable the plugin.

You should soon see the elasticsearch metrics reported to your dashboard.

Metrics and Tags

The table below lists each of the metrics gathered from the status endpoint.

Default Metrics

Namespace Description
elasticsearch.cluster.indices.shards_total The total number of all shards.
elasticsearch.cluster.initializing_shards The number of shards that are currently initializing.
elasticsearch.cluster.nodes.count.coordinating_only The total number of coordinating nodes in the cluster.
elasticsearch.cluster.nodes.count.data The total number of data nodes in the cluster.
elasticsearch.cluster.nodes.count.ingest The total number of ingest nodes in the cluster.
elasticsearch.cluster.nodes.count.master The total number of master nodes in the cluster.
elasticsearch.cluster.nodes.fs.disk_reads The total amount of data read from disk.
elasticsearch.cluster.nodes.fs.disk_writes The total amount of data written to disk.
elasticsearch.cluster.relocating_shards The number of shards that are currently relocating.
elasticsearch.cluster.status The elasticsearch cluster health status as a number (red = 1, yellow = 2, green = 3)
elasticsearch.cluster.unassigned_shards The number of shards that are not assigned to a node.
elasticsearch.nodes.fs.io.read_operations The total number of read operations.
elasticsearch.nodes.fs.io.write_operations The total number of write operations.
elasticsearch.nodes.indexing.stats.index_current The instantaneous indexing count.
elasticsearch.nodes.indexing.stats.index_failed The total count of failed indexing.
elasticsearch.nodes.indexing.stats.index_time The instantaneous indexing time (in ms).
elasticsearch.nodes.indexing.stats.index_total The incrementing indexing count.
elasticsearch.nodes.jvm.gc.collection.count Java GC stats, per node.
elasticsearch.nodes.jvm.gc.collection.time Java GC stats, per node.
elasticsearch.nodes.jvm.mem.heap_used_percent The percent value of JVM heap used.
elasticsearch.nodes.os.cpu.percent The percent value of CPU usage, per node
elasticsearch.nodes.search.stats.fetch_time The instantaneous search fetching time (in ms).
elasticsearch.nodes.search.stats.query_current The instantaneous searches count.
elasticsearch.nodes.search.stats.query_time The instantaneous search querying time (in ms).""
elasticsearch.nodes.search.stats.query_total The incrementing searches count.

Default Metric Tags

All Elasticsearch metrics are tagged with hostname. Instead of using this tag we recommend using the @host alias. If dynamic metrics are used (see exemplary task file provided with the plugin), then server name or hostname and port from elasticsearch will be added as a tag.

Optional Metrics

Optional metrics can be activated by editing the task yaml. For more information please read the SolarWinds Snap Agent configuration article.

Metrics group Description
indices Indices stats about size, document count, indexing and deletion times, search times, field cache size, merges and flushes
os Operating system stats, load average, mem, swap
process Process statistics, memory consumption, cpu usage, open file descriptors
jvm JVM stats, memory pool information, garbage collection, buffer pools, number of loaded/unloaded classes
thread_pool Statistics about each thread pool, including current size, queue and rejected tasks
fs File system information, data path, free disk space, read/write stats
http HTTP connection information
breaks Statistics about the field data circuit breaker
script Computing the grades stats based on a script

All available metrics list:

Namespace
elasticsearch.cluster.active_primary_shards
elasticsearch.cluster.active_shards
elasticsearch.cluster.active_shards_percent_as_number
elasticsearch.cluster.delayed_unassigned_shards
elasticsearch.cluster.indices.completion_size
elasticsearch.cluster.indices.count
elasticsearch.cluster.indices.docs_count
elasticsearch.cluster.indices.docs_deleted
elasticsearch.cluster.indices.field_evictions
elasticsearch.cluster.indices.field_size
elasticsearch.cluster.indices.filter_cache_evictions
elasticsearch.cluster.indices.filter_cache_size
elasticsearch.cluster.indices.id_cache_size
elasticsearch.cluster.indices.shards_primaries
elasticsearch.cluster.indices.shards_replication
elasticsearch.cluster.indices.shards_total
elasticsearch.cluster.indices.store_size
elasticsearch.cluster.initializing_shards
elasticsearch.cluster.nodes.count.coordinating_only
elasticsearch.cluster.nodes.count.data
elasticsearch.cluster.nodes.count.ingest
elasticsearch.cluster.nodes.count.master
elasticsearch.cluster.nodes.count.total
elasticsearch.cluster.nodes.fs.available
elasticsearch.cluster.nodes.fs.disk_io_op
elasticsearch.cluster.nodes.fs.disk_io_size
elasticsearch.cluster.nodes.fs.disk_read_size
elasticsearch.cluster.nodes.fs.disk_reads
elasticsearch.cluster.nodes.fs.disk_write_size
elasticsearch.cluster.nodes.fs.disk_writes
elasticsearch.cluster.nodes.fs.free
elasticsearch.cluster.nodes.fs.total
elasticsearch.cluster.nodes.jvm.max_uptime
elasticsearch.cluster.nodes.jvm.threads
elasticsearch.cluster.nodes.os.cpu.cache_size
elasticsearch.cluster.nodes.os.cpu.cache_size_in_bytes
elasticsearch.cluster.nodes.os.cpu.cores_per_socket
elasticsearch.cluster.nodes.os.cpu.count
elasticsearch.cluster.nodes.os.cpu.mhz
elasticsearch.cluster.nodes.os.cpu.total_cores
elasticsearch.cluster.nodes.os.cpu.total_sockets
elasticsearch.cluster.nodes.os.mem_total
elasticsearch.cluster.nodes.process_cpu_percent
elasticsearch.cluster.nodes.process_open_file_descriptors
elasticsearch.cluster.number_of_data_nodes
elasticsearch.cluster.number_of_in_flight_fetch
elasticsearch.cluster.number_of_nodes
elasticsearch.cluster.number_of_pending_tasks
elasticsearch.cluster.relocating_shards
elasticsearch.cluster.status
elasticsearch.cluster.task_max_waiting_in_queue_millis
elasticsearch.cluster.unassigned_shards
elasticsearch.nodes.breakers.estimated.estimated_size
elasticsearch.nodes.breakers.limit.limit_size
elasticsearch.nodes.breakers.overhead
elasticsearch.nodes.breakers.tripped
elasticsearch.nodes.completion.stats.fields.size
elasticsearch.nodes.completion.stats.size
elasticsearch.nodes.discovery.stats.committed
elasticsearch.nodes.discovery.stats.pending
elasticsearch.nodes.discovery.stats.total
elasticsearch.nodes.docs.stats.count
elasticsearch.nodes.docs.stats.deleted
elasticsearch.nodes.fields.stats.evictions
elasticsearch.nodes.fields.stats.fields.memory_size
elasticsearch.nodes.fields.stats.memory_size
elasticsearch.nodes.flush.stats.total
elasticsearch.nodes.flush.stats.total_time
elasticsearch.nodes.fs.available
elasticsearch.nodes.fs.free
elasticsearch.nodes.fs.io.operations
elasticsearch.nodes.fs.io.read_kilobytes
elasticsearch.nodes.fs.io.read_operations
elasticsearch.nodes.fs.io.write_kilobytes
elasticsearch.nodes.fs.io.write_operations
elasticsearch.nodes.fs.total
elasticsearch.nodes.get.stats.current
elasticsearch.nodes.get.stats.exists
elasticsearch.nodes.get.stats.exists_time
elasticsearch.nodes.get.stats.missing
elasticsearch.nodes.get.stats.missing_time
elasticsearch.nodes.get.stats.time
elasticsearch.nodes.get.stats.total
elasticsearch.nodes.http.current_open
elasticsearch.nodes.http.total_opened
elasticsearch.nodes.indexing.stats.delete_current
elasticsearch.nodes.indexing.stats.delete_time
elasticsearch.nodes.indexing.stats.delete_total
elasticsearch.nodes.indexing.stats.index_current
elasticsearch.nodes.indexing.stats.index_failed
elasticsearch.nodes.indexing.stats.index_time
elasticsearch.nodes.indexing.stats.index_total
elasticsearch.nodes.indexing.stats.noop_update_total
elasticsearch.nodes.ingest.stats.count
elasticsearch.nodes.ingest.stats.current
elasticsearch.nodes.ingest.stats.failed
elasticsearch.nodes.ingest.stats.time
elasticsearch.nodes.jvm.buffers.count
elasticsearch.nodes.jvm.buffers.total_capacity
elasticsearch.nodes.jvm.current_loaded_count
elasticsearch.nodes.jvm.gc.collection.count
elasticsearch.nodes.jvm.gc.collection.time
elasticsearch.nodes.jvm.mem.direct_max
elasticsearch.nodes.jvm.mem.heap_committed
elasticsearch.nodes.jvm.mem.heap_init
elasticsearch.nodes.jvm.mem.heap_max
elasticsearch.nodes.jvm.mem.heap_max
elasticsearch.nodes.jvm.mem.heap_used
elasticsearch.nodes.jvm.mem.heap_used_percent
elasticsearch.nodes.jvm.mem.non_heap_committed
elasticsearch.nodes.jvm.mem.non_heap_init
elasticsearch.nodes.jvm.mem.non_heap_max
elasticsearch.nodes.jvm.mem.non_heap_used
elasticsearch.nodes.jvm.peak_count
elasticsearch.nodes.jvm.threads_count
elasticsearch.nodes.jvm.total_loaded_count
elasticsearch.nodes.jvm.total_unloaded_count
elasticsearch.nodes.jvm.uptime
elasticsearch.nodes.merge.stats.current
elasticsearch.nodes.merge.stats.current_docs
elasticsearch.nodes.merge.stats.current_size
elasticsearch.nodes.merge.stats.total
elasticsearch.nodes.merge.stats.total_auto_throttle
elasticsearch.nodes.merge.stats.total_docs
elasticsearch.nodes.merge.stats.total_size
elasticsearch.nodes.merge.stats.total_stopped_time
elasticsearch.nodes.merge.stats.total_throttled_time
elasticsearch.nodes.merge.stats.total_time
elasticsearch.nodes.os.allocated_processors
elasticsearch.nodes.os.available_processors
elasticsearch.nodes.os.cpu.cpu.load_average
elasticsearch.nodes.os.cpu.percent
elasticsearch.nodes.os.mem.free
elasticsearch.nodes.os.mem.free_percent
elasticsearch.nodes.os.mem.total
elasticsearch.nodes.os.mem.used
elasticsearch.nodes.os.mem.used_percent
elasticsearch.nodes.os.refresh_interval
elasticsearch.nodes.os.swap.free
elasticsearch.nodes.os.swap.total
elasticsearch.nodes.os.swap.used
elasticsearch.nodes.percolate.stats.current
elasticsearch.nodes.percolate.stats.memory_size
elasticsearch.nodes.percolate.stats.queries
elasticsearch.nodes.percolate.stats.time
elasticsearch.nodes.percolate.stats.total
elasticsearch.nodes.process.cpu.percent
elasticsearch.nodes.process.cpu.total
elasticsearch.nodes.process.max_file_descriptors
elasticsearch.nodes.process.mem.total_virtual
elasticsearch.nodes.process.open_file_descriptors
elasticsearch.nodes.process.timestamp
elasticsearch.nodes.processes.id
elasticsearch.nodes.processes.refresh_interval
elasticsearch.nodes.query.cache.cache_count
elasticsearch.nodes.query.cache.cache_size
elasticsearch.nodes.query.cache.evictions
elasticsearch.nodes.query.cache.hit_count
elasticsearch.nodes.query.cache.memory_size
elasticsearch.nodes.query.cache.miss_count
elasticsearch.nodes.query.cache.total_count
elasticsearch.nodes.recovery.stats.current_as_source
elasticsearch.nodes.recovery.stats.current_as_target
elasticsearch.nodes.refresh.stats.total
elasticsearch.nodes.refresh.stats.total_time
elasticsearch.nodes.request.cache.evictions
elasticsearch.nodes.request.cache.hit_count
elasticsearch.nodes.request.cache.memory_size
elasticsearch.nodes.request.cache.miss_count
elasticsearch.nodes.script.stats.cache_evictions
elasticsearch.nodes.script.stats.compilations
elasticsearch.nodes.search.stats.fetch_current
elasticsearch.nodes.search.stats.fetch_time
elasticsearch.nodes.search.stats.fetch_total
elasticsearch.nodes.search.stats.open_contexts
elasticsearch.nodes.search.stats.query_current
elasticsearch.nodes.search.stats.query_time
elasticsearch.nodes.search.stats.query_total
elasticsearch.nodes.search.stats.scroll_current
elasticsearch.nodes.search.stats.scroll_time
elasticsearch.nodes.search.stats.scroll_total
elasticsearch.nodes.segments.stats.count
elasticsearch.nodes.segments.stats.doc_values_memory
elasticsearch.nodes.segments.stats.fixed_bit_set_memory
elasticsearch.nodes.segments.stats.index_writer_max_memory
elasticsearch.nodes.segments.stats.index_writer_memory
elasticsearch.nodes.segments.stats.memory
elasticsearch.nodes.segments.stats.norms_memory
elasticsearch.nodes.segments.stats.stored_fields_memory
elasticsearch.nodes.segments.stats.term_vectors_memory
elasticsearch.nodes.segments.stats.terms_memory
elasticsearch.nodes.segments.stats.version_map_memory
elasticsearch.nodes.store.stats.size
elasticsearch.nodes.suggest.stats.current
elasticsearch.nodes.suggest.stats.total
elasticsearch.nodes.suggest.stats.total_time
elasticsearch.nodes.threads.active
elasticsearch.nodes.threads.completed
elasticsearch.nodes.threads.count
elasticsearch.nodes.threads.largest
elasticsearch.nodes.threads.queue
elasticsearch.nodes.threads.rejected
elasticsearch.nodes.total_indexing_buffer
elasticsearch.nodes.translog.stats.operations
elasticsearch.nodes.translog.stats.size
elasticsearch.nodes.transport.rx_count
elasticsearch.nodes.transport.rx_size
elasticsearch.nodes.transport.server_open
elasticsearch.nodes.transport.tx_count
elasticsearch.nodes.transport.tx_size
elasticsearch.nodes.warmer.stats.current
elasticsearch.nodes.warmer.stats.total
elasticsearch.nodes.warmer.stats.total_time

Metrics changes

From ver 8 to ver 10 of the plugin:

Promoted to defaults:

  • /elasticsearch/cluster/nodes/count/coordinating_only
  • /elasticsearch/cluster/nodes/count/data
  • /elasticsearch/cluster/nodes/count/ingest
  • /elasticsearch/cluster/nodes/count/master
  • /elasticsearch/cluster/nodes/fs/disk_reads
  • /elasticsearch/cluster/nodes/fs/disk_writes
  • /elasticsearch/nodes/*/fs/io/read_operations
  • /elasticsearch/nodes/*/fs/io/write_operations
  • /elasticsearch/nodes/*/indexing/stats/index_current
  • /elasticsearch/nodes/*/indexing/stats/index_failed
  • /elasticsearch/nodes/*/indexing/stats/index_time
  • /elasticsearch/nodes/*/indexing/stats/index_total
  • /elasticsearch/nodes/*/jvm/gc/collection/*/count
  • /elasticsearch/nodes/*/jvm/gc/collection/*/time
  • /elasticsearch/nodes/*/jvm/mem/heap_used_percent
  • /elasticsearch/nodes/*/os/cpu/percent
  • /elasticsearch/nodes/*/search/stats/fetch_time
  • /elasticsearch/nodes/*/search/stats/query_current
  • /elasticsearch/nodes/*/search/stats/query_time
  • /elasticsearch/nodes/*/search/stats/query_total

No longer defaults:

  • /elasticsearch/cluster/active_primary_shards
  • /elasticsearch/cluster/active_shards
  • /elasticsearch/cluster/active_shards_percent_as_number
  • /elasticsearch/cluster/cluster_name
  • /elasticsearch/cluster/delayed_unassigned_shards
  • /elasticsearch/cluster/initializing_shards
  • /elasticsearch/cluster/number_of_data_nodes
  • /elasticsearch/cluster/number_of_in_flight_fetch
  • /elasticsearch/cluster/number_of_nodes
  • /elasticsearch/cluster/number_of_pending_tasks
  • /elasticsearch/cluster/task_max_waiting_in_queue_millis
  • /elasticsearch/cluster/unassigned_shards

Backwards incompatible renames of metrics:

  • /elasticsearch/cluster/nodes/fs/dev -> no longer supported
  • /elasticsearch/cluster/nodes/fs/dev -> no longer supported
  • /elasticsearch/cluster/nodes/fs/disk_queue -> no longer supported
  • /elasticsearch/cluster/nodes/fs/disk_queue -> no longer supported
  • /elasticsearch/cluster/nodes/fs/disk_service_time -> no longer supported
  • /elasticsearch/cluster/nodes/fs/disk_service_time -> no longer supported
  • /elasticsearch/cluster/nodes/fs/mount -> no longer supported
  • /elasticsearch/cluster/nodes/fs/path -> no longer supported
  • /elasticsearch/cluster/timed_out -> no longer supported
  • /elasticsearch/nodes/*/build_hash -> no longer supported
  • /elasticsearch/nodes/*/host -> no longer supported
  • /elasticsearch/nodes/*/ip -> no longer supported
  • /elasticsearch/nodes/*/jvm/jvm/mem/heap_committed -> /elasticsearch/nodes/*/jvm/mem/heap_committed
  • /elasticsearch/nodes/*/jvm/jvm/mem/heap_max -> /elasticsearch/nodes/*/jvm/mem/heap_max
  • /elasticsearch/nodes/*/jvm/jvm/mem/heap_used_percent -> /elasticsearch/nodes/*/jvm/mem/heap_used_percent
  • /elasticsearch/nodes/*/jvm/jvm/mem/heap_used -> /elasticsearch/nodes/*/jvm/mem/heap_used
  • /elasticsearch/nodes/*/jvm/jvm/mem/non_heap_committed -> /elasticsearch/nodes/*/jvm/mem/non_heap_committed
  • /elasticsearch/nodes/*/jvm/jvm/mem/non_heap_used -> /elasticsearch/nodes/*/jvm/mem/non_heap_used
  • /elasticsearch/nodes/*/jvm/jvm/uptime -> /elasticsearch/nodes/*/jvm/uptime
  • /elasticsearch/nodes/*/name -> no longer supported
  • /elasticsearch/nodes/*/os/arch -> no longer supported
  • /elasticsearch/nodes/*/os/name -> no longer supported
  • /elasticsearch/nodes/*/os/version -> no longer supported
  • /elasticsearch/nodes/*/threads/*/Count -> /elasticsearch/nodes/*/threads/*/count
  • /elasticsearch/nodes/*/transport_address -> no longer supported
  • /elasticsearch/nodes/*/version -> no longer supported
  • /elasticsearch/nodes/fs/io/read_operations -> /elasticsearch/nodes/*/fs/io/read_operations
  • /elasticsearch/nodes/fs/io/write_operations -> /elasticsearch/nodes/*/fs/io/write_operations
  • /elasticsearch/nodes/indexing/stats/index_current -> /elasticsearch/nodes/*/indexing/stats/index_current
  • /elasticsearch/nodes/indexing/stats/index_failed -> /elasticsearch/nodes/*/indexing/stats/index_failed
  • /elasticsearch/nodes/indexing/stats/index_total -> /elasticsearch/nodes/*/indexing/stats/index_total
  • /elasticsearch/nodes/jvm/gc/collection/count -> /elasticsearch/nodes/*/jvm/gc/collection/*/count
  • /elasticsearch/nodes/jvm/gc/collection/time -> /elasticsearch/nodes/*/jvm/gc/collection/*/time
  • /elasticsearch/nodes/jvm/jvm/mem/heap_used_percent -> /elasticsearch/nodes/*/jvm/mem/heap_used_percent
  • /elasticsearch/nodes/os/cpu/percent -> /elasticsearch/nodes/*/os/cpu/percent
  • /elasticsearch/nodes/search/stats/query_current -> /elasticsearch/nodes/*/search/stats/query_current
  • /elasticsearch/nodes/search/stats/query_total -> /elasticsearch/nodes/*/search/stats/query_total

When the APM Integrated Experience is enabled, AppOptics shares a common navigation and enhanced feature set with the other integrated experiences' products. How you navigate AppOptics and access its features may vary from these instructions. For more information, go to the APM Integrated Experience documentation.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.