Documentation forAppOptics

Docker

Overview

This plugin collects runtime metrics from Docker containers and its host machine. It gathers information about resource usage and performance characteristics.

This plugin is currently only available for x86_64 Linux.

Setup

The docker plugin is included with the SolarWinds Snap Agent by default, please follow the directions below to enable it for a agent instance.

Prerequisites

This plugin requires that the agent user solarwinds has access to the Docker API provided by the Docker daemon, which by default binds to a Unix socket owned by root and read/writable by the docker group. To give the agent access to the API, ensure the docker group exists and add the solarwinds user to the group:

Copy 
$ sudo groupadd docker
$ sudo usermod -aG docker solarwinds

You can get more information about this from the Docker post-install documentation.

Alternatively, this plugin can query the Docker API over HTTP if your Docker daemon is configured to listen on a TCP port. You can get more information about this from the Docker daemon configuration documentation.

To check that the agent has access to the Docker API, you can run the following command as user solarwinds which pulls a test image and runs it in a container:

Copy 
$ sudo -u solarwinds docker run hello-world

Once confirmed, the Unix or TCP socket should be given in the endpoint configuration setting (see below).

Configuration

The agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent, but requires you to provide the correct settings for your Docker installation. To enable the plugin:

  1. Make a copy of the docker example configuration file /opt/SolarWinds/Snap/etc/plugins.d/docker.yaml.example, renaming it to /opt/SolarWinds/Snap/etc/plugins.d/docker.yaml:
Copy 
$ sudo cp /opt/SolarWinds/Snap/etc/plugins.d/docker.yaml.example /opt/SolarWinds/Snap/etc/plugins.d/docker.yaml
  1. Update the /opt/SolarWinds/Snap/etc/plugins.d/docker.yaml configuration file with settings specific to your Docker install, for example:
Copy 
collector:
  docker:
    all:
     endpoint: "unix:///var/run/docker.sock"
     procfs: "/proc"
     interval: "60s"
     all_containers: false
load:
  plugin: snap-plugin-collector-aodocker
  task: task-aodocker.yaml

The key endpoint is a required setting and defaults to unix:///var/run/docker.sock. It should be set to the Docker API endpoint. If you want it to point to a specific host/port, set the value to sth like: tcp://docker:2375.

The key procfs is a required setting and defaults to /proc. It should be set to the proc filesystem path of the Docker host. Note that in order to collect certain Docker metrics, the agent user solarwinds would need read access to the corresponding proc filesystem data.

The key interval is optional and defaults to 60s. It sets time interval between subsequent metrics submissions and most probably you’ll never want to change that value.

The key all_containers is optional and defaults to false. If set to true then the collector would process metrics for all existing containers (not only running). Note that this may result in sending much more metrics, depending on your docker environment. For details please refer to: https://docs.docker.com/engine/api/v1.37/#operation/ContainerList (look for all parameter).

  1. Restart the agent:
Copy 
$ sudo service swisnapd restart
  1. Enable the Docker plugin in the AppOptics UI

On the Integrations Page you will see the Docker plugin available if the previous steps were successful. If you do not see the plugin, see Troubleshooting Linux.

Select the Docker plugin to open the configuration menu in the UI, and enable the plugin.

You should soon see the docker metrics reported to your dashboard.

Metrics and Tags

The tables below outline the default set of metrics collected by thei docker plugin along with the optional metrics available.

CPU and Memory Metrics

Namespace Description
docker.cpu.cpu_usage.total The total CPU time consumed
docker.cpu.cpu_usage.kernel_mode CPU time consumed by tasks in system (kernel) mode
docker.cpu.cpu_usage.user_mode CPU time consumed by tasks in user mode
docker.cpu.cpu_usage.usage_percent Percentage of the hosts's CPU the container is using
docker.cpu.throttling_data.nr_periods The number of period intervals that have elapsed
docker.cpu.throttling_data.nr_throttled The number of times tasks in a cgroup have been throttled
docker.cpu.throttling_data.throttled_time The total time duration for which tasks in a cgroup have been throttled
docker.memory.usage.max_usage The maximum memory used by processes in the cgroup
docker.memory.usage.usage_percent Percentage of the host's memory the container is using
docker.memory.statistics.active_anon The number of bytes of anon and swap cache memory on active LRU list
docker.memory.statistics.active_file The number of bytes of file-backed memory on active LRU list
docker.memory.statistics.cache The number of bytes of page cache memory
docker.memory.statistics.hierarchical_memory_limit The number of bytes of memory limit with regard to hierarchy under which the memory cgroup is
docker.memory.statistics.inactive_anon The number of bytes of anon and swap cache memory on inactive LRU list
docker.memory.statistics.inactive_file The number of bytes of file-backed memory on inactive LRU list
docker.memory.statistics.mapped_file The number of bytes of mapped file (includes tmpfs.shmem)
docker.memory.statistics.pgfault The number of page faults which happened since creation of cgroup
docker.memory.statistics.pgmajfault The number of page major faults which happened since creation of cgroup
docker.memory.statistics.pgpgin The number of charging events to the memory cgroup
docker.memory.statistics.pgpgout The number of uncharging events to the memory cgroup
docker.memory.statistics.rss The number of bytes of anonymous and swap cache memory
docker.memory.statistics.rss_huge The number of bytes of anonymous transparent hugepages

CPU and Memory Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container

Blkio Metrics

Namespace Description
docker.blkio.io_service_bytes_recursive Bytes transferred to/from the disk from all the descendant cgroups2
docker.blkio.io_service_time_recursive Total time between request dispatch and request completion for the I/O done from all the descendant cgroups2

Blkio Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container
major major number of the device
minor minor number of the device
operation Operation type

Network Metrics

Namespace Description
docker.network.<interface>.rx_bytes Number of bytes received over the network
docker.network.<interface>.rx_dropped Number of bytes dropped during receiving over the network
docker.network.<interface>.rx_errors Number of errors while receiving over the network
docker.network.<interface>.rx_packets Number of packets received over the network
docker.network.<interface>.tx_bytes Number of bytes sent over the network
docker.network.<interface>.tx_dropped Number of bytes dropped during sending over the network
docker.network.<interface>.tx_errors Number of packets sent over the network
docker.network.<interface>.tx_packets Number of errors while receiving over the network

Network Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container
network_interface Network interface

Container Metrics

Namespace Description
docker.container.count Number of containers total
docker.container.paused Number of containers paused
docker.container.running Number of containers running
docker.container.stopped Number of containers stopped

Container Metric Tags

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias

Optional Metrics

Optinal metrics can be activated by editing the task yaml. For more information please read the SolarWinds Snap Agent configuration article.

Connection, cpu, cpuset, memory, pids, and spec Metrics

Namespace Description
docker.connection.tcp.close The number of tcp connections in state "Close"
docker.connection.tcp.close_wait The number of tcp connections in state "Close_Wait"
docker.connection.tcp.closing The number of tcp connections in state "Closing"
docker.connection.tcp.established The number of tcp connections in state "Established"
docker.connection.tcp.fin_wait1 The number of tcp connections in state "Fin_Wait1"
docker.connection.tcp.fin_wait2 The number of tcp connections in state "Fin_Wait2"
docker.connection.tcp.last_ack The number of tcp connections in state "Listen_Ack"
docker.connection.tcp.syn_recv The number of tcp connections in state "Syn_Recv"
docker.connection.tcp.syn_sent The number of tcp connections in state "Syn_Sent"
docker.connection.tcp.time_wait The number of tcp connections in state "Time_Wait"
docker.connection.tcp6.close The number of tcp6 connections in state "Close"
docker.connection.tcp6.close_wait The number of tcp6 connections in state "Close_Wait"
docker.connection.tcp6.closing The number of tcp6 connections in state "Closing"
docker.connection.tcp6.established The number of tcp6 connections in state "Established"
docker.connection.tcp6.fin_wait1 The number of tcp6 connections in state "Fin_Wait1"
docker.connection.tcp6.fin_wait2 The number of tcp6 connections in state "Fin_Wait2"
docker.connection.tcp6.last_ack The number of tcp6 connections in state "Listen_Ack"
docker.connection.tcp6.syn_recv The number of tcp6 connections in state "Syn_Recv"
docker.connection.tcp6.syn_sent The number of tcp6 connections in state "Syn_Sent"
docker.connection.tcp6.time_wait The number of tcp6 connections in state "Time_Wait"
docker.cpu.cpu_shares The relative share of CPU time available to the tasks in a cgroup
docker.cpu.cpu_usage.per_cpu.<n>.value CPU time consumed on each nth CPU by all tasks 1
docker.cpuset.cpu_exclusive Flag (0 or 1) that specifies whether cpusets other than this oneand its parents and children can share the CPUs for this cpuset
docker.cpuset.cpus CPUs numbers that tasks in this cgroup are permitted to access
docker.cpuset.memory_exclusive Flag (0 or 1) that specifies whether other cpusets can share thememory nodes specified for the cpuset
docker.cpuset.memory_migrate Flag (0 or 1) that specifies whether a page in memory shouldmigrate to a new node if the values in cpuset.mems change
docker.cpuset.mems Memory nodes that tasks in this cgroup are permitted to access
docker.memory.cache Page cache including tmpfs
docker.memory.kernel_usage.failcnt The number of times the kernel memory allocation has reached thevalue set in kmem.limit_in_bytes
docker.memory.kernel_usage.max_usage The maximum kernel memory allocation by processes in the cgroup
docker.memory.kernel_usage.usage The total kernel memory allocation by processes in the cgroup
docker.memory.statistics.dirty The number of bytes that are waiting to get written back to the disk
docker.memory.statistics.hierarchical_memsw_limit The number of bytes of memory+swap limit with regard to hierarchyunder which memory cgroup is
docker.memory.statistics.swap The amount of swap currently used by the processes in this cgroup
docker.memory.statistics.total_active_anon The total bytes of anon and swap cache memory on active LRU list 2
docker.memory.statistics.total_active_file The total bytes of file-backed memory on active LRU list 2
docker.memory.statistics.total_cache The total bytes of page cache memory 2
docker.memory.statistics.total_dirty The total bytes that are waiting to get written back to the disk 2
docker.memory.statistics.total_inactive_anon The total bytes of anonymous and swap cache memory on inactive LRU list 2
docker.memory.statistics.total_inactive_file The total bytes of file-backed memory on inactve LRU list 2
docker.memory.statistics.total_mapped_file The total bytes of mapped file (includes tmpfs.shmem) 2
docker.memory.statistics.total_pgfault The total page faults which happened since the creation of the cgroup 2
docker.memory.statistics.total_pgmajfault The total page major faults which happened since the creation of the cgroup 2
docker.memory.statistics.total_pgpgin The total charging events to the memory cgroup 2
docker.memory.statistics.total_pgpgout The total uncharging events to the memory cgroup 2
docker.memory.statistics.total_rss The total bytes of anonymous and swap cache memory 2
docker.memory.statistics.total_rss_huge The total bytes of anonymous transparent hugepages 2
docker.memory.statistics.total_swap The total bytes of swap usage 2
docker.memory.statistics.total_unevictable The total bytes of memory that cannot be reclaimed 2
docker.memory.statistics.total_writeback The total bytes of file.anon cache that are queued for syncing to disk 2
docker.memory.statistics.unevictable The number of bytes of memory that cannot be reclaimed
docker.memory.statistics.working_set The number of bytes of memory being used and not dropped by the kernel
docker.memory.statistics.writeback The number of bytes of file.anon cache that are queued for syncing to disk
docker.memory.swap_usage.failcnt The number of times the swap space limit has reached the valueset in memorysw.limit_in_bytes
docker.memory.swap_usage.max_usage The maximum swap space used by processes in the cgroup
docker.memory.swap_usage.usage The total swap space usage by processes in the cgroup
docker.memory.usage.failcnt The number of times that the memory limit has reached thevalue set in memory.limit_in_bytes
docker.memory.usage.usage Total current memory usage by processes in the cgroup
docker.pids.current The current number of PID in the cgroup
docker.pids.limit The maximum number of PIDs in the cgroup
docker.spec.creation_time The time when container started
docker.spec.size_root_fs The total size (bytes) of all files in container
docker.spec.size_rw The size of the files created or changed from container base image.After container creation this should increase as files are created/modified.

Connection, cpu, cpuset, memory, pids, and spec Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container

1 docker.cpu.cpu_usage.per_cpu.* metrics also have the cpu_id tag

Hugetlb Metrics

Namespace Description
docker.hugetlb.<size>.failcnt The number of allocation failure due to HugeTLB limit
docker.hugetlb.<size>.max_usage Max "hugepagesize" hugetlb usage recorded
docker.hugetlb.<size>.usage Current usage for "hugepagesize"

HugeTlb Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container
size HugeTlb page size

Filesystem Metrics

Namespace Description
docker.filesystem.<device_name>.available The number of bytes available for non-root user
docker.filesystem.<device_name>.base_usage The base usage consumed by the container's writable layer
docker.filesystem.<device_name>.capacity The number of bytes consumable by container
docker.filesystem.<device_name>.device_name The block device name associated with the filesystem
docker.filesystem.<device_name>.inodes_free The number of available Inodes
docker.filesystem.<device_name>.io_in_progress The number of I/Os currently in progress
docker.filesystem.<device_name>.io_time The number of milliseconds spent doing I/Os
docker.filesystem.<device_name>.read_time The total number of milliseconds spent reading
docker.filesystem.<device_name>.reads_completed The total number of reads completed successfully
docker.filesystem.<device_name>.reads_merged The total number of reads merged successfully
docker.filesystem.<device_name>.sectors_read The total number of sectors read successfully
docker.filesystem.<device_name>.sectors_written The total number of sectors written successfully
docker.filesystem.<device_name>.type Type of the filesystem (e.g. vfs)
docker.filesystem.<device_name>.usage The number of bytes consumed by container
docker.filesystem.<device_name>.weighted_io_time The weighted number of milliseconds spent doing I/Os
docker.filesystem.<device_name>.writes_completed The total number of writes completed successfully
docker.filesystem.<device_name>.writes_merged The total number of writes merged successfully

Filesystem Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container
device_name Device name

Blkio Metrics

Namespace Description
docker.blkio.io_merged_recursive Total number of blkio requests merged into requests belongingfrom all the descendant cgroups2
docker.blkio.io_queue_recursive Total number of requests queued up from all the descendant cgroups 3
docker.blkio.io_serviced_recursive Number of I/Os (bio) issued to the disk from all the descendant cgroups 3
docker.blkio.io_time_recursive Disk time allocated to cgroup per device in milliseconds from descendantcgroups 3
docker.blkio.io_wait_time_recursive Total time the I/Os for cgroup spent waiting in scheduler queues forservice from all the descendant cgroups2
docker.blkio.sectors_recursive Number of sectors transferred to/from disk from all descendant group2

Blkio Metric Tags

Tag Name Description
creation_time The time when the container was started
docker_id The first 12 characters of the Docker container ID
full_id Docker container ID
hostname Name of the host. Instead of using this tag we recommend using the @host alias
image_name The name of docker image that the container has been created of
status The status of docker container
device_name Device name
major major number of the device
minor minor number of the device
operation Operation type

2 Hierarchical version of cgroups counter which in addition to the cgroup’s own value includes the sum of all hierarchical children’s values

3 Each blkio statistic exposes the read, write, sync, async and total values through the operation tag


See also

Read more about cgroups in Kernel documentation.

When the APM Integrated Experience is enabled, AppOptics shares a common navigation and enhanced feature set with the other integrated experiences' products. How you navigate AppOptics and access its features may vary from these instructions. For more information, go to the APM Integrated Experience documentation.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.