Prometheus

Overview

This integration collects metrics from from HTTP servers exposing metrics in Prometheus format.

Setup

The prometheus monitoring is accomplished by bridge plugin which is included with the SolarWinds Snap Agent by default. Follow the directions below to enable it for an agent instance. The bridge plugin utilize Telegraf Prometheus plugin.

Configuration

The agent provides an example task file to help you get started quickly, but requires you to provide the correct settings for your HTTP server(s) exposing metrics in Prometheus format. To enable the task:

1. Make a copy of the prometheus example task file task-bridge-prometheus.yaml.example, renaming it to task-bridge-prometheus.yaml:

   On Windows, using Explorer or PowerShell:

   copy "C:\ProgramData\SolarWinds\Snap\tasks-autoload.d\task-bridge-prometheus.yaml.example" "C:\ProgramData\SolarWinds\Snap\tasks-autoload.d\task-bridge-prometheus.yaml"

   On Linux using command line:

   sudo cp -p /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-prometheus.yaml.example /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-prometheus.yaml

2. Edit the task file with settings specific to your HTTP server(s) exposing metrics in Prometheus format, for example:

   ---
   version: 2
   schedule:
     type: cron
     interval: "0 * * * * *"
   plugins:
     - plugin_name: bridge
       config:
         prometheus:
           ## A list of urls to scrape metrics from.
           urls:
             - http://localhost:9100/metrics
             - http://otherhost:9100/metrics
           ## A list of Kubernetes services to scrape metrics from.
           # kubernetes_services:
           #   - http://my-service-dns.my-namespace:9100/metrics
           ## Kubernetes config file to create client from.
           # kube_config: "/path/to/kubernetes.config"
           ## Scrape Kubernetes pods for the following prometheus annotations:
           ## - prometheus.io/scrape: Enable scraping for this pod
           ## - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
           ##     set this to https & most likely set the tls config.
           ## - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
           ## - prometheus.io/port: If port is not 9102 use this annotation
           # monitor_kubernetes_pods: true
           ## Restricts Kubernetes monitoring to a single namespace
           ##   ex: monitor_kubernetes_pods_namespace = "default"
           # monitor_kubernetes_pods_namespace: ""
           ## label selector to target pods which have the label
           # kubernetes_label_selector: "env=dev,app=nginx"
           ## field selector to target pods
           ## eg. To scrape pods on a specific node
           # kubernetes_field_selector: "spec.nodeName=$HOSTNAME"
           ## Use bearer token for authorization. ('bearer_token' takes priority)
           # bearer_token: "/path/to/bearer/token"
           ## OR
           # bearer_token_string: "abc_123"
           ## HTTP Basic Authentication username and password. ('bearer_token' and
           ## 'bearer_token_string' take priority)
           # username: ""
           # password: ""
           ## Specify timeout duration for slower prometheus clients (default is 3s)
           # response_timeout: "3s"
           ## Optional TLS Config
           # tls_ca: /path/to/cafile
           # tls_cert: /path/to/certfile
           # tls_key: /path/to/keyfile
           ## Use TLS but skip chain & host verification
           # insecure_skip_verify: false
           ## Metric name prefix
           # bridge_prefix: "prometheus"
       publish:
         - plugin_name: publisher-appoptics

3. Restart the agent:

   On Windows command line:

   net stop swisnapd
   net start swisnapd

   On Linux command line:

   sudo service swisnapd restart

4. Enable the Prometheus integration in the AppOptics UI

   On the Integrations Page you will see Prometheus integration available if the previous steps were successful. It may take a couple minutes before the Prometheus integration is identified. Select the Prometheus integration to open the configuration menu in the UI, and enable it. If you do not see it, see Troubleshooting Linux..

Testing Integration

To check if and what metrics can be collected with given configuration, run bridge plugin in debug mode:

On Windows command line:

"C:\Program Files\SolarWinds\Snap\bin\snap-plugin-collector-bridge.exe" --debug-mode --plugin-config  "{\"prometheus\": {\"urls\": [\"http://localhost:9100/metrics\"]}}"

On Linux command line:

/opt/SolarWinds/Snap/bin/snap-plugin-collector-bridge --debug-mode --plugin-config "{\"prometheus\": {\"urls\": [\"http://localhost:9100/metrics\"]}}"

Metrics and Tags

The collected metrics will be prefixed with prometheus.* in AppOptics (or the configured value for bridge_prefix on the task configuration).

All metrics receive the url tag referencing the source URL in the configuration. If using Kubernetes service discovery, the address tag is also added indicating the discovered ip address.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.