Documentation forAppOptics

Varnish

Overview

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents.

This integration is only available for Linux platforms.

Setup

The varnish monitoring is accomplished by bridge plugin which is included with the SolarWinds Snap Agent by default. Follow the directions below to enable it for an agent instance. The bridge plugin utilize Telegraf Varnish plugin.

Prerequisites

The Varnish daemon must be started beforehand and configured to point to your HTTP service. By default, the Varnish service will run on port 6081, and the configuration file at /etc/varnish/default.vcl will need to be edited to point to your HTTP service.

Once the configuration file has been updated, reload the service:

service varnish reload

If you updated the service file to change the port Varnish runs on, you will have to reload the service file:

systemctl daemon-reload

Visit the Varnish website for more information.

Configuration

The agent provides an example task file to help you get started quickly, but requires you to provide the correct settings for your Varnish installation. To enable the task:

  1. Make a copy of the Varnish example task file /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml.example, renaming it to /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml:

    sudo cp -p /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml.example /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml
  2. Edit the task file to set which metrics to report and to ensure varnishstat is executable if needed, for example:

    If you wish to also collect logs for this service, uncomment the last section in the example task file. For more information on collecting logs, see the logs collector docs.

    ---
    version: 2
    
    schedule:
      type: cron
      interval: "0 * * * * *"
    
    plugins:
      - plugin_name: bridge
    
        config:
          varnish:
            ## If running as a restricted user you can prepend sudo for additional access:
            # use_sudo: false
            ## The default location of the varnishstat binary can be overridden with:
            # binary: "/usr/bin/varnishstat"
            ## By default, telegraf gather stats for 3 metric points.
            ## Setting stats will override the defaults shown below.
            ## Glob matching can be used, ie, stats: "MAIN.*"
            ## stats may also be set to "*", which will collect all stats
            stats: 
              - "MAIN.backend_*"
              - "MAIN.cache_*"
              - "MAIN.client_req"
              - "MAIN.fetch_*"
              - "MAIN.losthdr"
              - "MAIN.n_expired"
              - "MAIN.n_lru_*"
              - "MAIN.s_re*"
              - "MAIN.s_sess"
              - "MAIN.sess_*"
              - "MAIN.shm_*"
              - "MAIN.thread*"
              - "MAIN.uptime"
    
            ## Optional name for the varnish instance (or working directory) to query
            ## Usually appened after -n in varnish cli
            # instance_name: instanceName
    
        publish:
          - plugin_name: publisher-appoptics
    
    ## If you want to gather logs for this integration, uncomment the following section.
    #  - plugin_name: log-files
    #    config:
    #      file_paths:
    #        - /var/log/varnish/varnishncsa.log
    
    #    publish:
    #      - plugin_name: loggly-http-bulk
  3. Restart the agent:

    sudo service swisnapd restart
  4. Enable the Varnish integration in AppOptics

    On the Integrations Page you will see the Varnish integration available if the previous steps were successful. It may take a couple minutes before the Varnish integration is identified. Select the Varnish integration to open the configuration menu in the UI, and enable it. If you do not see it, see Troubleshooting Linux.

Testing Integration

To check if and what metrics can be collected with given configuration, run bridge plugin in debug mode:

/opt/SolarWinds/Snap/bin/snap-plugin-collector-bridge --debug-mode --plugin-config "{\"varnish\": {\"stats\": [\"MAIN.backend_*\"]}}"

Permission issues with running varnishstat

If you're seeing errors related to running varnishstat, you may need to set additional permissions for the solarwinds user.

  1. First, try adding the solarwinds user to the varnish group:

    usermod -a -G varnish solarwinds

    Confirm that the user has been updated:

    groups solarwinds
    solarwinds : solarwinds varnish

    Then restart the service:

    sudo service swisnapd restart
  2. If that doesn't work, try giving the solarwinds user sudo privileges:

    Set use_sudo to true in the /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml file.

    ## If running as a restricted user you can prepend sudo for additional access:
    use_sudo: true

    Then restart the service:

    sudo service swisnapd restart

Metrics and Tags

Default Metrics

The table below outlines the default set of metrics collected by the Varnish integration.

Namespace Description
varnish.backend_conn Successful backend connections
varnish.backend_unhealthy Backend connections not attempted due to unhealthy backend state
varnish.backend_busy Unsuccessful backend connections due to exceeding the maximum number of connections
varnish.backend_fail Backend connection failures
varnish.backend_reuse Backend connection reuses
varnish.backend_recycle Backend connection recycles
varnish.backend_retry Backend connection retries
varnish.cache_hit Number of requests served by the cache server
varnish.cache_hitpass Requests passed to the backend from the cache by choice
varnish.cache_miss Number of requests served by the backend server
varnish.client_req Good client requests received
varnish.fetch_head Backend HEAD requests
varnish.fetch_length Backend responses with Content-Length in the body
varnish.fetch_chunked Backend responses that were chunked
varnish.fetch_eof Backend responses with EOF in the body
varnish.fetch_bad Backend responses with a bad body
varnish.fetch_none Backend responses with no body
varnish.fetch_1xx Backend responses with no body due to 1xx responses
varnish.fetch_204 Backend responses with no body due to 204 responses
varnish.fetch_304 Backend responses with no body due to 304 responses
varnish.fetch_failed Backend response fetches that failed
varnish.fetch_no_thread Backend response fetches that failed due to no thread being available
varnish.losthdr HTTP header overflows
varnish.n_expired Number of expired objects
varnish.n_lru_nuked Number of LRU nuked objects
varnish.n_lru_moved Number of LRU moved objects
varnish.s_req Total requests seen
varnish.s_req_hdrbytes Total request header bytes
varnish.s_req_bodybytes Total request body bytes
varnish.s_resp_hdrbytes Total response header bytes
varnish.s_resp_bodybytes Total response body bytes
varnish.s_sess Total sessions seen
varnish.sess_conn Accepted client connections
varnish.sess_drop Dropped client connections
varnish.sess_fail Failed client connections
varnish.shm_records SHM records
varnish.shm_writes SHM writes
varnish.shm_flushes SHM flushes due to overflow
varnish.shm_cont SHM MTX contention
varnish.shm_cycles SHM cycles through buffer
varnish.threads Total number of threads
varnish.threads_limited Number of threads that hit the thread pool limit
varnish.threads_created Number of threads created
varnish.threads_destroyed Number of threads destroyed
varnish.threads_failed Number of threads that failed to be created
varnish.thread_queue_len Length of session queue for threads
varnish.uptime How long the child process has been running

Optional Metrics

All stats that varnishstat reports are available in AppOptics. To expose these, you will need to edit the stats configuration setting in the /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-bridge-varnish.yaml file. It is highly recommended to only expose metrics you are interested in as setting stats to "*" will cause all metrics (including those reporting no value/zero values) to start reporting to AppOptics at a 60 second resolution.

MAIN Metrics

Namespace Description
varnish.client_req_400 Client requests received, subject to 400 errors
varnish.client_req_417 Client requests received, subject to 417 errors
varnish.pools Number of thread pools
varnish.busy_sleep Number of requests sent to sleep because a busy object was found
varnish.busy_wakeup Number of requests woken after sleep on busy and rescheduled
varnish.busy_killed Number of requests killed after sleep on busy
varnish.sess_queued Client connections queued waiting for a thread
varnish.sess_dropped Client connections dropped waiting for a thread
varnish.n_object Number of object structs made
varnish.n_vampireobject Number of unresurrected objects
varnish.n_objectcore Number of objectcore structs made
varnish.n_objecthead Number of objecthead structs made
varnish.n_waitinglist Number of waitinglist structs made
varnish.n_backend Number of backends
varnish.s_pipe Total pipe sessions seen
varnish.s_pass Total passed requests seen
varnish.s_fetch Total backend fetches initiated
varnish.s_synth Total synthethic responses made
varnish.s_pipe_hdrbytes Total pipe request header bytes
varnish.s_pipe_in Total piped bytes from client
varnish.s_pipe_out Total piped bytes to client
varnish.sess_closed Number of client connections closed
varnish.sess_closed_err Number of client connections closed with an error
varnish.sess_readahead Number of client connections read ahead
varnish.sess_herd Number of times the timeout_linger triggered
varnish.sc_rem_close Number of client connection closes with REM_CLOSE
varnish.sc_req_close Number of client connection closes with REQ_CLOSE
varnish.sc_req_http10 Number of client connection closes with error REQ_HTTP10
varnish.sc_rx_bad Number of client connection closes with error RX_BAD
varnish.sc_rx_body Number of client connection closes with error RX_BODY
varnish.sc_rx_junk Number of client connection closes with error RX_JUNK
varnish.sc_rx_overflow Number of client connection closes with error RX_OVERFLOW
varnish.sc_rx_timeout Number of client connection closes with error RX_TIMEOUT
varnish.sc_tx_pipe Number of client connection closes with TX_PIPE
varnish.sc_tx_error Number of client connection closes with error TX_ERROR
varnish.sc_tx_eof Number of client connection closes with TX_EOF
varnish.sc_resp_close Number of client connection closes with RESP_CLOSE
varnish.sc_overload Number of client connection closes with error OVERLOAD
varnish.sc_pipe_overflow Number of client connection closes with error PIPE_OVERFLOW
varnish.sc_range_short Number of client connection closes with error RANGE_SHORT
varnish.backend_req Backend requests made
varnish.n_vcl Total number of loaded VCLs
varnish.n_vcl_avail Total number of VCLs available
varnish.n_vcl_discard Total number of discarded VCLs
varnish.bans Count of bans
varnish.bans_completed Number of bans marked ‘completed' (no longer active)
varnish.bans_obj Number of bans using obj.*
varnish.bans_req Number of bans using req.*
varnish.bans_added Bans added to ban list
varnish.bans_deleted Bans deleted from ban list
varnish.bans_tested Bans tested against objects during lookup
varnish.bans_obj_killed Objects killed by bans during lookup
varnish.bans_lurker_tested Bans tested against objects by the ban-lurker
varnish.bans_tests_tested Ban tests tested against objects during lookup
varnish.bans_lurker_tests_tested Ban tests tested against objects by the ban-lurker
varnish.bans_lurker_obj_killed Objects killed by the ban-lurker
varnish.bans_dups Bans superseded by bans added later to the ban list
varnish.bans_lurker_contention Number of times the ban-lurker waited for lookups
varnish.bans_persisted_bytes Bytes used by the persisted ban lists
varnish.bans_persisted_fragmentation Extra bytes in persisted ban lists due to fragmentation
varnish.n_purges Number of purge operations executed
varnish.n_obj_purged Number of purged objects
varnish.exp_mailed Number of objects mailed to expiry thread
varnish.exp_received Number of objects received by expiry thread
varnish.hcb_nolock HCB Lookups without lock
varnish.hcb_lock HCB Lookups with lock
varnish.hcb_insert HCB Inserts
varnish.esi_errors Edge Side Includes (ESI) parse errors
varnish.esi_warnings Edge Side Includes (ESI) parse warnings (unlock)
varnish.vmods Loaded VMODs
varnish.n_gzip Number of Gzip operations
varnish.n_gunzip Number of Gunzip operations
varnish.vsm_free Free VSM space
varnish.vsm_used Used VSM space
varnish.vsm_cooling Soon to be freed VSM space
varnish.vsm_overflow Data which does not fit in the VSM space
varnish.vsm_overflowed Total data which did not fit in the VSM space

MGT Metrics

Namespace Description
varnish.child_start Child processes started
varnish.child_exit Child processes that were stopped normally
varnish.child_stop Child processes that exited with an unexpected return code
varnish.child_died Child processes that died due to signals
varnish.child_dump Child processes that produced core dumps
varnish.child_panic Child processes that panicked

MEMPOOL Metrics

Namespace Description
varnish.busyobj.live In use
varnish.busyobj.pool In Pool
varnish.busyobj.sz_wanted Size requested
varnish.busyobj.sz_actual Size allocated
varnish.busyobj.allocs Allocations
varnish.busyobj.frees Frees
varnish.busyobj.recycle Recycled from pool
varnish.busyobj.timeout Timed out from pool
varnish.busyobj.toosmall Too small to recycle
varnish.busyobj.surplus Too many for pool
varnish.busyobj.randry Pool ran dry
varnish.req0.live In use
varnish.req0.pool In Pool
varnish.req0.sz_wanted Size requested
varnish.req0.sz_actual Size allocated
varnish.req0.allocs Allocations
varnish.req0.frees Frees
varnish.req0.recycle Recycled from pool
varnish.req0.timeout Timed out from pool
varnish.req0.toosmall Too small to recycle
varnish.req0.surplus Too many for pool
varnish.req0.randry Pool ran dry
varnish.sess0.live In use
varnish.sess0.pool In Pool
varnish.sess0.sz_wanted Size requested
varnish.sess0.sz_actual Size allocated
varnish.sess0.allocs Allocations
varnish.sess0.frees Frees
varnish.sess0.recycle Recycled from pool
varnish.sess0.timeout Timed out from pool
varnish.sess0.toosmall Too small to recycle
varnish.sess0.surplus Too many for pool
varnish.sess0.randry Pool ran dry
varnish.req1.live In use
varnish.req1.pool In Pool
varnish.req1.sz_wanted Size requested
varnish.req1.sz_actual Size allocated
varnish.req1.allocs Allocations
varnish.req1.frees Frees
varnish.req1.recycle Recycled from pool
varnish.req1.timeout Timed out from pool
varnish.req1.toosmall Too small to recycle
varnish.req1.surplus Too many for pool
varnish.req1.randry Pool ran dry
varnish.sess1.live In use
varnish.sess1.pool In Pool
varnish.sess1.sz_wanted Size requested
varnish.sess1.sz_actual Size allocated
varnish.sess1.allocs Allocations
varnish.sess1.frees Frees
varnish.sess1.recycle Recycled from pool
varnish.sess1.timeout Timed out from pool
varnish.sess1.toosmall Too small to recycle
varnish.sess1.surplus Too many for pool
varnish.sess1.randry Pool ran dry

SMA Metrics

Namespace Description
varnish.s0.c_req Allocator requests
varnish.s0.c_fail Allocator failures
varnish.s0.c_bytes Bytes allocated
varnish.s0.c_freed Bytes freed
varnish.s0.g_alloc Allocations outstanding
varnish.s0.g_bytes Bytes outstanding
varnish.s0.g_space Bytes available
varnish.Transient.c_req Allocator requests
varnish.Transient.c_fail Allocator failures
varnish.Transient.c_bytes Bytes allocated
varnish.Transient.c_freed Bytes freed
varnish.Transient.g_alloc Allocations outstanding
varnish.Transient.g_bytes Bytes outstanding
varnish.Transient.g_space Bytes available

VBE Metrics

Namespace Description
varnish.boot.default.happy Number of happy health probes
varnish.boot.default.bereq_hdrbytes Total request header bytes
varnish.boot.default.bereq_bodybytes Total request body bytes
varnish.boot.default.beresp_hdrbytes Total response header bytes
varnish.boot.default.beresp_bodybytes Total response body bytes
varnish.boot.default.pipe_hdrbytes Total pipe request header bytes
varnish.boot.default.pipe_out Total piped bytes to backend
varnish.boot.default.pipe_in Total piped bytes from backend
varnish.boot.default.conn Number of concurrent connections to backend
varnish.boot.default.req Number of backend requests sent

LCK Metrics

Namespace Description
varnish.backend.creat Created locks
varnish.backend.destroy Destroyed locks
varnish.backend.locks Lock Operations
varnish.backend_tcp.creat Created locks
varnish.backend_tcp.destroy Destroyed locks
varnish.backend_tcp.locks Lock Operations
varnish.ban.creat Created locks
varnish.ban.destroy Destroyed locks
varnish.ban.locks Lock Operations
varnish.busyobj.creat Created locks
varnish.busyobj.destroy Destroyed locks
varnish.busyobj.locks Lock Operations
varnish.cli.creat Created locks
varnish.cli.destroy Destroyed locks
varnish.cli.locks Lock Operations
varnish.exp.creat Created locks
varnish.exp.destroy Destroyed locks
varnish.exp.locks Lock Operations
varnish.hcb.creat Created locks
varnish.hcb.destroy Destroyed locks
varnish.hcb.locks Lock Operations
varnish.lru.creat Created locks
varnish.lru.destroy Destroyed locks
varnish.lru.locks Lock Operations
varnish.mempool.creat Created locks
varnish.mempool.destroy Destroyed locks
varnish.mempool.locks Lock Operations
varnish.objhdr.creat Created locks
varnish.objhdr.destroy Destroyed locks
varnish.objhdr.locks Lock Operations
varnish.pipestat.creat Created locks
varnish.pipestat.destroy Destroyed locks
varnish.pipestat.locks Lock Operations
varnish.sess.creat Created locks
varnish.sess.destroy Destroyed locks
varnish.sess.locks Lock Operations
varnish.smp.creat Created locks
varnish.smp.destroy Destroyed locks
varnish.smp.locks Lock Operations
varnish.vbe.creat Created locks
varnish.vbe.destroy Destroyed locks
varnish.vbe.locks Lock Operations
varnish.vcapace.creat Created locks
varnish.vcapace.destroy Destroyed locks
varnish.vcapace.locks Lock Operations
varnish.vcl.creat Created locks
varnish.vcl.destroy Destroyed locks
varnish.vcl.locks Lock Operations
varnish.vxid.creat Created locks
varnish.vxid.destroy Destroyed locks
varnish.vxid.locks Lock Operations
varnish.waiter.creat Created locks
varnish.waiter.destroy Destroyed locks
varnish.waiter.locks Lock Operations
varnish.wq.creat Created locks
varnish.wq.destroy Destroyed locks
varnish.wq.locks Lock Operations
varnish.wstat.creat Created locks
varnish.wstat.destroy Destroyed locks
varnish.wstat.locks Lock Operations
varnish.sma.creat Created locks
varnish.sma.destroy Destroyed locks
varnish.sma.locks Lock Operations

Tags

The table below outlines the default set of tags provided for each metric.

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias.
section Prefix of the original varnish stat (one of main, mgt, mempool, sma, vbe, lck)

Navigation Notice: When the APM Integrated Experience is enabled, SolarWinds Template Product shares a common navigation and enhanced feature set with other integrated experience products. How you navigate SolarWinds Template Product and access its features may vary from these instructions.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.