Logs Collector/Forwarder

Overview

This plugin is built on top of the modified Remote Syslog 2 engine which, as the official documentation says, tails one or more log files and sends syslog messages to a remote central syslog server. It generates packets itself, ignoring the system syslog daemon, so its configuration doesn’t affect system-wide logging. It provides a very broad scope of use cases such as:

• Collecting logs from servers & daemons which don't natively support syslog
• When reconfiguring the system logger is less convenient than a purpose-built daemon (e.g., automated app deployments)
• Aggregating files not generated by daemons (e.g., package manager logs)

It is a special plugin that forwards gathered logs to a remote log aggregation and analysis system. The officially supported systems for sending logs to are (both of them are SolarWinds products):

• Loggly
• Papertrail

The logs plugin also acts like a regular SWISnap plugin by collecting and submitting metrics (statistics of gathered metrics) to AppOptics.

Setup

The logs plugin is included with the SolarWinds Snap Agent by default, please follow the directions below to enable it for a Snap Agent instance. It can be enabled by default at the installation time by the official installer script by providing an optional --detect-logs parameter, it will also detect potential logs on the system.

Prerequisites

This plugin requires an active Loggly or Papertrail account.

Features

The logs collector supports the following logs sources:

• log file(s)
• Windows Events
• docker container logs

Configuration

The logs.yaml configuration template file

If plugins.d/logs.yaml exists, it is a relic of v1 logs collector and should be manually updated/migrated to match the v2 plugins configuration architecture. However, it will continue to work as its contents will be passed to tasks-autoload.d/task-logs-legacy.yaml which is a v2 task dedicated to keep the new logs collector backward compatible. In that case you will likely find some errors/warnings about failed attempts to start the v1 logs collector plugin and task, but swisnap task list should reflect the "legacy" task running.

If there exists a plugins.d/logs.yaml.example file, it can be safely ignored as it’s a disabled configuration file holding a deprecated configuration structure for v1 plugin. This file was created by the earlier versions of Snap Agent installer.

The logs-v2.yaml configuration template file

Snap Agent provides an example configuration files to help you get started quickly. Plugin configuration template file has to define the correct settings for your target Logs service.

Update the logs-v2.yaml configuration template (typically can be found in /opt/SolarWinds/Snap/etc/plugins.d on Linux or C:\ProgramData\SolarWinds\Snap\plugins.d\ on Windows) file with settings specific to your Logs server. Those settings will be automatically propagated to the every task that uses logs collector.

See the example below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
v2:
  collector:
    logs:
      all:

        sources:
          ## Define common settings for Loggly or Papertrail.
          common:

            ## Maximum line length to be written at once (in utf-8 characters)
            ## 0 means no limit
            ## 1024 is the default
            #max_line_length: 1024

        ## Tail and send logs from this host to Loggly or Papertrail.
        ## Loggly and Papertrail are also from the SolarWinds DevOps Cloud Product Family Suite.
        logging_service:

          ## Override the hostname used for logs reported by this agent. Defaults to the OS-provided hostname.
          #hostname: "myhost"

          timeouts:

            ## A timeout for connecting to logs-accepting API
            #connect: 30s

            ## A timeout for writing to logs-accepting API
            #write: 30s

          ## Sign up for a Loggly account at: https://www.loggly.com
          loggly:

            ## Loggly API token and host
            token: "SOLARWINDS_TOKEN"
            host: "logs-01.loggly.com"

            ## Loggly API port and protocol
            ## use 6514 with TLS or 514 with TCP
            port: 6514
            protocol: tls

            ## Path to Loggly public CA certificate. See https://www.loggly.com/docs/rsyslog-tls-configuration/ for reference.
            ## Uncomment this line if you want to use custom host certificate store.
            # ca_certificate_path: /path/to/your/certificate

          ## Sign up for a Papertrail account at: https://papertrailapp.com
          #papertrail:
            ## Papertrail API token
            #token: "PAPERTRAIL_TOKEN"

            ## Papertrail host and port details: change this to YOUR papertrail host.
            #host: "HOST.papertrailapp.com"
            #port: 12345
            #protocol: tls

Loggly

Can be set up at logging_service.loggly. Here are some details specific to this service:

• token is required and should be set to a proper SolarWinds Token. This token is automatically inserted during agent installation and can be the same used by AppOptics.
• host it’s optional and defaults to "logs-01.loggly.com".
• port it’s optional and defaults to 6514. Use 6514 with TLS or 514 with TCP (to be set via api_protocol setting).
• ca_certificate_path specifies path to custom CA certificate used with TLS. If commented out, default host certificate store is used.
• protocol it’s optional and defaults to "tls" which means that this plugin will use TLS-encrypted syslog by default. Other options are udp or tcp for unencrypted syslog.

See Loggly Syslog Endpoints for more information.

Papertrail

Can be set up at logging_service.papertrail. Here are some details specific to this service:

• token is required, but at present can be set to any non-empty string.
• host and port are required and should be changed to a valid Papertrail log destination. See Papertrail log destination for more information on creating and managing log destinations in Papertrail.
• protocol it’s optional and defaults to "tls" which means that this plugin will use TLS-encrypted syslog by default. Other options are udp or tcp for unencrypted syslog.

Timeouts

Can be set up at logging_service.timeouts.

• connect it’s optional and defaults to "30s". This is a timeout for connecting to logs-accepting API.
• write it’s optional and defaults to "30s". This is a timeout for writing to logs-accepting API.

Override hostname

• logging_service.hostname it’s optional and defaults to the OS-provided hostname. You can override it here to alter the hostname tag used for logs reported by this agent.

  This takes effect on the data submitted to a log aggregation service and does not affect the AppOptics metrics tagging.

Limit line length

• sources.common.max_line_length it’s optional and defaults to 1024. This is a maximum line length to be written at once (in utf-8 characters). 0 is a special value and means no limit is enforced.

  This setting is associated with max length of a line in the syslog format, not the raw log line in file.

Monitoring log files

Can be set up by filling up sources.log_files with the options as described below.

• new_file_check_interval it’s optional and defaults to "30s". This is an interval for looking for new files matching given pattern(s).

• file_paths it’s optional and defaults to an empty list (in that case no files will be monitored)

  This is a list of files to be watched:

  • path: file name or filename pattern. Wildcards such as /home/**/*.log are valid.
  • tag: an additional tag that will be applied to log lines read from the matching file(s).

  The files to be watched have to be readable by the solarwinds user/group.

• exclude_files_patterns it’s optional and defaults to an empty array. You can use it to provide one or more regular expressions to prevent certain files from being matched.

• exclude_lines_patterns it’s optional. There may be certain log messages that you do not want to send. These may be repetitive log lines that are "noise" that you might not be able to filter out easily from the respective application. To filter these lines, use exclude_patterns with an array of regular expressions.

The list of files to be monitored can be generated during the installation of the agent. Refer to Installation with log monitoring for more details. NOTE: Be careful when attempting to handle swisnapd logs as those might also contain log entries of the logs collector. To avoid an infinite recurrence effect you should apply the exclude pattern below by adding the ".*self-skip-logs-collector.*" pattern.

Collecting Windows Events

Can be set up by filling up sources.windows_events with the options as described below.

• windows_events it’s optional and enables the user to configure windows events that need to be traced and logged in an external logging service (e.g., loggly).

  • filters enumerates channels that are to be observed.

    Each channel can provide independent filters describing which messages should be passed to Loggly/Papertrail.

    Currently following fields are supported:

    • Level - level of event (e.g., Error, Warning, Information, Success Audit, Failure Audit)
    • EventId - event identifier
    • Source - application that triggered event (e.g., VSS, Winlogon)
    • Computer - computer on which event was triggered
    • User - user who triggered event
    • Message - message associated with the event

    For each field, either a single value or a list of possible values (as in the "Level" example below) can be provided. Field names and value(s) are case sensitive.

    Copy

    windows_events:
      filters:
      - channel: System
        Level: Error
      - channel: Application
        Level:
        - Error
        - Warning

  Default task related to windows events is located in C:\ProgramData\SolarWinds\Snap\tasks-autoload.d\task-logs-system.yaml.example.

  There are also special matchers:

  • range - allows to specify range of numbers instead of listing them one by one,
  • contains - checks whether a string contains a specific word
  • matches - checks whether string matches a regular expression

  range should be used only with EventId field, contains and matches are the most usable with Message fields, although might be used as a matcher with any other fields (requiring string argument). Example:

  Copy

  windows_events:
    enable: Yes
    filters:
    - channel: Application
      Level: Error
      EventId:
        - range:
            min: 50
            max: 53
        - 60
      Message:
      - matches: "event[0-9]{2,3}"
      - contains: message

Collecting Docker logs

Can be set up by filling up sources.docker with the options as described below.

• docker it’s optional and enables the user to configure docker containers logs that need to be traced and logged in an external logging service (e.g., loggly).

  • endpoint configures docker service endpoint. It’s optional and defaults to unix "unix:///var/run/docker.sock", windows "npipe:////./pipe/docker_engine".

  • timeout sets timeout for calling docker API. It’s optional and defaults to "10s".

  • logs defines the list of filters. It’s optional and defaults to getting all containers without option "-t" with default log options.

    Copy

    options:
      showstdout: true
      showstderr: true
      since: ""
      follow: true
      tail: all

    Each filter can provide independent filters describing which containers should be taken and with which log options messages.

    • filters sets filters for getting interesting containers. The filer can contain combination of two or more fileds like: "name" and "label". See https://docs.docker.com/engine/api/v1.39/#operation/ContainerList

      logs: - filters:

      name:

      nginx: true

      label:

      prod: true snap: true

    • options sets options for container logs. See https://docs.docker.com/engine/api/v1.39/#operation/ContainerLogs - stdout shows stdout logs. Defaults to "false" - stderr shows stderr logs. Defaults to "false" - since returns logs since this time as UNIX timestamp. Defaults to "time.Now". - until returns logs until this time as UNIX timestamp. Defaults to "0". - timestamp prints timestamp for every log line. Defaults to "false". - follow returns stream. Defaults to "false". - tail output specified number of lines at the end of logs, "all" or "number". Defaults to "all". - details shows extra details provided to logs. Defaults to "false".

    • allow_tty allows to read logs from container started with opitons -t.

    • exclude_variables allows the ignore container by setting container label. Second option for ignoring container is to set for container environment variable: -e "LOGS=ignore" or -e "LOGS=IGNORE"

    Copy

    docker:
      endpoint: unix:///var/run/docker.sock
      timeout: 10s
      logs:
      - filters:
          name:
            nginx: true
        options:
          showstdout: true
          showstderr: true
          since: ""
          until: "2030-04-22T15:19:01+02:00"
          timestamp: false
          follow: true
          tail: all
          details: false
        allow_tty: false
        exclude_variables:
        - not-log
      - filters:
          label:
            prod: true
        options:
          showstdout: false
          showstderr: true
          since: ""
          until: "2030-04-22T15:19:01+02:00"
          timestamp: false
          follow: true
          tail: all
          details: false
        allow_tty: true
        exclude_variables:
        - label-1
        - label-2

Task configuration files

There are also some example task files ready to be enabled for autoloading. Each of them is dedicated to handle a subset of what the collector provides:

Additionally, some of the example task files for other Snap Agent integrations also contain a section for configuring gathering logs related to a given service. See the following example (at the bottom):

---
version: 2

schedule:
  type: cron
  interval: "0 * * * * *"

plugins:
  - plugin_name: bridge

    config:
      redis:
        ## specify servers via a url matching:
        ##  [protocol://][:password]@address[:port]
        ##  e.g.
        ##    tcp://localhost:6379
        ##    tcp://:password@192.168.99.100
        ##
        ## If no servers are specified, then localhost is used as the host.
        ## If no port is specified, 6379 is used
        servers:
          - tcp://localhost:6379

        ## Optional TLS Config
        # tls_ca: /path/to/cafile
        # tls_cert: /path/to/certfile
        # tls_key: /path/to/keyfile

        ## Use TLS but skip chain & host verification
        # insecure_skip_verify: true

    publish:
      - plugin_name: publisher-appoptics

## If you want to gather logs for this integration, uncomment the following section.
#  - plugin_name: logs
#    config:
#      sources:
#        log_files:
#          file_paths:
#            - path: /var/log/redis*.log

#    publish:
#      - plugin_name: publisher-appoptics

To enable any of the tasks, make a copy of the example file xyz.yaml.example, renaming it to be a regular YAML file (xyz.yaml):

On Windows, using Explorer or PowerShell:

> copy "C:\ProgramData\SolarWinds\Snap\tasks-autoload.d\task-logs-docker.yaml.example" "C:\ProgramData\SolarWinds\Snap\tasks-autoload.d\task-logs-docker.yaml"

On Linux using command line:

$ sudo cp /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-logs-docker.yaml.example /opt/SolarWinds/Snap/etc/tasks-autoload.d/task-logs-docker.yaml

Restart SolarWinds Snap Agent

Service restart is needed for the changes to be applied.

On Windows command line:

> net stop swisnapd
> net start swisnapd

On Linux command line:

$ sudo service swisnapd restart

Enable the Logs plugin in the AppOptics UI

Don't forget to enable reading diagnostic/statistic metrics from the Logs collector. On the Integrations Page you will see the Logs plugin available if the previous steps were successful. If you do not see the plugin, see Troubleshooting Linux. Select the Logs plugin to open the configuration menu in the UI, and enable the plugin. You should soon see the logs metrics reported to your dashboard.

Metrics and Tags

The table below lists each of the metrics gathered from the status endpoint.

Default Metrics

The general rule for lines counters can be described using the following equations:

lines_total = lines_forwarded + lines_skipped
lines_forwarded = lines_succeeded + lines_failed

Default Metric Tags

All Logs metrics are tagged with hostname. Instead of using this tag we recommend using the @host alias. If dynamic metrics are used (see exemplary task file provided with the plugin), then server name or hostname and port from logs will be added as a tag.

Troubleshooting

Too many open files

When the collector stops forwarding lines and in SolarWinds Snap Agent logs (typically located in /var/log/SolarWinds/Snap/swisnapd.log) you notice errors like this:

ERRO[2019-04-18T13:35:30-04:00] time="2019-04-18T13:35:30-04:00" level=error msg="follower error" error="too many open files" self-skip-logs-collector= submodule=remote_syslog _module=plugin-exec io=stderr plugin=logs

it means your OS limits have to be leveraged to handle monitoring of all the requested files. You can determine the maximum number of inotify (an underlying files watcher) instances that can be created using:

cat /proc/sys/fs/inotify/max_user_instances

and then increase this limit using:

echo VALUE >> /proc/sys/fs/inotify/max_user_instances

where VALUE is greater than the present setting.

When you confirm that the limits are met and the logs collector/forwarder works as expected you should apply this new value permanently by adding the following to /etc/sysctl.conf:

fs.inotify.max_user_instances = VALUE

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.