Documentation forSolarWinds Observability SaaS

Forward Kubernetes logs (Papertrail)

For non-Kubernetes Docker hosts, see Forward Docker logs (Papertrail).

Use a standard log-collector DaemonSet to ship container logs from a Kubernetes cluster into Papertrail.

In Kubernetes, every pod's stdout/stderr is written to the node at /var/log/containers/*.log. A collector running as a DaemonSet, using one pod per node, tails those files, enriches each record with metadata (pod, namespace, container), and forwards it to Papertrail.

To get a destination and token in Papertrail, add a new log destination. The ingestion token is generated along with the destination. Use the collector endpoints for your account's region:

  • Syslog (TLS): syslog.collector.<REGION>.cloud.solarwinds.com:6514

  • HTTPS: https://logs.collector.<REGION>.cloud.solarwinds.com/v1/logs/bulk

Replace <REGION> with the data center your organization uses: na-01, eu-01, or ap-01.

Fluent Bit DaemonSet (syslog output)

Fluent Bit is a lightweight, Kubernetes-native collector. It ships as a DaemonSet, tails all container logs on each node, and its built-in kubernetes filter attaches pod, namespace, labels, and container metadata to every record.

Create values.yaml according to the example below, and then install with the official Helm chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit -f values.yaml

values.yaml

This example covers the following:

  • Reading and forwarding container logs (stdout/stderr)

  • Reading and forwarding file logs

  • Setting a proper structured data element for authentication

  • Setting a proper system, program, and severity

image:
  repository: fluent/fluent-bit
  tag: latest
  pullPolicy: IfNotPresent

service:
  flush: 2
  logLevel: warn

env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: SWO_TOKEN
    valueFrom:
      secretKeyRef:
        name: swo-ingestion-token
        key: token

tolerations:
  - operator: Exists

rbac:
  create: true

serviceAccount:
  create: true

extraVolumes:
  - name: state
    hostPath:
      path: /var/fluent-bit/state
      type: DirectoryOrCreate

extraVolumeMounts:
  - name: state
    mountPath: /fluent-bit/state

config:
  inputs: |
    # Container logs (stdout/stderr). Exclude Fluent Bit's own logs and system logs if you do not want them to be forwarded to SWO.
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*fluent-bit*.log,/var/log/containers/*_kube-system_*.log
        Parser            cri
        Tag               kube.*
        DB                /fluent-bit/state/containers.db
        Read_from_Head    true
        Mem_Buf_Limit     10MB
        Skip_Long_Lines   On

    # File logs. Set a proper Path where your pods are logging.
    [INPUT]
        Name              tail
        Path              /var/log/test-app/*.log
        Tag               app.*
        DB                /fluent-bit/state/application.db
        Read_from_Head    true
        Mem_Buf_Limit     10MB
        Skip_Long_Lines   On

  filters: |
    # The tail input plugin, when used without a parser, stores each raw line it reads into a field named log. This filter simply renames log to message so the syslog output can find it. The kube.* input doesn't need this because it uses the cri parser, which explicitly names the content field message rather than log.
    [FILTER]
        Name              modify
        Match             app.*
        Rename            log message

    # Queries the Kubernetes API server to enrich matching records with pod metadata.
    [FILTER]
        Name              kubernetes
        Match             kube.*

    # Runs a Lua function against matching records.
    [FILTER]
        Name              lua
        Match             *
        script            /fluent-bit/scripts/fluent-bit.lua
        call              add_sd

outputs: |
    [OUTPUT]
        Name                   syslog
        Match                  *
        Host                   syslog.collector.<REGION>.cloud.solarwinds.com
        Port                   6514
        Mode                   tls
        tls.verify             On
        Syslog_Format          rfc5424
        Syslog_Hostname_Key    k8s_host
        Syslog_Appname_Key     k8s_appname
        Syslog_Severity_Key    syslog_severity
        Syslog_Message_Key     message
        Syslog_SD_Key          solarwinds@41058

luaScripts:
  fluent-bit.lua: |
    # For details, see below.
    local SEVERITY = {
        debug = 7, info = 6, notice = 5,
        warn = 4, warning = 4,
        error = 3, err = 3,
        crit = 2, critical = 2,
        alert = 1, emerg = 0, emergency = 0,
    }

    function add_sd(tag, ts, record)
        record["solarwinds@41058"] = {token = os.getenv("SWO_TOKEN")}

        local msg = record["message"]
        if type(msg) == "string" then
            local level = msg:match('"level"%s*:%s*"([^"]+)"')
            if level then
                record["syslog_severity"] = SEVERITY[level:lower()] or 6
            end
        end

        local k8s = record["kubernetes"]
        if k8s then
            record["k8s_host"]    = k8s["pod_name"]
            record["k8s_appname"] = k8s["container_name"]
        else
            record["k8s_host"]    = os.getenv("NODE_NAME")
            record["k8s_appname"] = tag:match("%.([^.]+)%.[^.]+%.[^.]+$")
        end

        return 1, ts, record
    end

Notes

  • Syslog_Hostname_Key maps the pod name to the syslog hostname. This is shown as System in the Logs Explorer.

  • Syslog_Appname_Key maps the container name to the syslog appname. This is shown as Program in the Logs Explorer.

  • Syslog_Message_Key sends the JSON payload as the message body, so Papertrail can parse the embedded metadata into searchable fields.

  • The ingestion token is passed as syslog structured data (Syslog_SD_Key). This is how Papertrail authenticates the stream.

See the Fluent Bit syslog output and Kubernetes installation documentation for full options.

fluent-bit.lua

This function has three responsibilities:

  1. SD token: Injects [solarwinds@41058 token="TOKEN"] as the RFC 5424 structured data element.

    The syslog output plugin uses the field name (solarwinds@41058) as the SD-ID and the map value ({token = TOKEN}) as the SD params.

  2. Severity: Reads the level field and maps it to an RFC 5424 numeric severity (0-7) stored in syslog_severity.

  3. Metadata flattening: For kube.* records, copies kubernetes.pod_name and kubernetes.container_name into flat top-level fields k8s_host and k8s_appname.

my-app.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ...
  labels:
    app: ...
spec:
  containers:
    - name: ...
      ...
      volumeMounts:
        - name: app-log
          mountPath: /var/log/my-app

  volumes:
    - name: app-log
      hostPath:
        path: /var/log/my-app
        type: DirectoryOrCreate

Severity

Container stdout/stderr has no inherent syslog severity. The app's level (for instance, ERROR or WARN) is just text in the message or a JSON field. By default, Fluent Bit's syslog output stamps every record with a fixed level (syslog_severity_preset, default 6 = Info), so every log shows as Info in the Logs Explorer and real errors are indistinguishable.

To map the app's real level to the Logs Explorer's severity, use a Lua script to provide values for Syslog_Severity_Key, as shown in the example above. The exact code depends on your way of logging.

Facility

Facility can be left at its preset (user/local0). It is searchable as syslog.facility, but it carries no real meaning for container logs. There is no need to set it.

Namespace, labels, and other metadata

System and Program come from the syslog hostname/appname, but the rest of the Kubernetes metadata (namespace, labels, cluster, etc.) does not travel in the syslog header. To make it searchable, send the record as JSON in the message body. Papertrail automatically parses a JSON body and flattens it into searchable json.* fields. With the config above, you will have fields such as json.kubernetes.namespace_name, json.kubernetes.container_name, and json.kubernetes.labels.app.

Consider the following:

  • JSON-body fields are searchable but do not drive the System or Program columns, which read only the hostname/appname. The two mechanisms are complementary, using hostname/appname for the columns, and the JSON body for everything else.

  • Keep the JSON shallow. Papertrail caps nesting depth and collection size when flattening. Very deep or large label/annotation sets can be truncated. Forward labels but drop annotations, and avoid deeply nested structures.

Alternatives

Any collector that can run as a DaemonSet, enrich with Kubernetes metadata, and emit per-record syslog (or OTLP resource attributes) works the same way. The same rule applies: avoid per-batch HTTP headers for multi-pod collectors.

  • Grafana Alloy: loki.source.kubernetes + loki.source.file, forwarding via a syslog or otelcol component. Transformations are done in OTTL, which covers JSON parsing, attribute mapping, and severity extraction.

  • OpenTelemetry Collector: the filelog receiver + k8sattributes processor, setting host.name (pod) and service.name (container) as true resource attributes per record. OTLP carries these per record, so it avoids the per-batch problem entirely.

Fallback: Forward via a node-level syslog daemon

If the collector DaemonSet cannot reach the Papertrail collector directly (for example, because a network policy blocks egress on ports 6514 and 443), configure rsyslog on each node to tail the container log files from /var/log/containers/ and forward them to Papertrail over TLS. This is a node-level operation outside the Kubernetes control plane and is typically handled by the platform team. See Send logs using syslog (Papertrail) for configuration instructions.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.