Forward Kubernetes logs (Papertrail)
For non-Kubernetes Docker hosts, see Forward Docker logs (Papertrail).
Use a standard log-collector DaemonSet to ship container logs from a Kubernetes cluster into Papertrail.
In Kubernetes, every pod's stdout/stderr is written to the node at /var/log/containers/*.log. A collector running as a DaemonSet, using one pod per node, tails those files, enriches each record with metadata (pod, namespace, container), and forwards it to Papertrail.
To get a destination and token in Papertrail, add a new log destination. The ingestion token is generated along with the destination. Use the collector endpoints for your account's region:
-
Syslog (TLS):
syslog.collector.<REGION>.cloud.solarwinds.com:6514 -
HTTPS:
https://logs.collector.<REGION>.cloud.solarwinds.com/v1/logs/bulk
Replace <REGION> with the data center your organization uses: na-01, eu-01, or ap-01.
Fluent Bit DaemonSet (syslog output)
Fluent Bit is a lightweight, Kubernetes-native collector. It ships as a DaemonSet, tails all container logs on each node, and its built-in kubernetes filter attaches pod, namespace, labels, and container metadata to every record.
Create values.yaml according to the example below, and then install with the official Helm chart:
helm repo add fluent https://fluent.github.io/helm-charts helm repo update helm install fluent-bit fluent/fluent-bit -f values.yaml
values.yaml
This example covers the following:
-
Reading and forwarding container logs (stdout/stderr)
-
Reading and forwarding file logs
-
Setting a proper structured data element for authentication
-
Setting a proper system, program, and severity
image:
repository: fluent/fluent-bit
tag: latest
pullPolicy: IfNotPresent
service:
flush: 2
logLevel: warn
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SWO_TOKEN
valueFrom:
secretKeyRef:
name: swo-ingestion-token
key: token
tolerations:
- operator: Exists
rbac:
create: true
serviceAccount:
create: true
extraVolumes:
- name: state
hostPath:
path: /var/fluent-bit/state
type: DirectoryOrCreate
extraVolumeMounts:
- name: state
mountPath: /fluent-bit/state
config:
inputs: |
# Container logs (stdout/stderr). Exclude Fluent Bit's own logs and system logs if you do not want them to be forwarded to SWO.
[INPUT]
Name tail
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*fluent-bit*.log,/var/log/containers/*_kube-system_*.log
Parser cri
Tag kube.*
DB /fluent-bit/state/containers.db
Read_from_Head true
Mem_Buf_Limit 10MB
Skip_Long_Lines On
# File logs. Set a proper Path where your pods are logging.
[INPUT]
Name tail
Path /var/log/test-app/*.log
Tag app.*
DB /fluent-bit/state/application.db
Read_from_Head true
Mem_Buf_Limit 10MB
Skip_Long_Lines On
filters: |
# The tail input plugin, when used without a parser, stores each raw line it reads into a field named log. This filter simply renames log to message so the syslog output can find it. The kube.* input doesn't need this because it uses the cri parser, which explicitly names the content field message rather than log.
[FILTER]
Name modify
Match app.*
Rename log message
# Queries the Kubernetes API server to enrich matching records with pod metadata.
[FILTER]
Name kubernetes
Match kube.*
# Runs a Lua function against matching records.
[FILTER]
Name lua
Match *
script /fluent-bit/scripts/fluent-bit.lua
call add_sd
outputs: |
[OUTPUT]
Name syslog
Match *
Host syslog.collector.<REGION>.cloud.solarwinds.com
Port 6514
Mode tls
tls.verify On
Syslog_Format rfc5424
Syslog_Hostname_Key k8s_host
Syslog_Appname_Key k8s_appname
Syslog_Severity_Key syslog_severity
Syslog_Message_Key message
Syslog_SD_Key solarwinds@41058
luaScripts:
fluent-bit.lua: |
# For details, see below.
local SEVERITY = {
debug = 7, info = 6, notice = 5,
warn = 4, warning = 4,
error = 3, err = 3,
crit = 2, critical = 2,
alert = 1, emerg = 0, emergency = 0,
}
function add_sd(tag, ts, record)
record["solarwinds@41058"] = {token = os.getenv("SWO_TOKEN")}
local msg = record["message"]
if type(msg) == "string" then
local level = msg:match('"level"%s*:%s*"([^"]+)"')
if level then
record["syslog_severity"] = SEVERITY[level:lower()] or 6
end
end
local k8s = record["kubernetes"]
if k8s then
record["k8s_host"] = k8s["pod_name"]
record["k8s_appname"] = k8s["container_name"]
else
record["k8s_host"] = os.getenv("NODE_NAME")
record["k8s_appname"] = tag:match("%.([^.]+)%.[^.]+%.[^.]+$")
end
return 1, ts, record
end
Notes
-
Syslog_Hostname_Keymaps the pod name to the syslog hostname. This is shown as System in the Logs Explorer. -
Syslog_Appname_Keymaps the container name to the syslog appname. This is shown as Program in the Logs Explorer. -
Syslog_Message_Keysends the JSON payload as the message body, so Papertrail can parse the embedded metadata into searchable fields. -
The ingestion token is passed as syslog structured data (
Syslog_SD_Key). This is how Papertrail authenticates the stream.
See the Fluent Bit syslog output and Kubernetes installation documentation for full options.
fluent-bit.lua
This function has three responsibilities:
-
SD token: Injects
[solarwinds@41058 token="TOKEN"]as the RFC 5424 structured data element.The syslog output plugin uses the field name (
solarwinds@41058) as the SD-ID and the map value ({token = TOKEN}) as the SD params. -
Severity: Reads the level field and maps it to an RFC 5424 numeric severity (0-7) stored in
syslog_severity. -
Metadata flattening: For
kube.*records, copieskubernetes.pod_nameandkubernetes.container_nameinto flat top-level fieldsk8s_hostandk8s_appname.
my-app.yaml
apiVersion: v1
kind: Pod
metadata:
name: ...
labels:
app: ...
spec:
containers:
- name: ...
...
volumeMounts:
- name: app-log
mountPath: /var/log/my-app
volumes:
- name: app-log
hostPath:
path: /var/log/my-app
type: DirectoryOrCreate
Severity
Container stdout/stderr has no inherent syslog severity. The app's level (for instance, ERROR or WARN) is just text in the message or a JSON field. By default, Fluent Bit's syslog output stamps every record with a fixed level (syslog_severity_preset, default 6 = Info), so every log shows as Info in the Logs Explorer and real errors are indistinguishable.
To map the app's real level to the Logs Explorer's severity, use a Lua script to provide values for Syslog_Severity_Key, as shown in the example above. The exact code depends on your way of logging.
Facility
Facility can be left at its preset (user/local0). It is searchable as syslog.facility, but it carries no real meaning for container logs. There is no need to set it.
Namespace, labels, and other metadata
System and Program come from the syslog hostname/appname, but the rest of the Kubernetes metadata (namespace, labels, cluster, etc.) does not travel in the syslog header. To make it searchable, send the record as JSON in the message body. Papertrail automatically parses a JSON body and flattens it into searchable json.* fields. With the config above, you will have fields such as json.kubernetes.namespace_name, json.kubernetes.container_name, and json.kubernetes.labels.app.
Consider the following:
-
JSON-body fields are searchable but do not drive the System or Program columns, which read only the hostname/appname. The two mechanisms are complementary, using hostname/appname for the columns, and the JSON body for everything else.
-
Keep the JSON shallow. Papertrail caps nesting depth and collection size when flattening. Very deep or large label/annotation sets can be truncated. Forward labels but drop annotations, and avoid deeply nested structures.
Alternatives
Any collector that can run as a DaemonSet, enrich with Kubernetes metadata, and emit per-record syslog (or OTLP resource attributes) works the same way. The same rule applies: avoid per-batch HTTP headers for multi-pod collectors.
-
Grafana Alloy:
loki.source.kubernetes+loki.source.file, forwarding via a syslog orotelcolcomponent. Transformations are done in OTTL, which covers JSON parsing, attribute mapping, and severity extraction. -
OpenTelemetry Collector: the
filelogreceiver +k8sattributesprocessor, settinghost.name(pod) andservice.name(container) as true resource attributes per record. OTLP carries these per record, so it avoids the per-batch problem entirely.
Fallback: Forward via a node-level syslog daemon
If the collector DaemonSet cannot reach the Papertrail collector directly (for example, because a network policy blocks egress on ports 6514 and 443), configure rsyslog on each node to tail the container log files from /var/log/containers/ and forward them to Papertrail over TLS. This is a node-level operation outside the Kubernetes control plane and is typically handled by the platform team. See Send logs using syslog (Papertrail) for configuration instructions.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.