Documentation forSolarWinds Observability SaaS

Forward Fly.io logs (Papertrail)

How it works

You can forward Fly.io logs to Papertrail using the Fly log shipper, which runs Vector inside a Fly app and reads the organization's log stream.

The flyio/log-shipper image runs Vector in its own Fly app. Vector subscribes to the Fly organization's NATS log stream and forwards each log event to the sinks that are configured. The image contains a built-in papertrail sink, but that can only be used by customers with port-based syslog destinations available.

Token destinations use HTTPS, so this setup adds an http sink instead. The sink sends RFC 5424 messages to the HTTPS bulk endpoint. System, Program, and Severity are taken from each message, so every event is attributed to the Fly app and machine that produced it rather than to the shipper.

Prerequisites

  • A Papertrail ingestion token from a token-based destination.

    To get a destination and token in Papertrail, add a new log destination. The ingestion token is generated along with the destination. Use the collector endpoints for your account's region:

    • HTTPS: https://logs.collector.<REGION>.cloud.solarwinds.com/v1/logs/bulk

    Replace <REGION> with the data center your organization uses: na-01, eu-01, or ap-01.

  • The data center the organization uses: na-01, na-02, eu-01, or ap-01. This appears in the URL used to sign in, for example, my.na-01.cloud.solarwinds.com.

  • A Fly.io account with the organization whose logs should be forwarded, and flyctl installed and authenticated.

  • Outbound access to logs.collector.<REGION>.cloud.solarwinds.com on TCP port 443. (Replace <REGION> with your account's region.)

Setup

Create an empty directory, where you will add the two files below. Both are plain text files that are copied verbatim. Only the values in the final step differ based on your organization.

Step 1: Create papertrail-token.toml

This is the Vector sink configuration. Save it as papertrail-token.toml.

Do not name it papertrail.toml. That name already exists in the image and belongs to the port-based sink.

# Papertrail / SWO Logs token destination over HTTPS.
# Region is one of na-01, na-02, eu-01, ap-01 - see your sign-in URL.
# The body is RFC 5424, which is the only shape the collector parses per event,
# so System, Program and Severity come from each log line rather than being
# pinned to the shipper. A JSON body is stored verbatim with the sender IP as
# System. TIMESTAMP is NILVALUE ("-") because the collector stamps receipt time.

[transforms.papertrail_token_syslog]
  type = "remap"
  inputs = ["log_json"]
  source = '''
    pri = { "emerg": "8", "alert": "9", "crit": "10", "error": "11",
            "warn": "12", "notice": "13", "info": "14", "debug": "15" }
    prival = string!(get(pri, [downcase(to_string(.log.level) ?? "info")]) ?? "14")
    host = to_string(.fly.app.instance) ?? ""
    if host == "" { host = "-" }
    app = to_string(.fly.app.name) ?? ""
    if app == "" { app = "fly" }
    . = { "message": "<" + prival + ">1 - " + host + " " + app + " - - - " + encode_json(.) }
  '''

[sinks.papertrail_token]
  type = "http"
  inputs = ["papertrail_token_syslog"]
  uri = "https://logs.collector.${PAPERTRAIL_REGION}.cloud.solarwinds.com/v1/logs/bulk"
  encoding.codec = "text"
  auth.strategy = "bearer"
  auth.token = "${PAPERTRAIL_TOKEN}"
  # Vector batches up to ~10 MB by default; the collector rejects any request of
  # 5 MiB or more with "413 Bulk payload too large" and the whole batch is lost.
  batch.max_bytes = 4000000

The ${PAPERTRAIL_REGION} and ${PAPERTRAIL_TOKEN} placeholders are resolved from Fly secrets when the container starts, so the token is never stored in this file.

Step 2: Create fly.toml

Save the following as fly.toml in the same directory you created. Replace the app name and region as needed. The [[files]] block is what places the sink configuration inside the container.

app = "my-fly-log-shipper"
primary_region = "ams"

[[files]]
  guest_path = "/etc/vector/sinks/papertrail-token.toml"
  local_path = "papertrail-token.toml"

[metrics]
  port = 9598
  path = "/metrics"

Use local_path rather than raw_value. flyctl encodes the file content itself, so content that is already base64-encoded is encoded a second time, and Vector fails to start with a TOML parse error.

The configuration defines no services, so the app does not receive any public IP addresses, meaning nothing is exposed.

Step 3: Create the app and set the secrets

fly apps create my-fly-log-shipper

fly secrets set -a my-fly-log-shipper \
  ACCESS_TOKEN="$(fly tokens create readonly personal)" \
  ORG=personal \
  PAPERTRAIL_TOKEN=<TOKEN> \
  PAPERTRAIL_REGION=na-01
Secret Required? (Yes/No) Description
ACCESS_TOKEN Yes

Fly read-only organization token, used to read the log stream.

Create it with fly tokens create readonly <ORG>.

ORG Yes

Fly organization slug. Defaults to personal.

List organizations with fly orgs list.

PAPERTRAIL_TOKEN Yes Ingestion token from the token-based destination.
PAPERTRAIL_REGION Yes

Data center of the organization: na-01, na-02, eu-01, or ap-01.

An incorrect value produces HTTP 401 responses.

SUBJECT No

NATS subject to subscribe to. Defaults to logs.>, which forwards every app in the organization.

Set to logs.<APP_NAME>.> to forward a single app.

QUEUE No Queue name. Used when several shipper machines run for high availability so that each event is delivered once.

Both PAPERTRAIL_TOKEN and PAPERTRAIL_REGION must be set. If either is missing, the shipper discards the sink configuration at start-up and no logs are forwarded.

Step 4: Deploy

fly deploy -a my-fly-log-shipper --image flyio/log-shipper:latest

Because the sink configuration is injected at start-up, the published image can be updated independently. Redeploy whenever the sink configuration changes.

Verify ingestion

fly logs -a my-fly-log-shipper

The start-up output confirms that the configuration was injected and loaded:

INFO Writing file: /etc/vector/sinks/papertrail-token.toml
Configured sinks:
papertrail-token
INFO vector: Vector has started.

If papertrail-token is not listed, the sink configuration was removed because PAPERTRAIL_TOKEN or PAPERTRAIL_REGION is not set.

Generate a log event from any Fly app in the organization and confirm it in the Logs Explorer. The System column shows the Fly machine ID, the Program column shows the Fly app name, and the severity reflects the log.level of the event.

Port-based destinations

The log shipper also supports a port-based Papertrail destination through its built-in papertrail sink, which is enabled by setting the PAPERTRAIL_ENDPOINT secret to a host:port value. This method applies only to accounts that already have a port-based destination.

Remove or reset the configuration

To stop forwarding logs, stop the machine or delete the app with fly apps destroy my-fly-log-shipper. To change the destination, update PAPERTRAIL_TOKEN and PAPERTRAIL_REGION and redeploy. To change the message format, edit papertrail-token.toml and redeploy so that the updated file is injected again.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.