Ingesting logs from Amazon CloudWatch (Papertrail)
You are able to ingest logs directly from Amazon CloudWatch to Papertrail without employing the SolarWinds Observability Agent.
How it works
-
AWS captures log lines written by your application and stores them in a CloudWatch log group.
-
CloudWatch batches recent log events. Every few seconds, it packages up the new lines that arrived in the log group, compresses the batch using gzip, and base64-encodes it.
-
Based on a configured subscription filter, CloudWatch invokes a Lambda function, which receives the compressed package as its input.
-
The Lambda function unpacks and forwards. First, it decompresses and decodes the batch, restoring the individual log lines plus metadata, such as log group name, log stream name, and region. Then, it makes a single HTTPS
POSTto the Papertrail HTTP logs collector, sending all the log lines in the batch as the body, with the CloudWatch metadata attached as HTTP headers.
How Papertrail fills System and Program
The Papertrail Logs Explorer fills two key columns for each log line:
-
System, which is populated from
host.name. -
Program, which is populated from
service.name.
Both System and Program are set via the X-Otel-Resource-Attr HTTP header and apply to the whole batch. For CloudWatch logs, the natural mapping is below:
| Papertrail field | CloudWatch source | Example |
|---|---|---|
System (host.name) |
Log group name | /aws/lambda/my-function
|
Program (service.name) |
Log stream name | 2026/09/24/[$LATEST]abcdef
|
The log group identifies which service the logs come from. The log stream identifies which specific instance or execution produced the logs. Each Lambda invocation or ECS task gets its own stream. Both values are available directly in the CloudWatch event, so no extra configuration is needed.
Prerequisites for ingestion
Before you can begin ingesting logs from Amazon CloudWatch, you will need the following:
-
A Papertrail ingestion token
To get a destination and token in Papertrail, add a new log destination. The ingestion token is generated along with the destination. Use the collector endpoints for your account's region:
-
Syslog (TLS):
syslog.collector.<REGION>.cloud.solarwinds.com:6514 -
HTTPS:
https://logs.collector.<REGION>.cloud.solarwinds.com/v1/logs/bulk
Replace
<REGION>with the data center your organization uses:na-01,eu-01, orap-01. -
-
AWS CLI must be installed and configured with sufficient permissions, including Lambda, IAM, and CloudWatch Logs.
-
Your AWS account ID and the region where your log groups reside.
Setup log ingestion
Step 1: Write the Lambda function.
Create a file named lambda_function.py with the following content:
import base64
import gzip
import json
import os
import urllib.request
import urllib.error
HOSTNAME = os.environ["SWO_HOSTNAME"]
TOKEN = os.environ["SWO_TOKEN"]
def lambda_handler(event, context):
compressed = base64.b64decode(event["awslogs"]["data"])
payload = json.loads(gzip.decompress(compressed))
log_group = payload["logGroup"]
log_stream = payload["logStream"]
region = context.invoked_function_arn.split(":")[3]
body = "\n".join(e["message"] for e in payload["logEvents"])
url = f"https://{HOSTNAME}/v1/logs/bulk"
req = urllib.request.Request(
url,
data=body.encode("utf-8"),
method="POST",
headers={
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/octet-stream",
"X-Otel-Resource-Attr": (
f"cloud.provider=aws, "
f"cloud.region={region}, "
f"host.name={log_group}, "
f"service.name={log_stream}, "
f"aws.log.group={log_group}, "
f"aws.log.stream={log_stream}"
),
},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
if resp.status != 202:
raise RuntimeError(f"Unexpected status {resp.status}")
except urllib.error.HTTPError as e:
raise RuntimeError(f"HTTP {e.code}: {e.read().decode()}") from e
host.name and service.name are set from the CloudWatch event automatically. No manual configuration is required for each log group. A single Lambda deployment can serve multiple log groups.
Step 2: Create an IAM role for the Lambda function.
aws iam create-role \
--role-name swo-cloudwatch-forwarder \
--assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy \
--role-name swo-cloudwatch-forwarder \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
The AWSLambdaBasicExecutionRole policy allows the Lambda to write its own execution logs to CloudWatch.
Step 3: Deploy the Lambda function.
Package the function and create it. Replace the placeholders with your actual values:
| Placeholder | Actual value |
|---|---|
YOUR_ACCOUNT_ID
|
Your 12-digit AWS account ID |
YOUR_REGION
|
The AWS region where the log group lives Example: |
YOUR_COLLECTOR_HOSTNAME
|
The collector hostname for your data center Example: |
YOUR_INGESTION_TOKEN
|
The Papertrail ingestion token you created in the Prerequisites. |
zip function.zip lambda_function.py
aws lambda create-function \
--region YOUR_REGION \
--function-name swo-cloudwatch-forwarder \
--runtime python3.12 \
--role arn:aws:iam::YOUR_ACCOUNT_ID:role/swo-cloudwatch-forwarder \
--handler lambda_function.lambda_handler \
--zip-file fileb://function.zip \
--timeout 30 \
--environment '{"Variables":{"SWO_HOSTNAME":"YOUR_COLLECTOR_HOSTNAME","SWO_TOKEN":"YOUR_INGESTION_TOKEN"}}'
If the command fails with a role-not-assumable error, wait 10 seconds and retry. The IAM role propagation can take a minute.
Step 4: Allow CloudWatch to invoke the Lambda function.
Run the following to grant CloudWatch Logs permission to invoke the Lambda function for any log group in your account:
aws lambda add-permission \ --region YOUR_REGION \ --function-name swo-cloudwatch-forwarder \ --statement-id AllowCloudWatchLogs \ --action lambda:InvokeFunction \ --principal logs.amazonaws.com \ --source-account YOUR_ACCOUNT_ID
Replace YOUR_REGION and YOUR_ACCOUNT_ID.
You only have to run this once. It covers all the log groups you will add.
Step 5: Create a subscription filter.
The subscription filter tells CloudWatch to start sending new log events to the Lambda function. Replace YOUR_REGION, YOUR_ACCOUNT_ID, and YOUR_LOG_GROUP. Repeat for each log group.
aws logs put-subscription-filter \ --region YOUR_REGION \ --log-group-name "YOUR_LOG_GROUP" \ --filter-name swo-forwarder \ --filter-pattern "" \ --destination-arn arn:aws:lambda:YOUR_REGION:YOUR_ACCOUNT_ID:function:swo-cloudwatch-forwarder
The --filter-pattern "" forwards all log events. See CloudWatch filter pattern syntax if you want to forward only a subset.
After completing this step, new log events written to YOUR_LOG_GROUP will start flowing to Papertrail within seconds.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.