Archiving Logs to Amazon’s S3
Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.
After logs age past your log retention period, they are no longer accessible. If you still need to access them, you can facilitate log archiving by sending logs to an Amazon Web Services (AWS) S3 bucket. Logs in an S3 bucket are kept forever, or until you remove them. A copy of logs sent to an S3 bucket always exists in case it is needed for historical trend analysis, auditing, or other purposes. Log archiving is a service available on Loggly Pro and Enterprise tiers. The S3 bucket is a separate product maintained through AWS. SolarWinds cannot help you create or maintain accounts with AWS. We provide an overview of how to set up archiving here and point you to Amazon’s extensive documentation on all things AWS, where necessary.
Create an account on AWS
If you don’t already have one, you’ll have to create an Amazon account.
Create an S3 bucket
After you have set up an account you need to set up a bucket to send logs to. Check out Amazon’s documentation on setting up a new bucket. After the bucket has been set up, go to Loggly to set up logging.
Give permission to write to the bucket
After you have the bucket created, in AWS:
- Select the bucket in the buckets panel and click the Permissions tab.
- Click Add account
- In the Account field, enter
- Select all the boxes for List/Write objects and Read/Write bucket permissions.
- Click Save.
AWS provides additional documentation on editing bucket permissions.
Establish your new S3 bucket with Loggly
After you have set up an account and an S3 bucket, you need to provide Loggly with your credentials so it can write to the bucket. Only account owners can set up archiving within Loggly. If you are not an account owner, contact the account owner before attempting to continue. If you are the account owner go to the account page in Loggly and select Archiving. Enter the name of the S3 Bucket you created.
If your S3 bucket is located in a region that only supports Signature Version 4, a region endpoint is required. Please refer to the link below to find out which endpoint is best for you. For example, if your bucket is in Frankfurt, you can enter
<s3.eu-central-1.amazonaws.com> as your region endpoint. For more information about endpoints, see AWS service endpoints.
Loggly sends logs to your S3 bucket
After Loggly verifies access to your S3 bucket, it writes logs in batches every half hour. After the initial setup of an S3 bucket, it could take up to 8 hours before you start seeing logs in your bucket.
Access Your Logs
You can access your logs inside S3. The logs are uploaded to the bucket using following path format:
loggly/<YEAR>/<MONTH>/<DAY>/<HOUR>.<MINUTE>-<PART_NUMBER>.raw.gz. If you were looking for logs from 5/25/2020 they would be in folder
The easiest way to access logs is by going to the AWS Console > S3. Click on your bucket to view your files ordered by date. You can also use an S3 client from the command line. There are various clients available for OSX, Windows and *nix systems. At SolarWinds we use S3cmd, an open source command line tool for managing data stored with S3.
If logs are deleted from the search index, they are no longer accessible from the Loggly site.