Archiving Logs to Amazon’s S3

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

After logs age past your log retention period, they are no longer accessible. If you still need to access them, you can facilitate log archiving by sending logs to an Amazon Web Services (AWS) S3 bucket. Logs in an S3 bucket are kept forever, or until you remove them. A copy of logs sent to an S3 bucket always exists in case it is needed for historical trend analysis, auditing, or other purposes. Log archiving is a service available on Loggly Pro and Enterprise tiers. The S3 bucket is a separate product maintained through AWS. SolarWinds cannot help you create or maintain accounts with AWS. We provide an overview of how to set up archiving here and point you to Amazon’s extensive documentation on all things AWS, where necessary.

Create an account on AWS

If you don’t already have one, you’ll have to create an Amazon account.

Create an S3 bucket

After you have set up an account you need to set up a bucket to send logs to. Check out Amazon’s documentation on setting up a new bucket. After the bucket has been set up, go to Loggly to set up logging.

Give permission to write to the bucket

After you have the bucket created, in AWS:

Select the bucket in the buckets panel and click the Permissions tab.
Search for Access control list and click Edit.
Under Access for other AWS accounts, click Add grantee.
In the Grantee field, enter c1533c22deaba6b7e925aac6e1d58eeb7e2d0898e93725dc6f52000b96f48067
Select all the boxes for List/Write objects and Read/Write bucket permissions.
Click Save changes.

AWS provides additional documentation on editing bucket permissions.

Establish your new S3 bucket with Loggly

After you have set up an account and an S3 bucket, you need to provide Loggly with your credentials so it can write to the bucket. Only account owners can set up archiving within Loggly. If you are not an account owner, contact the account owner before attempting to continue. If you are the account owner, go to the account page in Loggly and select Archiving. Enter the name of the S3 Bucket you created.

Go to the Amazon Simple Storage Service documentation to find your S3 region for Signature Version 4.

If your S3 bucket is located in a region that only supports Signature Version 4, a region endpoint is required. Please refer to the link below to find out which endpoint is best for you. For example, if your bucket is in Frankfurt, you can enter <s3.eu-central-1.amazonaws.com> as your region endpoint. For more information about endpoints, see AWS service endpoints.

Set Encryption to S3-Managed if server-side encryption is enabled in your S3 bucket. Otherwise, set Encryption to None.

Loggly does not currently support Object Lock retention in S3 buckets.

Loggly sends logs to your S3 bucket

After Loggly verifies access to your S3 bucket, it writes logs in batches every half hour. After the initial setup of an S3 bucket, it could take up to 8 hours before you start seeing logs in your bucket.

Access Your Logs

You can access your logs inside S3. The logs are uploaded to the bucket using following path format: loggly/<YEAR>/<MONTH>/<DAY>/<HOUR>.<MINUTE>-<PART_NUMBER>.raw.gz. If you were looking for logs from 5/25/2020 they would be in folder loggly/2020/05/25/.

The easiest way to access logs is by going to the AWS Console > S3. Click on your bucket to view your files ordered by date. You can also use an S3 client from the command line. There are various clients available for OSX, Windows and *nix systems. At SolarWinds we use S3cmd, an open source command line tool for managing data stored with S3.

If logs are deleted from the search index, they are no longer accessible from the Loggly site.