Archiving Logs to Amazon’s S3
Once logs age past your log retention period, they’re no longer accessible. What if you need them? Well, we can facilitate log archiving by sending logs to an Amazon Web Services (AWS) S3 bucket. Logs in an S3 bucket are kept forever, or until you remove them, so you’ll always have a copy handy if you need them for historical trend analysis, auditing, or other purposes. Log archiving is a service that’s available on our Pro and Enterprise tier. The S3 bucket is a separate product maintained through AWS. We cannot help create or maintain accounts with AWS. We’ll give you the overview of how to set up archiving here and point you to Amazon’s extensive documentation on all things AWS, where necessary.
Step 1: Create an account on AWS
If you don’t already have one, you’ll have to create an Amazon account.
Step 2: Create an S3 bucket
After you’ve set up an account you’ll have to set up a bucket that we can send logs to. Check out Amazon’s documentation on setting up a new bucket. Ignore the "Set Up Logging" button. You’ll come back into our product to do that.
Step 3: Give us permission within AWS to write to the bucket
Once you have the bucket created:
- Select the bucket in the buckets panel and click the "Permissions" tab.
- Click the "Add account" button
- In the "Account" field, enter c1533c22deaba6b7e925aac6e1d58eeb7e2d0898e93725dc6f52000b96f48067
- Check all the boxes for "List/Write objects" and "Read/Write bucket permissions".
- Click "Save" button
Should you need further help with this, AWS has documentation on editing bucket permissions.
Step 4: Establish your new S3 bucket with Loggly
Now we come back to Loggly. Once you’ve set up an account and an S3 bucket, you’ll need to give us your credentials so we can write to the bucket. Only account owners can set up archiving within Loggly. If that’s not you, contact the account owner before you can continue. If you are the account owner go to the account page in Loggly and select archiving. Enter the name of the S3 Bucket you created.
If your S3 bucket is located in a region that only supports Signature Version 4, a region endpoint is required. Please refer to the link below to find out which endpoint is best for you. For example, if your bucket is in Frankfurt, you can enter
<s3.eu-central-1.amazonaws.com> as your region endpoint.
Step 5: We send logs to your S3 bucket.
That’s all you need to do. Once we verify access to your S3 bucket, we’ll write logs in batches every half hour. After you first set-up an S3 bucket it may take up to 8 hours before you start seeing logs in your bucket.
Step 6: Access Your Logs
When you’re ready, you can access your logs inside S3. The logs are uploaded to the bucket using following path format:
loggly/<YEAR>/<MONTH>/<DAY>/<HOUR>.<MINUTE>-<PART_NUMBER>.raw.gz. If you look for logs from 5/25/2020 they will be in folder
The easiest way is to log into your AWS Console and then open the S3 service. Click on your bucket to view your files ordered by date. You can also use an S3 client from the command line. There are various clients available for OSX, Windows and *nix systems. Here at Loggly, we use S3cmd, an open source command line tool for managing data stored with S3. Simple, no? Remember, once logs are deleted from our search index, they are no longer accessible from our site.
When the APM Integrated Experience is enabled, Loggly shares a common navigation and settings with the other integrated experiences' products. How you navigate Loggly and access its features may vary from these instructions. For more information, go to the APM Integrated Experience documentation.