Send Amazon ELB Classic Logs to Loggly

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

You can push Amazon Elastic Load Balancer (ELB) Classic logs to Loggly using an AWS Lambda Script. The script converts the ELB logs written to S3 into JSON format and then sends them to Loggly. ELB (Application) Logs are not supported at this time.

Alternatively, you may use Loggly's S3 ingestion service, which directly ingests ELB Classic logs into Loggly without requiring a Lambda function. The Loggly app pack for ELB contains popular dashboards and saved searches. It currently supports only ingesting logs using the S3 ingestion service and not a Lambda script.

Set up AWS

1. Get the Lambda Code

Clone the git repo

git clone https://github.com/cboscolo/elb2loggly.git
cd elb2loggly

Edit elb2loggly.js with your Loggly customer token and optional log tags. (You can set these as tags on the S3 Bucket that contains the logs.)

Install require npm packages.

npm install

Zip up your code

zip -r elb2loggly.zip elb2loggly.js node_modules

The resulting zip (elb2loggly.zip) is what you upload to AWS in Step 2 below.

2. Configure the Lambda Function

Go to the AWS Lambda Console. Click Create a Lambda function. Choose Upload a .ZIP file. Fill the following details.

Name: elb2loggly
Upload lambda function (zip file you made above in Step 1)
Handler*: elb2loggly.handler
Role*: In the drop down click "S3 execution role". (This opens a new window to create the role, click Allow)
Set memory at 128MB
Set Timer to 10 seconds. 

Configure Event Source to call elb2loggly when logs are added to the S3 bucket. Go to the AWS Lambda Console. Make sure the elb2loggly lambda function is selected, then click Actions > Add event source. Fill the following details.

Event source type: S3
Bucket: Choose the bucket that contains your ELB logs.
Event type: ObjectCreated (All) 

3. Configure ELB Logging

Under Load Balancers, go to the EC2 Management Console. Choose your ELB and scroll down to Access Logs. Click edit, and then set interval to 5 minutes and S3 Location to the bucket where you want to put your logs.

4. Tag Your Logs in S3

The Lamba script will look for your customer token as an S3 tag, which it uses to send data to your account. It also adds tags for Loggly, which makes the logs easier to find in a search. Using the S3 Management Console, click the bucket that contains your ELB logs. Under Properties > Tags, add the following tag:

Key: loggly-customer-token
Value: TOKEN
Key: loggly-tag
Value: aws-elb 

Replace TOKEN with your customer token from the source setup page.

5. Verify Events

Search Loggly events with the tag aws-elb over the past 30 minutes. It may take few minutes to index the events. If indexing doesn’t work, see Troubleshooting below.

tag:aws-elb 

Troubleshooting

If you don’t see any data show up in the verification step, then check for these common problems.

• Wait a few minutes in case indexing needs to catch up
• Make sure you’ve included your own customer token
• Make sure you have configured same roles as mentioned above.
• Go to your Lamda function in AWS Console and click on View logs in Cloudwatch in the Monitoring tab to view logs.
• If you still do not see your log, search for the "error" field with tag as aws-elb in your past 30 minutes logs.
• Search or post your own Amazon ELB logging questions in the community forum.

Learn how Loggly can help with all your AWS Log Management

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.