Documentation forSolarWinds Observability SaaS

Add logs from AWS

This topic covers using AWS Lambda to collect logs from your AWS cloud account. For information about the AWS Lambda metrics that can be gathered when monitoring your AWS cloud account, see Lambda metrics. For information about AWS Lambda instrumentation for service entities, see AWS Lambda instrumentation.

You can use SolarWinds Observability SaaS to monitor logs from your Amazon AWS cloud.

To configure logs, you need a Lambda function for your AWS account. You can deploy this using the AWS Cloud Account wizard.

The Lambda function is deployed by the CloudFormation template and supports CloudTrail Logs and VPC Flow Logs using CloudWatch Log Groups for providing the data to the Lambda function.

Configure the Lambda function

The Lambda function you configure is available only in the selected AWS region.

  1. In SolarWinds Observability SaaS, click Settings > Cloud Accounts > Amazon Web Services.

  2. Click the cloud account name to open the AWS Cloud Account wizard. Click Next to proceed to the Region & Services screen.

  3. On the Regions & Services screen, select the regions. Click Next.

  4. Follow the instructions on the Logs screen to deploy a Lambda function using the CloudFormation template.

Identify and upgrade outdated log-forwarding Lambda functions in SolarWinds Observability SaaS

You can identify and upgrade outdated log-forwarding Lambda functions using the AWS CloudFormation stack update flow launched from the Edit AWS Cloud Account wizard.

In SolarWinds Observability SaaS, click Settings > Cloud Accounts > Amazon Web Services. Click the cloud account to open the Edit AWS Cloud Account wizard. Navigate to the Logs step and expand a region to review the table of existing SolarWinds log-forwarding Lambda functions.

Information in the table

  • Resource name.

  • Version: Derived from Lambda tags.

    When there is no version tag present, SolarWinds Observability SaaS cannot confirm the version is current and treats it as outdated. The Lambda function is also treated as outdated when there is a newer version available.

  • Status (icons):

    • Green check mark: Up to date.

    • Yellow warning: Upgrade available or resource is legacy.

      A Lambda function is treated as legacy when it does not have CloudFormation stack tags, for example, aws:cloudformation:stack-name.

      Legacy Lambda functions are not associated with a CloudFormation stack. Even though they are functional, the wizard cannot upgrade them using a stack update.

    • Grey dash: Region does not have any Lambda function configured.

  • Actions:

    • View in AWS: Opens the Lambda function details page in the AWS Console.

    • Update in AWS: Opens the CloudFormation Update stack page in the AWS Console with the template URL pre-filled.

    • Delete in AWS: Displayed for legacy Lambdas that cannot be upgraded.

    • Configure in AWS: Displayed for regions without any configured Lambda.

Upgrade the Lambda function using CloudFormation stack update

  1. In SolarWinds Observability SaaS, click Settings > Cloud Accounts > Amazon Web Services.

  2. Click the cloud account to open the Edit AWS Cloud Account wizard, and navigate to the Logs step.

  3. Expand the region and on the row marked with a warning icon, click Update in AWS.

    This redirects you to the CloudFormation Update stack page in the AWS Console.

    1. Verify you are in the correct region.

    2. Verify the template URL is already pre-filled.

  4. Continue through the update steps, specifying correct parameters.

  5. Wait for the stack to reach the UPDATE_COMPLETE status.

  6. Return to the Edit AWS Cloud Account wizard in SolarWinds Observability SaaS and click Refresh or Refresh all.

    The resource shows a green check mark and the action switches to View in AWS.

Upgrade legacy Lambda functions

If SolarWinds Observability SaaS displays Cannot upgrade and the action is Delete in AWS:

  1. Click Delete in AWS to open the Lambda function page in the AWS Console.

  2. Delete the legacy Lambda function.

  3. Return to the SolarWinds Observability Edit AWS Cloud Account wizard and click Configure in AWS to deploy a new CloudFormation-managed version.

Troubleshooting upgrades

Upgrade fails due to subscription filter limits

CloudWatch Logs has a limit on subscription filters per log group. The AWS limit is currently 2. If the upgrade process attempts to create an additional subscription filter while the limit is reached, the stack update fails.

Resolve the issue:

  1. Go to CloudWatch > Log groups.

  2. Open the relevant log group.

  3. Click Subscription filters.

  4. If you see more than one filter pointing to the log-forwarding Lambda function, remove the additional or old filters.

  5. Retry the CloudFormation stack update.

Update link opens but looks wrong

If Update in AWS opens a page with an empty stack ID or missing template URL:

  1. Verify the Lambda function has CloudFormation stack tags. For example, aws:cloudformation:stack-name.

  2. Verify the Edit AWS Cloud Account wizard was opened on the correct account, role, and region.

  3. Refresh the wizard and try again.

Version doesn’t show

If the Version column shows a dash, the Lambda function has no recognized version tag, or it is still treated as outdated.

Resolve the issue:

  1. Verify the Lambda function has recognized CloudFormation stack tags. For example, aws:cloudformation:stack-name.

  2. In the SolarWinds Observability Edit AWS Cloud Account wizard, click Update in AWS to upgrade the Lambda function using CloudFormation stack update.

    In case of legacy Lambda functions, delete and redeploy the Lambda function.

Add triggers for log groups to be monitored with SolarWinds Observability SaaS

For each log group that you want to monitor, configure the log events in a log group to trigger the Lambda function. If you are using EKS or CloudTrail, send the logs and traced events to CloudWatch first, then set up your trigger.

After you complete the configuration, the AWS logs can be seen in the Logs Explorer.

Capture traced events

Traced events and logs from Amazon services, such as EKS and CloudTrail, can be sent to a log group in CloudWatch Logs. Once the logs are configured to go to CloudWatch Logs, a trigger can be used to send the CloudWatch logs to SolarWinds Observability SaaS.

Capture CloudTrail events

If you want to capture traced events from CloudTrail service and send them to SolarWinds Observability SaaS using CloudWatch log events, configure the CloudTrail events capture.

  1. In your AWS account, navigate to CloudTrail > Dashboard.

  2. Create a trail and specify required parameters for a new trail, or navigate to an existing trail.

  3. Under CloudWatch Logs, click the Enabled box and specify the target log group and access role.

Capture EKS logs

To make EKS logs available to CloudWatch and send them to SolarWinds Observability SaaS, see Logging for Amazon EKS - AWS Prescriptive Guidance in the Amazon AWS documentation.

After setting up, the following log groups will be available:

  • /aws/containerinsights/<CLUSTER_NAME>/performance
  • /aws/eks/<CLUSTER_NAME>/cluster
  • /aws/containerinsights/<CLUSTER_NAME>/application
  • /aws/containerinsights/<CLUSTER_NAME>/host
  • /aws/containerinsights/<CLUSTER_NAME>/dataplane

Configure log events to trigger the Lambda function

Configure log events that will trigger the Lambda function. With the newly created Lambda function open do the following:

  1. Click Add trigger.

  2. Select CloudWatch Logs in the drop-down menu.

  3. Either select the log group that serves as the event source or define a filter pattern.

  4. Add more triggers, if appropriate.

(Optional) Secure the API token and OTLP endpoint parameters

To prevent the API token from being viewed in the Lambda web console, encrypt the token.

  1. In the Lambda web console, navigate to SendLogsFunction in the Functions section.

  2. Click the Configuration tab, and click Edit in the Environment variables section.

  3. Click Enable helpers for encryption in transit and click Encrypt for the API_TOKEN environment variable. The Encryption in transit window opens.

  4. Select your AWS KMS key, copy the text from the Execution role policy, and click Encrypt.

  5. Repeat this operation for the OTLP_ENDPOINT environment variable.

  6. Set the USE_ENCRYPTION environment variable to yes and click Save.

  7. Navigate to the SendLogsFunction execution role in the IAM web console. This can be found in Configuration > Permissions > Execution role.

  8. In the Permissions tab, choose Add permissions > Create inline policy.

  9. In the Create policy window, activate the JSON tab and paste the execution policy previously copied from the Environment variable encryption dialog.

  10. Click Review policy, specify the policy name, and click Create policy.

  11. Navigate to the SendLogsFunction page and go to the Test tab.

  12. Select CloudWatch Logs from the drop-down and click Test to verify that encrypted parameters can be accessed by the function.

Remove the Lambda function

If the Lambda function was created by the Add AWS Cloud Account wizard, the function will be removed when you remove the AWS cloud account integration.

If you added the Lambda function manually and do not want to monitor the AWS cloud account anymore, remove the send-logs-app stack. Calling the function consumes resources and is subject to charge.

To remove the send-logs-app stack:

  1. Log in to your AWS Console.
  2. Navigate to CloudFormation > Stacks page
  3. Find the send-logs-app stack, click Delete.

The application stack and Lambda function associated with it will be removed from the account.