Send logs from Kubernetes clusters running on AWS Fargate

When you deploy the SWO K8s Collector to an AWS EKS cluster with a Fargate profile, by default, the SWO K8s Collector does not collect any logs from pods running on Fargate. If you wish to collect logs from pods running on Fargate, you must set up the log collection.

Prerequisites

Before you begin, make sure:

• You have an AWS EKS cluster with a Fargate profile.

  • Kubernetes version 1.23 or later.

  • Clusters using both EC2 node groups and Fargate profiles are also supported.

• You have permissions to configure the Fargate pod execution role.

• You have reviewed the instructions for deploying the SWO K8s Collector.

Set up Fargate logs

1. Configure the EKS cluster to send Fargate logs to AWS CloudWatch. For manual configuration, see Set up log collection manually. For automatic configuration, see Set up log collection using the SWO K8s Collector.

   • Set up log collection manually

     By following the steps below, you will create and manage Fargate logging configuration in the cluster outside of the SWO K8s Collector.

     1. Follow the instructions in Fargate logging to configure sending Fargate logs to CloudWatch.

        1. In step 2, append the following filter into the filters.conf section of the ConfigMap:

           [FILTER]
           Name modify
           Match *
           Add sw.k8s.cluster.uid <clusterUid>
           Add sw.k8s.log.type container

        2. Replace <clusterUid> with the same identifier you use for the cluster in SWO K8s Collector. This ensures the logs will be associated with the cluster in SolarWinds Observability SaaS.

        3. Optionally, you can adjust the log group name in section output.conf to better identity that the source of logs is the monitored cluster.

        4. Note the name of the log group in section output.conf.

        5. Start or restart all pods from which you want to receive logs. The created ConfigMap is applied only to pods that start after the modification.

           At least one pod has to be started or restarted before the next step. Otherwise, the log group may be missing.

     2. Follow instructions in Add logs from AWS to set up log forwarding from CloudWatch to SolarWinds Observability SaaS. When configuring the trigger, include the log group name you noted down in the previous step. The API token can be the same as for the SWO K8s Collector, or you can create a new one.

        If you have already configured log forwarding, for example, for collecting logs from EC2 instances, you can either set up a new Lambda specifically for cluster logs collection, or you can add an additional trigger to an already existing one.

   • Set up log collection using the SWO K8s Collector

     By following the instructions below, you will set up the SWO K8s Collector to create and manage the Fargate logging configuration in the cluster.

     1. Before installing the SWO K8s Collector Helm chart to the cluster, adjust the values.yaml file by adding the following setting:

        aws_fargate:
          enabled: true
          logs:
            enabled: true
            region: us-east-2 # use the region where the cluster is running.

     2. Install or update the SWO K8s Collector to the cluster. For more information, see Add a Kubernetes cluster.

     3. Download the CloudWatch IAM policy. You can also view the policy on GitHub:

        curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json 

     4. Create the IAM policy using the file downloaded in the previous step:

        aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

     5. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace 111122223333 with your account ID. Replace AmazonEKSFargatePodExecutionRole with your pod execution role (for more information, see Create a Fargate Pod execution role). If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, replace arn:aws: with arn:aws-us-gov::

        aws iam attach-role-policy \
           --policy-arn arn:aws:iam::111122223333:policy/eks-fargate-logging-policy \
           --role-name AmazonEKSFargatePodExecutionRole 

     6. Start or restart all pods from which you want to receive logs. The created ConfigMap is applied only to pods that start after the modification.

        At least one pod has to be started or restarted before the next step. Otherwise, the log group may be missing.

     7. Follow the instructions in Add logs from AWS to set up log forwarding from Amazon CloudWatch to SWO K8s Collector. When configuring the trigger, include the /swo/fargate/<cluster_name> log group. The API token can be the same as for the SWO K8s Collector, or you can create a new one.

        If you have already configured log forwarding, for example, for collecting logs from EC2 instances, you can either set up a new Lambda specifically for cluster logs collection, or you can add an additional trigger to the already existing one.

2. Verify that the logs are available in SolarWinds Observability SaaS. By default, all container logs are collected without any filtering.

Filter Fargate logs

You can filter logs sent to SolarWinds Observability SaaS using the following methods:

• Apply event filtering to the Lambda that is forwarding logs to SolarWinds Observability SaaS. For more information, see Lambda event filtering.

• Apply additional FluentBit filters in the ConfigMap that is deployed to the cluster during setup.

  Any changes to the ConfigMap will be applied only to pods that start after the modification.

Stop log collection

To stop sending logs from a Fargate EKS cluster to SolarWinds Observability SaaS, follow the instructions below.

1. Remove the Fargate logging configuration from the cluster.

   • If the Fargate logging configuration in the cluster is managed by the SWO K8s Collector, adjust the values.yaml file by removing the aws_fargate section and re-deploying the SWO K8s Collector to the cluster.

   • If the Fargate logging configuration in the cluster was created manually, remove the aws-logging ConfigMap from the aws-observability namespace.

2. (Optional) Remove the CloudWatch IAM policy applied to the pod execution role during the setup.

3. Remove the Lambda that forwards the logs to SolarWinds Observability SaaS.