Serverless

Major cloud computing vendors offer services that enable what is commonly referred to as serverless, where the provisioning and scaling of application infrastructure is handled automatically and typically in ephemeral resources. This makes efficient use of the underlying hardware, and allows the application developer to focus on business logic while reducing cost by paying only for needed capacity.

With AppOptics, you can monitor serverless applications deployed in AWS Lambda in several ways. There is an AWS CloudWatch Integration that can pull Lambda metrics, and now you can get enhanced metrics and distributed traces via our Lambda Forwarder and APM agents.

To get started, follow the steps below to install and configure the Forwarder to get enhanced metrics for your Lambda function. Once the Forward is set up, you can enable tracing for a Lambda function in one of the supported runtimes.

AppOptics Lambda Forwarder

The forwarder is provided as a Serverless Application Repository application named AppOptics-Lambda-Forwarder. Once installed as a Lambda function in your account, it is invoked by the CloudWatch Logs subscription filter(s) that you configure on the log group of your Lambda function. The forwarder ingests these log events to report enhanced metrics for your function; if tracing is enabled for your Lambda function, the forwarder also transforms the trace data found in the log events and reports them to AppOptics.

Prerequisites

We provide an install script to help set up the forwarder and subscription filter(s). This script needs to be run as an AWS user with a specific minimum IAM permissions.

The AWS CLI version 2 tool is a requirement for the install script and needs to be configured appropriately to perform the various CLI commands. See Configuring the AWS CLI in the Amazon Web Services documentation for more information.

Installation

The recommended way to set up the forwarder is to download the installation script we provide and run it in interactive mode, which pulls the AppOptics-Lambda-Forwarder application from Serverless Application Repository and installs it as a Lambda function in your account. The following commands will download and run the install script in interactive mode:

curl -sSO https://files.appoptics.com/appoptics-lambda-forwarder.sh
sh appoptics-lambda-forwarder.sh --action=install

You will be prompted to enter your AppOptics API Token, and one or more optional service mappings, which can also be set after the forwarder installation.

On successful installation, you should see the forwarder as a Lambda function that contains the string AoForwarderFunction in its function name.

Some helpful installer script options are as follows:

• The -h option prints all possible options and should help with the various tasks.

• For debugging purposes the script can be run with the --debug option. This will print all AWS CLI commands that are performed.

Subscription Filter Configuration

During or after the forwarder installation, you must set a subscription filter on the log group for your Lambda function to deliver its log events to the forwarder and so that the forwarder reports to the correct AppOptics service. Do this by using the installer script’s service mapping actions.

During installation

Interactive mode

If the forwarder is installed in interactive mode, the script will ask for one or more service mappings. A service mapping uses the form <function>:<service>[:<alias>] and describes the relationship between a Lambda function, AppOptics service, and optionally a host name alias.

Example of a service mapping in interactive mode:

Service Mapping (optional, format <function>:<service>[:<alias>], empty line when done): my-function:my-service

Non-interactive mode

If the forwarder is installed in non-interactive mode, you can pass in the service mapping as a parameter:

sh appoptics-lambda-forwarder.sh --action=install --interactive=no --api-token=my-api-token --service-mappings=my-function:my-service

After installation

Run the forwarder script with --action=add-mapping to add one more service mappings after installation. The format is the same as adding a service mapping during the installation.

Interactive mode

sh appoptics-lambda-forwarder.sh --action=add-mapping
Testing AWS CLI...done
Service Mapping (format <function>:<service>[:<alias>], empty line when done): my-function:my-service

Non-interactive mode

sh appoptics-lambda-forwarder.sh --action=add-mapping --interactive=no --service-mappings=my-function:my-service

Forwarder Configuration

The forwarder exposes the following configurable options as Lambda function environment variables. See Using AWS Lambda environment variables in the Amazon Web Services documentation for more information about environment variables.

The mandatory APPOPTICS_API_TOKEN is automatically set during installation via the CLI script, so these variables do not need to be modified in standard situations. You could choose to modify these settings to disable the enhanced metrics or to enable forwarder debugging messages.

• APPOPTICS_API_TOKEN
  • API token used to communicate with AppOptics
  • the default value is set during installation and can be changed later
  • mandatory
• APPOPTICS_REPORT_METRICS
  • a flag to enable/disable processing of the enhanced metrics
  • values are true and false
  • optional, default is true
• APPOPTICS_FORWARDER_DEBUGGING
  • a flag to enable debug logging for diagnostics purposes
  • values are false, true, and raw:
    • false: no extra logging, debugging turned off
    • true: log AO events and metrics that will be sent to the collector (BSON decoded)
    • raw: log raw JSON string that gets pushed by the CloudWatch subscription filter
  • optional, default is false

Enhanced Metrics

Once your Lambda function’s log group has been configured with the subscription filter, the forwarder will start reporting the following metrics.

Metrics

Each metric is the aggregate value along with the count for the reporting period.

Name	Description
AWS.Lambda.BilledDurationMS	The function invocation's billed duration in milliseconds. This is taken from the Report Log Billed Duration field.
AWS.Lambda.DurationMS	The function duration in milliseconds. This is taken from the Report Log Duration field.
AWS.Lambda.InitDurationMS	The time taken to initialize the function execution environment for a "cold start" invocation, in milliseconds. This is taken from the Report Log Init Duration field.
AWS.Lambda.MaxMemoryUsedMB	The memory used by the function in megabytes. This is taken from the Report Log Max Memory Used field.
AWS.Lambda.MemorySizeMB	The memory configured for the function in megabytes. This is taken from the Report Log Memory Size field.

Available Tags

Name	Description
@host	See Host aliases and the @host tag
environment	See Environments
hostname	Hostname which by default is the Lambda function name, but can be customized in the subscription filter.
service	AppOptics service name
version	Lambda function version
functionname	Lambda function name
region	Lambda function region

You can disable the reporting of these metrics via the Forwarder Configuration setting APPOPTICS_REPORT_METRICS=false.

Uninstall the Forwarder

The Forwarder and service mappings can be uninstalled using the appoptics-lambda-forwarder.sh script. Run it with the help option for details:

sh appoptics-lambda-forwarder.sh --help

We recommend that you first list and remove all service mappings, and then uninstall the Forwarder. For example:

sh appoptics-lambda-forwarder.sh --action=remove-mapping --service-mappings=funcA:serviceA,funcB:serviceB
sh appoptics-lambda-forwarder.sh --action=remove

Tracing

In the AWS Lambda environment, APM agents work by writing trace data to your function’s CloudWatch Logs log group. In order for the logged trace data to be transformed and reported to AppOptics, the AppOptics Lambda forwarder needs to be installed and properly configured, and a subscription filter needs to be configured for your Lambda function. Ensure that these prerequisites are met before proceeding with instrumenting your function.

When the agent is enabled and running in your Lambda function, you should see the trace data in CloudWatch Logs entries with the format below:

{
   "ao-data":{
      "events":[
         "5AEAAAJ...",
         "WQUAAAJ...",
         "HgEAAAJ..."
      ],
      "metrics":[
         "yAIAABV..."
      ]
   }
}

The following features are not supported by APM agents in the AWS Lambda environment:

• Runtime Metrics
• Transaction Filtering
• Trigger Trace

Enabling

See the links below on how to enable Lambda function instrumentation for each agent:

• Serverless (legacy agent) with the Go Agent
• Serverless (legacy agent) with the Java Agent
• Serverless (legacy agent) with the Node.js Agent
• Serverless (legacy agent) with the Python Agent

Sample Rate

The APM agents sample Lambda function requests (aka invocations) for tracing similar to the mechanism in non-AWS Lambda environments. However, instead of adaptive sampling where the rate is automatically adjusted, in a Lambda function the agents use a fixed rate. If your instrumented function is under heavy load and thus tracing at a high rate, it decreases the traces per minute for any non-Lambda services in your account.

Contact the AppOptics customer support team for help assessing factors such as expected function invocation rate and concurrency; these factors need to be considered before using internal settings to adjust the rate.