Lambda Troubleshooting Framework- Installation Details by Method

Lambda Troubleshooting Framework - Installation Details by Method

Planning for Install

What follows are some details about our Lambda monitoring product that will be relevant to planning for the install.

https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/compatibility-requirements-aws-lambda-monitoring/

Account linking with either API polling or Metric Streams integration (choose one, not both).

One you have an integration, it’s a good idea to run through an example. Our agents + Extension for shipping payloads are built into our Lambda layers. Note: The Extension layer is just the Extension without an agent. It is mostly for Go and .NET functions since we don’t have a layer with an agent for them.

You’ll get the most detail from our docs on what features are provided. There are two levels of serverless monitoring:

  1. Our AWS Serverless integration provides basic meta data about your functions: names, invocation counts, basic CloudWatch data. This data is available in the ServerlessSample event type for API polling integrations, and Metric for Metric Streams integrations. Metric data is also mapped to ServerlessSample so that it can still show up there.

    Monitoring AWS Lambda with Serverless monitoring | New Relic Documentation

    Note: If you choose the Metric Streams type of integration, our tooling for Lambda monitoring has not be updated to work with this yet. You’ll want to avoid using the CLI to newrelic-lambda integrations install or the Serverless plugin to enableIntegration: true since they will try to create an API polling type of integration. You could end up with two integrations for the same AWS account, which can cause issues.

  2. Our Lambda monitoring provides more detailed metrics from function invocations, similar to what you’d see from an APM agent. Note, we don’t store data in transaction events like with APM. Instead we use the Span and AwsLambdaInvocation events.

    Enable monitoring for AWS Lambda | New Relic Documentation

These Lambda Troubleshooting Framework posts goes into more detail:

With Lambda monitoring, you can see durations by invocation and it is also possible to generate a NRQL query to sort by span durations. If distributed tracing is enabled, 10% of invocations will generate a breakdown and highlight anomalous spans that take longer than average. This doc describes what you can see in the Lambda monitoring UI.

New Relic Lambda CLI

Note: Our CLI isn’t supported for use with the gov cloud at AWS. For a region like us-gov-west-1, you’ll need to use the manual methods described below.

Requirements

The AWS CLI version 2 is recommended.
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

# check for python 3
which python
python --version

# get config
aws configure list

# set config
aws configure

# verify UserId, Account, and Arn associated with current AWS profile
# this is how our CLI gets your AWS profile
aws sts get-caller-identity

# double check .aws profile
cat ~/.aws/config
cat ~/.aws/credentials

License Key vs Personal API Key

The personal api key is only used for the New Relic Lambda CLI in order to run some graphql queries which grab the license key for the account. It is that license key that should be associated with the function and the newrelic-log-ingestion function.

From the Lambda Monitoring Engineers:

The CLI needs your user API key, because it’s making several graphql requests, which need to be authenticated with that key. Among them is the account linking setup, and fetching your license key. The license key is what we use to send telemetry to New Relic.
We do not store your user API key, or use it after the command completes. So there’s no key rotation challenge there.
The CLI checks for an existing integration between your AWS account and your NR account. If one exists, it moves past that step.

Common Errors

See our README for some common errors and troubleshooting steps.

Updating

# check version
pip3 show newrelic-lambda-cli

# update newrelic-lambda-cli to make sure it is latest one
pip3 install -U newrelic-lambda-cli

Integrations Uninstall and Install

If you run into trouble with the integrations install, it can be helpful to start over by first uninstalling and then reinstalling. Before reinstalling, make sure all resources that were created before are torn down. This can be best accomplished by deleting the associated CloudFormation stacks, but you may also need to manually check for and clean up any stragglers from IAM roles or policies.

See our CLI repo for details on the integrations uninstall command.

# log ingestion function uninstall
newrelic-lambda integrations uninstall -r AWS_DEFAULT_REGION

When troubleshooting integration issues, it can sometimes be helpful to start fresh by uninstalling both the failing integration and the newrelic-log-ingestion function in your AWS account. This command will do both. The integration can alternatively be unlinked via the UI by going to New Relic One -> Infrastructure -> AWS -> Manage Services (for your integration) -> Unlink this account (at the bottom right).

Note: The integration can only be removed if no other AWS services depend on the integration. Otherwise, just uninstall the newrelic-log-ingestion function with the command describe in the section above.

# integration and log ingestion function uninstall
newrelic-lambda integrations uninstall -a NEW_RELIC_ACCOUNT_ID -r AWS_DEFAULT_REGION

For a new integration between a single AWS account in a particular AWS region and a single New Relic account, this will generate a new integration named like New Relic Lambda Integration - <AWS Account ID>. See our CLI repo for details on the integrations install command.

# integrations install for new integration in single aws region
newrelic-lambda integrations install -a NEW_RELIC_ACCOUNT_ID -k NEW_RELIC_API_KEY -r AWS_DEFAULT_REGION

For an existing integration, the following command will install the newrelic-log-ingestion function into your AWS account. The “linked account name” is the name of the integration found on the New Relic One -> Infrastructure -> AWS page.

Note: It’s important to make sure the “linked account name” is exactly correct, otherwise the CLI will not find it and try to create a new integration. This will then fail when it discovers that your AWS account has already been linked to New Relic.

# integrations install
newrelic-lambda integrations install -n LINKED_ACCOUNT_NAME -a NEW_RELIC_ACCOUNT_ID -r AWS_DEFAULT_REGION -k NEW_RELIC_API_KEY

The following command will additionally enable logs to be forwarded via the newrelic-log-ingestion function. It simply toggles the environment variable LOGGING_ENABLED to true on the function. Make sure your function has a subscription filter to send logs to the newrelic-log-ingestion function.

# integrations install with CloudWatch logs
newrelic-lambda integrations install -n LINKED_ACCOUNT_NAME -a NEW_RELIC_ACCOUNT_ID -r AWS_DEFAULT_REGION -k NEW_RELIC_API_KEY --enable-logs

The newrelic-log-ingestion function does not get updated automatically. You should perform regular updates on it manually. The following CLI command assists in making that process easier. See our CLI repo for details on the integrations update command.

# integrations update (only use to update the "newrelic-log-ingestion" function, not the New Relic integration)
newrelic-lambda integrations update

It may be necessary to specify which AWS profile to use and/or the AWS region if not set in your profile. There is also an option to disable the AWS permissions check to avoid the “chicken or egg” scenario where AWS requires certain permissions to be able to check if we have the necessary permissions.

newrelic-lambda integrations update -r AWS_DEFAULT_REGION -p AWS_PROFILE --no-aws-permissions-check

For AWS GovCloud

If integrating your AWS GovCloud account to New Relic, please follow these steps. For non-GovCloud regions, like us-west-2, follow these instructions.

CloudFormation Templates

Our New Relic Lambda CLI uses CloudFormation templates to create the following stacks on the newrelic-lambda integrations install command:

NewRelicLogIngestion
NewRelicLambdaIntegrationRole-<account-id>
NewRelicLicenseKeySecret

For EU regions

If connecting with our EU endpoint, you’ll need to specify the newrelic-lambda integrations install --nr-region eu option.

The following error is typically encountered when either the above EU region option is not specified, or the User API key does not have sufficient privileges to retrieve the license key on the New Relic account.

Could not retrieve license key from New Relic. Check that your New Relic Account ID is valid and try again.

List Functions

This will not only show a list of existing Lambda functions in your AWS account, but also whether they are showing as instrumented or not, which is quite useful to know for troubleshooting. See our CLI repo for details on the functions list command.

newrelic-lambda functions list

Add a Layer

See our CLI repo for details on the layers uninstall command.

# layers uninstall
newrelic-lambda layers uninstall -f FUNCTION_NAME

See our CLI repo for details on the layers install command.

# layers install
newrelic-lambda layers install -f FUNCTION_NAME -a NEW_RELIC_ACCOUNT_ID

# layers upgrade and enable function logs
newrelic-lambda layers install -f FUNCTION_NAME -a NEW_RELIC_ACCOUNT_ID -u --enable-extension-function-logs

Subscribe Your Function

See our CLI repo for details on the subscriptions uninstall command.

# subscription install
newrelic-lambda subscriptions uninstall -f FUNCTION_NAME

See our CLI repo for details on the subscriptions install command.

# subscription install
newrelic-lambda subscriptions install -f FUNCTION_NAME

# subscribe all functions at once
newrelic-lambda subscriptions install -f all

Does your function have any existing subscription filters?

aws logs describe-log-groups --log-group-name-prefix /aws/lambda/ --region <region>
aws logs describe-subscription-filters --log-group-name /aws/lambda/<function_name> --region <region>

If using the Extension, it is recommended to delete subscription filters that point to the newrelic-log-ingestion function to avoid duplicate logs being sent to New Relic.

aws logs delete-subscription-filter --log-group-name <value> --filter-name <value> --region <region>

CloudFormation and SAM Deploy

The New Relic Lambda CLI uses CloudFormation templates to create the integration, execution, and secrets roles. They can be used as a starting point for your functions. This is described in our docs.

Our Extension examples uses the SAM deploy method to deploy the Lambda function to AWS Lambda. It does this by utilizing several AWS services including: sam cli, aws cli, S3, and CloudFormation.

  1. We set some variables.
  2. Use sam build to create the container.
  3. Use aws s3 to create an S3 bucket.
  4. Store the sam package in the S3 bucket for your region and output the template yaml.
  5. Use aws cloudformation deploy to create the stack from the template file.
./deploy.sh <accountId> <region>
#!/bin/bash
accountId=$1
region=$2
echo "region set to ${region}"
sam build --use-container
bucket="newrelic-example-${region}-${accountId}"
aws s3 mb --region ${region} s3://${bucket}
sam package --region ${region} --s3-bucket=${bucket} --output-template-file packaged.yaml
aws cloudformation deploy --region ${region} --template-file packaged.yaml --stack-name NewrelicExampleNode --capabilities CAPABILITY_IAM --parameter-overrides "NRAccountId=${accountId}"

This process is essentially the same for each function language except for Go, which requires a special runtime, handler, and build tags for use on AWS. Go also forgoes using the sam cli completely and instead just zips up the whole package to S3.

That means we need to supply a CloudFormation template specifically for Go.

Note: Each example function contains a CloudFormation template.yaml which can be used as an alternative to the SAM deploy method. Just modify it to suite your needs.

It’s also possible to get CloudFormation templates from the sam cli:

sam build -t template.yml

Serverless Plugin

The serverless-newrelic-lambda-layers plugin utilizes CloudFormation templates for the creation of integration and secret manager roles. This is described in our docs.

The order of layers matters. It is recommended that the New Relic layer is added last, if more than one layer exists. This avoids the scenario where another layer overwrites a dependency in our layer.

To see more details on errors, set debug level for the plugin and the Extension.

custom:
  newRelic:
    debug: true      # plugin debug
    logLevel: debug  # function agent debug

One other thing you can do to increase verbosity and see error details is to use sls deploy -v.

Regarding the serverless.yaml, it is possible to reference both environment variables and config files within, like:

custom:
  newRelic:
    accountId: ${env:NEW_RELIC_ACCOUNT_ID}
    apiKey: ${env:NEW_RELIC_PERSONAL_API_KEY}
    enableExtension: true
    enableFunctionLogs: true
    enableIntegration: true
    linkedAccount: New Relic Lambda Integration - 1234567
    logLevel: debug

provider:
  name: aws
  region: ${file(myconfig.yaml):region}
  stage: ${file(myconfig.yaml):stage}

Note 1: The above config includes enableIntegration: true and a linkedAccount field, which should be set to the name of your existing integration in New Relic One -> Infrastructure -> AWS, otherwise it will attempt to create a new integration and fail when it discoveres that your AWS account has already been linked. There can only be one integration for a particular AWS account and New Relic account.

Note 2: There was a bug in our serverless plugin that prevented setting the region via a config file like above. This was fixed in v1.1.1, but please make sure you’re using v1.1.2 which includes another bug fix for the accountId field.

Note 3: We don’t have direct support for the NEW_RELIC_EXTENSION_LOG_LEVEL variable in our plugin yet. However, it can be added directly in your serverless.yaml with:

provider:
  environment:
    NEW_RELIC_EXTENSION_LOG_LEVEL: DEBUG

Our engineer says:

As long as it’s scoped to provider/environment, it’ll get picked up.

Terraform Templates

We have Terraform templates and examples for Node.js and Python functions in our Extension repository. This is described in our docs.

1 Like