AWS CloudWatch plugin Troubleshooting Framework Log Forwarding

This guide is intended for help troubleshooting logs from the configuration and data collection end. For example: missing logs and/or log messages, unparsed logs, config questions etc. If the issue resides in the UI then there is a separate troubleshooting guide.

Before reading further:

  • Ensure that you have installed and configured a compatible Log Forwarder. You can find a list of the different log forwarding options and how to install New Relic’s various plugins in our documentation here. Troubleshooting for individual forwarders can be found below.
  • Logs in Context is different from Log Forwarding. Setting up Logs in Context for your application simply adjusts your application’s logger to format logs as JSON with New Relic’s logging metadata. This metadata establishes context for your application logs and other features of your New Relic APM agent. Check out our Configure logs in context with APM agents doc for more detailed information.

General

  • Double check that you have the newrelic-log-ingestion lambda in your list of Lambda functions in AWS.

  • Ensure that the newrelic-log-ingestion lambda function is using the most recent version. You can check this by navigating to the newrelic-log-ingestion function in AWS and scrolling down to the “Tags” section. Look for “serverlessrepo:semanticVersion” the value for this tag is the version of the function.

  • Ensure you have the proper configuration set up via Environment Variables.

    • Navigate to the log ingestion lambda in AWS and scroll down to the “Environment Variables” section. Ensure you have properly included the required config options mentioned in the Configure your Lambda function section of our documentation.
  • Ensure that your log group is “Subscribed” to the log ingestion lambda. You can double check this by going to CloudWatch and checking your Log Groups.

    • If you do not see your log group subscribed to the log-ingestion-lambda you can find instructions on how to do so in the Create a Lambda trigger section of the Lambda logging docs

Troubleshooting

  • Check and see if logs made it to Cloudwatch.
    • This can be verified by going to CloudWatch in AWS, selecting “Log Groups” on the left menu, and clicking into the group in question. If there are no logs in Cloudwatch then there is nothing for the log ingestion function to forward. You will need to investigate why your logs aren’t making it to Cloudwatch.
  • Check that the log ingestion lambda noticed the log lines and tried to send them to New Relic.
    • The most obvious indicator for this is a log line from the log-ingestion-lambda function saying: “Log entry sent. Response code 202. url: https://log-api.newrelic.com/log/v1” Keep in mind that this function may also be sending instrumentation data for Lambda instrumentation so you’ll want to be sure the log line has the log-api.newrelic.com endpoint and not cloud-collector.newrelic.com which is for the Lambda side.
    • You’ll also want to note the response code, if it’s not a 202 response then something is wrong. 403 is the most common error code and occurs when the wrong license key is configured (or no license key). But if you see a 202 then you should move forward with assuming the data made it to New Relic.
  • If you can confirm that all the following questions are true and the issue is still occurring then it is likely the issue needs to be investigated on the backend via support ticket:
    • Is the log-ingestion-lambda installed?
    • Does it have the proper Environment variables?
    • Are there log groups subscribed to the function?
    • Do those log groups actually have new log lines?
    • Do the log-ingestion-lambda logs show “Log entry sent. Response code: 202. url: https://log-api.newrelic.com/log/v1”?
1 Like