AWS S3 Log Ingestion Lambda
AWS Serverless Application that sends log data from S3 to New Relic Logging.
To forward data to New Relic you’ll need access to your license key. You can access your license key by following the steps listed here: New Relic License Key.
You will also need to configure your AWS credentials so that they are accessible from your local terminal: Configuring the AWS CLI
Configure Lambda from AWS Severless Application Repository (SAR)
- Find “NewRelic-s3-log-ingestion” on the SAR (you have to check the “Show apps that create custom IAM roles or resource policies” box)
- Under “Application Settings” add your New Relic license key
- Acknowledge that this app creates custom IAM roles and press the “Deploy” button
- Wait for the “resources” to be created
- Navigate to AWS Lambda and open the functions
- Find “NewRelic-s3-log-ingestion” in the list and click on the function name
- Click on “Add Trigger” and selet the “S3” trigger
- Configure you “Bucket” with the bucket name of where you are logging to.
- Configure “EventType” to "all object create events
- Optionally configure “prefix” and “suffix”
- Select “enable trigger” and click “Add”
- Congratulations your Lambda is successfully set up!
- To test your lambda setup:
1. Upload a new log file to your s3 bucket
2. Go to New Relic Logs and view your test logs
Configure Lambda Using Serverless
Follow these steps below to deploy the log ingestion function manually.
- Clone this repository:
git clone <REPO>
- Install Serverless:
npm install -g serverless
- Install the serverless-python-requirements plugin:
- If not running Linux, install Docker and keep it running
- Retrieve your New Relic License Key
- Set the LICENSE_KEY environment variable:
- Set the SERVICE_NAME environment variable:
- Set the S3_BUCKET_NAME environment variable:
- [Optional] You can send logs from a specific location in the bucket only. Set the S3_Prefix (subdirectory name)
- Deploy the function:
New Relic hosts and moderates an online forum where customers can interact with New Relic employees as well as other customers to get help and share best practices. Like all official New Relic open source projects, there’s a related Community topic in the New Relic Explorers Hub. You can find this project’s topic/threads here:
Full details about how to contribute to
Contributions to improve s3-log-ingestion-lambda are encouraged! Keep in mind when you submit your pull request, you’ll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.
To execute our corporate CLA, which is required if your contribution is on behalf of a company, or if you have any questions, please drop us an email at email@example.com.
s3-log-ingestion-lambda is licensed under the Apache 2.0 License.
The s3-log-ingestion-lambda also uses source code from third party libraries. Full details on which libraries are used and the terms under which they are licensed can be found in the third party notices document.
Error: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
This may mean either docker is not running or it has not been correctly set up. Please ensure docker is runnnig on your machine.
An error occurred when creating the trigger: Configurations overlap. Configurations on the same bucket cannot share a common event type.
If this error message appears while adding a Lambda “Trigger”, ensure your s3 does not have another event of the same type on the bucket.
Ie, your s3 bucket cannot have multiple “all object create events” events.