Add custom event to already instrumented lambda functions

aandrusyk · August 20, 2020, 1:09am

I have a serverless application with instrumented lamda functions by serverless-newrelic-lambda-layers plugin. Is it possible to add some custom events? I know that adding custom event can be done by

newrelic.agent.record_custom_event('CustomEvent', {'foo': 'bar'})

The question is: should I initialize New Relic agent fist? Isn’t it already initialized by lambda layer? If it is already initialized by lambda layer, won’t it break this initialization? Should I install newrelic package into the virtualenv or I can use somehow newrelic package from the lambda layer?

OR

Custom event implies custom Lambda instrumentation with newrelic python agent and refuse of serverless-newrelic-lambda-layers?

kmullaney · August 25, 2020, 10:20pm

Hi @aandrusyk,

Thanks for posting your question here in Explorers Hub! My name is Keegan and I’m with the Lambda Support Team in Portland, Oregon.

Since our Lambda Monitoring utilizes the APM agents in serverless mode to record invocations, any custom attributes or events that you want to add would be done from the agent api. No need to remove our lambda layer.

https://docs.newrelic.com/docs/agents/python-agent/python-agent-api/recordcustomevent-python-agent-api

We detail this specific case in the manual steps for Python instrumentation for Lambda Monitoring.

https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/enable-serverless-monitoring-aws-lambda#python


@newrelic.agent.lambda_handler()

def handler(event, context):

    newrelic.agent.record_custom_event('CustomEvent', {'foo': 'bar'})

...

Looking forward to seeing how that works for you! If you get it to work, I’d love to hear about it.

kmullaney · August 26, 2020, 9:55pm

Hi @aandrusyk,

I wanted to follow up with some more details here. When adding custom events in a Python Lambda that has our Layer installed, you’ll still need to import/initialize the Agent for access to the API.

import json
import newrelic.agent

newrelic.agent.initialize()
@newrelic.agent.lambda_handler()

def handler(event, context):
  # TODO implement
  newrelic.agent.record_custom_event('CustomEvent', {'foo': 'bar'}) return {
    'statusCode': 200,
    'body': json.dumps('Hello from Lambda!')
    }

torrance.graham · January 31, 2022, 4:08pm

Is this still the appropriate way to record custom events on already instrumented lambdas using the latest NewRelic lambda layer?

@newrelic.agent.lambda_handler()

The above produces a deprecation warning today and I cannot find any documentation on proper agent calling from an already instrumented lambda.

Following the suggested approach above produces the following output:

[NR_EXT] Startup check failed: Vendored agent found at '/var/task/newrelic', a layer already includes this agent at '/opt/python/lib/python3.8/site-packages/newrelic'. Recommend using the layer agent to avoid unexpected agent behavior.

Also, does this endpoint take awhile to reflect during a NRQL query? My lambda is only sending custom events like 50% of the time despite the fact that I have the follow environment variables set as suggested here: Relic Solution: Record Custom Event Troubleshooting

NEW_RELIC_STARTUP_TIMEOUT: 10
NEW_RELIC_SHUTDOWN_TIMEOUT: 10

JCapriotti · February 7, 2022, 10:18pm

From what I can tell, the documentation doesn’t really exist. The only thing I found that appears to be “current” is this thread (linked from the documentation).

That said, I think there’s an expectation in general that when using a Lambda layer that includes libraries, your deployment package does not need to include them. So I think your code is correct… kind of annoying, but I think the solution is to remove the New Relic library from your deployment. I’d consider it a “development” library so you can still run it locally on your machine.

For your last part about recording custom events 50% of the time… that is worrying. Curious if anyone else has made a thread about that? We noticed it too, but was only when running locally so have been hoping it does not do that in an AWS environment.

-Jason

kmullaney · February 14, 2022, 10:34pm

Hey @torrance.graham,

What @JCapriotti said about removing New Relic from your deployment is correct. We want to ensure there is only one agent which generates the NR_LAMBDA_MONITORING payload. Our Extension looks for that payload and may not find it if it was generated by the agent included in your deployment. That could be the reason for it missing 50% of the custom events if half came from the agent in our layer and half from the agent in your deployment.

In my test Python 3.8 function, I found it to be unnecessary to include the @newrelic.agent.lambda_handler() line.

github.com

newrelic/newrelic-lambda-extension/blob/main/examples/sam/python/newrelic_example_python/app.py

from newrelic import agent

# In a python Lambda, the runtime loads the handler code as a module; so code in the top level
# of the module occurs once, during cold start.
print("Lambda Handler starting up")


def lambda_handler(event, context):
    # At this point, we're handling an invocation. Cold start is over; this code runs for each invocation.

    # This is an example of a custom event. `FROM MyPythonEvent SELECT *` in New Relic will find this event.
    agent.record_custom_event("MyPythonEvent", {
        "zip": "zap"
    })
    # This attribute gets added to the normal AwsLambdaInvocation event
    agent.add_custom_parameter('customAttribute', 'customAttributeValue')

    # As normal, anything you write to stdout ends up in CloudWatch
    print("Hello, world")

This file has been truncated. show original

We already import the agent, handler, and initialize the agent in our wrapper in the layer.

github.com

newrelic/newrelic-lambda-layers/blob/d4b6b82aa5ae767bdf5028504c67224362d8be4b/python/newrelic_lambda_wrapper.py#L17-L20

    
      
          import newrelic.agent  # noqa
          from newrelic_lambda.lambda_handler import lambda_handler  # noqa
          
          
newrelic.agent.initialize()

If you remove the agent in your deployment, does the warning disappear and 100% of custom events get captured?

Add custom event to already instrumented lambda functions

Follow Us