Create an alert for a daily lambda

Hello,

I have a AWS lambda process that I have instrumented that executes once a day. This lambda process is very important for the day to day function of my application.
I would like to set an alert for if the lambda fails to execute for whatever reason. I’m trying to set up the policy to alert me if the lambda fails to execute, however it seems there are limits on the window. The query I use to see the number of invocations is:

SELECT count(*) as ‘Invocations’ FROM AwsLambdaInvocation WHERE (entityGuid=‘XXXXXX’) SINCE 1 day ago

Is there any way to go about this?

Hi @rtaylor2 - Unfortunately, since our alerts evaluation system happens on a minutely basis when the data comes in infrequently anything over 60 minutes gap will need to use the proposed workaround suggested below:

Relic Solution: Extending the functionality of NRQL alert conditions beyond a single minute

In addition, the alert condition is based on a NRQL alert condition using count(). New Relic used to insert synthetic zeroes as a result for some queries, but ever since Streaming Alerts for NRQL conditions was introduced, synthetic zeroes are no longer inserted (the rules for when they were inserted was obtuse and opaque, so now you get exactly what the query returns).

Take a look at this article, which will explain why count() and uniqueCount() will never return a value of 0. The article also goes into a couple possible solutions depending on your use-case – these are new features which were also released with the Streaming Alerts Platform.

Depending on the situation you may need to set up a Loss of Signal (LoS) for the same duration as the threshold, and configure it to Open new "lost signal" violation when met. This will result in violations opening when the query stops returning numeric values and drops to 0/NULL or set up a Loss of Signal (LoS) for the same duration as the threshold, and configure it to Close all open violations when met. This will result in violations which close automatically, as you’d expect them to.

There is one caveat, which is covered in the article I linked as well: there needs to be a signal to lose, in order for LoS behavior to kick in. That is, there must be a numeric data point that then drops to NULL. Currently, given the data coming in right now, adding a LoS setting to this condition will not result in a violation opening – there would need to be at least one evaluation window with a non-NULL value first.

I was able to find a workaround by querying the invocations of the lambda using NerdGraph inside of a scripted synthetic monitor. Then asserting that the query on the invocation returned one. From my testing that seems to have done the trick.

3 Likes