Alerts not triggered anymore on Critical Violations

[demian.jaeger]

Hi Support

We have an alert condition defined as a NRQL query (id: 339091). When this is violated there should be an incident triggered.
However, we had yesterday a violation yesterday in the evening (CEST) and today at around 11am (CEST) which did not trigger an incident and thus left us unaware of the incident.

The same thing used to work yesterday during daytime.

Alert policy 47394 : https://one.eu.newrelic.com/launcher/nrai.launcher?pane=eyJuZXJkbGV0SWQiOiJhbGVydGluZy11aS1jbGFzc2ljLnBvbGljaWVzIiwibmF2IjoiUG9saWNpZXMiLCJwb2xpY3lJZCI6IjQ3Mzk0In0&sidebars[0]=eyJuZXJkbGV0SWQiOiJucmFpLm5hdmlnYXRpb24tYmFyIiwibmF2IjoiUG9saWNpZXMifQ&platform[accountId]=2458876
Violated query 339091: https://one.eu.newrelic.com/launcher/nrai.launcher?pane=eyJuZXJkbGV0SWQiOiJhbGVydGluZy11aS1jbGFzc2ljLnBvbGljaWVzIiwibmF2IjoiUG9saWNpZXMiLCJzZWxlY3RlZEZpZWxkIjoidGhyZXNob2xkcyIsInBvbGljeUlkIjoiNDczOTQiLCJjb25kaXRpb25JZCI6IjMzOTA5MSJ9&sidebars[0]=eyJuZXJkbGV0SWQiOiJucmFpLm5hdmlnYXRpb24tYmFyIiwibmF2IjoiUG9saWNpZXMifQ&platform[accountId]=2458876

[6MM]

Personally, I’d open a support ticket if it were me.

[sudo]

Hey @demian.jaeger,

I updated your ticket with a synopsis of what’s happening here, but I wanted to relay that information in this forum post as well.

Currently, the condition in this post has its evaluation offset set to 1 minute. Due to internal processing delays that can range anywhere between 1 and 3 minutes, our alert evaluators are missing a portion of the latent data needed to open an incident. By default, we set the evaluation offset to 3 minutes in our alert conditions, which is our recommended setting for all data being collected specifically by New Relic.

Adjusting this offset back to 3 minutes should resolve the issue you are seeing.

One of my teammates posted a great article that explains this in more detail. You can find it here: Relic Solution: Better Latent Than Never -- How Data Latency Affects NRQL Alert Conditions

Cheers!