NRQL Alert not triggering despite condition being met

Support: Alerts/Applied Intelligence (AI) Alerts
#1

We have an Alert Policy set up with an incident preference of “by condition”. Two of these conditions were met for the past few days, and no alert or event was generated, either in the notification channel or on the Alert Dashboard under Events or Incidents.

Opening the conditions themselves shows that the warning and critical evaluations displayed in the graph are correct, highlighting the correct time periods respectively.

One of these conditions is https://alerts.eu.newrelic.com/accounts/2364232/policies/22622/conditions/147799/edit?selectedField=thresholds

Alerts have worked in the past and I am not aware that the alerting configuration has been changed since.

Is this an issue with the setup or configuration somewhere?

#2

Hello @sven.riedel,

From the backend, I can see that some of the events returned by your query were dropped for being “late.”

NRQL alert conditions will evaluate the results of your query in one-minute slices, and by default, we look at data that’s three minutes old. That three-minute offset is a safeguard to account for most instances of data latency. However, if some data takes more than 3 minutes to reach your account, that data is subsequently “missed” by our alerts evaluation system, as it will have a timestamp matching a point in time that’s already been evaluated.

You can find more details on the subject here:

Knowing this, would you please raise the evaluation offset to 10 minutes to determine if a violation were to open with a higher offset value?

3 Likes
#3

Hi,
thanks for your reply. I’ll try increasing the offset and see if that changes anything.

Just so that I understand correctly: if the evaluation offset is X minutes it will always look at the data at now - X minutes, no matter if data may have already arrived for a later point in time? My impression was that it looked for the most recent data point up to X minutes ago.

The condition was red for almost 4 days without an event - so that would mean the data for exactly those examined timeslots was consistently late, correct? If so either our hosting platform is tardy in general or we’ve had very bad luck in this case.

#4

Alerts are triggering now as expected. Thank you for the pointer.