Alert policy stopped working

Hi,

One of our alert policy stopped creating incidents since November 29. We have not made any changes in the policy.

Alert policy id: 1533527
Policy name: ES-Log-Alerts

Can you please check why is it happening?

Hi @rahul16 -

It does seem strange that the alert policies aren’t working suddenly. Could you check the audit event type to see if there were any changes made in your New Relic account?

Hi,

Thank you for the update. But doesnt seems like an account wide issue. The alerts not working only for this specific policy. Others policies and alert conditions are working fine.

Hi,

I have gone through the “NrAuditEvent” report and no changes happened in mentioned policy between November 29th to December 08’th.

Hi @rahul16. Can you provide a permalink to a query that shows a time where the data violated the threshold but an incident did not open?

Hi,

Please find the permalink that shows the recent violation that didn’t trigger any incident.

https://onenr.io/0xVwg9gg0jJ

@rahul16 Thank you for providing that. It looks like the data points arrive inconsistently and pretty far apart. You are going to want to use Event Timer as the aggregation method rather than Event Flow.

Event Flow needs 2 data points before it will aggregate the data. If the ingested data is ‘gappy’ this can cause a long delay in incidents opening up. After 30 minutes, if a second point of data has not been received, the first point of data falls off into a ‘stale’ state and is no longer considered. What this means is that if your data is not coming in pretty consistently or has large window of time where it may not send anything to New Relic then you want to use timer.

Choose your aggregation method
New aggregation methods for NRQL alert conditions
Relic Solution: How Can I Figure Out Which Aggregation Method To Use?

Thanks. Alerts started coming after modifying the aggregation method. But I think there is some change in the timing(Delayed alerts). Let me check once again

1 Like

Hey @rahul16 - Thanks for sharing and glad your alerts started coming in again. Let us know if you need anything else.