Alerts are not firing anymore

[kostyantyn]

Hello

After your last change for the alerts notifications i was forced to recreate all the conditions for alerts sending and now i see that alerts are not firing anymore for logs and metrcis, though i still have the raised violations.
For example:

but in fact we had violations since 2:38 am but they were not sent to the channel

So, where is the root cause?
I have used two possible conditions to catch it but not none of them is working now, but it worked previosly before NR applied the changes in this section:

Please advise how to make it work now. Thanks.

p.s. I have checked the channel notifications by sending the test alert directly form there

and it works fine for me.
So, there is not an issue with the channel but looks like an issue with the NR logic and how it processes the violations.
p.s.s. and by the way, the day before yesterday it does not work for me at all. But yesterday i have managed to recive about 20 errors notifications for logs. But today it does not work againe. Please fix it, guys, cause i am 99% assured that problem is on your side! Cause it can not be normal that it works one day but stops to work on another day with the same configuration for the notifications!

[kostyantyn]

Well
As it turned out I still have one server (thanks God) where alerts still working somehow and it uses an old template

All the servers where I have changed to “event flow” (for better accuracy per your advice) now have stopped reporting the alerts at all. So your “better accuracy” does not work in an expected way. Please fix it ASAP.
Thanks.

[kostyantyn]

Well.
Looks like the issue was in not proper parameters after the switch from the old to the new alerts template.
After changing from “Event flow” to “Cadence” I have started to receive the notifications to Discord.

[Fidelicatessen]

Hi @kostyantyn

Cadence will certainly work for these, although you will likely get faster alerts with Event Timer.

When your data is sparse and inconsistent (log queries looking for errors is a great example of sparse, inconsistent data), Event Timer is your best bet. It will wait until your data shows up, and then only wait a certain amount of time after your data shows up (the “timer”) before aggregating the results and sending them to evaluate.

The reason Event Flow did not work is that it relies on subsequent aggregation windows being filled to determine that the first aggregation window is finished and ready for aggregation. If you haven’t already, I would recommend taking a look at the two posts I’ve made explaining how they work and which use-cases work best for each of them:

• New aggregation methods for NRQL alert conditions
• Relic Solution: How Can I Figure Out Which Aggregation Method To Use?

Again, Cadence will certainly work, but if receiving your notifications more quickly is important for this use-case, I would recommend trying out Event Timer with a 1-minute Timer setting (to start with). You can reduce the timer setting in increments to see what works, but the lower the timer setting, the sooner you will get your alerts.

[kostyantyn]

Thanks @Fidelicatessen
But I can set the cadence wait period 0 minutes and it will work immediately as well as the event timer. Or not?)

[Fidelicatessen]

@kostyantyn

That sounds like it’s working pretty quickly!

The difference is that Cadence will always use that delay (which includes the aggregation window itself, so it’s a lot like using SINCE 1 minute ago UNTIL now – the difference between evaluation offset and the Delay setting is explained in this article). You can’t make it go more quickly, even if your data is actually arriving faster than that.

Whether this is faster than Event Timer would require a bit of experimentation, but if your data points are coming in close together or you are only expecting a single data point per aggregation window, you could use Event Timer with a very low Timer setting (you can go as low as 5 seconds), which would result in faster alerts (lower Mean Time To Detect). The only risk with using a setting this low is that you might miss some data points that arrive a bit later. It looks like you’re only getting a single data point per window, however, so a 5-second timer would be accurate and very fast. You would essentially get your alerts within 5-10 seconds of New Relic receiving your data, instead of having to wait for a full minute.

In any case, I’m glad you found a setting that works well for you

[kostyantyn]

Changed to event timer. Will see the difference. Thanks.)