We have implemented several synthetic checks in New Relic, most of them have more than one alert policy. Sometimes not all policies seem to be triggered. So for instance the monitor fails it’s check and Policy 1 gets triggerd and all channels get notified, policy 2 also gets triggerd and all channels are notified. The “close” notification then triggers policy 1 but doesn’t trigger policy 2. Is this a known issue?
Hi @ravdvelde - Have you verified that the policy conditions are the same? For example, you may get differing results if Policy 1 is set to alert by entity and condition and Policy 2 is set to alert by condition only.
Both of the policies you linked have an Incident Preference of By Policy - Essentially this means that every condition in the policy that violates it’s threshold will be rolled up into 1 single incident. That being the first incident to have opened for conditions in that policy.
In your case, Policy All contains 63 conditions, and so whenever any of those violate, they’ll be attached to any other violations from that policy in the original incident.
This is important because Alerts only sends 3 notifications for an incident. When an Incident Opens, Closes, and is Acknowledged.
Individual violations do not trigger their own notifications.
It’s most likely that at the time a monitor fails, both policies recognise that, both open up a new violation, and both get added to an incident. But one of them is an already open incident, so the notifications aren’t sent for that individual violation.
I recommend checking out the below Level Up post regarding incident preference, and I think it would be most useful for you to configure your alert policy to be: Per Condition;