Not all policies are triggered on synthetic checks

Hi there,

We have implemented several synthetic checks in New Relic, most of them have more than one alert policy. Sometimes not all policies seem to be triggered. So for instance the monitor fails it’s check and Policy 1 gets triggerd and all channels get notified, policy 2 also gets triggerd and all channels are notified. The “close” notification then triggers policy 1 but doesn’t trigger policy 2. Is this a known issue?

Kind Regards,

Ramon

Hi @ravdvelde - Have you verified that the policy conditions are the same? For example, you may get differing results if Policy 1 is set to alert by entity and condition and Policy 2 is set to alert by condition only.

Hi @stefan_garnham,

Thanks for your response.
Yes i’ve verified that the policies are the same.

The difference is the amount of conditions and the notification channels.

Do you have more suggestions?

Thanks in advance.

Ramon

Sorry, but @RyanVeitch may be able to assist further.

1 Like

Thanks for mentioning me @stefan_garnham

@ravdvelde, can you share a link to your policies?

Hi @RyanVeitch,

Policy all: https://alerts.newrelic.com/accounts/1877656/policies/285935
Policy Prio 1 - Mijn ANWB: https://alerts.newrelic.com/accounts/1877656/policies/280226

@ravdvelde It looks like @stefan_garnham’s theory was correct.

Both of the policies you linked have an Incident Preference of By Policy - Essentially this means that every condition in the policy that violates it’s threshold will be rolled up into 1 single incident. That being the first incident to have opened for conditions in that policy.

In your case, Policy All contains 63 conditions, and so whenever any of those violate, they’ll be attached to any other violations from that policy in the original incident.

This is important because Alerts only sends 3 notifications for an incident. When an Incident Opens, Closes, and is Acknowledged.
Individual violations do not trigger their own notifications.


It’s most likely that at the time a monitor fails, both policies recognise that, both open up a new violation, and both get added to an incident. But one of them is an already open incident, so the notifications aren’t sent for that individual violation.

I recommend checking out the below Level Up post regarding incident preference, and I think it would be most useful for you to configure your alert policy to be: Per Condition;

2 Likes

@RyanVeitch thank you for your helpful answer.

I’ve misinterpreted the incident preference in New Relic.
I think changing the preference to by condition will help getting it to work as we would like it to.

In the mean time i’ve created a support ticket as well, what they’ve seen is that the notification channel email wasn’t send for this violation, so that’s something to look into.

I will close this issue though as this is a different issue.

Thanks again for all the help.
Same to @stefan_garnham.

2 Likes