I have created various monitors in NR synthetics, I have channels to report such as email, webhook. When we were testing the NR synthetics by bringing test websites up and down we did not receive any alert on email or in the webhook most of the time. Out of 10 times, we do received alerts it 3-4 times. Can someone point out the issue?
Hi @sourcefuse, have you been able to identify whether any alert violations were created from those failures?
Please note that an alert violation in and of itself will not send a notification. Notifications are sent when alert incidents are either opened, acknowledged, or closed.
Depending on the configuration settings of your alert policy, violations may roll up into preexisting incidents, preventing notifications from being sent on successive failures. You can read up on incident preference management here: Specify when New Relic creates incidents.
Let me know if this helps! If not, I’d like to see a link to an example monitor failure where you weren’t notified.
Hi jeffery ,
We have already created notification channels and are applied in the policy we are using .
Same policy have multiple notification channels ( EMAIL and webhook) .
Policy is applied on multiple monitors .
The alerts works flawlessly for a few times but suddenly they stop notifying on any of the notification channels when tested multiple times (policy name is “phonecall”).
Hi rdouglas ,
We created separate policies for all the monitors and now things seems fine atm . It looks like attaching same policy on multiple monitors was creating mess . Anyways here is the link for the policy which we were using .
Hi @devops23, it looks like that policy’s incident rollup preferences were set to By policy. This would mean that, across all conditions and entities in that single policy, only one incident would be open at a time – you wouldn’t receive notifications on subsequent monitor failures as long as at least one other monitor continued to fail.
The important thing to consider is that incidents are what control notifications – you will only receive a notification on a monitor failure if that failure (violation) opened an incident. Therefore, in a policy with multiple single-failure Synthetics alert conditions, something like By condition or By condition and entity might be closer to what you’re after.
@sschneider’s guide on this topic is also a great read: Relic Solution: Alert Incident Preferences are the Key to Consistent Alert Notifications.
Thanks for sharing the links and letting us know were the issue was.
You’re welcome @devops23, enjoy the rest of your day!