We had an alert violation trigger today but did not receive either email nor notification to PagerDuty as we usually do. Our last violation of the same alert was August 16th and worked well that time. The alert policy has not been modified since July nor have we changed our notification channels. Any help would be appreciated. Thanks!
I should add that we are getting notifications for other alerts, like Infrastructure alerts. It is just this one that did not trigger notifications today.
Could you drop in a link to the alert policy??
@RyanVeitch Here is the link:
Hey @afister - I think this is due to Incident Preference settings.
If you look at the Incidents List and look for all incidents from the
Production Services policy: See that here
You can see that over the past days there are pretty much always open incidents. Shortly after one closes another opens.
Your incident preference for that policy is set to
By Policy, this means that while those incidents are open, any violations of any other conditions in that policy will be rolled up into the one incident.
This is important because notifications are tied to incidents, not to violations.
So once that incident is open for infrastructure conditions (that you mentioned you were notified by), you will not be notified about the APM conditions you sent a link to.
If there are no open incidents when the APM condition is violated, then you will be notified.
You can read more about Incident Preference here:
I think the best setting for you is
Thanks @RyanVeitch ! I think this was the problem and I’ve changed the setting.
Fantastic! Thanks for confirming @afister