"Too many errors" does not trigger error notifications

###Alerts Question Template

  • Please describe what are you seeing:
    We had an incident caused by another vendor that resulted in a large concentrated flow of error status codes being returned from our api. We have error alerts set up to notify us when the total volume of errors exceeds 1% and 5% over a 5 minute (lowest available) timespan. While our api was returning these error status codes, we were unaware there was an incident because we received no notification from New Relic. After logging into the console, I saw this message at the top of our Error Analytics page:

  • How does this differ from what were you expecting to see?
    If there is an overflow of errors, we should ABSOLUTELY BE NOTIFIED. This is honestly unacceptable, and I hope it is just a simple configuration issue. I cannot stress enough that to pay for an apm solution that falls over and does not even notify us when there is a small incident is outrageous. Please tell me that there is a way to make this work. Cap the errors (fine), but send me a notification, preferably with huge words, bolded scary words such as TOO MANY ERRORS.

To reiterate, what i am looking for here is a notification of the “too many errors” condition. Also an explanation as to why I didn’t receive one in this instance.

No incident reported

  • If you aren’t seeing expected alert or data, please provide a link to the incident or violation (policy, condition, data app etc.)
    This should have flagged a notification - https://alerts.newrelic.com/accounts/1853390/policies/184045
    If unable to determine an accurate % based on error load, I should get a default notification saying as much.

Helpful Resources
Relic Solution: The key to consistent alert notifications

Troubleshooting downtime document

Hi @itadmin5

Thanks for contacting us about this! I’m eager to dive in and discover why this happened, but I’m going to need some possibly sensitive information, so I’m going to open a support ticket. Expect to see a notification in your email soon.