Thank you for your question, TripArtrip. I understand that recent New Relic incidents in which you received false alerts have led you to lose confidence in your alert policies and disable them. Unfortunately, disabling these policies has resulted in you missing an important alert when you did have an issue. When your systems are down, you expect New Relic to alert you. When they’re fine, the last thing you expect is New Relic to send you a false alert.
We are working diligently to address sources of instability in our platform so that you receive the alerts you’re supposed to receive, and no false alerts. At the same time, we know some issues are inevitable, so we also have mechanisms for suppressing false alerts in some cases that we leverage whenever we can. Clearly, our false alert suppression mechanisms are not perfect and we continue to refine them. However, we are making our best effort to suppress false alerts, so I recommend you keep enabled alert policies that you rely upon to notify you of important issues. We apologize for our incidents of instability, especially when those have resulted in false alerts getting through to customers like you.
One action you can take, however, to reduce the chances of receiving some kinds of false alerts, is to check the loss-of-signal settings on your NRQL alert conditions. Some false alerts are caused when data flow is interrupted in New Relic’s infrastructure, resulting in the alerting system triggering a loss-of-signal alert. You can read our documentation on how loss-of-signal alerts work, and decide if you really need that kind of detection for your use case, disabling loss-of-signal where it is unnecessary.