We are trying to setup 2 distinct alert categories for our new relic application. 1 is for Warning Alerts, which will be assigned to a PagerDuty Warning Channel, and another for Critical alerts, which will be assigned to a PagerDuty Critical Channel. What this requires is 2 alert policies that have to be assigned to the same application. It appears that an application can only be assigned to 1 Alert Policy. If so, how can I accomplish what I need to do here. We want to alert PagerDuty channels (Warning and Critical), based on our definition of the monitoring alerts we have defined.
You can only have one alert policy per application or server. I am going to submit a feature request for you.
We really appreciate feedback on the needs of our customers and use them to prioritize what we implement next.
This isn’t going to help you right now. One suggestion I can give is to take a look at our data api.
You can use to to pull information from us about your application and setup triggers for your applications if they fall below certain thresholds.
Here is a root of our doc page for this: https://docs.newrelic.com/docs/apm/apis
Let us know if you have any questions about any of this.
Thanks for the feedback. We won’t be using your API. We will use Rackspace monitoring to handle our Warnlng Level alerts and send an email notification to PagerDuty. Not an ideal solution, but based on your limited feature of assigning only 1 application to an Alert Policy, this is what we will be doing.
We appreciate you letting us know how you plan to handle the requirements for the time being.
We do take feature requests very seriously here at New Relic. It is my hope that the flexibility your require in your alerting will be available in a future update to our services.
We value your ideas on how we can improve our alerting or any of our services. Please let us know if you have other ideas for us. These requests are, in my experience, the most influential force driving our future development efforts.
+1 to this feature request. I was having a discussion with a co-worker the other day on this topic, and some work-arounds he has in place to get non-critical warnings into hipchat.
I think there’s 2 pieces to this:
- Allow an application to be part of multiple Alert Policies
- Allow “warning” alerts to go to separate Alert channels (right now only critical alerts go there).
Hi @adixon - thank you for your detailed input. I’ve sent this request off to our product managers.
Any movement on this since December? I just got out of a meeting requesting similar stuff. We’d like alternate alerting channels for Warning and Critical so that we could feed those into different On-Call queues.
Hey @Caleb, I’d recommend checking out our new Alerts beta. Such functionality is available in it: https://docs.newrelic.com/docs/alerts/new-relic-alerts-beta/configuring-alert-policies/defining-thresholds-trigger-alert
However, it is still required that a critical level is setup. This might not fit your needs exactly, but please give it a look and let us know how it functions for you. We’re happy to file a feature request on your behalf as well.
+1 to this feature request.
Here is a very common scenario for us:
- Warning breached - raise a ticket in JIRA (warning) & notify via Slack
- Critical breached - raise a ticket in JIRA (critical), notify via Slack & notify support via PagerDuty
At the moment the new Alerts system works fine for Critical, but means we don’t get notifications at the warning level.
The warning level in new Alerts is intended for situations where you want to monitor when an event passes a certain threshold, but you do not want to receive an alert notification for it. If you would like to receive multiple notifications, you can set up multiple conditions that target the same entity but have different critical thresholds. For example one condition could be for Error rate > 5% and another for Error rate > 10%.
From the description you gave it sounds like a potential solution would be to set up two Alerts policies, one for “Warning” breaches and another for “Critical” breaches. Since the conditions would be in separate Alerts policies you could configure a different set of notifications channels for each of them. In this example you would receive one set of notifications when the Error rate passed 5% and another set (potentially to different notification channels) when it exceeded 10%.
Thanks for letting me know how it is possible for this to be achieved - but I still think this feature request is incredibly useful. Its fine to go around duplicating your alerts when you have small numbers of them, but it becomes increasingly cumbersome to maintain when you have tens or hundreds. Also by setting your workaround, it would start to show up the apps in question as in a critical state on New Relic, when they are not - they are simply in a warning state.
thanks for the additional context, @andy.raines. We’ll get this feature request filed for you.
Is there any update on this? I’d like this for the same reasons as already discussed in this thread.