Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Caution Level Email

alerts-beta
feature-idea
new-alerts

#1

It would be great if we can have 2 levels - when “caution” level is reached, perform a smaller task such as sending an email, and then when “critical” level is reached, really raise the alarm.

Would this be possible? Currently I can only find a way of being alerted when the critical-threshold is reached.


New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Hi @mattu, this is not possible right now but something simple and straight forward like this is something we are looking into.

If you have more advanced routing / escalation requirements, our partners at OpsGenie, VictorOps and PagerDuty are worth looking into.


#3

@NateHeinrich I’d like to request a feature like this as well. Warning should be able to trigger an alert on different channels than critical alerts. For example: waring could trigger a slack message, while critical alert could trigger a Pagerduty alert. Alternatively, you could allow multiple “default” alert polities for the same services and servers, where each would have different thresholds and alert channels.


#4

@vmathew I went ahead and filed a feature request on your behalf.

Cheers
AJ


#5

Ditto, would like this feature as well.


#6

I’ll add your “ditto”, @ryanwalls! Thank you :slight_smile:


#7

Would love to see this feature. At the moment we have to configure two separate policies to get Warning alerts into one set of notification channels (e.g. slack, email) and Critical alerts into a different set of channels (e.g. PagerDuty).


#8

I’ve gone ahead and added your interest in this feature @tanya-outpace. Thanks!


#9

@tanya-outpace (or anyone else – @vmathew,@mattu ), can you describe what your internal workflow is for warning notifications?

Why would these notifications be useful?
How do you receive them (email, chat, etc.)?
Who receives them?
What action is taken on them?
How often do you think receiving warning notification would give you an early heads up to taking action vs. increasing alert fatigue?


#10

@NateHeinrich, we have monitoring configured for one of our plugins reporting event percentage drops compared to normal level (shows, clicks, etc.). We would like to have a more sensitive Caution level that in most cases is not really an issue and doesn’t require dev involvement but a business person can take a look and verify. We don’t want these to generate PagerDuty alerts that would wake somebody up but rather send e-mail to a specific DL or slack. But when we see larger percent drops we want to generate PagerDuty alerts because it might indicate an actual outage. This is very useful for alerts that are not very precise and generate a lot of false positives.


#11

As an example, disk space monitoring would be one area where this is useful.

At say, 75% full, it would be useful to have a warning email so I can schedule a time to fix it in the near future. No need to be woken up in the middle of the night. If it suddenly shoots up past 90%, we have something worth being woken up for.


#12

Have you considered configuring a policy that has a the threshold you’d like and only configuring an email channel for it to notify on?


#13

@parrott We have certainly considered it and ended up with 2 policies with 2 different notification channels - “Warning” policy using Critical level as a way of triggering warning emails and “Critical” policy using Critical level to generate pager duty alerts. This really defeats the purpose of having “Caution” level if all it’s good for is to color things yellow inside NR UI. I would much rather be able to configure different channels for Caution and Critical levels of the same policy. Hope that makes sense.


#14

@tanya-outpace I’ve filed a feature request on your behalf for this. Thanks for the details about your use case!


#15

I just want to add to this. I was thinking about it the other day.

It would be great to have a seperate notification channels for caution and alert.

That way cautions can go to Slack or some other chat.

And Alerts can go to email/sms/etc to ensure they are delivered.


#16

Hi @owenk

Thanks for your suggestion for different notification channels for caution and alert level notifications. I’ve gone ahead and forwarded a feature request on your behalf. :smiley_cat:


#17

We’re also interested in this feature, currently using the 2-policy workaround.

@NateHeinrich
Why would these notifications be useful?
Warning notifications can be investigated at a convenient time, rather than having the issue escalate via PagerDuty and require urgent attention i.e. responses to warning notifications can be managed.

How do you receive them (email, chat, etc.)?
HipChat, email

Who receives them?
Engineering team including engineering and product managers. Embedded SRE resource.

What action is taken on them?
Scheduled investigation, decide on an action to avoid a critical threshold from being hit.

How often do you think receiving warning notification would give you an early heads up to taking action vs. increasing alert fatigue?
It may increase certain types of alert fatigue in that we now expect to receive more alerts via email/chat. On the flipside, it may mean less alerts will be going via PagerDuty - in a sense this would combat fatigue re: pagerduty alerts.


#18

Wow! @mattfysh Loving all these details! Thanks—I will put in a feature request for you right now! :thumbsup:


#19

Hello,

We are testing new relic to replace our monitoring system and be able to trigger a warning alert is also a must-have for us.

The goal is to be pro-active. As said in previous comment, a disk alert for example allow the ops to clean the disk space during working hours and avoid an alert during the night.

We are testing the pagerduty integration, the goal is to create an alert on slack for all caution events on working hours and critical events only the night.


#20

Thank you for the helpful use case, @seraf - I am submitting a feature request for your right now. :slight_smile: