Relic Solution: Alert Incident Preferences are the Key to Consistent Alert Notifications

One of the common questions that gets raised in the community and support tickets is around when notifications are sent in New Relic Alerts. It definitely reminds us of the old “if a tree falls in the woods” adage: If an alert triggers but no notification was sent, did the alert really happen?

Obviously the answer is no. If you’re not receiving a notification, it’s really no use to you in managing and operating your software. Fortunately, we’ve found that in many of the cases we have encountered, the problem has been about configuration of Alerts policies, and more specifically choosing the right incident preference for your policy.

The Scenario:

Your site has an outage. You see it. Your customers see it. Even more concerning than the outage is that leading up to, during, and after the outage, not a single one of your server alerts sent you or your team a notification. Once you get service restored, you check on those Alert policies to see what happened. Sure enough, your policy is there and your alert conditions seem clear.

So what happened? To find out, let’s take a look at what goes into an Alert policy.

The Anatomy of an Alert Policy:

An Alert policy is made up of a few parts:

  • One or more conditions that define when violations should be opened and closed
  • Notification channels to determine who and how to send notifications to
  • Alert policy incident preferences to define how the violations emitted from the policy’s conditions will be grouped into incidents

Checking these areas first when you think an Alert notification should have been sent is the best first step.

Navigate to the Alerts product and select the policy you are interested in to view the conditions (which include the thresholds) and the notifications that have been established.

If everything looks like it is configured correctly for conditions and notifications, the last thing to check is the incident preference.

An incident is a collection of violations, and New Relic can greatly reduce alert noise because it sends notifications when incidents open, are acknowledge and closed. How you choose to create incidents is up to you and will affect how many notifications you will receive.

Here are the three options for how your policy can group violations into incidents:

  • By Policy: Only one incident will be open at a time for the entire policy. This creates the fewest number of alert notifications and requires immediate action and closing the incidents to be effective. If there is an active incident when this preference is selected, no new alert notifications can be created until the incident is closed.
  • By Condition: One incident will be open at a time for each condition in your policy. This configuration creates more alert notifications and is useful for policies containing conditions that focus on targets (resources) that perform the same job; for example, servers that all serve the same application(s).
  • By Condition and Entity: An incident will be created for every violation in your policy. This option creates the most alert notifications and is useful if you need to be notified of every violation or if you have an external system where you want to send alert notifications.

Note that when you create a new alert policy, the default incident preference is “By Policy.” Remember, if you have an open incident AND your alert incident preference is set to “By Policy,” no new alerts notifications will be sent. You’ll need to close the open incident before these notifications will send again. You may want to set the incident preferences to “By Condition” if the alert is truly important, or make sure that you add closing open incidents to your incident response playbooks.

Now that we’ve looked at settings that how and when notifications are sent, you should feel much more confident in knowing that the alert policy you set up will get your attention at the expected times.

15 Likes
Auto close the alerts once condition is not met
Relic Solution: Email Alerts and the New Relic Suppression List
Is there a way to keep getting failure notifications, if user misses the first failure notification
Preventing On-Call Burnout
Best Practice Guide: Alerts
Notifications not received - Synthetics & Infrastructure
New Relic Terraform Provider Destroy Issue
"Too many errors" does not trigger error notifications
Alert when multiple synthetic monitors are failing
Limit on Notification Channel Count for each type
How to get tags passed out of Custom Webhook JSON payload?
Alerts have stopped sending or showing in app
Up and Down Notifications
Understanding why baseline alert did not fire
Not Receiving Alerts From synthetics
How to eliminate 0% from alert analysis
What is the use of time_function in Alerts?
Didn't get notification about failures of monitor
Open Incidents by target
AWS Infrastructure Alert going wrong
Alert on Agent/MetricsReported/count not working
ALL email alerts goes in one Incident
Alert: Apdex has gone to zero
Alerts stopped working
Alert Policy is not showing while adding a Infra alert
What is the duration's unit while using in NRQL Alert Condition
Missing New Relic Alerts
Synthetics Monitor Check Failure no longer triggering Email and Slack channel sends
Synthetics Monitor Check Failure no longer triggering Email and Slack channel sends
Alerts not sending via Email
How to setup DB alerts?
Alert didn't get triggered
Send Alert Notification only when Incident is Open
Unable to set Alert on New Relic platform
Critical violation but we had not any notifications
Error Critical Alerting Not Working
Incident will not close automatically
Demotron for partners v2
Demotron for partners
Feature Request: Slack event button points to the event details instead of the overall incident page
Best Practice Guide: APM
Cannot disable/re-enable a condition for an alert policy
How to write NRQL query that is only executed at specific times
Relic Solution: How to Get Notified on Warning Violations
Process alerts notification
Stopped receiving Alerts in Slack
Unable to get details of critical priority alert violations via API
Alerts Generation is not consistent
Alerts not getting triggered
Question about Synthetics check alerting
Not all policies are triggered on synthetic checks
Alert has not triggered from a certain point
Sum of Query Results - Clarification
Use the values inside $TARGETS as a variable outside of $TARGETS
Alert condition not triggered notification event
Redirect notifications based on target for an Alert
No longer receiving alerts to email or PagerDuty integration
Not getting Alerts
What is the CPU Utilization in New Relic APM Alerts?
Alert conditions/violation not displayed under APM
NewRelic alert incident wrongly triggered (DELETED)
Email Alerts are not sent

I have created an alert policy as “By Condition and Entity” and selected some wrong URL to monitor.
I just got one notification alert but not alert email for every failure.

1 Like

Hey @yograj.patel - This is expected.

Setting your incident preference to By condition and entity will ensure that each entity you target with your condition (ie: Each app, monitor, host, etc…) will trigger its own incident.

If you set a monitor with a URL that will fail, it will trigger an incident for that monitor, but each other failure of that monitor will roll up as a violation under that 1 incident. That is, until that incident is resolved and closed.

There is currently no way to configure repetitive notifications for an already open incident.

Where mostly conditions can target multiple entities, the By Condition and Entity setting allows for the most amount of notifications. If you’re utilising synthetics alerts (which I’m assuming based on your post), each condition can only target one monitor (1 entity), and so an incident preference setting of By Condition and Entity is effectively the same as By Condition.

5 Likes

Is this still the case?

“There is currently no way to configure repetitive notifications for an already open incident”?

I’d like to have emails continue to go out until someone acknowledges an incident or it resolves.

Thanks,
Sam

Hey @sam.slingluff - This is still the case - I’ll get your +1 added to our feature idea for this. :slight_smile: