No notifications sent on Issue / Incident creation

[david119]

Hi there,

I’ve noticed that we are not receiving notifications for incidents anymore. I recently enabled alert muting, but the rules were “scheduled”, not “active” at the time of the incident.
Here you can see that there were issues raised around 9am, this is from the “Analyze > Issues and Activity > Issues” tab

It clearly shows that no notifications were sent for any of the paths.

If I select “Analyze > Issues and Activity > Incidents” I can see that incidents were created based off of these issues.

I read through the recommended forum post below, but it doesn’t seem to reference “Issues”

Relic Solution: Alert Incident Preferences are the Key to Consistent Alert Notifications

One of the common questions that gets raised in the community and support tickets is around when notifications are sent in New Relic Alerts. It definitely reminds us of the old “if a tree falls in the woods” adage: If an alert triggers but no notification was sent, did the alert really happen? Obviously the answer is no. If you’re not receiving a notification, it’s really no use to you in managing and operating your software. Fortunately, we’ve found that in many of the cases we have encountere…

I did check our policy incident preference, and it has been set to “By condition and signal”, so I would have expected these Issues / Incidents to all sent alerts.

Interestingly, when I navigate to the classic Incidents view, it does not show any incidents…?

We’ve created our alerts via Terraform if it helps

Anyone got any leads for diagnosing this?

Thanks!

[jeffrey_s]

Hi @david119, thanks for reaching out.

The Issues & activity page represents a list of Issues and Incidents in our newer data model (known as AIOps). In this data model, Incidents beget Issues, and notifications on open Issues are sent to various destinations based on rules configured in workflows. It doesn’t look like your account has any destinations or workflows configured, so no notifications will be sent.

Note that Alerts (classic) references our older Incidents and Violations data model, where Violations may beget Incidents depending on policy preferences. Here, it looks like you have some notification channels subscribed to various policies. Under this data model, any Incident that’s opened (or acknowledged, or closed) in a policy will send a notification to its subscribed channels.

From what you’ve provided, there are two things I’ve noticed:

• In your Alerts (classic) screenshot, the Incidents page has its time picker set to the last three hours. I extended out the timeframe on this page and found a variety of Incidents — some of which were muted, and some were not. Subsequently, I checked some of the non-muted incidents and I can see they sent notifications to their corresponding channels.

• In your Issues & activity screenshot, you’re seeing a list of high-priority Incidents. This maps to warning-priority Violations in the classic data model, and Violations of that priority do not open classic Incidents (and therefore do not send notifications).

That all being said… everything looks “fine” from my end but I’m glad to keep investigating. Could you perhaps link me to a classic Incident that didn’t send you any notifications?

[david119]

Hey Jeffrey,

Thanks for your response, it sounds like I need to get my head around the differences between Classic and AiOps alerts. We’re a pretty new customer so I think I’d just gotten my head around the Classic way of doing things before the new AI overview showed up

Forgive me if I’m missing something, but I’m not seeing a way to configure destinations in Terraform at the moment, is this yet to come?
https://registry.terraform.io/providers/newrelic/newrelic/latest/docs

Thank you for confirming that Warnings in the new data model do not send alerts, that is good to know moving forwards as we transition to the new model.

However, I understand that when you extended the time picker for Alerts classic you see more incidents. If I can rephrase my question [now that I am aware we will not get notifications from AiOps Issues] "why was an AiOps Issue created, but not a Classic incident?

The screenshots show the two pages with the same relative date picker results, and I do not know why the classic alert was not created in the first instance.

The issue and incident that I think is missing from the classic overview is this:
Issue: https://onenr.io/0a2wdB43zjE
Incident: https://onenr.io/0Z2R5OZ4oQb

In my mind, that should have created a classic incident, to which we would have been alerted, however, the incident was not raised (Even though that policy is by alert and signal and should open an incident for all thresholds)

Does that help to clarify my query at all?

Looking forward to your help,

Thanks,

David

[jeffrey_s]

Hi @david119, I recognize this is a lot to untangle (it’s quite a bit for myself as well) and I appreciate your thorough follow-up!

However, this isn’t necessarily what I’d attempted to convey:

Thank you for confirming that Warnings in the new data model do not send alerts, that is good to know moving forwards as we transition to the new model.

It’s actually the opposite — Warnings in the new data model can send notifications. They cannot in the classic data model, because they don’t create classic Incidents.

Here’s another breakdown of the processes that occur within Alerts and AIOps (assuming a by-condition-and-signal policy preference):

---

An alert condition has two thresholds: critical and warning. A breach of a threshold creates simultaneous, yet independent actions in Alerts (classic) and AIOps.

• When a critical threshold is breached:

  • A critical Violation opens in the classic data model, which spurs a (classic) Incident.

  • A critical Incident opens in the AIOps data model, which spurs a critical Issue.

• When a warning threshold is breached:

  • A warning Violation opens in the classic data model, which will roll up into an existing (classic) Incident if one is open. Otherwise, it will not do anything at all.

  • A high-priority Incident opens in the AIOps data model, which spurs a high-priority Issue.

---

Because the AIOps Incident/Issue you linked to is high-priority, we know that in Alerts (classic) it represents a warning Violation. As such, no classic Incidents were created due to its priority.

Warning violations in the classic data model are primarily for “decoration,” whereas in the new data model they’re more actionable and can be configured for notifications. That said, they also appear on the classic “Events” page, and I can find this one in particular if I search by its condition name here.

Does that make sense? If not please let me know, I’m still working out the best way to approach and define these two models.

[jeffrey_s]

Oh, and with regards to Terraform, I don’t believe there’s AIOps support yet but I’ll check in with a product manager.

[david119]

Hi @jeffrey_s

Yes I do think that makes sense. It sounds like my understanding of the classic system was still a little off then! (The joys of not being able to focus on NR for too long due to rushing around so many other areas of our business as well)

I think I’ll raise a task for us to move to the “new” system and ensure our alerts are working there, but we’d still like to maintain the Terraform provider, so any info on that would be appreciated.

In the old model, violations spawn incidents, whereas in the new model, incidents spawn issues.

I do wonder how many other users are going to find that change a little bit confusing> I’m counting myself lucky that we’re quite new and can continue learning without too much of a change in understanding (furthering it of course )

You’ve definitely helped me understand the gap in my knowledge there, thanks for your time today