List of manually closed incidents?

Is there a NRQL query I can run to list all of the incidents which have been manually closed?

Hi, @Larry.Collicott: The answer is… maybe.

Until recently, New Relic did not send any Alerts data to NRDB by default; you would have had to implement a workaround, such as using a webhook channel to send notifications to NRDB, or calling the Alerts API to get a list of violations and inserting them as custom events.

With the release of New Relic AI, however, there is a new (currently undocumented) event, NrAiIncident, that contains information about Alert incident state changes. It has a closeCause attribute, which may provide the information you are looking for.

Unfortunately, because it is currently undocumented, I don’t know if closeCause has a value for incidents that were closed manually, and if so, what that value is. I have a question in to the product manager, and will update this thread if I get an answer.

The new event should be documented within the next few weeks.

2 Likes

We are sending alerts data to NRDB, but it doesn’t seem to include anything about the manual closures.

I took a look at closeCause and see User, so that looks like what I need, but the IncidentId values do not match what we have in the AI incident page.

Right, the workaround methods do not have any information about how an incident was closed.

SELECT count(*) from NrAiIncident facet closeCause since 7 days ago

Does that work for you?

It’s USER for manually closed.

It does, but when in the results the incidentId doesn’t match what is over in AI. Does it match for you?

It doesn’t match the incident ID. Looks like they are matching the Alert Violation ID instead of the incident ID. Looks like a bug. I confirmed the ID with the API for incidents. If you pull up the incident ID you can see that the link to the violations is the ID in the NrAiIncident table for incidentId.

1 Like

@6MM you’re right the incidentId doesn’t match alert incident IDs in their current form, they do match violation IDs.

I’m going to bring in @bgoleno - the Product Manager who can likely help to explain the confusion here around the IDs matching Violations not Incidents.

I think it’s pretty important to have the incident id correct in the new event type. I’d like to also see the alert violation id present. There are other things I’d like to see there, but I don’t want to distract from the issue.

uh oh - @philweber done gone and spilled the beans.
The NrAiIncident has not been “released” / launched yet. There are still some changes being made to that new eventType. That is the first step in our AI Analytics initiative.

There is also another secret - We will be making some changes to the way we name various object in the alerts lifecycle next year. Since the AI Analytics events will be seen as a stable interface, we had to adopt the future naming standards now. Without going into details on that yet, there is one important thing to know…

NrAiIncident events are VIOLATION events. NOT Incident events.

As such, there are only 2 events for Violations (the NrAiIncident event) : open | close

The closeCause attribute has these values:
CONDITION_DELETED - Condition was deleted, and related violations were closed
EVALUATOR. - The signal recovered, and the violation closed automatically
EXPIRED - the Violation Time to Live setting expired
CONDITION_MODIFIED - Condition was modified, and related violations were closed
LOSS_OF_SIGNAL - Los of Signal Expiration threshold was triggered, and it closed open violations
USER - Closed Manually
CONDITION_DISABLED - Condition was disabled, and related violations were closed

2 Likes

Also, we are making some changes to this eventType still. Use at your own risk.
We will be documenting it within the next week or two as well.
One major change we will be making - instead of prefixing user metadata with “tag.” , we will be using “tags.” to be consistent across the product.

See my comment below,
It is not an Incident. It is a Violation ID.

That @philweber guy is pretty awesome.

(He did make it clear that it wasn’t finished and released yet.)

1 Like

I agree. That @philweber is awesome.

1 Like