Is there a NRQL query I can run to list all of the incidents which have been manually closed?
Hi, @Larry.Collicott: The answer is… maybe.
Until recently, New Relic did not send any Alerts data to NRDB by default; you would have had to implement a workaround, such as using a webhook channel to send notifications to NRDB, or calling the Alerts API to get a list of violations and inserting them as custom events.
With the release of New Relic AI, however, there is a new (currently undocumented) event,
NrAiIncident, that contains information about Alert incident state changes. It has a
closeCause attribute, which may provide the information you are looking for.
Unfortunately, because it is currently undocumented, I don’t know if
closeCause has a value for incidents that were closed manually, and if so, what that value is. I have a question in to the product manager, and will update this thread if I get an answer.
The new event should be documented within the next few weeks.
We are sending alerts data to NRDB, but it doesn’t seem to include anything about the manual closures.
I took a look at
closeCause and see
User, so that looks like what I need, but the
IncidentId values do not match what we have in the AI incident page.
Right, the workaround methods do not have any information about how an incident was closed.
SELECT count(*) from NrAiIncident facet closeCause since 7 days ago
Does that work for you?
It’s USER for manually closed.
It does, but when in the results the incidentId doesn’t match what is over in AI. Does it match for you?
It doesn’t match the incident ID. Looks like they are matching the Alert Violation ID instead of the incident ID. Looks like a bug. I confirmed the ID with the API for incidents. If you pull up the incident ID you can see that the link to the violations is the ID in the NrAiIncident table for incidentId.
@6MM you’re right the incidentId doesn’t match alert incident IDs in their current form, they do match violation IDs.
I’m going to bring in @bgoleno - the Product Manager who can likely help to explain the confusion here around the IDs matching Violations not Incidents.
I think it’s pretty important to have the incident id correct in the new event type. I’d like to also see the alert violation id present. There are other things I’d like to see there, but I don’t want to distract from the issue.
uh oh - @philweber done gone and spilled the beans.
The NrAiIncident has not been “released” / launched yet. There are still some changes being made to that new eventType. That is the first step in our AI Analytics initiative.
There is also another secret - We will be making some changes to the way we name various object in the alerts lifecycle next year. Since the AI Analytics events will be seen as a stable interface, we had to adopt the future naming standards now. Without going into details on that yet, there is one important thing to know…
NrAiIncident events are VIOLATION events. NOT Incident events.
As such, there are only 2 events for Violations (the NrAiIncident event) : open | close
The closeCause attribute has these values:
CONDITION_DELETED - Condition was deleted, and related violations were closed
EVALUATOR. - The signal recovered, and the violation closed automatically
EXPIRED - the Violation Time to Live setting expired
CONDITION_MODIFIED - Condition was modified, and related violations were closed
LOSS_OF_SIGNAL - Los of Signal Expiration threshold was triggered, and it closed open violations
USER - Closed Manually
CONDITION_DISABLED - Condition was disabled, and related violations were closed
Also, we are making some changes to this eventType still. Use at your own risk.
We will be documenting it within the next week or two as well.
One major change we will be making - instead of prefixing user metadata with “tag.” , we will be using “tags.” to be consistent across the product.
See my comment below,
It is not an Incident. It is a Violation ID.
That @philweber guy is pretty awesome.
(He did make it clear that it wasn’t finished and released yet.)
I agree. That @philweber is awesome.
Are we going to be getting incident id’s on this event or will be we getting another event type for Incidents?
I guess you didn’t like the ring that NrAiViolations had…
@JoiConverse - I cannot find any mention in the documentation on this event and there are questions around the data retention. There only appears to be 8 day’s worth of event data in my account. I removed the custom webhook we were using to record Alert events on 18th November and there are no NrAiIncident events before 23rd November
@JoiConverse - Any update on the data retention for NrAiIncident events please?
It’s not released yet.
@stefan_garnham There isn’t an update to share on this currently. @bgoleno and team know there is much interest in this feature and will update this thread, and corresponding documents, as soon as there is an update to share.
We will retain Alert Analytic events, such as this, for 15 months.
If you are experiencing otherwise, please let me know.