Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Create a Dashboard for Violations which are open RIGHT NOW which updates as violations are closed

insights
infrastructure

#1

I am attempting to create a dashboard which lists all open incidents (Similar to the ‘Open Incidents’ section under Alerts).

In this dashboard I want to select the hostname and the condition name for servers experiencing an open violation right now. I have created a violation condition named ‘Open Test’ for this. My current strategy is a query which goes:

SELECT hostname, conditionName FROM InfrastructureEvent WHERE conditionName =‘Open test’ AND violationUpdateType = ‘Opened’ Since 1 week ago

The problem with this is that when a violation is open New Relic creates an entry for it in InfrastructureEvent, and then when it is closed New Relic creates a new entry stating it is closed, rather than update the previous opened entry. This means that my query will obviously select all the entries where the violationUpdateType reads opened, meaning it will select entries where this occurred within the past week. This means serves which have violations that are right now closed will appear, as they were open at some point within the past week and those entries are still in the data and have not been altered.

I need a strategy that will allow me to select servers which are experiencing an open violation RIGHT NOW, which updates as the violation closes. Does anyone have any clever suggestions?


Feature Idea: INSIGHTS Dashboard to Show Alert Event When The Incident is in "Open" State & Goes Away when the incident is no longer in OPEN stage
Feature Idea: Retrieving ONLY open alerts in Insights
#2

Hi @jean-claude.carreno - luckily for you there is an in depth post on exactly how you can implement this using existing functionality to insert custom events. Until the alerts are actually Insights events this is the only method of achieving your requirements.


#3

Hi Stefan,

Thank you for this response, it was incredibly helpful. I have followed these instructions successfully, however I still have the same issue:

I have created a custom event called ‘alert’ which calls data from Alerts to INSIGHTS. The objective is to show a dashboard which shows violations which are open right now.

My query is: FROM alert SELECT condition_name WHERE current_state = ‘open’ since 1 week ago.

This dashboard however shows alerts which were once open within the last week, not ones which are open right now. If I check data explorer for alert, it is due to the exact same reason: A new entry is inserted when a violation is closed, however the old entry of it open still remains. This is the exact problem I ran into earlier. For eg:

Data:
10:10 am violation A Incident Closed
10:05 am violation B Incident Opened
10:00 am violation A Incident Opened

I want my query to only show violation B, as it is the only violation that is still open. However it will show A and B, even though A is closed, because in the past week it finds that it was once open. How do I get around this problem?


#4

Ah, sorry Jean-Claude, I read your post too fast and jumped to the wrong solution for you. I’ve thought about how you could approach this in Insights and I cannot think of how this could be implemented.

Viewing the posts on the thread I provided, I think that this is the closest answer you will currently get. I took the NRQL there and modified for some of the attributes that you have provided. All others will need to be reviewed.

SELECT latest(timestamp), latest(violationUpdateType), 
  latest(hostname), latest(condition_name)
FROM InfrastructureEvent
WHERE conditionName = 'Open test' 
FACET incident_id SINCE 1 week ago

#5

Hi Stefan,

A possible solution I thought of was to select every odd entry in the table, as this would mean the violation is still open. For example:

10:10 Violation A Closed
10:05: Violation B Opened
10:00 Violation A Opened

The fact that there is an odd number of entries for violation B means it has not yet been closed. This will work provided the following can never happen:

10:05: Violation A Opened (again)
10:00 Violation A Opened

Is there a query I can use to select odd entries?


#6

Violations are not re-opened as a violation is a unique event. You can verify this by checking the id’s for the violations and alerts.

There is no ability to select the results based on rownum.


#7

Unfortunately this does not take into account that an alert incident can have an ACKNOWLEDGE event. So determining which incident has an odd number of events actually won’t work, as you might miss incidents which were acknowledged.

I’m not sure there is a way, in fact, to craft this sort of report (all OPEN incidents from the past week) using Insights. I have already opened a feature request on your behalf, but I would also recommend opening a new thread on the Feature Ideas: Alerts section, as this is a great idea but would need some work on our end. Ideally, this would be possible using a single API call and would show all open incidents on your account, ignoring incidents that had already closed.


#8

I might have found a step in the direction of this open/closed violation, and that is adding uniques(current_state) to the query like my query below:

SELECT latest(details), latest(timestamp), uniques(current_state) FROM alert FACET (condition_id) where since 3 days ago

this produces a Json,

would it be possible to make a where clause that whipes out all results that have ‘closed’ in their members collection?

I think that this is what many people are looking for, but my knowledge on NRQL is to short to know if that is achievable.


#9

Have you tried adding:

WHERE current_state != 'closed'

#10

Yes I tried, and No, that is not the solution I’m looking for
as that will only whipe out the closed result from the collection.
but I also don’t want to see the the ‘open’ and ‘acknowledged’ result of an incident if that incident has a closed status.
in other words, what I want to achieve is to not pop up the incident number at all if that incident is actually closed.


#11

This is something I’ve tried many times to accomplish using the current feature set in Insights and I’ve come to the conclusion that it just isn’t possible. Hopefully, if it gets enough attention as a feature request, it’s something that can be implemented down the line.


#13

Here is a query that should do it, I think:

SELECT clamp_min(
         filter(count(*), WHERE current_state='open') -
         filter(count(*), WHERE current_state='closed'),
       0) AS 'Open Alerts'
FROM Alert SINCE 30 days ago

Here is what happens:

  • calculate the difference between open & closed alert events (the 2 filter functions)
  • use a longer time range (30 days), assuming most incidents should not run that long (i.e. if there was an incident opened 32 days ago and is still open it would not be counted); adjust to your needs
  • use clamp_min(*, 0) to avoid negative numbers (if there was an incident that opened 32 days ago and was closed 28 days ago it would count just a “closed” event for it, and all other incidents have been closed, it would lead to a negative difference)

The only problem is that with such a long SINCE, the chart will not auto update frequently on a dashboard. With 30 days it’s every 12 hours. So you won’t automatically see your new alerts immediately there.


#14

The only other way I can see is that you build a component on your side that regularly sends the current number of open alerts as insight event. It would have to poll the alerts API or track open/closed incidents via webhooks + keeping track of the total. Then send an event every minute or so, depending on how quick you need to see new alerts on the dashboard.


#15

I like this query @aklimets. It’s a good use of the new math functions (i.e. clamp_min) available in Insights. I think @jean-claude.carreno is looking for a widget that lists the conditionName attribute for each open violation. I still don’t believe this is possible with the current feature set.