Sending Alerts data to Insights

HI, @alex.tulikumwenayo: There are no ‘opened_at’ or ‘closed_at’ properties. New Relic Alerts generates separate events when an incident is opened or closed; the value ‘open’ or ‘closed’ is passed in the $EVENT_STATE attribute.

1 Like

Thank you @philweber. How can I include the duration of the violation?

I don’t think you can.

Can I send vía payload the affected server of the alert?
I’m using Infrastructure product. I want to facet the chart by affected server.

1 Like

FWIW, we have a recent alerts dashboard that uses the following NRQL to show recent alerts, including the duration of them:

SELECT max(timestamp) as 'Time',
       latest(current_state) as 'State',
       (max(timestamp)-min(timestamp))/60/1000 as 'Duration (min)',
       latest(severity) as 'Severity',
       latest(policy_name) as 'Policy',
       latest(details)
FROM alert
FACET incident_id
SINCE 1 day ago
LIMIT 99
6 Likes

Hi @tstansell - That is a great NRQL query. Can you add it to the NRQL Library or link to your post in the library?

2 Likes

Is it possible to group alerts by “target” application? I’d like to produce a chart like:

app-1 |XXXXXXHHHHHHH (grouped by severity)
app-2 |XXXXHHH

… but I can’t see the ‘targets’ attribute in alert webhook payload in Insights

The targets field is in an array. You will need a way to flatten them so you can send it to insights.

2 Likes

How can I “flat” this array on Custom Alert webhook? I can’t find this information on documentation :slightly_frowning_face:

We send it to a function that takes the array apart / flatten it, then send it to insights.

1 Like

I got the following error trying to test the webhook on the notification channel

    {

    response: 200,

    Unable to invoke Webhook. Webhook responded with HTTP status code 403.

    }

@David.Mijares - I took a look at your webhook notification channel, it seems like you are using X-Api-Key as your custom header.

That needs to be X-Insert-Key for the Insights API. Let us know your test goes if you get that updated.

6 Likes

you were right all good now

2 Likes

Fantastic - thanks for letting us know! :smiley:

1 Like

Hi @philweber - Just implemented this new alerts dashboard using your webhook. Awesome stuff!

I am trying to work out how I can show a Billboard type query that shows a count of the currently open and currently warning alerts (with thresholds set so it is red if more than one alert is open)
Its a bit tricky, as the records just get added, not updated .

So if I use this query :

SELECT count(incident_id) from alert WHERE severity = ‘CRITICAL’ and current_state = ‘open’

It shows that I have one open alert. But that alert has since been closed.
Any ideas how I do this?

Hi, @brett.howells: Since I posted this, the Alerts team has added a duration element to the JSON payload. I wonder it you can query for WHERE duration IS NULL?

Sadly not. It appears that the duration is also set for the open ones.

They do appear to be set to maybe a default value. My first open alert has a duration value of 3,653 and the second has 3,655 - where as the closed alert record has a value of 222,232

If the records are not updated, and this duration is set to a default or at least a low value, perhaps I could do something like:

SELECT count(incident_id) FROM alert WHERE duration /1000 <= 4 and current_state = ‘open’

Actually, just closed of an alert and that also doesnt work, because that record is still there.
I need to somehow take the latest timestamp for an incident ID into account?
Or not count it if there is another record that is closed with the same incident id?

Another thing I just noticed - Warning alerts do not appear to be sent to the new alert table in insights.
Only CRITICAL alerts get sent over. Is this because there is no alerting set on alert conditions?

One more thing @philweber - I can see a few people asking how we select the app name when posting to insights.
I am currently passing : “targets”: “$TARGETS”,
That looks like an array, so how do I either flatten that, or maybe just send the first target?

Hi, @brett.howells: This is covered earlier in the thread:

You will have to create your own API endpoint to receive the webhook from New Relic Alerts, modify the payload as desired, then post it to Insights.

2 Likes