Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Sending Alerts data to Insights

alerts
insights

#1

As I was teaching a customer about Insights this week, they had a “lightbulb moment”: “Ooh,” they asked, “can we use Insights to analyze Alerts data? Create a dashboard that shows how many incidents have occurred for each alert policy, and which conditions triggered each incident?” Turns out you can, and it’s surprisingly easy to set up.

In order to set it up, you’ll need the following information from your Insights account:

  • An Insert Key for the Insights API
  • Your New Relic account ID, which is included in the Insights events API endpoint (https://insights-collector.newrelic.com/v1/accounts/[account_id]/events)

Now that you have the necessary information, create a Webhook notification channel in Alerts. In the Base Url field, enter the Insights events API endpoint. Add a Custom Header named X-Insert-Key, and set its value to your Insights API Insert Key:

Next, customize the JSON payload that Alerts will send to Insights: add an additional field named eventType, and specify what you’d like to call your custom Insights event (I called mine alert):

You may optionally delete or rename attributes (the values to the left of the colons). Save your changes, and associate your new Webhook channel with one or more alert policies.

That’s it! Whenever a condition is violated on one of those policies, Alerts will post the details of the incident to Insights, where you may slice and dice it to your heart’s content.


Using Insights to calculate MTBF and MTTR
How can I query my events and make a report?
Question on Alert Events API
Create a Dashboard for Violations which are open RIGHT NOW which updates as violations are closed
Insert Synthetic Check results into Insights
Love our Lurkers Week DAY 4: Testimonials
Alerts Policy and Alert configuration
Feature Idea: New Alerts Incident as Queryable "Events" in Insights
Count of NR Incidents that cause an Violation with status Critical
Are Your Ops Standups Productive? Let's Fix That!
Violation condition for synthetics test?
Acknowledging Incident Directly from Insights Dashboard
Fetch Incident or Alerts\events based on Time stemp
Is it possible to get the open incident count from alert section
How to get all incident ID'S for a period of time?
Need get the average incidents alerts and metrics triggered in a month
Query used for 'Recent alerts' dashboard?
Implementing New Relic Alerts
How to measure throughput and alert if it impacts latency
How can I report the alerts
Monthly report of Same/similar alerts?
Webhooks - Can you include any data you want to query in a Webhook?
Is there a way to view all alerts that are connected to one server
#2

@philweber - this is an interesting use of alerts.

A feature request to have alert event data queryable by Insights would be more flexible. If I want to utilise this information AND another web hook then I would have to write an interface to handle both.


#3

Hi, @stefan_garnham: An alert policy can send notifications to multiple webhook channels; using the technique I describe does not prevent you from sending data to another webhook.


#4

+1 for making available via insights - much more robust to have things available across the account rather than custom configuration into every alert.

For this kind of incidents into insights approach with the current setup, I’d rather pull back the data via API calls and then poke everything back into a custom events table. A single script can iterate over every subaccount, every incident and every violation without requiring the individual application teams to correctly format and setup each alert webhook.


#5

@pault The new Alerts API allows programmatic reading and, if necessary, writing of webhook channels - so, with a bit of API setup, you don’t have to require individual application teams to correctly format each alert webhook.


#6

This looks like a great way to visualize alerts. We haven’t yet worked with Insights and I’m having issues creating the webhook appropriately. In the Create channel screen, webhooks only shows the Webhook URL, but not Base Url, Basic Auth, Custom Headers configuration sections. What am I missing? Is this a function of the new Alerts that doesn’t exist in legacy alerts?


#7

Hi, Logan: Yes, this technique requires the new Alerts.


#8

Thanks, this is good to know. We will await the new Alerts going live.


#9

Clever and useful if you have a small organization or few people using alerts. In a larger org it would be far more useful to have this built in so that all alerts are aggregated and there is no chance of human error in not including the channel.


#10

Great input, @eschumac! I would be happy to put in a feature request for you about this! :thumbsup: My product managers like to know this kind of info!


#11

This seems like a crazy workaround for something that should be in insights in the first place by default.

When New Relic sales was on site at our company and we were evaluating, one of the main selling points of Insights is that it collects the data of all the products into one query-able interface. Which is a powerful selling point, so why Alerts was left out of this seems crazy to me!

Is there a reason alerts data wasn’t included in insights from the beginning?

New Relic, you have the alerts data, you own Insights please please just bring Alerts under the insights umbrella like the rest of your products, and don’t make us do workarounds like this.

Thanks!


#12

Hi @jsprague, I totally agree with you. Bringing incident lifecycle from Alerts into Insights is on my roadmap. We’ve actually been doing it for ourselves for a while now and as you can imagine being able to query that data with NRQL is in fact awesome. I hope to be able to extend this to you all soon, I just don’t have an ETA at the moment.

However, I just created this thread If Alerts data was in Insights, what questions would you ask? and would love to know more about the questions you’d want to ask of the data.


#13

@NateHeinrich That is fantastic news! I’ll definitely reply to the other thread. Thanks!!


#14

Would have been super helpful to supply that JSON as text in addition to the screen grab!

EDIT: I take that back. When you click “Use Custom Payload” it is automatically populated with most of the JSON.


#15

This is a great solution @philweber As part of this, is it possible to have a widget to show incidents that have been open for 60mins or more …? The widgets in the example are great, but having this extra widget would be good from an OLA/SLA perspective.


#16

Hi, @LAMBERT: I’m not sure how you would do that. Alerts sends one event when an incident opens, and another when it is closed; I don’t know how to write a query that says, “Show me all incidents that were opened more than 60 minutes ago, and for which there is no corresponding close event.”

The closest I can come up with is this:

SELECT latest(timestamp), latest(current_state), 
  latest(account_name), latest(policy_name), 
  latest(condition_name), latest(details) 
FROM Alert 
FACET incident_id SINCE 1 hour ago

This will show you the most recent event for each incident; you may sort the resulting table by current_state to see which ones are still open.


Create a Dashboard for Violations which are open RIGHT NOW which updates as violations are closed
#17

Hi @philweber. That looks like a great insights dashboard to display the alert metrics and details.

Would you be able to provide the insights queries you used to build this dashboard? Have been playing around and can’t get it to work correctly.

Thanks :slight_smile:


#18

Sure, @DBarkho:

Incidents by Policy:
SELECT count(*) FROM Alert FACET policy_name

Incidents by Condition:
SELECT count(*) FROM Alert FACET condition_name

Incident Details:
SELECT timestamp, incident_id, policy_name, condition_name, details, severity FROM Alert

The procedure for adding click-to-filter is described in this post:


Query used for 'Recent alerts' dashboard?
Alerts Policy and Alert configuration
#19

Hi,

This is something which surely helps us.
However i would like to know if we will be able to display “targets” in the incident details query, when i try it states no value.
Basically along with alert details i would need to capture the target server which raised the raised.

Regards,
Maria


#20

We’re over 1.5 years from the original date of this post. Is there still not an integrated approach to getting Alert data into Insights? Or do we still have to set up each policy to send to a custom webhook? I thought this was on someone’s roadmap? :wink: @NateHeinrich