Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: Creating a Widget to Display Open Alert Violations

alerts
violations
rfb

#1

Introduction

Customers often ask how to display open alert incidents on a dashboard. You can configure New Relic Alerts to send incident data to Insights using a webhook, but that creates an event whenever an incident is opened, acknowledged, or closed.

The challenge

While this data is great for analyzing alert incidents after the fact — Which alert conditions are violated most frequently? Which entities generate the most violations? What is the mean time to resolution (MTTR)? — it is difficult or impossible to write a query that returns only incidents that are currently open. You can view a list of open incidents in the Alerts UI, but many customers would like to display this data on a dashboard, alongside other performance metrics.

The solution

Fortunately, the New Relic REST API includes endpoints that return lists of Alerts incidents and violations; these endpoints accept an only_open parameter to specify that they should return only items that are currently open. But how do you display this information on a dashboard?

The script below queries the REST API to get a list of open violations, then iterates over the list and generates an Insights custom event called AlertViolationsSample with information about each violation.

const request = require('request');

var headers = {
  'Content-Type': 'json/application',
  'X-Api-Key': $secure.API_KEY
};

var options = {
  url: 'https://api.newrelic.com/v2/alerts_violations.json',
  qs: { 'only_open': 'true' },
  headers: headers
};

var violations = null;

// Get open violations
request.get(options, 
  function(err, response, body) {
    if (response.statusCode == 200) {
      var data = JSON.parse(body);
      violations = data.violations;
      // For each violation...
      for (var i = 0; i < violations.length; i++) {
        // ...create Insights event
        insertViolationEvent(violations[i]);
      }
    }
  }
);

function insertViolationEvent(violation) {
  var headers = {
    'Content-Type': 'json/application',
    'X-Insert-Key': $secure.INSERT_KEY
  };
  var options = {
    url: 'https://insights-collector.newrelic.com/v1/accounts/' + $secure.ACCOUNT_ID + '/events',
    headers: headers
  }
  options['body'] = JSON.stringify(
    {
      'eventType': 'AlertViolationsSample',
      'id': violation.id,
      'label': violation.label,
      'duration': violation.duration,
      'policy_name': violation.policy_name,
      'condition_name': violation.condition_name,
      'priority': violation.priority,
      'opened_at': violation.opened_at,
      'entity.product': violation.entity.product,
      'entity.type': violation.entity.type,
      'entity.id': violation.entity.id,
      'entity.name': violation.entity.name
    }
  );
  request.post(options);
}

Once this data is in Insights, you may query it with NRQL to create a dashboard widget:

SELECT 
  latest(duration) / 60 AS 'Duration (mins)', 
  latest(condition_name), latest(label), latest(policy_name),
  latest(`entity.name`), latest(priority)
FROM AlertViolationsSample 
FACET id

Opportunities for enhancement

I have written my script as a New Relic Synthetics API test, taking advantage of Synthetics’ secure credentials feature. To use it as written, create secure credentials in Synthetics for your New Relic REST API key, Insights insert key, and your New Relic account ID. Then create a new API test monitor and paste the above code into the script editor. You should schedule the monitor to run from at least one location once per minute to minimize latency.

You may prefer to run the script elsewhere, such as at AWS as a lambda function, or even on your local computer. That is fine; you just need to replace the secure credential values in the script (beginning with $secure) with your account’s values.

My script queries open violations from the REST API; you may prefer to see open incidents instead. Again, no problem: change the API endpoint to https://api.newrelic.com/v2/alerts_incidents.json. You will also need to modify the values sent to Insights to correspond to those returned by the API.

Finally, if you have a large number of open incidents or violations, the REST API paginates the result. You will have to examine the response and see if there are additional pages; if there are, you can call the API again, passing a page parameter to request the desired page.


NRQL for current open alerts
Creating a status page in NewRelic
Create a Dashboard for Violations which are open RIGHT NOW which updates as violations are closed
Status dashboard for all the services
Display alerts or open incidents for multiple accounts
Incident Dashboard
Create a Dashboard for Violations which are open RIGHT NOW which updates as violations are closed
#3

FYI, the NRQL for the top 3 widgets is:

SELECT uniqueCount(id) 
FROM AlertViolationsSample 
FACET priority / condition_name / `entity.name`

#4

Awesome. Thank you @philweber. I have noticed that “No chart data available” is displayed when there is no open violations. I would prefer to display “0” instead. I wonder if there’s a way to do this?


#5

Hi, @walter.simba: For Violations by condition, you can do this:

SELECT 
  filter(uniqueCount(id), WHERE priority = 'Critical') AS 'Critical', 
  filter(uniqueCount(id), WHERE priority = 'Warning') AS 'Warning' 
FROM AlertViolationsSample 

That will also list critical violations first, even if the number of warning violations is higher. I do not know how to display 0 when there is no data for the other widgets.


#6

@philweber This is a helpful workaround, but I am also curious as to why NR has not added this data set to the Insights product. If NR already has access to it, why not expose it via Insights?


#7

I am not on the product team, so I don’t know the answer to that question. I am just a user of the products like you, finding ways to get things done. :slight_smile:


#8

@philweber this is a super solution. In relation to the pagination and maybe adding a start and end data,
I would like to add something like the following,
-G -d ‘page=36&start_date=2019-07-15T09:50:00+00:00&end_date=2019-08-15T09:50:00+00:00’
I know there are a total of 36 pages when I run the api call, so how best would I be able to incorporate that into the script to added above …?
I would run the script once for each page returned, ie. page=35 and run, then change to page=34 and run, and so forth


#9

Ignore that @philweber

I have amended the variable ‘options’ to the following,

var options = {
url: ‘https://api.newrelic.com/v2/alerts_violations.json?page=1&start_date=2019-07-15T09:50:00+00:00&end_date=2019-08-15T09:50:00+00:00’,
headers: headers
};