Relic solution: Alert conditions to monitor once in a day (or more) events

Some of the events that we have in our apps just happen once a day or even in bigger time gaps.

For those events normally alert conditions are created to detect if they failed to happen but due to the big data gaps, a “normal” alert condition is not able to monitor and create the incidents as expected.

Using the Signal lost threshold (up to 48h gap) is a possible workaround but it has some limitations:

  • If the event happens outside of the expected time frame it will reset the time count to the next event.

    • Eg: If the event that was supposed to happen at 06:00 am happens at 08:30 am, a 24h signal lost threshold would wait until 08:30 am of the next day to create the incident (not 06:00 am as was expected)
  • If the event fails twice in a row, the second incident wouldn’t be created as the signal didn’t return between the 2 failures

You can solve this issue using a Synthetics monitor, the Insights API, and the alert conditions💡!

As the Synthetic monitor is able to run a quick check time-to-time, we can configure it to, when inside the expected time window(s) check if the event(s) have happened and take action if it didn’t.

The scenario

You expect that every day at 16:00 one of your apps creates specific logs and submits them to New Relic and you want to have an alert to be created in case it doesn’t happen.

Creating the monitor

You are going to need a Scripted API monitor that will run time-to-time from your preferred location.
The time that the monitor will run depends on the query and how often you need that the event to be verified during the day.

Eg:

  • The query used in the example script below doesn’t have a SINCE clause, so it will return the default time (60 minutes)
  • Time window between the start and end times is 1 hour, so my monitor can run each 1 hour and still be able to verify the event inside the evaluation window.

Please find below the example of the script:

var assert = require('assert'); //Method to validate your the results
var myAccountID = '<YOUR_ACCOUNT_ID>';
var myQueryKey = '<YOUR_INSIGHTS_QUERY_KEY';

//The hours should consider UTC time.
var startHour = 16;
var startMinutes = 0;
var endHour = 17;
var endMinutes = 0;

//Hour to minutes
var start = startHour * 60 + startMinutes;
var end = endHour * 60 + endMinutes;

//Getting the current hour (as minutes)
const date = new Date();
const now =  date.getHours() * 60 + date.getMinutes();

//If the now is inside the right time window
if(start <= now && now <= end){
   
  //The query needs to be URI encoded as it will be included on the GET request
   var query = encodeURIComponent("FROM Log SELECT count(*)");
  
   var requestOptions = {
       //Define endpoint URI
       uri: 'https://insights-api.newrelic.com/v1/accounts/'+myAccountID+'/query?nrql='+query,
       //Define query key and expected data type.
       headers: {
       'X-Query-Key': myQueryKey,
       'Accept': 'application/json'
       }
   };

   //Make GET request, with the options and callback function.
   $http.get(requestOptions,parseResults);
}


/* This function gets the results of the GET request, 
   parses it and evaluates the information.

   Once evaluated the actions will be taken, on that example, make this request of the monitor fails.*/
function parseResults (err, response, body){

   assert.ok(response.statusCode == 200, 'Expected 200 OK response');  //Validate the response 
   var info = JSON.parse(body); //Parse the body 
   assert.ok(typeof(info.results) == 'undefined', 'The query didn't return any results');
   assert.ok(typeof(info.results[0].count) == 'number', 'Expected query result in the response'); //Validate the result of the query
   var logCount = parseFloat(info.results[0].count); //Parse it to float
  
   //If there are no logs make this request fail.
   if(logCount == 0){
       assert.fail('Logs not created!');
   }
}
  • Save and give a proper name to the monitor

IMPORTANT

The validation above done by the function parseResults needs to be adapted depending on the query used.

The object returned by the Insights API will change depending on the aggregator functions and other claused used in the query. Eg: FACET

So, check the payload returned and adapt the validation accordingly.


Creating the alert condition

As we have the monitor, that fails in case there are no logs, we just need to create the alert condition:

  • Type: Synthetics - Single failure
    Screenshot 2021-05-25 at 18.14.03

  • Target the monitor created above to be monitored
    Screenshot 2021-05-25 at 18.16.13

  • Give a proper name and save the condition
    Screenshot 2021-05-25 at 18.14.44

  • Save the condition


Important: the purpose of this article is to mention the idea and provide a guideline. The script used was adapted from the examples provided in the New Relic documentation * it should be adapted to your necessity.

Related content

11 Likes