Alert conditions to monitor once in a day (or more) events

Some of the events that we have in our apps just happen once a day or even in bigger time gaps.

For those events normally alert conditions are created to detect if they failed to happen but due to the big data gaps, a “normal” alert condition is not able to monitor and create the incidents as expected.

Using the Signal lost threshold (up to 48h gap) is a possible workaround but it has some limitations:

  • If the event happens outside of the expected time frame it will reset the time count to the next event.

    • Eg: If the event that was supposed to happen at 06:00 am happens at 08:30 am, a 24h signal lost threshold would wait until 08:30 am of the next day to create the incident (not 06:00 am as was expected)
  • If the event fails twice in a row, the second incident wouldn’t be created as the signal didn’t return between the 2 failures

You can solve this issue using a Synthetics monitor, the Insights API, and the alert conditions💡!

As the Synthetic monitor is able to run a quick check time-to-time, we can configure it to, when inside the expected time window(s) check if the event(s) have happened and take action if it didn’t.

The scenario

You expect that every day at 16:00 one of your apps creates specific logs and submits them to New Relic and you want to have an alert to be created in case it doesn’t happen.

Creating the monitor

  • You are going to need a Scripted API monitor
    • That will run every 1 hour from your preferred location
    • Which the script example is:
var assert = require('assert'); //Method to validate your the results
var myAccountID = '<YOUR_ACCOUNT_ID>';
var myQueryKey = '<YOUR_INSIGHTS_QUERY_KEY';

//The hours should consider UTC time.
var startHour = 16;
var startMinutes = 0;
var endHour = 17;
var endMinutes = 0;

//Hour to minutes
var start = startHour * 60 + startMinutes;
var end = endHour * 60 + endMinutes;

//Getting the current hour (as minutes)
const date = new Date();
const now =  date.getHours() * 60 + date.getMinutes();

//If the now is inside the right time window
if(start <= now && now <= end){
   
  //The query needs to be URI encoded as it will be included on the GET request
   var query = encodeURIComponent("FROM Log SELECT count(*)");
  
   var requestOptions = {
       //Define endpoint URI
       uri: 'https://insights-api.newrelic.com/v1/accounts/'+myAccountID+'/query?nrql='+query,
       //Define query key and expected data type.
       headers: {
       'X-Query-Key': myQueryKey,
       'Accept': 'application/json'
       }
   };

   //Make GET request, with the options and callback function.
   $http.get(requestOptions,parseResults);
}


/* This function gets the results of the GET request, 
   parses it and evaluates the information.

   Once evaluated the actions will be taken, on that example, make this request of the monitor fails.*/
function parseResults (err, response, body){

   assert.ok(response.statusCode == 200, 'Expected 200 OK response');  //Validate the response 
   var info = JSON.parse(body); //Parse the body 
   assert.ok(typeof(info.results[0].count) == 'number', 'Expected query result in the response'); //Validate the result of the query
   var logCount = parseFloat(info.results[0].count); //Parse it to float
  
   //If there are no logs make this request fail.
   if(logCount == 0){
       assert.fail('Logs not created!');
   }
}
  • Save and give a proper name to the monitor

Creating the alert condition

As we have the monitor, that fails in case there are no logs, we just need to create the alert condition:

  • Type: Synthetics - Single failure
    Screenshot 2021-05-25 at 18.14.03

  • Target the monitor created above to be monitored
    Screenshot 2021-05-25 at 18.16.13

  • Give a proper name and save the condition
    Screenshot 2021-05-25 at 18.14.44

  • Save the condition


Important: the purpose of this article is to mention the idea and provide a guideline. The script used was adapted from the examples provided in the New Relic documentation * it should be adapted to your necessity.

Related content

10 Likes