Different alerts for certain time periods

I have an alert for incoming requests for something. These requests happen every 5 minutes, but there are less requests during the night.

I have the following settings:

SELECT count(name) FROM Transaction WHERE transactionSubType = 'Custom' AND name = 'OtherTransaction/Custom/Requests'

When the query returns a value below 1 for at least 30 mins.

Now, this is sometimes triggered during the night when it is perfectly normal not to have any incoming requests. What I want is that I am not alerted during the night (threshold should be different), but during the period of 08.30 - 21.00 I am. I cannot use the baseline as it also alerts when I have more requests than the baseline. I only want be alerted when it goes below the baseline.

How do I accomplish this?

Hi @ssietses - By filtering the hours that you do not want covered, your NRQL alert should work as required. Use this post as a starting point:

@stefan_garnham maybe I don’t understand, but how can this query help? I don’t want to use average, I want to do a count. If I exclude the times I don’t want it will return 0 which will trigger the alert. But maybe I misunderstood?

So what I want is:
If I have 0 zero requests outside 08.30 - 21.00 timespan for the last 60 min I want to be alerted
If I have 0 zero requests inside 08.30 - 21.00 timespan for the last 15 min I want to be alerted

So inside ‘business hours’ I want to be alerted faster than outside these hours. How do I do this? thanks!!

The query shows you how to exclude hours that you are not checking:

WHERE ( hourOf(timestamp) NOT IN ('1:00','2:00','3:00','4:00')

2 Likes

Hi, @ssietses: You might also take a look at metric baseline alerts, which are designed for just this use case. Rather than triggering a violation when a metric crosses a fixed threshold, metric baselines compare the current value with historical behavior, and can notify you if the metric deviates by more than a selected amount.

1 Like

Hi Phil, I know, but it will also alert if we have more request than normal. And I only want to get alerted when it is lower than my base line.

Hi Stefan,

See the following query, it filters out all the results from these hours, which results in 0. Then I will get alerts.

1 Like

Of course it does, I was focused on the times, not the resulting value. I would suggest something like this as a workaround but I am not sure it will work based on the documentation not stating that filter is supported.

SELECT
  filter(100, WHERE hourOf(timestamp) in ('01:00', '02:00')) + count(name) from Transaction
WHERE
  transactionSubType = 'Custom' AND name = 'OtherTransaction/Custom/Requests'

If it doesn’t work, I’ll raise a feature request.

Hi, @ssietses: Another option is to use the REST API to change the duration at different times of the day. For example, at 08.30 you could run a script that sets the duration for the alert condition to 15 minutes:

curl -X PUT 'https://api.newrelic.com/v2/alerts_nrql_conditions/{condition_id}.json' \
     -H 'X-Api-Key: YOUR_ADMIN_API_KEY' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
  "nrql_condition": {
    "name": "Condition Name",
    "enabled": "true",
    "terms": [
      {
        "duration": "15",
        "operator": "below",
        "priority": "critical",
        "threshold": "1",
        "time_function": "all"
      }
    ],
    "value_function": "single_value",
    "nrql": {
      "query": "SELECT count(name) FROM Transaction WHERE transactionSubType = '\''Custom'\'' AND name = '\''OtherTransaction/Custom/Requests'\''",
      "since_value": "3"
    }
  }
}' 

At 21.00, you can run a script that updates the duration to 60 minutes:

curl -X PUT 'https://api.newrelic.com/v2/alerts_nrql_conditions/{condition_id}.json' \
     -H 'X-Api-Key: YOUR_ADMIN_API_KEY' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
  "nrql_condition": {
    "name": "Condition Name",
    "enabled": "true",
    "terms": [
      {
        "duration": "60",
        "operator": "below",
        "priority": "critical",
        "threshold": "1",
        "time_function": "all"
      }
    ],
    "value_function": "single_value",
    "nrql": {
      "query": "SELECT count(name) FROM Transaction WHERE transactionSubType = '\''Custom'\'' AND name = '\''OtherTransaction/Custom/Requests'\''",
      "since_value": "3"
    }
  }
}'
2 Likes

Thanks for the fast response!

It is not working though, it is just shifting the baseline to > 100 :expressionless:

Thanks, this is another option if there is not another way.

I am a bit sad that I have to write a service or use another product do this. And that this option is not available in New Relic alerts. If I was able to do: “When the query returns a value below 1 for at least 30 mins between 08:30 and 21:00.”, then my problem was solved.

But thanks for your replies anyways!

I use 100 as an example, as you didn’t want the alert firing during those hours if it reaches zero. If the count returns zero and the time is not between the hours you want, then the alert would fire.

I saw that I missed the timestamp hour calculation. I’ve edited the post to fix it.

1 Like

@stefan_garnham check my above screenshot again, as I understand it correctly, it should give >= 100 on 03:00 and 04:00, but before and on 05:00 it should go back to the right numbers, right? But as you can see it stays above 100…

Hi @ssietses,

Since your use case is distinctive, in that you want to be notified when this number flatlines at 0 but not during certain times of the day, the normal workarounds using hourOf(timestamp) will not work.

I would suggest setting up a couple of simple cron jobs that use the REST API to disable the alert condition overnight, then re-enable it in the morning. @philweber suggested something like this, but his idea involved extending the duration of the threshold, which doesn’t really help as you’ll still get false positives during the night even if you set the threshold duration to the maximum (2 hours). If you actually disable the alert condition altogether, then re-enable it in the morning (one cron job to disable, a 2nd cron job to re-enable), you should get exactly the results you’re looking for.

Instructions on how to use the REST API to enable/disable NRQL alert conditions can be found in this document.

I would also recommend posting the idea of configurable downtime windows for New Relic Alerts in the Feature Ideas section, since this would be the ideal solution.

I hope this helps!

2 Likes

Do we have solution for the above mentioned issue now?

Hello, @vkaushik1: All of the available solutions are discussed in this thread. You may:

1 Like

Hi, @ssietses: Revisiting this topic, I wanted to mention that metric baseline conditions now support evaluating whether a metric exceeds an upper or lower threshold, so they may work for your use case.

1 Like