Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Infrastructure alert for monitoring Disk Utilization - Peaks during weekend backups

feature-idea
infrastructure
nrql-alerting

#1

Hi guys,

I have the following case and I can’t find suitable solution on it.
We have Alerts for monitoring the Disk Utilization (Infrastructure Alerts). The conditions are created to monitor group of hosts (location-enviornment-application), which is perfect for us, but when it comes to peak of alerts because high Disk Utilization on some of them during the Weekend backups, I don’t know how to avoid the fake incidents (not fake, but not needed).

I have tried the NRQL Alerts with Baseline

but it seems that it won’t work for me. In the group of hosts I monitor I have 4 servers, but the NRQL alerts is watching them as one (We have many groups of hosts from 2 to 20 servres in them). What I want is as it is in the Infrastructure alerts, to monitor every host.

I am trying something with where_clause, but can’t think of a good logic to do what I want. Please if someone have an idea, share it with me :slight_smile:

Thank you,
Milen


New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Hi, @Milen.Palavrov: Try adding a FACET clause:

SELECT average(diskWriteUtilizationPercent)
FROM SystemSample
WHERE entityName LIKE '%pattern%'
FACET entityName

Now you should see each host individually.


#3

Hello @philweber,

We have tried it, but…

It cannot be Baseline with Facet.


#4

Hey @Milen.Palavrov - It looks like you have run into a limitation of baseline alerts. Such that NRQL Facets are not supported with Baselines.

I’ll get this use case added to our internal feature request with the Product Development team. :slight_smile:


#5

Hi @Ryan,

Thank you for your support !
I will wait for your update on fixing this. It is much appreciated.

Regards,
Milen


#6

Hey @Milen.Palavrov,

As an alternative solution while faceting in baselines is being implemented, you can also use our API to disable the alert condition while weekend backups are taking place.

If your backup process is able to run a command before and after the backup process, issue a call to disable the alert and re enable it after the maintenance is done.
You can disable an alert condition by switching the enable attribute to false.

Have a look at this example and document:
https://docs.newrelic.com/docs/infrastructure/new-relic-infrastructure/infrastructure-alert-conditions/rest-api-calls-new-relic-infrastructure-alerts#

Hope this helps,


#7

Hello @ccastro,

We were thinking about disabling the Alert while the backup is taking place, but it is not possible to add it as Pre and Post commands in the backup software.
Another qusetion, do you think to implement Scheduled outage (maintenance window) to be set in a Policy or Condition for specific periods not to monitor ?

Thank you,
Milen


#8

Have you considered changing the approach? You could then have a script to:

Disable the alert policy/condition
Run back up software
Enable alert policy/condition


#9

Hi @stefan_garnham,

The thing is, that we are not not responsible for the Backups and don’t have access to the backup software. Also we don’t know the timeframe when it is running.
Thank you for the suggestions. We will have this in mind and if the NRQL alerts with Facet clause for Baselins is not fixed soon we will try to use one of this approaches.

Thank you again!