I have a large suite of synthetic monitors, and I’m trying to create two levels of alert from them - a failure in a single monitor would be a warning but a failure in multiple monitors at the same time would be a critical.
Obviously the warning is simple, just put the synthetic checks into an alert and fire the alert on a per-condition basis.
The critical is more difficult. I’ve tried the following NRQL query:
SELECT count(*) from SyntheticCheck WHERE result != 'SUCCESS'
The problem is, this only fires when two or more monitors fail at exactly the same time. This is because my monitors run on schedules and don’t necessarily run at exactly the same time, which means that I can get multiple single-monitor failures but no combination critical alert unless the monitors fail at exactly the same time.
I’ think that the above is looking at the current value, which would be null if the monitor is not running. I want my alert to look at the latest result rather than the current one, but I have been having some trouble getting the latest() function to do what I want.
How does new relic implement the red/green statuses in the UI for synthetic monitors? Could I use this technique to create my alert?
If not, how would I achieve what I want?