For those following along at home, @Scott.Hancock emailed me a link to a condition with the following NRQL query (anonymized):
SELECT average(duration) FROM SyntheticCheck WHERE monitorName = 'MONITOR_NAME
The issue here is that Synthetics checks don’t run every minute, however; alerts looks at NRQL data on a minutely basis. So Alerts is seeing something like 5000, 0, 0, 0, 0, 6500, 0, 0, 0, 0 (totally made up numbers). Your condition is setup to look for a series of checks with a duration above the threshold you’ve set, but the data spikes up when there is a check and then comes through our system as a 0 for the minutes where there is no check.
Your best course of action is to use a sum of query results, or to be
at least once in but these both have tradeoffs. The former may result in some notification you would want to see being missed, and the latter may result in a more noisy condition than you’re hoping for.
If you have any other specifics about the use case you’re solving for, (eg “I want to know when N consecutive checks exceed the threshold”) then please let us know! If it’s the case that our current implementation doesn’t quite meet your needs we can submit a feature idea so that @nheinrich (the Alerts product manager) can make sure he hears your vote.