Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Monitor check status before running or alerting

alerts
feature-request
false-positive
synthetics
feature-idea

#1

When it’s time for a Synthetics monitor to run, it gets sent to the minion location, and added to a queue. If that queue is long, upwards of 60+ seconds that I’ve experienced, it could end up running against a site even if the monitor status has since been disabled. This feature idea would be one of the following options:

  1. Have the monitor job, either check back to the monitor status and see if it’s still okay to run, before it does it’s check.

  2. If that is too costly, then let the monitor job run like it does today, but if it hits a failure, check the monitor status to see if it’s active, before sounding the alarm/alert.

This would help solve false alerts about a site being down. As it is today, we call the API to disable a site monitor, wait 1 minute, and then turn off the site, but occasionally that monitor job is still in the minion location queue, and eventually turns on the site that we’ve turned off, and received ACK from NR API that it is disabled, but end up getting a monitor alert send to PagerDuty and waking up innocent team mates. :wink:

Other links to this issue:
My initial forum post: Disabled Synthetic site monitor still throws an alert
My support case Request: #312228


New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Worth noting, we are using the NewRelic public minions. I’m not sure if using private minions would help, or if we’d have ability to size/provision the private minions so they rarely queue.


#3

Hey @Mike.G - That’s a great feature idea. As you have assumed no feature currently exists for Synthetics minions to check the monitor status before running jobs. That is the same on Private Locations.

You are right in that you have full control over the provisioning and size of the private minions, so you can ensure you have enough compute power in place to handle the number of monitors you are running there.

All that said, public minions do scale up and down depending on the load they are working with. So in theory the public minion queue shouldn’t get so high that there are long delays for your monitors.

If you’d like, we can open up a support ticket for you to work with our Synthetics experts, who can help dig into the timings you are seeing, and that 60 second delay you mention.

For now however I have added a poll to this post for others to vote, and I have also filed a feature request internally on your behalf. :smiley:


#4

@RyanVeitch Thanks for your reply. I had a support ticket 312228 which is now closed. You can open a new ticket, I can reply with a couple alerts that happened after the monitor was disabled.


#5

@Mike.G You got it! You should have an email regarding that ticket now - one of our support team members will be in touch :smiley: