Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Set separate job rate to be used when monitor has failed

feature-idea

#1

We’d like to be able to assign two rates on monitors:

  1. The rate that is there now would apply whenever the last job was successful.
  2. The new rate would apply whenever the last job failed.

So, for example, I have a monitor that runs every 12 hours. If it fails, I’d like the job to run every 30 minutes so that when it is fixed, I don’t have to wait up to 12 hours to see if issue is resolved and for the alert incident to be auto-closed. As long as it keeps failing, the job will be run at the 30-min rate.

There is a Re-check button available on each synthetic failure report that allows to recheck on-demand, but we’d like an automated way. Here are some reasons we’d like to see this:

  • Sometimes when we have a failure on one monitor, the failure will occur on a few and occasionally many monitors due to same underlying issue. We won’t want to have to drill down into each failure to click Re-check button.
  • Sometimes an issue will be resolved after a delay (e.g. due to server restart, waiting for caches to expire). Rather than having to remember or set a reminder to run the re-check some minutes in the future, we can just know that it will automatically be checked again fairly soon, and until it is resolved.

New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Hey @marnee.derider - Firstly thanks for posting so much detail in your feature idea :slight_smile: It’s true that currently there is no option for variable monitor frequencies. I will get that filed up to the Synthetics Team.


I will say that, as soon as a monitor fails, we automatically queue up 2 additional runs of that monitor to take place immediately. This is a 3 strike rule that ensures that monitor failures you see are less likely to be a transient issue. The monitor then resumes it’s regular frequency.


When thinking about this, a potential solution comes to mind, using custom automations.

This would require you having a webhook alert channel, and a service listening for that webhook (could be a lambda function, or your preferred service).

My thoughts are that the monitor would fail, and send out a webhook notification telling your custom service Monitor Failed - Incident Open - that service parses that data and sends out an API request to the Synthetics API, updating the run frequency of that monitor from 12hrs to the 30 mins you desire.

The service can also listen for Monitor Success - Incident Closed, which would trigger it sending another API call to put the monitor back to the 12hr frequency.

I understand this is a hacky workaround, not ideal, but hopefully can help you while we await the feature idea getting an update. :slight_smile:


#3

I am not sure how to set up to get an email notification of replies in this forum (if that’s even possible). So, I didn’t realize I had a reply on this until today. I apologize for the delay.

Thank you for the suggested workaround. At this time, it’s not critical enough for us to spend the time to get that workaround set up, but I’ll keep it in mind.

My team will be adding a lot of new synthetics in the next weeks or months, so my team might eventually find this to be more of an issue in future and we might try the workaround later.


#4

Hey @marnee.derider - you can get notifications from the topics you are Watching,

which is an option you can select at the bottom of the thread:

Note though - that you will only receive emails if you are not active on the Explorers Hub at the time the notification occurs.


#5

Also - thanks for confirming that currently the workaround isn’t necessary - let us know if it ever becomes necessary and you have any questions about it. :slight_smile: