Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Synthetic minion shows failures for working web endpoints

privateminion

#1

Some tests are showing it is failing from a private location in our data center. Following is one of them.

https://synthetics.newrelic.com/accounts/1941126/monitors/5632130a-ef08-4647-97bb-afbbfab3b3e5

But when I tried doing curl to one of the endpoints that we monitor from the minion server its giving 200 response. I restarted the minion and then resolved the issue but after sometime issue pops up again for the exact same tests that has been failing (for minion) before. Its seems like a minion issue, But no idea what’s going on.

Does anyone has experienced such thing before ?
Need some help to fix this,

Thanks


#2

Hey @dinidu_sandaruwan

I just looked at the monitors in your account and I see that more than just the monitor you linked is failing.

It appears that every failing monitor from that private location is a heavyweight (Simple browser/Scripted browser/API Test).

While lightweight Ping monitors are working fine from your private location.


Of the failing monitors, I looked at 2, the one you linked to, and this one too.
The first monitor is failing with the error message: Internal Server Error
and the second is failing with: Job timed out after 240 seconds

I have a feeling that both of these are essentially the same, but the API test monitor is interpreting the failure as job timeout, and the simple browser is interpreting it as an Internal Server Error.

You may know that the maximum permitted job duration in Synthetics is 180 seconds. Typically when a job reaches that stage the monitor will throw a failure message Job times out after 180 seconds.

In your case your monitors are getting to 4 minutes (240 seconds) which is actually an enforced Docker container timeout.
This is enforced when something may be causing the Synthetics job to hang on longer than 180s, we forcefully shut down the docker container running that job after 240s to not leave that hung monitor there utilising resources.

When it comes to troubleshooting this, I think a good first step is to reboot the private minion. Hopefully doing that, which will restart the minion service on that host, will flush out any potential problems.

If that fails, you could try updating your minion to the latest version (See that here in the release notes)

If neither of those options work, it would be great to get some logs from the minion. Let me know if that is the case and I can get a ticket opened for you to share logs in.

-Ryan.


#3

Hi @RyanVeitch,

Thank you for your quick response,
I’m going to re deploy the minion with the latest as reboot didn’t work.

I’ll get back to you once it is done with the results


#4

Thanks @dinidu_sandaruwan let us know how you get on :smiley:


#5

@rdouglas and @RyanVeitch

Sorry for the late response. My Issue got fixed by redeploying the latest minion version. Thank you very much for your support. much appreciated !!!


#6

Fantastic! Thanks for letting us know @dinidu_sandaruwan :slight_smile: