Private Minion "Timed out waiting for validation results"

Support: Full-Stack Observability (FSO) Synthetic
,
#1

Is anyone familiar with this situation? Our private minion can no longer validate. Clicking the Validate button puts a job in the minion’s queue and I can see it complete successfully on the Synthetics minion dashboard, but in the script editor window it’s just stuck on: Running job in [Private Minion Name]... then after a while it times out and shows Timed out waiting for validation results.

Fortunately the regular timed runs still complete and the results are logged as normal, it only seems to be the validation function that’s not working. Any ideas or pointers would be greatly appreciated. Thanks!

#2

Hey @BigNStrauss - Can you drop a screenshot of the script log failure messages here when that validation fails? In addition, a link to the monitor would be helpful too, I’d like to look into what we may be able to see on that private minion. Thanks in advance!

#3

Hey @RyanVeitch here’s a screenshot of the failure message, as well as a screenshot of the private minion’s jobs to show that none are actually failing. I’ll private message you the link to the monitor. Thanks!

image

image

#4

Hey @BigNStrauss - I had a look into this a little bit, and I can’t see any reference of this error message coming up in the past, though we do currently have an active support ticket with another user who is facing the same issues.

It looks like we may need to get some logs & additional minion info. As such it may be best to get you into a ticket as well, in order to share some of that more private information.

Watch out in your inbox for that ticket email :slight_smile:

One thing to note is that you are running a somewhat old private minion (with a build timestamp of: April 2017).
The support engineers in the ticket will likely ask you to generate some minion logs, as well as ask you to upgrade to a more recent minion, in order to rule that out as a potential problem.

#5

@RyanVeitch Sounds good, I’ll open a ticket. Yes I had noticed that our minion is out of date. The engineer here who set it all up didn’t think the version would be causing this issue. But I’m fine with it being updated if needed. Thanks for the assistance so far.

#6

No worries - it may not be the minion version - I’m not sure right now, the devs working with our support engineers in the ticket will have a better idea :slight_smile:

Please do follow up here once you get to a resolution, I’m sure others would be interested to know, should they face similar issues.

#7

Yes, I’ll definitely report back with any findings.

1 Like
#8

Thanks! :smiley: @BigNStrauss

#9

Hi @BigNStrauss . did you ever get this resolved? Having the same issue running a NR scripted browser for internal service using a private minion.

#10

Hey @dcoleman1,

Thanks for posting here! The Timed out waiting for validation results is a common issue which can stem from several causes:

  • A lot of jobs queued waiting to run.
  • An undersized CPM to meet the job demand per minute.
  • Internal Engine Errors causing the CPM to run poorly.
  • Permissions issues due to SELinux or the /tmp mount that prevent non-ping jobs from running properly.
  • Installing the Docker CPM in a container orchestrator like ECS where “runner” containers become orphaned and unresponsive.
  • Insufficient IOPS for non-ping jobs to be able to save job results in a timely manner. This is especially true for AWS EC2 instances where the volume is < 33 GiB in size. IOPS will be 100 and the CPM could use up the available burst balance depending on jobs per minute.

Take a look at this rightsizing article to see if anything stands out to you.

If you can share a permalink to your private location, I can look up more specifics for you. Only New Relic Support and authorized users will be able to click through, so it’s okay to share permalinks here.

2 Likes