Synthetic private minion cannot be launched in EKS

We will keep the community posted if there is any update on this but in the meantime, please let us know how that works out for you and your team!

Hi,
Any update? I am facing similar issue.

Thanks.

Hi @cchokshi,

Other than the two options mentioned earlier (disable the BoundServiceAccountTokenVolume feature gate, or downgrade the K8s cluster to 1.20), our product manager does not anticipate having a fix for this issue until sometime next year. Though it’s possible it could arrive sooner if more and more people are affected by it, which I anticipate will happen as people upgrade to K8s 1.21+.

Since this issue is a blocker for some, especially those using EKS or OpenShift where the feature gate cannot be disabled, I wanted to be as clear as I could about the timeframe. Though it’s as clear as mud, sometime next year is our current best estimate based on the workload already assigned to the engineering team.

Please contact your account rep, if you have one, to try and escalate this issue. I will continue to do the same from within New Relic Support to help prioritize a fix.

Hi @kmullaney . This is a total blocker for us. We’re using AKS where you cannot switch off the BoundServiceAccountTokenVolume feature gate. AKS ver 1.20 will be end of life Feb 2022 (https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar) , nevertheless we already switched to 1.21 due to volume snapshot feature, so we’re totally screwed. Any chance to prioritize this issue ?

Hi @kmullaney, This has been a blocker for us where we have upgraded all our EKS clusters to 1.21. I have already escalated this to our account rep but there is no assurance of fix in the near future.
We have moved to having an EC2 instance with docker CPM but we lose scalability and the features that StatefulSet and pod minion offers. We expect a fix ASAP. I have reported this a long time ago. It should be in your roadmap at the top.

Thanks!

Hey @cin and @szd2013,

That’s totally reasonable. The escalations you’ve made via your account reps are also a good step to help prioritize this.

I couldn’t agree more that this kind of environmental blocker should be addressed quickly. I’ve been advocating for it and bringing it up in every meeting I have with the Synthetics engineering team.

If I hear anything new, I’ll keep ya’ll posted.

I appreciate that this is on your product manager, but this is a totally unacceptable timeline. 1.21 has been GA for 6 months and is only a few months out from being in maintenance mode.

Hi @christopher.duffin,

Thanks for contributing here! I’ve made sure everyone involved is aware of the Kubernetes EOL timeline for 1.20+.

1 Like

We are experiencing this issue also. We are trying to upgrade to 3.0.57 to mitigate CVE-2021-44228. Has anyone found a workaround?

Hi @pgrant1 ,

Hope the week has been going well so far! Thanks for posting!
Currently as mentioned earlier in this post that there is active advocation for this and those in the Synthetics engineering team are aware of the end of life for Kubernetes 1.20+

v3.0.57 does remediate CVE-2021-45046 - there are some great steps on updating the CPM here

Understood, I have read the steps on updating the CPM but we are using EKS 1.21, so this documentation is useless to us or am I missing something?

Hey @pgrant1 - hope the new year has been going well so far!

As @kmullaney mentioned we are actively advocating every-time we can as posted in earlier posts to unblock for EKS 1.21 - the Synthetics Engineering team is also still aware of Kuberenetes EOL timeline for 1.20+.

1 Like

That’s great news.

…Do you have an ETA on this perhaps?

Thanks,
Paul

An ETA would be good to have since we’re seeing the same issue as well.

Also blocked by this, any update appreciated.

+1 for not able to deploy private minion on EKS

Any update per chance?

Thanks,
Paul

We are actively working on this and should have a version of the minion that supports K8s 1.21+ by the end of this month.

1 Like

Great news @bpeck, Thank you for the update :+1:

Sounds like a fix is in the works for this, which is great as its blocking me as well.
Despite the title of this thread stating the problem is specific to EKS, I’m unable to run this in minikube at version 1.22.3…which I believe is also expected since you can’t shut off BoundServiceAccountTokenVolume. Looks like Minikube doesn’t start if you do that.

Whenever I can get this working in minikube I will get blocked by this because my production environments are both EKS and OKD which are both above 1.21. It’d be great if the pending fix worked out of the box on minikube at a version 1.21+ for those trying to evaluate this feature. Note: The current templates on master reference rbac.authorization.k8s.io/v1beta1 for the role and rolebinding api versions. Those need to go to v1 as part of this fix I think. I had to download and modify your templates before running just to get past errors related to that.

I may back my minikube down to version 1.15 just to see if that works temporarily but I couldn’t actually do anything with this in a real environment until the fix is available.