For some reason newrelic infrastructure monitoring for Kubernetes (RKE) not working as expected. After waiting more than hour to see that cluster appeared on integration page I’ve connected to kubernetes cluster and saw that newrelic kubelet pods restarting infinitely, if we are look into pod logs we can see this:
time=“2022-05-20T12:21:28Z” level=info msg=“Waiting for agent container to be ready…”
time=“2022-05-20T12:21:28Z” level=info msg=“New Relic Kubernetes integration Version: v3.2.0, Platform: linux/amd64, GoVersion: go1.17.9, GitCommit: 07996399732a02ea798be9df034446fbda254010, BuildDate: Mon May 16 11:56:34 UTC 2022\n”
time=“2022-05-20T12:21:28Z” level=info msg="Trying to connect to kubelet locally with scheme=“https” hostURL=“10.1.2.206:10250"”
time=“2022-05-20T12:21:28Z” level=info msg="Connected to Kubelet through nodeIP with scheme=“https” hostURL=“10.1.2.206:10250"”
time=“2022-05-20T12:22:01Z” level=error msg=“retrieving scraper data: retrieving kubelet data: kubelet data was not populated after trying all endpoints”
10.1.2.206:10250 is accessible from this container, I tried to
curl it from inside of kubelet container and it was able to connect to this endpoint.