Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Errors in newrelic/infrastructure-k8s pod - error trying to connect to: https://kubernetes.default/api/v1/nodes/$HOSTNAME/proxy/healthz


#1

Currently unable to get metrics into new relic using the infra agent as the pod is failing with the below issue

I am running Kubernetes: v1.9.6, newrelic-infra container: newrelic/infrastructure-k8s:1.0.0-beta2.3

example error:

time="2018-06-11T20:58:53Z" level=error msg="executing data source" data prefix=integration/com.newrelic.kubernetes error="exit status 1" plugin name=nri-kubernetes stderr="time=\"2018-06-11T20:58:53Z\" level=warning msg=\"Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json\"\ntime=\"2018-06-11T20:58:53Z\" level=panic msg=\"error trying to connect to: https://kubernetes.default/api/v1/nodes/ip-10-17-41-211.ec2.internal/proxy/healthz. Got error: Get https://kubernetes.default/api/v1/nodes/ip-10-17-41-211.ec2.internal/proxy/healthz: dial tcp: lookup kubernetes.default on 127.0.0.1:53: no such host \"\ntime=\"2018-06-11T20:58:53Z\" level=fatal msg=\"error trying to connect to: https://kubernetes.default/api/v1/nodes/ip-10-17-41-211.ec2.internal/proxy/healthz. Got error: Get https://kubernetes.default/api/v1/nodes/ip-10-17-41-211.ec2.internal/proxy/healthz: dial tcp: lookup kubernetes.default on 127.0.0.1:53: no such host \"\n" 

#2

@matt.burdan Thanks for opening the new thread on this issue. We are looking into this issue.


#3

any update on this? we are really wanting to get started testing this.


#4

@matt.burdan , could you retrieve the DNS configuration of the newrelic pods by executing the following steps?
1/ Retrieve the name of the new relic pod
kubectl get pods | grep newrelic
This will return the name of the newrelic pod(s).
Choose any of the pod name and copy for next step
2/ Show the DNS configuration
exec <name of newrelic pod> cat /etc/resolv.conf


#5

@jjoly here is the results:

[] burdz@~: kubectl exec newrelic-infra-poood cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 10.17.0.2
options ndots:2 timeout:1 attempts:2

#6

@matt.burdan Thanks for sending this.
It looks like it’s missing the search domains.
The results should look like the below, notice the line starting with search

[root@newrelic-infra-g42fz /]# cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5

I would advise to check issues with DNS. This page can help : https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

In particular the result of the below command is interesting :
kubectl exec -ti busybox -- nslookup kubernetes.default

Thanks


#7

I was able to get past this issue by setting dnsPolicy: ClusterFirstWithHostNet


#8

@matt.burdan Great news! Thanks for following-up and providing the details.