Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

No kubernetes info available for dashboards creations

insights
kubernetes

#1

Hi there. I’ve followed https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration#install and https://docs.newrelic.com/docs/infrastructure/new-relic-infrastructure/installation/install-infrastructure-linux to install Kubernetes monitoring integration and New Relic’s infra agent but running “SELECT * FROM K8sPodSample since 1 day ago” on Insights shows no info. I’ve tried to debug it as described in https://docs.newrelic.com/docs/integrations/host-integrations/troubleshooting/kubernetes-integration-troubleshooting-not-seeing-data whith the folloging results:

$ kubectl get pods --all-namespaces -o wide | grep newrelic
default newrelic-infra-13ctm 1/1 Running 2 1d 10.0.121.5 k8s-master-15972647-0
default newrelic-infra-m465k 1/1 Running 0 1d 10.0.121.7 k8s-master-15972647-2
default newrelic-infra-pcjc8 1/1 Running 1 1d 10.0.121.69 k8s-agentpool1-15972647-0
default newrelic-infra-v67m2 1/1 Running 0 1d 10.0.121.6 k8s-master-15972647-1
default newrelic-infra-w5nws 1/1 Running 1 1d 10.0.121.70 k8s-agentpool1-15972647-2
default newrelic-infra-xm3w3 1/1 Running 2 1d 10.0.121.68 k8s-agentpool1-15972647-1
$ kubectl get pods -n kube-system | grep kube-state-metrics
kube-state-metrics-2541278910-hr0t0 2/2 Running 0 1h
$ kubectl get daemonsets
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
newrelic-infra 6 6 6 6 6 1d

I’ve checked each box and the infra agent looks ok but I can’t confirm it because I couldn’t find any log for it, not even after looking for it with lsof.
I’m at lost here and maybe I’m missing something obvious, so please let me know if you guys have any ideas on what might be happening.

Best,
Juan Pablo Nogueira


#2

Hi @nogueiraj ,

Thanks for trying the Kubernetes integration! Could you send the logs of a pod?
$ kubectl logs newrelic-infra-13ctm

Thanks,
JF


#3

Hi @jjoly. Please check the attachment. Most errors look like:

time="2018-07-11T19:35:20Z" level=error msg="executing data source" data prefix=integration/com.newrelic.kubernetes error="exit status 1" plugin name=nri-kubernetes stderr="time=\"2018-07-11T19:35:20Z\" level=warning msg=\"Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json\"\ntime=\"2018-07-11T19:35:20Z\" level=panic msg=\"No data was populated\"\ntime=\"2018-07-11T19:35:20Z\" level=fatal msg=\"No data was populated\"\n"

but there are several others like:

time="2018-07-12T01:39:38Z" level=error msg="executing data source" data prefix=integration/com.newrelic.kubernetes error="exit status 1" plugin name=nri-kubernetes stderr="time=\"2018-07-12T01:39:08Z\" level=warning msg=\"Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json\"\ntime=\"2018-07-12T01:39:38Z\" level=panic msg=\"error trying to connect to: https://kubernetes.default/api/v1/nodes/k8s-master-15972647-0/proxy/healthz. Got error: Get https://kubernetes.default/api/v1/nodes/k8s-master-15972647-0/proxy/healthz: dial tcp 10.0.0.1:443: i/o timeout \"\ntime=\"2018-07-12T01:39:38Z\" level=fatal msg=\"error trying to connect to: https://kubernetes.default/api/v1/nodes/k8s-master-15972647-0/proxy/healthz. Got error: Get https://kubernetes.default/api/v1/nodes/k8s-master-15972647-0/proxy/healthz: dial tcp 10.0.0.1:443: i/o timeout \"\n" 
    time="2018-07-12T01:40:23Z" level=error msg="executing data source" data prefix=integration/com.newrelic.kubernetes error="exit status 1" plugin name=nri-kubernetes stderr="time=\"2018-07-12T01:39:53Z\" level=warning msg=\"Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json\"\ntime=\"2018-07-12T01:40:23Z\" level=panic msg=\"could not find node named \\\"k8s-master-15972647-0\\\". Get https://10.0.0.1:443/api/v1/nodes/k8s-master-15972647-0: dial tcp 10.0.0.1:443: i/o timeout\"\ntime=\"2018-07-12T01:40:23Z\" level=fatal msg=\"could not find node named \\\"k8s-master-15972647-0\\\". Get https://10.0.0.1:443/api/v1/nodes/k8s-master-15972647-0: dial tcp 10.0.0.1:443: i/o timeout\"\n"

I tried to attach a text file but looks like it isn’t possible - let me know if you need more info.

Best,
Juan Pablo Nogueira


#4

Hi @scott_nogueira ,

Could you try increasing the timeout for the connection?

You will need to edit the yaml file for the New Relic daemonset and redeploy it.
In the env: section:
ADD
- name: "TIMEOUT" value: "15000"
MODIFY (add TIMEOUT to the list)
- name: "NRIA_PASSTHROUGH_ENVIRONMENT" value: "KUBERNETES_SERVICE_HOST,KUBERNETES_SERVICE_PORT,CLUSTER_NAME,CADVISOR_PORT,NRK8S_NODE_NAME,KUBE_STATE_METRICS_URL,TIMEOUT"

Thanks


#5

I see the same errors. I added the env variables that you mentioned, but I still see the errors.


#6

Hey @aqeel.m.butt I just wanted to get some clarification here. Is this the same issue as ticket #311836? If so I’d like to continue working there until we can get a resolution.