Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Errors in newrelic-infra Kubernetes pods - no data in New Relic


#1

I am unable to get metrics into New Relic. I am running Kubernetes 1.9.3, using Kops in AWS. I have 3 masters and 2 nodes, with a single non-Kubernetes namespace that I created. I deployed kube-state-metrics to the kube-system namespace and I don’t see any errors in the pod. I then deployed the newrelic-infra daemonset, which deployed to the default namespace. For each pod created, I see the following errors, with the “executing data source” error repeating.

time=“2018-04-18T19:15:03Z” level=info msg=“Using Display Name: ip-172-31-9-55.us-west-2.compute.internal”
time=“2018-04-18T19:15:03Z” level=info msg=“New Relic Infrastructure Agent version 1.0.888 Creating Service (356.109µs)”
time=“2018-04-18T19:15:03Z” level=info msg=“Agent service manager started successfully. (386.062µs)” service=newrelic-infra
time=“2018-04-18T19:15:03Z” level=info msg=“New Relic Infrastructure Agent version 1.0.888 Initializing (541.208µs)”
time=“2018-04-18T19:15:04Z” level=info msg=“New Relic Infrastructure Agent version 1.0.888 Running (372.275434ms)”
time=“2018-04-18T19:15:04Z” level=error msg=“Getting the Sampling Path for plugin” data prefix=integration/com.newrelic.kubernetes entity=i-01b04d56b08e7b262 error=“Plugin not registered: integration/com.newrelic.kubernetes” plugin name=nri-kubernetes pluginID=integration/com.newrelic.kubernetes
time=“2018-04-18T19:15:04Z” level=error msg=“executing data source” data prefix=integration/com.newrelic.kubernetes error=“exit status 1” plugin name=nri-kubernetes stderr=“time=“2018-04-18T19:15:04Z” level=warning msg=“Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json”\ntime=“2018-04-18T19:15:04Z” level=warning msg=“Recoverable error group: containers data not found, possible data error in /stats/summary response”\ntime=“2018-04-18T19:15:04Z” level=warning msg=“No data was populated”\ntime=“2018-04-18T19:15:04Z” level=panic msg=“no data was populated”\ntime=“2018-04-18T19:15:04Z” level=fatal msg=“no data was populated”\n”


#2

Hi @toneil,

We would need to gather more information from your environment. Could you let us know the following details:

  • What version of Kubernetes are you running?
  • Is it a packaged solution (GKE, EKS, Azure Containers, Openshift, Tectonic, etc.)?
  • If not a cloud service, where are you running K8s (AWS EC2, Azure VMS, on-prem, etc.)?
  • How many nodes, namespaces, pods and containers is the cluster you want to monitor running?
  • What version of the New Relic Infrastructure Kubernetes integration (newrelic-infra container) are you running?
  • What version of kube-state-metrics are you running?
  • Do you have one or multiple kube-state-metric services deployed?
  • Have you made any changes to the deployment file provided by new relic? Which ones?
  • Have you set any authorization mode like RBAC?

Thanks,


#3

Hi @toneil , we have found the reason for this error and the team is working on a fix.


#4

I’m also interested in this as we are looking to use openshift 3.9/kube 1.9.

The newrelic site states to use kube-state-metrics 1.1 here:
https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration

But for kube 1.9, it mentions 1.1 isnt fully compatible?

So curious if newer kube-state-metrics can be used for kube 1.9/new relic agent?

thanks.


#5

@toneil We have just released version beta2.2 of the Kubernetes that should fix the issue for “containers data not found, possible data error in /stats/summary response”.


#6

@mshack you can also use kube-state-metrics 1.2. It’s now supported and the documentation has been updated.


#7

Im using kube-state-metrics 1.2 and newrelic-infra k8s v 2.3. I’m still getting:

me=“2018-06-19T14:19:19Z” level=error msg=“executing data source” data prefix=integration/com.newrelic.kubernetes error=“exit status 1” plugin name=nri-kubernetes stderr=“time=“2018-06-19T14:19:19Z” level=warning msg=“Environment variable NRIA_CACHE_PATH is not set, using default /tmp/nr-kubernetes.json”\ntime=“2018-06-19T14:19:19Z” level=panic msg=“No data was populated”\ntime=“2018-06-19T14:19:19Z” level=fatal msg=“No data was populated”\n”

kube-state-metrics v1.1 also is throwing same errors


#8

@Olsi.Birbo Are you using Openshift ?


#9

@mshack We have added the instructions to setup the Kubernetes integration on Openshift 3.9 : https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration#install-openshift-container-platform