Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Kubernetes objects not recognized

insights
kubernetes
k8s

#1

Hello,

I have a k8s cluster that’s running NR infrastructure agent following the documentation:

kubectl get pod -n utility
NAME READY STATUS RESTARTS AGE
kube-state-metrics-58f4cff6ff-nhv7d 1/1 Running 0 3d1h
newrelic-infrastructure-6gqnf 1/1 Running 0 5d2h
newrelic-infrastructure-cnnxn 1/1 Running 0 5d2h
newrelic-infrastructure-cx5rb 1/1 Running 0 5d2h
newrelic-infrastructure-p29h2 1/1 Running 0 5d2h
newrelic-infrastructure-v5zpx 1/1 Running 0 5d2h

Logs in every NR pod is similar to:

time=“2019-11-25T20:42:53Z” level=info msg=“Using Display Name: ip-10-20-172-247.us-east-2.compute.internal”
time=“2019-11-25T20:42:53Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Creating Service (920.308µs)”
time=“2019-11-25T20:42:53Z” level=info msg=“Agent service manager started successfully. (1.08493ms)” service=newrelic-infra
time=“2019-11-25T20:42:53Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Initializing (1.231294ms)”
time=“2019-11-25T20:42:53Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Running (274.683949ms)”
time=“2019-11-25T21:15:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container d093d424b74685d19ed29d2a9033b0d74c1b269c0b2b7302e001e6d5df164100 is not running”
time=“2019-11-25T22:01:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container cb91e84482e30431a1555e1101e41aa4469887f73f9d142ecf382f8aa066b71a is not running”
time=“2019-11-26T00:30:21Z” level=error msg=“metric sender can’t process 1 times” error=“InventoryIngest: events were not accepted: 400 400 Bad Request identity error error requesting internal key: Post http://infrastructure-identity-service.vip.cf.nr-ops.net/accounts/2236526: net/http: request canceled (Client.Timeout exceeded while awaiting headers)”
time=“2019-11-26T06:40:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 31c851b2d7f2d993efdd4188c846042174e2d3993c70f693e49d51b6f5678835 is not running”
time=“2019-11-26T08:30:04Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: EOF”
time=“2019-11-26T09:10:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 12b29e322aa18a8137cb64e5684ca04f565d044ae094f49273d31ee14caf0d97 is not running”
time=“2019-11-26T11:21:18Z” level=error msg=“metric sender can’t process 1 times” error=“InventoryIngest: events were not accepted: 400 400 Bad Request identity error error requesting internal key: Post http://infrastructure-identity-service.vip.cf.nr-ops.net/accounts/2236526: net/http: request canceled (Client.Timeout exceeded while awaiting headers)”
time=“2019-11-26T11:32:34Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 10.20.172.247:33682->162.247.242.5:443: read: connection reset by peer”
time=“2019-11-27T00:50:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 303bdf3ece3e4c81963c59509f5d56c23424afd3915afa34776c75e1b3c54f84 is not running”
time=“2019-11-27T04:44:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container cbd56aa5b809748093c491f995b7c0c42c1511488188fd92e7b14c42fb0b69c1 is not running”
time=“2019-11-27T08:48:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container c25b6ba48cdf2e2f94d20c496c97b48ae2665173fe4341744243a9e7c4070a45 is not running”
time=“2019-11-27T12:09:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 2c5acea44079f471a41a2e00742f2050b35563dc4b617253b372d9d6cdae042e is not running”
time=“2019-11-27T14:41:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container cdbeddc4cb16865f8afc3ff1fb9564f72169492d983578ce3369afd5cebc4817 is not running”
time=“2019-11-27T15:00:13Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 389e635f8abad6a1b52449962d6ac6f2c036a8fbe1c837354e726383ff0b16d4 is not running”
time=“2019-11-27T15:10:33Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container afe3ef6bee78dc65caee01c6479d8ce014c6a124480a053f9a6ca3ff8918e90a is not running”

time=“2019-11-25T20:55:14Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Creating Service (1.345799ms)”
time=“2019-11-25T20:55:14Z” level=info msg=“Agent service manager started successfully. (1.541941ms)” service=newrelic-infra
time=“2019-11-25T20:55:14Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Initializing (1.92056ms)”
time=“2019-11-25T20:55:14Z” level=info msg=“New Relic Infrastructure Agent version 1.3.18 Running (236.835134ms)”
time=“2019-11-25T23:09:35Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 10.20.172.65:36394->162.247.242.5:443: read: connection reset by peer”
time=“2019-11-26T17:01:25Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 10.20.172.65:51172->162.247.242.5:443: read: connection reset by peer”
time=“2019-11-27T15:20:14Z” level=error msg=“instantiating docker sampler process decorator” error=“Error response from daemon: Container 9f7d8a432a98a29b0c1a487097fc3eb722c6730b12670af3341077427c7117d6 is not running”
time=“2019-11-28T08:15:00Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 10.20.172.65:33636->162.247.242.5:443: read: connection reset by peer”
time=“2019-11-30T04:56:09Z” level=error msg=“metric sender can’t process 1 times” error=“Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 10.20.172.65:44624->162.247.242.5:443: read: connection reset by peer”

When I query SELECT * FROM K8sPodSample since 1 day ago I don’t even see the K8sPodSample.

Kube-state-metrics version is 1.7.2 - installed with helm.
NR agent image version: newrelic/infrastructure-k8s:1.9.4 - installed with helm.
Environment vars:
Environment:
NRIA_LICENSE_KEY: <set to the key ‘license’ in secret ‘newrelic-infrastructure-config’> Optional: false
CLUSTER_NAME:
NRIA_DISPLAY_NAME: (v1:spec.nodeName)
NRK8S_NODE_NAME: (v1:spec.nodeName)
NRIA_CUSTOM_ATTRIBUTES: {“clusterName”:"$(CLUSTER_NAME)"}
NRIA_PASSTHROUGH_ENVIRONMENT: KUBERNETES_SERVICE_HOST,KUBERNETES_SERVICE_PORT,CLUSTER_NAME,CADVISOR_PORT,NRK8S_NODE_NAME,KUBE_STATE_METRICS_URL,TIMEOUT

I don’t have any information about namespaces, pods, services but hosts with correct hostnames are recognized correctly. Does anyone experienced this?


#2

Hi @hero_dusan - The Infrastructure agent which is deployed as a daemonset as part of the Kubernetes On-Host integration requires the Docker daemon socket to be bind-mounted inside the newrelic-infra pod. For more information on these requirements please refer to our docs here.

If you run the same HELM command you used to deploy the integration except with the --dry-run --debug flag you should get an output which should show the docker.sock mount point under volumes:.

This should look similar to the below:

- name: host-docker-socket
          hostPath:
            path: /var/run/docker.sock

For more information on running this debug flag please refer to step 7 our docs here.

In addition, you should see the same path for the mount point when you run a describe on the newrelic-infra pod.

To see if the integration has been deployed properly with HELM you can run:

helm status [RELEASE_NAME]

HTH.


#3

Hi, I checked our config and it matches the config you described.
Here is an output:

kubectl describe ds newrelic-infrastructure  -n utility
Name:           newrelic-infrastructure
Selector:       app=newrelic-infrastructure,release=newrelic-infrastructure
Node-Selector:  <none>
Labels:         app=newrelic-infrastructure
                chart=newrelic-infrastructure-0.13.8
                heritage=Tiller
                mode=privileged
                release=newrelic-infrastructure
Annotations:    <none>
Desired Number of Nodes Scheduled: 4
Current Number of Nodes Scheduled: 4
Number of Nodes Scheduled with Up-to-date Pods: 4
Number of Nodes Scheduled with Available Pods: 4
Number of Nodes Misscheduled: 0
Pods Status:  4 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=newrelic-infrastructure
                    mode=privileged
                    release=newrelic-infrastructure
  Service Account:  newrelic-infrastructure
  Containers:
   newrelic-infrastructure:
    Image:      newrelic/infrastructure-k8s:1.9.4
    Port:       <none>
    Host Port:  <none>
    Limits:
      memory:  150M
    Requests:
      cpu:     100m
      memory:  30M
    Environment:
      NRIA_LICENSE_KEY:              <set to the key 'license' in secret 'newrelic-infrastructure-config'>  Optional: false
      CLUSTER_NAME:                  <redacted>
      NRIA_DISPLAY_NAME:              (v1:spec.nodeName)
      NRK8S_NODE_NAME:                (v1:spec.nodeName)
      NRIA_CUSTOM_ATTRIBUTES:        {"clusterName":"$(CLUSTER_NAME)"}
      NRIA_PASSTHROUGH_ENVIRONMENT:  KUBERNETES_SERVICE_HOST,KUBERNETES_SERVICE_PORT,CLUSTER_NAME,CADVISOR_PORT,NRK8S_NODE_NAME,KUBE_STATE_METRICS_URL,TIMEOUT
    Mounts:
      /host from host-volume (ro)
      /var/log from log (rw)
      /var/run/docker.sock from host-docker-socket (rw)
  Volumes:
   host-docker-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:
   log:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
   host-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
Events:            <none>

Still, we don’t have any but any kubernetes metric showed.


#4

Hello support,

I haven’t heard anything from you in last couple days. Do you have any suggestion what could be a reason for an issue we have?

Regards,
Dusan


#5

Hello @hero_dusan,

You might consider updating to a newer version of the Kubernetes integration than the one you are using.

https://docs.newrelic.com/docs/release-notes/platform-release-notes/host-integrations-release-notes/new-relic-integration-kubernetes-1102

This version includes a newer version of the infra agent, which may help here.

Regards