Kafka Consumer and Producer Monitoring in K8S

We’d like to monitor our Kafka consumers and producers in our Kubernetes cluster.
We installed the NewRelic bundle helm chart, current version, and configured the new relic infrastructure chart (that comes with it)

In regards to broker monitoring everything works as expected and described in the docs.

Here’s our Problem:
The producers and consumers’ data is not showing up in the Kafka dashboard.
The KafkaConsumerSample and KafkaProducerSample are absent from the metrics explorer.

The config from the newrelic-bundle-newrelic-infrastructure pods:

        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
          label.kafka-role: /producer/
      - name: nri-kafka
          CONSUMER_OFFSET: 1
          CLUSTER_NAME: ${discovery.clusterName}
          PRODUCERS: '[{"name": "${discovery.label.kafka-groupId}", "host": "${discovery.ip}", "port": ${discovery.ports.jmx} }]'
          ZOOKEEPER_HOSTS: '[{"host": "kafka-v230-zookeeper-headless.kafka.svc.cluster.local", "port": 2181}]'
          CONSUMER_GROUP_REGEX: '.*'

As advised we ran echo "*:*" | nrjmx -hostname <ip> -port <port > inside the infrastructure pod.
The command returned JMX data.

We ran nri-discovery and confirmed that the right label, kafka-role is set and that the relevant ports are exposed:

"ip": "<our-ip>",
"label.kafka-groupId": "<the-right-group>",
"label.kafka-role": "consumer-producer",  
"ports": {
   "0": 8080,
   "1": 9090,
   "http": 8080,
   "jmx": 9090

The logs of the newrelic-bundle-newrelic-infrastructure are clean.

Any help would be much appreciated :slight_smile:


@michael.ruhwedel The below configuration as outlined in our kafka documentation if set to the true value will stop all other collection and only collect the kafka offset data. And from the looks of it from the code here we do in fact convert whatever value goes into the ConsumerOffset variable to a bool value. So your configuration would have the CONSUMER_OFFSET option turned on.

I would advise removing that from your configuration and reloading it into the environment. Then you should start seeing those other sample start coming in. Let us know how that goes for you guys.

This topic was automatically closed after 365 days. New replies are no longer allowed.