Infrastructure On-host Integration Kafka Install

Ensure you have Installed the Infrastructure Agent, see this guide for more information.

  1. If Kafka is not running on Kubernetes or Amazon ECS, you must ensure that you install the infrastructure agent on a host that’s running Kafka. Otherwise:
  1. Java 8 or higher
  2. Total number of monitored topics must be fewer than 300
  3. It’s important that you JMX enabled on all brokers, Java consumers, and Java producers that you want monitored.
  4. Check if the server running the integration has access to the zookeeper.
  5. Ensure that the JMX and Zookeeper ports in the Kafka environment are network accessible to the hosts where the integration is installed.
  6. Ensure that Java_Home is accessible by the Infrastructure Agent. If it is not then you may see errors like so: JMX tool exited with error: exit status 127
  7. If you are not seeing any data coming through for the Kafka integration then ensure that there are active consumers. If there are no active consumers then you may need to manually generate some data for the Kafka instances so that it can be collected and reported by the integration.
  8. If you are seeing parsing errors in the logs then make sure all the attributes in the config file are populated and appropriate fields are commented out correctly as per the following github sample.
  9. If you are failing to collect consumer offset data and are seeing the following errors: Failed collecting consumer offset data: zk: node does not exist, such errors are seen when zookeeper configuration is not kept in the root zookeeper node. It can be configured with the zookeeper_path parameter. zookeeper_path is the path, in zookeeper, to the root node of the Kafka configuration. You could either leave the zookeeper_path blank (which defaults to /), or specifically set it to /.
  10. If you are seeing No JMX Connection errors then we recommend that you check your JMX settings by running the following pgrep java | xargs -n 1 -I {} /proc/{}/{environ,cmdline}

A post was split to a new topic: Monitoring Kafka & Zookeeper