Kafka-consumer-offsets Implementation

Support: Full-Stack Observability (FSO) Infrastructure
,
#1

Hi,

I’m trying to figure out how to implement kafka-consumer-offsets. The documentation mentioned it should only be running on one host at a time, but figuring out how to run it in an automated fashion on one host on a give cluster with repeatable process is challenging. I could put a check in when provisioning that say, only provisions it on the first broker instance. But that means if that broker goes down or away I lose all my metrics. We run consul in our environment, so I could use something like consul lock to only have a single running instance, however, I don’t see an easy way to run instances in separate processes.

I’m curious if anyone else has tackled this problem, and what they came up with for solutions?

-= Jay =-

#2

Hi Jay,

Thanks for writing in. We do recommend consumer offsets on only 1 node at a time due to the load that multiple nodes can bring. That being said, I do appreciate you outlining your use case here, so I’ll go ahead and create a feature request.