Consolidate Kafka Metrics in NRQL Query

Build on New Relic Data Queries & Synthetic Scripts
#1

I’m trying to graph consumer lag by service (Which maps to consumer group) to track what my total consumer lag is over a particular topic for a given consumer group. I can get an individual Partition avg over time with this:

SELECT average(consumer-fetch-manager-metrics.audit-0.records-lag) FROM KafkaMetrics FACET host SINCE 7 DAYS AGO WHERE appName = ‘audit-service’ TIMESERIES

But the topic has 32 partitions, and I want to summarize them all without having to add every single partition to the query. Is there a way to combine elements with something like consumer-fetch-manager-metrics.audit-*.records-lag ??

#2

Hi @jayeno, as all the consumer-fetch-manager-metrics.audit*.records-lag are reported as attributes then there’s no way to wildcard multiple attributes to calculate them together. As you are sending these as a Custom Event you could add an attribute that adds all data values together before sending them to us which would allow you to get the average of all while still being able to analyze each partition individually :slight_smile:

#3

That probably won’t work. These are sent per service, running many instances across multiple hosts. There’s no way to combine them before they get shipped. I didn’t realize these are a Custom Event, as we’re just using the standard Kafka integration, along with the NR APM Product. Is there any other option/data we can pull to monitor consumer lag on a topic without having to manually map every partition for every topic in a graph??

#4

Hi, @jayeno: The Kafka integration is open-source; you could build a custom version that aggregates data from multiple queues and combines them into a single attribute.

Other than that, I don’t have any suggestions.

#5

A post was merged into an existing topic: Writing Custom Kafka Metric in New Relic