Feature Idea: Support For Consumer Group Level Metrics

AWS MSK has introduced new metrics which are available in cloudwatch for the Managed Kafka Service which are detailed here: Amazon MSK Metrics for Monitoring with CloudWatch - Amazon Managed Streaming for Apache Kafka

Specifically there are new levels of metrics available for kafka ConsumerGroups, of which, in the default level monitoring we are particularly interested in:

  • EstimatedMaxTimeLag
  • MaxOffetLag

Is there a timeline/plan to introduce these metrics to the AWS Managed Kafka Integration for new-relic? https://docs.newrelic.com/docs/integrations/amazon-integrations/aws-integrations-list/aws-managed-kafka-msk-integration

New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

Hi @david.woosnam, welcome to the Explorers Hub. I have no info on whether we have plans to introduce those metrics to AWS Kafka integration but I have submitted a feature idea. I will update this thread if there is any word from our product development team on this.

Thanks Neal. Would be really useful.

If it’s any help as a supporting argument. I think in any situation where an enterprise is monitoring the health not just of the kafka internals, but the processes running on top of the kafka infrastructure, these are the crucial metrics to track. Without these, kafka can look healthy even though your processors are building up a backlog of work.

Great call @david.woosnam! It’s always good to include any justifications like this on the thread for review! I have also gone ahead and +1 this idea internally :slightly_smiling_face:

I have a similar need for these metrics. Specifically SumOffsetLag and MaxOffsetLag which are currently not available in New Relic. Wondering if there is any update on the timeline or plans for this feature since January?

Hi @Evie.May! This has been delivered as part of a new way to collect metrics from AWS (Metric Streams). It’s currently in beta since March 31st.


Do you know if there’s a way to join this beta or activate this feature?

is this feature available?

Hello @srontala - Indeed! It looks like this feature shipped back in April 2021!