Distributed tracing kafka batch listener

I am trying to use distributed tracing with kafka as per https://docs.newrelic.com/docs/agents/java-agent/instrumentation/instrument-kafka-message-queues. I am wondering if it is a fit for my use case. I have a kafkaconsumer that works in “batch” mode. It handles N kafka records at a time in a single transaction.

In my kafkaconsumer should I call

for(Header header: headers) {
        NewRelic.getAgent().getTransaction().acceptDistributedTracePayload(new String(header.value(), StandardCharsets.UTF_8));
    }

for each record in the batch before starting record processing? Processing some of these records might involve producing more kafka records to other batched listeners.

Here’s the scenario I’m really worried about. Batch 1 comes in with records a, b and c.

The handler is able to immediately handle records a and b but needs to send c to another topic for further processing. The handler for that topic gets batch 2 with records c, d and e. It finishes with c and d but sends e to another topic for further processing.

Will the trace for records a and b just go on forever? Are batch listeners just not a good fit for distributed tracing?

Hi @JBotuck,

Yes! You will need to call that portion of your code for every batch. As documented - here:

for(Header header: headers) {
    NewRelic.getAgent().getTransaction().acceptDistributedTracePayload(new String(header.value(), StandardCharsets.UTF_8));
}

when you do a kafka read, it will try to fill the batch up to the max batch size with records, where each could have different distributed tracing data.

For instance, ten different producers could produce a record and they might all get batched for read by a single consumer. So, if the consumer reads 10 records in the batch, each one will need to have its trace headers recorded. Also, you will need to ensure that you have all recent agents as well as the relevant config enabled on every application (producer and consumer apps).

As for whether the trace will go on forever - I don’t think so.

Hope that helps.

1 Like