Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Java Agent: Distributed Trace payload is empty!

java
javaagent
rfb

#1

I am using the Java Agent API to demonstrate Distributed Tracing between two services using AWS Simple Queueing Service (SQS).

When I try to create a distributed trace payload, the variable is empty! There is no clue in the logs as to what is going on. See my code snippet below

@Trace(dispatcher = true)
private void addMessageToQueue(String message) {
    // Get a distributed trace payload to include with the SQS message so we can see a trace
    String dtPayload = NewRelic.getAgent().getTransaction().createDistributedTracePayload().text();

    // Create a new message attribute to include the distributed trace payload
    System.out.println("distributed trace payload " + dtPayload);
    MessageAttributeValue msgAttribute = new MessageAttributeValue().withStringValue(dtPayload);
    Map<String, MessageAttributeValue> msgAttrMap = new HashMap<>();
    msgAttrMap.put("dtPayload", msgAttribute);

    SendMessageRequest send_msg_request = new SendMessageRequest()
            .withQueueUrl(queueUrl)
            .withMessageAttributes(msgAttrMap)
            .withMessageBody(message)
            .withDelaySeconds(5);

    sqs.sendMessage(send_msg_request);
}

I am able to verify that the agent is working and I can do things like read the Agent config and get the transaction name programmatically.

  • Java Agent Version: 4.10.0

#2

Hi @danielfitzgerald

I was able to duplicate this issue, however…it was caused by distribute_tracing being disabled. Could you verify in your newrelic.yml that enabled: true is configured for distributed_tracing ?

Cheers!
Matt


#3

Hi @MrMatt ,

I can confirm distributed_tracing = enabled in my yml config. I can confirm this by adding the following line to my code and the console output is ‘true’

System.out.println(NewRelic.getAgent().getConfig().getValue("distributed_tracing.enabled").toString());

I even tried adding the below flag to the VM options but it doesn’t work

-Dnewrelic.config.distributed_tracing.enabled

#4

Thanks @danielfitzgerald, I figured you had it enabled but had to check :slight_smile: I checked our code for this call and it looks like we’ll log any exceptions that occur if you set log_level: finest in your configuration. Could you set that log_level and look for one of those following lines:

"Created distributed trace payload: {0} for transaction: {1}", payload, this
OR
Unable to create distributed trace payload

The “Unable…” line should be preceded by the exception that occurred.

If you find those log lines and are able to upload them to this post I can definitely take a look!


#5

HI @MrMatt

In my log it says:

com.newrelic FINER: Not creating distributed trace payload due to null accountId.

#6

Hey @danielfitzgerald I’m going to move this discussion into a ticket since we’re going to need to request some private information, you should get an email about the ticket shortly. For anybody who comes across this discussion I’ll respond back here once we find the solution!


#7

Hi,

This is an update on how this particular issue was solved. The customer added this configuration to the newrelic.yml file:

sync_startup: true

And then agent was able to connect early enough to catch the call to Distributed Tracing. We believe this error ““The message attribute ‘dtPayload’ must contain non-empty message attribute type.”” in the New Relic logs was showing up because the call was happening right after the application had already started.

You can read more about this configuration setting here:
https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file#cfg-sync_startup


#8

I want to do this too. Where can I view the contents of the distribute context?


#9

hey @anthony.garo1 so you want to see distributed traces across services which communicate across AWS SQS? I put some code on GitHub which shows how you can enable this in your applications https://github.com/danifitz/newrelic-distributedtracing-sqs

Hopefully that helps, let me know if you have any questions!


#10

This is super helpful!
I really want to see what happens when you print the distribute context.
Is that in the documentation somewhere?
Specifically, this line:


#11

Hey @anthony.garo1 - the distributed trace payload looks like this

{"d":{"ac":"2181713","pr":1.402017,"tx":"5ae6fcb24a8fe37e","ti":1558710266339,"ty":"App","id":"c9b52f7864e09302","tr":"5ae6fcb24a8fe37e","sa":true,"ap":"5403826"},"v":[0,1]}

#12

This is exactly what I need! Thank you!

From your experience, is this payload flexible or customizable in any way?
The NR APIs look pretty standard

One other question, are the fields in the payload listed in the documentation?


#13

Hey @anthony.garo1 - the payload itself is not customisable AFAIK, it’s generated by the agents. What’s the use case for customising it?

I can’t find any documentation which explains the fields, closest I can find is this explanation of How Distributed Tracing works

By the way, I should share a gotcha. If you are using HTTP to fetch messages from the queue, you will see an issue with creating a trace because in service B, a distributed trace has already been created for the http interaction with the queue. A solution to this is to wrap the method which fetches messages with some code to momentarily disable the distributed tracing feature, like so:

newrelic.agent.config.distributed_tracing.enabled = false
sqs.receiveMessage(params, function(err, data) {
  newrelic.agent.config.distributed_tracing.enabled = true
  ...
})

#14

sorry that example is in Node.js, but hopefully illustrates what I mean.


#15

This is perfect. I appreciate the pro tip.