Span parentId not correct

I am working on setting up tracing with the c++ opentelemetry library (https://github.com/open-telemetry/opentelemetry-cpp). I manage to export traces into New Relic using the opentelemetry http exporter (as opposed to the grpc one). It mostly behaves as I would expect: all the spans in a trace share a traceId, the root span has no parentId, and the child spans have a parentId. However, the value of the parentId is not correct. By placing breakpoints, I have determined that the following json in the http request:

{“resource_spans”:
[{
“instrumentation_library_spans”:[{
“instrumentation_library”:{“name”:“foo_library”,“version”:“1.4.0”},
“spans”:[{
“attributes”:[
{“key”:“custom_attribute”,“value”:{“string_value”:“custom value”}},
{“key”:“parentId”,“value”:{“string_value”:“674695c1786e492f”}},
{“key”:“parent.id”,“value”: {“string_value”:“674695c1786e492f”}}],
“end_time_unix_nano”:“1653473808383162400”,
“kind”:1,
“name”:“f2”,
“parent_span_id”:“674695c1786e492f”,
“span_id”:“deec627e46294f43”,
“start_time_unix_nano”:“1653473785526072300”,
“trace_id”:“ebad050d74667129d3036d695577cab6”}]}],
“resource”:{
“attributes”:[
{“key”:“entity.name”,“value”:{“string_value”:“anton-test”}},
{“key”:“telemetry.sdk.name”,“value”: {“string_value”:“opentelemetry”}},
{“key”:“service.name”,“value”:{“string_value”:“anton-test”}},
{“key”:“telemetry.sdk.language”,“value”:{“string_value”:“cpp”}},
{“key”:“telemetry.sdk.version”,“value”:{“string_value”:“1.4.0”}}
]
}
}
]
}

However, what shows up in the New Relic dashboard is the following


Note that I have specifically added the parentId and parent.id attributes to the json in an effort to remedy the issue, and while this fixed the problem for parentId, it did not for parent.id. The opentelemetry specification seems to mention that parent.id should be set from parent_span_id in the json, but this is evidently not what happens here. Is this an issue on the new relic side, or on my side?

I am having the same problem, were you able to fix it @AntonApm?

Hey @AntonApm,

Welcome to the community.

While I am not positive where the issue is arising from as this is out of my scope, I am touching base with our engineers to see if this is something we support and if we can find a solution for you. We appreciate your patience and will get a response out to you shortly.

@german2 I was not able to fix this yet. Based on the response from @michaelfrederick I will wait for a bit and see what comes back from the New Relic engineers.

Hi @AntonApm and @german2

I can see that the support team are working on this, they will post here with any updates they have.

Should you have any additional updates or questions please do reach out.

1 Like

Hello AntonApm,

As noted this sounds a lot like the problem german2 is having with their open telemetry spans. The engineers were able to reproduce this behavior and they are working on figuring our where the problem is coming from. So I wanted to ask you for the same info @jberg asked for in that thread:

  • What version of the exporter-otlp-http exporter are you using?
  • Or, if using the exporter-trace-otlp-http exporter, what version are you using?
  • What version of the C++ OpenTelemetry client are you using?

Thanks,

Brian

Hi @bhensley,

To answer your questions, I use version 1.4.0 of the open telemetry library (Release OpenTelemetry C++ v1.4.0 · open-telemetry/opentelemetry-cpp · GitHub). Regarding your other questions, I do not know what distinction you have in mind with the two exporters you’re mentioning, but from the library mentioned above, I use the following class for exporting:
opentelemetry::v1::exporter::otlp::OtlpHttpExporter
This one is used by default in the example_otlp_http project from the open telemetry source code, which can be used to reproduce the issue if one feeds the exporter valid New Relic credentials.

Hello AntonApm,

I received some input from one of our open source engineers. At the moment the http/json flavor of OTLP support is fragile (and extremely sensitive to naming differences). It looks like that flavor of OTLP is the default for the OTel C++ library. The engineer recommends you try the http/protobuf flavor of OTLP by configuring the content_type setting in your exporter to use application/x-protobuf. See:

Please let us know if this is able to fix your misnamed parentIds.

Best,

-Brian

1 Like

Hi Brian,

Thanks for the suggestion. I have tried changing the content type, but as a result data does not show up in New Relic at all anymore. I have looked at what gets put in the http request to New Relic, and it still has the same kind of json structure as I mentioned in my original post, except that the span id, and trace id seem to be presented in a different format.

However, the official documentation says the open telemetry trace interface is now stable, so I do not see why basing a json record structure off of it would be problematic.

Hey there @AntonApm and @german2 ,

Thanks for reaching out to the New Relic Community! I think I might be able to shed some light on your situation. After going over the details above and discussing it with a few other team members, some interesting information regarding Open Telemetry itself came to light. If you go to the opentelemetry github and look under the “OTLP/HTTP” section, you’ll notice that the support for JSON in this case is still labeled as “experimental”. That being the case, I’ve included a doc HERE which should take you to an example and instructions on how to get OTLP running for C++ while using the recommended protobuf method.

I hope this information helps! Please don’t hesitate to reach out if you have any extra details in the mean time or questions!

Cheers