WebFlux All Web Transactions - /NettyDispatcher

Hello,

We have Spring WebFlux application (Spring Boot 2.2.6, NewRelic Java agent 5.10) and all our web transactions are actually “/NettyDispatcher”. How can we configure agent or New Relic APM to have them as “url”.

https://rpm.eu.newrelic.com/accounts/2480496/applications/36568434/transactions?type=app#id=5b225765625472616e73616374696f6e2f4e65747479446973706174636865722f4e6574747944697370617463686572222c22225d

For now as a workaround we added @Trace annotations for each controller method, but it’s not really what we want to have in future

Thanks for help

1 Like

Hey @serhii.homeniuk, Thank you for reaching out.

TBH, having the methods instrumented is the best way to deal with issues with Netty where Spring cloud gateway is involved.

I would also like to give you a little bit of a glimpse under the hood, regarding how our agent’s various instrumentation packages work together, and how the agent goes about starting and naming transactions. When an HTTP request first hits your server, assuming you’re running a supported web server (we support Netty 3.3.0.Alpha1 to 5.0.0.Alpha1), we mark a transaction start point and assign a name. That name is usually the page URI. But, as that request moves through your application, the agent will notice and recognize certain method calls that match one of its instrumentation packages. For example, it recognizes spring controllers that are annotated with the @Controller annotation.
As it does so, it renames the transaction that appears in APM, as per our transaction naming protocol.

So a single transaction trace might be named based on page URI, then renamed NettyDispatcher when it hits our Netty instrumentation, and then renamed again to a controller name when it hits our Spring instrumentation.

We do have instrumentation for Spring however, we don’t have an instrumentation package specific to Spring cloud gateway, which I see in your application. Please check out our Compatibility and requirements for the Java agent.

I understand you might not want to make code changes every time you install an agent on Netty, but you can leverage custom instrumentation editor where you have the capability to instrument methods through NR UI itself.

The other way to maybe mitigate this issue is by disabling auto_transaction_naming from the YAML file. This will just be a troubleshooting step. We, however, do not recommend this step as it might lead to and MGI. Relic Solution: Video - Understanding Metric Grouping Issues in 80 seconds and will start sending us unusual unique metric names.

Let me know if that helps. Cheers!

2 Likes

Hi @mlavania,

Thank you for response, but actually there is not Spring Cloud Gateway involved in this application - this is simple Spring WebFlux application with few endpoints.

I saw article that NewRelic support it:


but why cannot I achieve results like on screenshot?

Does it mean that I need to put @Trace everywhere in my app? Or even worse:
handle tokens and segments everywhere like mentioned in https://docs.newrelic.com/docs/agents/java-agent/async-instrumentation/java-agent-api-asynchronous-applications?

Do you have some real example how to achieve results similar to what we have with SpringMVC?

Thanks

Hi @serhii.homeniuk, thank you for your reply. Yes, we do have instrumentations for WebFlux however, as you see in the document, you might need to use @trace annotation or if the transactions are asynchronous then leveraging Token APIs is the best way moving forward. The reason is if you apply instrumentation (@trace) on these calls and when it hops on another thread, we lose it.

Our developers are continuously working on improving the instrumentation for /Netty. You can try disabling the auto_transaction_naming from the YAML file to mitigate the issue if the naming, in the codebase, is as per the transaction naming protocols..

Also, apologies, I went to the environment page of the application, saw a reference to spring-cloud-context, and thought you might have a cloud gateway.

Best,

Hi @mlavania

Thank you for quick response. We will try then to play around custom tracing then and auto_transaction_naming. And hope that soon in some of the next versions it will work out of the box.

Best Regards
Serhii

@serhii.homeniuk, it sounds good. Do let us know how it goes :slight_smile:

Cheers,

@serhii.homeniuk Have you had any luck with the custom tracing? Have a webflux environment and auto_transaction_naming=false seemed to break out the URIs better, but I’m not getting anything but the netty superficial transaction info, even with @Trace annotations on various methods.

@nyoung
From what I see having auto_transaction_naming=false is only think that works more or less acceptable. But as result I have all internal methods and external calls combined as one - NettyUpstreamDispatcher component in Transaction Trace view

I actually spend some time on it and only after adding Tokens and Segments (see https://docs.newrelic.com/docs/agents/java-agent/async-instrumentation/java-agent-api-asynchronous-applications#tokens) helped to get some traces inside, but it’s definitely not something I want to have.
Unfortunately as a result I got multiple strange items with bad time and this is not what I expected

For now I don’t know what really to do with it :disappointed:

I also tried adapt solution from: Distributed trace broken when using Spring cloud gateway
but it doesn’t work for me too.

Thanks for the follow-up. You are getting pretty similar results to what I’m seeing thus far.

@serhii.homeniuk, @nyoung, thank you for reaching out again and explaining the scenario. Could you guys please send in the permalink of the application in question. I just wanted to see what is been reflected in the UI.

To generate permalink: https://docs.newrelic.com/docs/using-new-relic/user-interface-functions/share-your-data/permalink-url-selected-time

Cheers.

@mlavania

here is link for web-flux controller transaction with auto_transaction_naming=false (transaction trace is wrong as you can see): https://rpm.eu.newrelic.com/accounts/2480497/applications/38046345/transactions?tw[end]=1589874060&tw[start]=1589787660&type=app#id=5b225765625472616e73616374696f6e2f5572692f637573746f6d65722f7370656369616c2d7061796d656e7473222c22225d&app_trace_id=9d5273fb-9926-11ea-9f9f-0242ac110007_3378_4115

here is link for custom trace with token linking (still transaction trace is not perfect): https://rpm.eu.newrelic.com/accounts/2480497/applications/38046345/transactions?tw[end]=1589990400&tw[start]=1589947200#id=5b224f746865725472616e73616374696f6e2f437573746f6d2f72657472696576652d7370656369616c2d7061796d656e7473222c22225d

Best,
Serhii

@serhii.homeniuk, apologies for the delayed response here. It is good to see that you have started getting the transactions other than one /NettyDispatcher. Yes, the transaction trace doesn’t look correct. Did you try instrumenting all the methods that are of interest or you have tried on a few?

Instrumenting parent methods might not auto instrument the ones been called under/by it. I would also suggest running a thread profiler to see the methods been captured and accordingly try instrumenting those to understand if it does some good to the transaction traces.

You can run a thread profiler from this JVM page: https://rpm.eu.newrelic.com/accounts/2480497/applications/38046345/profiles

Thanks.

@mlavania I instrumented all methods that I’m interest in, but still for some reason they are linked to NonWeb transaction that created via method annotation Trace with (dispatcher=true) and not linked to web transaction

and web transaction is just having huge “reminder” block where all remote calls happened

So as I understand I can also use rollupMetricName for grouping stuff - is it correct?
Can I then somehow put web transaction inside rollupMetricName?

@mlavania can I ask whether NewRelic team has some example project with proper instrumentation for webflux - because NewRelic announced support back in 2018 (https://blog.newrelic.com/product-news/newrelic-announces-support-spring5-webflux/) and I cannot get to the state that I saw on this blog article. Are there also any updates about plans for Netty instrumentation enhancements?

3 Likes

@serhii.homeniuk,

A ticket has been created to address you requests, please follow along there for updates.