How to associate background threads with request - Scala

I am migrating a Scala Play application to Dropwizard 1.0 using asynchronous Jersey back-end endpoints (using @Suspended AsyncResponse). Almost everything is running in a Future - DB calls (Slick 2.x), API calls to other services, etc. I’m using the 3.26.1 agent.

When it was running in Play, the agent was able to track the async activity just fine, but in the new framework New Relic reports clumps most of the processing under “AsyncProcessing”.

In hopes of improving granularity of reporting, I’ve made a custom ExecutionContext which allows me to wrap the async Future calls in my own logic, and I’m able to pass context from one thread to the next just fine, but I can’t find a way to help New Relic follow the chain of events.

For example, in the documentation for the @Trace asyc=true attribute, it says this should be possible:

“The user implicitly promises to register the object implementing traced method from within some existing New Relic Transaction before invoking the method and then to call the Transaction.startAsyncActivity API as the first action of the traced method.”

Unfortunately there is no “startAsyncActivity” method in the Transaction class, and I don’t see any reference to async in any of the other classes in the agent JAR.

Here’s how I would expect it to work, but the agent API seems to be missing some pieces.

  • When I’m about to queue up work on a background thread, I can call NewRelic.getAgent.getTransaction to remember which transaction was active.
  • When the background thread starts up, I’d like to call some kind of agent API like resumeTransaction and pass in the transaction that I know I belong to.
  • When the background thread finishes, I’d like to call some kind of agent API like suspendTransaction to let it know that any work this thread does later does not belong to my original transaction.

I have seen other examples online where they mention a method registerAsyncActivity but that doesn’t seem to exist in the Transaction class in 3.26.1.

I’m able to track all the background work myself but am simply not able to help New Relic connect the dots.

Is there something I’m missing, that would allow me to reconnect a new thread (Future) to a transaction that was started on a prior thread?

Jim

1 Like

A small update… I’ve worked on this more today and am able to get detailed trace information on my background Futures, but it results in 2 New Relic transactions for each request.

For example if I make a single request to “/user/1234”, in the New Relic UI I will see two transactions:

  1. I’ll see a transaction without any visibility into the real work, where most of the time is categorized into “AsyncProcessing”. The only nice thing about this transaction is that the transaction name is the URI template (like “/user/{userId}”) instead of the requested URI (like “/user/1234”)

  2. I’ll see another transaction with the actual URI of the request, like “/user/1234”, and with all the nice trace details of the asynchronous work broken out nicely (DB calls, API calls, etc).

I am pretty sure we end up with two transactions because I had to create my own implementations of the Request/Response classes so I could pass something to NewRelic.setRequestAndResponse().

I think I’d have a complete end-to-end solution for my system if there was an API like NewRelic.resumeTransaction(transaction) as requested in the first message above. Then I wouldn’t have to make custom request & response implementations, and wouldn’t have to worry about matching the name of the transaction, etc.

Although it’s nice to see the trace details again in New Relic, I’m nervous about pumping hundreds of thousands of unique transaction names into the UI (since my code doesn’t yet know how to find the generalized URI template annotated with @Path on my endpoints).

1 Like

After more work I’ve figured out how to detect the generic URL template ("/user/{userId}" instead of “/user/1234”) and am setting the New Relic transaction name.

Additionally all background pieces are now reported in New Relic. But there is still one gigantic problem:

New Relic treats all the individual background Futures as separate transactions. If my request processing chains together three Futures, then I end up with three transactions. This makes the throughput numbers very misleading, since we have some requests that take up to 20 different Future executions before completion (leading to a 20x increase in throughput).

Summarizing:

  • I can get the default Transaction before spawning any background threads, and …
  • I’ve successfully been able to propagate the request & response & generic transaction name to all background threads …
  • I just need a way to tell NewRelic that I’m part of that initial first Transaction, instead of always creating a new Transaction.
1 Like

Hi @jimm Thanks for your detailed contribution. We understand the problem and are actively working on getting a solution out soon. Unfortunately, we don’t have a concrete solution right now. You can keep an eye on the Java Agent release notes for any updates:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/java-release-notes

@jimm You really did get very far by yourself considering this is a known issue and is currently being worked on. Hopefully I will have an update on this soon!

Like @jfalleur said: keep an eye on future release notes for news on this fix! :blush:

Hi @jimm, I’m the Product Manager for the Java Agent here at New Relic. I want to reiterate that we really appreciate your involvement in this community and that we are indeed actively working toward a custom instrumentation API that can connect work across threads.

We are also working on Jersey instrumentation, which prompts me to ask: do you think you’ll use custom instrumentation to connect your application code across threads, or is your sole interest in Jersey?

1 Like

Hi Matt -

Ideally we wouldn’t have to use any custom solutions to track our code. I only went down that path because we have become so reliant on all the insight provided by New Relic that it feels like we’re driving blind without it.

We’re using:

I’d be happy to share the Scala source with you if you ever need it. It probably wouldn’t be directly helpful because it requires some effort on the part of the calling app to set up (use a custom ExecutionContext, register a servlet filter to capture Request/Response, register a Jersey filter to capture endpoint URL template, etc).

By the way, while I have your ear … I just noticed another issue with Jersey 2 async endpoints. In traces, the New Relic UI is incorrectly showing the HTTP response as 204 (no content), instead of recognizing the actual return code from the response.

Yesterday we turned off my custom tracing code because of the way it inflates the transaction count. When a request is broken into dozens of tiny transactions, it had a couple big side-effects in the New Relic UI:

  • It would bury errors. For example if one request required 20 transformations of a Future, an error could be affecting 100% of the API calls from the POV of the client, but in New Relic it would only show up as 1/20 of the transactions. When the error was less obvious, like only triggering on 1% of the requests, it would be nearly invisible.

  • It would reduce the overall response timings, because the UI aggregates and averages them over time. Our response timings were much smaller than reality, and our transaction counts were much higher than reality.

1 Like

Hi,

I’m giving a try to Finch library (https://github.com/finagle/finch) and I’m unable to trace async work. I think it would be awesome if gives us a solution so we can try HTTP libraries that new relic do not support.
Has there been any new improvement regarding this area?

Thanks,
Alex

Hey Alex,

As our Product Manager Matt previously mentioned, “we are indeed actively working toward a custom instrumentation API that can connect work across threads.” This API is not yet currently available but when released it should allow you to custom instrument the async transactions for frameworks that we don’t yet currently support for out of the box (such as Finch library).

Cheers,

Hi Alex,

I have submitted Finch support by the Java agent as a feature request to our product management team to be considered for a future release. If it is implemented, we will try to notify you. While we can’t guarantee when or if this feature will be implemented, we take customer requests very seriously and use them to prioritize which features we implement next. Thanks for helping us improve the product!

Bojana

1 Like

Please add RxJava to your list. In fact some generic way to wrap any Executor would be nice too.

Hi @amarch, I’ve submitted a feature request with your information attached.

Hey Jimm,

My application uses a single Jersey 2 async endpoint, but I’m having the same problem instrumenting it that you did. Can you offer more details on your solution? Code samples in particular would be nice - I get that it involves NewRelic.setRequestAndResponse, but I’m not sure I understand exactly what needs to be done to associate the background thread with the NewRelic transaction.

Thanks.

Hi gwilson - my solution was not very good because each piece of background work showed up in the New Relic UI as a distinct transaction. We ended up not using it because it threw our metrics way off. For example if we used 10 transformations of a Scala Future to fully process the request, we ended up with 10 transactions for every request, and each transaction appeared to be extremely fast because it didn’t account for any of the wait non-blocking waits.

But let’s say you want to go down that path… Are you using Scala or Java? The solution will be very different in each case. The general idea is to store the request & response when your endpoint is called, and then implement a way to pass that along to each piece of background work.

For myself, in Scala:

  • I implemented a custom ExecutionContext which could do this, and before it did any work it would call setRequestAndResponse, and then called clearRequestAndResponse() when finished.
  • I also had to implement a servlet filter which obtained the request & response in the first place.
  • I also had to implement a DynamicFeature which could compute the endpoint path (like “/some/path/123 (GET)”), so that the transactions representing the background work would still be grouped together with the main endpoint.
  • I also had to make an implementation of com.newrelic.api.agent.ExtendedRequest which pointed back to the servlet request.
  • I also had to make an implementation of com.newrelic.api.agent.Response which pointed back to the servlet response.

All this code was not very polished since I was deep in the “tinkering” stage where I was just experimenting with things to get it all to work. But if all this sounds good let me know and I can post it.

1 Like

Hey jimm, thanks for the in-depth response.

I’m on Scala, so it sounds like your experience is applicable to me. It looks like I might have an easier time of it since I do everything in one shot (eg. Future(foo).onSuccess(bar)) instead of mapping and flatmapping a number of smaller monads together, meaning I shouldn’t have an issue with multiple transactions.

So yes, it would be great to see how you implemented it - if you and your employer don’t mind it.

BTW, have you considered using Scalaz or Monix’s Task instead of the standard library’s Future? The most relevant difference is that you don’t pass an ExecutionContext/ExecutorService/whatever when calling “map”, the monad instead just uses the previous thread when mapping and only does a submit to the thread pool on flatMap. Depending on your application’s design, that might solve your multi-transaction problem.

1 Like

Grenville - I replied directly outside this forum, let me know if you don’t get the message!

1 Like

@jfalleur / @Linds / @jkeller - could you give an update on the progress? In Spring 2016 you mentioned that it was being actively worked-on.

It’s been a while so I’ll reiterate my problem:

  • When we use standard Scala futures, the work executes on a different thread from the original request and therefore doesn’t get associated with that transaction. The transaction shows up like “pool-11-thread-519095”, etc.
  • I made a custom ExecutionContext which allowed me to associate all the asynchronous work with the original request, but the agent provided no way for me to say “Hey New Relic agent, consider any subsequent work on this thread to be with transaction ID 123456” (no attach/detach to transaction). Therefore all the async work showed up as duplicates of the original transaction, which ballooned our throughput counts (each real-world request resulted in many tiny requests in New Relic, each request representing one tiny piece of async processing.)

As a baby step, we’d love some simple feature which still placed all the burden of tracking background work onto our own application, if only the agent supported three new APIs:

  • get the ID of the current transaction
  • tell the agent “this thread is now actually part of transaction {id}”
  • tell the agent “this thread is no longer associated with transaction {id}”

As a long-term step it would be awesome if this came for free out of the box, like so many other amazing things that the agent already does.

Thanks!

@jimm, the next Java agent should have improved support for Jersey/JAX-RS. We are also looking at the possibility of expanding access to the Java agent’s internal APIs. We should have a point release with improved Jersey support next week and please also keep an eye on our future release for opening of our APIs.

1 Like

It looks like some support for async was added in 3.37.0 https://docs.newrelic.com/docs/agents/java-agent/async-instrumentation/java-agent-api-asynchronous-applications

Hi!

I´m trying the version 3.37.0 (i need async support) but i get the following error:

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/newrelic/api/agent/Token
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212)
at com.google.common.cache.LocalCache.get(LocalCache.java:4053)

Caused by: java.lang.NoClassDefFoundError: com/newrelic/api/agent/Token
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
at
lCache$Segment.get(LocalCache.java:2208)
… 40 more
Caused by: java.lang.ClassNotFoundException: com.newrelic.api.agent.Token
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

The error occurs during deployment (Tomcat).

Is anyone using this version?