How to associate background threads with request - Scala

Hey Alex,

As our Product Manager Matt previously mentioned, “we are indeed actively working toward a custom instrumentation API that can connect work across threads.” This API is not yet currently available but when released it should allow you to custom instrument the async transactions for frameworks that we don’t yet currently support for out of the box (such as Finch library).

Cheers,

Hi Alex,

I have submitted Finch support by the Java agent as a feature request to our product management team to be considered for a future release. If it is implemented, we will try to notify you. While we can’t guarantee when or if this feature will be implemented, we take customer requests very seriously and use them to prioritize which features we implement next. Thanks for helping us improve the product!

Bojana

1 Like

Please add RxJava to your list. In fact some generic way to wrap any Executor would be nice too.

Hi @amarch, I’ve submitted a feature request with your information attached.

Hey Jimm,

My application uses a single Jersey 2 async endpoint, but I’m having the same problem instrumenting it that you did. Can you offer more details on your solution? Code samples in particular would be nice - I get that it involves NewRelic.setRequestAndResponse, but I’m not sure I understand exactly what needs to be done to associate the background thread with the NewRelic transaction.

Thanks.

Hi gwilson - my solution was not very good because each piece of background work showed up in the New Relic UI as a distinct transaction. We ended up not using it because it threw our metrics way off. For example if we used 10 transformations of a Scala Future to fully process the request, we ended up with 10 transactions for every request, and each transaction appeared to be extremely fast because it didn’t account for any of the wait non-blocking waits.

But let’s say you want to go down that path… Are you using Scala or Java? The solution will be very different in each case. The general idea is to store the request & response when your endpoint is called, and then implement a way to pass that along to each piece of background work.

For myself, in Scala:

  • I implemented a custom ExecutionContext which could do this, and before it did any work it would call setRequestAndResponse, and then called clearRequestAndResponse() when finished.
  • I also had to implement a servlet filter which obtained the request & response in the first place.
  • I also had to implement a DynamicFeature which could compute the endpoint path (like “/some/path/123 (GET)”), so that the transactions representing the background work would still be grouped together with the main endpoint.
  • I also had to make an implementation of com.newrelic.api.agent.ExtendedRequest which pointed back to the servlet request.
  • I also had to make an implementation of com.newrelic.api.agent.Response which pointed back to the servlet response.

All this code was not very polished since I was deep in the “tinkering” stage where I was just experimenting with things to get it all to work. But if all this sounds good let me know and I can post it.

1 Like

Hey jimm, thanks for the in-depth response.

I’m on Scala, so it sounds like your experience is applicable to me. It looks like I might have an easier time of it since I do everything in one shot (eg. Future(foo).onSuccess(bar)) instead of mapping and flatmapping a number of smaller monads together, meaning I shouldn’t have an issue with multiple transactions.

So yes, it would be great to see how you implemented it - if you and your employer don’t mind it.

BTW, have you considered using Scalaz or Monix’s Task instead of the standard library’s Future? The most relevant difference is that you don’t pass an ExecutionContext/ExecutorService/whatever when calling “map”, the monad instead just uses the previous thread when mapping and only does a submit to the thread pool on flatMap. Depending on your application’s design, that might solve your multi-transaction problem.

1 Like

Grenville - I replied directly outside this forum, let me know if you don’t get the message!

1 Like

@jfalleur / @Linds / @jkeller - could you give an update on the progress? In Spring 2016 you mentioned that it was being actively worked-on.

It’s been a while so I’ll reiterate my problem:

  • When we use standard Scala futures, the work executes on a different thread from the original request and therefore doesn’t get associated with that transaction. The transaction shows up like “pool-11-thread-519095”, etc.
  • I made a custom ExecutionContext which allowed me to associate all the asynchronous work with the original request, but the agent provided no way for me to say “Hey New Relic agent, consider any subsequent work on this thread to be with transaction ID 123456” (no attach/detach to transaction). Therefore all the async work showed up as duplicates of the original transaction, which ballooned our throughput counts (each real-world request resulted in many tiny requests in New Relic, each request representing one tiny piece of async processing.)

As a baby step, we’d love some simple feature which still placed all the burden of tracking background work onto our own application, if only the agent supported three new APIs:

  • get the ID of the current transaction
  • tell the agent “this thread is now actually part of transaction {id}”
  • tell the agent “this thread is no longer associated with transaction {id}”

As a long-term step it would be awesome if this came for free out of the box, like so many other amazing things that the agent already does.

Thanks!

@jimm, the next Java agent should have improved support for Jersey/JAX-RS. We are also looking at the possibility of expanding access to the Java agent’s internal APIs. We should have a point release with improved Jersey support next week and please also keep an eye on our future release for opening of our APIs.

1 Like

It looks like some support for async was added in 3.37.0 https://docs.newrelic.com/docs/agents/java-agent/async-instrumentation/java-agent-api-asynchronous-applications

Hi!

I´m trying the version 3.37.0 (i need async support) but i get the following error:

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/newrelic/api/agent/Token
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212)
at com.google.common.cache.LocalCache.get(LocalCache.java:4053)

Caused by: java.lang.NoClassDefFoundError: com/newrelic/api/agent/Token
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
at
lCache$Segment.get(LocalCache.java:2208)
… 40 more
Caused by: java.lang.ClassNotFoundException: com.newrelic.api.agent.Token
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

The error occurs during deployment (Tomcat).

Is anyone using this version?

I’m using 3.37.0 without this problem. You might want to ensure that the version of the JAR that is deployed in your production environment is at version 3.37.0, not just the thin agent facade JAR that you bundle in with your app.

1 Like

Hey @mauro.capolupo - Did @jimm’s suggestion work for you! Feel free to click the check box icon if that solved the problem. Thanks!

With the new 3.37.0 async support, I was able to successfully track our Scala Future processes, and also our asynchronous javax.ws.rs.client.Client calls.

At Porch we have a tradition in the engineering department called Dev Day. Every few months the devs get two or three days to focus on something we’re passionate about, so I took advantage of that to get this working in a Scala app. I was just able to prove the concept this morning and it will take time to get the code cleaned up into something that I’d want to share, but I’ll include more in this thread later with details.

In turned out to be 20x more complicated than I expected. Here’s what I had to implement:

  • A custom Scala ExecutionContext which can propagate the NR Transaction to subsequent threads. This utilized the new Token feature added in 3.37.0 to “link” the next unit of work, and expire it afterwards.
  • A custom servlet Filter to create an object that stores my working data (the NR Transaction and a unique ID that helps me track everything).
  • I had to wrap my existing javax.ws.rs.client.Client instances in a proxy, so that I could create a NR Segment when a client API call is being configured, while we’re still inside a thread that is linked to an active transaction. (This also required me to proxy javax.ws.rs.client.WebTarget.) I wanted to solve this with a ClientRequestFilter but async filters are invoked on a totally different thread pool, and I had no way to link back to the original NR transaction.
  • A custom ClientRequestFilter, which can hydrate the NR Segment with additional data when the Jersey client is ready to make the request. For example it passes HttpParameters to segment.reportAsExternal().
  • A custom ClientResponseFilter which allows me to call segment.end() when a client API call completes.
  • Finally I also had to augment our async endpoints so that we capture the NR Transaction at the beginning of our processing. I wanted to do this with the servlet Filter but my filter was getting called before the true transaction had been created. Luckily we already route our async endpoints through a utility so it was easy to add this and have it affect the entire app.
4 Likes

https://s3.amazonaws.com/uploads.hipchat.com/75009/1168625/pLiCb1nvXFOOYLa/upload.png

4 Likes

Did you share your code in some project on github? Maybe newrelic developed something more easy to use or it has plan to export this logic with just a simple annotation? @Linds

I haven’t published it anywhere yet since I have been simplifying it a little bit since the original proof of concept.

1 Like

Thanks @jimm. I will follow this post! It is really interesting!

1 Like

HI @jimm, I’d be very interested to see an example of this code if you are happy to share it.

cheers

1 Like