Feature Idea: Python ThreadPool transaction tracing

Hey,
I’m using Pyramid and the newrelic agent v2.72.1.53 on py.3.4

In one of my transaction, I’m calling concurrent.futures.ThreadPoolExecutor to make a thread pool to execute certain external requests. Turns out I don’t see these requests as “external services” in the newrelic apm UI. Is there a way I can instrument these calls (perhaps with a decorator) to tie these threads to the current transaction so that they correctly show up?

If it’s not possible, is it on the roadmap at some point? I feel like the functionality has to be there somewhere in the python agent already.

Thank you

New Relic edit

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

@joelm Thanks for posting to the forum. Would you please send us a permalink to one of your transactions? To create a permalink to any page within the New Relic user interface, scroll to the bottom and click ‘Permalink’ all the way on the right next to ‘Kiosk Mode.’ This will show us the exact page and time period that you are observing.

Also, can you please elaborate on whether you’re spawning a new thread in the transaction to make the external call? If so, I’ll move forward with the feature request. At this time, the Python agent is unable to properly instrument code that starts in one thread and ends in another thread.

Here’s a sample trace: Permalink

So like, the request starts in one thread and obviously ends in the same thread, but yes, that thread is spawning other threads (which make the various rpc calls) and waits for them to complete. This is a basic method of parallelization… I’d love to have the feature request at the very least yeah.

@joelm Thanks for elaborating. A transaction must have everything run through and complete on a single thread for tracing. Our product developers would definitely love to hear your input on how this would work better for you as a feature idea. You can post your idea right here in the OTC. This will allow other users to participate by adding their support and use cases for your feature idea as well. Here is a link to the Python agent feature idea section, where you can add your use case for your proposed feature:

https://discuss.newrelic.com/c/feature-ideas/feature-ideas-python-agent

We have the same use case. We have a web request which is calling out to 4-5 additional REST APIs. We use a ThreadPoolExecutor to multithread those requests, but would love to have them show up in the same transaction. I was hoping that I could do something like

trans = current_transaction()
start_thread(trans)

# now in the thread
with trans.nested_transaction():
  do_stuff()

It seemed like transactions have some notion of nesting.

Excellent use case, @jhorman! Thanks for sharing—I have added a poll so that you can vote as well!

have someone found a workaround using the python agent api ?

I used this decorator:

@newrelic.agent.background_task()
def function_which_runs_in_other_threads

In one of my transaction, I’m calling concurrent.futures.ThreadPoolExecutor to make a thread pool to execute certain external requests. Turns out I don’t see these requests as “external services” in the newrelic apm UI. Is there a way I can instrument these calls (perhaps with a decorator) to tie these threads to the current transaction so that they correctly show up?

@joelm Have you found a way to get this to work ?

@eugene.kovalev problem with background_task() is that it creates a new transaction…