Feature Idea: NR does not display time spent in requests module

Please paste the [permalink][1] to the page in question below:

Please share your agent version and other relevant versions below:

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

Hi, I am running our Django 1.8 server with gunicorn/gevent worker settings and have a NR agent with version

Since we run everything in gunicorn settings, we never had to directly spawn gevent greenlets programmatically but we recently had such needs and had a piece of code that spawns greenlets that make GET requests using python “requests” module.

Code works in parallel just fine like before but the issue now is that time spent making GET requests now shows under the “python” portion. Before the NR agent classified such requests based on where we are making GET requests to.

Is there any workaround so that the NR agent shows fine-grained details of our GET requests?

New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

Hi @nate_kim The issue here is that the GET requests you are sending are now happening in spawned greenlets, which the agent does not have insight into. Once the work is finished, and the greenlets return to their spawning function, all that time looks like time spent in “python” since all the agent was able to see is the function (that spawned the greenlets) waiting.

This is why you used to have visibility into the work being done in the request, and now you do not. Unfortunately, we do not have a workaround to get insight into this type of use-case at this time.

Hi agoetz,

thanks for your explanation. I do, however, have a question lingering in my mind.
If I compare the before and after, the only thing that’s different is how we spawn greenlets.

Before, Gunicorn did that for us underneath the hood, after I spawned greenlets programatically. Since we use the gevent worker settings for Gunicorn, I get that it essentially does the same thing as programmatically spawning greenlets for each incomming request. How did the NR agent capture such activity before?

Does it have to with monkey patching? Now that I spawn greenlets myself, I do monkey patch everything (thread, socket, and others especially) and I kinda guess that the NR agent now probably runs in its own greenelt instead of a thread and thus is limited.

I would graciously appreciate your elaboration.

thanks in advance!

Sure I can provide some more detail @nate_kim

As the Python agent works now, when a transaction is created (either from a Web request, or from starting a Background Task), the agent expects that all work done in that transaction will be done on a single thread (or greenlet). A transaction should run entirely on the same thread on which it starts. If work is farmed out to a separate thread in the middle of a transaction (either by spawning a new thread or greenlet, or by using a thread pool), then any traced function that runs in that separate thread will not be recorded.

Essentially, the agent does not have a way to associate work done in the spawned thread with the transaction that was started in the original thread.

As your previous experience shows, simply using the gevent worker with gunicorn generally does not cause any problems with our instrumentation. In the typical use case, gunicorn spawns a separate greenlet to handle each request, and each web transaction runs entirely in that spawned greenlet.

It is only in cases where a transaction is started, and then new threads (greenlets) are spawned, that cause instrumentation and API calls to go unrecorded, since the activity in these new threads (greenlets) goes unseen by the agent.

Is there any way for the new relic python agent to support this use case? Or no?

@km-pg Thanks for posting to the forum. If you are starting a transaction in one thread and completing the transaction in another thread, the Python agent would not be able to support this use case.

I’d be happy to create a feature request on your behalf for this functionality. Let us know if you have any questions.

1 Like