Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Force data to be sent in background job

backgroundjob
python

#1

newrelic.core.agent INFO - New Relic Python Agent (2.66.0.49)

I’m trying to instrument a legacy code that use pyres without luck.

The problem seems to be that methods I want to instrument are forked by the main worker thread and killed before new relic agent could sent the data.

I tried different approaches from the documentation. No luck.
Tried sleeping the job thread , still no luck

The basic workflow:

  • Start worker (long runnig process)
  • for each job found from a redis server, fork the process to run the job
  • jobs run fast (1 - 3s)

here are some logs, you can see that there is a multiple agent being created and registered, but they never send data over to newrelic.

2016-07-13 17:54:56,823 (37026/MainThread) newrelic.core.agent INFO - New Relic Python Agent (2.66.0.49)
2016-07-13 17:54:59,368 (37026/NR-Activate-Session/Ping-tornado) newrelic.core.data_collector INFO - Successfully registered New Relic Python agent where app_name='Ping-tornado', pid=37026, redirect_host=u'collector-159.newrelic.com' and agent_run_id=u'105936786689599589', in 2.50 seconds.
2016-07-13 17:54:59,370 (37026/NR-Activate-Session/Ping-tornado) newrelic.core.data_collector INFO - Reporting to: https://rpm.newrelic.com/accounts/1379331/applications/24665329
2016-07-13 17:55:21,483 (37085/MainThread) newrelic.core.agent INFO - New Relic Python Agent (2.66.0.49)
2016-07-13 17:55:23,945 (37085/NR-Activate-Session/Ping-tornado) newrelic.core.data_collector INFO - Successfully registered New Relic Python agent where app_name='Ping-tornado', pid=37085, redirect_host=u'collector-159.newrelic.com' and agent_run_id=u'105936786689601031', in 2.41 seconds.
2016-07-13 17:55:23,946 (37085/NR-Activate-Session/Ping-tornado) newrelic.core.data_collector INFO - Reporting to: https://rpm.newrelic.com/accounts/1379331/applications/24665329
2016-07-13 17:58:00,211 (37161/MainThread) newrelic.core.agent INFO - New Relic Python Agent (2.66.0.49)

#2

Hi @lucas.stephanou, thanks for posting to our forum! First, I’d like to
verify if you see the PIDs from the worker processes and the master process in the logs, or is it just showing master processes?

We do have a way to force the agent to register. You mentioned trying different approaches from the documentation - have you tried forcing the registration of the agent? This method allows you to force the agent to fully register with our data collectors before running your tasks to ensure that the data is reported.


#3

Yes, I tried forcing the registration.

If you take a look at the logs I provided, the register is successful.
I think the way pyres works(killing the forked process maybe the reason newrelic agent didn’t send any data.

before the agent is able to report data, the forked process is killed.

about the PIDs, it SEEMS both master and forked process get their own agents registered.


#4

@lucas.stephanou, I took a closer look at pyres, and it looks like you’re correct. Part of the issue we face when attempting to instrument pyres with the Python agent is that when using pyres, as you noted, the worker is killed with os._exit(), which means our final harvest at shutdown doesn’t run.

We currently do not have instrumentation for pyres. But, if we did provide pyres instrumentation, there’s still the downside of forcing an agent registration and shutdown for every job processed. If each job only take 1-3 seconds, the agent overhead of registration/shutdown may be considerable.

If you want to attempt a workaround yourself, you could 1) wrap your job in newrelic.agent.background_task, and 2) call newrelic.agent.shutdown_agent(timeout=<num_seconds>) in the worker process after the task runs, but call it outside of the job function.

https://docs.newrelic.com/docs/agents/python-agent/customization-extension/python-management-api#shutdown_agent

Lastly, I’ll go ahead and file a feature request on your behalf for pyres instrumentation.


#5

Yeah,

I tried some more approaches without a good way to make this work.

I’m currently working to convert jobs from pyres to celery. Pyres didn’t see updates for a long time. Which make more sense.

thanks anyway!


#6

Let us know if you need anything else, @lucas.stephanou ! Always happy to help troubleshoot. :blush:


#7

@Linds Was there ever any support added for Pyres? I am having the same issues as OP, but can’t seem to find any other documentation on this.


#8

@kristofers, the Python agent doesn’t include instrumentation for Pyres. I’d be interested to know have you tried any of the workarounds suggested in this thread?