No transaction traces for the background tasks in Tornado 4 application

Please paste the permalink to the page in question below:

https://rpm.newrelic.com/accounts/1527797/applications/40831428

Please share your agent version and other relevant versions below:

newrelic==2.82.0.62
python 2.7

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

Hi there,

I’m trying to instrument newrelic agent for a non-web, background-only, tornado 4 service.

The architecture of the service is very simple:
the service is a rabbitmq consumer implemented using pika (https://pika.readthedocs.io/en/0.10.0/examples/tornado_consumer.html).

The on_message handler is wrapped in @gen.coroutine.

My goal is to track each on_message execution as a separate transaction.

I’ve tried the following:

  1. In my newrelic.ini I added the following line
feature_flag = tornado.instrumentation.r3 
  1. Also, disabled motor hook as it was throwing errors
[import-hook:motor]
enabled = false
  1. Enabled debugging
log_file = stderr
log_level = debug
  1. In my code I instrumented newrelic agent as follows
        @gen.coroutine
        def on_message(self, channel, method, props, message):
            """Process a task."""
            application = newrelic.agent.application()
            if getattr(props, 'content_type', None) == 'application/json':
                try:
                    message = json.loads(message)
                except ValueError:
                    LOGGER.warn('Invalid message format: %r', message)

            with newrelic.agent.BackgroundTask(application, name=message['method'],
                                               group='Task'):
                try:
                    self.count_receive(message, props)
                    yield self.process(message, props)
                except HttpAccessTokenRefreshError:
                    # Client has revoked his token, in that case remove the task from queue
                    self.statsd.get_counter('exception.revoked.count').increment()

                except Exception as exc:  # noqa
                    self.statsd.get_counter('exceptions.count').increment()
                    self.log_exception(exc)
                    self.retry(message, method, props)
                else:
                    self.channel.basic_ack(method.delivery_tag)
  1. I run the service using the newrelic-admin cli tool like this
    NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-python -m service
  2. I can see the basic stats reported to the newrelic dashboard (host CPU/Memory usage) but not the transaction traces. There are no errors in the log output

Here is some more technical information:

  1. newrelic agent logs https://gist.github.com/vood/8acc4ef119e2df44d333fd5012ff834a
  2. pip freeze output https://gist.github.com/vood/48a46610a24a5a7e618102214872a88e
  3. python version 2.7.13

Any help appreciated!

Per the documentation on the BackgroundTask generator (linked below), it appears you are missing the register_application call. If using the generator, the register_application code is required in order to ensure the agent is started up.

https://docs.newrelic.com/docs/agents/python-agent/supported-features/python-background-tasks#registration

This might look like the following pseudo code. Note the application = newrelic.agent.register_application(timeout=10) addition.

import newrelic.agent

import tornado.ioloop
from tornado import gen

@gen.coroutine
def run_all():
    ...code...

if __name__=='__main__':
    application = newrelic.agent.register_application(timeout=10)
    with newrelic.agent.BackgroundTask(application, 'Hello World'):
        tornado.ioloop.IOLoop.current().run_sync(run_all)

Give that a try and let us know if things still are not reporting as expected.

Thanks for the reply. I’ll give it a try and will let you know if it helped.