[Python] Calling record_custom_metric from crontab not working

I have a django app with crontab to run a function and record a custom metric. Unfortunately, the log never makes it to New Relic. I have a log showing the function is being called, and there are no exceptions caught/logged.

When I SSH into my apps container and run the code directly I see the logs in New Relic after a minute or two. Any thoughts on why within crontab it is not logging as expected?

Hi, @techops23: How long does the function take to run from crontab? New Relic agents send their data once per minute; perhaps the function does not run long enough to give the agent a chance to upload its data. You might try calling newrelic.agent.shutdown_agent() at the end of your function, to tell the agent to send any pending data.

Interesting to know. Let me give this a try and report back, thanks for the reply.

It seems this has not resolved the issue. I can see my logs that the function has been called a few times via the cron timer since the last deploy and no exceptions were caught.

You might try enabling logging for the agent and see if it records any helpful information.

would calling newrelic.agent.initialize() within the cron make a difference?

I don’t know, you can try it!

I have set the log_level to debug and output my logs to a file locally, unfortunately I don’t see much that is useful to me. I am debugging through my crontab function and I can see the app name and license key. I was able to get these values after calling newrelic.agent.initiaize('newrelic.ini') within my cron function and inspecting the value of newrelic.agent.register_application().

I also ran newrelic-admin validate-config ./newrelic.ini and verified that connected. But even when running the crontab function locally I am not seeing those logs appear in NR.

Edit: The app name i provided in the newrelic.ini file was mismatched (bad escapes) and I see the data being sent from my local.

I think I have resolved the issue. I was not placing newrelic.agent.initialize() high enough in my code. I was trying to call initialize within my handler but it needs to be out of that scope right after the imports.

1 Like

I am getting inconsistent results. The first couple attempts sent to NR fine, but subsequent attempts have not. After some experimenting what I’ve noticed is that when running in debug mode and setting a breakpoint within my code it will send to NR. Without me stopping execution, it seems like initialize hasn’t fully finished before the call to record_custom_metric

Is there a way to ensure or wait for the agent to be fully initialized?
Here is the code that is calling to record_custom_metric

import newrelic.agent


class NewRelicApplication:

    # private inner class
    class __NewRelicApplication:
        def __init__(self):
            self.application = newrelic.agent.register_application()

    instance = None

    def __init__(self):
        if not NewRelicApplication.instance:
            NewRelicApplication.instance = NewRelicApplication.__NewRelicApplication()

    @classmethod
    def record_custom_metric(cls, metric, unit):
        application = NewRelicApplication().instance.application
        newrelic.agent.record_custom_metric(metric, unit, application)

Where are you calling newrelic.agent.initialize()?

# various imports
newrelic.agent.initialize()

class Command(BaseCommand):

    def handle(self, *args, **options):
        try:
            with connection.cursor() as cursor:
                cursor.execute((
                    "SELECT <stuff>;"
                ))
                count = cursor.fetchone()
                # This is the class that was posted in my last reply
                NewRelicApplication.record_custom_metric('Custom/mymetrics', count)
                newrelic.agent.shutdown_agent()
        except Exception as e:
            print(f'Error: {e}')

some non-relevant names and values have been changed

According to the docs:

For WSGI and application script files, place the initialize call before all imports, with the exception of the sys import and updates to the sys.path.

Maybe try adding an initialize() call at the top of your NewRelicApplication class as well?

I moved the initialize() call to the top of the handler file as well as the file containing NewRelicApplication but no luck.
I still can only get the log to send if my debug breakpoint stops for a couple seconds before proceeding.

Above the imports?

Correct

import newrelic.agent
newrelic.agent.initialize()

import os  # noqa: E402
import time  # noqa: E402
from django.core.management.base import BaseCommand  # noqa: E402
...

Have you read the documentation on register_application()? There are a lot of details there about when to call it: register_application (Python agent API) | New Relic Documentation.

I have found a consistent solution, although, it is not my preferred choice. I added a time.sleep(10) right before calling my record_custom_metric method.
This even happens when the first 3 lines in my file are

import newrelic.agent
newrelic.agent.initialize()
newrelic.agent.register_application()

If there is another way to wait just long enough for the initialize/register_application to finish, I am all ears. Thanks!

Edit: would register_application (Python agent API) | New Relic Documentation adding a timeout here wait before app execution?

Maybe you can use one of the techniques here to wait until application is not null:

Try it! And please let us know what you learn.