Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Details for slow transactions that never completed

longtransactions
transactions
feature-idea

#1

Please paste the [permalink][1] to the page in question below:

https://rpm.newrelic.com/accounts/678477/applications/3067859/transactions?show_browser=false&tw[end]=1499106240&tw[start]=1498847040&type=scheduled+job

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

We use the New Relic node.js agent (specifically, newrelic.createBackgroundTransaction() and newrelic.endTransaction()) to capture transactions for scheduled jobs that we run on various schedules, which works great. We can generally get a view of which jobs are slow, and the specific instances of those jobs that are slowest show up in the list of slow Transaction Traces.

However, we have one job in particular that we recently discovered is taking so long that it never actually completes, and newrelic.endTransaction() is never called. The transaction is still in progress when the node process is terminated (generally as a result of deploying a new version of the application each week). This job does not seem to appear at all in the list of transactions. Are incremental traces of transactions which are started but never end sent from the agent to New Relic? If so, is there some way to see these incomplete transactions? If not, do you have any advice for how we can ensure that these transactions are captured in some way even if the node process is terminated before the transaction completes?


New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Hi, @scottgilroy: Can you call newrelic.endTransaction() from a process.on('exit') handler?


#3

@philweber I could try that. Will it be a problem that the call to newrelic.endTransaction() happens outside of the context of the transaction? That is, will the correct transaction be ended? Also, will the newrelic agent correctly persist/upload transactions in a scenario like this where the process is being terminated, despite the typical behavior of only periodically uploading traces?

Here is what our code looks like:

function defineJobs(agenda) {
    // delayed require to avoid problems with app/newrelic.js module configuration file not found
    const newrelic = CONFIG.newrelicKey && require('newrelic');

    _.each(jobs, function(details, key) {
        agenda.define(details.name, function(job, done) {
            let doJob = function doJob() {
                Promise.cast(details.fn(job)).then(() => {
                    done();
                }, done).done();
            };

            if (newrelic) {
                const originalDone = done;
                done = function() {
                    // end the newrelic transaction before telling agenda that the job is done
                    newrelic.endTransaction();
                    originalDone.apply(undefined, arguments);
                };

                // wrap the function, using the name of the job (as opposed to key) to be consistent with logs and db
                doJob = newrelic.createBackgroundTransaction(details.name, 'scheduled job', doJob);
            }

            doJob();
        });
    });
}

#4

@scottgilroy Thanks for posting to the Community. Can you please upgrade your agent version to 2.0.1 (more info here) and try implementing the following code snippet?

function defineJobs(agenda) {
    // delayed require to avoid problems with app/newrelic.js module configuration file not found
    const newrelic = CONFIG.newrelicKey && require('newrelic');
    const transactions = new Set();

    process.on('beforeExit', () => {
      const endings = [];
      for (const tx of transactions) {
        endings.push(new Promise((resolve) => tx.end(resolve)));
      }
      Promise.all(endings).then(() => newrelic.shutdown({collectPendingData: true}));
    })

    _.each(jobs, function(details, key) {
        agenda.define(details.name, function(job, done) {
            let doJob = function doJob() {
                const tx = newrelic ? newrelic.getTransaction() : null;
                if (tx) {
                  transactions.add(tx);
                }
                Promise.cast(details.fn(job)).then(() => {
                    if (tx) {
                      tx.end(() => transactions.remove(tx));
                    }
                    done();
                }, done).done();
            };

            if (newrelic) {
              newrelic.startBackgroundTransaction(details.name, 'scheduled job', doJob);
            } else {
              doJob();
            }
        });
    });
}

This should allow you to instrument the transaction. Let us know if you have any questions.


#5

Thanks @hakimu! I just saw your response. That is very helpful! We haven’t upgraded to the newrelic 2.x agent yet, but I’ll look into this. Does the agent keep track of pending background transactions in some way that would allow me to do iterate through all transactions and end them without the need for manually keeping track of pending transactions? We have other places in our app where we create background transactions, so keeping track of all of them would be non-trivial and error prone. Related question: isn’t this behavior of ending pending transactions on exit something that would be universally useful and could be implemented by the newrelic agent instead of by each application? I’m curious if this is something your team has considered adding as a feature, or might pursue in the future.

Thanks again!


#6

Hey @scottgilroy! What you described above is something that has been prioritized by the dev teams—it’s not on the current roadmap but remains a topic of conversation.

I will pass along your interest in it to our Product Managers! :thumbsup: Thanks for reaching out!


#7

We just ran into a somewhat similar situation where we had threads hanging. In our case it was in java with the thread itself creating a transaction but never finishing the task. Having New Relic notify us when a transaction has been open for more than a set amount of time would have been much better than having to find the issue the hard way. Please let Dev know that there are multiple fans of this idea :grin:


#8

Thanks for letting us know and sharing your use case, @jason.mathison! Be sure and vote above!