Memory leaking only with Node.js agent installed

Moderator Edit - Please check out this thread for more information!


Currently having exactly the same problem. Spent the better part of 2 days trying to debug what I thought was an application level memory leak and seems that by removing require('newrelic') memory usage stabilises.

Using Ubuntu 14.04, pm2 and newrelic 1.14.5

Would be great to get this sorted.

1 Like

Hi @imjoshholloway - our Node agent engineers are actively working on this issue, and are always looking for example applications experiencing a memory leak with the Node agent included that might help reveal the parameters of the issue. Is that something you would be willing to share with us? (If you’d like to share it but need to do so privately, we can create a ticket for you in our system.)

Hi @alexis I’m willing to help in anyway I can. If you want to create a support ticket so that we can discuss what you guys need that would be great as the application isn’t publicly available as such (all internal tools).

Hi @imjoshholloway, thanks for being willing to help out. For your privacy I’ve created a ticket for your in our system where you can provide more information. You should receive an email shortly notifying you of the creation of the ticket. If you don’t see the email, please check your spam folder first, then let us know.

Just checking if there has been any movement on this. I have the same problem running a clustered Node app using newrelic 1.14.5 and Node 0.10.28.

Conditionally requiring the newrelic module only for workers in the cluster as suggested in Node.js Agent Memory Leak did not work for me. The only thing that does seem to alleviate the problem is disabling SSL as suggested in Node.js Agent Memory Leak.

I also have heap dumps confirming that there are a number of slabBuffers stemming from https connections not getting GC’ed when SSL is on. Let me know if this might help you find a solution to the problem.

On a related note, have you looked into merging https://github.com/newrelic/yakaa/pull/2 for the keep alive agent? I have stepped through the code and confirmed that the author of the PR is right, there is no property named ‘destroyed’ in the socket object in Node versions <= 0.10.28. The property to check for is named ‘_destroyed’. This does not have any effect in the memory issue with SSL though.

Are you on an AWS Micro instance by chance?

I’ve found that the NPM agent requires a certain amount of ram to run/profile correctly. If you have a small amount of RAM this will appear to be a memory leak.

As soon as I upgraded to a T2.Medium instance my “leak” went away…

EDIT: Here’s my production server ram running express3:

1 Like

My app is also hosted in AWS but in a cc2.8xlarge instance with 60GB of RAM and running Express 4.10.7.

Hi @devnull, have you tried some of the additional suggestions here? https://docs.newrelic.com/docs/agents/nodejs-agent/troubleshooting/large-memory-usage I saw that turning off SSL helped; how about reducing the TLS slab buffer size?

Hi,
Is there an estimation for a fix on this issue?
I’m also experiencing this memory leak.
I use a node.js cluster deployed on heroku with latest newrelic agent (1.14.5)

I prefer not turning off SSL…

We are seeing this behavior as well. When comparing one server with the agent enabled to another with the agent disabled, the monitored server’s memory use is growing by 500mb/hour more. On our memory usage graph it is clear that the slope is much greater when NR monitoring is enabled. We are watching memory levels and may need to discontinue monitoring due to this leak.

Has the New Relic team been able to reproduce this?

Hi @todd_saylor, it looks like you’ve created a ticket in our system and we’ve been working with you there. We’ll continue to work with you through that medium so that we can exchange detailed logs and code information.

So I keep chasing these tickets trying to find what’s going on with the NR memory leak…
Where can I see the status of this issue, or should I open a ticket myself for the same thing?
Thanks

I am experiencing the same issue. Idle running on a Heroku 1X instance the memory usage will grow steadily until the memory max is hit. Turning off SSL is not acceptable as this is a production server.

Our developers are investigating this memory leak as a top priority. We’ve identified a bug in Node.js core which has been one of the contributors to the memory leak issue in the agent. You can see our pull request on this bug here: https://github.com/joyent/node/pull/9064/files. We’re continuing to research other contributing factors.

Any updates on this? I have a production setup for a socket.io server and I want to track my socket calls.

After adding in the node newrelic agent, I see the memory climbing steadily. As shown in the charts attached, after commenting out require('newrelic') on the commit 8371d, memory stabilizes.

I can confirm this issue still exists and since removing SSL is not an option for us, removing NewRelic was the only option. We’d love to continue using NewRelic with our Node app but we cannot while this problem exists.

In our case, we were able to determine that the specific version of memcached libraries we were using (0.2.8) contained a memory leak. Simply downgrading those libraries one version made a significant difference, to the point where it isn’t even clear that there is a problem. There may still be memory leaks but the scope is nowhere near what we were facing prior to this discovery. This is no longer blocking us.

Since New Relic resources were also attached to those leaked objects, when monitoring was on we experienced a noticeably larger leak. That is to say, enabling NR monitoring magnified the existing problem.

We initially used this approach to help identify the leaking objects: http://strongloop.com/strongblog/how-to-heap-snapshots/

1 Like

So was there a memory leak on node newrelic agent? Or was it just amplifying the leaks on application code?

In our case it was the memcached libraries that were leaking, and the NR agent was amplifying that. This might be the case at other sites as well – perhaps not our specific memcached leak, but some other leak which is also being amplified by NR (since they will be attaching objects wherever network calls are made).

2 Likes

The memory leak in its current state is believed to be an issue node core. The problem is some OpenSSL (and node) objects are being leaked on outbound HTTPS requests. This is provable against google.com (making it not a misconfiguration of newrelic servers). The problem exists always with HTTPS, but is amplified with the agent because we use custom certs on outbound connections back home and those are being leaked too.

I got pulled into some other things that needed to be released but should be back on the memory leak hunt soon. Once I’ve developed a full fix for it in core, I’ll also use that patch in our outbound connections to newrelic servers. The trick is, we need to know exactly what the patch looks like in core so we can reliably detect if we need to inject it or not for our connections, which means I can’t really apply any fix in the agent until something has been settled on for core.

1 Like