Memory leaking only with Node.js agent installed

It’s worth noting that once I disabled SSL I also saw a significant increase in application performance under heavy concurrent loads. Even with 0.12.0, ssl on node appears to still have a lot of overhead and the ssl reporting to new relic can negatively impact your app.

Fortunately we don’t send URL params or custom metrics so turning SSL off is a viable option for us…and we will probably keep it off even if this memory leak is fixed due to the performance increases.

Is there any update on this memory leak?

Turning off clustering, or turning off SSL: neither of these are really options for us. We’re hoping to scale up a production instance of an application that’s been under development but this is really a blocking issue as we need a monitoring solution.

Turning off clustering, or turning off SSL is not an option for us either.
NewReleak…

Hi all - we are still working on this. It is an issue in Node core; however, our engineers take this seriously and are looking into what we can do here.

Essentially this update is the same as the earlier one:

I don’t have an ETA on changes here; this is important and we are actively working on it.

Is the leak still not a priority?

The leak is one of the highest priorities. That said other things do come up and sometimes we have to react to them.

If the problem was in our agent, the fix would likely be very simple, but the problem exists in node core and makes it vastly more difficult to fix for a number of reasons. Part of that is having reproduction cases that meet cores standards. Currently there is a provable leak when you hit google.com but core would like a test that doesn’t involve hitting an external server. This means recording packets and timing and trying to replay as exactly as possible to show the problem using a node based TLS server, (only some TLS servers cause the race conditions)

Another part of it is, that the race condition involves both the tls module and the net module. Neither of which is particularly simple because it is a confluence of several duplex streams (which are really 2 different streams all with their own state). Plus a couple other state orchestration objects. I am working with core folks on this issue.

Just to reiterate: the issue is with node core, the agent is using all the APIs properly and not leaking the memory itself. In fact you can find cases of other libraries running into this bug (such as aws.js) they eventually stopped using custom certs on their outbound, which makes the leak much smaller, but still exists. We are considering going that route but there would still be a leak, just slower, and we have have a few reasons why we ship our own certs instead of trusting what node ships with.

2 Likes

Hey @wraithan. Just wondering if there is an ETA for the fix for this?

Fully appreciate that it’s not an issue with the Node agent itself. However, we can’t deploy New Relic at the moment due to the leak (node 0.10.37)

@briandela_barn2door Since it’s a bug in Node.js core, we can’t provide an ETA on a fix.

However, we did make some changes to the Node.js agent in version 1.18.0 that should mitigate the impact of the Node.js core issue on the agent:

https://docs.newrelic.com/docs/release-notes/agent-release-notes/nodejs-release-notes

We recommend upgrading to this version of the agent when you have the chance.

Upgrading to 1.18.x did not help with our memory leak, it does not stop leaking.

Same problem here. We cannot put this into production until a solution is found, or we will have to investigate an alternative service.

@nzpost @kmiyashiro is it possible that either of you would be willing/able to provide us with a reproduction of your respective situations?

Is there any update on this?

I’m also concerned about it.

Hey all, while we are attempting to work around this issue; ultimately, the memory leak bug is in Node core and the fix needs to come from the Node team themselves.

I know that this is incredibly frustrating and I share your frustration. We want this to work for you without any such memory issues - and without any issues at all!

Unfortunately, this bug does need to be fixed in Node core to alleviate the issue. :neutral_face:

If this is a leak in node core, can someone point me to the corresponding GitHub issue?

Unfortunately, that would take quite a while to figure out since we don’t know which code paths are causing the leak. Our API is pretty large and it would take hours to create an isolated test case.

Hi bc_alex. Here is the GitHub pull request that was mentioned earlier in this thread.

1 Like

Hey, what is the status on getting this merged into the node core? I would like to have newrelic running asap.

Hi @mlyons I’ll see what I can find out; however, it may be up to the Node team at this point…

Hey all, I wanted to follow up since I said I was going to check this out. I don’t have an update at the moment; I wish I had more information and I can only begin to imagine how frustrating this is. We will update if more information becomes available.