Over the weekend, we’ve started seeing very high memory usage of our PHP applications, particularly in long-running PHP non-web transactions. After a while of investigation, we’ve narrowed the problem down to the New Relic PHP agent update. If we pass the PHP flag to disable it with PHP’s
-d option then the leaks vanish.
We’ve tried different New Relic settings, thinking the culprit might be transaction trace data, but only disabling the agent entirely seems to mitigate the leaks. Downgrading to the previous 8.x version fixes the problem for us.
To paint a picture of the severity of these leaks, I’ve seen a process that normally uses like 100-120MB use up 1.5 gigabytes of memory. Another process used up all 8GB of RAM on the server and 2GB of swap space, forcing me to stop the VM and double its memory to prevent further production incidents.
Other folks have mentioned segfaults, here we’ve got memory leaks - perhaps writing sections of the agent in Rust might be something the team could look into.
This is actually not the first time we’ve seen issues like these in the New Relic PHP agent, and as a result, we have become hesitant to deploy NR PHP agent package updates in production. I have found the NR team very responsive to fixing things like these in the past, after submitting a support ticket, but I no longer have the means to do so.
So YMMV but if anyone has the latest PHP agent installed, and is running background tasks, my tip would be to check for increased memory usage on your servers.