NewRelic Agent v1.39.1
- One of our Express app route do open a Mongoose cursor (http://mongoosejs.com/docs/api.html#querycursor_QueryCursor) to fetch documents (might be ~100k), pipe them into a CSV stream then pipe them into Express response.
- This is working super fine with
NEW_RELIC_ENABLED=false, memory goes from 100mo to 130mo, and stays that way during the streaming of the CSV (which can take up to a minute).
- When NewRelic is enabled, the memory goes from 100mo to 500mo (our provider’s, Heroku, limit) in less than 10 seconds (during the CSV is being streamed) and the app then crash.
- We tried disabling instrumentation for that route (
NEW_RELIC_IGNORING_RULES='.*format=csv.*') but it keeps leaking the same
- According to your documentation, there is a known issue with MongoDB cursors (https://docs.newrelic.com/docs/agents/nodejs-agent/troubleshooting/troubleshooting-large-memory-usage-nodejs#mongo), but i’m not so sure it is related since Mongoose handles the cursor closing properly (https://github.com/Automattic/mongoose/blob/4.10.4/lib/querycursor.js), and the memory leak happen as soon as the cursor is being streamed anyway.
- Our only solution right now is to disable NewRelic completely
- Is there a way to fix this issue? or at least to make NewRelic ignore that transaction / Mongoose cursor methods?
New Relic edit
- I want this, too
- I have more info to share (reply below)
- I have a solution for this
We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.