10X Slowdown After Enabling NewRelic Agent for Node.js

Node Agent Question Template

  • Node Version: 8.11.4
  • What language agent version are you using? 5.6.3
  • Describe what are you seeing? Describe how that differs from you were expecting to see?
    After enabling the Node.js agent on our API we see a 10X slowdown (200ms to 2000 ms) in our API responses. Disabling the agent immediately resolves the issue (environment variable adjustment) so it appears to be directly related to the agent. I’ve attempted to tune the agent to sample less frequently but that has not altered the performance characteristics. Config is an internal store that provides the stated values per environment.

newrelic.js file:

exports.config = {/**
	 * Array of application names.
	 */
	app_name: [config.get("newrelic.appName")], //varies by environment
	/**
	 * Your New Relic license key.
	 */
	license_key: config.get("newrelic.licenseKey"),
	/**
	* Level at which to log. 'trace' is most useful to New Relic when diagnosing
	* issues with the agent, 'info' and higher will impose the least overhead on
	* production applications.
	*/
	logging: {
		level: config.get("newrelic.level") //Value is set to info
  },
	/**
	 * When true, all request headers except for those listed in attributes.exclude
	 * will be captured for all traces, unless otherwise specified in a destination's
	 * attributes include/exclude lists.
	 */
	allow_all_headers: true,
	attributes: {
		/**
		 * Prefix of attributes to exclude from all destinations. Allows * as wildcard
		 * at end.
		 *
		 * NOTE: If excluding headers, they must be in camelCase form to be filtered.
		 *
		 * @env NEW_RELIC_ATTRIBUTES_EXCLUDE
		 */
		exclude: [
			'request.headers.cookie',
			'request.headers.authorization',
			'request.headers.proxyAuthorization',
			'request.headers.setCookie*',
			'response.headers.cookie',
			'response.headers.authorization',
			'response.headers.proxyAuthorization',
			'response.headers.setCookie*'
		]
	},
	transaction_events: { 
		max_samples_per_minute: 10 //Added to attempt to address the performance issue
	}
};

@ross15

Can you send me a permalink to the jump in response time that you’re seeing? To create a permalink to any page within the New Relic user interface, scroll to the bottom and click ‘Permalink’ all the way on the right next to ‘Kiosk Mode.’ This will show us the exact page and time period that you are observing.

Also, can you let me know what node framework you are using (if any)?

I can’t provide a permalink because the difference is newrelic installed vs new relic uninstalled. On one of our routes (and this is true across the board) our response before enabling new relic was 200 ms and then after installing it the time jumped to 2000ms.

We’re utilizing a variety of frameworks but for serving requests we’re using Restify@4.2.0.

Hi @ross15, I assume this increase happened when the App first started reporting, so even if we have the name or ID of the App when it was reporting, we may be able to do some digging in Insights.

In the meantime, it may be worth installing the newest version of the agent and enabling Troubleshooting logs for a few minutes to see if there are any obvious errors.

I’ve enabled trace logging on the application and upgraded to the newest version of the agent. Here’s the requested permalink: https://rpm.newrelic.com/accounts/1301612/applications/171306266?tw[end]=1556230375&tw[start]=1556228575

Some additional context that may or may not be relevant: the application is deployed on heroku and has the add on setup. The add-on attempted to assign a different API key than what we had previously. When utilizing the appropriate API key the app starts showing up as valid in our accounts page but doesn’t receive any data and our application is dramatically slower.

We’re also now seeing max call stack size exceeded errors manifesting in the environment. Disabling newrelic appears to make the issue go away.

Hi @ross15

Thanks for that clarification. I think the best way to move forward in troubleshooting this issue will be to create a support ticket so we can gather these logs and investigate further. I will create the ticket from our end and contact you through there with further instructions.