Application slows down considerably after enabling .NET agent

I just installed and enabled NewRelic .Net Agent for our webapplication which uses ASP.Net forms and .Net Framework v4.7. The application is hosted as Azure app service.

I can see the app listed in APM, but the application slows down considerably (at least 10x times slower than usual) and leads to request timeouts. I tried to disable the agent and everything works normal.

I also tried to disable the following via newRelic.config
crossApplication Tracerenabled = false
browserMonitoring autoinstrument= false

But it didn’t help much, the performance is still slow. What else can I try ?

Hello @prakash.r,

Thanks for your post!

I wanted to mention that it is expected that response time can be impacted during the first minute or so the agent is starting up while the agent is cataloging the methods it needs to instrument.

This is due to the nature of the instrumentation process - our profiler (or any code-level profiler, for that matter) forces the application to go through the JIT process which gives our agent the opportunity to insert its instrumentation code.

However I wouldn’t expect the slowness to continue after that initial period, although there is a nominal amount of increased overhead with the agent.

Can you provide a permalink so I can take a look at the app? Only support will be able to view the link.

Thanks!

I have also found that start time within docker doubles also once the agents enabled

Thanks for the detailed info. I aware that the initial run might be slower as the agent is injecting the instrumentation logic. But we do see consistent slowness leading to timeouts after the initial few minutes. Here’s a permalink if you can take a look. Its running in an Azure app service.

Pls note I have disabled the transactiontracer, crossApplicationTracer and browserMonitoring via newrelic.config

Initial application load:
https://rpm.eu.newrelic.com/accounts/2470512/applications/22113483?tw[end]=1574646527&tw[start]=1574644727

When we tried to login to the application.
https://rpm.eu.newrelic.com/accounts/2470512/applications/22113483?tw[end]=1574647176&tw[start]=1574645376

@ntierney May I know if there is any update to the issues I raised ?

Hi @prakash.r

You can also remove all of the instrumentation from the extensions folder however this will remove all instrumentation. If you notice an improvement you can then re add in the xml files one at a time untill you notice an impact on performance again.

This may help you determine which instrumentation is causing the most overhead to your application

Glen

Removed everything from the extensions folder. Application was still slow
But when I checked the logs I see a

NewRelic WARN: The executing user, IIS APPPOOL\xxx, has insufficient permissions to collect Windows Performance Counters.
NewRelic WARN: No GC performance counters were instantiated. GC Metrics will not be captured.

But we did not have any such permission issues when we used Application Insights previously with the same instance.

Hi @prakash.r,

Sorry to hear things are still slow after removing the instrumentation. Regarding that warning you mentioned, it just means that the user under which your app pool runs does not have the permissions required to gather metrics like Garbage Collection. More information here:

I’d like to open a ticket to investigate the performance issue further. Look out for an email shortly, and we’ll circle back here once we’re able to resolve the issue.

Thanks!

Just to circle back after resolving this, when using the NuGet package in this case the agent was being placed in the bin folder of the application.

When the agent files are placed in the bin folder, it causes the app domain to restart (or the app pool to recycle) constantly, and the agent is restarting along with it, causing the agent to always be operating in the more overhead-intensive startup process. This is probably related to logs being written by the agent, and when any files in the bin folder (including subfolders) are modified, it triggers an application domain restart or application pool recycle. This is brought up in the following article:

If you’re not able to use the Azure extension, placing the agent files outside of the bin folder should also resolve this performance issue.