Microsoft provides a number of tools and configuration options for testing and debugging .Net applications. One of these options is to compile an application or module in debug mode. This can be set in Visual Studio or via the compilation element in the app’s config file. While debug mode is valuable when testing and preparing an application or module for release, applications and modules should always be compiled in release mode before being moved to prod.
There is a known performance tax incurred by using debug mode, or modules in debug mode, in any .Net app. Microsoft has some great blog posts that go into detail about the what and whys of this performance impact, here and here. In addition to the basic performance impact of debug mode, instrumenting an app, or modules, in debug mode with the .Net Agent incurres an additional performance impact, typically manifested as higher CPU and/or memory usage.
Simply put, when you are running in debug mode optimizations are disabled and the profiler and agent have to operate in a configuration that they were not designed to work in. Below are the ways the profiler and agent benefit from release optimizations and why there are penalties when using any code manipulating profiler with debug turned on:
Method in-lining - this is where the code of a function is rolled into the caller, making things like property accessors free - big performance gains.
CPU register allocation - this is important to the profiler and thread profiling specifically because method args are stored in CPU registers.
Array index checking elimination - The agent makes heavy use of collections and if this optimization is missing then where we loop through collections there will always be an index check.
Dead code removal - when running in debug all code paths must be evaluated. When optimized some code paths are removed and never considered because they are either always true or always false. Some of the logic in these conditions can be costly even though the contents of the conditional will never be used.
Loop unrolling - in cases where the agent is looping over a very small set of something having debug turned on causes us to incur a branch misprediction penalty. Instead the code will have to go through a series of interpretive analysis when it runs into a conditional.
For these reasons one of the best first steps you can take when investigating performance issues with your app in general and specifically while instrumenting your app with the .Net Agent, is to confirm that the application and all the modules it uses are compiled in release mode. One of the ways we check for this is by gathering a memory dump as it will show if the app is compiled in debug mode as well as listing any modules that are in debug mode.
The Sysinternals tool ProcDump provides a simple way to gather a memory dump and the Microsoft tool DebugDiag can be used to analyze a memory dump. Additionally this MSDN page can help you set the correct setting in Visual Studio for compiling your applications and modules in release or debug mode.