Process Monitor, or procmon, is a powerful Windows tool that for some reason still isn’t included in Windows. It allows you to inspect processes in real time to see exactly what they are doing which can be very valuable in the world of Microsoft errors like “The data is the error”. The latest version can be found here:
The best part? There is no install! It is a standalone .exe that comes with a readme in a zipfile.
I’ve created a couple different real world problems with the .NET Core Agent to demonstrate how to use this tool to figure out why it is not reporting. These would apply to the .NET Framework Agent as well but for .NET Core you often get no errors or logs to start your troubleshooting so procmon is invaluable. This is an IIS hosted .NET Core process which is often simply “dotnet.exe” but you may also be interested in the app pool process, “w3wp.exe”, or your custom built .exe. Because this is hosted in IIS I am doing multiple IISRESETs to capture the startup of the application which is when it attempts to load New Relic. A console app would need to be restarted itself.
Step 1: Starting procmon for the first time
My biggest complaint is that when you first start procmon it starts recording the entire system. Every action of every process on the system. This can quickly cause it to become unresponsive so IMMEDIATELY after starting it, stop recording. You can use the little spyglass icon to stop/start recording in the top left which I highlighted:
Step 2: Clear results and set a starting filter:
Next, clear the results to remove noise and set a filter to the process you want to monitor. I’ve highlighted the clear (left) and filter (right) icons. The filter is made of three parts. The first dropdown is the column to filter on, then the verb, then the filter term. I recommend starting with “Process Name” because in our case we normally know what process we are interested in:
With the filter in place, click the spyglass icon again to start recording and restart you app. In my case, this meant an
IISRESET followed by hitting the site once in my browser to “wake up” the app in IIS.
Step 3: Filter your results to identify the problem:
You should have a big long list of every action your process took after recording. Be sure to stop recording once you have some data to avoid too much noise.
The “Process Start” operation will show us the environment variables of the process when it started. This is very helpful for us because the .NET Agent relies on a number of environment variables. If you don’t see it, add a filter of the “Operation” type and select “Process Start” from the dropdown:
Double clicking on a line in the results will open details page. Here we can confirm what the environment variables are from the actual process: No more guessing if it is correct!
The “Process” tab is also useful giving details like the user this particular operation executed as. This is important for getting file permissions correct.
Step 4: Use different filters to get different results:
The environment variables look fine so lets add another filter, this time for “PATH” “CONTAINS” “newrelic”. This will show us everything the dotnet.exe did where the path includes the term “newrelic”. This should include our logfiles, .dlls, and more. We can also leave our “Operation” filter in place and just click the checkbox to disable it. This is handy in case we want to go back to it in the future:
Here we found our problem! The process can’t find the NewRelic.Profiler.dll at that location. Lets double check the file system:
And there is our problem. The environment variable had no space in NewRelic but the file system does! Easy mistake to make and hard to spot. Lets fix that issue, clear the results, and get another recording after an IISRESET.
Step 5: Fix a problem, look for more
With the same filters in place we can easily see a restart of our process scoped down to what we are interested in. And here we see a permission problem, another common problem with the .NET Core Agent. My process is running as the “App Pool Identity” which is a member of the IIS\IUSRS group:
Step 6: What success looks like
After giving full control to the IIS\IUSRS group another round of clearing the results, turning recording back on and IISRESET. Here we see a successful output with the process loading the NewRelic.Profiler.dll and writing logfiles. One thing I want to highlight is that the PID of our process is visible and used in the name of the profiler log:
Step 7: Saving results for later
One last thing to point out is saving. You can save your results to share with others. By default this will save only what results you are currently displaying but you can select “All Events” to save even filtered out events. This is useful in case someone wants to see events you have filtered out at the time of saving but makes the save file much larger:
Procmon is a very powerful tool but is very noisy and pretty daunting at first. The trick is to expect some errors. You can even see a Name Not Found message above in the successful results. That doesn’t mean it is causing any issues, just that Windows decides to look there for a config file as well.