Net Core APM agent intrumentation not working

We’ve recently deployed to Net Core 3.1 (with ASP Net Core) and have lost all APM metrics.

The latest APM Agent is installed and I have read and followed the setup docs a number of times, and yet our sites are not showing up in the APM.
We have an older Net Framework web app showing metrics but none of the Net Core apps are reporting.
Diagnostics look generally ok, although it does report a DotNet requirement check:

Detected failed DotNet Agent requirement checks. Failed checks: DotNet/Requirements/NetTargetAgentVersionValidate

I’ve read the linked document and can see no reason the apps shouldn’t be reporting.
They are Kestrel hosted by IIS.
The latest Agent is installed (and I have restarted IIS)
All docs suggest they should be auto-instrumented but so far nothing.
Their configs all point to the correct APM name
They explicitly enable the agent in code (via the API)
As a check I have also enabled the environment setting in their web config in an attempt to get them to report but again no luck.

What am I missing? It shouldn’t be this difficult?
We’ve had all of these apps reporting before for years in their Framework incarnations. It’s frustrating that this failure is occurring and there is so little reported that I can actually look at to find out what is wrong.

I’ve also checked and we still don’t have browser monitoring either (as per my previous post)
The installations just seem really flaky and unreliable which is not what you want out of your monitoring software

Hi, @mikesmith: I am sure it is frustrating to have things not work as they should, without any clear error message or other indication of what is wrong! If you have not already done so, you might try the troubleshooting steps here:

Hi @philweber thanks so much for your response.
Looks like we have got a little further but I need to confirm something before proceeding.

Firstly it’s frustrating that despite multiple searches I hadn’t seen that document before. The documents available on the main docs site make no mention of these steps and troubleshooting tips. If anything they make it very clear that all one needs to do to instrument .Net Core is “just install the APM Agent” and it should work out of the box and yet in our case it does not. I don’t think we have a particularly special environment. We are Windows + IIS + Net Core 3.1.
Linking to this troubleshooting doc would have probably helped me far sooner.

I have discovered that our web process is not running with all of the environment variables. We have profiler logs (old) and no newrelic_agent logs. Upon checking the process in Process Explorer we have one environment variables (CORECLR_ENABLE_PROFILING=1) but I think this is because I have hardcoded it in the web.config environment variables. I only did this because this is the ONLY hint regarding environment variables in the documentation that I could find.
Further Googling led me to check the environment variables set in the Registry for the WAS process and the W3SVC process. These variables should be used by the processes when they start up and from what I understand the New Relic Agent installer places all of the correct variables in here. Lo and behold those variables are indeed present. But for some unknown reason our processes in IIS are not picking them up.
I am loath to hardcode ALL of these variables into our web config environment variables. We also run as an app pool identity so setting them manually in the Windows UI is also out of the question.

My main concern is really why the APM Agent has not been able to set these variables properly so that our IIS worker services pick them up? Something still seems wrong.
Like I said I can hardcode them into the configs, but because this has never ever been mentioned in the standard docs I feel this shouldn’t be required and I would rather cure the cause of the issue than paper over the cracks.

Any further assistance you might be able to provide would be very much appreciated.

Many thanks,
Mike

Hi, @mikesmith: I have opened a support ticket for you, so our support engineers can help troubleshoot your issue (I am just a trainer, not a support engineer).

I will pass your feedback to our documentation team about linking to the troubleshooting guides from the main docs. Thank you for your feedback, and your patience!

1 Like

Thanks @philweber for your help. I appreciate you opening the ticket and I look forward to hearing back from the support team.

Hope you have a great rest of your day.

Hi @mikesmith - If you have changed your IIS App pool user, as we do, you will need to ensure that the IIS user has access to read those keys as well. I wrote a powershell script to ensure that occurs and I set it as a start up script when the server restarts.

2 Likes

Hi @stefan_garnham thanks for your feedback too.
We deployed our sites with the application pool identities (as I believe that is default) again in the understanding that NR APM would be able to set everything up for them out of the box.
When I checked however I suspect you might be right as there are no explicit permissions set on those keys for the app pools or the IIS_IUSRS group. I’ve tested setting the group permission on the keys and initially it looks like it might work? I’ll be testing it more thoroughly out of hours.
So if that is the case then you will have helped enormously! It also will confirm to us that NR APM is not setting up out of the box as promised. It feels janky having to mess with permissions in the registry just to get this to work!
I’ll update later hopefully.

Update time!
Setting the registry permissions explicitly has not fixed the issue.
In some ways this is good(?) because that means that’s not the fix. However we still have the majority of our asp net core sites not reporting. The processes are not loading the NewRelic.Profiler.dll references.
Please could this issue be expedited with the support team as we are currently in the dark with almost all of our live sites.

Hi, @mikesmith: Have you received the emails related to your support ticket? If not, you might check your spam folder. You may communicate directly with support in the ticket.

Thanks @philweber. I’ve checked my inbox and spam folders and not received any emails from support.

@dmorris, can you help @mikesmith with his .NET Core issue, and figure out why he is not receiving emails from his support ticket? Thank you!

2 Likes

Hi @mikesmith

I sent a response to your support ticket back on Tuesday. I’m not sure why you’re not getting the emails, but you could just log in to support.newrelic.com and check the responses in the support portal directly.

Thanks,

Don

1 Like

The issue here was that IIS on the host servers had not yet been fully reset since installing the agent.

A reset had been performed through the IIS manager UI, but that type of reset is not as complete as a command-line iisreset command.

After issuing that command, the missing variables loaded into the process correctly and the agent started producing logs for Mike’s applications.

3 Likes