Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Method discovery for custom instrumentation

dotnet
custom_instrumentation
.netagent

#1

.NET Framework
IIS/ASP.NET application
Windows 2012 R2

.NET agent version: 6.18.139.0

The application loads 786 DLLs: Microsoft, 3rd party + custom

Using the following article as guide:

I am configuring <log level=”all” /> in newrelic.config. The NewRelic.Profiler. <pid>.log file is showing entries with “Possibly instrumenting” statements.

BUT - The application is getting YSD from the amount of instrumentation.
YSD when <log level=”finest” />

No YSD when <log level=”debug” />.

Discovery Questions:
(1) How can I configure the method discovery to only include certain modules by a RegEx or wildcard pattern?
(2) How can I configure the method discovery to excluded certain modules by a RegEx or wildcard pattern?
(3) Is there any other way to do method discovery?

Instrumentation Questions
(4) Any existing New Relic tools to take method discovery and automate creating extension .XML files?
(5) The documentation and extension.xsd file only define exactMethodMatcher.
Is it possible to instrument all methods in a class by:
(5a) Ignoring element exactMethodMatcher?
(5b) Use a wildcard for exactMethodMatcher like the following

<?xml version="1.0" encoding="utf-8"?>
<extension xmlns="urn:newrelic-extension">
  <instrumentation>
	 <!-- instrument CustomInstrumentDemo.Controllers.SecondController.SecondExample -->
	<tracerFactory>
	  <match assemblyName="CustomInstrumentDemo" className="CustomInstrumentDemo.Controllers.SecondController">
		<exactMethodMatcher methodName="*" />
	  </match>
	</tracerFactory>
  </instrumentation>
</extension>

#2

Hi @Jeffrey.Hyson,

I’m not sure what “YSD” means in this context, but here are the answers to your explicit questions.

(1) How can I configure the method discovery to only include certain modules by a RegEx or wildcard pattern?

The profiler logs will always log any and every method signature passed to it by the .NET Core runtime when the logging level is set to finest. This is not configurable.

(2) How can I configure the method discovery to excluded certain modules by a RegEx or wildcard pattern?

Same answer as above.

(3) Is there any other way to do method discovery?

The only other method is to inspect the source code of the application you’re trying to monitor itself.

(4) Any existing New Relic tools to take method discovery and automate creating extension .XML files?

New Relic does not maintain any tools for this purpose. I believe a third-party developer created something along those lines called “Pablisimo” and posted it to Github, but I have no information on how it works.

(5) The documentation and extension.xsd file only define exactMethodMatcher.
Is it possible to instrument all methods in a class by:
(5a) Ignoring element exactMethodMatcher?
(5b) Use a wildcard for exactMethodMatcher like the following

No, there is no support for “wildcarding” in the XML files and the exactMethodMatcher element is required. The agent will only match exact method signatures, explicitly defined in the instrumentation files.

Hope this helps.

Don


#3

@dmorris

YSD = Yellow Screen of Death

with “all” or “finest” defined I am unable to start the application for the discovery to capture the methods I would like to instrument.

Sounds like you are saying for
(A) Discovery it is all or nothing without the ability to exclude assemblies.
(B) Instrumentation the product does not make it easy to generate the required configuration data - it’s all manual considering the first problem I just mentioned.


#4

Hi @Jeffrey.Hyson

Yes this is correct

Discovery is all or nothing. it would be a very manual process

YSD thats a new one for me :smiley:


#5

Thanks @Nemo, that’s a shame :frowning:

I’m sure I cannot be the first person looking to have these two points as features in the product. Perhaps you can create a usability feature request and bring it to the attention of your product management team.

Maybe YSOD, not YSD
https://www.google.com/search?q=ysod


#6

@Jeffrey.Hyson -

Although I know app performance can be affected by “All” logs I don’t think I’ve ever seen or heard of a YSOD because of the logging level.

That’s an interesting idea to apply “all” level profiler logging to specific assemblies only. I can certainly make that feature request.

One other thing to consider. For IIS hosted apps, when you turn up the logging level to “All” in the “global” newrelic.config file, it affects both the profiler logs and the agent logs.

If you have an app-local newrelic.config file (one at the same level as web.config) you could try setting that log level to “Info” and the “Global” seeing to “All”. That way the profiler logs will be detailed but the agent logs will be just info. I don’t know if that might prevent the YSOD. Maybe worth a try.


#7

@Tjd

Still get a YSOD.

What should the contents of the app-local newrelic.config file be?
(A) Identical to the global newrelic.config except for the log level
OR
(B) Just the log level information

<configuration>
  <log level="info" />
</configuration>

Was the global/local newrelic.config file a working theory or do you know it should behave this way? If I look at the log files how can I confirm the local/global settings were picked up?

All my tests showed the log level is only picked up by the global newrelic.config file
<-- New logging level set: Info
OR
<-- New logging level set: Trace


#8

Hi @Jeffrey.Hyson,

The local newrelic.config file should have the log level set to info. The global C:\ProgramData\New Relic\.NET agent\newreli.config should have this set to all.

After putting the local newrelic.config in the same directory as the web.config for the application, did you restart the application? This means either performing an iisreset or recycling the application pool that the application is in. The only setting in the newrelic.config that you can change that doesn’t require an application restart is the log level setting. However, in order to get an application to pick up a local newrelic.config in the first place, after it’s first been placed in the local directory, the application must be restarted.

You can tell where the application is getting its newrelic.config settings from by looking in the agent log for the application. You’ll find the logs in the C:\ProgramData\New Relic\.NET agent\Logs directory. If there’s more than one application on the system, open IIS and find the site ID for the application (right-click the site and select Manage Website>Advanced Settings). The agent log for that site will look something like newrelic_agent__LM_W3SVC_X_ROOT, where X is the site ID number. If the application is the child of a root site, find the site ID first, then look for something like newrelic_agent__LM_W3SVC_X_ROOT_ChildApplication, where ChildApplication is the name of the application as it is displayed in IIS. The line you’re interested in is:

NewRelic INFO: Reading configuration from

After this message, the full path to the newrelic.config being used for the application will be recorded. Be sure to find the very last such entry in the log file as that provides this information for the most recent restart of the application. If something has changed between restarts, only the most recent output will be accurate.


#9

@kyle

The application is still generating a YSOD.

I will call the original suggestion from the New Relic team a failed experiment to prevent the YSOD when the global level newrelic.config log level="all" and the application level newrelic.config log level="info".

  • With the YSOD only 767 of the 786 assemblies are loaded.

  • The NewRelic.Profiler.5812.log file is showing 53,970 instances of “Possibly instrumenting:” of which 40,614 are for a NewRelic.*.dll assembly.

  • None of the application’s assemblies are getting profiled because each one is encountering the following error.

    [Error] 2019-06-17 11:27:54 Win32 function call failed. Function: metaDataImport->FindTypeDefByName HRESULT: 0x80131130
    [Error] 2019-06-17 11:27:54 Failed to tokenize method signature: class System.Reflection.Assembly System.CannotUnloadAppDomainException::LoadAssemblyOrThrow(string)
    [Error] 2019-06-17 11:27:54 An exception was thrown while attempting to inject into a module.

Having a feature to only profile certain assemblies IMO would possibly prevent a YSOD; certainly, there is no reason for the New Relic agent to profile itself 40K times.

Three log files are generated
(1) C:\ProgramData\New Relic.NET Agent\Logs\newrelic_agent__LM_W3SVC_1_ROOT_MyApp.log
(2) C:\ProgramData\New Relic.NET Agent\Logs\newrelic_agent_w3wp.log
(3) C:\ProgramData\New Relic.NET Agent\Logs\NewRelic.Profiler.5812.log

The 3rd file NewRelic.Profiler.5812.log is the one where the “Possibly instrumenting:” statements are recorded.

When using only a global newrelic.config log level="all"
(1) has FINEST
(2) has FINEST
(3) has TRACE

When using a global newrelic.config log level="all" and the application level newrelic.config log level="info"
(1) has INFO
(2) has FINEST
(3) has TRACE

The configuration work performed only disabled FINEST in the first log file which has no impact on Agent profiling which seems to be the root cause of YSOD.

How can Profiling be done which will record the desired information and not generate a YSOD?