Building Custom Instrumentation Tracer Factories From .NET Agent Log Files

This post is intended to show you how to extract values for your custom instrumentation files from the .NET agent’s logs. It is not intended to be a “how to” on custom instrumentation. For more details on custom instrumentation, please see the New Relic documentation. If you are trying to add custom instrumentation to a non-IIS application (like a console app or a background process) there is also this forum post that might help you out.

If you’re having problems setting up your assembly names, class names, method names, or method parameters in your custom instrumentation files, there is a tried and true trick for gathering the exact information you need using the .NET agent profiler logs.

Essentially what you’ll do is have the agent create a very detailed set of logs that report every method it sees, regardless of whether it instruments those methods or not. Then you’ll search those logs for the method you want to instrument and pull out all of the required information for your custom instrumentation. This ensures that what you put in your custom instrumentation file exactly matches what the agent is seeing when deciding if it should instrument a given method.


To follow this guide you’ll need:

Thing You Should Know

  • In a standard installation, logs are found in %ALLUSERSPROFILE%\New Relic.NET Agent\Logs

  • The newrelic.config file is located either in the root folder of your application, or at %ALLUSERSPROFILE%\New Relic.NET Agent. If the file exists in your application’s root folder, use that one.

  • Custom instrumentation files are created by you and placed in C:\ProgramData\New Relic.NET Agent\Extensions. Use this documentation to create your custom instrumentation file.

Warning: Don’t leave your logging level set to “all” for longer than is necessary in order to gather the information you need. “All” level logs are very verbose and can fill a lot of space very quickly. Don’t forget to set them back to “info” when you’re done.

TIP: If you have a lot of files in your log folder, it may help to move them to a different folder or sub-folder while performing this task, just to de-clutter the folder and give you fewer files to sort through.

Generate the logs you’ll need

Only methods that are actually instrumented by the agent will be logged when using the default logging level. To see all of the methods that the agent considers, you need to set your agent’s logging level to all.

Open the newrelic.config file for your application and look for the line that says:

<log level=”info” />

And change it to:

<log level=”all” />

Save and close the config file.

The profiler will only see your application’s methods right before they are JIT compiled at startup, so restart your application’s host process (IIS, WCF, etc) in order to force the agent to profile the application.

Next, exercise your application in a way that will make it call the methods you want to instrument so that they’ll be seen by the profiler.

After you’re done, change the logging level back to “info” in your newrelic.config file and restart your host process again.

Find the Process ID (PID) of your application

Getting the Process ID (PID) of your application is pretty simple if you have access to the machine your process is running on. In this case you can just open a command prompt and run the tasklist command (while the application is running). This should provide you with a list of running processes and their associated PIDs.

If you can’t gather the PID from the command line, you can also find it in the agent log for your application. Just open the log file associated with your application in a text editor and search for the most recent line that looks like the following.

2018-07-18 19:12:13,301 NewRelic INFO: Your New Relic Application Name(s): My Application

Once you find that line, look just before it and you should see a line that looks like this:

2018-07-18 19:12:11,269 NewRelic INFO: The New Relic .NET Agent v<agentVersion> started (pid 1152) for virtual path '/MyPath'

In this line you’ll find the PID for your application’s process (in this example, 1152).

Find the profiler log for your application and extract the data you need.

Now that you have all level logs and the PID of your application, you are ready to gather the information you need for your custom instrumentation file from the logs.

Find the profiler log for your application by looking for the file named:


where <PID> equals the PID for your application. This log contains entries for, among other things, the methods you want to implement in your application.

Open this log and look for lines like the following. If you know the names of the methods you want to instrument, just search for the method name, otherwise, you can look for the string “Possibly instrumenting”.

[Trace] 2018-04-25 22:22:23 Possibly instrumenting: (Module: C:\Program Files\New Relic\.NET Agent\NewRelic.Agent.Core.dll, AppDomain: /LM/W3SVC/1/ROOT/insightstesting-1-130990117326409037)[NewRelic.Agent.Core]Newtonsoft.Json.Serialization.JsonPropertyCollection.TryGetValue(System.String,Newtonsoft.Json.Serialization.JsonProperty&)

This entry may look daunting at first glance, but getting the info you need from it is actually pretty simple.

You can ignore everything in the line up to the square brackets ( [ ] ).

  • Inside the square brackets you will find the assembly name for your custom instrumentation file (in this example, NewRelic.Agent.Core). Paste this into the assemblyName attribute of your Tracer Factory in your custom instrumentation file.

  • The next portion, up to the opening parenthesis, is the fully-qualified path to the method. Everything up to the last dot preceding the parenthesis (Newtonsoft.Json.Serialization.JsonPropertyCollection) is the value you place in the className attribute of your Tracer Factory.

  • The section between the last dot and the opening parenthesis is the method name (TryGetValue). Paste this into the methodName attribute of your Tracer Factory.

  • Inside the parenthesis are the optional method parameters (System.String,Newtonsoft.Json.Serialization.JsonProperty&) that can be used in your Tracer Factory if you wish, but are not required.

Putting it all together

From the information you gathered, you can now construct the Tracer Factory for your custom instrumentation file. A completed Tracer Factory from the example above would look something like this:

<tracerFactory metricName="Custom/MyTransaction">
  <match assemblyName="NewRelic.Agent.Core" className="Newtonsoft.Json.Serialization.JsonPropertyCollection">
    <exactMethodMatcher methodName="TryGetValue" />        

You’re Done!

There you have it: a fully functioning Tracer Factory that can be used to instrument the chosen method. Paste your Tracer Factory into its proper place in your custom instrumentation file and the .NET agent will instrument this method after your next host process restart.

Note that instrumented methods must be called from within a transaction in order to be passed to the New Relic servers. This is outside the scope of this post, but you should keep it in mind when instrumenting new methods. Custom transaction documentation can be found here if you need it.

For more information on creating custom instrumentation files, please see the New Relic documentation.

Hope this helped!


A post was merged into an existing topic: Relic Solutions: Building Custom Instrumentation Tracer Factories From .NET Agent Log Files