Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: Troubleshooting XML-based Custom Instrumentation Issues

level-up

#1

So, you added some custom instrumentation for your .NET application with a custom XML file, but you’re not seeing the results you expected. This could mean a lot of things, but the first thing to do is make sure there aren’t any problems with the XML file itself.

If you haven’t created your custom instrumentation file yet, start here.

For the rest of you, let’s get started.

Is your XML in the right directory?

Custom XMLs should be placed in the extensions folder.

  • .NET Framework agents installed on Windows servers use C:\ProgramData\New Relic\.NET Agent\extensions (make sure you do not use the extensions folder in the Program Files path).

  • .NET Core agents, or any agent installed with the Agent.Api NuGet package use the extensions folder inside the directory where the agent was installed.

If you’re not sure if you’re in the right directory, look for existing .xml files in the same directory with names like NewRelic.Providers.Wrapper.*.xml. If you see files like that, you’re in the right place.

Is your XML valid?

The easiest way to make sure your custom instrumentation file is valid (free of structural issues) is to validate it against the supplied extensions.xsd schema provided with the agent. This file is located inside the agent’s extensions folder (the same folder where you place your custom .xml file).

There are many tools for validating an xml against its schema (.xsd). This post isn’t intended to be a tutorial on that. If you don’t have a dedicated tool installed on your machine, there are free “xml to xsd validator” sites online that can help.

If your xml does not validate against the xsd file, it will not be parsed by the agent and your instrumentation will not be applied.

Lets get some logs!

Now we’ll generate some profiler logs that will allow us to see exactly what the agent is doing with your xml file when your application runs.

Open the global newrelic.config file for your agent.

  • Framework agents installed with the msi, you’ll find it at C:\ProgramData\New Relic\.NET Agent\.
  • All other agents (.NET Core or installed via the Agent.Api Nuget package) it will be in the directory where the agent was installed.

If you are using localized config files in the root folder of your app, do not use those ones for this process. Only use the global newrelic.config file.

In that file, find the line

`<log level="info" />`

and change it to

`<log level="finest" />`

Save the file, then restart your application and exercise it in a way that calls the methods you’re trying to instrument in your xml file.

While your application is running, take note of the process ID (PID) your application is executing under (if your application is running in IIS, note the PID of the w3wp.exe process hosting your app).

The logs you generated should be located at:

  • .NET Framework agents - C:\ProgramData\New Relic\.NET Agent\Logs
  • .NET Core agents or agents installed with NuGet - In the logs folder inside the directory where the agent was installed.

Note, if you don’t see logs being generated in that directory, stop and go check out this post.

Get the right log file

Grab the PID your application was running under and look for a log called

NewRelic.Profiler.[PID].log

What to look for in the logs

Is the agent seeing the method being called?

In that log file, search for your method name and you should see a line that fits this pattern…

[Trace] <<DateTime>> Possibly instrumenting: (Module: <<path/to/your/module>>, AppDomain: <your domain>>)[assembly]assembly.class.method(parameters)

If you don’t see that line in the file, your method was not called in the time-frame the logs were collected. You’ll need to go back to the “Lets get some logs!” section and try again.

The important parts of this line are the Possibly instrumenting and the [assembly]assembly.class.method(parameters).

  • Possibly instrumenting means that the method was called and the .NET CLR presented the method signature to the agent for possible instrumentation.
  • [assembly]assembly.class.method(parameters) is the exact method signature that must be present in the XML file for the agent to match your xml to the actual method signature. I do mean exact (case matters here).

In that method signature, compare the following:

  • The portion inside the brackets ([assembly] in our example) should map to the assemblyName= attribute in your xml file.

  • The next portion, up to, but not including the method name (assembly.Class in our example), should map to the className= attribute in your xml file.

  • The next portion, up to, but not including the opening parenthesis (method in our example), should map to the methodName= attribute in your xml file.

  • The parameters inside the parenthesis are not required in the XML unless you are trying to instrument only certain cases of an overloaded method. In that case, map them to the parameters= attribute of the <exactMethodMatcher> element.

If all of those element match exactly between the method signature in the logs and the one in the xml file, you should then see another line shortly after the Possibly instrumenting line which is identical with the exception that “Possibly instrumenting” has been replaced with “Instrumenting method”.

[Trace] <<DateTime>> Instrumenting method: (Module: <<path/to/your/module>>, AppDomain: <your domain>>)[assembly]assembly.class.method(parameters)

If you see the “Instrumenting method” line for your method signature, then you can be sure that your custom XML file is being picked up by the agent and applied to your application.

Some common what-if scenarios

SCENARIO 1: I don’t see my method in the profiler log at all! There is no line that says “Possibly instrumenting” and contains my method’s signature.

This means that the .NET CLR never presented the method to the agent for possible instrumentation.

Check the following:

  • Make sure you’re looking at the right log file. The name of the file should contain the PID of your application’s process at the time the logs were collected.

  • Make sure the method you’re trying to instrument is actually being called at the time the logs are being generated.

  • Make sure the logs were generated during an application restart. This all happens at JIT time, so make sure that’s covered.

SCENARIO 2: In the profiler log I get the line that has “Possibly instrumenting” and my method signature, but I don’t get a second line that says “Instrumenting method”.

“Possibly instrumenting” means the .NET CLR saw the method and asked if the agent wanted to do anything with it. “Instrumenting method” means the agent said “yes, I have instrumentation for that method”. So, this scenario means the agent did not find valid instrumentation for that method.

Check the following:

  • Your XML is in the right directory (see " Is your XML in the right directory?" above).
  • Your XML is valid (see “Is your XML valid?” above).
  • The values in your xml match the method signature the CLR reported exactly.

SCENARIO 3: In the profiler log I get both the “Possibly instrumenting” and “Instrumenting method” lines for my method, but I don’t see anything showing up in the New Relic UI.

This situation could be caused by many things that are bigger than this post’s scope (network connectivity, application naming, application duration, etc), but in the context of troubleshooting your custom instrumentation file, check the following:

  • Make sure you’re not trying to instrument the main() method (or similar). A transaction consists of everything that happens between the time an instrumented method is called and when it returns. Since the main() method stays alive for the entire lifecycle of the application, it never returns and nothing is ever reported to New Relic. Instrument things inside main(), not main itself.

  • Make sure you’re using (or not using) the name attribute correctly in your xml file. name is an attribute of the <tracerFactory> element and tells the agent if it should create a transaction or not when the method is called. If it is set to the value
    "NewRelic.Agent.Core.Tracer.Factories.BackgroundThreadTracerFactory",
    then it will create a new transaction when the method is called (if one does not exist already). If there’s already a transaction running at the time the method is called, it will add the method call as a segment to the existing transaction.
    If the name attribute is omitted all together, the method will only be added to a transaction that already exists at the time the method is called. It will not create a new transaction if one does not already exist.
    All metrics from the agent must be contained within a transaction. If there’s no transaction, the data won’t get reported to your account.

I hope this has been helpful. Happy monitoring!


#2

When the log is set to finest my worker role becomes unstable and it does not work. Anyone experiencing this issue?


#3

Hey @hfutloo - It looks like you’re working directly with the experts in our support team via a private ticket - let us know how that goes for you :smiley: