Relic Solution: Troubleshooting Attribute-based Custom Instrumentation Issues

So, you added some custom instrumentation to your .NET application with in-code attributes, but you’re not seeing the results you expected. This could mean a lot of things, but the first thing to do is make sure the agent is instrumenting the code as expected.

If you haven’t added in-code attributes to your application yet, start here.

For the rest of you, let’s get started.

Lets get some logs!

First, we’ll generate some profiler logs that will allow us to see exactly what the agent is doing when the application runs.

Open the global newrelic.config file for your agent.

  • Framework agents installed with the msi, you’ll find it at C:\ProgramData\New Relic\.NET Agent\.
  • All other agents (.NET Core or agents installed via the Agent.Api Nuget package) it will be in the directory where the agent was installed.

If you are using localized config files in the root folder of your app, do not use those ones for this process. Only use the global newrelic.config file.

In that file, find the line

`<log level="info" />`

and change it to

`<log level="finest" />`

Save the file, then restart your application and exercise it in a way that calls the methods you’re trying to instrument in your xml file.

While your application is running, take note of the process ID (PID) your application is executing under (if your application is running in IIS, note the PID of the w3wp.exe process hosting your app).

The logs you generated should be located at:

  • .NET Framework agents - C:\ProgramData\New Relic\.NET Agent\Logs
  • .NET Core agents or agents installed with NuGet - In the logs folder inside the directory where the agent was installed.

Note, if you don’t see logs being generated in that directory, stop and go check out this post.

Get the right log file

Grab the PID your application was running under and look for a log called

NewRelic.Profiler.[PID].log

What to look for in the logs

In that log file, search for your method name and you should see a line that fits this pattern…

[Trace] <<DateTime>> Possibly instrumenting: (Module: <<path/to/your/module>>, AppDomain: <your domain>>)[assembly]assembly.class.method(parameters)

If you don’t see that line in the file, your method was not called in the time-frame the logs were collected. You’ll need to go back to the “Lets get some logs!” section and try again.

The important parts of this line are the Possibly instrumenting and the [assembly]assembly.class.method(parameters).

  • Possibly instrumenting means that the method was called and the .NET CLR presented the method signature to the agent for possible instrumentation.
  • [assembly]assembly.class.method(parameters) is the method signature of your instrumented method

If your attribute instrumentation is correctly in place, you should also see another line, soon after the first which is identical except that “Possibly instrumenting” has been replaced with “Instrumenting method”.

[Trace] <<DateTime>> Instrumenting method: (Module: <<path/to/your/module>>, AppDomain: <your domain>>)[assembly]assembly.class.method(parameters)

If you see both of those lines, the agent is seeing your instrumentation and applying it.

Some common what-if scenarios

SCENARIO 1: I don’t see my method in the profiler log at all! There is no line that says “Possibly instrumenting” and contains my method’s signature.

This means that the .NET CLR never presented the method to the agent for possible instrumentation.

Check the following:

  • Make sure you’re looking at the right log file. The name of the file should contain the PID of your application’s process at the time the logs were collected.

  • Make sure the method you’re trying to instrument is actually being called at the time the logs are being generated.

  • Make sure the logs were generated during an application restart. This all happens at JIT time, so make sure that’s covered.

SCENARIO 2: In the profiler log I get the line that has “Possibly instrumenting” and my method signature, but I don’t get a second line that says “Instrumenting method”.

“Possibly instrumenting” means the .NET CLR saw the method and asked if the agent wanted to do anything with it. “Instrumenting method” means the agent said “yes, I have instrumentation for that method”. So, this scenario means the agent did not find valid instrumentation for that method.

  • Check to be sure your code actually contains the attribute instrumentation, as outlined in this documentation.

SCENARIO 3: In the profiler log I get both the “Possibly instrumenting” and “Instrumenting method” lines for my method, but I don’t see anything showing up in the New Relic UI.

This situation could be caused by many things that are bigger than this post’s scope (network connectivity, application naming, application duration, etc), but in the context of troubleshooting your attribute instrumentation, check the following:

  • Make sure you’re not trying to instrument the main() method (or similar). A transaction consists of everything that happens between the time an instrumented method is called and when it returns. Since the main() method stays alive for the entire lifecycle of the application, it never returns and nothing is ever reported to New Relic. Instrument things inside main(), not main itself.

  • Make sure you’re using the right attribute ([Transaction] vs [Trace]). The [Transaction] will create a new transaction when the method is called (if one does not exist already). If there’s already a transaction running at the time the method is called, it will add the method call as a segment to the existing transaction. The [Trace] attribute will only add the method call to an existing transaction. It will not create a transaction if one does not already exist.
    All metrics from the agent must be contained within a transaction. If there’s no transaction, the data won’t get reported to your account.

I hope this has been helpful. Happy monitoring!

5 Likes