Relic Solution: Naming web transactions in Java (or "how to self-mitigate Java MGIs")

Naming web transactions in Java (or “how to self-mitigate Java MGIs”)

New Relic has worked hard to provide a Java monitoring solution that works out of the box. In most supported frameworks we are able to pick up web transaction, datastore and external request data with little to no configuration, just install our agent, sit back and watch the metrics roll in. However, there are cases where the agent isn’t able to determine a good, meaningful name to assign to web requests. Anyone who has dealt with a metric grouping issue in WebTransaction namespace is probably familiar with this issue.

For those who aren’t, I’m going to start by defining a few useful terms, and by providing a bit of background as to how the Java agent names web transactions.

Important terminology:


The prefix assigned to an APM metric. Every web transaction in APM is prefixed by “WebTransaction/”, followed by a category that indicates what gave the transaction its name. For example, if you’re using Servlets, you’ll likely see metrics prefaced with “WebTransaction/Servlet/”, and if you’re using Spring you’ll likely see metrics prefaced with “WebTransaction/SpringController/”. Non-web metrics also have a prefix, for example Datastore requests are prefaced with “Datastore/” and External requests have the prefix “External/” or “ExternalTransaction/”.

Transaction start point

The method that is marked as the beginning of a web request or background task. Also sometimes called the “Dispatcher”, as in HttpDispatcher, or the method that handles incoming web requests. Most APM namespaces other than “WebTransaction/” or “OtherTransaction/” (for background tasks) are ‘scoped’ to these namespaces. In other words, if you don’t have a transaction start point, you also can’t pick up metrics that are scoped to the web transaction, like Datastore or External requests.

Scoped vs. Unscoped metrics

When APM reports a datastore call, it actually reports two metrics. One is the datastore call scoped to a specific web transaction, and the other is the same datastore call, but across all web transactions. The same is true with external requests, or any other metric that appears on your APM transactions’ breakdown tables. To understand this, imagine two different pages which both reach out to the same datastore. Maybe they both access the same table in the same way, so the database call itself looks identical. In APM, you’ll wind up with 3 metrics. The first shows the performance of that datastore call for the first page, the second shows the performance of that database call for the second page, and the third shows the performance of that datastore call across all pages. The first two are called ‘scoped’ metrics because they are scoped to a specific endpoint. The third is called an ‘unscoped’ metric, because it is the aggregate of all the scoped metrics.

Metric grouping issue (MGI)

This page explains what an MGI is better than I can:

A web transaction’s many names.

Now for some background on how the Java agent names web requests.

Possibly foolish assumptions

For this example, I’m going to assume a few things.

  1. I’m assuming you’re using a supported application server (we’ll say Tomcat), so that the agent recognizes your HttpDispatcher as a transaction start point.
  2. I’m also assuming you’re using Servlets, that you’re serving the page source to the client as a JSP, and that you’re using JaxRS to route specific URLs to specific method handlers. I chose this particular combination of frameworks because it’s one where the agent is almost guaranteed to give good, meaningful transaction names out of the box.

In this example, when a web request first hits the JVM, it is handled by your HttpDispatcher, which passes an HttpRequest object. This contains the page URI, and the agent uses this to name the incoming request. However, almost immediately that request is going to pass through a series of servlets / servlet filters, and each time it hits one of these, the agent is going to try to rename the request to match the servlet’s ‘doGet’ method. At some point in the request, a JSP is going to be loaded, either a static page or a series of JSP templates (header, footer, etc.), and the agent will attempt to rename the request based on the JSP name.

Finally, at some point in the request we’ll hit your JaxRS framework layer. JaxRS uses @Path annotation to match a specific URL or range of URLs to a specific class and method. Our agent will use the content of the @Path annotation to rename the web request.

Don’t worry if you’re lost at this point, I’m about to provide a concrete example of this in action. For now, just know that there is an order of precedence to these names. A Servlet or Servlet Filter name overrides a URI name. A JSP name takes precedence over a Servlet name, and a Framework name (like JaxRS) takes precedence over a JSP name. This hierarchy is explained in detail in this doc:

Transaction naming in action

To see this in action, here’s a simple application that uses JaxRS to route web requests:

This application does one thing – it matches requests against specific URLs to specific methods. If I spin this app up locally, and ping the ‘http://localhost:8080/RESTfulExample/users/12345’ endpoint, the request is handled by the ‘getUserById’ method. If I ping ‘localhost:8080/RESTfulExample/users/books/999’, the request is handled by the ‘getUserBookByISBN’ method. This bit of magic is accomplished by the @Path annotation:

public class UserRestService {

	@Path("{id : \\d+}") //support digit only
	public Response getUserById(@PathParam("id") String id) {

	   return Response.status(200).entity("getUserById is called, id : " + id).build();


@Path("{id : \d+}")’ is regex – ‘\d’ stands for ‘numeric digit’, and the ‘+’ means ‘one or more’, so any numeric string prefaced with /users/ will hit this method.

I have this app running locally. When I ping the following URLs, I see the following transactions in APM:

  • /RESTfulExample/users : 31%20PM
  • /RESTfulExample/users/vip : 50%20PM
  • /RESTfulExample/users/12345 : 23%20PM
  • /RESTfulExample/users/abcdefg : 07%20PM
  • /RESTfulExample/users/username/a9 : 15%20PM
  • /RESTfulExample/users/books/999 : 28%20PM

Here’s a full-page screenshot:

As you can see, whatever appears in the @Path annotation also appears in the APM Web Transaction name.

If you look in the agent logs, you can see the request being renamed.

Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINEST: Request initialized: /RESTfulExample/users/123456
Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINER: Setting transaction name using filter name: Tomcat WebSocket (JSR356) Filter
Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "Tomcat WebSocket (JSR356) Filter" for transaction com.newrelic.agent.Transaction@41d62d9e using LEGACY scheme
Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINER: Setting transaction name using servlet name: resteasy-servlet
Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "resteasy-servlet" for transaction com.newrelic.agent.Transaction@41d62d9e using LEGACY scheme
Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "/users/{id : \d+} (GET)" for transaction com.newrelic.agent.Transaction@41d62d9e using LEGACY scheme
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINER: Setting transaction name using filter name: Tomcat WebSocket (JSR356) Filter
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "Tomcat WebSocket (JSR356) Filter" for transaction com.newrelic.agent.Transaction@34e63d17 using LEGACY scheme
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINER: Setting transaction name using servlet name: resteasy-servlet
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "resteasy-servlet" for transaction com.newrelic.agent.Transaction@34e63d17 using LEGACY scheme
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "/users/{id : \d+} (GET)" for transaction com.newrelic.agent.Transaction@34e63d17 using LEGACY scheme
Dec 27, 2018 19:42:50 -0800 [54712 42] com.newrelic FINER: Setting transaction name using filter name: Tomcat WebSocket (JSR356) Filter
Dec 27, 2018 19:42:50 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "Tomcat WebSocket (JSR356) Filter" for transaction com.newrelic.agent.Transaction@313dd1be using LEGACY scheme
Dec 27, 2018 19:42:50 -0800 [54712 42] com.newrelic FINER: Setting transaction name using servlet name: resteasy-servlet
Dec 27, 2018 19:42:50 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "resteasy-servlet" for transaction com.newrelic.agent.Transaction@313dd1be using LEGACY scheme
Dec 27, 2018 19:42:50 -0800 [54712 42] com.newrelic FINEST: Setting transaction name to "/users/{id : \d+} (GET)" for transaction com.newrelic.agent.Transaction@313dd1be using LEGACY scheme
Dec 27, 2018 19:42:49 -0800 [54712 42] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@34e63d17 finished 2ms /RESTfulExample/users/12345

Occasionally you’ll also see this:

Dec 27, 2018 19:42:48 -0800 [54712 42] com.newrelic FINEST: Not setting the transaction name to  "/users/{id : \d+} (GET)" for transaction com.newrelic.agent.Transaction@41d62d9e using LEGACY scheme: a higher priority name is already in place. Current transaction name is users/{id : \d+} (GET)

which indicates that there’s already a name of equal or higher priority set (in this case, the names are identical, but you’ll see similar behavior with different names if your request hits multiple differently-named servlets or JSPs).

So you can see that in ideal situations, the agent watches as a request hits different aspects of your code base that it recognizes, and tries to improve the transaction name at each point. The goal here is to create a single web transaction per code path, so that you can clearly see the performance of different code paths. This is an example of how things work when everything goes according to plan.


The best laid plans (or “what could go wrong?”)

Now, let’s take a look at what can go wrong. In the example above, I’m using multiple framework elements that the agent recognizes out of the box (Tomcat, Servlets, JaxRS with @Path annotation). But what if you aren’t using a supported framework? What if you’ve written your own custom code? In these cases, the agent may not be able to assign a meaningful framework name to your web transactions.

We can simulate this behavior by disabling different instrumentation packages in the agent. Let’s pretend for a moment that the agent doesn’t support JaxRS. If I pass the following in my newrelic.yml file:

  common: &default_settings
        enabled: false

I’ve effectively disabled our JaxRS instrumentation, so now the agent doesn’t know about @Path annotation. If I repeat my test, pinging the six endpoints I used in the first example, I see this in APM:

What happened? My application is routing all of my web requests through a single servlet. Without the JaxRS naming, instead of each code path getting a different name, all of my requests are lumped together under a generic servlet name. I think you’ll agree that this is less than useful. I can’t tell which endpoint is performing fast, and which is slow.

We can take this a step further. If I disable automatic transaction naming entirely, thus:

    enable_auto_transaction_naming: false

then the agent will use the page URI to name all requests.


You may be thinking at this point “Ah! URI naming, that’s what I want. One name per URL!” If so, read to the end of this post. You’re about to see why we don’t set automatic transaction naming to false by default. It can cause more problems than it’s worth. URI naming is very likely to create metric grouping issues

That means if I ping the following URIs:


each one shows up as a separate web transaction.

This is a metric grouping issue. Now the number of web transactions equals the number of users. There’s a limit on the number of unique metric names in APM, capped at 300,000 metrics per application. It’s easy to imagine a large customer base including more than 300,000 unique user names.

More importantly, now your data is fragmented. A single code path is being displayed as thousands of different transactions, each with a slightly different name. It is no longer easy to tell which code path is performing slowest.

So what do you do if you find yourself in this situation? The simplest solution is to implement custom names. There are four ways to overwrite the transaction name set by the Java agent, and all four take priority over framework naming. I am going to show you how to use all four.

1 Like

XML instrumentation

XML instrumentation is basically just a list of classes and methods, which extends the list of classes and methods the agent recognizes out of the box. The big advantage of XML instrumentation is that it can be implemented without changing any code. The downside, when it comes to naming, is that Java’s XML instrumentation doesn’t allow you to set the transaction name to whatever you want. Instead, it allows you to set the transaction name to the name of the targeted class and method.

To name a transaction, two things are needed.

  1. You must tell the agent to rename the transaction. This is done by including the <nameTransaction/> node within the <pointcut> node of the XML file.
  2. You must know the full package / class / method name, as it appears at runtime, and include this in the <className> and <method> nodes of the XML pointcut.

And here we run into a common snag – what if I don’t know my class and method names? Maybe this isn’t our code, maybe I’m used to a front-end URI view of my application and the whole reason I need New Relic is to gain visibility into what’s happening under the hood. Luckily, we can use APM to view this information.

For our example, we’ll use the getUser method. Our full package name is shown here:

What if I don't know my package name?

Since we’re renaming an existing endpoint, and JaxRS is a recognized framework, the method I want to use to rename the request is already shown as a scoped metric on the /users (GET) transaction. This would not be the case in an entirely unsupported framework, in which case I would probably use our Thread Profiler or X-Ray Session features to expose this data, or simply go and look at the code base (assuming I have access). The nice thing about our thread profiler is that it doesn’t rely on the agent’s instrumentation packages. Instead, it polls java.lang.Thread to grab a snapshot of exactly what methods are on the call stack at a given moment, and uses these snapshots to build the thread profile. That means it can expose package / class / method names that the agent wouldn’t recognize out of the box.

It also integrates with our Custom Instrumentation Editor to make it especially easy to select which methods to instrument and use them to build a custom XML instrumentation file. I won’t cover the CIE in this lesson, but I will show you the easiest way to build and deploy XML instrumentation, once we know the full name of the class and method we are targeting.

If I go to the Instrumentation page under Settings in APM, and click on ‘add instrumentation’, I see the following:

I can enter the class and method name we grabbed earlier (including the full package name), and I can check the ‘name the transaction…’ checkbox. The other check box, ‘start the transaction…’ corresponds to the transactionStartPoint property of the XML pointcut, and is for marking the transaction start point in frameworks where we don’t recognize your HttpDispatcher. You can use these in conjunction to start and name a transaction when a specific method is called. This is the easiest option to generate transaction data in an entirely unsupported environment.

In our case, checking ‘start the transaction…’ doesn’t matter, as there’s already an existing transaction in progress when this method is called, we really just want to change the name. When I click on ‘Add to instrumentation queue’, the screen changes to this:

From here, we’ll click on the ‘Export xml’ box. The file that is downloaded should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<extension xmlns="" xmlns:xsi="" xsi:schemaLocation="newrelic-extension extension.xsd" name="extension-example" version="1.0" enabled="true">
    <pointcut transactionStartPoint="true" ignoreTransaction="false">

This is our custom XML instrumentation file. You can see that transactionStartPoint="true" is set within the <poincut> node, and that <nameTransaction/> appears directly under it. This is what checking the '‘name the transaction…’ and ‘start the transaction…’ check boxes did. You can also see our full class / package name in the <className> node, and the method name within the <method> node.

If I click on ‘Deploy instrumentation changes’, the next time I ping ‘http://localhost:8080/RESTfulExample/users’ my method name should change from ‘WebTransaction/Uri/RESTfulExample/users’ to ‘WebTransaction/Custom/’. In fact, I’ll do this now so I can show you a screenshot of the change:

This is functionally identical to downloading the XML file, and then dropping it in the /newrelic/extensions folder on my local system.

There are limitations to XML naming. For one, you can’t set the transaction name to whatever you want, you have to set it to the name of an existing method on the request path. We’re relying on your application having a meaningfully-named method for every code-path, and a one-to-one relationship between code path and method handler. For another, there are ways of compiling classes in Java (such as class Enhancers) which result in a different class name each time the class is executed at runtime. XML instrumentation is pivoting off of the class name that the JVM sees, which is not necessarily the same class name that the developer sees. XML instrumentation doesn’t accept a wildcard, so you can’t easily target classes that are given dynamic names at runtime. For a similar reason, you can’t target abstract classes / methods with XML (since the abstract method isn’t actually executed at runtime), although you can target the abstract classes implementation. There are also options in the XML pointcut to target an interfaceName (which will instrument any class that implements that interface) or a methodAnnotation (which will instrument any class that is annotated with said annotation). These advanced uses of XML are beyond the scope of this level-up post, but are described in detail here.

Java agent API – @Trace annotation

Speaking of annotations, we define our own, and we can use it to set the transaction name. The agent looks for a couple of different annotations. It will respond to @NewRelicIgnoreTransaction by ignoring any request that calls the annotated method. It will respond to @NewRelicIgnoreApdex similarly by ignoring any request that hits the annotated method, but only for the purpose of Apdex calculations.

For renaming purposes though, the only one we need to be concerned with is @Trace, specifically @Trace(metricName=“YourMessageHere”, dispatcher=true).

@Trace annotation, like XML instrumentation, is really just a way of telling the agent which classes and methods are important, and should be traced. It extends the list of classes and methods that the agent instruments out of the box. It does require a code change, and it requires adding our API to your application’s class path, but it’s easy to apply, and doesn’t get in the way of your code. Remember, annotations aren’t executed, so no time is added to your method call. In the absence of our agent, the annotation does nothing. But, if our agent happens to be running on the JVM, we’ll react to the @Trace annotation by wrapping that method as we would were we to target it with XML.

In fact, the two are surprisingly similar. In XML, I set transactionStartPoint="true" to set the transaction start point, and nameTransaction to name it. With @Trace annotation, I pass dispatcher=true to start the transaction, and whatever value I pass to metricName is used to name the web request.

Note that unlike XML, I can set the name to whatever I want. Also, I have to set both the transaction start point (dispatcher=true) and a metricName value in order to rename the overall web transaction. If I just set a metricName value without the transaction start point, it will change the scoped metric name that appears on the web transaction’s breakdown table, not the transaction itself.

Let’s see this in action. First off, I’ll need to add our Java API to my test application’s class path. I’m using maven to compile my app, so doing this is a simple matter of adding the newrelic-api.jar file in my /newrelic root folder as a compile-time dependency in my pom.xml file. I won’t show this part because how you add dependencies almost certainly varies, but for those who are interested here’s how to add our API.jar to a pom.xml file.

Now all I have to do is add an import statement to my main .java file importing the API Trace class, and I can access @Trace annotation. I’ll add this to the getUserByName method.

import com.newrelic.api.agent.Trace;

public class UserRestService {
	@Trace(dispatcher=true, metricName="/getUserByName")
	public Response getUserByName(@PathParam("name") String name) {

		return Response.status(200)
				.entity("getUserByName is called, name : " + name).build();


The only line I’ve added is @Trace(dispatcher=true, metricName="/getUserByName"). Once I recompile and redeploy, the next time I ping ‘http://localhost:8080/RESTfulExample/users/aaa’ I should see this:

Note that before, all my user names were showing up in my transaction names. That’s a potential security issue in addition to being a metric grouping issue. Now, all requests that hit getUserByName will be named /getUserByName, no matter which user. I can clearly see the performance of the code path.

Java agent API – setTransactionName()

This one’s an easy one. You can set an API call within a method, to call out to our agent. There are a lot of neat things you can do with the API, but since the topic of this post is naming web transactions, I’m only going to focus on one.

We’re going to use the setTransactionName API call to rename the getUserById method. Like with @Trace annotation, we have to add the newrelic-api.jar to the class path, and we have to import the API. In this case, I’m using a wildcard in my import statement to import the entire API, not just the Trace annotation class.

import com.newrelic.api.agent.*;

public class UserRestService {

	@Path("{id : \\d+}")
	public Response getUserById(@PathParam("id") String id) {

		NewRelic.setTransactionName("Custom", "getUserById");
		return Response.status(200).entity("getUserById is called, id : " + id)


The only line I added was this: NewRelic.setTransactionName("Custom", "getUserById");

Once I recompiled and redeployed, this is what I see in APM:

The advantage of using the API is accessibility. If I have access to the code, I can control every aspect of the metric name. I can set the prefix (in this case /Custom), I can choose to set one name for one result of an if clause and a different name for a different result. I can grab just about any string that is accessible to a given method and use it for the name. Java is an encapsulated language, so usually class specific attributes are only visible within that class, but since we’re placing a call from inside that class, we can see its local attributes.

Which leads me to another common question. In the example above, I just turned a bunch of different URI metrics:


into a single metric named /getUserById. But what if I need to know the user ID? Generally, APM is used to track the performance of code paths, not individual users, but it could be useful to know how often one particular user logged in last week. Here, I’ll just point you to a recent release note: Java Agent 4.9.0

  • The ‘request.uri’ attribute is now included in Transaction events and corresponding Transaction Error events. As a result this attribute can now be queried in Insights. It may be excluded via the exclude list in the attributes stanza of the yaml config file. It is also excluded when High Security Mode is enabled.

So for all those who are worried that using class and method based naming will obfuscate data that’s visible with URI naming, don’t worry. The page URI is still in Insights, and you can query it. If I run this against my test app in Insights:

SELECT count(*) FROM Transaction WHERE `request.uri` LIKE '/RESTfulExample/users/%' SINCE 3 days ago TIMESERIES AUTO FACET `request.uri`

I see the following chart, which shows all of the requests against /RESTfulExample/users/* endpoints, broken out by user ID.

This is true whether the actual transaction name was set to the page URI or to a class and method name, so at the end of the day, no data need be lost by grouping transactions together.

I’ll also point out one other API endpoint: addCustomParameter. If I were to pass up the user ID as a separate custom parameter, I could query it directly in Insights.

1 Like

Request Attribute Naming

For those who have been following along especially close, you may have noticed that I mentioned four ways of renaming transactions at the beginning of this post, and we’ve only covered 3. You may also have noticed that the top entry in our list of naming priorities, above even the Java API, is ‘Request Attributes’.

This refers to HTTP request attributes, like the page URI. Our agent looks for a couple of different attributes on the HTTP request, and takes actions based on their presence. One is com.newrelic.agent.IGNORE, which tells the agent to ignore a specific HTTP request entirely. Another, com.newrelic.agent.APPLICATION_NAME, when used in conjunction with automatic application naming , lets you set what application name a given HTTP request is reported under in APM. This actually overrides the application name set in the newrelic.yml file.

For the purposes of metric naming, we’ll focus on just one: com.newrelic.agent.TRANSACTION_NAME. If you pass this as part of a request, the agent will use it to name the web transaction in APM. It’s that simple.

Unfortunately, I don’t have a great way to demonstrate this option using the test app I used for the rest of this post. That is because my JaxRS class isn’t passing the HttpRequest object directly, so I can’t just call request.setAttribute. However, were I to call this from a servlet’s doGet method, or any other method that does handle the HttpRequest object directly, it would be a single line of code:

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    request.setAttribute("com.newrelic.agent.TRANSACTION_NAME", "MyTransactionName");

and whatever I passed in place of “MyTransactionName” would show up in APM as the web transaction name.

The nice thing about request attribute naming is that it can be added to the HTTP request at any point during the life of the request. That means in cases where you can’t edit the code directly (say you’re using a 3rd party service), it isn’t too hard to stick a servlet filter in front of the application, and do your naming logic there. I’ve even seen customers add New Relic HTTP request attributes at their proxy layer, like this.

And that’s it. You now know all of the ways I know of to control transaction naming in the Java agent. Note that any of these methods can be used to turn our metric grouping issue:



back into a couple of cleanly named code paths;


When first setting up monitoring on a Java application, I would recommend giving a bit of thought to how you want to organize your data. Remember that if the agent doesn’t recognize your framework, you’re likely to see transaction names based on URI, or possibly no transactions at all, and if it’s the former, any dynamic elements in the URI are likely to fragment a single code path into many separate transactions. If you do encounter a metric grouping issue, we’ll let you know when you’ve hit 300,000 unique metric names, but by then you may have a fair amount of historic data built up, and custom dashboards, alerts, etc. built around that data. It’s much easier to give your app a clean naming scheme when it’s first spun up than to go back and clean up a large metric grouping issue after it has exploded.

If you have further questions regarding metric grouping issues, or transaction naming, feel free to post your questions here or reach out to our support department.

Thank you all, and happy naming!

1 Like