This is a followup to article Relic Solution: Resolving .NET Metric Grouping Issues Part I. Be sure to read that article before proceeding with this one.
This article presents a few Insights queries you can run to diagnose the root cause of your metric grouping issue ticket. It will also map solutions to the results of those queries. You might even find the queries useful outside the context of metric grouping issues.
The queries shown below are applicable to transaction name related metric grouping issues only.
These queries don’t apply to issues related to database, message broker, external, custom, etc. metrics - those still require analysis and resolution by Support, usually through the use of grouping rules (although an issue due to custom metrics can be solved by adjustments to your agent API calls).
Your app must be collecting Transaction events. This may not be the case if you’ve disabled the collection of events in the agent’s config or on the Insights “Manage data” screen.
So, assuming we are looking at a transaction name related metric grouping issue and transaction events are being collected for the app of interest, here we go…
Transaction names as a function of status code
The following query lists the count of unique transaction names as a function of status code:
SELECT uniqueCount(name) FROM Transaction where appId = <appID> and name like 'WebTransaction/Uri%' SINCE 1 month ago facet `response.status`
The query as shown applies to asp. net apps. Change “Uri” to “ASP” for asp. net core apps. And remember, we are using the Uri and ASP categories because these are normally the ones associated with metric grouping issues. They imply the agent was unable to determine a meaningful transaction name for the request.
The query may result in a table like this:
Yikes! In this case the agent created over 48k unique transaction names due to redirects in the application. That would certainly create a metric grouping issue.
This likely means you are on an older agent version and you need to upgrade to the latest agent version to prevent a future issue.
Newer agent versions “group” those transactions into names like
This grouping actually applies to transactions with status codes >= 300 so if you see a large number of transactions names related to 40xs or 50xs status codes then the solution is the same - agent upgrade.
A brief aside
For fun, add the
TIMESERIES clause to the end of the query. You may need to adjust the timespan as well.
That’ll show the timing of things - was it a one-off penetration test or is it an ongoing issue?
In any event, and agent upgrade will prevent a future metric grouping issue from occurring.
Another brief aside
Here is an alternative query:
SELECT uniqueCount(name) from Transaction where appId = <appId> since 1 month ago facet transactionSubType,response.status limit 100
The results of this query might look something like this:
That query shows the number of unique transaction names as a function of transaction category (aka transaction sub-type) and response status code. In general, metric grouping issues occur in the Uri category but occasionally happen in other categories like MVC, which the above query may surface, and leads us to our next topic.
What about MVC transaction names?
There is one fly in the agent-upgrade ointment here. Let’s say you suspect the metric grouping issue is due to a large number of MVC web transaction names. And from our previous query you discover a large number of unique MVC transaction names related, say, to a penetration test.
Unfortunately, the agent upgrade won’t help in this case. You’ll need to engage with New Relic support to formulate a solution here.
Asp. net core + 204 = CORS
Let’s say you have an asp. net core app and you run the query:
SELECT uniqueCount(name) FROM Transaction where appId = <appID> and name like 'WebTransaction/ASP%' SINCE 1 month ago facet `response.status`
and you find a large number of names associated with the 204 status code.
This usually means you are using CORS middleware to process preflight requests. Older .NET agents will generate a uri based (category “ASP”) transaction name for each preflight request, resulting in a bunch of unique names.
The solution here is an agent upgrade. Starting with 8.16 the agent ignores asp.net core preflight requests.
But the status code is 200!
Ok, you’ve run one of the status code queries and you find a large number of unique uri based transaction names associated with a 200 response code - that seems odd. Here are a couple of queries to see a few of the names:
SELECT count(name) as 'Count' FROM Transaction where appId = <appId> and name like 'WebTransaction/Uri%' and `response.status`= '200' since 1 month ago facet name limit 2000
or the equivalent
SELECT count(name) as 'Count' FROM Transaction where appId = <appId> and transactionSubType = 'Uri' and `response.status`= '200' since 1 month ago facet name limit 2000
One of those queries should give you an idea of the types of requests/transactions involved and their frequency of occurrence. Substitute
Uri if you are analyzing an asp. net core app.
Here is a sample result:
There could be several reasons for this case (solutions follow):
You are using an app framework, like NancyFx ,that is not supported by the agent. In this case the agent provides no instrumentation specific to the unsupported framework and defaults to generating transaction names based on the request uri. It doesn’t know how to otherwise name transactions in this case.
Your app is serving up static files or is otherwise processing requests outside of any code the agent instruments which means the agent can only give a uri based transaction name. Maybe you have a custom middleware or custom http module processing all requests?
Your app is WebApi and the transaction names reflect your WebApi endpoints but the names aren’t controller/action based. You may see parameters/guids in the name which are causing the name explosion and you may even see
*in the transaction name (
*represents an attempt by the agent to group an unruly transaction name segment).
You are processing CORS preflight requests in an asp. net app. The results of the following query are suggestive of this case:
SELECT count(*) from Transaction where appId = <appId> since 7 days ago facet cases(where name not like 'WebTransaction/Uri%' as 'Not Uri', where name like 'WebTransaction/Uri%' as 'Uri') TIMESERIES
Two time series lines, labeled “Uri” and “Not Uri”, tracking each other suggests CORS preflight request handling may be occurring. The line labeled “Uri” may be the preflight requests.
Case 1 and 2 solutions
A solution for cases 1 and 2 is to “group” transaction names using the SetTransactionName call.
However, an issue with this approach is where to make the API call. For an asp. net app you might be able to call it from an event handler in global.asax (say
Application_BeginRequest) or in an http custom module. Or maybe the application framework provides a hook from which to make the call.
For an asp .net core app you can create a custom middleware that makes the API call.
If the API call isn’t feasible then grouping rules created by Support many be the only option.
Case 3 solution
Case 3 above may reflect the need for a configuration change documented here. This applies only to .net framework applications. In this case, the part of agent responsible for naming WebAPI transactions may have shut down because the newer async request processing pipeline introduced in .NET 4.5 has not been activated. The following entry in the agent’s log will confirm this issue:
NewRelic WARN: The method InvokeActionAsync in class System.Web.Http.Controllers.ApiControllerActionInvoker from assembly System.Web.Http will not be instrumented. Some async instrumentation is not supported on .NET 4.5 and greater unless you change your application configuration to use the new ASP pipeline. For details see: https://docs.newrelic.com/docs/agents/net-agent/troubleshooting/missing-async-metrics
Case 4 solution
An agent upgrade won’t help in this case. If possible, detect these requests and call the IgnoreTransaction API to ignore them.
It’s me, not you
Occasionally a metric grouping issue is created by improper use of the SetTransactionName call. In this case you may be including guids or other highly variable components in your custom name.
Let’s reuse a previous query to surface this issue:
From Transaction select uniqueCount(name) where appId = <appId> facet transactionSubType since 1 month ago
This query shows the number of unique names by transaction category. If a category related to your custom transaction naming stands out then you’ll need to adjust your API call to eliminate variable components in your names. Often the category will be “Custom” but you might have chosen something else.
When Custom is not custom
This is an unusual case! If you are using the Owin pipeline and you are not using an agent supported middleware like WebAPI the agent will generate uri-based transaction names and their category will be “Custom”.
This can be confusing given the discussion in the previous section. This is not the result of the misuse of the SetTransactionName API call. In fact, this case calls for the use of SetTransactonName to group names effectively.
For example, if you find the agent is generating the following transaction names:
WebTransaction/Custom/v1/resource/Transaction/123456789 WebTransaction/Custom/v1/resource/Transaction/987654321 WebTransaction/Custom/v1/resource/Transaction/019283746 . . .
You’ll want to call SetTransactionName (from a custom middleware) to group things as, for example:
So you’ve upgraded the agent, used the SetTransactionName API call, and/or fixed your custom transaction naming. How do you evaluate the effectiveness of your changes with the metric clamp still in effect?
The good thing is a clamp only affects metrics, not Insights Transaction events. Transaction names found in Insights Transaction events will reflect changes you’ve made. This means you can evaluate your proactive efforts with suitable Insights queries even with the metric clamp in effect.
Once you’ve evaluated the effectiveness of your changes you can ask Support to
release the “metric clamp”.
Now you should be good to go - once the clamp is lifted metric name changes due to app enhancements or agent upgrades will be reflected in your New Relic charts.
Let’s hope you never receive another metric grouping issue ticket from New Relic. But if you do, you’ll know what to do.