Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: Resolving .Net Metric Grouping Issues Part I


#1

Normally, customers file tickets for issues they are experiencing or questions they have. But there is one class of tickets created by New Relic Support that look like this:

This ticket was created due to a very large number of metric names being created for your dotnet application ‘Some-application’ at https://rpm.newrelic.com/accounts/<account_number>/applications/<application_id>.

There may be more metrics than are useful to display and continuing to add metrics to your account database would reduce the effectiveness of charts in the user interface and may cause performance issues within RPM. A limit has been put in place that will only allow through existing metrics…

This is the somewhat mysterious Metric Grouping Issue ticket! This post is meant to dive more deeply into the reasons for this ticket, actions you should take, and solutions to the underlying issue.

Be sure to read the followup article: Relic Solution: Resolving .Net Metric Grouping Issues Part II as a follow up to this article. It presents techniques for identifying causes and provides solutions to common metric grouping issues.

So now what?

  1. Read https://docs.newrelic.com/docs/agents/manage-apm-agents/troubleshooting/metric-grouping-issues. I would also recommend reading Relic Solution: Metric Grouping Limits and Relic Solution: Video - Understanding Metric Grouping Issues in 80 seconds. After reading those articles please return here for .NET agent specific information!
  2. Reply to the ticket! Don’t ignore it - the ticket may auto-close if you do. The above message means a “metric clamp” has been put in place on the specified application which means no new metrics can be created (although data for existing metric names will continue to report). If you make a code change that would introduce a new metric then you will never see that metric as long as the clamp is in place. It’s important to resolve these issues as quickly as possible.
  3. When you reply to the ticket ask for clarification on the exact nature of the issue. There are various causes of metric grouping issues and we can help determine the cause and solution. We may need your help though so be prepared to answer our questions as well. We may ask about the application type, your use of agent API calls and custom instrumentation, and even agent debug logs to examine agent behavior.

Can I See These Metrics?

Not in the New Relic UI. Not all of them anyway. Not even many of them. You can view a fair number of web transaction metrics by going to Transactions>Show all transactions table… Maybe enough to give some idea of what’s going on. We can send you a complete list of the problematic metrics.

Insights queries can surface a sample of transaction names. See Relic Solution: Resolving .Net Metric Grouping Issues Part II for more information and sample queries.

Solutions

It may seem odd to discuss solutions before discussing causes but we want to get one thing out of the way: New Relic Support created metric grouping rules, discussed below, are the least desirable solution to resolving transaction name related metric grouping issues, which are the most common type!

Having your application group metrics via agent API calls, discussed next, is the best solution when possible. It could be a simple agent upgrade, which may implement appropriate grouping, will prevent future issues.

Grouping via agent API calls

It’s preferable for your code to group transaction name metrics using the SetTransactionName API call.

In some cases an alternative is to ignore transactions using the IgnoreTransaction API call.

We understand that an API call is not always possible but it should be seriously investigated before resorting to grouping rules.

Grouping via Support created rules

One solution is for New Relic Support to write rules that “group” metric names. These transform a metric name into another name. For example we might create a rule to transform the following:

WebTransaction/Uri/api/v1/example-12345
WebTransaction/Uri/api/v1/example-54321
WebTransaction/Uri/api/v1/example-21435

To

WebTransaction/Uri/api/v1/example-*

This way we can “group” possibly thousands of unique names into one name preventing a future issue.

Although rules may seem like the way to go they really should be a last resort for resolving transaction name related metric grouping issues for the following reasons:

  • They can be tricky to implement.
  • They are prone to breaking on changes to the application and the .NET agent.
  • They remove control of the transaction naming from the application developers.
  • They require New Relic Support to implement and maintain.
  • They are based on string pattern matching only - no other context is available (e.g. response status codes). This may cause rules to group too many or too few names.

However, rules are usually the only solution for most non-transaction name related metric grouping issues, for example, those related to Datastore, External, MessageBroker, and other metrics.

Common causes of metric grouping issues

Ok, now on to common causes of all kinds of metric grouping issues:

  1. The .NET agent is unable to give transactions a “good” name. This is the most common cause of metric grouping issues for .NET apps and is discussed below.
  2. Your app makes use of many SQL databases and/or tables or you are using temporary tables. This can create an explosion of Datastore metrics.
  3. Your app makes many external calls - perhaps it’s crawling the web which results in thousands of unique external call metrics and creates an issue in the External metric space.
  4. The application in question may make api calls to another application which itself is experiencing a web transaction naming related MGI. This can create an issue in the ExternalTransaction metrics of the calling app.
  5. You may be using a message broker (e.g. RabbitMQ) in such a way that causes our agent to create thousands of unique MessageBroker metrics.

Transaction name related MGIs

The inability of the agent to give transactions a meaningful name is the commonest cause of metric grouping issues for .NET applications.

The agent tries to name a transaction after the framework in use, for example, ASP, WebApi, MVC, etc. It does this by instrumenting certain methods used by a framework. By doing so, it can determine information about a transaction like which controller/action was invoked or the virtual path to the aspx code file and name the transaction accordingly.

In other words the agent categorizes (e.g. WebApi, ASP, MVC, WCF, etc) and names a transaction after the executed code path (e.g. the controller/action, aspx virtual file path, WCF service method, etc).

To see a transaction’s category select the Transactions tab, hover over a transaction name, and view the category in the displayed tooltip.

If the app is an asp. net application and the agent can’t give the transaction a name that reflects the executed code path then it will categorize the transaction as Uri and name it after the request uri. A complete metric name might look like this: WebTransaction/Uri/api/v1/asp-net-example/21435

If the app is an asp. net core application and the agent can’t give the transaction a name that reflects the executed code path then it will categorize the transaction as ASP and name it after the request uri. A complete metric name would look like this: WebTransaction/ASP/api/v2/asp-net-core-example/54321

In either case the uri-based naming can lead to thousands of unique web transaction metric names and a metric grouping issue. This is because each request uri likely contains parameters, guids, and other unique information that becomes part of the transaction name.

A brief aside - transactions and request processing pipelines

The agent knows when requests begin and end for the following request processing pipelines:

  • asp .net
  • asp .net core
  • OWIN
  • WCF

This means the agent can create transactions for requests that enter these pipelines. Beyond that, if a request does not execute any further code instrumented by the agent then the transaction will be given a uri-based category and name as discussed above.

Also, if an application does not utilize one of these pipelines then the agent won’t be able to create transactions at all and will need the help of custom instrumentation to define transaction boundaries. But that is the subject of another article.

Transaction name MGI common causes and solutions

Here are some common causes of transaction name metric grouping issues and their solution. This is not a complete list. We’ll work with you to determine a cause not listed in this table.

Cause Solution
Requests that result in status code errors, i.e. response status code ≥ 400. This is typically due to page-not-found errors (404) often as a result of penetration tests. Agent upgrade. This is primarily an issue for older agents. Agents 6.20+ categorize these transactions as “StatusCode” and name them after the status code, e.g. “WebTransaction/StatusCode/404”
Requests that result in 30x status codes. These too can be related to penetration tests or other application configuration/behavior. An agent upgrade can prevent a future issue. Starting with 8.15 the agent groups these as, for example, “WebTransaction/StatusCode/302”
CORS preflight requests can cause the agent to give the transaction a Uri-based name. This is most common in asp .net core apps using CORS middleware but can also occur in an asp. net application that handles CORS requests. The solution for asp. net core apps is to upgrade to agent 8.16 which will ignore those requests (an upgrade won’t solve the issue for other pipelines).
A framework unsupported by the agent, like NancyFx, will cause the creation of Uri-based transaction names. Calling SetTransactionName, if possible, is the best solution. Also ask us to file a feature request for support of your favorite framework.
Static file or static asset related requests can generate thousands of unique Uri-based transaction names. Calling SetTransactionName, if possible, is the best solution. This may need to be done in Global.asax or a custom http module.
Your application may be misusing the SetTransactionName API call by including guids or other unique information. Don’t include unique information in your transaction names. Instead, use the API call AddCustomParameter to add transaction specific data as transaction attributes.
Upgrading an app that previously targeted .NET 4.0 or earlier to targeting .NET 4.5+ without certain application configuration updates can cause the agent to use a uri-based transaction name. This occurs most often for WebApi and MVC applications. See this page: https://docs.newrelic.com/docs/agents/net-agent/troubleshooting/missing-net-async-metrics for solutions.

What about those other types of MGIs?

Oh right. I mentioned there could be metric grouping issues in the Datastore, External, MessageBroker, and possibly other metric spaces.

These types of issues will likely require grouping rules. However, we’ll discuss the issue with you to determine the optimal solution.

An issue with ExternalTransaction metrics will call for a solution in the called application. Again, we’ll investigate the offending application and propose solutions.

What’s next?

Be sure to read the followup article: Relic Solution: Resolving .Net Metric Grouping Issues Part II as a follow up to this article. It presents techniques for identifying causes and provides solutions to common metric grouping issues.


Relic Solution: Resolving .NET Metric Grouping Issues Part II