APM transaction grouping problem

Check out the permalink

Hi, I have a complicated problem. I want to use NewRelic to give me insights on how fast/slow my API responses, when, and where so I can then improve response speeds and improve my customer’s experience.

I’ve provisioned and setup NR in my web app (which is a Node.js app build with the Adonis.js framework but the problem that I have is that the majority of my transactions end up like this:

This is problematic for me because I can’t get insight into my API requests. At first I thought this was caused my Web Socket long pooling requests so I’ve set NR to ignore them in the handler function (see example below)

This did not have the intended effect so after some Googling around I found out about “Metric Grouping Issues” and so I thought if I setup a global middleware that would “name” all my transactions using the URL, it would solve this issue but I read some forum responses saying that its best to let NR do auto naming.

I’m at a lost and I don’t really know what else I can do to prevent having a big root bucket where all requests fall into.

If I switch the view to be percentage based it shows me that the 95% and 99% percentile is 26sec, the timeout for a request which indicates that long pooling from the socket is not effectively being ignored.

Hi @verdi - When the information is collected as metrics, which is the view on your screenshot, then grouping is applied. I am fairly certain that this is not the case when you query the Transaction event information. I think you can check this using the below query as an example.

SELECT uniques(name) FROM Transaction WHERE appName = '[Your APM app name]' SINCE 1 week ago LIMIT MAX

You could then use a count(*) query FACET name over the time period to provide a breakdown.

Thanks @verdi and @stefan_garnham for your posts here. This is a great topic for Explorers Hub since many people face this exact Metric Grouping Issue (MGI) hurdle!

I’d first like to direct you to this excellent post by New Relic’s @jfalleur that details what’s going on here.

https://discuss.newrelic.com/t/relic-solution-video-understanding-metric-grouping-issues-in-80-seconds/71941

TL;DR

The best way to solve this is with careful planning and implementation of a good transaction naming scheme using newrelic.setTransactionName. Using setTransactionName, opposed to regex matching rules, will make future changes to application code much less likely to break the naming scheme.

Having fewer than 10 unique names will assist in seeing where performance can be improved in your app. Ideally, I’d limit this to the 5 most important types of transactions rather than diluting results across (what can easily become) 1000 names, which then get rolled up into /* to prevent performance issues in the UI.

Sometimes a big enough MGI can trigger a deny-new-metrics rule, which does just what it says. It denies new metrics from coming into APM in order to prevent an explosion of unique names. Fortunately, that hasn’t happened for your app, @verdi.

If you do suspect new metrics are not showing up in APM, or if you receive an automated MGI email from New Relic describing as such, definitely contact NR Support to work with us on resolving the MGI and then we can disable that deny-new-metrics rule.

Below are the Insights queries I use when analyzing an app facing an MGI. Insights is quite useful here because it will show all the transactions hidden behind the rolled up /* in APM.

  1. Count of unique names by transactionSubType like Custom (which indicates names set by API calls).
    SELECT uniqueCount(name) from Transaction where appId = [your appId] since 1 week ago facet transactionSubType
  2. The same as number 1, but viewed over time.
    SELECT uniqueCount(name) from Transaction where appId = [your appId] since 1 week ago facet transactionSubType TIMESERIES max
  3. Unique names with socket somewhere in the name.
    SELECT uniques(name) from Transaction where appId = [your appID] since 1 week ago LIMIT MAX where name LIKE '%socket%'

While setTransactionName can be a useful to for renaming transactions with a lot of variation in their names, sometimes it is equally useful to ignore certain transactions altogether. To accomplish that, I recommend utilizing the newrelic.js config file where we can add a section like this:

rules: {
  ignore: [
    '^\/sockjs\/.*',
    '^\/socket\.io\/.*\/xhr-polling/'
  ]
}

FYI, here’s a default config file with all available options listed:

https://github.com/newrelic/node-newrelic/blob/master/lib/config/default.js

Happy renaming and ignoring!

3 Likes