Different throughput values on APM and Insights

Please paste the [permalink][1] to the page in question below:


Please include any NRQL you are using:

SELECT count(*) FROM Transaction WHERE appName = ‘my-service’ SINCE ‘2018-03-14 12:35:00 CET’ UNTIL ‘2018-03-14 13:45:00 CET’ TIMESERIES 1 minute

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

APM Overview page presents different throughput then the above query executed on Insights.
In my case, the difference is 25K rpm (on APM) vs 16K rpm (NRQL).
What NRQL returns true throughput?

Some additional questions:
Does events sent to Insight are sampled? And APM presents the data aggregated by NR agent?
If so:
Is sample rate constant?
Does sampling is impacted by the current traffic?

Hi there @LJK - Welcome to the Explorers Hub!

So there are a couple of different reasons why you would see different data in APM vs. Insights.

First - the two products actually work with two different types of data. APM works with Metric data and Insights works with Events data. There’s more information about the two in this documentation.

You are right that the APM data is sampled, and because of that, there is no real way to get both APM and Insights to align down to the second.

Let me know if that does not clear things up.

I still have some doubts.

Does Insight’s Transaction table contains all transactions which happened on the service or only its sample?

The throughput is a number of transactions per unit of time. It should not matter if events or metrics data are used.
I am not talking about slight differences. On the screenshot below, you can see tremendous difference of 8000 rpm. Which data I can trust?

Insights (16K rpm):

or APM (24.5K rpm):

Both charts relate to the same time period (unfortunately I used different time zones : CET and GMT which could be misleading but should not impact the results).

Hello, @LJK: Insights does cap the number of transaction events it collects per minute; I don’t know that the maximum number of events is documented. You may use the following NRQL to see the total number of transactions:

SELECT count(*) + sum(`nr.omittedCount`) 
FROM Transaction WHERE appName = 'My Application' 
SINCE '2018-03-14 12:35:00 CET' UNTIL '2018-03-14 13:45:00 CET' 

Update: Sorry, looks like nr.omittedCount may only be visible to New Relic personnel.

1 Like

Here is a previous thread with additional detail:

Hi philweber,
thanks for confirming that events are sampled. I should not monitor throughput with NRQL though.

BTW, sum(‘nr.omittedCount’) does not work in my case. It is constantly zero. I am using agent ver 3.40.

Hmm. I tested my query on your data, using the link you provided in your original post. You might try widening your time window, or removing the WHERE appName = ... clause.

Update: Sorry, looks like that attribute may only be visible to New Relic personnel.