Data from New Relic UI not matching up with data pulled from API

I am trying to create my own internal dashboard of our app metrics.

I am using both the REST API for the metrics data, and the nerdgraph API for Events data. For both APIs, the data I am pulling does not match up with the data I am seeing in the New Relic UI.

For instance with the Nerdgraph API, I have this query in my discovery dashboard:

I copy pasted this NRQL query into my API request. However, the data I am getting is as follows:

The graphs are generally pretty close, but they are still different.

I am also trying to recreate the daily SLA reports using this query:
SELECT (count(*)) AS ‘Page Views’, (average(duration) * 1000) FROM PageView FACET dateOf(timestamp) since 12 days ago

This also does not match up with the data I am seeing in the SLA reports within New Relic.

As for the REST API, I am trying to recreate the web transactions time graph in the main dashboard using this query:
curl -X GET ‘${APPID}/metrics/data.json
-H ‘X-Api-Key:${APIKEY}’ -i
-d ‘names[]=HttpDispatcher&values[]=average_response_time’

obtained from here. This data is also discrepant (I was only able to upload 2 screenshots)

What is causing this discrepancy? I am performing all the requests at 2 hour time intervals, so my pulled data is at a 1 minute resolution, which matches with what I am seeing in the UI.

Hi @alexander.dewolff - I noticed that the timeseries are not matching. Try changing the NRQL to TIMESERIES 16 minutes to match that chart below. You may also want to add WITH TIMEZONE with the correct selection for PST as well.

SELECT average(duration * 1000) AS 'Response time' FROM Transaction TIMESERIES 16 minutes SINCE 1 hour ago WITH TIMESERIES 'America/Los Angeles'

Not sure on the SLA report, although there is a post about replicating SLA reports that you could adjust

1 Like

@alexander.dewolff - Was Stefan’s latest comment helpful? Let us know if that helped you to match up the data :smiley:

1 Like

Hi Stefan, thanks for the reply!

I’m not sure what you mean by setting the timeseries to 16 minutes, as I am pulling data from the API using TIMESERIES 1 MINUTE (I’m only showing the start time in my graph, but the corresponding end time is 1 minute later). I added the TIMEZONE Los Angeles like you suggested, and specified 1 minutes, and the data still does not match.

This is the raw result I get from my API pull using this query: SELECT average(duration * 1000) FROM Transaction TIMESERIES 1 minute SINCE 1 hour ago WITH TIMEZONE 'America/Los_Angeles'

Could the issue be the second at which the report is generated? e.g., the API report is calculated at 8:07:22, and the UI report is calculated at 8:07:45?

As for the SLA report, that is the post and code I used to build my query which does not match the UI.

The time that the queries run would only affect the start and end times of the data returned, not the actual data for a specific time period in that timeframe.

@RyanVeitch - I think this needs some dedicated time from a support resource.

1 Like

Hi @alexander.dewolff -

This is interesting, but it’s important to understand that you are querying 2 different sources of data that are not expected to match exactly.

It’s likely that it’s related to long running Transactions; i.e. Events (your insights data) are just placed into the time bucket where they started and metrics (the API data) record the duration in the bucket they ended.

So anything spanning the minute boundary won’t match.

While I don’t expect the difference between API & NRQL to be as high as it is for you, this is the most likely cause of that.

Hi @RyanVeitch

I was under the impression that the NerdGraph API queries events data from the same source as the Insights dashboard. Is this not the case?

The Rest API that I am querying for the HttpDispatcher average_response_time also does not match the response time graph of the graph in the APM Overview panel. Do these two also not query the metrics data from the same source?

Thanks for the reply @alexander.dewolff

I didn’t see from your screenshots that you were taking data from the nerdgraph API and failing to match it up against Insights.

That does absolutely sound wrong. Your statement is true; that the NerdGraph API queries events data from the same source as the Insights dashboard. Is this not the case?

As is true the statement that the Rest API does indeed query metric data from the same source as the APM UI.

I feel as though this needs a much deeper investigation, and so I have created a ticket for you with our Support engineers who can try to dig in to this.

It’s likely that they will ask for a bigger data set example from you, so it may help to preempt that by sending over some data when you get that ticket link in your inbox.

Thanks! Let us know if you come to a solution with the Support team. :smiley:

1 Like

Thanks for the help @RyanVeitch!

1 Like

No problem @alexander.dewolff - I’m excited to hear what solution you come to in your ticket