Missing data in NR dashboard

Hello,

I would like to discuss an issue we’re having with our NR dashboard for a while.
We’ve noticed that our metric is sometimes not visible in the NR dashboard.
I can see there is a pattern:


As you can see, there are these gaps with missing data. The problem is, we do not stop sending data from our application. I will share with you more print screens:

FEB 21 06:00 PM - FEB 22 06:00 AM

FEB 21 06:00 PM - FEB 21 09:00 PM

FEB 21 06:00 PM - FEB 21 06:20 PM

So as you can see, when I select a certain time range, it is pretty clear that we are missing data and all of our widgets do not take those data that are not visible into account when calculating any type of metric. However when I significantly decrease the time range, you can see that the data is actually there.
Is there something we’re doing wrong on our side? How come I cannot see all the data that are continuously coming to the NR Metrics API? It causes our metrics to be very misleading, since every day there is a several hours long gap where the data sent to NR Metric API are ignored.

Thank you very much for checking,

Tamara

Hi @dev86

Although I can’t see what is being queried (without that knowledge I’m left guessing), this is behaving as if cardinality limits are being hit. Documentation on that can be found at this link.

When you hit this limit on your account, it stops metric aggregation for the next 24 hours. Note that the raw data is still there, it’s just not being aggregated. Any timeseries graph over 1 hour will display aggregated data by default. On the other hand, if the graph shows a time window of 1 hour or less, raw data will be shown (which would explain why data isn’t showing up for longer time ranges, but shows up fine if you query an hour or less).

Will you try running the longer queries with RAW at the end? So long as you are querying less than 48 hours, RAW will force the raw, un-aggregated data to show up.

Let us know how it goes!

2 Likes

Thank you very much @Fidelicatessen!

It was indeed the cardinality limit violation. For other people I will sum it up:

Cardinality = unique metric = unique combination of metric name and its attributes.

The cardinality limit is 100K per metric per day.

In my case, I was sending latency in one of my attributes. Since the load in my app is pretty high, I reached 100K unique metrics pretty easily (because of all these unique latency values). I fixed this “issue” by removing latency from my metric and I am no more hitting this limit and my metrics (aggregations) are visible all the time.

3 Likes

@dev86

Wahoo! Glad I could help out!

Thanks for circling back around and letting us know what the outcome was :smiley: