Relic Solution: Downsampling and Retention of Infrastructure Data

If you have already read my level-up post about Data Visualization in the Infrastructure UI, this is a more advanced look at how data retention and downsampling work in Infrastructure. Most of this is only visible through Insights, but you may come up against some of it with custom time windows in the Infrastructure UI.

What is downsampling?

Downsampling is when we take several buckets worth of data and aggregate them by taking an average. This occurs in realtime, as the data comes in.

So when your Infrastructure agent is running, collecting data to send to New Relic every 5 seconds (or 10 seconds for Process Samples)**, once per minute a downsample happens and 12 raw data points (6 for Process Samples) are aggregated into a 1-minute downsampled bucket. Once 10 of those 1-minute buckets are collected, they are aggregated into a 10-minute bucket, and so forth.

The raw data stays around and is queryable for a certain amount of time (7 days, actually). In order to view the raw data, however, you must use a time window of 59 minutes or less in Insights or on the Infrastructure dashboard. After 7 days, the raw data disappears, leaving the 1-minute downsampled buckets which can be queried in Insights. However, using a time window of greater than 6 hours will cause 10-minute buckets to be used. Once the data is 30 days old, those 1-minute buckets disappear, but 10-minute buckets can still be queried. 10 minute buckets never disappear – they will stick around for as long as your data retention allows.

Keep in mind that data retention is 13 months for Pro subscriptions and 3 months for Essentials.

The following table outlines the smallest bucket size that can be viewed based on the length of the time window used in a query, and how long that bucket size is retained.

Hopefully this information will help you to set up custom dashboards to show you exactly the data you want, with exactly the granularity you need to make good business decisions!

** As of version 1.3.1 of the Infrastructure agent (release notes here), ProcessSample and StorageSample have had their default polling frequencies raised to 20 seconds, while NetworkSample’s default frequency was raised to 10 seconds. This change is mainly to decrease the footprint of the agent. You can change these polling frequencies to their original values with the metrics_process_sample_rate, metrics_storage_sample_rate and metrics_network_sample_rate config settings.


A post was merged into an existing topic: Feature Idea: Increase maximum buckets for TIMESERIES