Feature Idea: Custom retention for Insights Custom Events by eventType

Summary

I would like to be able to set days of retention by eventType within the Event data retention management page for Insights Custom Events.

This would allow me to have long-term retention on some events (6-12 months), while purging events that I only need for a shorter period (24 hours to 7 days, for example).

Backgound

I create a lot of custom events with New Relic Insights, using the Insert API. These events are extremeley useful to my business, but some of them are only needed for a short period, while I’d like to keep or create other events that are retained for longer, so that I can query and create dashboards that go back several months.

Due to the way Insights licensing & usage works, my “bucket” fills up with events containing data that could be aggregated into a much smaller number of events, and retained for longer.

Feature Request Details

If I could set custom retention by event type, and have some events purged, then I could keep the total number of events within the licensed amount, and while also having longer retention for other events where necessary.

In order for this to be most useful, I would want to create additional “aggregated” custom events from the higher quantiy / more frequent timeseries events, to allow longer retention of important information, but at less detail. These would be retained for a longer period.

If I could use New Relic itself to create the aggregated events automatically, that would be even better, but it wouldn’t be hard to query the Insights API, and then just feed the results back in as new Custome Events.

Alternatives

It would appear that this could be achieved via a “hack” by using Sub-Accounts: Event data retention | New Relic Documentation

It looks like a sub-account could be used for each set of Insights Custom Events that require a different retention. However, this is obviously a “sledgehammer to crack a nut” scenario.

Aggregation

An additional feature, or possibly even an alternative, would be to specify custom aggregation for events. This is apparently already built-in to APM, Browser, and Mobile, in some fashion. I forsee the need to aggregate in other ways as well (sum, for example), and to specific the attribute / facets that are aggregated.

From: Overview of data retention and components | New Relic Documentation

New Relic Docs

Useful links:


New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

1 Like

Thank you for submitting such a complete account of your needs. I have added a poll to get the interest from other Explorers and informed our product team. :slight_smile:

1 Like

6 posts were split to a new topic: Custom events questions

Since posting this, I was informed of the relatively new Metrics feature in New Relic, and the following capabilities:

It turns out that a large quantiy of my data that I’m sending to Insights is actually metric data, and not truly events. So this seems like a good match for a good part of this.

The Metrics product also apparently has extended retention, and aggregation built-in:

However, a large number of my events in Insights are indeed actual “events”, and the data has a high degree of cardinality, and while useful metrics can be created from this data, the ability to query the raw data using NRQL is very valuable.

But the timeframe for keeping the full dataset for some high-volume events could still be reduced, while the timeframe for some other low-volume events extended, to allow longer retention & raw NRQL queries of the lower-volume data, which means this feature request is still quite relevant.

Also, it is still not yet clear whether Metrics will be sufficient for my needs, and the pricing, limits, and details on automatic aggregation are either unclear or completely missing from the New Relic website, at this time.

Thanks for the extra detail on this @mkopala

For Metrics, the limits are documented here: https://docs.newrelic.com/docs/data-ingest-apis/get-data-new-relic/metric-api/metric-api-limits-restricted-attributes

As for pricing, that’s information our sales team can help with here
But… if you have a regular paid account with no metrics subscription, you get:

Additional account conditions: Trial and paid accounts receive a 100k DPM and 100k cardinality limit for trial purposes. Metric API limits apply at the individual account level

(Taken from: here)