Export Insights data for historical tracking

Hi,

Does anyone have ideas on how I can extract insights data periodically, problem is we have a fixed retention but need to keep insights stats data for atleast 1 year for customer audit purpose, and its a big task to do that.

We want to take snapshots of the data monthly to keep them as archived for any questions on the dashboard from end customer.

any idea if this can be achieved without taking a cost hit.

thanks
Manas

1 Like

Hi @MKhanna - The options available to you that I can think of are:

  1. Increase your data retention period to cover 12 months or your customer’s auditing requirements.
  2. Export the data using the Insights API to add to your archive datastore.
  3. Create a pdf/image of the dashboard covering the period as an archive.

Note that New Relic dashboards only work from the New Relic data, you cannot configure them to access data from anywhere else.

2 Likes

Hey @stefan_garnham thank you for the quick response,
For 1: I think at this point we cannot do that as it will be a running cost hit which will impact the budget.
For 2: I tried that but seems I get only 99 events even though my query is for 1 month or quarter.

but for step 3 is there any automated way you can recommend may be using Synthetic. I was thinking like doing a script to take screen shot of each dashboard and storing it somewhere. ( but not clear on how and where to store and what is that somewhere?)

But that can be a good use case so if I do $browser.takeScreenshot(); for each of my dashboard now i need some ideas on how to save it in pdf on a server of work station and run this monthly.

Any ideas are much appreciated.

thanks
Manas

NRQL queries have default results returned which can be increased by using the LIMIT clause. If you are expecting more results then you will have to break the query into smaller chunks and make more calls.

I haven’t found anything on the forum showing how to save a dashboard in code so that would be for you to investigate.

Hi @stefan_garnham thank you for the suggestion. I am also trying to figure out how to create a screen shot set to save locally or have the logs with screen shot saved in a file for later validation.

thanks

1 Like

I think your best and most reliable option is going to be using the Query API to automate event retrieval. This returns a JSON which can be parsed programmatically, potentially saving you a large amount of time and effort. I’d prefer this method over running queries manually or archiving images that take a large amount of space and are nearly impossible to parse or search. You can find more information on using the Query API here: https://docs.newrelic.com/docs/insights/insights-api/get-data/query-insights-event-data-api

2 Likes

Option 1 - pay for more retention…

Option 2 - poke the API’s on a regular basis. This is the kind of thing I’ve done in the past to solve a couple of areas where on-premise data has been a requirement. We still have and use insights heavily for more general purpose visualisations)

You’re going to want to loop round the insights data types that you want making the following type of query:

SELECT ... FROM .... since 'YYYY-MM-DD HH:MM:SS' until 'YYYY-MM-DD HH:MM:SS' LIMIT 1000

If you get 1000 results back, you’ve overflowed and need to repeat the query with a shorter time range until you get <1000 results. If you don’t get, say over 500 results then you’ll want to make your time interval longer on the next query so you make efficient use of the API.

This can be made via API call and you can easily iterate this approach across a number of accounts and a number of data stores in insights. Next time you run the query, just pick up again from “last known timestamp - 5 mins” and cary on.

Given you get JSON data back, poking this into an ELK stack is really trivial. Just make sure you add the account number / account name into the records before pushing the data in.

Being a good API citizen, don’t think of this as an “end of month” thing, more of a regular feed of your data from NR into your existing datastore. You want to be trickling the data across on a little and often basis rather than doing a big end of month batch.

If you only need archive log data then this should be all you need. If you need visuals then you’re going to have to pick a new tool to recreate what you’ve got in insights. In this area, if you’re looking for something good looking, free and with the same kind of really short learning curve as insights, you’re probably going to be disappointed. Good tools aren’t free and free tools aren’t as easy to learn.

6 Likes

Thank you @pault and @babbott for your replies.

I have been digging more into this and come up with this approach, kindly share your thoughts.

  1. Create a Query in insights that will give me data table for apdex monthly.
  2. Extract that data and feed it back to insights as a custom data set.
  3. by adding each month I will have this data with 1 records per month for apdex value that I can use for historical.

Only thing Is I need to script all this for automation and self processing. my end goal is to have apdex per month per environment at end of every month. So say in end of march, have jan, feb march apdex per environment and keep going.

Unfortunately retention for now is a bottle neck the option above is all I can think off.

Kindly share your thoughts and inputs.

Help is much appreciated.

thanks
Manas

2 Likes

Hi @MKhanna - You are aware that apdex performance for applications is a weekly report that you can receive? You can even view a sample report to see if it meets your requirements before starting to write your own solution.

3 Likes

Hi @stefan_garnham thank you for the information. I am aware but I cannot figure out how to use this for yearly metrics.

on other note, any tips on query for that weekly report?

thanks
Manas

If you do this at the end of each month, will you not still have the same retention issue you’re trying to solve? This puts you back in the “buy more retention” or “store it outside of new relic” options.

You mention to start with that you only require information for audit purposes. A later comment suggests " 1 appdex value per month (per app?)" The first requirement suggests you have an infrequent ad-hoc requirement to evidence data. The second suggests you have a very small set of data you’ll need to evidence. Neither yet suggest you have a visualisation challenge to solve?

You’ll want to run something like the following on a daily basis via the API:

SELECT apdex(duration, t: 2) FROM PageView SINCE 70 days ago 
facet appName, MonthOf(timestamp) LIMIT 1000

This makes a few assumptions:

  • You have a fixed appdex of 2 across all apps
  • The appdex granularity is per app, not based on transaction types, URIs etc
  • Your looking at browser appdex, not APM
  • You have less than 333 applications you want to record appdex against.

If any of these aren’t appropriate, you should get the gist of how to tweak the NRQL to meet your requirements…

Once you receive this data, you’ll want to ignore anything from “the month before last”. It won’t be accurate as it will only be looking at the last few days of the month. You can always trust “this month and last month” figures as the appdex FACETing always covers those months entirely

Take the results and store this in whatever is your internally supported / default database. Just set the table unique keys to app name, timestamp and appdex and do something like an “INSERT … ON DUPLICATE KEY UPDATE appdex”

The reason for running across “at least 2 months” is so that once you put this in place, you don’t need to panic about random failures happening at worse-case timing and leaving you with data gaps. As long as failures don’t persists for 2 months, your data will be sound. The other benefit is that your “this month” figures will be constantly updated during the month.

As long as you only have an audit requirement and not a visualisation requirement, you should now have everything you need?

2 Likes

Thanks @pault if I get the data out to my database then we loose the consistency with the visualization.

I figured out one more thing to take screen shots of the page using syntehtics, so basically do a script to run check on dashboard and take screen shot, and that will be on end of month, so that will be there for 1 year i think.

Thanks for the help and suggestions.

SOOO many great things shared here in this thread! Thank you @MKhanna, @pault, and @stefan_garnham! :blush:

1 Like

Forgot about this post, if you are reporting on your apdex metrics then you may find this is more like the SLA Reports if you have an agent which records apdexPerfZone.

4 Likes

Thanks for the addition and the razor sharp memory @stefan_garnham

1 Like