Use the New Relic AWS Billing Integration to add cost as a new metric and show what your business is spending in AWS. Help manage your AWS services. Include this information in Insights alongside other New Relic products.
New Relic products manage a lot of things. But, are you using New Relic to manage your AWS cloud costs? That’s right, through our AWS Billing integration, cost is now a metric you can monitor and manage. To begin with, let’s take a look at this AWS Overview Dashboard screenshot with comments…
Since New Relic added the AWS Billing integration around May 19th I’ve had quite a few folks ask me how this integration works, where can examples can be seen, and why someone would use this.
So, the basic questions were boiled down to these:
- Can I show how much this application is costing me in AWS?
- Can I use this information to help me reduce my AWS spend?
- Can I be notified when I cross various monetary thresholds?
In this post I’ll give the basics of how to setup the AWS Billing integration both from the NR side as well as the AWS side. I’ll show some examples of Insights AWS Billing information using both a Dashboard and Data App and give some examples of how this information can be used. I’ll also give an example of how our customers can use our Alerts functionality to send them notifications based on their AWS services usage.
By the end of this post each of the above questions will be answered. But first, the basics - SETUP!
Set up the New Relic Side
If you haven’t already setup the AWS integration follow the documentation located here.
After that make sure your New Relic account has the Billing integration checked off starting with Figure 1 below.
Next, check off the AWS Billing checkbox shown in Figure 2 below. Then select Save changes at the bottom of the page.
Note: You may integrate more than one AWS account. The red marked “aAWS” shows what name I used for this New Relic/AWS integration.
Set Up the AWS Side
To setup the AWS Billing integration follow the documentation located here. It will walk you through configuring AWS to allow New Relic to capture billing information.
If you stop here you’ll only receive billing information from AWS. Specifically, the actual spend for the AWS services you consume. It is pretty basic. However, if you continue and setup Budgets in AWS you’ll be able to show some pretty useful information alongside other New Relic products.
First things first. Within AWS you have the ability to Tag the various AWS services you use. This post won’t go into the what, why, and how of Tagging in AWS. But, for this post take one application you’re using AWS for and Tag each component with a Key/Value pair that makes sense. Example: for each EC2, S3 bucket, ELB, EBS, Route 53, VPC, RDS, etc. you are using for that one application Tag it with:
Once you’ve Tagged your services, you’ll need to wait a little bit for AWS to get everything updated. Next, go to Billing within the AWS Console and create a Budget as seen in Figure 3.
Fill out the Budget details. Name example: Application [Name]
- You can create either a Cost (think money) or Usage Budget (think service consumed - example: number of EC2 hours used)
- If you don’t set an End date your budget will continue monthly
Next, Refine your budget. As shown in Figure 4 you’ll want to check off Tag and fill out the Key you want to look for. For our example use Application. Tip: sometimes you just have to start typing your Key before AWS displays your Keys in the drop down box. If AWS doesn’t find your Key you haven’t waited long enough for AWS to get everything updated after you applied your Key/Value pairs to your AWS services.
After you select the Key you will be presented with all of the Values for that key (i.e. your applications). In Figure 5 below you can see that I named my Applications A, B, C, and D. You may only see the one application you tagged [Application Name].
Select the Application for this Budget. You can skip the Notifications section, you’ll be doing this in New Relic Alerts.
Select Create in the lower right to save you budget.
Repeat these steps and create another Budget for the State/Production key value pair.
Back to the New Relic side. With that done you will start to receive AWS Billing information about your budgets within 1-4 hours. To view the basic AWS Billing Dashboards go to Infrastructure/Integrations and select the Billing (Budgets) Dashboard link next to the Billing integration.
Show Me [where I’m spending] the Money!
Once New Relic receives the AWS Budget information you can create Insights Dashboards and Data apps that use this information.
Figure 6 shows a high level overview of two applications hosted in AWS. This overview shows the number of EC2 instances that are part of each application as well as the Actual and Forecasted costs. Also included is an Infrastructure chart showing the average of some CPU metrics for each application.
At the bottom are three charts showing AWS Billing information for AWS services tagged with Production and Development. Also, a total AWS Monthly Budget is displayed.
You can use this information to also point out where you might reduce your AWS spend. For example, if your EC2 instances are oversized you’ll see that your CPU metrics are low. This is a good indication that you have an opportunity to scale down the size of your EC2 instances or reduce the number instances running.
Figure 7 is an example of a Data app showing more information regarding just one Application. Included in this are the addition of other AWS services related to this application (S3 and EBS) as well as New Relic Synthetics information (Resources load time).
Send New Relic Alerts
New Relic also offers Alerts. With Alerts you have the ability to receive notifications when your AWS spend exceeds various levels. A simple example is to be alerted when your AWS actual costs exceed a specific amount. You can also create alerts to notify you when you exceed 90% (warning) of your budgeted amount and another one when you exceed 100% (critical).
We started with three basic questions. Let’s go over them one by one.
Can I show how much this application is costing me in AWS?
Using the tagging ability AWS offers along with an AWS Budget you can show not only your actual AWS spend per application, but also the forecasted amount.
Can I use this information to help me reduce my AWS spend?
Showing your AWS spend alongside New Relic product information, Infrastructure for example. you can better understand your AWS usage. If Infrastructure shows that your EC2 instances are not heavily used (i.e. low CPU usage) you can either reduce their size or remove instances until your usage is higher and more in line with your AWS spend.
Can I be notified when I cross various monetary thresholds?
New Relic Alerts allows for multiple and varied threshold levels that will notify the proper people that your AWS spend has reached or breached specific levels.
The New Relic AWS Billing integration is very powerful in that it gives you and your business more visibility and control over what and how you use the AWS services that you run your business on.
But wait, there’s more. Don’t forget, you can see this on the Insights mobile app for free! Check out the two screen shots below…
The one on the left shows me the number of EC2 instances and the actual as well as forecasted AWS spend for Application A (other information is just a scroll away). The right one shows all of the AWS Budgets as well as the total number of EC2 instances I’m running.
PROTIP: Get this data by Region
From the AWS ‘My Billing Dashboard’ - Go to ‘Budgets’ and ‘Create budget’.
Fill out the ‘Budget details’ section - selecting ‘Cost’, etc.
In the ‘Refine your budget’ section select ‘Availability Zone’ and then select the zone or zones you want this budget to track.
Note 1: You will ONLY see zones that have AWS Services in them that are incurring cost. For the example below I have services running in 5 out of 6 zones. You will need to go back and edit this budget if you ever add services in a zone that didn’t initially display.
Note 2: You won’t have to edit the budget if a zone that once was incurring costs, later doesn’t. The budget will still have that zone.
Note 3: You can see where you can future separate Region budgets into each availability zone.
- Select ‘Create’ to finish this budget. New Relic will pick up this budget on the next AWS Billing integration poll. For me, it only took about 5 min to get data into Insights.
Here’s what it looks like in Insights for the two Region budgets I created.
NRQL for the US East widget above:
) as '$ Actual', latest(provider.forecastedAmount
) as '$ Forecast', max(provider.limitAmount
) as '$ Limit'FROM FinanceSample WHERE provider='BillingBudget' andprovider.budgetName
= 'US East (N. Virginia)'