Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: How to Use the Infrastructure Alerts REST API to Its Maximum Potential - Part 1: Exclusion Filtering

restapi
newrelic-infra
infra
rest
alerts
api
levelup
infrastructure

#1

When you build Infrastructure Alerts conditions you’ll have criteria which the conditions need to satisfy so that they’re useful to you. The UI, however, might have certain limitations when helping you satisfy that criteria. If you run into these limitations then you’ll want to use the Infrastructure Alerts REST API to help you get around them.

In a series of posts I will go over each workaround individually. Even more exciting is that each of these workarounds can be combined in any way you think is necessary. For this reason you may notice that the posts sometimes repeat the same information. Please have a look at the Infrastructure Alerts REST API documentation to make sure you have a general familiarity with the technology I’ll be talking about:

Our first example pertains to exclusion filtering. Exclusion filtering is similar to an SQL clause of NOT IN, NOT LIKE, or !=. In the Infrastructure UI you are able to create filter sets defined by certain attributes. You can then create alert conditions based on them:

However, the Infrastructure Alerts UI is not able to ignore a subset of attributes in order to provide a more finely tuned filter set unless a new attribute is created for the hosts which you don’t want to select. Say you have a small subset of a large and useful filter set which has hosts with ‘DB’ somewhere in their hostnames. If all you want to do is select the whole filter set and exclude that handful of DB hosts then you can create a query which uses AND hostname NOT LIKE '%DB%' and then put this query into a POST API call. Here is an example query you can keep in mind:

SELECT (memoryUsedBytes/memoryTotalBytes)*100 FROM SystemSample WHERE `environment` = 'PROD' AND `hostname` NOT LIKE '%DB%'

Here is what the same query might look like in a POST API call. I will break it up to label its anatomy and then show you the completed call. Here’s where the curl is invoked and the alert condition is given a general classification:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{
      "type":"infra_metric",
      "name":"Non-DB Memory Usage Percent",
      "enabled":true,
      "policy_id":{policy_id},

Memory usage percent is calculated instead of hard-coded. When we set our "event_type" as "SystemSample" the query is automatically going to FACET by hostname because hostname is the domain of SystemSample:

      "select_value":"(memoryUsedBytes/memoryTotalBytes)*100",
      "event_type":"SystemSample",

Here’s where you can use exclusion filtering with NOT LIKE. The tricky part here is that each and every single quote must be escaped with '\'. Backticks do not have to be escaped:

      "where_clause":"(`environment` = '\''PROD'\'' AND `hostname` NOT LIKE '\''%DB%'\'')",

Next, the threshold will be violated if a value goes above what he have set. The "time_function" of "all" represents a threshold time function of for at least whereas a "time_function" of "any" represents a threshold time function of at least once in. The critical threshold below is the equivalent of above 95 for at least 10 minutes while the warning threshold is the equivalent of above 90 for at least 20 minutes:

      "comparison":"above",
      "critical_threshold":{
         "value":95,
         "duration_minutes":10,
         "time_function":"all"
      },
      "warning_threshold":{
         "value":90,
         "duration_minutes":20,
         "time_function":"all"
      }
   }
}'

Here is the completed API call:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{
      "type":"infra_metric",
      "name":"Non-DB Memory Usage Percent",
      "enabled":true,
      "policy_id":{policy_id},
      "event_type":"SystemSample",
      "select_value":"(memoryUsedBytes/memoryTotalBytes*100)",
      "where_clause":"(`environment` = '\''prod'\'' AND `hostname` NOT LIKE '\''%DB%'\'')",
      "comparison":"above",
      "critical_threshold":{
         "value":95,
         "duration_minutes":10,
         "time_function":"all"
      },
      "warning_threshold":{
         "value":90,
         "duration_minutes":20,
         "time_function":"all"
      }
   }
}'

Your condition "name", "policy_id", "select_value", "where_clause", and threshold values will probably all be different from what you see here.

Here are the other workaround posts which are coming up:

Part 2: Compound Alert Conditions
Part 3: FACET more than 500 hosts
Part 4: Cloud Integration Metrics & Evaluation Offset


Feature Idea: Add 'does not contain' filter to infrastructure & it's alerts
Infrastructure agent reporting high server memory usage because Lock pages in memory set on SQL server
Feature Idea: Allow filtering by exclusion
#2

You do realize that you’re proposing literal workarounds as a solution to problems in your own product? This is quite ridiculous. Just fix the problem! Don’t make paying customers jump through hoops.


Relic Solution: How to Use the Infrastructure Alerts REST API to Its Maximum Potential - Part 2: Compound Alert Conditions
Relic Solution: How to Use the Infrastructure Alerts REST API to Its Maximum Potential - Part 3: FACET More Than 500 Entities
Relic Solution: How To Use The Infrastructure Alerts REST API to Its Maximum Potential - Part 4: Cloud Integration Metrics & Evaluation Offset
#3

Hey @rdeknijf - You’re right! I absolutely agree with you, this is a workaround to a limitation. We do have Feature Ideas logged here for that, but of course our engineering teams need to prioritise the features they implement. I can’t say that exclusion filtering as a feature that is on a roadmap for the near term, but please know that our product team is aware of that limitation.

Gene’s goal with this post is to help with achieving exclusion filtering while we await that to be built in to the product. I’ll definitely get a +1 added to the feature request on your behalf though.

Feel free to DM me if you’d like to chat further about either this, or more generally about our feature idea process. :slight_smile:

-Ryan.