Relic Solution: How To Use The Infrastructure Alerts REST API to Its Maximum Potential - Part 4: Cloud Integration Metrics & Evaluation Offset


Our last examples pertain to cloud integrations. It’s possible to monitor resources which are hosted by cloud providers such as AWS. The data we display within the integration comes from the cloud provider. The Infrastructure Alerts UI metric selection dropdown is only going to show you metrics it knows about because the metrics have already hit your New Relic account. But what happens if you know about some hypothetical future metrics which have not been sent to your New Relic dashboard yet? How would you be able to proactively build alerts around them if you can’t select them from the dropdown? Fortunately this can be defined using the REST API. The how-to for this is provided in our official documentation:

Evaluation Offset

When trying to use NRQL alerts in conjunction with cloud integrations it can be difficult to avoid false positives. There must be some way to account for potential latency. With NRQL alert conditions we have what’s called Evaluation Offset. It’s basically a grace period during which Alerts waits for latent data to come in so that it can avoid false positives. The UI allows a minimum offset of 1 minute and a maximum of 20 minutes. The default offset is 3 minutes which is the equivalent of a query evaluating data SINCE 3 minutes ago UNTIL 2 minutes ago. More information is provided in the documentation:

When you string together the evaluation pipelines of NRQL alerts and cloud integrations then you need to pay attention to which polling interval New Relic is using to grab the cloud integration data and set your evaluation offset accordingly. Here are the polling intervals for AWS, GCP, and Azure:

Using the Infrastructure Alerts REST API, however, allows the alert condition to automatically snap to whichever evaluation offset is most appropriate. Here is an example of a REST API call which sets up a cloud integration alert condition:

curl -X POST '' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
    "name":"Low Credits",
    "where_clause":"`provider.cpuCreditBalance.Sum` != 0",

You can tell it’s cloud integration-related because it’s using the ComputeSample event which comes from EC2. This is the same payload as the one in Part 2:

With this knowledge you can proactively define alerts around metrics which have not populated yet and, if desired, avoid having to manually set your evaluation offset.

Part 1: Exclusion Filtering
Part 2: Compound Alert Conditions
Part 3: FACET more than 500 hosts