Customer Solution: New Relic Alerts Configurator

When you have dozens of services running across multiple environments, configuring the various alerts for each service is a lot of repeatable work which is also error-prone. As it turns out, human data entry can be pretty inconsistent! Here at Ocado, we wanted to find a way to configure alerts policies programmatically, which would ensure that our Alerts policies are consistent, and our configuration is not changed by an accident.

To solve this problem, our Panda Team built the New Relic Alerts Configurator. With this tool, we are able to automate whole process. It reads configuration from repository and performs required changes to Alerts policies via New Relic REST API. We have extracted common alerting policies to a shared space that is used by all services and separated services that have unique policies.

Changing both common and unique parts of the configuration became really easy. This had tremendous impact on our monitoring process. Introducing any change in Alerts policies now takes very little time so we feel confident to fine tune our policies to check for specific abnormal behaviours. This in turn allowed us to act quickly whenever service had problems. That means that we are able to save man-hours in setting up alerts, AND we are able to make our alerts more meaningful.

We found it so useful that we wanted to share it with other New Relic customers. Here’s the summary available in the GitHub repo:

New Relic Alerts Configurator can be used to configure NewRelic alerts for your application. Instead of defining alerts through UI you can define them in code. It allows you to automatize alerts configuration, easily recover them in case of wipe out and have full history of changes in your version control system.

We hope you find it as useful as we have and look forward to hearing about how you are able to use it!


New Relic edit:

We want to extend a huge thanks to the Panda Team at Ocado for making this available to New Relic customers! Please note that this is offered for use as-is without warranty. You are free to use and modify as needed. It has been created for use with New Relic Alerts, but is not a supported product of New Relic.

10 Likes