Behind the scenes in Infrastructure Alerts creation
Creating an Infrastructure alert condition is easy through the UI. What is not apparent, however, is what is going on behind the scenes when you’re setting up your filtering in the alert condition.
When you set up a filter as part of the process of creating an Infrastructure alert condition, what’s really happening is that you’re creating a
WHERE clause in an invisible NRQL query.
- Maybe your filter set is hostname contains “dev”. Behind the scenes, this becomes
WHERE hostname LIKE '%dev%'.
- Perhaps you’ve chosen to filter by region, so your filter looks like AWS REGION: us-west-2. The hidden NRQL query now has
WHERE awsRegion = 'us-west-2'added to it.
Trouble in paradise
I hope that is clear and easy to understand. With simple filter elements, the
WHERE clause in the hidden NRQL query is short and it gets the job done admirably.
The problem comes in when the filtering in an Infrastructure alert condition gets complex. Since a NRQL query is being used behind the scenes to implement your filter set, the 4096-character limit on NRQL queries comes into play. If your
WHERE clause starts getting too big, this can stop you from adding any further filter elements.
For example, if your filter set involves selecting hostnames manually, each hostname needs to be added to the
WHERE clause. This can get very long, very quickly. If your hidden NRQL query is approaching the 4096-character limit, you may not be able to add any further filter elements to your Infrastructure alert condition. This can really ruin your day if you have an alert condition with (for example) 159 hosts manually added by hostname and you suddenly find yourself unable to save the alert condition when trying to add further hostnames to the filter.
Best practice for filtering in Infrastructure alert conditions
Manually selecting hostnames to add to an alert condition is difficult to scale and, once you have added enough hostnames and your hidden NRQL query’s length approaches 4096 characters, will eventually cause an error when trying to save the condition.
Due to this, we recommend using attributes to filter on instead. Using pre-existing attributes is one option, but this will not always be able to meet your needs (i.e. if you want all the hosts running SQL Server). Using part of the hostname is another option that will keep your
WHERE clause manageable (see the example above), but may also fall short of meeting your needs.
This is a great time to implement custom attributes, which will allow you to add more hosts to any alert condition by simply adding the custom attribute to the host’s Infrastructure config file. Removing hosts also becomes simple, since by removing the custom attribute from the host’s config file, it will be automatically removed from any alert conditions which use that attribute as a filter.
I hope this helps to better understand how Infrastructure alert filtering works behind the scenes, and how to more easily set up smoothly scaleable Infrastructure alert conditions.