Relic Solution: How To Achieve Compound Alert Behavior

Support: Alerts/Applied Intelligence (AI) Alerts
, , , ,
#1

Compound alerts

Compound alerts are cool, and a commonly requested feature. What are compound alerts? Simply stated, it’s when you set up multiple thresholds that are dependent upon one another. For example, I could set up a condition that would violate when my app’s CPU usage went over 90%, and another alert condition that would violate when my app’s memory usage went over 90%. A compound alert would not open an incident unless both states were present.

Up until recently, it has not been possible to set up compound alerts out-of-the-box in New Relic. Now that we have workflows, however, you can do it – I’ll show you how below.

How to get notified on a compound alert

Configuring conditions

  1. Select Alert conditions (Policies) on the sidebar.
  2. Choose an existing policy or create a new one to contain your conditions.
  3. Create conditions according to your needs in the policy.
  4. Make sure Incident Preference is set to By policy.
  5. Make sure the Correlate and suppress noise box is NOT checked.
  6. (optional) If you only want “compound alert” notifications, make sure that no notification channels are attached to the policy.

Configuring destinations

  1. Select Destinations on the sidebar.
  2. Create one or more destinations where you would like to be notified.

Configuring a workflow

  1. Select Workflows on the sidebar.
  2. Click on Add a workflow to create a new workflow.
  3. Under Select issues, click Build a query.
  4. Click Select or enter attribute.
  5. In the Filter your data section in the new workflow choose attribute conditionFamilyId, operator exactly matches, and enter the value of the first condition ID (for the condition id location see Screenshot 1 below).
  6. Click on the + AND button and repeat the process with the second condition id (see Screenshot 2 for an example of what this should look like).
  7. In the Notify section choose your configured destination and decide what the payload is going to look like.

Screenshot 1:

Screenshot 2:

**

And that’s it! This workflow will only open an Issue (and thus send notifications) when both conditions have opened an incident.

8 Likes
Alerting based upon multiple related conditions
Calculate percent of users affected by some problem
Create concurrent conditions for alerting
Use the result of two monitors to trigger an alert
#2

Dear @Fidelicatessen ,
Thanks for this topic!
Could you please answer the following question:
I get notifications when the last condition from the compound alert ended.
Is it possible to get a notification, in the beginning, when all the conditions are met?

[Example of what do we have right now:
The workflow contains two conditions C1, and C2;
Let’s say C1 started violation at 10 am and ended at 11 am
C2 started violation at 10:30 am and ended at 11:30 am
Then I will get the notification at 11:30 am (when C2 ends)
What I would like to have, is to receive the notification at 10:30 am when C1 was violated and C2 just started ]
Best,
Roman

#3

Hi @Roman.Semenko

Sorry about the misunderstanding! I just learned that this is a limited release – we’ll work on getting your account(s) added to the feature as soon as possible.

1 Like
#4

[update]
Thanks, now I am getting a notification when the last needed condition starts.

1 Like
#5

Hey @Roman.Semenko,

Thank you for letting us know you are now getting the notification like expected. Please reach out if there is anything else we can help with. I hope you have a great day!

#6

Hi,

Thanks for providing this feature. Is it possible to use this “compound alert behaviour” when building up queries through NRQL? I essentially wanted to have an alert that gets triggered only when Condition1 AND Condition2 are met. I’ve tried doing this by using sub-queries only to be told that sub-queries are also not supported in streaming alerts =(

This has been widely requested on this thread Alerting based upon multiple related conditions but I’m not sure that this feature provides a way of doing it.

Thanks!

#7

Hi @leonardonickel

Unfortunately, you can’t yet do this through alert condition configuration. But you can effectively simulate the behavior with your notifications. This method will not change the behavior of incidents and issues opening, but it will change how you get notified, so that notifications only go out when there is an issue containing incidents from both Condition1 and Condition2.

If you let us know the use-case you’re trying to achieve with a compound alert condition, I will upvote the feature request with your information. Please include details on why the method detailed above isn’t working for you.

1 Like
#8

Hi @Fidelicatessen thanks for your quick reply.

First of all I’m super new to New Relic alerts so apologies in case I say something stupid =P

My use-case is quite common and I’ve seen many people requesting the same feature from New Relic Alerts: I’d like to be able to have a query in my alert NRQL with the following conditions:

  • amount of errors divided by the amount of requests in a given period
    AND
  • a minimum amount of requests in the given period so in case of low traffic we don’t trigger an alert when there’s, for instance, 4 requests and 2 errors which would configure 50% of errors.

I’ve initially tried with sub-queries only to discover that sub-queries are not supported in streaming alerts. Then I’ve tried with something similar to a UNION clause but that is not supported either. Then I’ve found out this thread Alerting based upon multiple related conditions where other people were asking for alerts based on multiple conditions that led me to this “Compound alerts” feature.

My alert configuration is pretty simple, I have the NRQL and then a threshold of 5% for the logic mentioned above. However without being able to combine conditions or use sub-queries it generates a lot of false positives (like yesterday we had 2 requests and one resulted in error hence the alert got triggered). I’m sending below my alert configuration but I don’t think it helps much, it’s a standard one.

Hope my comments make sense, thanks for your time!

Screenshot 2022-08-05 at 16.25.23
Screenshot 2022-08-05 at 16.25.231884×1520 260 KB

1 Like
#9

Hi @leonardonickel

Thank you for your detailed response! You absolutely did not say anything stupid, and this is valuable information for us folks on the product side of things.

I do have a question for you, though: will the method detailed at the top of this thread work for you? If you only get notified when both conditions are alerting (e.g. request count above a certain number AND error rate above 5%), will that fulfill your use-case? If not, please let me know why so I can include that in my write-up.

Thanks again for including good details. It’s very clear to me what you’re trying to do, and hopefully the method that uses workflows to silence your notifications unless both thresholds are being breached works for you (but I’m very interested to know why it doesn’t, if it doesn’t).

#10
#11

Hi @Fidelicatessen the solution in my opinion looks more like a work around instead of a proper compound alert behavior ? Is that the recommended why to achieve compound alerting at this moment ?

I am working in a NR set up with many teams in the same account and would not be possible to advice that as there is a limitation of workflows also still does not smell as the right approach to me.

Thanks

#12

Yes, that’s exactly right. This is a way to achieve the behavior in most use-cases, although it is not proper “compound alerting.”

#13

@Fidelicatessen Hello! We’d love to try out this compound alerting approach, but I’m having an issue finding how to build a query as part of creating a workflow. Is this something that needs to be enabled in my account?

Thanks for the guide!

#14

Hi, @ethan.evans1: Just click on Select or enter attribute under Filter data:

SCR-20221101-g26

#15

Thanks @philweber! I’ve given that a shot, but I’m still not able to get the workflow to fire with the following settings:

image
image1148×494 33.9 KB

Both of the conditions have active issues created. I’m worried that the workflow is not triggering because it is looking for one issue that has both of the IDs. Have you been able to use similar settings and achieve the compound alerting behavior?

#16

Hi @ethan.evans1

Both of the conditions have active issues created. I’m worried that the workflow is not triggering because it is looking for one issue that has both of the IDs.

That’s exactly it. Workflows always searches for Issues that exactly match the filter set configured. In this particular case, it’s looking for an issue that contains at least one incident from each of those conditions.

In other words, if there is one issue with an incident from one condition, and another issue with an incident from the other condition, then the workflow will not fire – it explicitly needs one issue with incident(s) from both conditions.

The easiest way to make sure this happens is to use the By policy issue creation preference setting in the policy that contains both conditions. If both conditions are not in the same policy, this isn’t going to work reliably.

Have you been able to use similar settings and achieve the compound alerting behavior?

Yes, but only when both conditions are in the same policy and the policy’s issue creation preference is set to By policy (so that only one issue will be created in the policy, and all condition’s incidents will roll up into it).

This does not work perfectly – for instance, one incident may open for entity A, and another incident will open for entity B, but you’ll get notified even though neither entity A or B met both criteria. Keep in mind this is a workaround to simulate true compound alerting behavior.

2 Likes
#17

@Fidelicatessen Thanks for the great explanation! Workflows seem super flexible, so it’s cool they support some new behaviors like this. I’ll give it a shot!

1 Like
#18

Hi @ethan.evans1

Thanks for the feedback! As usual @Fidelicatessen explanation is on point :man_superhero:

1 Like
#19

Thanks for this tutorial was the solution to my problem!

Just one question:

How to terraform the AND conjunction of the two conditionFamilyId


resource "newrelic_workflow" "workflow-example" {
  name = "workflow-example"
  muting_rules_handling = "NOTIFY_ALL_ISSUES"

  issues_filter {
    name = "Filter-name"
    type = "FILTER"

    predicate {
      attribute = "accumulations.conditionFamilyId"
      operator = "EXACTLY_MATCHES"
      values = # This is a list of values but how I can use the AND operator to 
               # trigger the condition of the two accumulations.conditionFamilyId? <===========HERE
    }
  }

  destination {
    channel_id = newrelic_notification_channel.webhook-channel.id
  }
}
#20

@jorge.llanos
I’ve been working to build up Workflows for this and if you do this… it will add 2 filters to your workflow with an “AND” between them.

resource "newrelic_workflow" "workflow-example" {
  name = "workflow-example"
  muting_rules_handling = "NOTIFY_ALL_ISSUES"

  issues_filter {
    name = "Filter-name"
    type = "FILTER"

    predicate {
      attribute = "accumulations.conditionFamilyId"
      operator = "EXACTLY_MATCHES"
      values = ["condition1ID"]
    }
    predicate {
      attribute = "accumulations.conditionFamilyId"
      operator = "EXACTLY_MATCHES"
      values = ["condition2ID"]
    }
  }

  destination {
    channel_id = newrelic_notification_channel.webhook-channel.id
  }
}
2 Likes