Adding an Alert to an Event Alert

We’ve been using the Events API to add in data from our Java/Test-NG framework to track activity on some of the APIs we have in DEV/QA. I have an alert that checks on specific ones and raises an Alert and sends a notification to Teams when the API goes down. For now we just close the incident on the Alert after a couple of hours, and it’s ok for the moment.

I want to add an alert. or notification when things are better. I was hoping to set up something that checks for an open incident and then the status of the API, if it looks good after some time then we should be up and I can send an all good notification. Or, if there is a way to send a Teams notification when the incident closes that would be best.

I haven’t digested all the new Alert documentation yet, is there something that can do what I want within the same alert (two messages) or something that checks for the Down Alert and then when I get a pass condition can send out an It’s Up Alert.

Hey @FurmaniukM

Alerts Incidents can each have a max of 3 notifications. When the incident opens, when it is acknowledged, or when it closes.

Violations do close automatically when the reverse of the condition is met.

A basic example of that is; a condition looking at response times going above 1s for at least 5 minutes is triggered.

Once that applications response time is below 1s, for a continued 5 minute period, the violation recovers, which can trigger the incident to close.

Once that incident closes, a notification is sent.

So I don’t think you need to configure anything extra. You should be receiving an Incident Open AND an Incident Closed notification.

1 Like

We typically don’t acknowledge, people handle the issue outside the alert. Perhaps I am mixing things but for this NRQL type alert I have a policy that sends me an Alert when it opens. Currently I have an auto close when we are no longer in the threshold. If I can send a notification for when it gets closed where do I set that?

I’m not seeing an option to add a notification on auto close.

There is no setting for that - even when you manually close a violation / incident, a close notification is automatic, and should go to all of the same notification channels as the alert open notifications.

If you can share a link to an example closed incident, I can check whether or not it has sent closed notification.

1 Like

Thanks Ryan.

This is one incident, with the NRQL alert I have them auto close after a few hours as I haven’t found a way to automatically do it in the Alert and send a notification.

https://one.newrelic.com/launcher/nrai.launcher?pane=eyJuZXJkbGV0SWQiOiJhbGVydGluZy11aS1jbGFzc2ljLmluY2lkZW50cyIsIm5hdiI6IkluY2lkZW50cyIsImluY2lkZW50SWQiOiIxNTExNjg3ODEifQ==&sidebars[0]=eyJuZXJkbGV0SWQiOiJucmFpLm5hdmlnYXRpb24tYmFyIiwibmF2IjoiSW5jaWRlbnRzIn0=&platform[accountId]=1626857&platform[timeRange][duration]=604800000&platform[$isFallbackTimeRange]=true

Hey @FurmaniukM

I see in that condition you have set to auto-close violations after 2hrs. However the incident you send did not need that 2hrs to close. As you can tell by the timeline below, the entire incident lifecycle was ~29 mins.

If you look at the events tab in the link you sent, you can see the sequence that took place.

So the timeline here is:

Event Time
Critical Violation Open 11:47am
Incident Open Notification 11:50am
Violation recovers 12:02pm
Incident Closed Notification 12:19pm

This close notification will go to that exact same Webhook channel you have set up for the policy.

Could you check on where these notifications are intended to go? There may be some logs on that side to say why you may not be getting closed notifications.

1 Like

These notifications got to a Teams channel using a webhook I have setup.

I do see both notifications, with the channel notifications they mention that the Alert has been triggered, and in this case something is down. So I get two down alerts.

For the Close I would like to have something that says it’s up. Can I script that in the notification? That would actually solve my issue.

I signed up for the Alert course but it’s not for a couple of weeks, I’m hoping to fill in some knowledge gaps at that point.

Hey @FurmaniukM

That’s great! The Alerts course with New Relic University is a good one, I’m sure you’ll learn a lot!

As for your current situation - as we discussed, there are 3 possible, but 2 guaranteed notifications in an alerts lifecycle. Open and Closed are the guaranteed ones, and Acknowledged is optional.

All of these notifications hit the same notification channels.

You can see here that your webhook going to teams is set up with a custom payload.

This payload is hard-coded to say ‘Service is down’, so even for a closed notification you will always get that ‘Service is down’ message.

Webhooks do have optional variables you can use that come from the incident. So your notifications can be more closely related to the events taking place.

More specifically in your case I think $EVENT_STATE is the one for you, that’ll allow your notification to tell you OPEN / CLOSED / ACKNOWLEDGED.

https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/alert-notifications/customize-your-webhook-payload#variables

1 Like

Thanks this is what I thought I would have to do, is there an example somewhere about how to do this in the JSON payload?

Thanks!

Hi, @FurmaniukM: You may find this post helpful:

1 Like

Thanks Phil, I have looked at that but it is more focused on setting up the message with fields, I am not seeing anything in there that helps me branch the message depending on the EVENT_STATE so I can send two different messages to the same channel with the same notification using the JSON structure.

Hey @FurmaniukM

The JSON in the example that Phil sent includes the title line:

"title": "**$CONDITION_NAME incident $EVENT_STATE**"

That is, pulling in the event state Opened / Closed / Acknowledged

That should get you what you need to your Teams channels