Question about Synthetics check alerting

Hello, I have 2 synthetic checks, check-A and check-B. Check-A sends alert to Team-1 and Check-B sends alert to team-2. Check-B is a dependant of check-A, means it is expected to fail if check-A fails.
Is there a way I can stop check-B from creating an incident/sending alert to team-2, if there is a incident already open for check-A ?

@pradeesh.pezheril Sorry you have been waiting awhile for a response from our community. I’m going to bring this back to the attention of our support team. Thanks for your patience!

Neal Mc

1 Like

I’m following this.

My scenario would be a back-end webservice that uses an on-prem SQL instance is called by a front-end public website and internal tools (CRM). My 1st synthetic checks validate .com website is up/page loads. 2nd synthetic checks if webpage that calls back-end service is up/loads. We are now adding monitor for SQL to ensure no timeouts, no deadlocks, no CPU/Memory high consumption.

If SQL has timeout issues I’m going to also receive alerts on 2nd synthetic failing. I don’t want to get Synthetic alerts if the system already knows SQL is timing out.

It would cause me to waste resources/time digging into why Synthetic is failing when other team already knows about SQL.

So it would be helpful to somehow correlate the checks and only fire one alert. I do want to see the issue in Dashboards and SLA but do not need the extra email alerts.

Is there documentation on best-practice to setup alert correlation?

Hi @pradeesh.pezheril, @Craig.Tarr,

One way that I see of doing this is by creating the alert conditions in the same policy and setting the incident preference of the policy to By Policy.

This means that the first time a condition is violated under a policy an incident is opened and you will get notified. If any subsequent conditions or entities violate under the same policy they will be rolled up under the same incident but you will not get any additional notifications for these new violations.

For more information on incident preference type please refer:

2 Likes

Thanks for answering @zahrasiddiqa, however that doesn’t solve my problem. I want the alerts to be sent to different teams based on what check is failing. Plus, it has to respect the dependency between checks/conditions and should not create additional incident.

Hi, @pradeesh.pezheril: One way to do this is to have the alert policy for Check-A use a webhook notification to create a custom event when an incident is opened or closed. Then use a scripted browser for Check-B to query NRDB and see if Check-A currently has an incident open. If so, do not execute the check.