Open Incidents by target

Hi all,

I have reviewed the documentation about Alerts and it seems there is no way to group violations by targets. The behaviour I need is open only 1 incident by target regardless of the condition it fails.

Example:
Policy 1 has condition1, condition2 and condition3 is applied to host1 and host2. If host1 goes down, I only want one incident grouping all its conditions. I assume if the host is down, condition 1,2 and 3 will be violated. And if host2 is down, I would want another incident regarding host2 issues. I think with the default Incident preference, I would be opened only 1 incident with all the violations.

Thanks in advance

Hi @juanalbacar -

You’re right in your final point that:

I think with the default Incident preference, I would be opened only 1 incident with all the violations.

By default an incident preference is set to By Policy, meaning that any violations, no matter the target, will be rolled up into one incident.

You can set that incident preference to By condition and entity, which will allow you to break your alert incidents out, such that any violation, for any entity, gets it’s own incident.

It sounds like that may be what you are looking for.

This Level Up Post is a great resource for understanding Incident Preferences.

thanks @RyanVeitch for your response.
I think in the situation when a server is down, in that case I will get n incidents. Our SRE team is going to be mad… you know.

Is there any way to group incidents? or relate them?

I completely understand that, Alert Fatigue is a real problem.

I think it’s important to run a frequent alert audit, make sure that anything that can wake up your teams is actually something they need to be woken up for.

Regardless of that - In terms of the right incident preference for you - that depends on your needs,

By Policy =>

You’ll get 1 Incident for any violation within that policy for any condition and any target. Everything rolls up into that incident until such time that it is closed - paving the way for a new one.

By Condition =>

You’ll get a new incident for the first violation in any condition within the policy. If that condition throws a number of violations while that first incident is still open, those violations will roll up into that incident.

By Condition and Entity =>

You’ll get a new incident for the first violation for any target in the conditions within the policy. If for example you have conditions targeting an APM app and that app has 5 violations in in short succession, you’ll still only get 1 incident for that so long as that first incident is still open, since it’s the same condition, and same target.

There is no way to get a incident per target, regardless of condition.

I think for you it may be best to go By Condition - You’ll get verbose notifications, but not too verbose.