Alert Showing Critical Violation but not Opening an Incident

Hello there :slight_smile: We have an alert that tracks the error rate of our lambda functions, and we have noticed that, despite it showing that there are critical violations, it still does not open up an incident for these violations.

The policy this condition is in is configured to categorize incidents by condition and entity, and we have tried recreating the condition and the policy in attempts to get the rule to open up an incident, but with no dice :confused: Anyone knows what might be happening here?

Hi @newrelic134 could you please share a link to your alert condition so that we could take a further look into this for you?

Thanks @nmcnamara ! Here is the link.

@nmcnamara Do you guys happen to have any update on this? :slight_smile:

Hi @newrelic134,

It has been a while since I have supported Alerts (I primarily support Synthetics these days), but I have had a chance to look into this and I believe I have an idea of what your issue here might be.

Your condition is based on this query: SELECT sum(provider.errors.Sum) * 100 / sum(provider.invocations.Sum) FROM ServerlessSample FACET provider.functionName``. It is set to open a violation when query returns a value above 3 for at least 60 minutes.

Looking at a 1 minute TIMESERIES view of the data in Dashboards, I can see that some facets here are returning 0 values and I believe that might be causing some issues here. Adding a signal lost threshold and gap filling where the 0 values are might solve your issue here. Any chance you’d be willing to give that a shot and let me know your findings?

1 Like

@Masen You’re a life saviour! :smiley: Gap filling with the previous value solved the issue! Thank you so much!! :raised_hands: