Alert Severity the same for OPEN/CLOSED Incidents

Please paste the permalink to the page in question below:

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

I have a successful webhook integration from NR to ServiceNow. It will open up events when a synthetic URL Check fails and when it comes back up. But, in the payload the severity of the event, regardless of down/up, is the same: Critical. Shouldnt the severity of the “UP/RECOVERY” event be good or OK?

Hi @wademorris thanks for your question. New Relic currently does not offer 1st-class ServiceNow support and this issue is symptomatic of that. New Relic returns INFO, WARN, or CRITICAL so it’s likely seeing a boolean positive value as CRITICAL. The documentation on payload values can be found here:

Since this is the case it might be necessary to add a programmatic layer at the ServiceNow endpoint to parse this and convert the payload from CRITICAL to ‘good’ or ‘OK’.

I encourage you to read this post I wrote about webhooks which may help you troubleshoot future issues:

Cheers

I don’t think it has anything to do with ServiceNow. The payload that gets sent over from NR contains the wording “CRITICAL” with the UP/Recovery event. I can’t program Service Now to change that because the initial down event would be changed too.
I verified this behavior by implementing an endpoint with requestb.in and sending failures and recovery messages to it and both of them contain “CRITICAL” for the severity.

@wademorris in this case CRITICAL refers to the condition’s threshold that has been violated. The chain of logic here is that you’re getting a notification for an incident opening, incidents open when a new critical threshold has been violated and settings dictate that the violation should not be rolled up into an existing incident, the notification is sent telling you that a new incident has opened as a result of a critical violation. Incidents only close when all critical violations associated with the incident have closed, so the notification of incident close is also related to a critical violation.

This would probably be more obvious if we supported functionality for warning level violations to open incidents (triggering notifications) and that’s something that is being considered as a possible feature at some future point. No promises on whether that will materialize, though!

I completely understand your confusion and the request for a change to the payload. My explanation is purely that (explanation) and not justification :smiley: I think that the best course of action here is to file this as a feature-idea so that your feedback can make it up to @NateHeinrich, who is the Alerts PM. Please let me know if you have any other questions about what we’re doing and why.

Hello, I am working on similar case of integration between New Relic and Service Now.

Service Now can receive my alerts by webhooks and open it once New relic detects any failure, but Service Now is not closing it automaticaly because New Relic uses “Closed” instead of “Closing” (current_state: $EVENT_STATE).

Do you have any idea how to solve this? or any recommendation to do it?

thank you

@Renato.Malvino - There’s no way to alter the closed - closing text the webhook sends, and since we don’t have a direct integration with service now, there is limited support we can provide for this.

With that said, it feels to me that a feasible option could be to capture the webhook before it reaches service now, alter the text in a 3rd party tool and then send that on to Service Now?

(maybe a lambda function could be triggered by the webhook, read the data and change it to suit your needs, then forward it to Service Now).

We created a custom Scripted REST API in ServiceNow to take in the New Relic Payload and transform it in to something more useful. You should be able to alter it that way.

3 Likes

Thanks for sharing that @wademorris

As an alternative, you can also use Servicenow event mapping . ServiceNow uses Severity “0-clear” that causes the event to close . You can map New Relic event state (Closing) to service now severity (0-clear) in Servicnow event mapping.

3 Likes

Sounds like that could work well! Thanks for sharing @psharda :slight_smile: