Custom webhook payload for Infrastructure alerts using labels?

Support: Alerts/Applied Intelligence (AI) Alerts
, , , ,
#1

We’re building out some alerting and event management standards for Infrastructure metrics. One of the values we’d like to pass it an out-of-band notification schedule (basically a name of a schedule like “ServerOncallSchedule”) that is then parsed out by our event management system and used to trigger that notification.

I see how to use labels in filter sets and filter sets in alerts, but I am trying to not make alerts too complex.

Is there any way to pass infrastructure labels via a custom webhook? I see that labels are passed in APM-based alerts via the webhook payload, but not in Infrastructure.

#2

Hi there!

While you can use labels in alerts that target APM applications, you can’t do this as yet with Infrastructure. Infra uses custom attributes which are handled differently. Perhaps a separate notification channel would work better for this use-case?

Regards,

Paul

#3

So we got a little sneaky to solve this problem :wink: We do event management in a 3rd party platform. Because we standardize the names of our notification schedules to align with the names of the receiving team, we were able to only pass a value that indicated which schedule was being passed in the Policy name.

For example, if my team was called App1SupportTeam our notification group would be App1SupportTeam-RS1. (RS = Recipient Schedule)

Our policy names are formed using the team name : [New_Relic_Module] : Recipient_Schedule : Policy_Type : <Optional_Text>. The hardest part was staying within 64 character limit :wink:

So a policy name might look like App1SupportTeam:APM:RS1:E or App1SupportTeam:APM:RS1:E:WebServices Our alert enrichment pulls apart the policy name and can assemble the recipient schedule as App1SupportTeam-RS1.

Why did we choose to go this route? It was easier to audit policy names vs auditing labels in our environment. Your mileage may vary, but as we often preach in our space, DATA IS HARD usually following by GARBAGE IN, GARBAGE OUT. :smile:

Be good data stewards. It makes everyone’s life easier.

1 Like
#4

Very sneaky indeed @jbiggley. Thank you so much for sharing your workaround here. I am certain that will help solve a lot of problems.

1 Like