How to get tags passed out of Custom Webhook JSON payload?

###Alerts Question Template

• Please describe what are you seeing:

I have added tags to my entities, in addition to the default tags imported from the agent. I expect to see these tags as part of alerts within NR and sent to external systems via webhooks.

• How does this differ from what were you expecting to see?

No tags are present in the custom webhook payload sent in an alert.

• If you aren’t seeing expected alert or data, please provide a link to the incident or violation (policy, condition, data app etc.)

I created a custom webhook with all available JSON objects (as seen in docs) and none contain a tags object. There is an empty label object within the $TARGETS object but I’ve never seen it populated.

Please provide guidance on how to pass all tags out of a custom webhook.

Helpful Resources
Relic Solution: The key to consistent alert notifications

Troubleshooting downtime document

Hello @doug17, The only variables that can be used in the webhook payload are the ones listed in this document .

I understand that you are looking to setup dynamic custom tags for the webhook payload. At this point, our product does not support this.

You may be able to work around this issue by adding a runbook URL to the condition and create a dashboard to show these details, once a certain clustername is entered (by using facet linking ).

You can also add static values to your webhook, but these would be simple static values, not variables.

If the target of the condition (and hence the violation) is a cluster entity, then clustername will be included in the $TARGETS variable. If you facet on a tag, that will be populated in the target’s name field.

This is a tremendous gap in the product, especially with this new API and tagging feature. Makes things very difficult especially when other vendors have rich tagging capabilities across their UI and alert notifications/webhooks.

What should be included in the Targets:Labels object? It’s blank every time.

Please pass this along to the most appropriate product owner. I’m happy to help with guidance and feedback for the customer’s I represent who are trying to use NR with all of the best practices available.

Thanks for sharing your thoughts here. We always appreciate valuable feedback - and we’ll absolutely get this in front of the Alerts product manager. If there is anything you would like to add, feel free to direct message me, and I can add on to the communication to the Alerts PM.

Is this still the case with alerts?

It is! Could you outline your use case (if different to that already shared in this thread), so that I can get your +1 added with as much info as possible for our product managers to review.

With our use case we are using tags in our synthetic monitoring and want to pass through to our monitoring system in order to enrich the alert with our CMDB and provide additional information. We initially were using labels. From my understanding those were supported in the alert but have been deprecated.

Hi @dave.nelson1,

Labels were deprecated, it’s true, but they were automatically migrated to tags. You can use them in NRQL queries by simply adding tag. to the label name. So, for example, if you had a label of environment: prod, you would query your new tag like this:

SELECT * FROM SyntheticCheck WHERE tags.environment = 'prod'

We did have a problem where Synthetic tags were not coming through in the $TARGETS variable in a webhook payload. However, that issue has been addressed, and you should be able to use that variable in your webhook payload to get all the tags associated with the Synthetics monitor that caused the incident.

I hope this helps!

Hi @Fidelicatessen, thanks for sharing the WHERE tags.environment = 'prod', I don’t think I’ve seen that documented anywhere. Do you happen to have a link to learn more about it?

I ask because I have a number of a tagged APM applications with Purpose:Migration, but I can’t seem to return any results with the following query.

FROM Transaction SELECT count(*) FACET appName WHERE tags.Purpose = 'Migration'

Hi @rishav.dhar

Currently, only Synthetics events (like SyntheticCheck) can use tags in this way to filter on a query. The Transaction event doesn’t yet have tags included on it, although our engineering team is working on getting this added.

I don’t think there is any documentation on this yet, but I’m going to reach out to the documentation team to request that they put some information in about how to query tags – thanks for mentioning this!