Relic Solution: How-to: Customize workflow payload

Hurray, New Relic has a new and improved templating solution for notification and it can be used with many destinations: webhook, servicenow, email…

The new templates use handlebars.js which has the advantage of being open source, some users may be familiar with it already, documentation might be readily available outside of New Relic.

One caveat from porting prior handlebar knowledge or following external documentation too much, is that handlebar capabilities are dependent on what helpers have been installed (IE: if you search for ‘handlebar math operator’ you can’t apply these solutions). New Relic supports the ‘built-in’ helpers and provides 4 additional helpers: JSON, Equality, Contains, Replace

https://docs.newrelic.com/docs/alerts-applied-intelligence/notifications/message-templates

JSON helper should be used as much as possible, it flattens an object into properly formatted json.
The only situation where it cannot be used is if you need to concatenate 2 values together:
IE:

message_key":"{{accumulations.tag.dbInstanceIdentifier.[0]}}-{{accumulations.policyName.[0]}}-{{accumulations.conditionName.[0]}}",

A few tips before you rush and build custom payloads:

1- Think of issues as an array of incidents, this implies that any metadata coming from an incident is available as an array in the issue.

Hence, many variables will be arrays (ie: all tags), and you may access the first element of the array using variableName.[0] or loop over the array.

2- In order to know what metadata is available, dump it all first to a webhook and explore what you get.

A great tip to explore what’s available in the metadata is to dump the root objects you see in the editor autocomplete, and use the JSON helper to flatten it, then you can observe the output in your webhook.
I always start by exploring what’s in

“accumulations”: {{json accumulations}},
“annotations”: {{json annotations}},
“labels”: {{json labels}},
“entitiesData”:{{json entitiesData}},

Remember some values are only available when some entities types are part of the issue, or when the issue is in a particular state (IE: duration is only available for the close notification).

3- The content that will be available depends on the entity types that are contained in the issue

The content of the issue is dynamic, and might not all be there every time when the issue is first notified on.

Often policies may contain conditions on many entity types and mostly when there is a mix of infrastructure and application, the metadata available will be vastly different (think all these aws/k8s tags).
Trying to access array content of an undefined variable will fail the Handlebar execution and NO notification will be sent (see tip #7). You should ALWAYS check whether your variables are defined

{{#if accumulations.tag.aws.dbInstanceIdentifier}}{{json accumulations.tag.aws.dbInstanceIdentifier.[0]}}{{else}}“NA”{{/if}}

Always add an {{else}} and output some string, this makes it obvious when something isn’t working as expected.

4- For the 'payload Editor, the Send Test Notification and the Test Workflow to behave the best they can, create a workflow with a rather strict filter, make sure the workflow catches only the issues you want to build that payload for.

5- Enricher syntax IS NOT the same as the payload customization syntax, you cannot use the same variables.
Enrichments are set up to work with variable name, and doesn’t support handlebar syntax.
Enricher syntax is basically ‘variable names with brackets around them’.
https://docs.newrelic.com/docs/alerts-applied-intelligence/applied-intelligence/incident-workflows/custom-variables-incident-workflows/
Many of the variables are lists or ‘sets’ (arrays), unlike handlebars, enricher cannot access the content of the array, you must use the entire array, hence many enrichers query will use ‘where attribute IN {{array}}’

SELECT latest(ec2State) FROM ComputeSample where provider = 'Ec2Instance' and entityName in {{entitiesData.names}}

Important things to know about enrichers:

  • for Email, Slack and Pagerduty, the enrichment comes as an image, like a screenshot, that cannot be scrolled: for these we recommend to use enrichers queries that output a graph rather than string values.

  • for Webhook, Service Now, Jira, Event Bridge, the enricher will return the results in JSON format. Be aware the payload Editor won’t access your enricher’s JSON properly (in the payload preview it only shows the first numerical value)
    preview payload

But a real Notification will have the entire JSON

Gotchas with Enricher:

→ Enrichers won’t work well in the ‘Test Notification’, Enrichers will work well in the ‘Test Workflow’
→ They run inside the Incident intelligence Account, so if you have a cross account environment (not recommended at this time), you should your workflow on that particular account, else the enricher will not work when the workflow processes an issue from another account.
→ When using an enricher variable, in addition to the query result, the json object returned contains the value that was used in place of the variable. Because of that the result object is structured differently when the enricher query uses a variable or not.
To access the return value of a query like

FROM Transaction SELECT count(*) WHERE entityGuid in {{entitiesData.ids}}

You will use

{{enricherName.result}}

To access the return value of a query like

FROM Transaction SELECT count(*)

You will use

{{enricherName}}

6- Troubleshooting syntax with the editor is difficult, a way to do this is to divide and conquer, back up your json text and test it, line by line, or few lines at a time.

7- New Best Practice
Create an alert to notify your New Relic administrators of notification errors, ideally to more than one channel.

FROM NrIntegrationError select count(*) where category=‘NotificationError’

This will alert if for some reason the notification endpoint isn’t available, or some error happened during payload transformation. Recipients will be warned in real time that they may be missing notifications so they can fix the issue.

8- if you need to send a JSON as a string value, you can escape like this

3 Tips on HandleBars syntax, (this doesn’t mean you should not read the guide, it’s not very long)

1- triple-stash {{{ is a valid handlebar syntax to get HTML-escaped output, ie ‘&’ is not transformed into ‘&’
hence don’t be surprised if the payload editor insist that you add a space or a new line. You may encounter this problem when trying to make your script denser, or when copying and pasting portions of code

no triple-stash at the end

no triple-stash at the beginning

Use new lines

or use spaces

2- {{{{ is also valid handlebar syntax to escape handlebar itself (ie: display handlebar code as text without interpreting it)

3- WhiteSpace Control or ~

The payload output must be valid json, hence, it cannot have spaces or new lines where JSON doesn’t want them, and you likely do not want extra characters in your values.

This may lead you to take the easy solution: write very dense handlebar scripts, aka “put it all on one line”, this may lead you to syntax validation error, and just makes it hard to read

Adding spaces or new lines breaks json compatibility

Use the ‘whitespace control’ character before or after any handlebar section to ignore the white spaces before or after

3 Likes

Is this supported for Slack destinations? Finding many scenarios in which we want to edit our notifications that are sent to slack. If this isn’t possible with customising the workflow payload, is there another solution that could help us?

Thanks