Plan to upgrade Alert Notification Channels to Workflows and Destinations

:triangular_flag_on_post: Please follow this page for updates on the upgrade process.

Workflows and Destinations capabilities are fully deployed, and they are the official method for configuring and receiving notifications.

Workflows and Destinations are an upgrade, and direct replacement of, the legacy alert notification channels, and the policy-to-channels relationship.

We will be upgrading all accounts to use the new capabilities. We will do so in a way that will give API users time to update their automation, while trying to minimize the effort and disruption for UI-managed accounts.

The Critical Details

  • The current APIs for Notification Channels and Channels-to-Policy will remain available through June, 2023.

  • From August 2022, through January 2023, we will upgrade each account by automatically creating workflows and destinations to mirror the existing policy-to-channels relationship. See the current schedule below.

  • There should be minimal disruption to alert notifications during the upgrade process.

  • After the upgrade, you can review and optimize your workflow configurations. Since workflows are more flexible and efficient than the policy-to-channel relationship, you may be able to implement a more efficient model. The ideal situation would be for you to learn about workflows and destinations, and implement a model this best suits your observability processes.

  • A small number of accounts have configurations that can not be automatically upgraded. Your account teams will be contacting you if are one of these accounts. A New Relic specialist will be available to assist.

  • Accounts will be upgraded based on the notification channel types that are being used in the account. We will migrate whole accounts so that the account is not left in an awkward mixed state that uses both old and new capabilities. However, an organization may have sets of accounts that are in different upgrade states.

  • There is not a “User” destination type in the new system. “User” channels will be upgraded to a workflow that triggers an email destination and a mobile push destination.

  • User Channels are considered to be deprecated. User Channels are a “V1 Users” capability. After accounts are upgraded to “V2 Users,” User channels will not be created for new users. The existing, active User channels will continue to function until they are upgraded to workflows that have an email destination and a mobile push destination.

Current Schedule

The schedule below is a forecast as of 7/14/2022 . This schedule is subject to change as the process evolves. This schedule will be updated if there are changes to the plan. Please follow this page.

Start Date End Date Upgrade Account Cohort
8/21/2022 8/23/2022 DONE: Accounts that ONLY use email channels
8/29/2022 9/01/2022 DONE: Accounts that use user channels alone, or in combination with the above (email).
10/10/2022 10/13/2022 Accounts that use Slack channels alone, or in combination with any of the above
10/25/2022 10/31/2022 Accounts that use PagerDuty channels alone, or in combination with any of the above
11/1/2022 1/15/2023 No upgrades to support holiday preparedness
1/15/2023 1/31/2023 Webhooks, xMatters, Opsgenie, VictorOPs

Resource Links

9 Likes

Good to hear, and thanks for the early heads-up. Could you share more details around Terraform support of these recent developments?

1 Like

Hi @Rishav_old

There will be API aliases for notification channel API calls, so that they create destinations instead. Since the API calls will remain the same, Terraform will be unaffected (at least until June of 2023).

2 Likes

Terrific, thank you.

Are Custom Webhook Notification channels also being migrated to Workflows as-is?
I noticed there is still variance between the Webhook variables vs the Workflow variables available.

1 Like

Logged in to post about custom payload for Webhooks - new destination doesn’t seem to provide a way to use custom payload nor custom headers, how this will be handled?

1 Like

@OLopez1 , yes - that is the intention and we are working through all of the remaining details and testing now. The requirement is that the Webhook payload should remain exactly as it is today. There are some additional variables that are not available right now, but those will be fully plumbed through before the upgrade . The current forecast for Webhook auto upgrade is the second half of January 2023.

2 Likes

Thanks for the reply! That is reassuring and great to hear that the expectation is that it will be a 1-to-1 with Workflow Webhooks. We have several Webhooks that report back Alert data to NRDB for Analytics / Reporting purposes and we have not been able to migrate to Workflows since not all variables are there. Adding to that, would you know if there will be an “Issues” Event type? Currently, The NrAiIncident Event type is tied to “Incidents” (formerly Violations), not Issues, which is the reason we are relying on that “Alert” Webhook for the Alert Quality Management process.

@OLopez1 , yes. We are working on the NrAiIssue events now. There will also be an NR Ai Analytic Event for Notifications (probably NrAiNotification, but tbd). I would LOVE for us to get NrAiIssue published before the January upgrade date, but I can not commit to that yet.

1 Like

@gromovd1 , The webhooks in the new system are actually far more flexible and configurable than the old webhook template. However, they are in a different logical location. With workflows and destinations, the destination object is where the connectivity is configured, but not the message format. This allows for the same destination to receive different message formats, or have different rules for the content.
The webhook template itself is defined in the workflow configuration. If you are using the GraphQL API, it is technically considered a “channel”, which is a join of the workflow object and the destination object.

Here is the documentation for how to create message templates:
https://docs.newrelic.com/docs/alerts-applied-intelligence/notifications/message-templates

Related, we are also in early preview mode for allowing email subject lines and email bodies to be templatized as well.

1 Like

@bgoleno Thanks for pointing to the relevant documentation - because UI is not helpful at all to explain that payload is configured in Workflows.
Will existing webhooks automatically convert later on or I need to convert them myself?

1 Like

@gromovd1

New Relic will automatically convert them, although we will be doing accounts that use webhooks last, so it won’t be in the next few months.

That said, if you would like to convert them yourself before then, you are perfectly welcome to do that!

Ok, got it, will look into converting myself.
Thanks

And, the new Terraform resources should be available in the next couple of weeks. I will update here when those are available in the latest provider version.

So I tried to convert my policies which use webhook notification to workflows, however I am missing a vital piece of information - how to include infrastructure alerts into workflows?
Looks like infrastructure alerts are still connected to policies and it is not possible to create a workflow with filter based on conditionName - these only include synthetics.
To replicate (or, I’d say, duplicate) same setup, I had to create workflow with policyName filter, which make it look like workflows are redundant and only exist to complicate notification channels setup :frowning:
It would be more logical to be able to create conditions separately and then use them as filters for workflows… That or I am missing entire logic here.

Hi, @gromovd1: It is possible to create a workflow that targets Infrastructure condition names, but the conditions do not appear in autocomplete immediately after they are created. You may manually type the condition name and select Enter new, or wait a few minutes until it appears in the list.

1 Like

Ok, great, good to know! I will test this out and hope that the experience in condition selection would be improved soon.

Hi @gromovd1

I hope you are well.

Just wanted to check in and see if this has improved ?

Remember to check out the New Relic Essentials campaign, for you chance to win a OLDED Nintendo Switch and a $500 charitable donation among three amazing non-profits.

All you need to do is reply to the New Relic Essentials: Alerts thread with a doc/ video/ community post, or any other New Relic resource that has helped you better understand Alerts.

No, it did not improve. Autocomplete drop-down still only lists synthetics unless exact name of the condition is typed in.
I also think this would be a non-starter for accounts with thousands of conditions. I tried this in the large account I manage and selecting condition (or any other filter for that matter) via simple drop-down is bad experience. Ideally, there should be a way to navigate and filter full list of conditions/policies/other properties.

In regards to replacement webhooks,

There are significant differences between the existing notification channels and the new webhooks. They are not the same, not even close.

The new format is more powerful, but is using a 3rd party framework with a New Relic overlay (similar to how Synthetics works with selenium / node).

Ignoring that all of the existing payload needs to be changed since there is no alignment, and is not the same between old and new format,

We have spent the last 2 months trying to convert / get them to work and have had no success yet.

The new frame work does not send data in strings and only in array format. This means any system that can’t process arrays format cannot accept the new web hooks. This includes XMatters, Service Now (if not using the integration) … I’ll take a guess this includes VictorOps & OpsGenie which is why they are now getting dropped.

Since the string function hasn’t been exposed / implemented by New Relic (RFE) from the framework,
The only option is to send data as ‘json’ and or this dodgy conversion from array to string, which puts everything as numeric based place holders from the array.

We are now having to re-write all our processing logic in the receiving system.

If you haven’t already, start now.