Plan to upgrade Alert Notification Channels to Workflows and Destinations

Hello @peter.ralston,

Welcome back to the community, it is good to see you again!

We appreciate such a detailed post. I wanted to reach out and let you know I have relocated your post to this thread. One of our Relics pointed out this would be a better place for it and it will also make sure we get you the right help and insight.

@gromovd1 re : "workflows are redundant and only exist to complicate notification channels setup "

To clarify, workflows will be entirely replacing the Notification Channels all together, soon there will no longer be an option to specify the notification configuration linked to policies. The workflow filter can use the policyName or policyId attributes in an Issue if you want to maintain that, or as Phil points out, you can use conditionName specifically. Now, remember, that if you use conditionName, and then you change the name of the condition, the linkage will be broken. conditionID is a more stable filter, but less readable.

I believe the auto complete works off of the presence of attributes and tags in recent issues. If you create a new condition, I think the name will not be available in the auto complete until an issue exists for it.

@gromovd1 , creating one workflow per condition is an anti-pattern, unless you truly have different notification logic for each condition.

The best practice for how to configure your workflow model depends on where you send alert notifications to, and how your teams are oriented.
If you are using pager duty, and you have dev ops teams who own certain services, the best practice is to add a “team” tag to each Alert condition, and then create one or more workflow per team to route notifications to different endpoints based on the Issue Priority.

There is a UI for adding tags to NRQL Conditions. You must use GraphQL API or the GraphiQL Explorer UI to add tags to other types of Alert Conditions.

That’s a great point which shows why current UI approach isn’t the best. In “legacy” setup policy, conditions and notification channels are automatically linked with id.
New approach requires USER to map policy or condition name to id manually and then insert it in workflow filter somehow.
In my view, as I explained, the best would be to allow user to lookup entity by name using appropriate, full screen interface and then link it up with numeric id.

I have different notifications for different set of conditions - which were combined logically into policies before - so I am using workflow with a set of policies for now - that matches legacy setup.
As for using tags - that would be great when infrastructure conditions would support them… As of now, they rely on legacy policies.

1 Like

@peter.ralston If you are using webhooks with custom payload, you can use the following code to convert array to a string… I had to do this for Teams:

"value":"{{#each entitiesData.names}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}"
1 Like

thanks @gromovd1 . I agree with your recommendations, and I have passed them along to the appropriate PM as a feature request.
To Clarify, tags are available for Infra Alert conditions, it is just that you can not add / edit them via the UI. this is because Infra conditions will be converted to NRQL conditions next year. We are currently building a scope and signal selector UI for NRQL conditions that will be equivalent to what Infra conditions has.

@gromovd1 and @peter.ralston ,

Regarding formatting the webhook…

  1. if you are hitting a wall, and have a dedicated account team, please reach out to them. We are willing and able to assist. Please do not struggle alone.
  2. check out this opsgenie template shared in another post. It demonstrates some formatting approaches that will address your concerns. We will be adding these template examples to our docs:
    OpsGenie via webhook

I am good now with custom payload for Team Webhook which I created - I also posted it in another thread.
The only issue is that some previously available variables don’t exist anymore, ex. human-readable timestamps…

1 Like

I made a feature request for human readable timestamp formats.

2 Likes

hi @gromovd1 these variables exist on the issue payload
image

1 Like

@mmoser2 Thanks for pointing this out - this is new and seems to have been released in the last couple of days. I modified my template for MS Team payload to include Activated time so it matches old template.

{
  "@context": "http://schema.org/extensions",
  "@type": "MessageCard",
  "title": "New Relic Issue" ,
  "text": {{ json workflowName }},
  "potentialAction": [
    {
      "@type": "OpenUri",
      "name": "View Issue",
      "targets": [
        {
          "os": "default",
          "uri": {{json issuePageUrl}}
        }
      ]
    }
  ],
  "sections": [
    {
      "activityTitle": {{ json annotations.title.[0] }},
      "activityImage": "https://static.intercomassets.com/avatars/792690/square_128/newrelic_logo-1476726027.png?1476726027",
      "activitySubtitle": "{{priority}} violation {{state}} activated at {{issueActivatedAtUtc}}",
      "facts":[
            {
               "name":"impactedEntities",
               "value":"{{#each entitiesData.names}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}"
            },      
            {
               "name":"totalIncidents",
               "value":{{ json totalIncidents }}
            },  
            {
               "name":"trigger",
               "value":{{ json triggerEvent }}
            },   
            {
               "name":"isCorrelated",
               "value":{{ json isCorrelated }}
            },    
            {
               "name":"alertConditionNames",
               "value":"{{#each accumulations.conditionName}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}"
            },    
            {
               "name":"alertPolicyNames",
               "value":"{{#each accumulations.policyName}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}"
            }    
      ]
    }
  ]
}

1 Like

Add a link to the resources section for
Relic Solution : Patterns for Implementing Alerts Workflows

1 Like

I have one more question regarding Custom Webhooks. Currently, I’ve noticed Workflow Webhooks don’t actually trigger the “Acknowledgement” event in NR (I’ve only seen “open”, “active” and “closed”), whereas the legacy Webhook did trigger an “Acknowledgement” event that we then use for Alert Analytics in our Dashboards. I am about to build a NOC custom dashboard and one of their hard requirements is to be able to visualize “acknowledge” events and track Acknowledgement engagement rate. This requirement is keeping me from using the Workflow solution since it does not trigger these type of events. Is this going to be fixed anytime soon (are events other than “open” , “active” and “closed” will be supported)? Should I stick with the legacy Webhooks for now and wait until legacy Webhooks are seamlessly migrated? Thanks for your help!

1 Like

Hi @Oscar.Lopez5 acknowledged events are planned along with other issue events, this planned to be implemented and documented this Q.

1 Like

Can we get more detail on OpsGenie migration? Someone has published a webhook version, but it is inadequate as a replacement for existing notification integrations.

Is there a reason that PagerDuty has a dedicated workflow and Opsgenie does not?

2 Likes

Hi @laurence

Thanks for reaching out,

I notice you have several posts from the same day asking the same question. Please follow along OpsGenie missing from Workflow Destination for all updates on OpsGenie.

As a gentle reminder we would request users not create multiple posts asking the same question as it can create confusion.

Wishing a great day!

1 Like

Would be great to seen screenshots of what a before and after will look like for the new workflows and destinations after an alert is migrated. As our first batch in cohort 1 will be done by NR engineering - we would like to give our user base insight as to what to expect before hand.

2nd Item of note - it appears when a new workflow destination is created it goes into “read-only” mode. Is this going to be updated to reflect the current permissions on the alert that was migrated ? That is a lot of work for my team to maintai.

Hi @Brenda.Stensland1

Thanks for reaching out, I hope you are well.

Thank you for this feedback as well as highlighting some potential challenges. I will be sure to pass this onto the product team.

Please also note in my previous post I have linked OpsGenie missing from Workflow Destination. In which it mentions that currently our team is working with OpsGenie to get this all updated.

To my understanding, as OpsGenie is 3rd party the changes need to be confirmed from their side before we can confirm them. However this may take some time, updates will be posted via the previous linked post.

Additionally I have created a case on your behalf with the support engineer team to ensure you get the support you on the 2nd issue highlighted “destination is created it goes into “read-only” mode”, as this is out of scope. Note they will reach out via this post with their findings.

Wishing you a great day.

Hello @bgoleno ,

Currently, our slack notification still shows the following notice

Notice
We are upgrading notification channels to a more powerful feature called “workflows”.
Existing notification channels will be automatically upgraded to workflows. Read about the plan and schedule in the link below.
Upgrade plan | Learn about workflows | More about the new Slack Application

I have read your detailed explanation above, and I have few questions:

  • Should we, the customers, do something in our end?
  • Is there anything that we have to do in our end?
  • If we are going to create a new alerts/notification, what should we do? Is there any terraform available for this?

The first 2 questions are similar, I just wanted to know, and make sure that everything (slack notifications, alerts) is still working after the migration.

Regards,
Laurentius

There is nothing that you must do proactively for this.
You may choose to convert to using workflows in advance of this process if you wish, or just wait for us to do it for you.

I would recommend that you create new alerts using workflows and destinations.
The terraform resource for workflows will be available very soon.

1 Like