OpsGenie via webhook

Good morning all,
Like many of us here, we use ops genie in our daily process, so losing the integration into ops genie was disturbing, so trying to understand how to create a webhook alternative was something I needed to try and understand. I have not seen a writeup of what that process looks like, so I wanted to write my findings on the chance it might help someone else. Hopefully it helps someone else.

To create a webhook, you need to create a destination, so

Alerts&AI → Destinations → Webhook
Enter a webhook name - OpsGenie Alerts (or whatever you like)
For endpoint url, this is the api endpoint you want to hit. I needed alerts so I used https://api.opsgenie.com/v2/alerts
(Ops Genie documents their api here, https://docs.opsgenie.com/docs/api-overview, and you can find different targets and json packages defined here.)
Choose do not authorize here, we need to address this in the workflow to follow.
Click update destination

Now go to Alerts & Api → Workflows → Add a workflow
Enter a workflow name = OpsGenie Alert Flow (or whatever you like)
Filter, don’t choose send all issues, come up with a specific case to test with, I did where policyName=mytestpolicy so that I could control the ingest rate while testing.
Skip enriching for today, this sounds really great, but I have not gotten to this yet.
Mute Issues : set to whatever you like
Notify, click webhook
set name = OpsGenie Alert Workflow
and choose the destination from the drop down that we created in the previous step
Add a custom header, (this is the authorization piece),
The name of the key is Authorization
The value should look like this string, GenieKey ########-####-####-####-############, you will replace the #s with an OpsGenie integration key. It was an integration key that I needed for alerts, so within OpsGenie, I went to integrations and added a new api integration and used the key provided there.

This is my example of a payload template

{
“source” : “NewRelic”,
“issue_id”: {{ json issueId }},
“responders”:[
{
“name”:“Solutions Reliability”,
“type”:“team”
}],
“message”: “My text {{ annotations.title.[0] }}”,
“details”: {
“self_url”: {{ json issuePageUrl }},
“state”: {{ json state }},
“is_correlated”: {{ json isCorrelated }},
“created_at”: {{ createdAt }},
“updated_at”: {{ updatedAt }},
“activated_on”: {{ activatedAt }},
“closed_at”: {{#if issueClosedAtUtc}} {{ json issueClosedAtUtc }} {{else}}“None”{{/if}}
},
“incident_ids”: “{{#each incidentIds}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}”,
“priority”: “P4”,
“impactedEntities”: {{json entitiesData.names}},
“totalIncidents”: {{json totalIncidents}},
“state”: {{ json state }},
“trigger”: {{ json triggerEvent }},
“isCorrelated”: {{ json isCorrelated }},
“createdAt”: {{ createdAt }},
“updatedAt”: {{ updatedAt }},
“sources”: {{ json accumulations.source }},
“alertPolicyNames”: {{ json accumulations.policyName }},
“alertConditionNames”: {{ json accumulations.conditionName }},
“workflowName”: {{ json workflowName }}
}

At this point, you should be able to send a test notification and in turn see a result in the OpsGenie alert window, go ahead and click update message and then update workflow and you should be good.

Thanks to Mohan R, for his help and support on a couple hiccups.

Troubleshooting:
If you test the workflow and you get a 401, validate your documentation regarding . There are two types of OpsGenie api keys, so be sure you double check the docs and confirm you are using the correct apikey vs integration api key.

Template Notes:
Most of the time in your template, you are just setting json values,
“self_url”: {{ json issuePageUrl }}
But if you need to concatenate some text, the syntax is a little different than you might expect, so watch for this, surround the string with quotes and drop the json keyword.
“message”: “My text {{ annotations.title.[0] }}”
priority within NewRelic is warning and critical, but in OpsGenie it needs to be P1 - P5 for it to be passed correctly. That will probably need to have a #if added, or just have 2 separate workflows, but just wanted to call it out.

Weaknesses/Additional Comments:
What I have done is 1 directional at this time, it places alerts into ops genie. If you need OpsGenie to NewRelic you will have to create OpsGenie webhook calls back into NewRelic unfortunately.

I’ll add more as I learn more. Please feel free to do the same and I can update this and hopefully OpsGenie will be able to update their integration for NewRelic’s changes to come.

Anyhow, all the best and good luck!

Tim

7 Likes

Great write up! Thanks for sharing.

There are issues with this template and it doesnt function as a good enough replacement for the Opsgenie Direct integration. These are all issues between the choices New Relic has made and How things interact with the Opsgenie API.

Noticeable issues:

  • Doesnt maintain state, so alerts cannot auto close.
  • If you correct the template to use the opsgenie alias field = the New Relic Incident ID field the counter increases but again wont close because of state. Without that change it just dupes alerts.
  • Regardless of API Key type the alerts come in over the Default API, so templates and filters dont work.
  • Opsgenie is built on Tags and New Relic tags arent available in the template builder, they are also not easy to build into the correct string in the builder if you wanted to manually add them.
1 Like

Hi @Elmer.Hahn

Thanks for reaching out I hope you are well.

Thank you for the feedback, I will be sure to pass this into the product team. Im sure it will be of great benefit to the team to have this insight from our customers.

Wishing you a great day.

You are correct, this is simply the ingest piece of the alert. To be able to maintain state, linking, acknowledgement, closings (all the things we have today), webhooks that call the NewRelic API would have to be created within OpsGenie and more than just this one in NewRelic. I don’t have cards on my board to allow me to work on this, but I do think it is possible with what I have seen. Unfortunately, until OpsGenie updates their plugin, I think it is going to be a manual hackathon.

I am really hopeful that NewRelic and OpsGenie can work together to prevent us from having to try to implement something like this to solve a self-inflicted breaking change.

Tim

Hi @Tim.Walter2

Thanks for reaching out, and the feedback here.

Currently our team is working with OpsGenie, we hope to have this resolved soon. I will be sure to highlight this to the relevant team.

@dcody Any updates on the New Relic - Opsgenie integration or a ticket that I can watch?

Hi @chris.chung - The Product team has managed to create and close New Relic incidents in OpsGenie. The following instructions and working template should work for you.

Configuring Opsgenie side of the integration

  1. Go to your team’s dashboard from Teams,
  2. Select Integrations, and select Add integration.
  3. Follow the rest of the steps to complete the integration.
  4. Go to Settings > Integrations. Search for New Relic and select Add.
  5. Specify who is notified of New Relic incidents using the Responders field. Auto-complete suggestions are provided as you type.
  6. Copy the API key.
  7. Select Save Integration.

Destinations

  1. Create a webhook destination and paste the api key
    https://api.opsgenie.com/v1/json/integrations/webhooks/newrelicv2?apiKey={your API key here}

Workflows

  1. Choose the webhook notifier, and the destination you have created
  2. This template should work, feel free to edit accordingly
{
      "tags": "tag1,tag2",
      "teams": "team1,team2",
      "recipients": "user1,user2",
      "payload": {
        "condition_id": {{json accumulations.conditionFamilyId.[0]}},
        "condition_name": {{json accumulations.conditionName.[0] }},
        "current_state": {{#if issueClosedAtUtc}} "closed" {{else if issueAcknowledgedAt}} "acknowledged" {{else}} "open"{{/if}},
        "details": {{json issueTitle}},
        "event_type": "Incident",
        "incident_acknowledge_url": {{json issueAckUrl }},
        "incident_api_url": "N/A",
        "incident_id": {{json issueId }},
        "incident_url": {{json issuePageUrl }},
        "owner": "N/A",
        "policy_name": {{ json accumulations.policyName.[0] }},
        "policy_url":  {{json issuePageUrl }},
        "runbook_url": {{ json accumulations.runbookUrl.[0] }},
        "severity": {{#eq "HIGH" priority}} "WARNING" {{else}}{{json priority}} {{/eq}},
        "targets": {
        
         "id": {{ json entitiesData.entities.[0].id }},
        "name": {{ json entitiesData.entities.[0].name }},
        	"type": "{{entitiesData.entities.[0].type }}",
          	"product": "{{accumulations.conditionProduct.[0]}}"
        },
        "timestamp": {{#if closedAt}} {{ closedAt }} {{else if acknowledgedAt}} {{ acknowledgedAt }} {{else}} {{ createdAt }} {{/if}}
      }
}

There is a gap that needs to be closed and it is to create a trigger event when the issue is acknowledged. Engineering is working on this but not sure of the ETA at this time.

1 Like

What is the reason for using the V1 API which is long deprecated over the V2 one which the New Relic Opsgenie Channel uses?

https://api.eu.opsgenie.com/v2/json/newrelic?apiKey=

I’m having problems with this integration.
I have used this template and template provides by OpsGenie and it does not work