Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: Troubleshooting Alerts Webhook Notification Channel Issues

alerts
notifications
webhook

#1

Introduction

This is a continuation of the great post my colleague @gjohnson had written, namely

In this post we will go into depth and provided a framework on how to go about troubleshooting Alerting Webhook Notification issues.

I will be using Slack Webhook for this example. However, this framework should work for virtually any Webhook. Without further ado, let’s get started.

Creating a Slack Webhook

First, we’ll need to configure Slack Webhook by following the steps outlined in the Slack documentation:

Creating & Testing New Relic Slack Webhook Notification Channel

Once we have configured the Slack Webhook, we’re going to test it by send a New Relic Test Notification. For this you’ll need to create a Webhook Notification Channel like so:

Clicking on the Send a test notification we should have received a dialog box with the results of the test:

Interpreting Test Notification Results

Let’s break down the two messages in the dialog box. The top message, response: 200, is New Relic stating that it was successfully able to send the test notification. The bottom message, Unable to invoke Webhook. Webhook responded with HTTP status code 400, is the response New Relic received from the Slack endpoint.

Ok, but a HTTP status code 400 is not very helpful. Looking up what a 400 HTTP Status code could entail. We’re told that HTTP 400 Bad Request should return some string in the response body. Looking at the bottom message from the test result, we don’t see any returned string. The string is actually returned and is captured on the New Relic backend. It looks something like this:

            "request.uri": "/notification/test",
            "requestor": "Sebastian Betea",
            "response": "Unable to invoke Webhook. Webhook responded with HTTP status code 400.",
            "response.headers.contentType": "application/json",
           ...
            "webhook_auth_provided": "false",
            "webhook_channel_name": "Slack_Webhook_Test",
            "webhook_error_response": "no_text",
            "webhook_error_status_code": "400",
            "webhook_notification_id": "NOTIFICATION_TEST",

Notice the line "webhook_error_response": "no_text". That is the response string mentioned in the Slack documentation. We’ll revisit what the no_text error response means and how to address it. But first let address how one might get access to the error response since it’s not available in the New Relic UI.

Capturing Test Notification Payload

What we’re going to do next is capture the payload the New Relic Test Notification sends to the Slack endpoint. Then we’re going to reconstruct the request and send it via Postman (curl is another option), which will provide us with the error response code we’re looking for.

For capturing the payload you may use any synthetic endpoint site of your choice. I have used webhook.site and requestbin and they work well. I’m going to use RequestBin in this example.

First we’ll need to create a RequestBin endpoint.

Once you’ve created a RequestBin endpoint, you need to copy the Endpoint url and paste that in the New Relic Webhook Notification Channel Base Url:

Now Save and Send a test notification. If successful you should receive a similar message:

Switch back to the RequestBin site. You should see a new POST request with the payload that was sent:

Constructing Payload / Retrieving Slack Error Response

Now we launch Postman (you may also do this with curl) and construct the request (using the payload from RequestBin) that we are going to send to the Slack endpoint. Hit the Send button. You’ll notice the Slack error response was returned, namely no_text:

Interpreting Slack Error Response

Now that we’ve got the Slack error response, no_text, what exactly does that mean and how do we address it? According the to Slack documentation a text attribute/field is required (most time, but definitely in this situation) when posting a message:

Addressing The Slack Error Response

So how do we address that? We’ll need to modify our New Relic Webhook Notification channel by adding a custom payload and adding a text field/attribute:

We then Save and Send a test notification:

Which results in this message in the Slack channel:

Custom_Payload_Example_Result

But wait, what happened to the rest of the fields/attributes that were present in the custom payload? While the payload was valid JSON, it didn’t conform to the Slack standards/syntax. For that I direct you to the Slack documentation linked below. You’ll need to make sure that the payload conforms to Slack requirements:

Using New Relic Slack Notification Channel Payload as starting point

That being said, you could create a New Relic Slack Notification Channel. Use the same base url as the Webhook notification channel. Then use your RequestBin url to send a Test Notification. Capture the payload. Then use the captured payload a starting point:

Which results with the following message in the Slack channel:

Conclusion

Phew, that was a lot of information! Hopefully it was helpful and informative. But most important I hope this post will come in handy the next time you run into problems with Alerts Webhook Notifications Channels.