This is a continuation of the great post my colleague @gjohnson had written, namely
In this post we will go into depth and provided a framework on how to go about troubleshooting Alerting Webhook Notification issues.
I will be using Slack Webhook for this example. However, this framework should work for virtually any Webhook. Without further ado, let’s get started.
Creating a Slack Webhook
First, we’ll need to configure Slack Webhook by following the steps outlined in the Slack documentation:
Creating & Testing New Relic Slack Webhook Notification Channel
Once we have configured the Slack Webhook, we’re going to test it by send a New Relic
Test Notification. For this you’ll need to create a Webhook Notification Channel like so:
Clicking on the
Send a test notification we should have received a dialog box with the results of the test:
Interpreting Test Notification Results
Let’s break down the two messages in the dialog box. The top message,
response: 200, is New Relic stating that it was successfully able to send the test notification. The bottom message,
Unable to invoke Webhook. Webhook responded with HTTP status code 400, is the response New Relic received from the Slack endpoint.
Ok, but a
HTTP status code 400 is not very helpful. Looking up what a 400 HTTP Status code could entail. We’re told that
HTTP 400 Bad Request should return some string in the response body. Looking at the bottom message from the test result, we don’t see any returned string. The string is actually returned and is captured on the New Relic backend. It looks something like this:
"request.uri": "/notification/test", "requestor": "Sebastian Betea", "response": "Unable to invoke Webhook. Webhook responded with HTTP status code 400.", "response.headers.contentType": "application/json", ... "webhook_auth_provided": "false", "webhook_channel_name": "Slack_Webhook_Test", "webhook_error_response": "no_text", "webhook_error_status_code": "400", "webhook_notification_id": "NOTIFICATION_TEST",
Notice the line
"webhook_error_response": "no_text". That is the response string mentioned in the Slack documentation. We’ll revisit what the
no_text error response means and how to address it. But first let address how one might get access to the error response since it’s not available in the New Relic UI.
Capturing Test Notification Payload
What we’re going to do next is capture the payload the New Relic
Test Notification sends to the Slack endpoint. Then we’re going to reconstruct the request and send it via Postman (
curl is another option), which will provide us with the error response code we’re looking for.
First we’ll need to create a RequestBin endpoint.
Once you’ve created a RequestBin endpoint, you need to copy the
Endpoint url and paste that in the New Relic Webhook Notification Channel
Send a test notification. If successful you should receive a similar message:
Switch back to the RequestBin site. You should see a new
POST request with the payload that was sent:
Constructing Payload / Retrieving Slack Error Response
Now we launch Postman (you may also do this with
curl) and construct the request (using the payload from RequestBin) that we are going to send to the Slack endpoint. Hit the
Send button. You’ll notice the Slack error response was returned, namely
Interpreting Slack Error Response
Now that we’ve got the Slack error response,
no_text, what exactly does that mean and how do we address it? According the to Slack documentation a
text attribute/field is required (most time, but definitely in this situation) when posting a message:
Addressing The Slack Error Response
So how do we address that? We’ll need to modify our New Relic Webhook Notification channel by adding a custom payload and adding a
Send a test notification:
Which results in this message in the Slack channel:
But wait, what happened to the rest of the fields/attributes that were present in the custom payload? While the payload was valid JSON, it didn’t conform to the Slack standards/syntax. For that I direct you to the Slack documentation linked below. You’ll need to make sure that the payload conforms to Slack requirements:
Using New Relic Slack Notification Channel Payload as starting point
That being said, you could create a New Relic Slack Notification Channel. Use the same base url as the Webhook notification channel. Then use your RequestBin url to send a
Test Notification. Capture the payload. Then use the captured payload a starting point:
Which results with the following message in the Slack channel:
Phew, that was a lot of information! Hopefully it was helpful and informative. But most important I hope this post will come in handy the next time you run into problems with Alerts Webhook Notifications Channels.