Alerts Troubleshooting Framework- Notifications

  1. You will need a link to an incident for this.

  2. Keep in mind that warning violations do not open incidents and therefore you can’t be notified for warning violations.

  3. Check to make sure an incident opened at that time. You can use the Events tab in the incident in the Alerts & AI UI to check if the violations were opened – do a search on the condition name.

  4. Check the alert condition’s policy – you want to find out what the incident preference is. If it is By Condition or By Policy, it’s possible that there was already an incident open, and that the violation simply rolled up into it.

  5. If you see that a notification should have been sent (the violation got its own incident or at least opened a fresh one), check the Events tab in the incident UI to see if there were any issues sending any of the notifications. All should say something along the lines of “NOTIFICATION SENT.”

    • If an email shows as sent confirm with your email admins that New Relic emails are not being blocked or reach out to New Relic Global Technical Support.
  6. Try sending test notifications- Test alert notification channels

If there’s an issue with the notification channel, the UI should show a message with detailed information.

If the notification test responds with the top message as response: 200, its New Relic stating that it was successfully able to send the test notification. The bottom message is the response New Relic received from the notification endpoint.

  1. Troubleshooting webhooks- There isn’t much New Relic can do from our end if we see a response: 200 as the top message. Here are some suggestions you can use to troubleshoot your webhooks -

  2. Relic Solution: Troubleshooting webhook channels with RequestBin

  3. Relic Solution: All About Webhooks!

  4. Relic Solution: Troubleshooting Alerts Webhook Notification Channel Issues

  5. If you’ve made it this far, and it still looks like there are issues with notifications being sent, please reach out to New Relic Global Technical Support. Make sure to tell us what you tried and what you found along with a detailed description of the issue, this will speed up troubleshooting.

2 Likes