Alert when dashboard doesn't display metrics

I have setup multiple Dashboards that display the metrics. These dashboards are updated once a day. On few odd days these dashboards fail to display the metrics since the source that generates these metrics fails.

I need to create an alert whenever any of the dashboards doesn’t display metrics.
e.g if metrics is not displayed on any of the dashboard on current date (say 8pm ) I would want to send an alert to few folks via chat notification

Hi, @Pradeep.Godi: I don’t think it is possible to create an alert condition based on whether a dashboard can display something, but you can create a condition based on whether the source of the metrics is reporting. Can you be more specific about what kinds of metrics the dashboard displays?

Hi @philweber ,

I have jupyter notebooks that generate metrics e.g avg streaming duration by users, avg number of users etc which is later sent to NR tables. The dashboard then displays these values in the form of line chart or bar chart.

You could create an NRQL alert condition based on a query like this:

SELECT count(*) FROM YourCustomEvent

or

SELECT count(`your.metric.name`) FROM Metric

Be sure to tick the box to alert on loss of signal.

I read through the NRQL alert condition but I couldn’t find the setup that meets my requirement.
Below are the end to end steps that are involved in my dashboard monitoring setup

  1. Generate the metrics externally everyday (once a day at 6pm)
  2. Push the metrics to New Relic Tables
  3. The dashboards display the latest metrics
  4. Now I want to check at 7pm everyday if latest metric is available . If a record for the current date is not available , then generate an alert and send it over an email/chat notification.

You could create a Synthetics API test that executes an NRQL query to check for your data. Have the script fail if the query returns no data. Set the Synthetics test to run once a day, and configure a Synthetics alert condition to notify you if the test fails.

2 Likes

@Pradeep.Godi

Phil’s solution is sound. Here’s another option:

Create a NRQL alert condition counting those results that come in once per 24 hours. Set Loss of Signal to 25 hours (or maybe 24.5 hours). If the event does not show up when it is supposed to, a “Signal Lost” violation will open.

2 Likes