Guide: alert on custom metrics

I wanted to record a custom metric and then alert on it. It took me a while to figure this out, so I wrote my findings up as a guide.

My Ruby code:

NewRelic::Agent.record_metric('Custom/my_service/queue_size', 56)

The first thing I tried was

SELECT count(*)
FROM Metric
WHERE metricName = 'Custom/my_service/queue_size'
TIMESERIES

That returns no results. I’m not quite sure why custom metrics recorded via record_metric don’t show up in the normal Metric dataset, but they don’t. There’s something about “Timeseries” versus “Dimensional” metrics but I don’t understand the difference. Nor does the Ruby client seem to have a way to report “Dimensional” metrics.

When I use the Data Explorer, these custom metrics show up under “Timeseries Metrics”. That generates the NRQL

SELECT average(newrelic.timeslice.value) * 1000
AS `Custom/my_service/queue_size`
FROM Metric
WHERE metricTimesliceName = 'Custom/my_service/queue_size'
AND `entity.guid` = 'NTI5ODl8…'
SINCE 30 MINUTES AGO
TIMESERIES

For some reason, all that shows up are some zeros:

new-relic-timeseries-average-nrql

My hypothesis is that since my test data is so sparse, the average(…) is seeing a lot of empty entries and dividing small number / very large number.

If I change the average(…) * 1000 to count(…) then I get something more like what I expect:

If I then click “Create Alert”, it converts the original NRQL to

SELECT count(newrelic.timeslice.value)
AS `Custom/my_service/queue_size`
FROM Metric
WHERE metricTimesliceName = 'Custom/my_service/queue_size'
AND `entity.guid` = 'NTI5ODl8…'

That seems to work for an alert.

3 Likes

@jamesr1 Thank you for sharing this guide! I’m sure it will be helpful information for other community members :slight_smile: This will be one of our topics of the weeks.

1 Like