Relic Solution: Custom Metric Alerts on the Linux Server Monitor

NOTE: This Post references the Linux Server Monitor, New Relic’s old server monitoring solution which is now deprecated.

Please refer to Infrastructure for the latest in New Relic’s Server Monitoring capabilities.

In New Alerts for the Linux Server Monitor, several common metrics may be alerted on:

  • CPU Usage %
  • Memory Usage %
  • Disk I/O %
  • Fullest Disk %
  • Load Average per minute

However New Relic also offers the ability to set a Custom Metric Alert on raw metrics produced by the local agent. This guide will demonstrate how to create an alert on Disk Fullness for a disk that is not the fullest disk. Access to these features will depend on subscription. Here we will discuss the use of the Linux Server Monitor, Custom Metric Alerting in New Alerts, and the Insights Data Explorer for Metrics.


Overview

First we will determine the correct metric by using the Insights Data Explorer. Then we will create the Alert in New Alerts. In the screenshot below you can see there are two disks. We will create the Alert for /mnt/ebs0, which is only 10% full, while the root disk is 72% full.


Locate the metric to alert on:

  1. Go to Insights.
  2. Select the Data Explorer on the left side.
  3. Choose the Metrics tab at the top.
  4. Choose Servers.
  5. Choose the particular Server to search for the root partition on.
  6. In the Suggested Searches, choose Filesystem. This populates the search bar with ^Filesystem/
  • Note that another option is Disks however this will refer to Disk I/O metrics, rather than Disk Fullness metrics.
  1. All Filesystem metrics on this disk are displayed in the metric list below. For example, to alert on the bytes used on the disk /mnt/ebs0, you would choose: System/Filesystem/^mnt^ebs0/Used/bytes.
  • Note that any forward slash / is replaced with the wildcard character ^ in metric names. This ensures that the disk path does not interfere with the metric path segmentation. This is discussed in the forum post Why is there a ^ in my metric?
  1. If you click on this metric, it will add a chart to the right that shows how this metric has performed recently.
  2. Confirm this is the right metric by viewing the same chart in the Servers UI. For example, go to the Disks tab on this particular server, and view the utilization chart. It should match this chart in Insights.


Create the Custom Alert:

  • Go to NR Alerts by choosing Alerts^New from from the right side of the header.
  • Create a new Alert Policy.
  • Create a new Alert Condition on the Policy.
  • Choose Servers for the product and Metric as the type of condition.
  • Select the Server Name as the entity.
  • In thresholds, choose:
  • When a target server: Enter metric name
  • Metric name: System/Filesystem/^mnt^ebs0/Used/bytes or whatever metric you chose above.
  • Note: there is no validation on this field so we highly recommend you confirm the chart shown in Insights for this metric matches the chart shown for the disk in the UI.
  • Leave the defaults: has an average value and above and for at least.
  • Here, enter the percentage of disk fullness to alert on. The unit for this value must be the same as the unit represented in the metric. Do not add any unit in the field, perform any calculations, or use scientific notation.
  • Edit the time for the metric to be above this threshold if desired.
  • Add a warning threshold if desired. Notifications are not sent when a warning threshold is crossed.
  • Name the Condition.
  • Create the Condition.

When creating an Alert on disk fullness, keep in mind that some Linux distributions keep 5% of disk space in reserve for root, as discussed on the forum post Relic Solution: How the Linux Server Monitor Calculates Disk Fullness. If your system reserves this disk space, you may need to adjust the percentage in the Alert accordingly.

1 Like