Kubernetes: Metric Update

We are moving away from using the container_memory_usage_bytes metric in favor of container_memory_working_set_bytes for memory monitoring within our Kubernetes capabilities.

Why?

The container_memory_usage_bytes metric isn’t an accurate indicator for out of memory (OOM) prevention as it includes cached data (i.e., filesystem) that can evict in memory pressure scenarios. However, the container_memory_working_set_bytes metric excludes cached data and is what Kubernetes uses for OOM/scheduling decisions, making it a better metric for monitoring/alerting memory saturation.

What is changing?

This change will be reflected within New Relic One as follows:

  • Kubernetes integration: Predefined alert policy. The “Container memory usage % is too high” alert condition now uses the new memoryWorkingSetUtilization attribute instead of (memoryUsedBytes/memoryLimitBytes)/100. Please note that existing alert conditions won’t be automatically updated. If you are using our predefined alert policy, we strongly recommend you to update it accordingly.

  • The following widgets are now using working set bytes in Kubernetes dashboard:

    • Resources Used
    • Container Memory Usage - % Used vs Limit
    • Container MBytes of Memory Used
    • Node Resource Consumption
  • Golden metrics definition for containers will now use memoryWorkingSetBytes

  • The Kubernetes cluster explorer now shows the container’s memory consumption compared to the limit, using the WorkingSetBytes metric in the Pod preview

  • New alerts can now be created based on:

    • Memory Working Set Bytes
    • Memory Working Set Utilization
    • Requested Memory Working Set Utilization
5 Likes