Mismatch between K8sContainerSample.memoryUsedBytes and actual memory usage in container

We have a K8S cluster (EKS) running the NRIA to report on pod metrics. The K8sContainerSample.memoryUsedBytes metric for a container seems to be way different than what is actually being used in that container.

For example, in one pod, NR is reporting 10G of used memory, but when I log into that pod, it’s only using about 4G

GiB Mem : 61.477 total, 0.871 free, 4.037 used, 56.569 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 56.754 avail Mem

Another pod is reporting 16G of memory used, but top is reporting only 2.3G used
GiB Mem : 61.477 total, 34.126 free, 2.391 used, 24.960 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 58.402 avail Mem

I can’t find any docs on what K8sContainerSample.memoryUsedBytes actually means and how it’s gathered.

Hey @jschutt it looks like you’ve already talked to @Fidelicatessen about this issue but I’d like to close the loop here. The value K8sContainerSample.memoryUsedBytes will pull the usageBytes metric from the /stats/summary endpoint of the kubelet, scoped to the container. As you noted that value included the linux buffers and cache.

Hi @avialpando - I’ve just come across this post and wondered if you could supply a little more information. The K8sContainerSample.memoryUsedBytes metric clearly sounds like it should show us the memory a container is using - but having been woken up in the middle of the night last by an alert based on this metric, I’d like to find out how we can actually work out how much memory a container is using (rather than the memory being allocated to it by the node OS). Is there a metric that can do this in the K8sContainerSample?

Hey @adam.cheney - sorry for the delay here - I’ll try to follow up with an answer for you soon!

Hi @adam.cheney, thanks for your patience on this. The memoryUsedBytes attribute should tell you how much memory a container is using whereas memoryRequestedBytes would indicate how much memory is allocated by the Pod. Would you mind sharing the alert incident that opened in this case so we can take a look to see if there is any unusual behaviour? Cheers

Hi @rdouglas,

Sorry, I missed this response when it arrived and have been away on holiday.

The incident was https://alerts.newrelic.com/accounts/1962024/incidents/80790416. When I ran ‘kubectl top’ against the pod it was compaining about, it showed much less memory use than NR was indicating.

Hello @adam.cheney, It looks like the alert condition that violated targets the memoryUsedBytes and the memoryLimitBytes. memoryLimitBytes represents the limit bytes of memory defined for the container in the pod specification. memoryUsedBytes shows the memory that is actually used by the container.

We’re also experiencing this mismatch between K8sContainerSample memoryUsedBytes and what k8s 'kubectl top ’ reports with memoryUsedBytes always being higher.

The default settings on the NR k8s alert policy uses memoryUsedBytes:
(memoryUsedBytes/memoryLimitBytes)*100 > 95

This is causing us to receive false alerts since we have our requests/limits tuned to what ‘kubectl top’ reports.

Is there a metric we can use that matches what ‘kubectl top’ reports?

Hello @david.lo1,

Thanks for reaching out regarding the memoryUsedBytes metric from K8sContainerSample.

I can confirm that the memoryUsedBytes metric is gathered from the kubelet’s /stats/summary endpoint, and it does include cached memory that can be freed up by the pod. The memory usage that you’re seeing from kubectl is memoryWorkingSetBytes , which is the memory used that can’t be freed up and is the value evaluated by OOM Killer to clean up processes when a host’s memory resources are exhausted.

The Kubernetes integration introduced collecting memoryWorkingSetBytes in v1.7.0. If you are not on that version already, update, and let us know if memoryWorkingSetBytes is what you are looking for.

Regards

@sbetea Thanks for the info. memoryWorkingSetBytes looks more inline with what I expect to see.

Hi @david.lo1,

How did you configure alert integration to use this memoryWorkingSetBytes? When I tried to configure custom alert using K8sContainerSample metric, it allows only to set the exact threshold using bytes/kb/mb etc. I cannot set relative thresholds like 95% or 85% of the limit.

In general, I would like it to be the default metric in Kubernetes integration as cached/buffered memory is considered to be “AVAILABLE” and should not cause false alarms.

Hi @sergei.shishov, I didn’t end up modifying the default Kubernetes integration alerts. I also see what you’re seeing, where it’s not possible to use memoryWorkingSetBytes on an alert while also using percentage thresholds.