Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Mismatch between K8sContainerSample.memoryUsedBytes and actual memory usage in container

insights

#1

We have a K8S cluster (EKS) running the NRIA to report on pod metrics. The K8sContainerSample.memoryUsedBytes metric for a container seems to be way different than what is actually being used in that container.

For example, in one pod, NR is reporting 10G of used memory, but when I log into that pod, it’s only using about 4G

GiB Mem : 61.477 total, 0.871 free, 4.037 used, 56.569 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 56.754 avail Mem

Another pod is reporting 16G of memory used, but top is reporting only 2.3G used
GiB Mem : 61.477 total, 34.126 free, 2.391 used, 24.960 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 58.402 avail Mem

I can’t find any docs on what K8sContainerSample.memoryUsedBytes actually means and how it’s gathered.


#2

Hey @jschutt it looks like you’ve already talked to @Fidelicatessen about this issue but I’d like to close the loop here. The value K8sContainerSample.memoryUsedBytes will pull the usageBytes metric from the /stats/summary endpoint of the kubelet, scoped to the container. As you noted that value included the linux buffers and cache.