Mismatch between K8sContainerSample.memoryUsedBytes and actual memory usage in container

We have a K8S cluster (EKS) running the NRIA to report on pod metrics. The K8sContainerSample.memoryUsedBytes metric for a container seems to be way different than what is actually being used in that container.

For example, in one pod, NR is reporting 10G of used memory, but when I log into that pod, it’s only using about 4G

GiB Mem : 61.477 total, 0.871 free, 4.037 used, 56.569 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 56.754 avail Mem

Another pod is reporting 16G of memory used, but top is reporting only 2.3G used
GiB Mem : 61.477 total, 34.126 free, 2.391 used, 24.960 buff/cache
GiB Swap: 0.000 total, 0.000 free, 0.000 used. 58.402 avail Mem

I can’t find any docs on what K8sContainerSample.memoryUsedBytes actually means and how it’s gathered.

Hey @jschutt it looks like you’ve already talked to @Fidelicatessen about this issue but I’d like to close the loop here. The value K8sContainerSample.memoryUsedBytes will pull the usageBytes metric from the /stats/summary endpoint of the kubelet, scoped to the container. As you noted that value included the linux buffers and cache.