Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Confusing information used memory between server monitoring ui and "top"


#1

I find confusing information used memory between server monitoring ui and used memory in “top”.
It shows huge different size used memory.

I not understand how to interpreted used memori on server monitoring ui and used memory
in “top” from this link



#2

Hi @kopra_katapedia,

Our memory overview graph should match what is reported by the free command or from /proc/meminfo

The calculation is:

mem_total  = "MemTotal" from /proc/meminfo
mem_free   = "MemFree"  from /proc/meminfo
mem_used   = mem_total - mem_free

buffers = "Buffers"  from /proc/meminfo
cached  = "Cached"  from /proc/meminfo
kern = buffers + cached

The overview graph is showing

percent_used = (mem_used - kern) / (mem_used + mem_free)  

as the physical memory used percentage.

The basic idea is that the Linux OS will use free Ram for caching and buffers because it isn’t being used. If needed it will give it back. So you want to factor that into your calculations.
Here is a nice site that gives a high level explanation of it:
http://www.linuxatemyram.com/


#3

I undertstand very well what Linux say about memory (with the free command in particular the cached that is “used” and not “free”).
About this, I think that “free, top, vmstat, etc.” are right and you are wrong !!!
Example : I have a system with 132 GB RAM.

  • DB2 process use 64 GB RAM (that is declared and allocated)
  • Java Application Server process use 48 GB RAM (that is declared and allocated)
  • the remaining 24 GB RAM are for the other processes (also for other caches etc.).

NewRelic says that I have used only 30% memory ( used 30.5% 40 GB / 130 GB ) … calculated from this :
[root@d761 ~]# free
total used free shared buffers cached
Mem: 136169852 134982452 1187400 0 399816 93100616
-/+ buffers/cache: 41482020 94687832
Swap: 8388600 1132568 7256032
[root@d761 ~]#

For me that is not correct and I risk to misunderstand that nNOT ALL MEMORY is used (while it is USED as LINUX top say !!!).

Can we do a configuration parameter or something other to correct this wrong situation ?!?!?
Thanks, Adriano


#4

Hi @adriano_mari,

I quickly ran through the math for you:

mem_total = 136169852
mem_free = 1187400
mem_used = 136169852 - 1187400 = 134,982,452

buffers = 399816
cached = 93100616
kern = buffers + cached = 93500432

percent_used = (mem_used - kern) / (mem_used + mem_free)
percent_used = (134,982,452 - 93500432) / (134,982,452 + 1187400)
percent_used = 41482020 / 136169852 = 0.30

.30 from this calculation is equivalent to 30%

That matches what free is claiming. As for why this isn’t matching what you allocated. The processes are not actually using it, so it is free to be used in buffers and cache unless they actually need it.
You can see this on the server here: https://rpm.newrelic.com/accounts/820115/servers/5459750/processes

Put some load on the server and the OS will take the buffers and cache and give it to the processes that have it allocated and actually need it.

Right now, I believe everything is behaving and reporting correctly. If you have any further concerns, please let us know.


#5

Yes … correct is correct. I think the concept is wrong.
Linux (free, top etc.) says the right concept.
Infact, (as you suggest), if you see the processes … we can see that :

  • db2 processes use 64 GB ca.
  • java processes (also GlassFish application server) use 32 GB ca.
  • … and so on …
    we are already over 96 GB … (not 40 GB …).

I think the right view (for me that need to know if my memory is used or not) is that :
the system USE that memory (for caching … ok … but using that !!!).

My optimal system (tuned system), use 99% memory since few minute from starting services … (as Linux can do by tuning memory processes and automatically caching and buffering [that is different from caching]).
If I see that the total RAM is used only at 30%, (as you say), I would go to tune processes (like java A.S. or Db2) to using more memory … so I obtain … SWAP ???

No, that is not correct. The right view is Linux view. This is also my opinion.
Anyway … thanks and regards.
Adriano


#6

Chris,

When reviewing our own systems, we find the same as the original poster. Our systems are running most efficiently when our cache is full; however, this triggers the alert thresholds unnecessarily.

For example, we received an alert for a system that was >90% utilization. When including buffers and cache, the system was at 97% utilization; however, this cache is used to speed up application performance. Using the correct (linux-preferred) calculation, our system was at 40% RAM utilization.

Restarting the application cleared 37% of our RAM utilization dropping the totals, including cache, from 97% to 61%; however, only a 7-point drop of our actual total RAM usage was observed, from 40% to 33%, impacting the system by forcing it to re-read the cache required for performance.

I would like to see the alert metrics fixed. Impacting performance in this way is unwarranted, this is also an impact to the teams relying on proper alert behavior.

Brian Beaudoin

P.S. Information about how cache is used and the correct amount of “free” RAM can be seen at http://serverfault.com/questions/85470/meaning-of-the-buffers-cache-line-in-the-output-of-free.

P.P.S. Re-reading the contents and the post and examining our alerts further, I see our alerts are unrelated to server RAM utilization but are related to the PHP OPcache plugin. The alert messages are confusing as the title is “% Memory Used > 90” requiring and requires further drill-down within the New Relic interface to determine this is related to a plugin monitoring application usage and not the server itself.

Hopefully this information may help others who are confused by alerts related to RAM usage when the source is not apparent.


#7

@brian_beaudoin Thanks for the extra information sharing. It really helps the community. I’ve always been a fan of the explanation offered at http://www.linuxatemyram.com/ but that might be a result of my enjoyment at imagining a RAM hungry penguin on a rampage (pun intended)


#8

@brian_beaudoin @parrott
3 years later into the future and I got waken up 2am for this same miscalculation.
The OS reserved a nice bunch of memory for cache and the reported usage percentage went over 90% when, in reality, 15% was in the cache.
If you guys are still around… did you do anything to solve this?


#9

Hey there @federico.boerr - The Linux Server Monitor has been deprecated for a while now, can add clarity on where you’re getting alerts from? I presume you got Infrastructure Alerts?


#10