Comment on Help with understanding memory usage discrepancy
vk6flab@lemmy.radio 1 day ago
Linux aggressively caches things.
4 GB of RAM is not running out of memory.
If you start using swap, you’re running into a situation where you might run out of memory.
If oomkiller starts killing processes, then you’re running out of memory.
Is there a good way to tell what percent of RAM in use is used by less important caching of files that could be closed without any adverse effects vs files that if closed, the whole app stops functioning?
Basically, I’m hoping htop isn’t broken and is reporting I have 8GB of important showstopping files open and everything else is cache that is unimportant/closable without the need to touch SWAP.
I am guessing that htop is in the wrong.
If you look in /proc/meminfo, you’ll see a MemAvailable field and a MemTotal field.
It looks to me like this is approximately (I didn’t check top’s source) what top is using: MemTotal - MemAvailable.
stackoverflow.com/…/difference-between-memfree-an…
Rik van Riel’s comments when adding MemAvailable to /proc/meminfo:
/proc/meminfo: MemAvailable: provide estimated available memory
Many load balancing and workload placing programs check /proc/meminfo to estimate how much free memory is available. They generally do this by adding up “free” and “cached”, which was fine ten years ago, but is pretty much guaranteed to be wrong today.
It is wrong because Cached includes memory that is not freeable as page cache, for example shared memory segments, tmpfs, and ramfs, and it does not include reclaimable slab memory, which can take up a large fraction of system memory on mostly idle systems with lots of files.
Currently, the amount of memory that is available for a new workload, without pushing the system into swap, can be estimated from MemFree, Active(file), Inactive(file), and SReclaimable, as well as the “low” watermarks from /proc/zoneinfo.
However, this may change in the future, and user space really should not be expected to know kernel internals to come up with an estimate for the amount of free memory.
It is more convenient to provide such an estimate in /proc/meminfo. If things change in the future, we only have to change it in one place.
Looking at the htop source:
github.com/htop-dev/htop/blob/main/MemoryMeter.c
/* we actually want to show "used + shared + compressed" */ double used = this->values[MEMORY_METER_USED]; if (isPositive(this->values[MEMORY_METER_SHARED])) used += this->values[MEMORY_METER_SHARED]; if (isPositive(this->values[MEMORY_METER_COMPRESSED])) used += this->values[MEMORY_METER_COMPRESSED]; written = Meter_humanUnit(buffer, used, size);
It’s adding used, shared, and compressed memory, to get the amount actually tied up, but disregarding cached memory, which, based on the above comment, is problematic.
top, on the other hand, is using the kernel’s MemAvailable directly.
gitlab.com/procps-ng/procps/-/blob/…/free.c
printf(" %11s", scale_size(MEMINFO_GET(mem_info, MEMINFO_MEM_AVAILABLE, ul_int), args.exponent, flags & FREE_SI, flags & FREE_HUMANREADABLE));
Thank you for the detailed explanation
No problem. It was an interesting question that made me curious too.
This is the job for the OS.
You can run most Linux systems with stupid amounts of swap and the only thing you’ll notice is that stuff starts slowing down.
In my experience, only in extremely rare cases are you smarter than the OS, and in 25+ years of using Linux daily I’ve seen it exactly once, where oomkiller killed running mysqld processes, which would have been fine if the developer had used transactions. Suffice to say, they did not.
I used a 1 minute cron job to reprioritize the process, problem “solved” … for a system that hadn’t been updated for 12 years but was still live while we documented what it was doing and what was required to upgrade it.
tal@lemmy.today 1 day ago
Well, you could want to not dig into swap.
a_fancy_kiwi@lemmy.world 1 day ago
That’s pretty much where I’m at on this. As far as I’m concerned, if my system touches SWAP at all, it’s run out of memory. At this point, I’m hoping to figure out what percent of the memory in use is unimportant cache that can be closed vs important files that process need to function.
eager_eagle@lemmy.world 1 day ago
That’s a swap myth. Swap is not an emergency memory, it’s about creating a memory reclamation space on disk for anonymous pages (pages that are not file-backed) so that the OS can more efficiently use the main memory.
The swapping algorithm does take into account the higher cost of putting pages in swap. Touching swap may just mean that a lot of system files are being cached, but that’s reclaimable space and it doesn’t mean the system is running out of memory.
a_fancy_kiwi@lemmy.world 1 day ago
From what I can tell, my system isn’t currently using swap at all but it does have 8GB of available swap if needed.
To make sure I’m following what you are saying, if I upgraded my system to 64GB and changed nothing else, and let’s assume ZFS didn’t trying caching more stuff, would there still be a potential for my system to use swap just because the system wanted to even if it wasn’t memory constrained?
EarMaster@lemmy.world 1 day ago
If that’s the case you should look into your swappiness settings. You can set this to zero meaning the swap will only be used if you’re actually out of memory, but as others have noted that is maybe not a healthy decision…
a_fancy_kiwi@lemmy.world 1 day ago
I’m currently not in a situation where swap is being used so I think my system is doing fine right now. I’m not against swap, I get it’s better to have it than not but my intention was to figure out how close is my system getting to using swap. If it went from not using swap at all to using it constantly, I’d probably want to upgrade my ram, right? If nothing else just to avoid system slow downs and unneeded wear on my SSD
victorz@lemmy.world 1 day ago
It’s just that the system freezes for me when I used to run out of memory when I had only 32 GB of memory. Then I couldn’t do anything and had to hard reset the computer with its reset button. Then it would be nice to have a little bit of swap to kill some stuff before literally everything just stops working.
B0rax@feddit.org 1 day ago
Was that on Linux?