man8alexd

I think Kahneman acknowledged problems with one chapter about priming, not the entire book.

Powered by Movable Type.

I have already pointed you to /proc/pressure/memory. Prometheus node_exporter tracks it as node_pressure_memory_waiting_seconds_total and node_pressure_memory_stalled_seconds_total. You'll see them spiking when you experience stalling.

You can also track variables from /proc/vmstat exported as node_vmstat_allocstall_(normal|dma32|movable), node_vmstat_pgscan_direct, node_vmstat_pgsteal_direct - you'll see them spike when the system enters direct reclaim.

Install and configure systemd-oomd and track its log - it will activate on memory pressure.

To track code page eviction/re-faults specifically, you'll need to write some eBPF code, there are no direct measurements for this available.

You seem to confuse user-space memory allocations (malloc) with the kernel memory page allocation and mapping. The processes don't need to allocate any new memory with malloc. They get stuck when accessing memory that is not mapped to a physical memory page or when they enter a syscall that needs in-kernel memory allocation.

The assumption that a system gets unresponsive due to code pages being reclaimed doesn't make any sense. It is a common myth, cargo-cult knowledge. The kernel doesn't discard any active referenced pages at all, and the active code pages are the last that are getting reclaimed.

The proper explanation is relatively straightforward. Normally, any memory request for a process is served by getting a page from the free memory pool. But when the system free memory goes below the low watermark (calculated from vm.min_free_kbytes and vm.watermark_scale_factor), the kernel stops giving the pages from the free memory pool and instead goes into direct reclaim, trying to find unused pages. While the kernel does reclaim, the allocating process remains stalled. The kernel can't return control to the process - there is no page mapped that the process tried to access. This applies to any types of pages - anon, file-backed, code pages. So, under the condition of free memory below the low watermark, any process trying to allocate memory gets stalled. This is measured as memory pressure stalls (PSI) in /proc/pressure/memory.

The problem with the system being stuck in an unresponsive state is that the design choice for the kernel OOM handling is to be optimistic and keep the workload running as long as the kernel is making any progress reclaiming pages. The direct reclaim process needs to fail to make any progress 16 times before the OOM killer is invoked.

The solution is to use cgroup memory limits to avoid getting into the situation of memory exhaustion and use user-space oomd to recover from this situation earlier.

Value 60 is not an arbitrary number. It makes the kernel reclaim 140 file pages for each 60 swapped-out anon pages. This proportion is supposed to reflect the relative performance cost between sequential file access and random swap I/O on HDD. For SSDs, the value 100 is more appropriate.

> I don't see a reason why the default shouldn't be 0 then on modern systems.

The zero value makes reclaim very unbalanced and is not appropriate for generic systems. This will cause the kernel to discard all the page cache, slowing down the system with uncached I/O, while keeping all unused anonymous pages in memory.

What do you think this value of 60 means? Also, as I said, in extreme situations, when the system is right before invoking the OOM killer, swap is used regardless of `vm.swappiness` setting.

In simple cases, instead of using `mlock` just set cgroup memory.min

Active code pages with the "Accessed" bit set are specifically excluded from being moved to inactive LRU when active LRU is being shrunk. Instead, they are moved to the head of the active LRU. All other pages are moved from active LRU to the inactive LRU unconditionally, even if they have the "Accessed" bit set. So, no, the active code pages are not going to be eventually evicted.

I'll try again. Why do you think that unresponsiveness is caused by code pages being discarded? Do you have evidence for this? Have you considered that there are other mechanisms that cause programs to become unresponsive under memory pressure?

All code pages in the active file LRU are excluded from the reclaim.

The assumption that data pages can always be easily discarded is incorrect. Dirty data pages can't be discarded until they are written back. Recently accessed data pages are also not discarded until all inactive data and code pages are discarded.

Could you clarify what exactly is "not true"? Also, could you explain what exactly your evidence proves? What do you think happens when the system becomes unresponsive under memory pressure, and why?

vm.swappiness doesn’t work the way most people assume it does. and it is ignored under heavy memory pressure.