When a Proxmox host runs out of RAM, the failure mode is often not a clean “memory full, kill one process” event. Instead, it can become a cascading resource collapse where memory pressure creates storage pressure, storage pressure slows guests, guest retries create even more storage pressure, and the whole node becomes progressively less responsive. This is commonly described as a spiral of doom (SOD)

Why this happens

A Proxmox host is not just running virtual machines. It is also running:

• the Linux kernel
• the Proxmox management stack
• storage services
• network stack
• qemu processes for KVM guests
• possibly lxc containers
• filesystems, page cache and metadata caches
• sometimes zfs, or ceph which have their own memory appetites

When RAM becomes scarce, Linux tries very hard to avoid killing processes immediately. That behaviour is usually helpful, but on a virtualisation host it can become destructive because the host tries to reclaim memory by doing more disk I/O at the exact moment the system is already under pressure.

The core chain of events

The typical sequence is:

1. Free RAM becomes critically low
2. Kernel memory reclaim starts aggressively
3. Dirty pages are flushed to disk
4. Anonymous memory is swapped out
5. Disk latency rises and I/O queues deepen
6. VM disk operations slow down or time out
7. Guests retry I/O and generate more writes
8. Host needs even more reclaim and swap activity
9. CPU time is spent waiting on I/O rather than doing useful work
10. System responsiveness collapses
11. Eventually the OOM killer may terminate qemu processes or other services

That is the spiral.

What the kernel is trying to do

Linux memory management is designed to reclaim memory before it gives up and invokes the OOM killer.

The kernel will typically attempt to reclaim memory by:

• dropping reclaimable page cache
• writing dirty cache pages to disk so they can be freed
• shrinking slab caches where possible
• swapping out less-active anonymous pages
• scanning memory repeatedly looking for reclaimable pages

On a normal server, this may be survivable. On a hypervisor, it is much more dangerous because many workloads are generating disk activity at once, and the hypervisor itself depends on low-latency storage just to keep guest I/O flowing. This is not isolated to local storage, NFS or iSCSI exhibits the exact same latency trap. 

Why page flushing makes things worse

Some RAM is used for file-backed cache and dirty pages. Dirty pages are memory pages containing data that has been modified but not yet written to storage.

Under memory pressure, the kernel starts forcing those pages out to disk. That means:

• more synchronous and asynchronous writeback
• more pressure on the storage stack
• more waiting for I/O completion
• less spare IOPS and bandwidth for guest traffic

If the storage is already busy, writeback slows down. If writeback slows down, memory cannot be reclaimed quickly. If memory cannot be reclaimed quickly, the kernel scans harder and tries more aggressively. That alone can make the host feel hung.

If you're using ZFS or CEPH, which both try and do an end-run around Kernel buffering, then you have another problem domain which is now also exhibiting high IO and disk pressure IN ADDITION to the kernel which is doing it's very best to free up space. 

Why swapping is especially bad on a hypervisor

Once reclaiming cache is not enough, the kernel starts swapping anonymous memory to disk.

That creates a second problem:

• the host now needs disk I/O not only for guest virtual disks
• it also needs disk I/O just to move host memory pages in and out of swap

This is very expensive because swap I/O is usually random and latency-sensitive. If a swapped-out page is needed again, the process stalls until that page is read back.

Why guest VMs experience disk I/O trouble

A guest VM does not see the host’s memory crisis directly. It sees symptoms:

• virtual disk writes take much longer than usual
• reads are delayed
• filesystem journal commits stall
• application timeouts occur
• database transactions slow or fail
• kernel logs inside the guest may report blocked tasks or I/O timeouts

From the guest’s point of view, its virtual disk has become very slow or intermittently unavailable.

That leads to retry behaviour in several layers:

• guest operating system block layer retries
• filesystem retries or journal recovery work
• database retry logic
• application retry logic
• clustered software trying to resynchronise or reconnect

Those retries often create more I/O, not less. 

Why retries amplify the problem

Retries are supposed to recover from temporary delays, but under host memory exhaustion they make the underlying bottleneck worse. The host still intends to commit the writes or action the reads, but the guest has given up, timed out and is actively re-sending the block requests. That's amplification on a varying scale, and all the while the guest is buffering further reads and writes, using more of it's RAM for those buffers. 

A common pattern looks like this:

• guest write is delayed
• guest times out and retries
• application queues more work
• buffered writes accumulate
• once the guest gets CPU and I/O again, it pushes even more requests
• host storage queue grows further
• host reclaim and swap take even longer

This creates positive feedback:

• low RAM causes high I/O
• high I/O causes slow reclaim
• slow reclaim causes more swap
• more swap causes slower guest I/O
• slower guest I/O causes retries
• retries cause even higher I/O

That is why the problem snowballs rather than stabilising. The guest doesn't know the host is 'backed up', it sees a failing storage subsystem and attempts to get it's data written and continue. 

Why the host can become almost completely unresponsive

Once the system enters severe reclaim and swap thrashing, several things happen together:

• CPUs spend time in memory reclaim and I/O wait
• the storage queue remains saturated
• management commands block waiting on disk or memory
• SSH sessions freeze
• the web UI stops responding
• Promox services crash
• Cluster events stall
• even logging can become delayed because log writes need I/O

At this stage the node may still technically be alive, but operationally it behaves like a hang.

Why the OOM killer often acts late

Administrators sometimes expect the OOM killer to intervene early and solve the problem. In practice, Linux usually tries many reclaim paths first.

The OOM killer tends to act only when:

• the kernel decides memory really cannot be reclaimed fast enough
• allocation requests are failing in a way that cannot be satisfied
• the system has exhausted practical reclaim options

By that point, the machine may already be thrashing badly.

On a Proxmox host, the OOM killer may then target:

• a qemu-system-* process
• a less protected userspace daemon
• some other large memory consumer

If it kills a qemu process, that is effectively a VM crash.

So the final stage of the spiral is often:

• long period of severe host-wide pain
• then abrupt guest death when OOM finally starts killing processes

The role of storage latency

The whole spiral is really driven by latency amplification.

A small increase in storage latency causes:

• reclaim to complete more slowly
• swap-in and swap-out to complete more slowly
• guest disk requests to wait longer
• application queues to build up
• more dirty memory to accumulate
• more work to be flushed later

This is why hosts with fast NVMe may survive longer under memory stress than hosts using slower SSDs or RAID on spinning disks. The same memory shortfall can be catastrophic on one storage platform and merely unpleasant on another.

Why prevention matters more than recovery

Once the host is in full swap thrash, recovery without disruption is difficult. Even simple administration may not work reliably. In many cases the realistic outcomes are:

• wait and hope reclaim catches up
• live with severe guest outages
• OOM kills one or more guests
• force reboot the node

That is why prevention is much more important than trying to manage a host already in the spiral.

Practical ways to avoid the spiral

1. Keep real RAM headroom on the host

Do not allocate RAM so tightly that normal cache growth or temporary workload spikes exhaust free memory.

As a rule of practice:

• reserve RAM for the host itself, which should be in the range of 20%
• reserve extra RAM for storage caching and burst activity with write-back and read-ahead
• always consider HA activity and make sure it cannot spin up more guests than the node supports

2. Be cautious with memory overcommit

Overcommit is not free capacity. It is a bet that not all guests will actively use their memory at the same time. If you can't be sure, don't overcommit. 

3. Treat swap as emergency space, not working memory

Some swap is useful because it gives the kernel breathing room. Too much reliance on swap on a hypervisor is dangerous.

Good practice:

• keep swap available
• do not design the host assuming regular heavy swap use is acceptable
• monitor swap-in and swap-out activity, not just total swap size
• adjust swapiness to keep usage low

4. Separate high-contention storage where possible

If practical, avoid placing all of the following on one slow device set:

• guest virtual disks
• host swap
• heavy writeback workload
• backups or replication targets

5. Tune ZFS/CEPH carefully if used

On ZFS-backed Proxmox hosts, ARC can consume substantial RAM. If the host is memory-constrained, ARC limits may need to be set conservatively. 

On Ceph Proxmox hosts, OSD Cache targets can be adjusted to prevent the same and free up RAM at the expense of performance. There are calculations that can be made for this, but testing is essential.

6. Monitor leading indicators

Useful metrics include:

• free and available memory
• swap utilisation
• swap-in and swap-out rate
• disk queue depth
• storage latency
• %wa
• VM balloon pressure
• writeback activity
• OOM and reclaim events in kernel logs

7. Avoid concurrent heavy jobs on a tight host

The spiral is often triggered by a combination such as:

• backup job
• snapshot activity
• replication
• scrubbing (hw or sw)
• guest patching
• database checkpointing
• storage rebuilds
• memory growth in several VMs

Scheduling matters.

Summary

The “spiral of doom” on a Proxmox host is a feedback loop between memory exhaustion and storage saturation, and we see it regularly at the HelpDesk. The key point is that on a hypervisor, memory pressure does not stay a memory problem. It rapidly becomes a whole-system performance collapse because both the host and all guests are competing for the same storage path at the worst possible time.