Proxmox: Windows VM Fills Host Swap
A Windows VM is running and the PVE host's swap fills up until the whole host gets sluggish. There is no switch to "forbid" the VM from doing this — the VM never writes to host swap in the first place. The host kernel swaps out the VM's memory when physical RAM runs short, so the fix is removing that memory pressure, or marking the VM's memory as unswappable.
This is the prevention side of Clear Swap Without Reboot, which covers draining swap once it has filled.
How It Works
From the host's perspective, a VM is one QEMU process. When RAM runs short, the kernel moves pages of that process out to swap — the same way it would for any other process.
Windows guests are the worst offenders for two reasons. Windows zeroes its free memory at boot, writing to every page the OS considers free — from the host's view, every page is now dirtied and needs real memory. On top of that, Windows fills leftover RAM with prefetched file cache (SysMain). A Windows VM therefore sits at close to its full configured allocation minutes after boot, while a Linux guest with the same config might only occupy a fraction — memory a guest never touches is never allocated.
Without a working balloon driver, none of that memory is ever given back. Add a second VM, a backup job, or a hungry ZFS ARC, and the kernel starts evicting QEMU pages to swap.
Pre-requisites
- Root access to the PVE host
- For the balloon driver: the virtio-win ISO attached to the Windows VM
- The ZFS ARC section applies to hosts with ZFS storage only
Check Which VM Is in Swap
Total swap usage is in free -h. Per-VM usage is more useful — every running VM has a PID file, and the kernel reports per-process swap:
for p in /var/run/qemu-server/*.pid; do echo "VM $(basename $p .pid): $(grep VmSwap /proc/$(cat $p)/status)"; done
Then do the budget math: the sum of all VM memory allocations, plus roughly 1 GB for Proxmox itself, plus the ZFS ARC maximum must fit into physical RAM. If it does not, the kernel will swap — no tuning parameter changes arithmetic.
Limit the ZFS ARC
On many homelab hosts the overcommit is not the VMs — it is ZFS's read cache. The ARC grows up to zfs_arc_max and competes with QEMU processes like any other memory user. Check its current size:
Old installs carry a 50% ARC default
Since PVE 8.1 the installer caps the ARC at 10% of RAM (maximum 16 GiB). Hosts installed earlier keep the old ZFS default of 50% of RAM unless a limit was set manually — on a 64 GB host that is a silent 32 GB claim. Check before you blame the VMs.
Set a permanent limit in /etc/modprobe.d/zfs.conf. The value is in bytes; a common formula is 2 GiB base plus 1 GiB per TiB of pool capacity. This example sets 8 GiB:
If the root filesystem is on ZFS, rebuild the initramfs and reboot to activate:
Verify after the reboot:
Set Up Ballooning for Windows
Ballooning lets the host take memory back from a running guest instead of swapping it out. When host RAM usage crosses 80%, PVE tells guests to release memory, down to each VM's configured minimum. The mechanism runs inside the guest — and Windows ships no virtio balloon driver, so on a stock Windows VM ballooning silently does nothing.
Configure a memory range
Maximum stays your normal allocation; minimum is what the VM must always keep. In the GUI: VM → Hardware → Memory → set Minimum memory below Memory, leave the Ballooning Device enabled. Or on the CLI:
Install the driver in the guest
Mount the virtio-win ISO in the VM and run the virtio-win-gt-x64 installer — it includes the balloon driver.
Install the balloon service
The driver alone responds to balloon requests; the service additionally reports the guest's real memory usage to the host, which the auto-ballooning logic relies on. From an elevated prompt in Windows:
Verify: the memory value on the VM's Summary page in the PVE GUI now matches Windows Task Manager instead of showing the flat maximum.
Set the minimum honestly
The minimum is what Windows actually needs under load, not a wishful number. If the host balloons the guest down too far, the guest runs out of memory and kills or pages its own applications — the problem moves, it does not disappear. Ballooning is also incompatible with PCI passthrough and adds a small overhead; the Proxmox docs discourage it for critical VMs.
Reduce Swappiness
With the budget fixed, tell the kernel to prefer dropping cache over swapping processes:
The reasoning, the permanent setting, and draining already-filled swap are covered in Clear Swap Without Reboot.
Hugepages: Memory That Cannot Be Swapped
Everything above reduces pressure; none of it forbids the kernel from swapping the VM. If one VM must never touch swap, back it with hugepages — hugepage-backed memory is allocated in full at VM start and the kernel has no mechanism to page it out:
2 uses 2 MiB pages, 1024 uses 1 GiB pages, any prefers 1 GiB with 2 MiB fallback. The trade-offs:
- The full allocation is reserved the moment the VM starts, touched or not.
- Ballooning no longer applies — the allocation is fixed.
- On a fragmented host the VM start can fail because contiguous memory for the pool cannot be found. Starting the VM early after host boot avoids this;
qm set 100 --keephugepages 1keeps the pool reserved across VM restarts.
What Does Not Help
- Windows pagefile settings. The guest's
pagefile.syslives on the guest's virtual disk and has no connection to host swap. Disabling it only removes Windows' own safety margin. - In-guest RAM cleaner tools. Memory freed inside Windows is not returned to the host — a page once dirtied stays allocated in the QEMU process. Only the balloon driver hands memory back.
- Disabling swap on the host. Removes the symptom and the safety valve with it: on real memory exhaustion the OOM killer terminates a process without warning, plausibly a VM. A small swap partition with low swappiness stays empty in normal operation and buys warning time instead.
Verify
After the changes, watch the host through a full workload cycle — all VMs booted, a backup run, SysMain settled:
Swap used should stay near zero, and the per-VM check from above should report VmSwap: 0 kB for the protected VMs.
Common Issues
| Symptom | Cause | Fix |
|---|---|---|
| PVE GUI still shows the flat maximum despite balloon driver | Balloon service not installed | Run blnsvr.exe -i as Administrator in the guest |
| Ballooning never triggers | Minimum equals maximum, or Ballooning Device disabled | Set minimum below maximum, re-enable the device |
| Guest applications crash when host memory gets tight | Balloon minimum set too low | Raise the minimum to what Windows needs under load |
VM start fails with hugepage allocation failed |
Host memory fragmented | Start the VM earlier after boot, set keephugepages |
| Swap refills although VMs were shrunk | ARC unlimited on a pre-8.1 install | Set zfs_arc_max in /etc/modprobe.d/zfs.conf |