Skip to content
ToxicPinePublic

About

Memory-optimized NixOS VM runner with selective cross-guest sharing of kernel and Nix store pages.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SuperVM

Warning

Experimental Demo, Not Audited! Works on x86_64.

SuperVM is an aggressively memory-optimized runner for heterogeneous VMs. SuperVM VMs can run different packages, services, etc, but must use NixOS. Any software packages from public Nix caches are stored exactly once on disk and cached once in shared memory across all guests; private state remains private.

nix run .#supervm -- prepare ./my-vm
nix run .#supervm -- launch ./my-vm

Yet, each guest retains its own kernel, distinct writable Nix store, and state that is invisible to every other guest.

Why this exists

SuperVM is a proof of concept for seeing how far VM memory efficiency can be pushed under production constraints using selective kernel-page merging and policy-controlled sharing of Nix package files. It demonstrates that we can exercise granular control over what enters shared memory, avoiding broad cache-side-channel exposure. The same mechanism has near-zero discovery cost at runtime: known pages are mapped directly rather than found by scanning memory for duplicates, so the safer approach is also faster than scan-based deduplication.

Why it is so efficient

Firstly, SuperVM makes the core kernel text and rodata in each guest's memory KSM-mergeable, so VMs running the same kernel share one physical copy of those pages. Notably, no other guest memory is eligible. This cuts the overhead of the read-only kernel image without scanning changeable or secret data, where copy-on-write churn would hurt performance and merging could create side channels.

Secondly, Nix packages can annotate which files, including binaries, are safe to share; unmarked files never enter the shared cache. SuperVM applies that policy file by file, serving annotated files from one common cache instead of copying them into every VM. This captures most of the memory savings without exposing secret-handling code to cross-guest cache side-channel attacks.

SuperVM is built around a patched crosvm, which provides the memory mappings these mechanisms need. Its stripped build has Firecracker-class overhead: 6.7 MiB of measured host memory per VM outside guest RAM, versus 2.8 MiB for Firecracker.

How much memory it saves

Fleet measurements

Measured in idle fixed-count runs at 1, 4, and 32 VMs on 512 MiB minimal NixOS guests, including runtime infrastructure and resident host caches.

Deployment memory

Aggregate physical memory, with shared pages counted once:

VM count LameVM SuperVM
1 416.7 MiB 480.0 MiB
4 1183.6 MiB 776.0 MiB
32 8329.9 MiB 3512.8 MiB

At 32 VMs the deployments measure 8.1 GiB against 3.4 GiB.

Shared mappings (not per-VM)

Shared, once per host SuperVM
DAX pool (mapped) ~50 MiB
Metadata index ~9.8 MiB

DAX window overhead

Earlier guest-side changes cut an idle guest's host-resident RAM from 270 MiB to 118 MiB without KSM, mainly by reducing DAX window page metadata from 129 MiB to 5 MiB; these measurements predate dynamic DAX metadata allocation.

Charts

Idle Memory Consumption

Run it

Use a Linux host with Nix flakes and KVM.

Preparation builds the guest and adds its packages to the shared store. Launching boots that prepared guest with the state folder as its private layer.

Prepare first:

nix run .#supervm -- prepare ./vm-a
nix run .#supervm -- prepare ./vm-b

Then, start the VMs in two terminals:

nix run .#supervm -- launch ./vm-a
nix run .#supervm -- launch ./vm-b

The optional --profile flag selects a NixOS configuration:

nix run .#supervm -- prepare --profile ./machines#web ./vm-web
nix run .#supervm -- launch ./vm-web

launch boots the last successfully prepared runner without evaluating the guest again.

launch --dax-window=BYTES sets the DAX window size (default 1 GiB). It costs the guest 16 MiB of RAM per GiB, so size it to the files a guest maps at once, not to the store:

nix run .#supervm -- launch --dax-window=536870912 ./vm-a

Select --dax=never, --dax=inode (the default), or --dax=always while preparing:

nix run .#supervm -- prepare --dax=always ./vm-a
nix run .#supervm -- launch ./vm-a

Compare with vanilla crosvm

lamevm runs the same NixOS guest with mainline microvm.nix and nixpkgs crosvm:

nix run .#lamevm -- prepare ./baseline-vm
nix run .#lamevm -- launch ./baseline-vm

As with SuperVM, select a profile while preparing; launch reuses the prepared runner without evaluating it again:

nix run .#lamevm -- prepare --profile ./machines#web ./baseline-web
nix run .#lamevm -- launch ./baseline-web

For a fair comparison, run multiple instances of each, since SuperVM's benefit comes from sharing common pages between guests. If you just want to see how much performance SuperVM can achieve, pass --dax=always so every common store page is shared. Alternatively, to test the production-style fine-grained sharing policy used to avoid cross-guest cache side channels, use the default --dax=inode mode and add an $out/nix-support/fsmeta file to packages whose files should be shared. Each line in fsmeta has the form snix.dax <path-relative-to-the-package-root>, for example snix.dax bin/example; listed files enter the shared cache, whereas unlisted files stay out.

About

Memory-optimized NixOS VM runner with selective cross-guest sharing of kernel and Nix store pages.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages