Skip to content

microvm-local.md

Latest commit

 

History

History
199 lines (150 loc) · 7.6 KB

microvm-local.md

File metadata and controls

199 lines (150 loc) · 7.6 KB
Raw

Running the microVM runtime locally

The microVM sandbox class (ateworker-microvm: a Kata guest on Cloud Hypervisor) needs /dev/kvm, which takes some extra setup compared to the default gVisor path. This guide covers just that delta: getting a KVM-capable Docker environment — on Linux, or on Apple Silicon macOS via Lima — then running the microVM counter demo and verifying a guest-memory snapshot round-trip.

Prerequisites

Complete the Quickstart (Development) in the README first — it covers the base tooling and the default (gVisor) path this guide builds on. For background on the runtime, see architecture.md and hack/microvm-assets/README.md.

Option A: Linux host with KVM

Works on bare-metal Linux or any cloud VM with nested virtualization enabled (e.g. GCE N4/N4D instances with nested virt, or equivalent on other clouds).

1. Verify KVM

ls -la /dev/kvm
# Expected: crw-rw---- 1 root kvm 10, 232 ... /dev/kvm

grep -cE '(vmx|svm)' /proc/cpuinfo   # >0 means CPU virt support (x86)

hack/create-kind-cluster.sh probes for KVM by running a root container with --device /dev/kvm, which works out of the box with a standard (rootful) Docker install. With rootless Docker the container's root is remapped to your user, so the probe fails with permission denied — use rootful Docker instead, or open up the device with sudo chmod 666 /dev/kvm.

2. Create the cluster

./hack/create-kind-cluster.sh
# Look for: "/dev/kvm found: microVM (kata + cloud-hypervisor) support will be enabled."

Once the control plane is up, atelet advertises the device on each KVM-capable node, which is what places microVM workers:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.ate\.dev/kvm}{"\n"}{end}'

3. Run the microVM demo

./hack/run-microvm-demo-kind.sh

This is a one-shot bring-up: it deploys the control plane, installs the cluster-wide microVM deps via hack/install-microvm-deps.sh — assembling the guest runtime assets for your architecture (skipped if already present under bin/microvm-assets/), staging them into the in-cluster rustfs bucket, and applying the microvm SandboxConfig — then deploys the demo worker pool + template.

4. Verify

kubectl get pods -n ate-demo-counter-microvm
kubectl get workerpools -A
kubectl ate get actor-templates -a ate-demo-counter-microvm

Expected:

NAMESPACE                  NAME                                 DESIRED  READY  AVAILABLE
ate-demo-counter-microvm   workerpool.ate.dev/counter-microvm   1        1      1

ATESPACE                   NAME              GOLDEN TAG                             ERROR   AGE
ate-demo-counter-microvm   counter-microvm   b9f6bd93-3c5a-4b64-9d5e-2f8a1c7d0e42           1m

The template is ready once the GOLDEN TAG column is non-empty (the value is a UUID); ERROR means the golden build failed, and -o yaml shows the full error message. An empty GOLDEN TAG with no ERROR means the golden build — a full guest boot plus checkpoint — is still running.

Option B: Apple Silicon macOS via Lima

Lima can run a Linux VM with nested virtualization, exposing /dev/kvm to Docker (and therefore to the kind node) inside the VM. This is a well-trodden path — much of Substrate's development happens on macOS via limactl.

Important

Apple's Virtualization framework supports nested virtualization only on M3 and later — on earlier Apple Silicon (M1/M2), Lima fails with [hostagent] Starting VZ ... FATA exiting. Fall back to Option A on a Linux host.

1. Install Lima and the Docker CLI

brew install lima docker

2. Launch Lima with nested virtualization

Start the docker-rootful template with nested virtualization and a writable home mount:

limactl start --name=docker-nested template:docker-rootful --cpus=8 --memory=16 --nested-virt --mount-writable

If prompted to edit the configuration, verify it includes at least:

cpus: 8
memory: "16GiB"
nestedVirtualization: true
networks:
  - vzNAT: true
mounts:
  - location: "~"
    writable: true

The writable home mount lets the kind/ko workflows write into your checkout, 8 CPUs / 16 GiB is a comfortable floor for the control plane plus a microVM worker, and vzNAT gives the VM outbound networking under the vz VM type.

On arm64 (amd64 is unaffected), Linux 6.19 and newer clear the active state of GICv3 LPIs when folding list registers on a VM exit (upstream commit 73c9726975af). Under nested virtualization, hardware EOI in the L2 microVM guest then leaves the interrupt priority active in ICH_AP1R0_EL2 and blocks subsequent virtio interrupts. Enable kvm-arm.vgic_v3_group1_trap=1 on the Lima guest kernel so L1 KVM traps Group 1 EOI writes and clears the active priority in software:

limactl shell docker-nested sudo tee /etc/default/grub.d/99-vgic-trap.cfg <<'EOF'
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:-} kvm-arm.vgic_v3_group1_trap=1"
EOF
limactl shell docker-nested sudo update-grub
limactl stop docker-nested
limactl start docker-nested

3. Assemble the arm64 assets inside the Lima VM

Assemble the arm64 assets in the guest because macOS does not ship zstd by default:

limactl shell docker-nested

cd <your substrate checkout>    # visible via the writable home mount
./hack/microvm-assets/assemble.sh
exit

The script downloads about 590 MB and takes a minute or two, leaving ~285 MB in bin/microvm-assets/arm64/. That directory is shared with the host through the home mount — the demo script will find the assets there and skip re-assembling.

4. Point the Docker CLI at Lima and bring everything up (on macOS)

export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"
echo 'export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"' >> ~/.zprofile

cd <your substrate checkout>

./hack/create-kind-cluster.sh
./hack/run-microvm-demo-kind.sh

Verify as in Option A, step 4.

Trying it out

On completion, run-microvm-demo-kind.sh prints next steps: create an actor from the counter-microvm template, hit the in-RAM counter, then suspend and resume it and confirm the count continues — proving the guest-memory snapshot round-tripped. The flow is the same as the README Quickstart, just with the microVM template; see the counter demo's microVM steps for background. Note that an actor template showing a GOLDEN SNAPSHOT in the verify step already exercises the runtime end-to-end — the golden snapshot requires a full guest boot and checkpoint.

Troubleshooting

Symptom Root cause Fix
/dev/kvm: permission denied during the kind KVM probe Rootless Docker: the probe container's root is remapped to your user, which can't open the device (660 root:kvm) Use rootful Docker, or sudo chmod 666 /dev/kvm before ./hack/create-kind-cluster.sh
Lima: [hostagent] Starting VZ ... FATA exiting on M1/M2 Apple's Virtualization framework supports nested virtualization only on M3 and later Use an M3+ Mac, or a Linux/KVM host (Option A)
MicroVM golden build or actor boot hangs on arm64 Linux 6.19+ L1 KVM drops LPI active state on VM exits, stalling virtio interrupts in nested guests Add kvm-arm.vgic_v3_group1_trap=1 to the Lima VM kernel command line and reboot (Option B, step 2)