Skip to content

fix(firecracker): VMs outlived their runs and snapshots - #52

Merged
chxmxii merged 1 commit into
mainfrom
fc-vm-leak
Oct 12, 2026
Merged

chxmxii merged 1 commit into
mainfrom
fc-vm-leak

Conversation

@chxmxii

@chxmxii chxmxii commented Oct 12, 2026

Copy link
Copy Markdown
Member

Bug: every Firecracker run and every snapshot left its VM running. Present since the Firecracker backend landed in v0.3.

Cause

Teardown happens in box __fc, which already runs inside podman's user namespace. Remove killed the VM through inPodmanNS, which enters that namespace with nsenter --user --target <pause>. Entering the user namespace a process is already in fails:

$ podman unshare nsenter --user --mount --preserve-credentials --target <pause> -- true
nsenter: reassociate to namespaces failed: Invalid argument

The error went to /dev/null, so crun kill never ran. Only box down, which runs on the host, stopped a VM. Each leaked VM kept its RAM and its memory snapshot file alive; on the test machine ten of them (from builds and runs) pinned about 4 GB.

Fix

inPodmanNS runs the command directly when the process is already inside the namespace (_CONTAINERS_USERNS_CONFIGURED is set there, by podman and by box's own nsenter path). That fixes every caller, not just this one.

Test

  • The escape suite gains nothing left behind: after a run ends, no VM for the sandbox may remain other than the up one.
  • It fails on the old binary (2 VMs left: the snapshot's and the run's) and passes with the fix on firecracker strict, podman strict and krun standard, with no escapes.
  • By hand: a firecracker build (snapshot), three runs, then up/down → 0 Firecracker processes, 0 crun containers and 0 VM directories left.

Every Firecracker run and every snapshot left its VM running. Teardown
happens in box __fc, which already runs inside podman's user namespace, and
inPodmanNS tried to enter that namespace again through nsenter: setns into
the user namespace a process is already in fails with EINVAL, the error went
to /dev/null, and crun kill never ran. Only box down, which runs on the host,
stopped a VM. Each leaked VM held its RAM and kept its memory snapshot file
alive; on the test machine ten of them pinned about 4 GB.

inPodmanNS now runs the command directly when the process is already in the
namespace. The escape suite gains a check that no VM outlives its run: it
fails on the old binary (two VMs left, the snapshot's and the run's) and
passes on all three backends with this fix.

Present since the Firecracker backend landed in v0.3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chxmxii chxmxii added the bug Something isn't working label Oct 12, 2026
@chxmxii
chxmxii merged commit 16adbab into main Oct 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant