Skip to content

runtime: prove isolation per backend; firecracker needs no krun - #56

Merged
chxmxii merged 1 commit into
chxperiments:mainfrom
KoLDXr00T:fix/isolation-proof-per-backend
Oct 23, 2026
Merged

chxmxii merged 1 commit into
chxperiments:mainfrom
KoLDXr00T:fix/isolation-proof-per-backend

Conversation

@KoLDXr00T

@KoLDXr00T KoLDXr00T commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #4, which introduced the cached isolation proof this fixes. Found while running the escape suite for #55.

The bugs

The isolation check that gates every run, exec and up (EnsureIsolated) is cached once per box home. The cache key is RuntimeIdentity(), which on Linux fingerprints krun, crun and podman whatever the backend.

  1. box up fails on a firecracker-only host. RuntimeIdentity requires krun on PATH and returns no 'krun' on PATH, although firecracker never runs krun. The self-hosted kvm runner has krun installed, which hid this.
  2. A proof made on one backend clears every other. A firecracker sandbox passing the check stores the one token, and a podman sandbox then skips its own check, so its krun path was never proven. podmanBackend.Preflight checks only that krun exists and leaves libkrun to this check. On a host whose crun lacks libkrun, the podman sandbox could therefore run as a plain container on the host kernel, cleared by the firecracker proof. This takes a broken toolchain plus a mix of backends (which backend: auto makes ordinary), and it defeats the one guard against that fallback.
  3. The Firecracker build and its guest kernel are not in the fingerprint. A change to them did not force a re-check.

The fix

  • RuntimeIdentity(backend) fingerprints only what that backend runs:
    • podman and krun: krun and the crun it resolves to (path, size, mtime, --version with its +LIBKRUN tag);
    • firecracker: the pinned Firecracker and kernel hashes (fcAssets checks both before every use), the tools image tag, and crun, which runs the VMM;
    • all backends: podman's version and the backend name.
  • verify.json holds one proof per backend ({"backends": {...}}), written via a temp file and rename. A cache in the old single-token format reads as empty, so each backend re-verifies once after upgrade. That costs one double boot and never clears a run.
  • MarkVerified takes the spec, so box build and box verify record the proof under the sandbox's backend.
  • box doctor reports a row per proven backend. box inspect --security-report reports the proof for the sandbox's own backend.
  • SECURITY.md says each backend is proven on its own.

Checked

  • Unit tests:
    • a firecracker proof doesn't clear podman or krun, and proving podman doesn't drop the firecracker proof;
    • the old and corrupt cache formats read as unverified, and a fresh proof replaces them;
    • with only crun and podman on PATH, the firecracker identity works while podman and krun refuse;
    • an upgraded crun changes the firecracker identity;
    • all three backends get distinct identities.
  • go test -race ./..., go vet, gofmt.
  • CI on the fork (check, vulncheck, three fuzz targets): https://github.com/KoLDXr00T/bluebox/actions/runs/39483914902
  • Escape suite, firecracker, strict and standard, on a GitHub-hosted runner with no krun on PATH (a step fails the job if krun is present): no escapes, one expected warning in standard mode (the VMM runs as your own UID): https://github.com/KoLDXr00T/bluebox/actions/runs/39483949749. Before this fix, the same setup failed at up with no 'krun' on PATH: https://github.com/KoLDXr00T/bluebox/actions/runs/39482241469

Not run: the podman and krun backends' escape suite. That needs crun built with libkrun, i.e. the self-hosted kvm runner, and none is registered yet (#31).

Left as is

box doctor still marks a missing krun as FAIL on a firecracker-only host. That's now only cosmetic, since up works, but it misleads. It could become a warning that names the backends needing krun, as a follow-up.

The isolation check that gates every run was cached once per box home,
keyed on a fingerprint of krun, crun and podman, whatever the backend:

- box up failed on a host with only the firecracker backend, with "no
  'krun' on PATH", although firecracker never runs krun.
- a proof made on one backend cleared every other. A firecracker sandbox
  passing the check let a podman sandbox run without its own krun path
  ever being proven, and the podman backend leaves that to this check:
  on a host whose crun lacks libkrun, it could run as a plain container
  on the host kernel.
- the Firecracker build and its kernel were not part of the fingerprint.

RuntimeIdentity now takes the backend and fingerprints only what that
backend runs: krun and the crun it resolves to for podman and krun; the
pinned Firecracker and kernel hashes and crun for firecracker. verify.json
keeps one proof per backend. A cache in the old single-token format reads
as empty, so each backend re-verifies once. box doctor reports each
backend's proof.
@chxmxii
chxmxii merged commit d9f4fa2 into chxperiments:main Oct 23, 2026
5 checks passed
chxmxii pushed a commit that referenced this pull request Oct 23, 2026
…change (#57)

GitHub's hosted Ubuntu runners expose /dev/kvm, and the firecracker backend
needs nothing more than KVM, vsock, podman and crun. escape.yml runs the
escape suite there under strict and standard isolation, on every pull
request and push to main as well as nightly; each job takes about a minute.
Until now the suite ran only on the self-hosted kvm runner, nightly, and
that runner has not picked up a job in days.

The podman and krun backends need crun built with libkrun, which those
runners lack, so security.yml keeps running them on the self-hosted runner.

Until #56 lands, box up needs krun on PATH whatever the backend, so the job
symlinks krun to crun; the step says to remove that once it does.
chxmxii pushed a commit that referenced this pull request Oct 29, 2026
govulncheck has failed on main since 2026-10-23: GO-2026-6809 in
crypto/x509, reachable from box serve, is fixed in Go 1.26.10, and go.mod
pinned 1.26.9, which CI and releases build with.

escape.yml symlinked krun to crun until #56 made box stop needing krun for
firecracker. #56 has landed, so the step now fails if krun is on PATH
instead, and the job stands for a firecracker-only host.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants