Skip to content

vm: rebuild owned networking before launch when the host lost it - #51

Merged
CMGS merged 3 commits into
masterfrom
fix/cni-recover-on-start
Sep 23, 2026
Merged

CMGS merged 3 commits into
masterfrom
fix/cni-recover-on-start

Conversation

@CMGS

@CMGS CMGS commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Problem

A host reboot removes every network namespace and TAP device, while the VM record and disks survive. After that, every vm start of a VM whose networking cocoon-macos created (--net cni, or the bridge/tap auto modes) fails before QEMU runs. launch enters the persisted netns and opens the persisted TAP, and nothing rebuilds them. The unquiesce after start is a no-op when the netns is missing. The only way out today is vm rm plus recreate.

Fix

launch now checks the VM's owned NIC through the cocoon network provider before it starts QEMU. When the provider's Verify fails, launch runs Prepare and then Add with AddRecover. This is the same recovery cocoon runs for its own VMs at start.

  • The NIC keeps its MAC. For CNI, that is the MAC set on the in-netns interface.
  • The provider's existing network record is reused.
  • The rebuilt TAP name and netns path are written back into the VM record before it is saved.
  • A failed rebuild fails the start instead of launching a guest with half-built networking.
  • A VM with a user-supplied --tap is untouched.
  • Provisioning and recovery now share one single-queue VM config, so the recovered TAP keeps the queue layout QEMU's -netdev tap expects.

The new TestStartChecksOwnedNetworkBeforeQEMU checks that starting an owned-network VM goes through the provider first and fails before QEMU when that check fails.

Evidence

$ GOWORK=off make lint
==> golangci-lint GOOS=linux
0 issues.
==> golangci-lint GOOS=darwin
0 issues.

$ GOWORK=off make fmt-check
FMTCHECK rc=0

$ GOWORK=off go test -race -count=1 ./...        # darwin
ok  	github.com/cocoonstack/cocoon-macos/cmd	1.303s
ok  	github.com/cocoonstack/cocoon-macos/cmd/image	1.540s
ok  	github.com/cocoonstack/cocoon-macos/cmd/vm	2.344s
ok  	github.com/cocoonstack/cocoon-macos/home	2.007s
ok  	github.com/cocoonstack/cocoon-macos/qemu	2.303s
NO-FAIL-LINES

$ go test -race -count=1 ./...                   # golang:1.27 container, go1.27.1 linux/arm64
ok  	github.com/cocoonstack/cocoon-macos/cmd	1.018s
ok  	github.com/cocoonstack/cocoon-macos/cmd/image	1.018s
ok  	github.com/cocoonstack/cocoon-macos/cmd/vm	6.589s
ok  	github.com/cocoonstack/cocoon-macos/home	1.019s
ok  	github.com/cocoonstack/cocoon-macos/qemu	1.195s
NO-FAIL-LINES
--- PASS: TestStartChecksOwnedNetworkBeforeQEMU (0.03s)

$ GOOS=linux asl ./...
ASL-LINUX rc=0
$ GOOS=darwin asl ./...
ASL-DARWIN rc=0

# mutation check: recovery call removed from launch, same container
    net_linux_test.go:95: Start error = launch qemu: exec: "qemu-system-x86_64": executable file not found in $PATH, want the network recovery failure before qemu launch
--- FAIL: TestStartChecksOwnedNetworkBeforeQEMU (0.02s)

Notes

  • Pending Linux verification. The actual netns and TAP rebuild after a host reboot has not been exercised yet. It needs a Linux host with CNI plugins and a bridge. This change is verified by unit tests and static gates only.
  • The CNI address may change. No guest IP is persisted, so a rebuilt CNI NIC takes a fresh lease from the IPAM plugin. The guest may come back with a different address.
  • A missing bridge now fails the start. Starting an owned bridge-mode VM now builds the bridge provider, which requires the bridge device to exist. If it is missing, start fails with recover network: bridge <dev>: ... instead of launching QEMU on a TAP with no bridge.

A host reboot wipes every netns and TAP while the VM record and disks
survive, so every later start of a VM with cocoon-owned networking
(CNI, or bridge/tap auto mode) failed before QEMU ran: launch entered
the persisted netns and opened the persisted TAP, and nothing rebuilt
them. The only way out was rm plus recreate.

launch now verifies the owned NIC through the cocoon provider and, when
it is missing, re-runs Prepare and Add with AddRecover, the same
sequence cocoon's own start uses. The NIC keeps its MAC, the provider's
existing network record is reused, and the rebuilt TAP and netns path
are written back to the record before it is saved. A failed rebuild
fails the launch, since half-built networking strands the guest.

The single-queue VMConfig moves into one helper that provisioning and
recovery share, so the recovered TAP keeps the queue layout QEMU's
-netdev tap expects.
@CMGS
CMGS merged commit 5943df8 into master Sep 23, 2026
3 checks passed
@CMGS
CMGS deleted the fix/cni-recover-on-start branch September 23, 2026 16:34
CMGS added a commit that referenced this pull request Sep 24, 2026
…ps, docs (#52)

* build: bump github.com/cocoonstack/cocoon 22b7392e16b0→1951f352e9f2, github.com/gofrs/flock v0.13.0→v0.13.1, github.com/projecteru2/core v0.1.3→v0.1.5, oras.land/oras-go/v2 v2.6.1→v2.6.2

* fix: reject a double quote in the VNC password

HMP parses an argument that starts with a double quote as a quoted string, so a password like "abcdef" was stored as abcdef.

* review: tighten comments and log the missed power-down at warn

Drop dead gosec/errcheck nolint directives, restated DataDisks and VNC proxy comments, and the nonexistent export op; cut the gc godocs to one fact each; the guest ignoring the power-down is a degraded stop, so it logs at warn like cocoon's stop escalation.

* review: take the stdlib v4 UUID for the SMBIOS identity and fold the partition wait predicate

* cut: drop the gc live-VM predicate and the empty-Add guard

vmInUse never matched: the collectors only ask about IDs Resolve left after dropping every recorded VMID, sweep holds the gc lock and every VM lock so no record appears before Collect, and the bridge collector passes an 8-char prefix.
provisionNet's empty-result guard never fired: both Add backends return one config per spec on success and AddRange(0, 1) is one spec.

* docs: name the per-VM disk and OpenCore overlay, the start-time network rebuild, and the e2e script

cli.md named overlay.qcow2 and a copied OpenCore; the VM dir holds disk.qcow2 and an OpenCore overlay. networking.md gains the #51 rebuild of lost owned networking. The e2e.sh header names the script it is.

* fix: fail a qmp-input command on a QMP error or a closed socket

cmd dropped error replies and EOF, so a failed screendump left OCR reading the previous /tmp/_ocr.ppm and a rejected key or click still exited 0.

* cut: drop qmp-input's ocrdclick op

Its only callers, the picker double-clicks in build-qemu-macos.sh, went away in 5ef9948; nothing in the workspace calls it.

* review: list qmp-input's live ops, fold the agree_button docstring, and take f-strings

* build: pin cocoon to dcec1fa and take cliutil.GroupCommand for the command groups

cocoon #264 made GroupCommand the one owner of the group contract (NoArgs + help); the two groups here now use it. Checked on the built binary: vm and image print help with rc 0, vm bogus and image bogus fail with unknown command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant