Skip to content

pool, store: dead warm VMs, shared staging, cache reclaim, volume unmount budget - #427

Merged
CMGS merged 23 commits into
mainfrom
fix/warm-store-volume
Nov 3, 2026
Merged

CMGS merged 23 commits into
mainfrom
fix/warm-store-volume

Conversation

@CMGS

@CMGS CMGS commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes from a whole-repo correctness read of the warm pool, the checkpoint stores and volume teardown, plus one promote-path speedup. Stacked on #420.

Fixes

  • Dead warm VM handed out. A warm VM whose VMM exited stayed in the pool: /v1/info and the mesh counted it, refill never replaced it, and a claim got it with 200 and then 502s. The poll, the daemon snapshot and exit events now evict a warm VM reported present and not running, after one Inspect confirms it, destroy it and refill. A node with warm VMs and no claims still polls.
  • no_balloon ignored after a restart. The golden stamp did not cover it, so flipping it adopted the old golden. It is stamped only when set, so default goldens stay adoptable.
  • A guest's own disk-full parked every pool. A warmup that printed no space left on device matched the node capacity signature. A guest exit error no longer carries one.
  • Shared checkpoint_dir: a restart deleted a peer's in-progress staging. The startup sweep now removes only staging older than the generation grace.
  • dir commit pointer not durable. meta.json is fsynced and its directory synced before the archive path drops the VM.
  • S3 fetch cache never reclaimed. A record deleted by another node, or recreated in cache by a canceled fetch's detached flight, stayed on disk until restart. The generation sweep drops a cache unused within the grace whose record is gone from the bucket. The bucket is never touched.
  • A delete that succeeds now vetoes an in-flight heal, so a deleted checkpoint cannot come back after a 204.
  • Volume timeouts answered 500. Attach, setup and detach failures carry the context error, so the documented 504 and 408 hold.
  • Volume unmounts capped at 2 s. A flush longer than 2 s lost its unsynced writes while release succeeded, and live removal failed at 2 s against a documented 30 s. Each unmount now runs on its volume's 30 s budget; once one unmount runs out, a release skips the rest (their markers stay), so a wedged guest costs at most 30 + 2 s.

Performance

  • Promote on the dir store reads holes. A chunk with no data takes the precomputed zero sum instead of a 16 MiB read; digests are unchanged and still match the s3 backend. 10 GiB disk with 200 MiB of data, interleaved 8 rounds, p50: 1.149 s -> 0.099 s (macOS APFS), 1.142 s -> 0.133 s (Linux overlay).

Cost

Nothing on the claim path. One vm list per minute on a node holding warm VMs and no claims; one file and one dir fsync per dir publish; one GET per idle cached id per sweep.

Tests

Each fix has a regression test that fails against the old code or a mutant of the fix. go test -race on pool, engine and store; golangci-lint and asl clean on darwin and linux.

Update

Later commits on this branch:

  • Refill kick after a fresh golden (regression from pool: let a golden's guest finish starting before its snapshot #415). The ready path neither refilled nor kicked, so warm pools filled one 2 s tick late whenever the golden finished after that tick. Interleaved on hardware, with and without the kick: port-e2e 6.0 -> 4.0 s, archive-e2e 46 -> 41 s; against main on another cocoon version, egress lane 9.1 -> 7.1 s.
  • A golden whose clones keep failing is retired and rebuilt. The stamp cannot see a Cloud Hypervisor rollback, and a golden built on v55 and restored on v54 failed every clone with every claim answering 500, forever. After 8 clones in a row fail, whether a refill's or a claim's, the golden's stamp is removed and it is rebuilt, at most once per 30 s. A node-capacity failure, a canceled claim, a failed probe and a promoted template's clone do not count; the directory stays for a claim still cloning it. Hardware: warm 2 and warm 0 pools both recover by themselves, claims answer 500 until the 8th failure, cold-boot during the rebuild, then clone.
  • A warm VM claimed during its eviction goes to recovery instead of waiting for the next poll. With cocoond_socket a dead warm VM is evicted about 0.15 s after its VMM exits, without it up to 10 s; a warm claim stays unprobed, so one landing inside that window answers 200, then 502 until vmm_restart acts. docs/deploy.md states this.
  • The hourly sweep covers the template view on a shared store, which keeps its own fetch cache.
  • The periodic sweep removes aged staging, so a crash leftover on a local root does not wait for a later restart.
  • The port reader starts at 4 KiB and grows on the first frame that needs it. Preview, 1 KiB response: 93 -> 64 us p50, 444 -> 89 KB allocated per request; 4 MiB response: 0.15-1% slower at p50, within its run-to-run spread.
  • The liveness poll no longer skips a node with no claims and no warm VMs.

Merge this before #428: the golden paragraph there describes this branch.

CMGS added 23 commits November 3, 2026 16:24
…ing it out

Liveness watched only claims, so a warm VM killed by OOM or a guest panic stayed in the pool: info and the mesh counted it, refill never replaced it, and a claim got it with 200 and then 502. The poll, the daemon snapshot and an exit event now evict a warm VM reported present and not live, after an inspect confirms it, destroy it and kick refill. The poll no longer skips when only warm VMs exist.
The engine passes --no-balloon only on vm run, so a golden keeps the balloon setting it was booted with; flipping no_balloon and restarting adopted the old golden silently. The field is stamped only when set, so default goldens stay adoptable.
Warmup returns the guest's exit with up to 4 KiB of its output, so a warmup that filled the guest's disk printed 'no space left on device' and parked refill on every pool as at_capacity. A guest exit error now never matches a node-capacity signature.
A restarting node removed every .tmp under the root, so on a shared checkpoint_dir it deleted a peer's in-progress staging, and the peer's publish could then commit a partial record. Staging now ages out by the same GenerationGrace rule the generation sweep uses.
Publish wrote meta.json.tmp and renamed it unsynced, and the archive path destroys the VM right after, so a host crash could leave an empty meta.json and lose the claim. The pointer now goes through ReplaceFileSync: file fsync, rename, record dir fsync, once per publish and off the claim path.
…the bucket

Delete removed only this node's cache dir, so a record deleted by another node, or a canceled fetch's detached download landing after a delete, left a cache dir that nothing reclaimed until disk_reserve tripped. SweepGenerations now removes a cached id unused within GenerationGrace whose meta.json is gone. The bucket itself stays untouched.
DeleteCheckpoint vetoed a pending heal only when the record was absent; when another path published the record while the heal staged, the delete answered 204 and the heal then republished it.
…4 and 408 hold

The engine hides the context error: cocoon exits 'signal: killed' and a silkd call returns a closed-connection read error, so the handler's errors.Is checks never matched and a timeout or cancel answered 500. applyVolume and detachVolumeRun now wrap the context error into a failure when it is set. The live detach's VolumeMutationTimeout moves from engine.DetachVolume to detachVolumeRun, beside the attach's, so the deadline is the caller's to see.
Each umount was capped at VolumeCallTimeout, a slice chosen so one wedged umount could not spend a shared quiesce budget and every mount still got tried. A flush longer than 2 s then failed: release dropped unsynced writes while answering success, and live removal failed at 2 s under a documented 30 s ceiling. The umount now inherits the caller's deadline: live removal the detach's VolumeMutationTimeout, and release a VolumeMutationTimeout per volume, which still tries every mount after a wedged one.
hashChunk read and hashed every byte of a sparse export, so a promote on the default 10 GiB disk spent most of its digest time on holes. A chunk with no data under SEEK_DATA now takes the cached zero sum for its size, through the seek helper the s3 scan already uses; a filesystem that cannot answer keeps the read. Interleaved A/B on a 10 GiB file with 200 MiB of data, p50: macOS APFS 1.149 s -> 0.099 s, Linux arm64 overlay 1.142 s -> 0.133 s, tmpfs 1.148 s -> 0.331 s; digests equal.
…wering

With a 30 s budget per writable mount, a wedged guest held a release for 30 s per mount, up to 242 s for eight. Once an unmount runs out its budget, the guest is treated as wedged: the remaining unmounts are skipped, their markers stay, and the sync fallback runs as before, so the worst case is 30 + 2 s. A busy mount, which fails fast, still lets the other mounts be tried.
…rame needs it

DialGuestPort allocated a 341 KiB reader per dial, and preview dials once per request, so a 1 KiB response allocated about 440 KB. The reader now starts at the 4 KiB handshake cap and is replaced once by a full-size reader the first time a frame overflows it, with the consumed prefix fed back in front of the conn. Interleaved A/B through DialGuestPort with keep-alives off, p50 and allocated bytes per request: 1 KiB 93 us/444 KB -> 64 us/89 KB on macOS, 121 us -> 81 us on Linux arm64; 4 MiB 4.11 ms -> 4.12 ms and 2.79 ms -> 2.80 ms, 708 KB -> 712 KB.
…next tick

The adopted and stale paths kick refill, the ready path did not. Since #415 a fresh golden completes about 1.98 s after Run, so the first warm clones waited for the 2 s refill tick. On cocoon-test2 one kick cut port-e2e from about 6.0 s to 3.95-4.06 s and the archive leg from 46 s to 41 s.
A golden that cannot be cloned was never rebuilt: on cocoon-test2 a golden built under Cloud Hypervisor v55 and adopted after a restart on v54 failed every vm.restore with a memory manager error, and every claim answered 500. The stamp cannot cover the hypervisor without reading cocoon internals, so the general cause is handled instead: when a pool's refills of its current golden fail refillFailStreak times running (capacity failures do not count), the golden is retired as a stamp mismatch retires it and the normal build runs. A cause that survives the rebuild retires it again no sooner than buildRetryDelay later, and the first successful refill ends the streak.
…build

Retirement left refillFails and nextRefill in place, so the rebuilt golden inherited the streak: one refill in flight, and once the 30 s retirement gate had passed, which a slow desktop build outlasts, a single failed first refill retired a working golden again. Late failures of the old golden's clones also counted against the new one. Retirement now resets the streak and its backoff, and only a refill of the current golden counts toward it.
Retirement keyed on refillFails, which also counts warm-probe timeouts and warmup failures, so an overloaded host retired a golden whose clones succeed and every claim then cold-booted until the rebuild. A clone-only streak, counted under the same conditions and reset by any successful clone or a retirement, now gates retirement; refillFails and its backoff are unchanged. A Cloud Hypervisor rollback fails the clone call itself, so it is still caught.
…g it

Retirement renamed the golden directory at once, but a claim clone holds claimClones for the whole clone and dropPool waits for that; a claim mid-clone got ENOENT and answered 500. Retirement now removes only the stamp, clears goldenDir and imageID and bumps goldenGen; the rebuild's exportGolden replaces the directory, as the image-change and reload paths already do. dropPool keeps its own body again.
SweepStaging runs only at startup and keeps staging younger than the generation grace, and SweepGenerations never touched it, so a restart during an archive, checkpoint or promote kept that partial export until a later restart an hour after it. Both sweeps now share one pass over the root that removes staging older than the grace and runs generation retention.
…is its own

Checkpoints and templates stage and cache under separate directories, so stopping after the checkpoint view left a template deleted on another node cached until restart. The early return dates from when the s3 sweep did nothing.
…of waiting for the poll

An exit event read the watched claims, then scanned the warm pool. A VM claimed between those reads, or while evictWarm inspected it, was in neither list, so nothing recovered it until the next poll, 60 s with the stream live. The event now scans the warm pool first, and an eviction that finds its VM gone from the pool hands it to recoverDown, where recoverVMM re-checks under the Transition lock. evictWarm takes the VM name read under m.mu, since a claimed VM's name can change.
Refill never runs for a pool with no warm target, although its golden is still built, so after a Cloud Hypervisor rollback every clone-tier claim of that pool answered 500 until an operator removed the golden. A claim's clone of the pool's current golden now counts under refill's rules: a success resets the streak, a failure adds one, and a canceled claim, a capacity failure, a stale generation, a failed probe and a promoted template's clone say nothing about it. Hot path: warm claims are untouched; a clone-tier claim takes m.mu once more after its clone.
@CMGS
CMGS force-pushed the fix/warm-store-volume branch from 5d43609 to d0fd405 Compare November 3, 2026 08:24
@CMGS
CMGS changed the base branch from review/post-417 to main November 3, 2026 08:24
@CMGS
CMGS merged commit 4b7d574 into main Nov 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant