Repository navigation
pool, store: dead warm VMs, shared staging, cache reclaim, volume unmount budget - #427
Merged
Merged
Conversation
This was referenced Nov 3, 2026
…ing it out Liveness watched only claims, so a warm VM killed by OOM or a guest panic stayed in the pool: info and the mesh counted it, refill never replaced it, and a claim got it with 200 and then 502. The poll, the daemon snapshot and an exit event now evict a warm VM reported present and not live, after an inspect confirms it, destroy it and kick refill. The poll no longer skips when only warm VMs exist.
The engine passes --no-balloon only on vm run, so a golden keeps the balloon setting it was booted with; flipping no_balloon and restarting adopted the old golden silently. The field is stamped only when set, so default goldens stay adoptable.
Warmup returns the guest's exit with up to 4 KiB of its output, so a warmup that filled the guest's disk printed 'no space left on device' and parked refill on every pool as at_capacity. A guest exit error now never matches a node-capacity signature.
A restarting node removed every .tmp under the root, so on a shared checkpoint_dir it deleted a peer's in-progress staging, and the peer's publish could then commit a partial record. Staging now ages out by the same GenerationGrace rule the generation sweep uses.
Publish wrote meta.json.tmp and renamed it unsynced, and the archive path destroys the VM right after, so a host crash could leave an empty meta.json and lose the claim. The pointer now goes through ReplaceFileSync: file fsync, rename, record dir fsync, once per publish and off the claim path.
…the bucket Delete removed only this node's cache dir, so a record deleted by another node, or a canceled fetch's detached download landing after a delete, left a cache dir that nothing reclaimed until disk_reserve tripped. SweepGenerations now removes a cached id unused within GenerationGrace whose meta.json is gone. The bucket itself stays untouched.
DeleteCheckpoint vetoed a pending heal only when the record was absent; when another path published the record while the heal staged, the delete answered 204 and the heal then republished it.
…4 and 408 hold The engine hides the context error: cocoon exits 'signal: killed' and a silkd call returns a closed-connection read error, so the handler's errors.Is checks never matched and a timeout or cancel answered 500. applyVolume and detachVolumeRun now wrap the context error into a failure when it is set. The live detach's VolumeMutationTimeout moves from engine.DetachVolume to detachVolumeRun, beside the attach's, so the deadline is the caller's to see.
Each umount was capped at VolumeCallTimeout, a slice chosen so one wedged umount could not spend a shared quiesce budget and every mount still got tried. A flush longer than 2 s then failed: release dropped unsynced writes while answering success, and live removal failed at 2 s under a documented 30 s ceiling. The umount now inherits the caller's deadline: live removal the detach's VolumeMutationTimeout, and release a VolumeMutationTimeout per volume, which still tries every mount after a wedged one.
…rr (govet shadow)
hashChunk read and hashed every byte of a sparse export, so a promote on the default 10 GiB disk spent most of its digest time on holes. A chunk with no data under SEEK_DATA now takes the cached zero sum for its size, through the seek helper the s3 scan already uses; a filesystem that cannot answer keeps the read. Interleaved A/B on a 10 GiB file with 200 MiB of data, p50: macOS APFS 1.149 s -> 0.099 s, Linux arm64 overlay 1.142 s -> 0.133 s, tmpfs 1.148 s -> 0.331 s; digests equal.
…wering With a 30 s budget per writable mount, a wedged guest held a release for 30 s per mount, up to 242 s for eight. Once an unmount runs out its budget, the guest is treated as wedged: the remaining unmounts are skipped, their markers stay, and the sync fallback runs as before, so the worst case is 30 + 2 s. A busy mount, which fails fast, still lets the other mounts be tried.
…rame needs it DialGuestPort allocated a 341 KiB reader per dial, and preview dials once per request, so a 1 KiB response allocated about 440 KB. The reader now starts at the 4 KiB handshake cap and is replaced once by a full-size reader the first time a frame overflows it, with the consumed prefix fed back in front of the conn. Interleaved A/B through DialGuestPort with keep-alives off, p50 and allocated bytes per request: 1 KiB 93 us/444 KB -> 64 us/89 KB on macOS, 121 us -> 81 us on Linux arm64; 4 MiB 4.11 ms -> 4.12 ms and 2.79 ms -> 2.80 ms, 708 KB -> 712 KB.
…next tick The adopted and stale paths kick refill, the ready path did not. Since #415 a fresh golden completes about 1.98 s after Run, so the first warm clones waited for the 2 s refill tick. On cocoon-test2 one kick cut port-e2e from about 6.0 s to 3.95-4.06 s and the archive leg from 46 s to 41 s.
A golden that cannot be cloned was never rebuilt: on cocoon-test2 a golden built under Cloud Hypervisor v55 and adopted after a restart on v54 failed every vm.restore with a memory manager error, and every claim answered 500. The stamp cannot cover the hypervisor without reading cocoon internals, so the general cause is handled instead: when a pool's refills of its current golden fail refillFailStreak times running (capacity failures do not count), the golden is retired as a stamp mismatch retires it and the normal build runs. A cause that survives the rebuild retires it again no sooner than buildRetryDelay later, and the first successful refill ends the streak.
…build Retirement left refillFails and nextRefill in place, so the rebuilt golden inherited the streak: one refill in flight, and once the 30 s retirement gate had passed, which a slow desktop build outlasts, a single failed first refill retired a working golden again. Late failures of the old golden's clones also counted against the new one. Retirement now resets the streak and its backoff, and only a refill of the current golden counts toward it.
Retirement keyed on refillFails, which also counts warm-probe timeouts and warmup failures, so an overloaded host retired a golden whose clones succeed and every claim then cold-booted until the rebuild. A clone-only streak, counted under the same conditions and reset by any successful clone or a retirement, now gates retirement; refillFails and its backoff are unchanged. A Cloud Hypervisor rollback fails the clone call itself, so it is still caught.
…g it Retirement renamed the golden directory at once, but a claim clone holds claimClones for the whole clone and dropPool waits for that; a claim mid-clone got ENOENT and answered 500. Retirement now removes only the stamp, clears goldenDir and imageID and bumps goldenGen; the rebuild's exportGolden replaces the directory, as the image-change and reload paths already do. dropPool keeps its own body again.
SweepStaging runs only at startup and keeps staging younger than the generation grace, and SweepGenerations never touched it, so a restart during an archive, checkpoint or promote kept that partial export until a later restart an hour after it. Both sweeps now share one pass over the root that removes staging older than the grace and runs generation retention.
…is its own Checkpoints and templates stage and cache under separate directories, so stopping after the checkpoint view left a template deleted on another node cached until restart. The early return dates from when the s3 sweep did nothing.
…of waiting for the poll An exit event read the watched claims, then scanned the warm pool. A VM claimed between those reads, or while evictWarm inspected it, was in neither list, so nothing recovered it until the next poll, 60 s with the stream live. The event now scans the warm pool first, and an eviction that finds its VM gone from the pool hands it to recoverDown, where recoverVMM re-checks under the Transition lock. evictWarm takes the VM name read under m.mu, since a claimed VM's name can change.
Refill never runs for a pool with no warm target, although its golden is still built, so after a Cloud Hypervisor rollback every clone-tier claim of that pool answered 500 until an operator removed the golden. A claim's clone of the pool's current golden now counts under refill's rules: a success resets the streak, a failure adds one, and a canceled claim, a capacity failure, a stale generation, a failed probe and a promoted template's clone say nothing about it. Hot path: warm claims are untouched; a clone-tier claim takes m.mu once more after its clone.
CMGS
force-pushed
the
fix/warm-store-volume
branch
from
November 3, 2026 08:24
5d43609 to
d0fd405
Compare
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from a whole-repo correctness read of the warm pool, the checkpoint stores and volume teardown, plus one promote-path speedup. Stacked on #420.
Fixes
/v1/infoand the mesh counted it, refill never replaced it, and a claim got it with 200 and then 502s. The poll, the daemon snapshot and exit events now evict a warm VM reported present and not running, after one Inspect confirms it, destroy it and refill. A node with warm VMs and no claims still polls.no_balloonignored after a restart. The golden stamp did not cover it, so flipping it adopted the old golden. It is stamped only when set, so default goldens stay adoptable.no space left on devicematched the node capacity signature. A guest exit error no longer carries one.checkpoint_dir: a restart deleted a peer's in-progress staging. The startup sweep now removes only staging older than the generation grace.dircommit pointer not durable.meta.jsonis fsynced and its directory synced before the archive path drops the VM.Performance
dirstore reads holes. A chunk with no data takes the precomputed zero sum instead of a 16 MiB read; digests are unchanged and still match the s3 backend. 10 GiB disk with 200 MiB of data, interleaved 8 rounds, p50: 1.149 s -> 0.099 s (macOS APFS), 1.142 s -> 0.133 s (Linux overlay).Cost
Nothing on the claim path. One
vm listper minute on a node holding warm VMs and no claims; one file and one dir fsync perdirpublish; one GET per idle cached id per sweep.Tests
Each fix has a regression test that fails against the old code or a mutant of the fix.
go test -raceon pool, engine and store; golangci-lint and asl clean on darwin and linux.Update
Later commits on this branch:
cocoond_socketa dead warm VM is evicted about 0.15 s after its VMM exits, without it up to 10 s; a warm claim stays unprobed, so one landing inside that window answers 200, then 502 untilvmm_restartacts.docs/deploy.mdstates this.Merge this before #428: the golden paragraph there describes this branch.