pool: accepted windows from the 2026-11 audit, with reopen triggers #433

Activity

1. CMGS commented on Nov 4, 2026

   

   CMGS

   on Nov 4, 2026

   ContributorAuthor

   More actions

   Update after #436, #437 and #438.

   Closed: "An env write during an archive". #437 rewrites the row when a write lands while the archive intent is set (TestAnEnvWriteDuringTheArchiveReachesItsRow).

   Added, same format:

   Window	What happens	Size	Reopen when
   A sandboxd restart during a hibernate dump	The cocoon CLI dies on SIGPIPE at its next log line after the snapshot lands. The VMM stays paused, the claim reads running, exec answers 502 at 10 s. A retried hibernate answers 204 in about 2.6 s and the wake is intact. cocoon documents that it does not ignore SIGPIPE, so sandboxd does not adopt that snapshot	Any restart during a dump, 5 of 5 runs under strace. A dump past cocoon's own 10 minute limit reaches the same state	A restart during a dump must not need a client retry. An automatic retry of the journaled hibernate at restart is the option, not built
   A restarted sandboxd during a dump does not listen yet	The listener opens only after the dump ends. The stack is a rename blocked on writeback; the file is not identified	18-128 s in three runs	A restart has an availability target
   A wake or release behind a long dump	An exec, or a release of a claim with a registry row, waits on the claim's transition lock	Up to the dump budget, 15 minutes plus its 3 minute grace	A caller needs a bound shorter than the dump
   A hung refill clone	Clone and clone-snap now take the dump budget, so a hung one holds a refill_concurrency slot	Up to 18 minutes, one slot per hung clone, loud	Refill starves behind a hung clone
   max_running slots are first come	A wake past the cap answers 429 with no queue and no reservation	Until a running claim stops	A tenant needs a reserved share
   A moved claim with no owner found	SDK Close returns nil after a 404 when Lookup finds no owner, so the claim lasts to its deadline, or to its retention once archived	No deadline when archive_delete_after_seconds is 0	Evacuated claims leak in practice

   Follow-up tied to cocoon: once cocoon's capture cancel ships (cocoon #297, merged, not released), a canceled capture waits for the resume for up to 10 minutes. dumpTermGrace (3 minutes) must then exceed that, or SIGKILL lands first. v0.1.22 does not include this.

   The moves in #438 kept every behaviour above.
2. CMGS commented on Nov 5, 2026

   

   CMGS

   on Nov 5, 2026

   ContributorAuthor

   More actions

   Update after #455 (merged 14b4692). Windows that the post-#447 audit found and kept without a mechanism, in the same format:

   Window	What happens	Size	Reopen when
   A received claim's restart while vm inspect keeps failing	The target retries the claim every 2 s and logs each try. That claim's release, reap and evacuation stay gated until inspect answers; nothing else waits	As long as the cocoon CLI cannot inspect; hardware with one injected failure: one retry, then liveness restarted the guest	Inspect outages long enough for gated claims to show up
   A source killed in the final round while the target writes its replica row	The old replica checkpoint, already cleared from the source journal, is never reclaimed by sandboxd	One orphan archive checkpoint per such crash, left to the operator reclaim tool	Orphan archive checkpoints show up in the reclaim report
   A source killed after the target committed	Its move record stays started until a node's hourly sweep deletes it; a new move of that claim overwrites a started record	Up to an hour, no effect on moves	Never, by design
   claims.json unwritable for over an hour during a move	The hourly sweep can delete an aborted record the source still journals; after a restart the source finds no record and drops its claim	Needs a full or read-only data dir for an hour, which also stops every claim, release and hibernate on the node	A node runs with an unwritable data dir that long
   A move round with a non-regular tar entry	The target writes it as a plain file; only a peer holding the fleet token can send one	[contrived]	The control port's token stops being the trust boundary
3. CMGS commented on Nov 6, 2026

   

   CMGS

   on Nov 6, 2026

   ContributorAuthor

   More actions

   Windows the static audit of 2026-11-06 (PR #481) reasoned through and left without a mechanism. Each is bounded and loud or self-converging.

   Window	What happens	Size	Reopen when
   tenants/pg cold-lookup limiter	One 200/s limiter per node covers tokens it has not cached. A flood of random bearer tokens, or a cache flush (Replace, listener reconnect, 100k cached names), answers 503 tenant store down; retry to uncached tenants	Until the flood stops, or about N/200 s after a flush	Legitimate 503s from the limiter in production
   Peer heal budget split	The 30-minute heal budget is shared by the owners left, so a healthy owner of a very large checkpoint can time out when many owners are tried	Fails only above about 28 GB at 1 Gbit/s with 8 owners	Heal timeouts on large checkpoints
   A refused overlay address release, then a restart	The hourly retry is in memory; a restart before it lands leaves the PG row allocated	One address per meta-store outage at release time followed by a restart inside the hour; each refusal logs a Warn	tun address exhaustion or repeated release warnings
   A PG Assign whose reply is lost after commit	The claim runs without tun (Warn) and one address stays held	One address per lost reply inside the 3 s query timeout	The same
   64 KiB per queued inbound tun frame	A stalled guest reader pins up to 1024 frames (64 MiB) per claim	Documented at sendQueue	sandboxd RSS growth under tun load
   tun send drops during the stop drain	Frames charged before a send that fails while the underlay closes	The shutdown drain only	—
   A tun disable during a checkpoint, fork or replica capture	No tun_down reaches the guest; its tun0 carries nothing	An API disable racing a capture (documented)	Guests left with a dead tun0 in practice
   A deterministic inject refusal after a warm pop	The popped warm VM is destroyed and refilled	One per 400 claim; our SDKs do not retry a 400	A client retrying 400s drains a pool
   rewriteArchiveRow after two refused replaces	Returns nil	Needs two stale-epoch refusals on a row this node owns while the caller holds Transition	—
   clearArchiveIntent after a failed commit	Skips the orphan checkpoint delete, since the durable journal may still name it	ENOSPC plus a release before the next restart	Orphan archive checkpoints in the store
4. CMGS commented on Nov 6, 2026

   

   CMGS

   on Nov 6, 2026

   ContributorAuthor

   More actions

   Update after #482 (merged 72770b5), which changes one row and adds three windows it kept without a mechanism.

   Changed: "max_running slots are first come". With archive_failover, the wake of an archived claim that max_running or disk_reserve refuses now moves to a peer with room. The owner answers 404, and the client's Lookup finds the adopter. A hibernated claim, or any claim on a node without archive_failover, still answers 429 with no queue and no reservation. The reopen trigger is unchanged.

   Added, same format:

   Window	What happens	Size	Reopen when
   The quiet period after a failed capacity hand-off	A node runs one hand-off at a time. After one fails, it asks no peer for 30 s, so an archived wake it refuses in that time answers 429 even if a peer has freed room since	Up to 30 s per node after each failed hand-off	Archived wakes answer 429 while a peer had room
   A peer's room is checked when it adopts, not reserved	If the adopter fills up before the client's wake arrives, that wake is refused there and may be handed on again	One Lookup round trip	Archived claims bounce between full nodes
   Peers are asked in order	The wake that runs a hand-off waits up to 5 s for each peer that does not answer, and calls queued on that claim's transition lock wait with it	5 s per unreachable peer, on full nodes only	A wake latency target on a cell with unreachable peers

   The gossiped room flag that would replace the quiet period was ruled out on 2026-11-06: the claim redirect already skips peers that are full on max_running, so the flag would have one user.