Skip to content

review: event stream cap, billing order, replica interval, silkd session and pty, compat cuts - #469

Merged
CMGS merged 17 commits into
mainfrom
review/post-467
Nov 5, 2026
Merged

CMGS merged 17 commits into
mainfrom
review/post-467

Conversation

@CMGS

@CMGS CMGS commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Second whole-repo static audit, on main at bcca6c8 (after #459-#467): every source file and all 13 flows re-read in full under every lens, with the rules added after #457's misses (every OK window row states its end state, every holder of a shared lock is listed, env-gated tests are checked against CI, tests that assert a loss are checked against the docs, cross-language caps are checked at the boundary). No hardware or E2E run in this round.

Bugs fixed (each with a test that fails on the old code)

Commit Bug
engine: read cocoon daemon's opening snapshot of a node past 3,000 VMs cocoon sends every VM record in one SSE data: line (~300 bytes each); the 1 MiB line cap broke the stream at every connect on a full node, so liveness fell back to polling for good. Test: a 5,000-VM snapshot (1.26 MB line) synced 0 VMs before.
pool: record a release's usage event once the release commits The release event was written after the teardown (volume quiesce, vm rm); a restart or crash in that window left a claim event with no end, billed up to the deadline.
pool: load a forked claim's tenant for operator forks too The quota check reads the tenant cache only; an operator fork skipped the lookup that fills it, so after a cache flush the tenant's max_claims read as none. Move admission already did the lookup.
pool: answer an invalid PUT /v1/pools spec as invalid config A bad pool limit answered "invalid fork count".
server: count a checkpoint claim handed to a peer sandboxd_claim_redirects_total missed checkpoint redirects.
test: fix the pool benchmarks … run every benchmark once in go-test Four pool benchmarks failed their cleanup invariants since #436/#459; no CI step ran -bench. make go-test now runs each benchmark once.
pool: keep a claim's last replica attempt across ticks it cannot replicate A claim that idles and wakes inside every replicate_seconds never came due, had no registry row, and was lost with its node.
silkd: end a session command when its shell exits A background job holding the session's stdout kept a command hung after the shell exited (exit, set -e, OOM), its io lock held and the session never reaped.
silkd: open the pty master and slave close-on-exec Every pty shell inherited a spare master; confirmed in a linux/arm64 container (silk-1-masters before, 0 after).
pool: refuse a move while a failed save capture is still settling moveBlocker refused an unsettled hibernate intent but not an unsettled save.

Removed (no backward compatibility)

  • The info frame's proto field and KeepAliveProto in Rust, Go, Python and the fixture: nothing reads it since the SDKs stopped probing.
  • The empty-Layer egress fallback and the Pooled callback it needed (records older than review: whole-repo round on main 893bf20 #216).
  • json/v2's case-insensitive name matching that the v2 move kept: config, request bodies and MCP tool arguments now take exact names (a case-variant key is an unknown member).

Cleanups

Docs and godocs matched to the code, including leftovers #457 itself left (the Python mount row, pre-proto-2 sentences, the images trigger path, the secret valuation godoc); restating comments removed; single-caller helpers inlined in the reader router, SDK and silkdtest; an unreachable preview guard and dead lint exclusions removed; test checks that could not fail or could panic on the failure path fixed; the fixture test now ranges the decoder tables.

Numbers

Production -55 net (+86 -141), tests +35 net, docs +24 -30. Comment lines in Go and Rust: +11 -25.

Kept without code

  • Reader VMs after an image re-import: cocoon's DiskAttachSnapshot refuses a snapshot whose layers and blobs differ from the reader VM's, so the read wakes the claim; no sandboxd mechanism needed.
  • Reconcile clearing an unlisted hibernate intent on a stopped VM: cocoon persists the snapshot before it terminates the VMM, so a live capture CLI always shows the VM running [contrived].
  • A release racing finalize's arm (a root release of a claim id within the ~ms before its claimant gets the reply) [contrived].
  • TestS3BackendContractRealEndpoint never runs in CI: MinIO images are no longer publicly pullable; picking a replacement S3 server is an owner decision.
  • Unmeasured perf leads: perf: unmeasured leads from the second whole-repo audit (after #467) #468.

Gates

  • GOWORK=off make go-lint (5 modules x linux/darwin): 10 x 0 issues. (golangci cache cleaned first: a warm cache panics SA4023 in e2e)
  • asl -forwarder=false ./... both GOOS, all 5 modules: clean
  • GOWORK=off make go-test (race, plus one pass of every benchmark): green
  • Rust make lint + make test on macOS and in rust:1 linux/arm64: green (silkd 165 tests, boot/init 17)
  • Python 3.12: sdk/python 273, openai 10, langchain 15 passed

CMGS added 17 commits November 5, 2026 18:14
cocoon sends every VM record in the sync event's one data line, about 300 bytes a
record, so the 1 MiB line cap broke the stream at every connect on a full node
and liveness fell back to polling vm list for good.
…ter its teardown

The claim is durably gone at the commit; a restart or crash during a slow teardown
(volume quiesce, vm rm) left a claim event with no release, which the folding rule
bills up to the claim's deadline.
… applies

The quota check reads the tenant cache only; an operator fork skipped the lookup
that fills it, so after a cache flush or restart the tenant's max_claims read as
none and the fork overshot it. A removed tenant still does not stop the operator.
…, and run every benchmark once in go-test

Four benchmarks seeded claims around addClaimLocked or without a store row and
failed the cleanup invariants; no CI step ran -bench, so nobody saw it.
…ropping the live-pool fallback for records older than the field

Every claim records its layer at commit since #216 and adopt and move carry it, so
the fallback served only pre-#216 records; the Pooled callback threaded through
outbound existed for it alone.
…icate

replicateOnce dropped the attempt time whenever a claim was hibernated, archiving
or settling, so a claim that idles and wakes inside every interval never came due,
had no registry row, and was lost with its node.
…2's exact names, dropping the v1 case-insensitive matching the v2 move kept

A case-variant key is now an unknown member: config and strict bodies reject it,
other bodies ignore it.
…ell's stdout EOF

A background job started in the session holds the stdout pipe past the shell, so
an exit, a set -e failure or an OOM kill of the shell left the command hung, its
session's io lock held and the session never reaped.
…ts jobs do not inherit them

libc openpty leaves both fds inheritable; every pty shell held a spare master, so a
job outliving the shell never saw its terminal hang up.
Drops the Python mount row and the pre-proto-2 sentences #457 left, the images
trigger path, and the old-template digest note; corrects the data_dir entries,
the drain error, the shared-store probe, idle upstream conns, env snapshot reads,
lineage markers, secret valuation, exit codes, Ready, the tenant_claims help,
Metas, the checkpoint fallback and the Python lookup and keep_alive docs.
…test, drop an unreachable preview guard and lint exclusions for directories that do not exist
…ad error, scope table contexts to their subtest, cover every decoder with a fixture, and drop dead or always-true checks

The fixture test now ranges the decoder tables instead of a hand list a new verb
could miss; the watcher garbage test asserts the error it promises; smoke and
bench tools type their 409 check and stop racing the handshake reassignment.
moveBlocker refused an unsettled hibernate intent but not an unsettled save; a move
that won the transition over the save's settle carried SaveSnap to the target and
left a listed fork snapshot on the source that the startup sweep keeps.
@CMGS CMGS changed the title review: second whole-repo static audit after #467 (event stream cap, billing order, replica interval, silkd session and pty, compat cuts) review: event stream cap, billing order, replica interval, silkd session and pty, compat cuts Nov 5, 2026
@CMGS
CMGS merged commit 8ee9366 into main Nov 5, 2026
4 checks passed
@CMGS
CMGS deleted the review/post-467 branch November 6, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant