pool: Manager carries most of the node's state machine #419

Activity

1. CMGS commented on Nov 3, 2026

   

   CMGS

   on Nov 3, 2026

   ContributorAuthor

   More actions

   Direction after measuring the tree with every open audit PR applied (#420, #422, #424, #425, #426, #427, #428, #429, #430, #431 plus a follow-up cleanup). Regex counts, so treat them as approximate. No change lands from this comment.

   The shape now

   • pool: 9440 production lines, Manager with 362 methods, m.mu.Lock at 130 sites (claims and lifecycle 38, archive and failover 35, volumes 20).
   • 255 of the 362 methods never write m.mu in their own body. They hang off Manager for its fields, not because they hold its lock.

   Distinct method pairs called across areas, caller on the left:

   	claim	warm	failover	capture	volume
   claim	116	25	19	3	28
   warm	14	69	5	1	1
   failover	27	17	138	6	9
   capture	8	8	2	68	4
   volume	1	1	0	0	48

   • Claims and failover call each other (27 and 19 pairs), write nearly all types.Sandbox fields between them (35 and 39 sites) and share m.claimed (21 and 19 uses). The recent archive rollback, intent settle and unfence bugs all crossed that line.
   • Volumes call out least, and catalog, volumeRuns and volumeErrors are almost only theirs. Their admission still needs m.mu.
   • Heal, the template cache and the record locks already have their own locks and data. Only their fields sit on Manager.

   Answers to the open questions

   • Goal: fewer things to hold in the head, not independently testable packages. The hard bugs live on the state machine across areas, which a package split would not test.
   • m.mu: stays one lock. Claim admission, quota and volume holds rely on its atomicity, and a second lock needs a cross-lock order.
   • Hot path: the steps below add no interface call and no lock. Each PR still shows the interleaved A/B this issue asks for.

   Proposal, one PR per step, behaviour-preserving, stop and assess after each

   1. Group Manager's fields into named structs with a stated owner and lock: volumes, heal, templates, recLocks, failover (fence, lease, replicas, registry), sweeps (the *Sweep atomics), limits (the semaphores).
   2. Move methods onto those small types, in this package: heal, record locks, the template cache, the volume catalog and any sweep that owns its data, so calls read m.heal.pull(...).
   3. Split the large files by flow: failover.go into row, adopt and intent; claim.go into claim, release and reap; refill.go into golden and refill.

   Not proposed

   • Failover behind an interface (candidate 4): the coupling above is too dense, and an interface would hide rollback orderings that are visible today.
   • A registered list of tick loops (candidate 3): five tickers do not repay the indirection.
   • Volumes as a subpackage: admission needs m.mu. The catalog moves in step 2 as a type in this package, which gets most of the gain.

   Production lines stay about flat, possibly slightly up for the type declarations. This is about reading the code, not about reducing it.

   Timing

   After the open audit PRs merge, to avoid rebasing every one of them over a mechanical move.
2. CMGS commented on Nov 3, 2026

   

   CMGS

   on Nov 3, 2026

   ContributorAuthor

   More actions

   Every PR named above is merged, and #432 is the follow-up cleanup.

   #433 lists the windows the audit accepted in claims, warm pools and failover. The steps here keep those behaviours, so a reader does not have to re-derive them during a move.
3. CMGS commented on Nov 4, 2026

   

   CMGS

   on Nov 4, 2026

   Last edited by CMGS

   ContributorAuthor

   More actions

   Status after #438, which ran steps 1 to 3 in part, together, not one PR per step as proposed. Behaviour-preserving moves, proved by sorted-line comparison per commit.

   Step 1, fields: sweeps groups the single-flight guards; recs (recordLocks), heals (healVetoes) and ckMarks (ckMarkers) own their state.

   Step 2, methods: the record locks, the heal vetoes, the archive delete markers and the volume catalog's holder bookkeeping are now methods of their own types, in this package.

   Step 3, files: refill.go splits into refill.go, golden.go and provision.go; reap.go and archive_delete.go are new; the pool type joins its methods in watermark.go; types.go splits into env.go and volume.go.

   Not done:

   • failover.go (723 lines) into row, adopt and intent.
   • claim.go (664 lines) into claim and release.
   • The field groups volumes, templates, failover and limits.
   • Heal and template cache methods onto their own types.

   Hot path: warm claim, 8 interleaved rounds with the order swapped, pinned to one NUMA node: T-B p50 +0.0015 ms [-0.0068, +0.0097], p90 -0.0016 ms [-0.0158, +0.0127]. Clone tier, 6 rounds: T-B p50 -0.311 ms [-3.175, +2.553]. No measurable cost.

   The shape now, regex counts: 357 Manager methods, 140 m.mu.Lock sites, 9757 production lines. Manager's field count depends on how the groups are counted, so it is not compared with the 95 above.

   This stays open for the steps not done.