Skip to content

docs(fleet): self-contained control plane design and implementation plan - #42

Merged
forhappy merged 2 commits into
codex/fleet-operations-foundationsfrom
codex/fleet-control-plane-plan
Oct 2, 2026
Merged

forhappy merged 2 commits into
codex/fleet-operations-foundationsfrom
codex/fleet-control-plane-plan

Conversation

@forhappy

@forhappy forhappy commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Define a supported Cellule control plane that reduces application-team integration and operational work, with explicit contracts for large fleets and production release evidence.

  • Ship a controller binary/container, CLI, bundled UI, and worker SDK. The supported integration requires no custom journal, observer, transport, election, or maintenance implementation from application teams.
  • Specify partition leases, atomic fleet and installation budgets, cross-partition movement, complete snapshot/delta inventory, and terminal-evidence archiving. These require versioned extensions to the existing framework contracts.
  • Cover capacity control, disruption limits, bulk maintenance, rolling upgrades, identity rotation, tenant isolation, restore-generation fencing, and operational diagnostics.
  • Define proposed qualification profiles up to 10,000 nodes and one million Cells, including failover, noisy-fleet isolation, historical churn, and a 72-hour soak. These are release targets, not measured capacity claims.
  • Provide ordered implementation packages CP0–CP13, source boundaries, acceptance commands, and a release evidence ledger that rejects skipped baseline capabilities.

docs/fleet-control-plane-audit.md records 18 design gaps, their evidence, design resolutions, implementation packages, and remaining qualification requirements. docs/fleet-control-plane-plan.md incorporates those resolutions into the self-contained implementation handoff.

Scope and dependency

This PR changes two documentation files and remains stacked on #37 (codex/fleet-operations-foundations). It consumes the canonical fleet safety contracts and explicitly requires versioned amendments before partitioning, inventory compaction, or restored-generation authorization can be enabled. Current framework limits remain authoritative until those extensions are implemented and qualified.

The documents do not implement the control plane or certify the unfinished native fleet/maintenance work. Production readiness requires the specified implementation, native fault tests, provider qualification, scale measurements, and operator acceptance. Retarget to main after #37 merges.

Validation

Checks passed in an isolated checkout of this PR's branch with only the documentation update:

  • python3 scripts/check-doc-links.py: 1,171 local links and anchors resolve.
  • python3 scripts/check-doc-rust-fences.py: 110 documented Rust snippets parse.
  • git diff --cached --check: passed.
  • Updated documents match the audited source copies; whitespace, code-fence balance, and JSON examples checked.

No runtime or process tests were run for this documentation-only change. Proposed binaries, APIs, and commands are clearly identified as implementation targets.

@forhappy
forhappy merged commit 353904e into codex/fleet-operations-foundations Oct 2, 2026
4 checks passed
forhappy added a commit that referenced this pull request Oct 4, 2026
* feat(fleet): add durable movement foundations and bounded reconciler

Add journal-backed fleet operations, retained intents and enrollment contracts, resource-backed receiver preparation, finite host actions, fresh actor inspections, and a caller-driven movement controller. Include the local SQLite journal example, focused evidence, CI model coverage, and the full remaining fleet implementation plan.

* feat(fleet): run real three-node overload and controller reconstruction

Share the leased-node admission-pressure scenario between the executable and tests. Move a bounded journal-backed batch to two receivers, reopen the controller client, verify original outcomes and restored state, fence old source handles, and join all runtimes and journal jobs. Preserve incomplete production role coverage and remaining plan scope. Fix the owner-loss fixture to inspect the retained original fencing source and require empty resource ledgers.

* fix(fleet): fence unaccepted dispatches atomically before retry

* fix(docs): compile website examples with target dependency artifacts

* feat(fleet): resume lost releases under a replacement controller

* fix: use atomic try_update across supported Rust toolchains

* feat(fleet): cordon maintenance nodes before settled evacuation

* feat(runtime): quiesce Cell foreground work for maintenance

* feat(fleet): release quiesced maintenance Cells canonically

* feat(fleet): plan busy maintenance from configured Cell envelopes

* docs(fleet): record busy demand and merged-source evidence

* test(fleet): cover Effect and Activity maintenance recovery

* feat(fleet): fence boot startup with durable enrollment

* fix(runtime): join read replica closure across retained clones

* feat(fleet): pin exact reader sources before enrollment

* feat(fleet): own durable reader enrollment and retirement

* feat(fleet): confirm every follower fence before maintenance close

* feat(fleet): own requested maintenance rotation through host drain

* feat(fleet): prepare exact follower enrollment and retain retirement fences

* feat(fleet): journal managed followers before native enrollment

* feat(fleet): bind planning to the complete durable enrollment roster

* feat(fleet): observe retained reader enrollment through paused I/O

* feat(fleet): expose bounded managed follower enrollment progress

* docs(fleet): self-contained control plane design and implementation plan (#42)

* docs(fleet): add self-contained control plane implementation plan

* docs(fleet): close control plane production design gaps

* test(fleet): retain reader fixture admission and live boot headroom

* feat(fleet): expose retained durability supervisor progress

* feat(fleet): retain request-bound native snapshots

* feat(fleet): drive real count convergence with native observations

* feat(fleet): refresh running boot intent before lease renewal

* test(qualification): capture canonical roots after bounded publication observation

* feat(fleet): expose canonical reader lifetime observations

* fix(fleet): bind reader continuation to complete native states

* perf(node): avoid repeated verification during canonical decoding

* fix(readers): reconcile retained enrollment work periodically

* fix(readers): preserve reconciliation across lease boundaries

Charge temporary reader inventory as metadata so pre-lease startup and fenced producer repair keep their canonical owner healthy. Preserve native admission checks and add public regressions for both boundaries.

Make snapshot retry registration explicit and inject model timeouts at observed before/after acceptance boundaries. Retain the original deadline, permit, source-error and resource assertions. Record rejected Linux runs and final native/Linux qualification evidence.

* fix(host): retain node task joins across shutdown retries

* fix(host): retain complete drain attempts across caller cancellation

* fix(fleet): distinguish source release refusal from unknown close

Native pressure eviction can invalidate a selected source before an accepted fleet release reaches its actor. Tag exact-position preflight refusals before canonical deactivation and preserve the original cause. The executor records Rejected so existing cancellation can join unused receiver credit. Canonical close and prior accepted actions without original results remain Unknown. Add public regressions for both boundaries; preserve required two-movement evidence and record its remaining reliability gap.

* test(runtime): qualify Cron dispatch across maintenance handoff

* fix(fleet): inspect and adopt actual clean-release successors

* fix(fleet): avoid competing with oldest-first pressure eviction

Pass actor recency through the advisory demand and prefer recently used settled Cells on active pressure donors. Keep native eviction, authority checks, drain ordering, capacity bounds and safe refusal unchanged.

Make the native fixture adversarial and deterministic. Preserve the reproduced before failure and 20 paired passing replays. Broad verification records the remaining count-equilibrium failure and the unfinished W1-W10 delivery streams.

* fix(host): bind native closing to checked boot withdrawal

* fix(qualification): defer evidence writes beyond arrival clocks

* feat(host): evacuate managed readers with checked replacements

* feat(host): collect and recheck full native fleet inventories

* feat(host): traverse and recheck foreign follower authority

* feat(host): verify managed follower evacuation against live replacements

* feat(host): verify cross-node fleet role coverage

* feat(host): retain role coverage in reconciler observations

* feat(runtime): retire canonically recovered follower ensembles

* feat(host): publish recovered follower enrollment closure

* feat(host): publish failed boot retirement with process evidence

* feat(host): close failed receiver reader enrollments

* feat(host): persist and revalidate reader replacement evidence

* feat(host): persist follower replacement evidence and relocate minion

* feat(runtime): expose complete pinned recovery manifest inventory

* feat(host): confirm original process closure before recovery

* feat(runtime): retain original owner history before departure

* feat(runtime): verify complete bounded catalog traversals

* feat(fleet): retain complete original failed-boot writers

Join original process evidence and authenticated complete catalogs before retaining all full owner Controls in bounded canonical pages. Commit the immutable operation/process set atomically in minion’s existing accepted SQLite owner; preserve original bytes across replay, reconstruction, lost replies and cancellation.

Verify 1,586 workspace tests and 218 minion tests on an isolated snapshot, with workspace Clippy and API docs denying warnings. Metadata grants no successor-prefix, availability or maintenance settlement rights; those integration and qualification streams remain open.

* feat(runtime): retain canonical acquisition metadata before admission

Keep the exact successful claim input and claimed or recovered position through immutable bounded origin metadata. Preserve ordinary authority and rollback, exact replay, storage errors and accepted receiver work across canceled waiters and lost replies.

Verify 1,595 workspace and 218 minion tests on an isolated snapshot, plus local LTX, all-target checks, Clippy and API docs with warnings denied. Historical metadata supplies no complete prefix, availability or maintenance settlement proof; those fleet streams remain open.

* feat(runtime): verify exact root prefixes before fleet serving

Retain additive native prepared-root lineage before root authority CAS.
Prove exact released/recovered prefixes through publication and compaction,
verify bounded complete origin inventories, and recheck native serving.
Charge transient proof metadata through the existing runtime memory ledger.

Keep minion canonical and document proof limits and remaining W1-W10 work.

Verification: 1,609 workspace tests and 218 minion tests passed; 36 workspace
cases ignored. Local LTX: 54 passed. Workspace all-target/all-feature check,
Clippy and API docs passed with warnings denied; all static gates passed.
747 frozen Rust/Cargo paths match the tested isolated source manifest.

* feat(fleet): verify original sealed recovery suffix before serving

* fix(publication): overlap durable lineage with native uploads

Create fresh lineage without an absence GET; reconcile conflicts through one verified additive ETag merge. Join lineage retention with native uploads under existing I/O admission, preserving original errors and waiting for both before publication. Reuse an exact private confirmation before authority CAS; retain the canonical path for external and rebased proposals.

* fix(ltx): compose final predecessor before lineage retention

Verified private compaction must name the original authority predecessor before metadata retention. Carry that native factory context through one preparation and remove it from immutable read views. Cover 41 real publications through compaction with zero lineage reads and one lineage write per proposal.

* feat(fleet): reload complete committed original writer inventory

* feat(fleet): collect complete original boot suffix inputs

* docs(fleet): record suffix checkpoint and remaining delivery

* feat(fleet): verify all original writers against native successors

* docs(fleet): document native successor collection and repair cookbook lock

* docs(fleet): record successor verification and remaining delivery

* feat(fleet): retain original successor proofs in planner observations

* test(fleet): qualify inherited recovery and stabilize lease fixtures

* fix(ci): satisfy Rust 1.99 error-size and slice lints

* feat(fleet): retain fresh failed-boot closure observations

* fix(fleet): pin process confirmation to retired boot evidence

* feat(host): retain fresh role evacuation proofs in fleet observations

* docs(fleet): record qualified role observation and remaining delivery

* Observe original accepted fleet executor work

* Fence source enrollment after maintenance closure

* fleet: retain original maintenance enrollments

* fleet: match every original maintenance policy obligation

* fleet: discover every native maintenance donor policy

* fleet: confirm original enrollment nonexecution

* fleet: join exact original reader requests

* fleet: retain exact final reader roots

* fix(runtime): preserve replica rotation across retries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant