Skip to content

feat(fleet): durable movement foundations and bounded reconciler - #37

Merged
forhappy merged 97 commits into
mainfrom
codex/fleet-operations-foundations
Oct 4, 2026
Merged

forhappy merged 97 commits into
mainfrom
codex/fleet-operations-foundations

Conversation

@forhappy

@forhappy forhappy commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Latest published checkpoint — 9110095

  • Merge 85f731e integrates main at 4c1c982 and resolves the three reported conflicts. The actor retains fleet admission kinds alongside main's optional request permits. Both LTX documentation contracts are preserved. PR mergeability was confirmed after both pushes.
  • Replica retries now retain original placement positions when a candidate is removed. A real three-process one-refusal regression fails with the original cursor at a 7/5 split and passes with the corrected cursor at 6/6. The ordinary balance assertion, 12 successful reads, receipts, recruitment/refresh, eviction and joined shutdown remain unchanged. CI explicitly runs the regression and verifies its selector exists.
  • Frozen pre-merge source passes all 13 Rust commands and nine static gates: 1,683 framework passes, 320 minion passes (2,003 distinct), local LTX 54, Axum without defaults 16 and cookbook 256 plus check/Clippy/docs/builds. Focused/process repeats are excluded. Its 1,117 Rust/Cargo paths and original failures/campaigns are retained under /Users/haipingfu/Workspace/crabbuild-target/cellule-smoke-f0a15a6/evidence.
  • The merged correction passes all-target/all-feature workspace check, formatting/boundary/layout/schema/docs checks (136 Rust snippets, 1,302 links), and both ordinary and one-refusal three-process scenarios. Both have 6/6 read distribution and joined hosts. Its exact 1,122 Rust/Cargo paths, source archive, native binary hash and logs are retained under /Users/haipingfu/Workspace/crabbuild-target/cellule-pr37-merge-4c1c982/evidence.
  • Full current-head CI is required and is still running. Parent f0a15a6 fails Compose balance, a minion follower-observation roster deadline, leased forwarded-command/c1 throughput (88.40% versus the unchanged 90% gate), and the routing aggregate. Object-only routing passes (minimum gated throughput 94.21%). The unchanged comparator reproduces both original 178-file campaigns. The hosted smoke artifact lacks an attempt trace; the new regression isolates the retry defect without establishing which transient refusal occurred there. The separate follower deadline and performance causes remain open.

The complete W1–W10 goal remains unfinished. Highest priorities remain current-head CI, source/failed-owner reader and follower succession with durable process/accepted-work evidence, SettleRoles/Finalize, receiver-session recovery and minion maintenance/receiver-loss commands, then W9/W10 campaigns and exercised runbooks. SourceSuccessor blocks and SettleRoles/Finalize remain refused. See the current progress record.


Previous final-root checkpoint

Current checkpoint: exact final reader root

Canonical executable: crates/cellule-host/minion; Cargo target fleet_operations.

  • Preserve the exact last installed Cell/incarnation-scoped root in the native reader's existing shared state through successful refresh, detachment and joining. Derive receipts from that same root. Failed, cancelled or uninstalled refreshes cannot replace it with a source publication.
  • Expose the root in native lifecycle observations and exact original ReaderEnrollmentRetirement; retain it after peer clone joining and shutdown. Native reader inventory fingerprint v2 binds every root field. Prior ephemeral continuations refuse their normal topology check; persisted enrollment/evidence bytes and signed peer contracts remain unchanged.
  • A new real native regression covers source publication versus installed view, failed/successful refresh, detachment, peer clones and released ledgers. Existing cancelled-query/close cases verify final root retention. A new fingerprint regression distinguishes every exact root field, including a changed digest at identical counters.
  • The real host handoff now refreshes before joining the original reader, drains its exact busy writer through canonical release, restores another runtime, preserves SQL readback and verifies the final root through the successor's canonical lineage/origin path. A substituted digest with unchanged scope/counters refuses.
  • Two isolated one-variable negative controls fail at the intended assertion: retaining the previous root after refresh, and omitting the exact digest from the inventory hash. Restore production byte-identically before broad qualification.

Verification

All 13 fresh Rust commands and nine static gates pass on the isolated Rust 1.99 snapshot with its mounted target. All 1,117 committed Rust/Cargo paths match the qualified manifest.

Check Result
Framework, all features, locked 1,682 passed; zero failures; 36 documented ignores.
Canonical minion 320 passed; zero failures or ignores.
Distinct framework/minion passes 2,002; focused repeats, negative controls and overlapping local LTX excluded.
Local LTX / Axum without default features 54 / 16 passed.
Features/targets, Clippy, warning-denied API docs, minion build Passed.
Fresh cookbook All 256 tests, check, Clippy, warning-denied docs and binary builds passed.
Static gates Passed; 136 Rust snippets and 1,280 local links.

Manifest SHA256: 42c25e30a926cabb324f48d3f9ea632b1ce47ffb1ab9418e634843b6949dd104.

Preliminary failures correct a test-only position construction, an invalid idle-transfer assumption after the new command, and insufficient fixture memory for the newly exercised origin verifier. The exact busy handle now joins/releases through ordinary drain; the receiver is provisioned for the verifier's existing metadata envelope. No idle grace, production admission, profile, gate or expected evidence was weakened. Original failed logs/source remain retained.

Exact source archives/manifests, commands, original preliminary failures, negative sources/logs and restored qualification are retained under /Users/haipingfu/Workspace/crabbuild-target/cellule-reader-final-root-bd381d6/evidence.

CI and remaining delivery

Published parent bd381d6 remains MERGEABLE; all checks are terminal: 32 successes, two failures and three documented skips. Workspace, follower/object capacity, MSRV, contracts, cookbook quality/scenarios, Compose smoke, model, website and fuzz checks pass.

  • Leased routing passes; lowest gated throughput is 92.68% of baseline.
  • Object-only forwarded-command and local-command throughput at concurrency 16 fail at 89.83% and 88.60%, against the unchanged 90% gate. Latency gates pass; the routing aggregate fails.
  • The unchanged comparator reproduces both original 178-file campaigns. Original binaries, manifests, stage rows, file hashes and failed job logs are retained. Baseline and candidate binaries are byte-identical to their respective earlier campaigns, whose failing lanes differed. Publication cost, provider/runner variation and execution order remain hypotheses; no independent performance correction is established.

Parent CI does not qualify this new source. Fresh current-head CI is required. The earlier overload endpoint failure's cause remains open despite the passing subsequent focused/full/hosted runs; preserved diagnostics do not independently establish a fix.

The full W1–W10 goal remains active and unfinished. SourceSuccessor remains blocking; SettleRoles/Finalize remain refused. Exact root identity and native joining supply no complete current replacement policy, accepted external-work joining, durable process retention or physical maintenance completion.

Priority Remaining stream
0 Qualify current-head CI; independently diagnose/correct leased routing and the original overload failure.
1 Compose exact installed reader joining/final-root derivation with fresh source/failed-owner reader and follower succession policy, durable provider/process retention and complete original/current requests.
2 Finish accepted native/external work and SettleRoles/Finalize with original action joining before terminal drain handoff, native Stopped, withdrawal, boot retirement and committed completion; finish Cron/Blob owners and fault matrices.
3 Finish receiver-session loss/recovery/adoption, refusal/unknown supervision, sustained convergence and minion maintenance/receiver-loss commands.
4 W9 process/provider/mixed-binary/load campaigns; W10 exercised runbooks and rollout/rollback.

See the full plan, execution evidence, and exact reader contract.

Add journal-backed fleet operations, retained intents and enrollment contracts, resource-backed receiver preparation, finite host actions, fresh actor inspections, and a caller-driven movement controller. Include the local SQLite journal example, focused evidence, CI model coverage, and the full remaining fleet implementation plan.
Share the leased-node admission-pressure scenario between the executable and tests. Move a bounded journal-backed batch to two receivers, reopen the controller client, verify original outcomes and restored state, fence old source handles, and join all runtimes and journal jobs. Preserve incomplete production role coverage and remaining plan scope. Fix the owner-loss fixture to inspect the retained original fencing source and require empty resource ledgers.
…s-foundations

# Conflicts:
#	scripts/check-web-rust-examples.py
@forhappy
forhappy merged commit fa548bb into main Oct 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant