Repository navigation
docs: explain explicit same-policy drained recovery #41
Description
Source engine PR: RandomDevelopment/ci-fleet#146
Source engine issue: no separate engine issue; recovery was introduced by PR #146.
Status: initial engine implementation merged; correction PR147 pending validation and independent review. No template publication, adopter promotion or host acceptance.
Originally reviewed engine PR146 head: b8673d926a31b741c094f246d35446093d32da42.
Merged engine commit: 1be1d4c94f931d3cc2848c181e266a81cf115f7a.
Post-merge blocker: RandomDevelopment/ci-fleet#146 (comment) identifies the current failed transaction record poisoning otherwise healthy drained rollback verification. Do not treat the earlier clean review or merge as resolving this later finding.
Correction: RandomDevelopment/ci-fleet#147 . Current proposed correction head278c805edd0fcbad3c061c1842ea06a20e84c239 passed independent exact-diff follow-up review (source review only); complete exact-head validation and trusted review evidence remain separate. Terminal publication and checked failure fallback are now covered; double storage failure must explicitly STOP without completion and is not durably critical health or power-loss durability. Initial diagnostic regression head5a54b8e29f41a796992271664951ee75658525ad failed as expected on hosted Linux run40035764756. Final reviewed correction head and merged correction: pending, not supplied by this tracking record.
Gap: operators need adoption and failure guidance for opt-in remote-reconcile.sh --ref FULL_LOWERCASE_40_SHA. Normal default-HEAD timers remain unchanged. Recovery must remain drained, same controller/repository/current runtime engine and unchanged configured capacity/resources/routing/fleet policy; only additive managed_images for the unchanged engine is allowed. Approval, required checks, publication validation, installer transaction receipts, full health and durable LKG readback remain mandatory. It cannot grant admission or active-pilot evidence.
Affected template files: docs/UPDATING.md, docs/MANAGED-IMAGES.md, README.md link if appropriate. Embedded engine documentation: docs/EXPLICIT-DRAINED-RECOVERY.md. No schema, initializer, fleet examples or automatic pin change is needed.
Acceptance checks:
- Explain mutating --ref syntax, malformed/incompatible option rejection, exact commit selection without default-branch changes, unchanged timer behavior and --installed-ref remaining check-only.
- Explain policy-denial, unknown transaction, health-failure and LKG-persistence-failure outcomes; rollback only to verified drained baseline, no automatic admission and no fabricated recovery metadata.
- Require fresh health output with matching process exit and preservation/readback of trusted baseline metadata before live installer approval/checkpoint checks.
- Require verified current-transaction rollback state persistence/readback before the rollback probe, unchanged last-success timestamp, independent host-health checks still effective, and terminal failure reporting preserved. Missing/stale state cannot authorize recovery success. Await the independently reviewed correction before publishing these guarantees.
- Link the immutable reviewed/merged corrected engine guide once available; state this does not work on older pinned managers.
- Run template documentation/link checks and applicable standalone regression checks.
Capability staging: no new desired-state capability field; stage a new reviewed supporting executing manager with compatible runtime/checkpoint recovery using the existing authorized procedure. Selecting candidate source does not upgrade an executing older manager. Preserve approval/review and image-publication policy.
Migration: no data-schema migration. Require independent engine review, complete isolated Linux validation, separately authorized immutable manager pin/adoption, and live-host acceptance. Focused command-double tests are not full host validation. Do not merge/release the template, alter private config, stage hosts or admit work under this issue.
Documentation impact: update adoption, image-bootstrap and recovery guidance. Engine availability, template publication, adopter promotion and live-host acceptance remain separate milestones. This update records the merged initial engine and pending correction; it is not a release or activation authorization.
Activity
Nickfost commentedon Oct 22, 2026
Pending engine PR #146 correction head: b8673d926a31b741c094f246d35446093d32da42 (not merged). Adoption guidance must require fresh health output with matching command exit, and preserved trusted baseline record before live approval/checkpoint checks. Installed-drained only; old 312 manager unsupported; image-free baseline needs matching protected activation checkpoint. No bootstrapability, publication, adoption or live-host acceptance claimed.
Nickfost commentedon Oct 31, 2026
Related installed-baseline recovery guidance now has actual engine implementation under review in RandomDevelopment/ci-fleet#159, exact unmerged head
96432fd23b115bbe07cf94660ba7c669b17b76f4, base05221379641b252001654dd4b4f564f12d286742. Complete source acceptance/merge remain pending, so this is not a selected engine pin or available rollout feature.remote-reconcile.sh --accept-installedfetches only the locked installed commit, preserves approval and installed checks, stops on drift, and requires fresh state-aware full health plus LKG write/readback. It never repairs configuration or applies the retained older baseline. This supplies a distinct path when an independently installed healthy configuration needs accepting without replaying an incompatible older LKG. Existing normal/check-only/no-op and drained --ref behavior stay separate.Required actual standalone update: docs/UPDATING.md and relevant recovery/image guidance, with an immutable link to the final reviewed merged engine guide. No schema, initializer, capability evidence or automatic runtime pin change is needed. Explain old-manager incompatibility, independent trusted executing source, prior metadata preservation, failure/hold requirements and separation from storage admission/workload retirement. Require exact-source full upstream and initialized-adopter suites, independent review, then actual private guidance import/tests before live use. Preserve executor/deployer pins, published release records and active fleet/evidence. This thread owns the actual sequential update after engine acceptance; an issue comment alone is not completion.
Nickfost commentedon Oct 31, 2026
Current engine159 successor is
49b43ae0b4fcc901a0b9766baf832eb80a431356, with an added refusal for image-free/staged acceptance before the installed check. Its52 reconciliation and34 existing recovery methods passed; fresh current-head review/full VM acceptance remain gates, so no supporting merged pin is selected. Actual local standalone preparation now edits docs/UPDATING.md and docs/MANAGED-IMAGES.md on docs/installed-recovery-guidance, based exactly on merged1bdef39. It documents activated managed-image policy, independent executing source, preserved old recovery evidence and stopped/held failure semantics. Ordinary-CI runtime/policy, executor, fleet, schema, validator, historical records and releases stay unchanged in this preparation. Final actual merged source/immutable link, complete upstream/adopter suites and independent review are still required before publishing/merging the actual guidance or importing it privately.Nickfost commentedon Oct 31, 2026
The prepared guidance now has an explicit single completion owner. Frozen local commit
e435c389544ef99750cdd68d629e293a53da8fb2, based on accepted template1bdef39a6932ef9ae2dfdddff34956a9ca8e1e80, was transferred with independently verified tree, parent, patch, bundle and file hashes. Its original checkout remains frozen.The same preparation is reused in a separate checkout, with local successor
3f8c02b644b184709f5bda4c8ce0f278e4fa938f. Onlydocs/UPDATING.mdanddocs/MANAGED-IMAGES.mddiffer from accepted main. The guidance now states active healthy/warning versus matching drained/disabled maintenance/warning, health report/exit consistency, and independent supporting command-source semantics. It also explains that LKG replacement can precede a later readback or terminal-state failure, so operators must inspect actual persisted results and retain prior recovery evidence and automation holds.Ten local Markdown link/anchor checks, both conventional commits, committed-history secret scanning and whitespace passed. Configuration, schema, validator, initializer, evidence, workflows, all test inventories, ordinary-CI baseline840, executor/deployer689 and historical publication records are unchanged. This is clean, local, unpushed preparation, not accepted or published template support.
Engine PR159 remains draft at
49b43ae0b4fcc901a0b9766baf832eb80a431356; its required full acceptance and merge remain independently owned. The preparation notice and final supporting identity stay unset until the actual accepted merge. Final immutable-source guidance, current-head independent review and both complete new-source upstream/initialized-adopter suites remain required. The reserved execution VM has not been accessed or released. Actual accepted template, private import/full acceptance, then authorized host use retain their order. Previous840/template1bdef/private acceptance is preserved and is not being rerun.Nickfost commentedon Oct 31, 2026
Supporting engine PR159 is accepted and merged normally as
e61831c72787df57e76db3030ab7bb0b0986d369, tree7a03ba7376e7e29bb820b168a42bd2a25ad2369f, matching reviewed/fully validated source49. Required run41228217633 attempt1 passed all eight stages with no skips; the nine-member artifact and all stage hashes were exported/verified. Artifact SHA25661de08486c06e567cd34f5136e2a67afd9efa3914508838075eb367e73428811.Actual-template work is now draft PR #46 at
5f53e2755afad161af0721ea6aff3c206b97e221. Only UPDATING and MANAGED-IMAGES change. The immutable merged helper guide is linked as a separate supporting executable pin; ordinary runtime/policy840, executor/deployer689, schema, validator, initializer, evidence, workflows, all inventories and historical publication records remain unchanged.Independent review of predecessor46d found a valid documented-artifact regression caused by the relative helper path. The existing three documentation cases reproduced the failure; successor5f53 makes the external supporting-checkout invocation explicit, and all three pass. The checker and tests are unchanged. The finding was fixed and resolved, and fresh exact-source review is requested. No full suite ran on46d.
The worker owner explicitly released the existing VM for sequential complete final-source upstream and regenerated initialized-adopter suites. Those suites remain pending successor review; their180-second command budgets and protected executor689 remain unchanged. Actual template acceptance/merge and private import/full acceptance still precede live helper use. No release, host operation or storage-retention acceptance is claimed.
Nickfost commentedon Oct 31, 2026
The installed-baseline guidance follow-up is implemented and accepted in merged PR #46, normal merge
da9e34ef5ccce4b0c48ffa0d18440ee7451cf963at 2026-10-31T22:15:35Z. REST and local Git verify tree91fcc19547521e9695617091d9c80e460b69f067equals reviewed and fully validated head5f53e2755afad161af0721ea6aff3c206b97e221; ordered parents are accepted main1bdef and that reviewed head.The two guides link actual merged supporting helper
e61831c72787df57e76db3030ab7bb0b0986d369. Its executed source remains separate from unchanged ordinary runtime/policy840 and executor/deployer689. Schema, validator, initializer, evidence, workflows, complete inventories and historical publication records are unchanged. No template release/tag was published.Both complete new-source upstream and regenerated strict initialized-adopter suites passed all11 stages. Both17-member bundles, all16 indexed files per bundle, all11 stage-log hashes, exact identities and required inventories were independently verified after export. Expected and actual skips are empty. Upstream bundle SHA256
eb6e8c7c07050686686dec1a6b989f3bed0d5f3577d6f15ca1b129b3e9e87dff, report30a9242c82ac4db1ece4d9e52f93c583aea82c719aa9263cad69187d7602eecf. Initialized-adopter bundle10800a8ed8b11278d7ed5843af5be2a43d382b7d8ea6871d4cac42c35451b35d, report586f74f5b5f6cf2ab10a5fc27828f77cdb7431ddc93b24c3f02edaa890711573. The fixture's genuine one-root zero base applies only to that new fictional repository.Current-source Codex review completed without findings before full execution and again after marking ready at22:14:19Z. The earlier external-path finding was reproduced, fixed and resolved without changing the checker or tests. All command budgets remain180 seconds. Final role/executor/account/daemon/image/key/cgroup cleanup passed, and the execution slot was explicitly returned to the operational owner.
Use the exact merged template through the documented unpublished reviewed-revision import. Preserve the actual private predecessor and private corpora/harness/configuration/evidence/workflows/pins/history. Actual private full acceptance and separately authorized helper use remain next gates. No live acceptance, workload retirement, storage admission or scheduled-reclamation proof follows from these repository results.