Skip to content

fix(reconcile): isolate current failure during rollback health - #147

Merged
Nickfost merged 10 commits into
mainfrom
fix/rollback-health-current-transaction
Oct 24, 2026
Merged

Nickfost merged 10 commits into
mainfrom
fix/rollback-health-current-transaction

Conversation

@Nickfost

@Nickfost Nickfost commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Summary

Corrects PR146's post-merge rollback-health finding: the current transaction's failed record made a fresh health probe reject a correctly restored drained baseline.

  • After validated applied LKG restoration, write/read back a nonterminal record with rejected desired SHA and restored applied SHA before probing rollback health.
  • Keep real health checks, historical failure severity and the last-success timestamp unchanged; no bootstrap suppression or health-policy change.
  • Require checked terminal rolled_back publication before completion. If rejected, attempt one checked failed fallback with the known identities; if that is also unverified, emit an explicit operator STOP and no completion marker.
  • Scope strict persistence to these rollback paths; retain unrelated best-effort state handling.
  • Verify production state-reader/evaluator behavior with synthetic host collection, including LKG write/no-write/readback, transient state, terminal state, independent health and double-persistence failures.

Storage failure can leave a nonterminal record when both terminal and fallback publication fail. That outcome is explicitly unverified, not a guarantee of durably critical health. Write/readback verification does not claim power-loss durability.

Regression evidence

  • Initial RED 5a54b8e29f41a796992271664951ee75658525ad: 40035764756, production-evaluator regression failed 2 != 3 after restored LKG (27 tests,1 failure).
  • Expanded RED 5baffba8cff96eefd6fd4bd09a67e8a88ceff456: 40037887744, strict transient-state/LKG/identity regressions (31 tests,6 failures).
  • Independent exact-diff review rejected intermediate a34b73303dc241389f733f1942e59a57d7e835a1 for unchecked terminal persistence, despite its separate clean Codex response. That finding was not waived.
  • Terminal-persistence RED df02d262bc51c7772f8c31e185152ad39f6db39b: 40039640847, 33 tests,6 failures.
  • Expanded terminal/double-failure RED d01ada1bb6857d107714a9640c808419986de2f4: 40041084169, 34 tests,8 failures. Earlier injection erroneously matched an empty LKG target; corrected before this RED, with production source unchanged.
  • Current correction: 278c805edd0fcbad3c061c1842ea06a20e84c239. Complete exact-head hosted validation and fresh independent review are pending, not accepted.

Cancelled intermediate runs are not GREEN evidence. Local absolute Git Bash syntax,11literal Python heredocs, outer/nested fixture syntax and git diff --check pass. Hosted Linux provides real integration execution; no managed fleet host is used for tests.

Documentation and template impact

Updated docs/EXPLICIT-DRAINED-RECOVERY.md: checked transient/terminal state, explicit double-failure STOP, persistence limits, unchanged host-health gates and real-evaluator fixture boundary. Reused template issue41, preserving its adoption requirements and separating merged initial M from the pending correction. No schema, desired-state, image, capability, routing or workflow change.

Scope

Owner authorized only narrow public fix, regression tests and follow-up PR for independent review. Keep unmerged. No deployment, private-input edit, primary operation, admission or manager substitution is authorized. Source-only analysis, review and checks are not host acceptance.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please independently review exact head a34b733 for the post-merge PR146 rollback-health finding4355687989. In particular assess checked transient-state persistence/readback, both final-health and LKG persistence failure paths, correct desired/restored identities, preservation of last_success_at, independent health failures and unchanged historical failed/rolled_back severity. Real production-evaluator regressions and retained RED runs are described in the PR. Current full suite is running, not claimed passed. Please provide an exact-commit terminal result. No merge, deployment or host action is requested.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T03:09:23.384205Z 278c805 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: a34b73303d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Nickfost

Copy link
Copy Markdown
Member Author

Independent exact-diff review found a terminal persistence gap in a34b733: best-effort rolled_back finalization can leave the temporary reconciling record behind. This is actionable despite the separate Codex clean result on a34. Regression-only head df02d26 now covers terminal write, apparent missing-write and readback failure in both restoration paths, requiring no completion marker, correct restored identities, unchanged success timestamp and critical ordinary health. Hosted RED run40039640847 is pending. Do not accept a34 or merge this draft; correction and fresh exact-head review remain required. No deployment or host operation.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review final correction278c805edd0fcbad3c061c1842ea06a20e84c239. Your clean result on priora34 does not cover an independently found terminal-persistence gap; that finding was not waived. Added regressions reproduced terminal write/missing-write/readback failures in both rollback paths plus double-failure STOP (correctedRED40041084169). This head checks terminal rolled_back state and one failed fallback, emits no completion if unverified, preserves desired/restored identity and last_success_at, keeps health.py unchanged, and documents storage/power-loss limits. Please inspect whole correction and give exact-terminal result. Full exact-head validation still required; no merge/deployment/host action requested.

@Nickfost

Copy link
Copy Markdown
Member Author

Independent follow-up exact-diff review passed for278c805edd0fcbad3c061c1842ea06a20e84c239: no security concerns or logic errors. The previous terminal-persistence finding is fixed in both affected paths; checked terminal state and failed fallback, explicit STOP on double uncertainty, preserved identities/last_success and unchanged health policy were verified from exact source. Source review only: no test or deployment acceptance. Nonblocking suggestion retained: extend fallback no-op/readback double-failure fixtures. No scope expansion or new source change planned for that suggestion. Full exact-head validation40041887940 and fresh Codex result still pending. Keep PR unmerged and all host/private/admission holds.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 278c805edd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Nickfost

Copy link
Copy Markdown
Member Author

Preparation verification complete for exact278c805edd0fcbad3c061c1842ea06a20e84c239. Latest complete hosted run40041887940 attempt1 SUCCESS, convention120714785397 and full integration120714826032SUCCESS. Actual log executes all8fullstages including container-build-and-cleanup; actual checkout4b2c0f33c9252ceb405099d71bb4f94ae1299420 has reviewed tree08ca791362e252a53b19a4684407f591d5a5981f and exactbase/headparents. Independent follow-up exactdiff reviewPASS. Exact trustedbot199175422 receipt6377119196 and completedclean summary6376688010 verified; public recognizer wording acceptance remains separate, no parser change or transferred private exception. No merge/deployment authorized by this record. PR remains draft and unmerged while owner rollout decision/current private input acceptance is outstanding.

@Nickfost
Nickfost marked this pull request as ready for review October 24, 2026 03:05
@Nickfost
Nickfost merged commit 456c92e into main Oct 24, 2026
3 of 4 checks passed
@Nickfost
Nickfost deleted the fix/rollback-health-current-transaction branch October 24, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant