fix(storage): classify duplicate-alias fan-out siblings as ineligible, not fatal - #3368
Merged
Merged
Conversation
…, not fatal ## Problem Discovered live 2026-07-28, minutes after deploying polylogue-ihc8's fix (PR #3326, already merged): the daemon's raw-authority frontier census crashed with `RuntimeError: duplicate alias lacks an exact strategy proof: stale raw is not the currently accepted head of this logical source key`, holding the sole writer lock for 543 seconds before failing the whole pass with 10 other queued daemon actors starved behind it -- repeating every retry cycle. `_inspect_duplicate_raw_identity` (storage/repair.py) already returns `status="ineligible"` (never raises) for exactly this shape: several sessions can share one stale native-id-inclusive raw as their accepted head (forked/subagent/resumed sessions replaying the same parent JSONL, polylogue-ihc8's own fix), but only ONE of them can ever fold onto the single available canonical twin. Once that fold lands, every other sibling's own re-inspection legitimately returns "ineligible" (e.g. "canonical raw is already an accepted head"). This is exactly the N:1 fan-out follow-up polylogue-dmvo flagged as needing live confirmation. `_classify_frontier` (storage/raw_reconciler.py) treated any status outside {eligible, already_repaired} as a fatal proof violation and raised -- crashing the *entire* frontier census over one row's expected, benign non-eligibility, not just that row's own classification. ## Solution Add an explicit `duplicate_item.status == "ineligible"` branch that classifies the row as `UNRESOLVED_PROVENANCE`/`NONE` (non-executable, carrying the specific ineligibility reason) instead of raising. Genuinely unexpected statuses (anything not in {eligible, already_repaired, ineligible}) still raise -- this only stops a *known, designed* terminal state from crashing the census. New regression test (`test_duplicate_alias_ineligible_proof_does_not_crash_the_whole_census`) monkeypatches `_inspect_duplicate_raw_identity` to force the exact ineligible status directly, proving the fix rather than relying on reproducing the live multi-pass timing that produces it naturally (not reliably reproducible in a single-pass fixture). Anti-vacuity: reverting the new branch makes this test fail with the exact live RuntimeError message. ## Verification - `devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py` -> 10 passed - `devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate"` -> 85 passed - `mypy polylogue/storage/raw_reconciler.py` -> clean - `devtools verify --quick` -> exit 0 Ref polylogue-dmvo, polylogue-ihc8 Co-Authored-By: Claude <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe duplicate-alias frontier classifier now treats ChangesDuplicate-alias frontier census
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Sinity
added a commit
that referenced
this pull request
Jul 28, 2026
… follow-up polylogue-dmvo's original crash is fixed and confirmed live (PR #3368): one fan-out sibling (896c6b64) successfully folded onto its canonical raw in production. A new, distinct postflight-invariant bug surfaced immediately after (raw_authority.py:1417) -- filed as polylogue-ewfp with full live evidence rather than attempted as a third same-session patch, given its complexity and that the underlying archive data is safe. Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 28, 2026
…#3369) ## Summary Fix a live production bug (discovered immediately after deploying PR #3368) where the raw-materialization pass still crashed on the polylogue-ewfp postflight invariant when multiple duplicate-alias fan-out siblings are selected together in the same apply batch. ## Problem At a single pre-apply census snapshot, the canonical raw is still genuinely dangling for every fan-out sibling, so a batch selection can select more than one sibling's fold together in the same apply_raw_authority_frontier call. The first plan applied commits and claims the canonical; the second plan's own apply then legitimately finds status="ineligible" -- a permanent, not transient, outcome. The code only handled eligible/already_repaired; any other status fell through to raise, which the caller's generic exception handler labeled RETRYABLE, and the postflight check then required that exact plan_id to survive byte-identical forever -- but a sibling's evidence legitimately changes once its canonical is claimed, tripping the check on every subsequent pass. Observed live: a 405-second writer-lock hold ending in an uncaught RuntimeError, with 9 other queued daemon actors starved behind it. ## Solution Extend the existing already_repaired no-mutation-but-resolved pattern to also cover ineligible: commit the empty transaction and return normally so the caller records this plan as EXECUTED (permanently resolved) rather than RETRYABLE. ## Verification - devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py -> 11 passed (new regression test reproduces the exact live batch shape; anti-vacuity confirmed by reverting and observing the exact live error message) - devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate" -> 86 passed - mypy polylogue/storage/raw_reconciler.py -> clean - devtools verify --quick -> exit 0 Ref polylogue-ewfp, polylogue-ihc8 Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 28, 2026
…ification (#3370) ## Summary Fix the final layer of the live incident chain (PRs #3368, #3369 fixed two real bugs and measurably improved the writer-hold from 543s to 210s): crash-recovery finalization was applying the same strict "no plan may change" invariant as a normal apply, which a stuck multi-day-old census can never satisfy once its classification legitimately moves on. ## Problem recover_interrupted_raw_authority_frontier runs on every daemon startup and finalizes every still-planned census. One census had sat unfinalized across multiple days. A stopped-daemon read-only diagnostic showed the affected sessions now classify as unresolved_provenance/refine_quarantined_raw (the underlying raw got quarantined by an unrelated safety mechanism) -- not duplicate_alias anymore, as they were when the census was originally planned. finalize_raw_authority_census's postflight check required every retryable/carried-forward plan to survive byte-identical in the fresh post-census, regardless of whether this was a normal apply or crash recovery. This left the census permanently unable to finalize: every recovery attempt raised on the same stale plan_ids, on every single daemon restart. ## Solution Skip the byte-identical-persistence check when interrupted=True. Correct for a normal apply; wrong for crash recovery, whose entire purpose is reconciling against current ground truth after an arbitrary gap. ## Verification - devtools test tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_duplicate_raw_identity_repair.py -> 41 passed - mypy polylogue/storage/raw_authority.py -> clean - devtools verify --quick -> exit 0 Ref polylogue-ewfp, polylogue-ihc8 Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 28, 2026
… follow-up polylogue-ewfp's crash is fixed and confirmed live: PR #3370 (crash-recovery finalize tolerance) resolves the true recurring-crash mechanism after #3368/#3369 fixed two adjacent layers of the same stack. Verified: the stuck census finally finalized (outcome=success), every previously-starved daemon actor now runs cleanly, no more crash across multiple subsequent passes. Filed polylogue-zaiz for the distinct, unattempted residual: 3 fan-out sessions now correctly classify as refine_quarantined_raw (a separate actuator/workflow) rather than crashing -- non-urgent, daemon operates normally around it. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix a live production bug (discovered minutes after deploying
polylogue-ihc8's fix) where the daemon's raw-authority frontier census crashes entirely on the N:1 duplicate-alias fan-out follow-up case tracked bypolylogue-dmvo.Problem
_inspect_duplicate_raw_identityalready returnsstatus="ineligible"(never raises) when a fan-out sibling's own fold isn't currently possible — e.g. the single available canonical twin was already claimed by a different sibling's fold._classify_frontiertreated any status outside{eligible, already_repaired}as a fatal proof violation and raised, crashing the entire frontier census over one row's expected, benign non-eligibility.Observed live:
daemon writer released actor=maintenance.raw_materialization wait_s=0.000588 hold_s=543.959183 outcome=error queued=10— a 9-minute writer-lock hold ending in an uncaughtRuntimeError, repeating every retry cycle, starving 10 other queued daemon actors.Solution
Add an explicit
ineligiblebranch that classifies the row asUNRESOLVED_PROVENANCE/NONE(non-executable, reason preserved) instead of raising. Genuinely unexpected statuses still raise.Verification
devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py→ 10 passed (new regression test monkeypatches the proof helper to force the exact ineligible status; anti-vacuity confirmed by reverting the branch and observing the test fail with the exact live error message)devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate"→ 85 passedmypy polylogue/storage/raw_reconciler.py→ cleandevtools verify --quick→ exit 0Ref polylogue-dmvo, polylogue-ihc8
Co-Authored-By: Claude noreply@anthropic.com
Summary by CodeRabbit