Skip to content

fix(storage): classify duplicate-alias fan-out siblings as ineligible, not fatal - #3368

Merged
Sinity merged 1 commit into
masterfrom
fix/duplicate-alias-ineligible-crash-dmvo
Jul 28, 2026
Merged

fix(storage): classify duplicate-alias fan-out siblings as ineligible, not fatal#3368
Sinity merged 1 commit into
masterfrom
fix/duplicate-alias-ineligible-crash-dmvo

Conversation

@Sinity

@Sinity Sinity commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

Fix a live production bug (discovered minutes after deploying polylogue-ihc8's fix) where the daemon's raw-authority frontier census crashes entirely on the N:1 duplicate-alias fan-out follow-up case tracked by polylogue-dmvo.

Problem

_inspect_duplicate_raw_identity already returns status="ineligible" (never raises) when a fan-out sibling's own fold isn't currently possible — e.g. the single available canonical twin was already claimed by a different sibling's fold. _classify_frontier treated any status outside {eligible, already_repaired} as a fatal proof violation and raised, crashing the entire frontier census over one row's expected, benign non-eligibility.

Observed live: daemon writer released actor=maintenance.raw_materialization wait_s=0.000588 hold_s=543.959183 outcome=error queued=10 — a 9-minute writer-lock hold ending in an uncaught RuntimeError, repeating every retry cycle, starving 10 other queued daemon actors.

Solution

Add an explicit ineligible branch that classifies the row as UNRESOLVED_PROVENANCE/NONE (non-executable, reason preserved) instead of raising. Genuinely unexpected statuses still raise.

Verification

  • devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py → 10 passed (new regression test monkeypatches the proof helper to force the exact ineligible status; anti-vacuity confirmed by reverting the branch and observing the test fail with the exact live error message)
  • devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate" → 85 passed
  • mypy polylogue/storage/raw_reconciler.py → clean
  • devtools verify --quick → exit 0

Ref polylogue-dmvo, polylogue-ihc8

Co-Authored-By: Claude noreply@anthropic.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of duplicate records that cannot be safely repaired.
    • Prevented one ineligible duplicate from interrupting classification of other records.
    • Clearly marks unresolved records as non-actionable while allowing eligible duplicates to remain available for repair.

…, not fatal

## Problem

Discovered live 2026-07-28, minutes after deploying polylogue-ihc8's fix
(PR #3326, already merged): the daemon's raw-authority frontier census
crashed with `RuntimeError: duplicate alias lacks an exact strategy proof:
stale raw is not the currently accepted head of this logical source key`,
holding the sole writer lock for 543 seconds before failing the whole pass
with 10 other queued daemon actors starved behind it -- repeating every
retry cycle.

`_inspect_duplicate_raw_identity` (storage/repair.py) already returns
`status="ineligible"` (never raises) for exactly this shape: several
sessions can share one stale native-id-inclusive raw as their accepted head
(forked/subagent/resumed sessions replaying the same parent JSONL,
polylogue-ihc8's own fix), but only ONE of them can ever fold onto the
single available canonical twin. Once that fold lands, every other
sibling's own re-inspection legitimately returns "ineligible" (e.g.
"canonical raw is already an accepted head"). This is exactly the N:1
fan-out follow-up polylogue-dmvo flagged as needing live confirmation.

`_classify_frontier` (storage/raw_reconciler.py) treated any status outside
{eligible, already_repaired} as a fatal proof violation and raised --
crashing the *entire* frontier census over one row's expected, benign
non-eligibility, not just that row's own classification.

## Solution

Add an explicit `duplicate_item.status == "ineligible"` branch that
classifies the row as `UNRESOLVED_PROVENANCE`/`NONE` (non-executable,
carrying the specific ineligibility reason) instead of raising. Genuinely
unexpected statuses (anything not in {eligible, already_repaired,
ineligible}) still raise -- this only stops a *known, designed* terminal
state from crashing the census.

New regression test
(`test_duplicate_alias_ineligible_proof_does_not_crash_the_whole_census`)
monkeypatches `_inspect_duplicate_raw_identity` to force the exact
ineligible status directly, proving the fix rather than relying on
reproducing the live multi-pass timing that produces it naturally (not
reliably reproducible in a single-pass fixture). Anti-vacuity: reverting
the new branch makes this test fail with the exact live RuntimeError
message.

## Verification
- `devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py` -> 10 passed
- `devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate"` -> 85 passed
- `mypy polylogue/storage/raw_reconciler.py` -> clean
- `devtools verify --quick` -> exit 0

Ref polylogue-dmvo, polylogue-ihc8

Co-Authored-By: Claude <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9053242c-8afa-4014-8313-b4520badf905

📥 Commits

Reviewing files that changed from the base of the PR and between 3d68cc4 and a1e85d2.

📒 Files selected for processing (2)
  • polylogue/storage/raw_reconciler.py
  • tests/unit/storage/test_duplicate_raw_identity_repair.py

📝 Walkthrough

Walkthrough

The duplicate-alias frontier classifier now treats "ineligible" proof results as unresolved provenance instead of raising an error. A regression test confirms the census continues classifying eligible and ineligible siblings independently.

Changes

Duplicate-alias frontier census

Layer / File(s) Summary
Classify ineligible proofs and validate sibling handling
polylogue/storage/raw_reconciler.py, tests/unit/storage/test_duplicate_raw_identity_repair.py
Ineligible duplicate-alias proofs produce UNRESOLVED_PROVENANCE items with no actuator and the recorded reason, while the regression test verifies eligible siblings remain executable and the census does not raise.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main fix and matches the code changes.
Description check ✅ Passed The description covers Summary, Problem, Solution, and Verification, with only non-critical template sections omitted.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/duplicate-alias-ineligible-crash-dmvo

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Sinity
Sinity merged commit 21bd302 into master Jul 28, 2026
3 checks passed
@Sinity
Sinity deleted the fix/duplicate-alias-ineligible-crash-dmvo branch July 28, 2026 09:37
Sinity added a commit that referenced this pull request Jul 28, 2026
… follow-up

polylogue-dmvo's original crash is fixed and confirmed live (PR #3368):
one fan-out sibling (896c6b64) successfully folded onto its canonical raw
in production. A new, distinct postflight-invariant bug surfaced
immediately after (raw_authority.py:1417) -- filed as polylogue-ewfp with
full live evidence rather than attempted as a third same-session patch,
given its complexity and that the underlying archive data is safe.

Co-Authored-By: Claude <noreply@anthropic.com>
Sinity added a commit that referenced this pull request Jul 28, 2026
…#3369)

## Summary
Fix a live production bug (discovered immediately after deploying PR #3368) where the raw-materialization pass still crashed on the polylogue-ewfp postflight invariant when multiple duplicate-alias fan-out siblings are selected together in the same apply batch.

## Problem
At a single pre-apply census snapshot, the canonical raw is still genuinely dangling for every fan-out sibling, so a batch selection can select more than one sibling's fold together in the same apply_raw_authority_frontier call. The first plan applied commits and claims the canonical; the second plan's own apply then legitimately finds status="ineligible" -- a permanent, not transient, outcome. The code only handled eligible/already_repaired; any other status fell through to raise, which the caller's generic exception handler labeled RETRYABLE, and the postflight check then required that exact plan_id to survive byte-identical forever -- but a sibling's evidence legitimately changes once its canonical is claimed, tripping the check on every subsequent pass.

Observed live: a 405-second writer-lock hold ending in an uncaught RuntimeError, with 9 other queued daemon actors starved behind it.

## Solution
Extend the existing already_repaired no-mutation-but-resolved pattern to also cover ineligible: commit the empty transaction and return normally so the caller records this plan as EXECUTED (permanently resolved) rather than RETRYABLE.

## Verification
- devtools test tests/unit/storage/test_duplicate_raw_identity_repair.py -> 11 passed (new regression test reproduces the exact live batch shape; anti-vacuity confirmed by reverting and observing the exact live error message)
- devtools test tests/unit/storage/ -k "raw_reconciler or raw_authority or duplicate" -> 86 passed
- mypy polylogue/storage/raw_reconciler.py -> clean
- devtools verify --quick -> exit 0

Ref polylogue-ewfp, polylogue-ihc8

Co-Authored-By: Claude <noreply@anthropic.com>
Sinity added a commit that referenced this pull request Jul 28, 2026
…ification (#3370)

## Summary
Fix the final layer of the live incident chain (PRs #3368, #3369 fixed two real bugs and measurably improved the writer-hold from 543s to 210s): crash-recovery finalization was applying the same strict "no plan may change" invariant as a normal apply, which a stuck multi-day-old census can never satisfy once its classification legitimately moves on.

## Problem
recover_interrupted_raw_authority_frontier runs on every daemon startup and finalizes every still-planned census. One census had sat unfinalized across multiple days. A stopped-daemon read-only diagnostic showed the affected sessions now classify as unresolved_provenance/refine_quarantined_raw (the underlying raw got quarantined by an unrelated safety mechanism) -- not duplicate_alias anymore, as they were when the census was originally planned.

finalize_raw_authority_census's postflight check required every retryable/carried-forward plan to survive byte-identical in the fresh post-census, regardless of whether this was a normal apply or crash recovery. This left the census permanently unable to finalize: every recovery attempt raised on the same stale plan_ids, on every single daemon restart.

## Solution
Skip the byte-identical-persistence check when interrupted=True. Correct for a normal apply; wrong for crash recovery, whose entire purpose is reconciling against current ground truth after an arbitrary gap.

## Verification
- devtools test tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_duplicate_raw_identity_repair.py -> 41 passed
- mypy polylogue/storage/raw_authority.py -> clean
- devtools verify --quick -> exit 0

Ref polylogue-ewfp, polylogue-ihc8

Co-Authored-By: Claude <noreply@anthropic.com>
Sinity added a commit that referenced this pull request Jul 28, 2026
… follow-up

polylogue-ewfp's crash is fixed and confirmed live: PR #3370 (crash-recovery
finalize tolerance) resolves the true recurring-crash mechanism after
#3368/#3369 fixed two adjacent layers of the same stack. Verified: the
stuck census finally finalized (outcome=success), every previously-starved
daemon actor now runs cleanly, no more crash across multiple subsequent
passes.

Filed polylogue-zaiz for the distinct, unattempted residual: 3 fan-out
sessions now correctly classify as refine_quarantined_raw (a separate
actuator/workflow) rather than crashing -- non-urgent, daemon operates
normally around it.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant