fix(storage): make membership-replay conflicts retry-eligible by type - #3646
Merged
Sinity merged 2 commits intoAug 3, 2026
Merged
Conversation
…ion type Extends test_bundle_replay_respects_unconvertible_single_session_head's failing-branch assertions to check parse_error starts with "MembershipReplayConflictError:" instead of only checking it is non-None. This is red against current code: apply_raw_membership_classification's two head-conflict guards (storage/sqlite/archive_tiers/revision_governance.py) still raise a plain RuntimeError, so parse_error is recorded as "RuntimeError: membership replay cannot replace a head with unresolved byte-append evidence: ..." -- a message whose exact wording has already drifted once since PR #2718 introduced this guard under different phrasing ("... cannot replace an unconvertible byte head"), and storage/repair.py's raw-materialization retry-candidate query can only ever match by literal message text. A production Codex session (019f49d8-0185-7c43-8793-db6e57db13e1, polylogue-5iz4) hit this exact guard under the #2718-era wording in July and has never been retried since: repair.py's candidate query treats any parse_error outside a two-item literal allowlist as permanently terminal. Ref polylogue-5iz4
Problem: a production Codex session (native_id 019f49d8-0185-7c43-8793-db6e57db13e1, 804 raw_sessions rows from incremental full-snapshot capture of a growing rollout.jsonl) has zero sessions in index.db. Its largest revision (90,822,451 bytes) has never been parsed; the next-largest (90,156,590 bytes, confirmed byte-for-byte prefix of the largest) carries parse_error 'RuntimeError: membership replay cannot replace an unconvertible byte head' from PR #2718 (2026-07-12). That exact message no longer exists anywhere in the codebase -- #3211 and the later polylogue-miwv fix narrowed and reworded both of apply_raw_membership_classification's head-conflict guards. Reading the live archive read-only confirmed: (1) the actual codex parser handles the real 90.8MB blob fine (29,280 messages, no crash); (2) raw_revision_heads has zero rows for this logical_source_key, so the guards' "existing_head is not None" branches cannot fire on a fresh attempt; (3) storage/repair.py's raw-materialization candidate query excludes any parse_error other than two hardcoded exact-match transient-error strings ('OperationalError: database is locked', 'decode:...No such file or directory') as permanently terminal. The bug is not a live crash loop -- it is a stale, retry-blocking parse_error frozen under message wording that predates two subsequent refactors, with no mechanism that ever clears or re-evaluates it. Solution: give apply_raw_membership_classification's two head-conflict raises (storage/sqlite/archive_tiers/revision_governance.py) a dedicated MembershipReplayConflictError(RuntimeError) type, following the existing ActiveByteRevisionChainError precedent. mark_raw_parse_failed records parse_error as f"{type(exc).__name__}: {exc}", so this gives repair.py's retry logic a message-wording-independent marker instead of an allowlist of brittle exact-message strings. Both the SQL candidate query and the _raw_materialization_retryable_missing_blob_error Python gate (the real single source of truth -- the SQL clause is a pre-filter, not merely an optimization; a row that clears the SQL WHERE is independently re-checked there) now recognize a 'MembershipReplayConflictError:' parse_error prefix as retry-eligible. Alternatives rejected: matching on the existing message text more broadly (e.g. a regex over "cannot retire"/"cannot replace") was rejected because it re-creates the exact fragility that caused this bug -- any future rewording of the guard's message silently breaks the match again. A dedicated exception type is the only marker guaranteed stable across message edits. Verification: - devtools test tests/unit/sources/test_live_batch_support.py -k test_bundle_replay_respects_unconvertible_single_session_head -- red before the fix (2 failed: parse_error was plain "RuntimeError: ..."), green after (4 passed). - devtools test tests/unit/storage/test_repair.py -k test_raw_materialization_retries_membership_replay_conflict_failure -- new test, confirms a MembershipReplayConflictError-prefixed parse_error is retry-eligible while a sibling row carrying the OLD plain-RuntimeError text (anti-vacuity control) remains excluded. - devtools test tests/unit/storage/test_repair.py tests/unit/sources/test_live_batch_support.py -- 142 passed, 3 failed; confirmed by reverting this fix and rerunning the same 3 tests that they fail identically on unmodified code (test_append_multi_session_payload_is_ rejected_before_index_write, test_full_ingest_skips_durably_excised_ content_without_aborting_batch, test_full_ingest_writes_archive_with_ route_observability -- all fail with an unrelated "did not replay to exactly one session" RuntimeError from _parse_raw_revision_chain). - Read-only against the live archive (/realm/db/polylogue, mode=ro): confirmed the actual codex.parse_stream() call succeeds on the real 90,822,451-byte blob (29,280 messages), and confirmed raw_revision_heads/raw_revision_applications have zero rows for this logical_source_key today. Ref polylogue-5iz4
|
Warning Review limit reached
Next review available in: 10 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Sinity
added a commit
that referenced
this pull request
Aug 3, 2026
…cale (#3666) ## Summary Adds a regression test reproducing the real production shape behind polylogue-5iz4 (many incremental full-snapshot captures of one growing Codex session, plus a same-identity duplicate arriving from a second "incident recovery" path) and demonstrates that `MembershipReplayConflictError` is a transient, recoverable refusal, not a permanent one. ## Problem polylogue-5iz4's AC required a redacted structural fixture reproducing the real 804-revision Codex session's crash shape before the retry-eligibility fix in PR #3646 could be verified against something closer to the real shape than the original small #2718 pin (2-3 message sessions). PR #3646 fixed the retry-eligibility bug (a stale `parse_error` text that permanently excluded the raw from every future rebuild) but did not add the growth-chain-scale fixture the bead's AC calls for, and its own scope note said it could not claim the retried classification actually succeeds without live evidence. Reading the live archive read-only (`/realm/db/polylogue`, `mode=ro`) confirmed the real 22 full-revision blobs for this session form ONE clean, unforked byte-prefix chain (every smaller blob is an exact prefix of every larger one, verified pairwise) and that `raw_revision_heads`/`sessions` are empty for this identity today. Reproducing the actual `MembershipReplayConflictError` needed the same mechanism as the existing `test_bundle_replay_respects_unconvertible_single_session_head` pin (a same-identity bundle arriving at a second path, colliding with a QUARANTINED dangling append fragment hanging off the accepted head), scaled up to a growth chain with many messages instead of 2-3. ## Solution Added `test_growing_file_incident_recovery_duplicate_recovers_after_head_advances` to `tests/unit/sources/test_live_batch_support.py`. It: 1. Establishes an accepted head via 25 real incremental single-session ingests (growth-chain scale). 2. Introduces a same-identity "incident recovery" bundle at a second path plus a dangling QUARANTINED append fragment, reproducing `MembershipReplayConflictError` (fail-closed, correct behavior). 3. Confirms the recorded `parse_error` carries the retry-eligible `MembershipReplayConflictError:` marker `storage/repair.py` recognizes (polylogue-5iz4 / #3646). 4. Clears the interfering accepted head (matching the live archive's own empty `raw_revision_heads` row for this identity) and retries — the same durable raw now succeeds and reaches the index with the expected message_count. Notable finding during construction: `storage/repair.py`'s offline `repair_raw_materialization` reprocesses every retained typed-`'full'` raw for a `logical_source_key` on every pass (including the accepted head's own cohort), which re-establishes the interfering head before ever reaching the colliding raw in the same pass — so the recovery in this test is demonstrated via the live watcher's own retry path (`_ingest_full_paths_sync` again), not via `repair_raw_materialization`. That offline-repair gap is noted directly in the test as a real follow-up rather than papered over. **Root cause conclusion for the live production session**: the retry-eligibility fix already merged in #3646 is the actual fix. #3646's own read-only investigation already confirmed the codex parser succeeds on the real 90.8MB blob and that `raw_revision_heads` has zero rows for this identity today, so a fresh rebuild-index pass over the real archive should reach the simple byte-chain replay path directly (the real content is one clean chain) without ever re-tripping the membership-governance guard. No further production code change was found necessary this session; this PR is fixture/regression-test work only. ## Verification - `devtools test tests/unit/sources/test_live_batch_support.py -k test_growing_file_incident_recovery_duplicate_recovers_after_head_advances` — 1 passed, verified stable across 4 repeated runs. - `devtools test tests/unit/sources/test_live_batch_support.py -k "revision_replay or membership"` — 7 passed. - `devtools test tests/unit/storage/test_repair.py -k membership` — 2 passed. - `devtools verify --quick` — exit 0. Ref polylogue-5iz4 Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Aug 19, 2026
…4019) ## Summary A Codex rollout that grows in place accumulates both `full` snapshots and `append` fragments under one logical source. Append fragments are deliberately never parsed for identity, and the parser-census receipt written for them was `failed` — a status that satisfies neither branch of the census-complete gate. Every raw-materialization pass therefore re-selected the fragment, rewrote the same receipt, and reported "paused until the persisted parser census completes" forever, which stopped raw-replay planning for **every** `full` snapshot sharing the authority component. This receipts a byte-governed fragment as the complete observation it is, so those components converge. ## Problem `polylogue-39kcs` (bucket B7 of the dyica classification — the only genuine gap in the 111-failure population): codex session `019f49d8-0185-7c43-8793-db6e57db13e1` is acquired, parsed, and censused but absent from `index.db`. It is the only codex logical source with a parsed raw missing from the index out of 1448 checked. The `parse_error` the bead quotes — `RuntimeError: membership replay cannot replace an unconvertible byte head` — is **not** the live cause. That wording no longer exists in the codebase (PR #3646 replaced it with the typed, retry-eligible `MembershipReplayConflictError`); it is a legacy string frozen on one raw row. Two measurements ruled the membership-replay hypothesis out entirely: - Running `classify_historical_full_revision_streams` over the 22 real full blobs resolves a **clean, unambiguous byte-prefix chain** — every revision `byte_proven`, head `dd0cbb34b9` (90,822,451 bytes, the current on-disk file). Nothing about this cohort is "unconvertible" today. - Reconstructing the cohort in a scratch archive from the real blobs and driving `backfill_historical_revision_evidence` materializes the session with 29,280 messages, even with the live archive's 21 stale `raw_session_memberships` rows (`decision IS NULL`, `revision_authority='quarantined'`) staged first. The actual blocker is upstream of replay, in the census/planning handshake: - `_persist_revision_census` routes every `source_index < 0` raw straight to the byte-authority membership receipt (`BYTE_AUTHORITY_CENSUS_DETAIL`) without parsing it — correct, appends are governed by byte revision authority. - `record_current_parser_source_census` then computed `complete=False` for that raw (no parsed sessions, no membership rows, not a typed non-session artifact) and wrote `status='failed'` carrying the byte-authority detail. - `uncensused_historical_revision_raw_ids` accepts only `complete` + `detail LIKE 'parser-observed:%'`, or `failed` at the current resource-blocked fingerprint. The fragment's receipt matches neither, so it is reported uncensused on every pass, forever. Reproduced against the real shape: with the component rebuilt from the live blobs, six consecutive whale-escalation passes each returned `success=False` with `Raw replay planning paused until the persisted parser census completes for 22 relevant raw(s)` and zero progress. The live source has 767 such fragments. This is the "converged or explicitly blocked" invariant failing in the silent direction: the pass claims unfinished census work for a raw the census can never say anything more about, instead of typing it as the authority debt it is. ## Solution `polylogue/storage/sqlite/archive_tiers/revision_governance.py` — `record_current_parser_source_census` now treats an append-position raw (`source_index < 0`) whose membership census is the byte-authority governance receipt as having an authoritative empty identity set, exactly like the existing `typed_non_session` branch it sits beside. The receipt becomes `complete` / `parser-observed: append fragment governed by byte revision authority`. The fragment's `revision_authority` is untouched — it stays `quarantined`, still counted as durable authority debt. Only the census receipt changes, from "we have not finished looking at this" to "we looked, and byte governance owns it". The `source_index < 0` guard keeps the branch off full snapshots even if one somehow carried the marker, matching how `storage/repair.py` already reads the same detail. Alternative rejected: special-casing the byte-authority detail inside `uncensused_historical_revision_raw_ids` instead. That leaves the durable receipt lying about its own status and makes every other consumer of `raw_authority_parser_census` disagree with the gate. Nothing in revision-authority refusal semantics changed: the byte-chain classifier, the polylogue-52l2 retired-sibling guard, the polylogue-eqnv source-path guard, and both `MembershipReplayConflictError` refusals are untouched. Two existing tests asserted the old receipt shape and were updated with the reason recorded inline — they were encoding the livelock, not protecting an invariant. `test_raw_materialization_reports_uncensused_append_fragments_as_pending_debt` now asserts the census reaches quiescence while the fragment is still reported as `append authority quarantine` debt; its three authority-bucket transitions (pending → quarantined → fragment) are unchanged and still pass. ## Verification Red first. `test_raw_materialization_converges_component_with_byte_governed_append_fragment` drives the real `repair_raw_materialization` entry point and failed on the parent commit at the census gate: ``` E AssertionError: assert ('80be922c7b2...62bf34ee473',) == () Left contains one more item: '80be922c7b289e66fa7776dad5efc1f6f4c7dae4e92e0521cff9a62bf34ee473' ``` Green after the fix, and the whole affected surface with it: ``` devtools test tests/unit/storage/test_repair.py tests/unit/storage/test_revision_replay.py → 140 passed in 19.02s devtools test tests/unit/sources/test_revision_backfill.py tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_archive_readiness.py → 167 passed in 27.49s devtools test tests/unit/cli/test_status.py tests/unit/cli/commands/test_status.py tests/unit/core/test_readiness_capability.py tests/unit/daemon/test_daemon_status.py tests/unit/daemon/test_parse_prefetch.py tests/unit/daemon/test_raw_materialization_parse_stage_equivalence.py tests/unit/maintenance/test_raw_authority_reset.py tests/unit/maintenance/test_inactive_candidate_durable_barrier.py tests/unit/pipeline/test_archive_ingest_shared_raw.py tests/unit/storage/test_incremental_rebuild_equivalence.py tests/unit/storage/test_durable_migrations.py → 400 passed in 41.93s devtools test tests/unit/scenarios/test_codex_804_live_proof.py -k red_mutation → 3 passed devtools verify --quick → exit=0, no "out of sync" ``` That set is every test file in the repo that references `raw_authority_parser_census` or `parser-observed`. Against the real 019f49d8 shape, in a scratch archive built from the live blobs (21 distinct full revisions + append fragments bound as the live rows are bound, live blob store read-only, live archive never written): - before: six whale passes, `success=False`, `index sessions: (0,)` - after: `attempt 0: success=True repaired=1`, `index sessions: (1,)`, `('codex-session:019f49d8-0185-7c43-8793-db6e57db13e1', 29280)` At the daemon's ordinary 64 MiB envelope the component is still explicitly resource-blocked (`plan_deferred_count 1.0`, "retry through bounded stream-safe whale pass"). That is the correct typed blocked state and is what routes it to the whale pass — the invariant now holds at both envelopes. **Gate substitution (deliberate, per lane policy).** The final plain `devtools verify` was skipped. Policy is to run `devtools why` first and skip the full run when it predicts a bootstrap; it did: ``` devtools why selection: bootstrap (absent) reason: native environment 'polylogue-e962f60d05f5eb...' is absent ``` Re-provisioning this lane's venv changed the testmon environment digest, so no graph exists for it and a plain verify becomes a full-corpus bootstrap (~45 min measured median) rather than a warm affected-selection run. The focused `devtools test` selections above plus `devtools verify --quick` are this PR's gate in its place. The coordinator records the merge receipt centrally at head. **Also not run:** `test_codex_804_live_proof.py::test_sanitized_codex_804_revision_recovery_proof` (900s timeout, `storage_scale`). It is provably unaffected — `tests/infra/whale_fixtures.py` never sets `source_index` or `revision_kind`, so every raw it acquires is `source_index=0`, and the new branch is gated on `source_index < 0`. Its three fast red-mutation guards do pass. ## Bead disposition matrix | Bead | Disposition | Evidence | | --- | --- | --- | | `polylogue-39kcs` | satisfied | Root cause identified and fixed (`67ab67c52`); red-first production-route regression test (`9ccfaa893`); the real 019f49d8 cohort converges to 29,280 messages where it previously livelocked. | `Ref polylogue-dyica` — AC#2 ("parser or lifecycle defects have production-route regression tests that fail when the original failure is reintroduced") is satisfied for this defect by the new test. The rest of dyica (the backup-gated live reclassification pass for the 110 benign rows, AC#1/#4) is untouched and stays open. <!-- polylogue-pr-scope:v2 { "assigned_beads": [ "polylogue-39kcs" ], "dispositions": [ { "bead_id": "polylogue-39kcs", "disposition": "satisfied", "evidence": [ { "kind": "test", "ref": "tests/unit/storage/test_repair.py::test_raw_materialization_converges_component_with_byte_governed_append_fragment" }, { "kind": "commit", "ref": "9ccfaa893" }, { "kind": "commit", "ref": "67ab67c52" }, { "kind": "command", "ref": "devtools test tests/unit/storage/test_repair.py tests/unit/storage/test_revision_replay.py" } ], "successors": [] } ], "mutated_beads": [], "scope_digest": "778a5991c2362d6487d63c578af293dbd1722d2bb82e5fa4fcb5ce425cedac4d", "scope_kind": "bead", "version": 2 } -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apply_raw_membership_classification's two head-conflict guards (storage/sqlite/archive_tiers/revision_governance.py) now raise a dedicatedMembershipReplayConflictError(RuntimeError)instead of a plainRuntimeError, andstorage/repair.py's raw-materialization retry-candidate logic (both the SQL query and the_raw_materialization_retryable_missing_blob_errorPython gate) recognizes that type by its stableparse_errorprefix as retry-eligible.Problem
A production Codex session (native_id
019f49d8-0185-7c43-8793-db6e57db13e1, 804raw_sessionsrows from incremental full-snapshot capture of a growingrollout.jsonl) has zero sessions inindex.db. Its largest revision (90,822,451 bytes) has never been parsed; the next-largest (90,156,590 bytes, confirmed byte-for-byte prefix of the largest) carriesparse_error = 'RuntimeError: membership replay cannot replace an unconvertible byte head'from PR #2718 (2026-07-12).That exact message string no longer exists anywhere in the codebase — PR #3211 and the later polylogue-miwv fix narrowed and reworded both of
apply_raw_membership_classification's head-conflict guards. Reading the live archive read-only (/realm/db/polylogue,mode=ro) confirmed three things:codex.parse_stream()on the raw bytes succeeds, 29,280 messages, no crash.raw_revision_headshas zero rows for thislogical_source_keytoday, so the guards'existing_head is not Nonebranches cannot currently fire for this session on a fresh attempt.storage/repair.py's raw-materialization candidate query excludes anyparse_errorother than two hardcoded exact-match transient-error strings ('OperationalError: database is locked','decode:...No such file or directory') as permanently terminal.So the bug is not a live crash loop — it is a stale, retry-blocking
parse_errorfrozen under message wording that predates two subsequent refactors, with no mechanism that ever re-evaluates or clears it.repair.py's retry-candidate query permanently excludes this raw from every future rebuild, independent of whether the underlying governance conditions that produced the error still apply.Solution
MembershipReplayConflictError(RuntimeError)inrevision_governance.py, following the existingActiveByteRevisionChainErrorprecedent, and changed both head-conflictraisesites to use it.archive.py's imports and__all__.mark_raw_parse_failedrecordsparse_errorasf"{type(exc).__name__}: {exc}", so this givesrepair.py's retry logic a message-wording-independent marker instead of an allowlist of brittle exact-message strings._raw_materialization_retryable_missing_blob_error(the real single source of truth — the SQL clause is a pre-filter, not merely an optimization; a row that clears the SQLWHEREis independently re-checked in Python) to recognize a'MembershipReplayConflictError:'parse_errorprefix as retry-eligible.Alternatives rejected: matching on the existing message text more broadly (e.g. a regex over "cannot retire"/"cannot replace") was rejected because it re-creates the exact fragility that caused this bug — any future rewording of the guard's message silently breaks the match again. A dedicated exception type is the only marker guaranteed stable across message edits.
Scope note: this fixes the retry-eligibility bug (the reason this raw is permanently excluded from every rebuild). It does not itself guarantee the retried classification succeeds for every one of the 804 rows — the live evidence (no
raw_revision_headsrow, parser succeeds on the real bytes) strongly suggests the largest two revisions will now classify and index successfully on the nextpolylogue ops maintenance rebuild-indexpass, but I did not run that pass against the live archive (read-only access only, per task constraints) and cannot claim the full 804-row churn compacts without that live evidence.Verification
devtools test tests/unit/sources/test_live_batch_support.py -k test_bundle_replay_respects_unconvertible_single_session_head— red before the fix (2 failed:parse_errorwas plain"RuntimeError: ..."), green after (4 passed).devtools test tests/unit/storage/test_repair.py -k test_raw_materialization_retries_membership_replay_conflict_failure— new test; confirms aMembershipReplayConflictError-prefixedparse_erroris retry-eligible while a sibling row carrying the OLD plain-RuntimeErrortext (anti-vacuity control) remains excluded.devtools test tests/unit/storage/test_repair.py tests/unit/sources/test_live_batch_support.py— 142 passed, 3 failed; confirmed by reverting this fix and rerunning the same 3 tests that they fail identically on unmodified code (test_append_multi_session_payload_is_rejected_before_index_write,test_full_ingest_skips_durably_excised_content_without_aborting_batch,test_full_ingest_writes_archive_with_route_observability— all fail with an unrelated"did not replay to exactly one session"RuntimeErrorfrom_parse_raw_revision_chain, pre-existing and untouched by this change).devtools verify --quick—20260803T113615Z-quick-2698640-140a86d3, 22/22 steps passed, exit 0./realm/db/polylogue,mode=ro): confirmed the actualcodex.parse_stream()call succeeds on the real 90,822,451-byte blob (29,280 messages), and confirmedraw_revision_heads/raw_revision_applicationshave zero rows for thislogical_source_keytoday.Ref polylogue-5iz4