fix: converge raw components holding byte-governed append fragments

[Sinity]

Summary

A Codex rollout that grows in place accumulates both full snapshots and
append fragments under one logical source. Append fragments are deliberately
never parsed for identity, and the parser-census receipt written for them was
failed — a status that satisfies neither branch of the census-complete gate.
Every raw-materialization pass therefore re-selected the fragment, rewrote the
same receipt, and reported "paused until the persisted parser census completes"
forever, which stopped raw-replay planning for every full snapshot sharing
the authority component. This receipts a byte-governed fragment as the complete
observation it is, so those components converge.

Problem

polylogue-39kcs (bucket B7 of the dyica classification — the only genuine gap
in the 111-failure population): codex session
019f49d8-0185-7c43-8793-db6e57db13e1 is acquired, parsed, and censused but
absent from index.db. It is the only codex logical source with a parsed raw
missing from the index out of 1448 checked.

The parse_error the bead quotes — RuntimeError: membership replay cannot replace an unconvertible byte head — is not the live cause. That wording no
longer exists in the codebase (PR #3646 replaced it with the typed, retry-eligible
MembershipReplayConflictError); it is a legacy string frozen on one raw row.
Two measurements ruled the membership-replay hypothesis out entirely:

• Running classify_historical_full_revision_streams over the 22 real full
  blobs resolves a clean, unambiguous byte-prefix chain — every revision
  byte_proven, head dd0cbb34b9 (90,822,451 bytes, the current on-disk file).
  Nothing about this cohort is "unconvertible" today.
• Reconstructing the cohort in a scratch archive from the real blobs and driving
  backfill_historical_revision_evidence materializes the session with 29,280
  messages, even with the live archive's 21 stale raw_session_memberships
  rows (decision IS NULL, revision_authority='quarantined') staged first.

The actual blocker is upstream of replay, in the census/planning handshake:

• _persist_revision_census routes every source_index < 0 raw straight to the
  byte-authority membership receipt (BYTE_AUTHORITY_CENSUS_DETAIL) without
  parsing it — correct, appends are governed by byte revision authority.
• record_current_parser_source_census then computed complete=False for that
  raw (no parsed sessions, no membership rows, not a typed non-session artifact)
  and wrote status='failed' carrying the byte-authority detail.
• uncensused_historical_revision_raw_ids accepts only complete +
  detail LIKE 'parser-observed:%', or failed at the current
  resource-blocked fingerprint. The fragment's receipt matches neither, so it is
  reported uncensused on every pass, forever.

Reproduced against the real shape: with the component rebuilt from the live
blobs, six consecutive whale-escalation passes each returned success=False
with Raw replay planning paused until the persisted parser census completes for 22 relevant raw(s) and zero progress. The live source has 767 such fragments.

This is the "converged or explicitly blocked" invariant failing in the silent
direction: the pass claims unfinished census work for a raw the census can never
say anything more about, instead of typing it as the authority debt it is.

Solution

polylogue/storage/sqlite/archive_tiers/revision_governance.py —
record_current_parser_source_census now treats an append-position raw
(source_index < 0) whose membership census is the byte-authority governance
receipt as having an authoritative empty identity set, exactly like the existing
typed_non_session branch it sits beside. The receipt becomes
complete / parser-observed: append fragment governed by byte revision authority.

The fragment's revision_authority is untouched — it stays quarantined, still
counted as durable authority debt. Only the census receipt changes, from "we
have not finished looking at this" to "we looked, and byte governance owns it".
The source_index < 0 guard keeps the branch off full snapshots even if one
somehow carried the marker, matching how storage/repair.py already reads the
same detail.

Alternative rejected: special-casing the byte-authority detail inside
uncensused_historical_revision_raw_ids instead. That leaves the durable
receipt lying about its own status and makes every other consumer of
raw_authority_parser_census disagree with the gate.

Nothing in revision-authority refusal semantics changed: the byte-chain
classifier, the polylogue-52l2 retired-sibling guard, the polylogue-eqnv
source-path guard, and both MembershipReplayConflictError refusals are
untouched.

Two existing tests asserted the old receipt shape and were updated with the
reason recorded inline — they were encoding the livelock, not protecting an
invariant. test_raw_materialization_reports_uncensused_append_fragments_as_pending_debt
now asserts the census reaches quiescence while the fragment is still reported
as append authority quarantine debt; its three authority-bucket transitions
(pending → quarantined → fragment) are unchanged and still pass.

Verification

Red first. test_raw_materialization_converges_component_with_byte_governed_append_fragment
drives the real repair_raw_materialization entry point and failed on the
parent commit at the census gate:

E   AssertionError: assert ('80be922c7b2...62bf34ee473',) == ()
      Left contains one more item: '80be922c7b289e66fa7776dad5efc1f6f4c7dae4e92e0521cff9a62bf34ee473'

Green after the fix, and the whole affected surface with it:

devtools test tests/unit/storage/test_repair.py tests/unit/storage/test_revision_replay.py
  → 140 passed in 19.02s
devtools test tests/unit/sources/test_revision_backfill.py tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_archive_readiness.py
  → 167 passed in 27.49s
devtools test tests/unit/cli/test_status.py tests/unit/cli/commands/test_status.py tests/unit/core/test_readiness_capability.py tests/unit/daemon/test_daemon_status.py tests/unit/daemon/test_parse_prefetch.py tests/unit/daemon/test_raw_materialization_parse_stage_equivalence.py tests/unit/maintenance/test_raw_authority_reset.py tests/unit/maintenance/test_inactive_candidate_durable_barrier.py tests/unit/pipeline/test_archive_ingest_shared_raw.py tests/unit/storage/test_incremental_rebuild_equivalence.py tests/unit/storage/test_durable_migrations.py
  → 400 passed in 41.93s
devtools test tests/unit/scenarios/test_codex_804_live_proof.py -k red_mutation
  → 3 passed
devtools verify --quick
  → exit=0, no "out of sync"

That set is every test file in the repo that references
raw_authority_parser_census or parser-observed.

Against the real 019f49d8 shape, in a scratch archive built from the live blobs
(21 distinct full revisions + append fragments bound as the live rows are bound,
live blob store read-only, live archive never written):

• before: six whale passes, success=False, index sessions: (0,)
• after: attempt 0: success=True repaired=1, index sessions: (1,),
  ('codex-session:019f49d8-0185-7c43-8793-db6e57db13e1', 29280)

At the daemon's ordinary 64 MiB envelope the component is still explicitly
resource-blocked (plan_deferred_count 1.0, "retry through bounded stream-safe
whale pass"). That is the correct typed blocked state and is what routes it to
the whale pass — the invariant now holds at both envelopes.

Gate substitution (deliberate, per lane policy). The final plain
devtools verify was skipped. Policy is to run devtools why first and skip
the full run when it predicts a bootstrap; it did:

devtools why
  selection: bootstrap (absent)
  reason: native environment 'polylogue-e962f60d05f5eb...' is absent

Re-provisioning this lane's venv changed the testmon environment digest, so no
graph exists for it and a plain verify becomes a full-corpus bootstrap (~45 min
measured median) rather than a warm affected-selection run. The focused
devtools test selections above plus devtools verify --quick are this PR's
gate in its place. The coordinator records the merge receipt centrally at head.

Also not run:
test_codex_804_live_proof.py::test_sanitized_codex_804_revision_recovery_proof
(900s timeout, storage_scale). It is provably unaffected —
tests/infra/whale_fixtures.py never sets source_index or revision_kind, so
every raw it acquires is source_index=0, and the new branch is gated on
source_index < 0. Its three fast red-mutation guards do pass.

Bead disposition matrix

Bead	Disposition	Evidence
polylogue-39kcs	satisfied	Root cause identified and fixed (67ab67c52); red-first production-route regression test (9ccfaa893); the real 019f49d8 cohort converges to 29,280 messages where it previously livelocked.

Ref polylogue-dyica — AC#2 ("parser or lifecycle defects have production-route
regression tests that fail when the original failure is reintroduced") is
satisfied for this defect by the new test. The rest of dyica (the backup-gated
live reclassification pass for the 110 benign rows, AC#1/#4) is untouched and
stays open.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a4639dc1-9425-44ed-8a5d-fe77d1157ec3

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}