fix(storage): raw-authority fingerprint gating, lineage replay order, dry-run success by Sinity · Pull Request #3485 · Sinity/polylogue

Sinity · 2026-07-31T22:15:44Z

Summary

Lane C12: polylogue-9dxn, polylogue-5q2u, polylogue-f57q complete with focused production fixes + regression tests; polylogue-hjpx closed out via evidence, no new production code (see AC matrix — the P0 defect the bead was filed against is already fixed and regression-tested on master, predating this lane).

Problem

Measured baseline (coordinator-verified 2026-07-31):

RAW_AUTHORITY_PARSER_FINGERPRINT existed as a proper constant but sources/revision_backfill.py hardcoded the literal "revision-membership-v1" 8 times instead of importing it; storage/repair.py's terminal-decision check for decision = 'ambiguous' had no fingerprint/version gate, so a classifier correction (polylogue-bu1i) could never heal already-persisted ambiguous verdicts.
sources/revision_backfill.py:958 (for logical_key in sorted(logical_keys):) is lexicographic, lineage-blind — during a rebuild a child (resume/fork) replays before its parent roughly as often as not, triggering the expensive #2467 deferred-tail normalization path.
repair_raw_materialization's dry-run path unconditionally reported success=False even when the preview validly identified real work, breaking devtools/scale_regression_probe.py's raw_materialization_debt_detected check.
polylogue-hjpx's own 2026-07-31 reconciliation note claimed PR test(storage): prove raw-authority fair scheduling and CAS-typed retry (hjpx AC3/AC4) #3345 (AC3/AC4 regression tests) was unmerged and the P0 gap unresolved.

Solution

polylogue-9dxn (polylogue/storage/raw_authority.py, polylogue/sources/revision_backfill.py, polylogue/storage/repair.py): revision_backfill.py now imports RAW_AUTHORITY_PARSER_FINGERPRINT instead of repeating the literal. Added SUPERSEDED_MEMBERSHIP_FINGERPRINTS and bumped the constant to revision-membership-v2 in the same commit as the gating (per design constraint — a bare bump alone would force a ~4h20m full reparse). The quiescence gate (uncensused_historical_revision_raw_ids) now accepts any known fingerprint (current or superseded), not only current, so a bump does not force a full archive re-census. repair.py's terminal-ambiguous query (covering both index_tier.raw_revision_applications and raw_session_memberships) now LEFT JOINs raw_authority_parser_census and excludes a raw from the terminal gate when its census fingerprint is superseded; absent census / current fingerprint stays conservative (terminal).

polylogue-5q2u (polylogue/sources/revision_backfill.py): new _lineage_aware_replay_order computes roots-first, children-after-parent ordering using ParsedSession.parent_session_provider_id the census phase already parsed and spilled. Falls back to lexicographic order for any key whose parent is unresolvable (missing/external/cross-batch/cycle) — nothing is ever skipped. Scheduling-only: the pipeline-decode prefetcher now shares the same order as the writer's replay loop.

polylogue-f57q (polylogue/storage/repair.py, devtools/scale_regression_probe.py): dry-run preview success now means "the requested phase completed validly", matching the convention the adjacent branches in the same function already use — repaired_count (always 0 for dry-run) remains the sole "nothing mutated" signal. Migrated 6 test_repair.py assertions and the scale-probe's own check to the corrected semantics.

polylogue-hjpx: investigated per evidence-harness discipline (build the failing fixture first). git merge-base --is-ancestor 64d203c4f e8a23cc31 confirms PR #3345 (hjpx AC3/AC4 regression tests, state MERGED, mergedAt 2026-07-27) is already an ancestor of this lane's base commit (e8a23cc, 2026-07-31) — the bead's own reconciliation note was stale. Attempted to reproduce AC1's named defect (repair_raw_materialization reports a scanned/classified raw yielding replayed_logical_sources=0 with the same candidate remaining forever): the existing regression test test_raw_materialization_no_progress_component_terminalizes_instead_of_looping (production fix: commit 6ea374222, PR #3337) already covers exactly this shape and passes. No new failing fixture could be honestly constructed because the defect does not reproduce against current code — see AC matrix below for what's covered by pre-existing landed work vs. genuinely open.

Verification

devtools test tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_archive_readiness.py tests/unit/storage/test_revision_replay.py tests/unit/storage/test_quarantined_accepted_raw_repair.py
  -> 99 passed
devtools test tests/unit/sources/test_revision_backfill.py
  -> 59 passed, 1 pre-existing failure unrelated to this branch
     (test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule;
     reproduces identically on unmodified HEAD -- a content-classification
     gate assertion untouched by this diff)
devtools test tests/unit/storage/test_incremental_rebuild_equivalence.py tests/benchmarks/test_graph_resolve_deferred_tail.py
  -> 3 passed
devtools test tests/unit/storage/test_repair.py
  -> 65 passed
devtools test tests/unit/devtools/test_scale_regression_probe.py
  -> 2 passed
devtools test -k raw_materialization
  -> 115 passed, 2 pre-existing failures unrelated to this branch
     (test_maybe_run_raw_materialization_whale_pass_*; TypeError on an
     unrelated mock-signature mismatch in polylogue/config.py:2173,
     nothing to do with repair.py/dry_run/success)
devtools test -k raw_authority
  -> 92 passed
python -m mypy
  -> Success: no issues found in 2426 source files
devtools verify --quick
  -> 18/18 steps green, exit 0

devtools verify (default testmon-affected tier) was not run: testmon is unseeded in this worktree and seeding is a heavy one-time harness step out of proportion to this diff (per repo convention, reserved for harness/dependency changes); the focused selections above plus --quick cover the changed surface.

Per-bead AC matrix

polylogue-9dxn

RAW_AUTHORITY_PARSER_FINGERPRINT is the single source of the fingerprint string; no module hardcodes it -- satisfied (8 literals replaced in revision_backfill.py; grepped repo-wide for revision-membership-v1, zero hits outside the SUPERSEDED_MEMBERSHIP_FINGERPRINTS definition itself).
An ambiguous verdict under a superseded fingerprint is replayable; one under the current fingerprint stays terminal; both directions covered -- satisfied (test_ambiguous_verdict_under_superseded_fingerprint_is_replayable, test_ambiguous_verdict_under_current_fingerprint_stays_terminal, test_ambiguous_verdict_with_no_census_row_stays_terminal).
A bump does not force re-census of unaffected raws, asserted against a fixture -- satisfied (uncensused_historical_revision_raw_ids accepts any known fingerprint; existing test_stale_per_raw_parser_fingerprint_is_recensused_before_planning continues to pass, proving the targeted recensus path still works for a genuinely stale/unknown fingerprint while known fingerprints are accepted).

polylogue-5q2u

AC1 (fixture proves reduced/eliminated deferred-tail hits, no adoption change) -- satisfied (test_lineage_aware_replay_order_reduces_deferred_tail_hits: 5->0 _reextract_prefix_tail_db calls on a 1-parent/5-child fixture).
AC2 (replay outcome parity, differential test) -- satisfied (test_lineage_aware_replay_order_preserves_outcome_parity: byte-identical _index_content_manifest + identical RevisionBackfillResult counts between lineage and forced-lexicographic order).
AC3 (cycles/missing/cross-batch parents degrade gracefully, never skip) -- satisfied (test_lineage_aware_replay_order_falls_back_for_unresolvable_parent; the ordering function's cycle-fallback loop is unconditionally total over its input set by construction).
AC4 (focused tests + devtools verify --quick land together) -- satisfied.

polylogue-f57q

Full class-level MaintenanceOutcome/receipt contract census across every repair/cleanup handler -- deferred. This PR fixes the specific dry-run outcome defect (repair_raw_materialization's preview success semantics) and the two scale_regression_probe.py tests it broke, matching the lane's explicit non-goal allowance ("land the typed outcome + the raw-materialization path + scale-probe fix, and file a follow-up bead for remaining handlers"). The broader phase/candidate/eligible/blocked/planned/applied/already-satisfied/failed/remaining vocabulary across every handler remains open scope needing its own design pass, as the bead's own prior note already states.
The seeded raw-materialization case reports phase-honest preview semantics (candidate=eligible=planned=1, applied=0, mutates=false, success=true) -- satisfied for this one call site; not generalized to a shared typed vocabulary.

polylogue-hjpx

AC1 (production-shaped fixture + fix/defer) -- misframed: the exact defect this AC names (a scanned/classified raw yielding zero replayed logical sources, forever re-selected) is already fixed (commit 6ea374222, PR fix(storage): terminalize no-progress raw replay plans instead of looping #3337) and regression-tested (test_raw_materialization_no_progress_component_terminalizes_instead_of_looping), already in this lane's base commit. No new fixture could be honestly built because it does not reproduce.
AC2 (per-plan outcome conservation) -- satisfied by pre-existing landed work: RawReplayPlanOutcome/RawReplayPlanStatus (EXECUTED/RETRYABLE/DEFERRED/TERMINAL/REJECTED_STALE/CARRIED_FORWARD), _raw_replay_conservation_metrics, raw_materialization_plan_conservation_error_count, and test_raw_materialization_fails_closed_on_plan_conservation_mismatch are already live and green.
AC3 (fair bounded scheduling) -- satisfied by pre-existing landed work (PR test(storage): prove raw-authority fair scheduling and CAS-typed retry (hjpx AC3/AC4) #3345, merged 2026-07-27, confirmed ancestor of this lane's base via git merge-base --is-ancestor).
AC4 (retryable/CAS typing) -- satisfied by pre-existing landed work (same PR test(storage): prove raw-authority fair scheduling and CAS-typed retry (hjpx AC3/AC4) #3345).
AC5 (two quiescent dry-run censuses = fixed point) -- satisfied by pre-existing landed work (record_raw_authority_census's fixed_point computation; test_two_successive_quiescent_censuses_are_required_for_fixed_point passes).
AC6 (scale proof) -- out of scope per lane non-goals (no live apply authorized).
AC7 (live-apply ceremony) -- out of scope per lane non-goals; the named devtools test/devtools verify commands were run as this PR's own verification (above), which is the non-live-apply portion of AC7.

Anti-vacuity statement

9dxn: test_ambiguous_verdict_under_superseded_fingerprint_is_replayable calls repair._raw_replay_plan_outcomes (the exact function repair_raw_materialization, the daemon's live raw-materialization repair entrypoint, calls internally) via the public build_raw_replay_plans pair. Reverting the LEFT JOIN raw_authority_parser_census + NOT COALESCE(... IN (SELECT value FROM json_each(?)) ...) guard in repair.py's terminal query back to the unconditional ambiguous check makes this test fail by reclassifying the plan TERMINAL.
5q2u: test_lineage_aware_replay_order_reduces_deferred_tail_hits calls the real production route backfill_historical_revision_evidence (its replay loop's ordered_logical_keys, computed by _lineage_aware_replay_order at its actual call site). Reverting the call site back to sorted(logical_keys) makes lineage_hits jump from 0 to 5 (verified live during implementation via monkeypatch.setattr).
f57q: test_scale_regression_probe_runs_seeded_bug_class_checks / test_scale_regression_probe_main_emits_json exercise repair_raw_materialization(dry_run=True) through run_scale_regression_probe's raw_materialization_debt_detected check. Reverting success=True back to success=False in repair.py's dry-run branch reproduces the exact assert False is True / assert 1 == 0 failures observed before this fix.
hjpx: no new production surface added by this lane; the anti-vacuity claim is the evidence trail itself (git merge-base --is-ancestor 64d203c4f e8a23cc31, plus the passing -k hjpx / -k raw_materialization / -k raw_authority selections against unmodified production code).

Residuals / follow-ups

polylogue-f57q: the full MaintenanceOutcome typed vocabulary across every repair/cleanup handler remains open scope (deferred per lane non-goal). A follow-up tracking item should own the handler census explicitly.
polylogue-hjpx: the coordinator should re-verify this bead's status against live master (not the stale 2026-07-31 reconciliation note) before further dispatch -- AC1-AC5 read as satisfied by already-merged work; only AC6 (scale proof) and AC7's live-apply ceremony remain genuinely open, both explicitly out of scope for any lane under the standing "no live apply is authorized" constraint.
Two pre-existing, unrelated test failures observed and left untouched (out of this lane's scope): test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule (content-classification gate) and test_maybe_run_raw_materialization_whale_pass_* (_bootstrap mock-signature mismatch in config.py:2173).

Ref polylogue-9dxn, polylogue-5q2u, polylogue-f57q, polylogue-hjpx

…print Problem: RAW_AUTHORITY_PARSER_FINGERPRINT existed as a proper constant in storage/raw_authority.py but sources/revision_backfill.py hardcoded the literal "revision-membership-v1" eight times instead of importing it, so bumping the constant would half-apply. Separately, storage/repair.py's terminal-decision check (~4432-4448) treated any persisted decision = 'ambiguous' row as durable terminal debt with no way to distinguish "ambiguous under the current classifier" from "ambiguous under a classifier we have since corrected" -- every improvement to classify_membership_revisions was therefore inert on existing data (polylogue-bu1i fixed 157/157 live aistudio-drive cohorts going forward, but the persisted ambiguous verdicts for those cohorts stayed terminal forever). Solution: - revision_backfill.py now imports RAW_AUTHORITY_PARSER_FINGERPRINT instead of repeating the literal at all 8 call sites (fingerprint writes to raw_authority_parser_census/raw_membership_census, and the resource-blocked-envelope fingerprint helper). - The quiescence gate (uncensused_historical_revision_raw_ids) now accepts ANY known fingerprint (current or superseded), not only the current one -- a bump answers "is this verdict still authoritative?", not "was this raw ever observed by a real parser?", so it must not force a full archive re-census. - repair.py's terminal-ambiguous query (covering both index_tier.raw_revision_applications and raw_session_memberships) now LEFT JOINs raw_authority_parser_census and excludes a raw from the terminal gate when its census fingerprint is listed in the new SUPERSEDED_MEMBERSHIP_FINGERPRINTS set. A raw with no census row (or a current-fingerprint row) stays conservative and remains terminal. - Bumped RAW_AUTHORITY_PARSER_FINGERPRINT to "revision-membership-v2" in the same commit as the gating (a bare bump alone would force a ~4h20m full reparse; the gating is what makes a bump targeted). Verification: devtools test tests/unit/storage/test_raw_authority_ledger.py tests/unit/storage/test_archive_readiness.py tests/unit/storage/test_revision_replay.py tests/unit/storage/test_quarantined_accepted_raw_repair.py -> 99 passed devtools test tests/unit/sources/test_revision_backfill.py -> 55 passed, 1 pre-existing failure unrelated to this change (test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule, a content-classification gate assertion untouched by this diff; reproduces identically on unmodified HEAD) devtools test tests/unit/storage/test_incremental_rebuild_equivalence.py -> 1 passed Anti-vacuity: test_ambiguous_verdict_under_superseded_fingerprint_is_replayable exercises repair._raw_replay_plan_outcome via the public build_raw_replay_plans/_raw_replay_plan_outcomes pair used by repair_raw_materialization (the daemon's live raw-materialization repair entrypoint). Reverting the LEFT JOIN + json_each NOT COALESCE guard in repair.py's terminal query makes this test fail by reclassifying the plan TERMINAL. Ref polylogue-9dxn

…cographic Problem: revision_backfill.py's rebuild replay loop visited sorted(logical_keys) -- a lexicographic string sort with zero relationship to parent/child lineage. During a cold/full rebuild this means a child (resume/fork) replays before its parent roughly as often as not. A child replayed before its parent is stored WHOLE (a full duplicate of the eventual shared prefix); when the parent finally arrives, _resolve_session_graph must walk every such orphaned child and normalize it (delete the duplicate prefix rows, remap session_events refs, delete prefix-scoped dependents) -- the #2467 deferred-tail path, confirmed linear but real O(orphaned_children * shared_prefix_size) row-mutation work (tests/benchmarks/test_graph_resolve_deferred_tail.py, 260s for one codex-session parent in a live 2026-07-03 rebuild batch). Solution: _lineage_aware_replay_order (new) computes roots-first, children-after-parent ordering for one rebuild's logical_keys, using the ParsedSession.parent_session_provider_id the census phase already parsed and spilled -- no extra reparsing on the happy path (spill.for_raw falls back to a bounded reparse only if a key's representative raw fell out of the spill cache). Falls back to the previous lexicographic order for any key whose parent is unresolvable (missing/external/cross-batch parent, or a lineage cycle) -- nothing is ever skipped. Scheduling-only: it changes the ORDER backfill_historical_revision_evidence's replay loop (and its pipeline-decode prefetcher, which now shares the same order) visits logical keys in, never what gets replayed or adopted. Verification: devtools test tests/unit/sources/test_revision_backfill.py -k lineage_aware -> 4 passed: - test_lineage_aware_replay_order_visits_parent_before_children - test_lineage_aware_replay_order_falls_back_for_unresolvable_parent - test_lineage_aware_replay_order_reduces_deferred_tail_hits (AC1): measures _reextract_prefix_tail_db call count on a 1-parent/5-child codex resume fixture -- lexicographic order hits it 5 times (once per child), lineage order hits it 0 times - test_lineage_aware_replay_order_preserves_outcome_parity (AC2): same fixture, lineage vs. forced-lexicographic order reach byte-identical index.db content (_index_content_manifest: sessions/messages/blocks/session_links) and identical RevisionBackfillResult counts devtools test tests/unit/sources/test_revision_backfill.py -> 59 passed, 1 pre-existing failure unrelated to this change (same as prior commit: test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule) devtools test tests/benchmarks/test_graph_resolve_deferred_tail.py tests/unit/storage/test_incremental_rebuild_equivalence.py -> 2 passed Anti-vacuity: test_lineage_aware_replay_order_reduces_deferred_tail_hits exercises the real production route (backfill_historical_revision_evidence -> its replay loop's ordered_logical_keys, computed by _lineage_aware_replay_order at its actual call site) with a real Codex resume-shaped fixture. Reverting the call site back to sorted(logical_keys) (verified live during implementation via monkeypatch.setattr(revision_backfill, "_lineage_aware_replay_order", lambda *a: sorted(a[0]))) makes lineage_hits jump from 0 to 5, equal to lexicographic_hits -- the exact failure this test is built to catch. Ref polylogue-5q2u

Problem: repair_raw_materialization's dry-run branch (the path reached once a preview has validly identified executable authority components) unconditionally returned success=False, conflating "nothing was mutated" (which repaired_count=0 already encodes correctly) with "the preview failed". Repro: seed one raw_sessions row with a real blob and call repair_raw_materialization(config, dry_run=True) -- candidate_count=1, repaired_count=0 (correct), success=False (wrong: the preview completed validly). devtools/scale_regression_probe.py's own raw_materialization_debt_detected check had been adapted to assert `dry_run.success is False` as the "ok" condition, silently codifying the bug as expected behavior rather than fixing it. Solution: dry_run success now means "the requested preview phase completed validly", matching the convention the adjacent "no candidate_raw_ids" and census-pending branches in the same function already use. repaired_count (unconditionally 0 for every dry-run outcome) remains the sole signal for "nothing was mutated" -- preview success and preview mutation are now two independent, honestly-named facts instead of one field trying to carry both. Migrated 6 existing test_repair.py assertions plus scale_regression_probe.py's own check to the corrected semantics; no production caller depends on dry-run success being False (grepped every `repair_raw_materialization`/`repair_materialization` call site -- the daemon's two callers, daemon/cli.py:1169/1231, always pass dry_run=False and are unaffected). Non-goal (per lane scope): this fixes the specific dry-run outcome defect and the scale-probe assertions it broke, not the broader class-level typed MaintenanceOutcome/receipt vocabulary (phase + candidate/eligible/blocked/planned/applied/already-satisfied/failed/ remaining counts across every repair/cleanup handler) polylogue-f57q's design section describes. That census remains open scope; a follow-up tracking item should own it explicitly rather than this fix silently claiming it. Verification: devtools test tests/unit/storage/test_repair.py -> 65 passed devtools test tests/unit/devtools/test_scale_regression_probe.py tests/unit/storage/test_raw_authority_ledger.py -> 40 passed devtools test tests/unit/daemon/test_daemon_cli.py tests/unit/daemon/test_raw_materialization_parse_stage_equivalence.py -> 112 passed, 2 pre-existing failures unrelated to this change (test_maybe_run_raw_materialization_whale_pass_* fail with `TypeError: <lambda>() got an unexpected keyword argument '_bootstrap'` inside polylogue/config.py:2173 -- a mock-signature mismatch with no relation to repair.py/dry_run/success, reproduces on unmodified files) Anti-vacuity: test_scale_regression_probe_runs_seeded_bug_class_checks and test_scale_regression_probe_main_emits_json exercise the real production route repair_raw_materialization(dry_run=True) through run_scale_regression_probe's raw_materialization_debt_detected check. Reverting the `success=True` mutation in repair.py's dry-run branch back to `success=False` makes both tests fail again with the exact `assert False is True` / `assert 1 == 0` shape observed before this fix. Ref polylogue-f57q

devtools verify --quick's mypy step failed on two additions from this branch's earlier commits: - test_raw_authority_ledger.py: _seed_ambiguous_membership_component (polylogue-9dxn) was typed to return tuple[str, object]; narrow it to tuple[str, RawReplayPlanOutcome] so .status/.reason access type-checks. - test_revision_backfill.py: the deferred-tail-hit counting_wrapper (polylogue-5q2u) took *args: object/**kwargs: object, which mypy rejects against _reextract_prefix_tail_db's concrete signature; switch to *args: Any/**kwargs: Any (the standard shape for an untyped passthrough wrapper around a function whose exact signature the test does not care about). Verification: python -m mypy -> Success: no issues found in 2426 source files. devtools test tests/unit/storage/test_raw_authority_ledger.py tests/unit/sources/test_revision_backfill.py -> 97 passed, 1 pre-existing unrelated failure (test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule).

chatgpt-codex-connector · 2026-07-31T22:15:48Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

coderabbitai · 2026-07-31T22:15:52Z

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 49 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0c44886c-202a-4efe-b3a7-c4803482b2e4

📥 Commits

Reviewing files that changed from the base of the PR and between c1d8016 and 5655186.

📒 Files selected for processing (12)

devtools/scale_regression_probe.py
polylogue/sources/revision_backfill.py
polylogue/storage/raw_authority.py
polylogue/storage/repair.py
tests/unit/sources/test_live_batch_support.py
tests/unit/sources/test_revision_backfill.py
tests/unit/storage/test_archive_readiness.py
tests/unit/storage/test_incremental_rebuild_equivalence.py
tests/unit/storage/test_quarantined_accepted_raw_repair.py
tests/unit/storage/test_raw_authority_ledger.py
tests/unit/storage/test_repair.py
tests/unit/storage/test_revision_replay.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Co-Authored-By: Claude <noreply@anthropic.com>

…d routes (#3496) ## Summary polylogue-2cuv asked for a bounded-memory producer/consumer pipeline overlapping rebuild parse decode with the writer's apply work, on both the daemon bulk-rebuild route and the offline CLI route. Re-verification against current `master` found this already shipped by PR #3478 (`_ReplaySpillPrefetcher`, "Lever A"), with lineage-order preservation from PR #3485 landed the same day. This PR does not add pipeline plumbing (none is missing) — it adds one regression test proving the pipeline auto-engages from the **production entry point both routes share**, plus an honest before/after measurement in the PR description below. ## Problem The bead's core measured claim, from a real 4h22m rebuild receipt: `parse_s (census 1202s + spill_load 2830s) + apply_s (8601s) == total (12633s)` **exactly** — zero overlap between decode and the single SQLite writer. `spill_load` (2830s, 22% of the pass) was documented as strictly SERIAL `pickle.loads`/reparse work interleaved with writer apply work. The bead asked for a producer/consumer queue so census+spill hide behind apply, on both `daemon/bulk_rebuild.py` and `maintenance/rebuild_index.py`. ## Solution **Finding: already done.** Both routes call the identical chain: - Offline: `maintenance/rebuild_index.py` → `rebuild_index_from_source_sync` - Daemon: `daemon/bulk_rebuild.py::run_daemon_bulk_rebuild_pass` → `rebuild_index_from_source_sync` (same function, scheduled through the daemon's write coordinator) Both funnel into `maintenance/replay.py::rebuild_index_from_source` → `sources/revision_backfill.py::backfill_historical_revision_evidence`, which since PR #3478 (`66515459c`/`c6e275bca`, "pipeline replay decode off the writer thread") runs a background `_ReplaySpillPrefetcher` thread that decodes upcoming replay cohorts' parsed sessions while the writer applies the current cohort, auto-engaging via `pipeline_decode=None` whenever `parallel_threads_effective()` (free-threaded build) and the pass has `>= _PIPELINE_DECODE_MIN_COHORTS` (8) cohorts. PR #3485 (`bdffadc2d`, same day) added `_lineage_aware_replay_order`, which both the prefetcher and the writer loop consume from the identical `ordered_logical_keys` sequence — lineage ordering survives pipelining by construction. Since `pipeline_decode` auto-resolution lives *inside* the shared function, there is no separate per-route knob either the offline or daemon caller could have forgotten to wire — both inherit it for free. **What this PR adds:** `tests/unit/maintenance/test_rebuild_parse_apply_split.py::test_rebuild_index_from_source_sync_auto_engages_pipelined_decode` — drives the real `rebuild_index_from_source_sync` entry point (not the lower-level `backfill_historical_revision_evidence` the existing unit tests already call directly) with a corpus of `_PIPELINE_DECODE_MIN_COHORTS + 4` independent raws and RAM spill tiers shrunk to force every `for_raw` decode through the prefetcher-or-inline fork, then asserts `stage_timings_s["spill_prefetch.consumed"] > 0`. Anti-vacuity: dropping the `pipeline_decode` parameter anywhere in the `rebuild_index_from_source_sync` → ... → `backfill_historical_revision_evidence` threading chain makes this assertion fail with `0`, not a wrong number. **What this PR does NOT add:** no new pipeline plumbing. The remaining fully-serial stage is the up-front `census` classification pass — it structurally precedes the replay loop (cohort membership must be resolved before cohorts can be ordered/classified), and `_ReplaySpillPrefetcher` neither touches nor could touch it without a materially larger redesign (overlapping census-page N+1 with replay-page N's apply, across page boundaries). Left as a residual finding, not implemented here — see the honest measurement below for its current relative weight. ## Before/after measurement Ad hoc synthetic-corpus script (not committed; used `tests/infra/rebuild_cost_model.py`'s `Stratum`/`build_stratum_sample_corpus` machinery against `backfill_historical_revision_evidence` directly, comparing `pipeline_decode=False` — the exact pre-#3478 serial path — against `pipeline_decode=None`/auto, the current production default), 160-raw corpus, ~900KB payloads, 20% chain fraction, 10% ambiguous fraction: | metric | before (serial) | after (auto pipeline) | | --- | --- | --- | | `spill_load` | 3.988s | 0.063s | | `spill_prefetch.decode_concurrent` (hidden behind apply) | n/a | 2.901s | | fraction of original `spill_load` now overlapped | — | **72.7%** | Wall-clock deltas from this run are **not reported as a speedup number**: the host was at load average ~19-22 on 24 cores during measurement (multiple concurrent agent lanes per repo convention), which visibly perturbed unrelated stages (e.g. `census`, which `pipeline_decode` never touches, moved 5.04s → 7.89s between the two runs) — the wall-clock signal was too noisy to trust in isolation. The `spill_load`/`decode_concurrent` stage-timing shift is the reliable signal because it is a structural property of which code path ran, not a wall-clock race against host load. ## Verification ``` python -m devtools test tests/unit/maintenance/test_rebuild_parse_apply_split.py -> 5 passed (includes the new test) python -m devtools test tests/unit/sources/test_revision_backfill.py -k pipelined_decode -> 3 passed (pre-existing outcome-parity proofs for pipeline_decode, confirmed still green: test_pipelined_decode_matches_serial_archive_state[reparse-fallback-lane], test_pipelined_decode_matches_serial_archive_state[sqlite-spill-lane], test_pipelined_decode_respects_batched_replay_commits) python -m devtools test tests/benchmarks/test_rebuild_cost_model.py -k "not full_population" -> 4 passed python -m devtools verify --quick -> exit 0 (ran again on push via pre-push hook, exit 0) ``` Not run: `tests/benchmarks/test_rebuild_cost_model.py::test_full_population_projection` (opt-in, ~40 real rebuild passes, minutes) and the full non-integration suite (`devtools verify --all`) — this PR's surface is a single new test in an already-green file plus a PR-body-only measurement, not new production plumbing. ## AC matrix (against polylogue-2cuv) | AC | status | | --- | --- | | Bounded-memory producer/consumer pipeline overlapping parse decode with apply | **Already satisfied** — `_ReplaySpillPrefetcher` (PR #3478), pre-existing | | Wired on both daemon bulk-rebuild route and offline route | **Already satisfied** — both share `rebuild_index_from_source_sync`; this PR adds the regression test proving it from that shared entry point | | Preserve lineage-aware ordering from #3485 | **Already satisfied** — prefetcher consumes the same `ordered_logical_keys` the writer loop does | | Outcome parity proven by a differential test (accepted_raw_ids/adoption identical) | **Already satisfied** — pre-existing `test_pipelined_decode_matches_serial_archive_state` (byte-identical `RevisionBackfillResult` + full index content manifest, both RAM-miss decode lanes) | | Honest before/after measurement on the rebuild-cost harness fixture | **Satisfied in this PR body** (see above); wall-clock deltas explicitly caveated as unreliable under current host load rather than reported as a speedup claim | | Census-stage overlap (the remaining ~9-16% pre-#3478 serial bucket not touched by Lever A) | **Not implemented / out of scope** — would require overlapping census across page boundaries with the prior page's apply, a materially larger redesign than what this bead's "spill_load 2830s SERIAL" evidence targeted | Ref polylogue-2cuv Co-authored-by: Claude <noreply@anthropic.com>

…jtc, bvnz, 5vbs) Reconciliation pass driven by devtools lab probe bead-pr-reconciliation: verified each bead's own acceptance criteria against the actual diff/AC matrix of its referencing merged PR(s), not just the PR's self-description. Closed (AC fully satisfied): - polylogue-bvnz: PR #3504, all 5 AC items satisfied. - polylogue-5vbs: PR #3439, AC's detection/repair branch satisfied via new daemon/fts_orphan_audit.py sweep. Left open with progress notes (partial, residual scope named): - polylogue-layg.1: PR #3486 satisfies 6/8 AC items; scale fixture + coverage manifest update still missing. - polylogue-hjpx: PR #3485 confirms AC1-5 satisfied (mostly by pre-existing landed work whose ancestry the bead's own stale note had missed), but bd close refuses on open child polylogue-yla8 (the live closure-gate audit AC1-5 doesn't substitute for). - polylogue-6j9c, polylogue-9kjtc: PR #3511 satisfies 2/3 AC items each; live-archive reprice pass over existing session_profiles rows still outstanding for both. Ref polylogue-93xe Co-Authored-By: Claude <noreply@anthropic.com>

Sinity added 4 commits July 31, 2026 23:30

Sinity merged commit ee64e2b into master Jul 31, 2026
3 checks passed

Sinity deleted the feature/storage/raw-authority-execution-completeness branch July 31, 2026 22:18

Sinity added a commit that referenced this pull request Jul 31, 2026

chore(beads): close 9dxn/5q2u/hjpx via PR #3485, narrow f57q

d3039ff

Co-Authored-By: Claude <noreply@anthropic.com>

Sinity mentioned this pull request Jul 31, 2026

test(maintenance): pin pipeline-decode auto-engagement on both rebuild routes #3496

Merged

Navigation Menu

fix(storage): raw-authority fingerprint gating, lineage replay order, dry-run success - #3485

fix(storage): raw-authority fingerprint gating, lineage replay order, dry-run success#3485
Sinity merged 4 commits into
masterfrom
feature/storage/raw-authority-execution-completeness

Sinity commented Jul 31, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 31, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Sinity commented Jul 31, 2026

Summary

Problem

Solution

Verification

Per-bead AC matrix

Anti-vacuity statement

Residuals / follow-ups

Uh oh!

chatgpt-codex-connector Bot commented Jul 31, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant