test(maintenance): pin pipeline-decode auto-engagement on both rebuild routes

[Sinity]

Summary

polylogue-2cuv asked for a bounded-memory producer/consumer pipeline overlapping rebuild parse decode with the writer's apply work, on both the daemon bulk-rebuild route and the offline CLI route. Re-verification against current master found this already shipped by PR #3478 (_ReplaySpillPrefetcher, "Lever A"), with lineage-order preservation from PR #3485 landed the same day. This PR does not add pipeline plumbing (none is missing) — it adds one regression test proving the pipeline auto-engages from the production entry point both routes share, plus an honest before/after measurement in the PR description below.

Problem

The bead's core measured claim, from a real 4h22m rebuild receipt: parse_s (census 1202s + spill_load 2830s) + apply_s (8601s) == total (12633s) exactly — zero overlap between decode and the single SQLite writer. spill_load (2830s, 22% of the pass) was documented as strictly SERIAL pickle.loads/reparse work interleaved with writer apply work. The bead asked for a producer/consumer queue so census+spill hide behind apply, on both daemon/bulk_rebuild.py and maintenance/rebuild_index.py.

Solution

Finding: already done. Both routes call the identical chain:

• Offline: maintenance/rebuild_index.py → rebuild_index_from_source_sync
• Daemon: daemon/bulk_rebuild.py::run_daemon_bulk_rebuild_pass → rebuild_index_from_source_sync (same function, scheduled through the daemon's write coordinator)

Both funnel into maintenance/replay.py::rebuild_index_from_source → sources/revision_backfill.py::backfill_historical_revision_evidence, which since PR #3478 (66515459c/c6e275bca, "pipeline replay decode off the writer thread") runs a background _ReplaySpillPrefetcher thread that decodes upcoming replay cohorts' parsed sessions while the writer applies the current cohort, auto-engaging via pipeline_decode=None whenever parallel_threads_effective() (free-threaded build) and the pass has >= _PIPELINE_DECODE_MIN_COHORTS (8) cohorts. PR #3485 (bdffadc2d, same day) added _lineage_aware_replay_order, which both the prefetcher and the writer loop consume from the identical ordered_logical_keys sequence — lineage ordering survives pipelining by construction.

Since pipeline_decode auto-resolution lives inside the shared function, there is no separate per-route knob either the offline or daemon caller could have forgotten to wire — both inherit it for free.

What this PR adds: tests/unit/maintenance/test_rebuild_parse_apply_split.py::test_rebuild_index_from_source_sync_auto_engages_pipelined_decode — drives the real rebuild_index_from_source_sync entry point (not the lower-level backfill_historical_revision_evidence the existing unit tests already call directly) with a corpus of _PIPELINE_DECODE_MIN_COHORTS + 4 independent raws and RAM spill tiers shrunk to force every for_raw decode through the prefetcher-or-inline fork, then asserts stage_timings_s["spill_prefetch.consumed"] > 0. Anti-vacuity: dropping the pipeline_decode parameter anywhere in the rebuild_index_from_source_sync → ... → backfill_historical_revision_evidence threading chain makes this assertion fail with 0, not a wrong number.

What this PR does NOT add: no new pipeline plumbing. The remaining fully-serial stage is the up-front census classification pass — it structurally precedes the replay loop (cohort membership must be resolved before cohorts can be ordered/classified), and _ReplaySpillPrefetcher neither touches nor could touch it without a materially larger redesign (overlapping census-page N+1 with replay-page N's apply, across page boundaries). Left as a residual finding, not implemented here — see the honest measurement below for its current relative weight.

Before/after measurement

Ad hoc synthetic-corpus script (not committed; used tests/infra/rebuild_cost_model.py's Stratum/build_stratum_sample_corpus machinery against backfill_historical_revision_evidence directly, comparing pipeline_decode=False — the exact pre-#3478 serial path — against pipeline_decode=None/auto, the current production default), 160-raw corpus, ~900KB payloads, 20% chain fraction, 10% ambiguous fraction:

metric	before (serial)	after (auto pipeline)
spill_load	3.988s	0.063s
spill_prefetch.decode_concurrent (hidden behind apply)	n/a	2.901s
fraction of original spill_load now overlapped	—	72.7%

Wall-clock deltas from this run are not reported as a speedup number: the host was at load average ~19-22 on 24 cores during measurement (multiple concurrent agent lanes per repo convention), which visibly perturbed unrelated stages (e.g. census, which pipeline_decode never touches, moved 5.04s → 7.89s between the two runs) — the wall-clock signal was too noisy to trust in isolation. The spill_load/decode_concurrent stage-timing shift is the reliable signal because it is a structural property of which code path ran, not a wall-clock race against host load.

Verification

python -m devtools test tests/unit/maintenance/test_rebuild_parse_apply_split.py
  -> 5 passed (includes the new test)
python -m devtools test tests/unit/sources/test_revision_backfill.py -k pipelined_decode
  -> 3 passed (pre-existing outcome-parity proofs for pipeline_decode, confirmed still green:
     test_pipelined_decode_matches_serial_archive_state[reparse-fallback-lane],
     test_pipelined_decode_matches_serial_archive_state[sqlite-spill-lane],
     test_pipelined_decode_respects_batched_replay_commits)
python -m devtools test tests/benchmarks/test_rebuild_cost_model.py -k "not full_population"
  -> 4 passed
python -m devtools verify --quick -> exit 0 (ran again on push via pre-push hook, exit 0)

Not run: tests/benchmarks/test_rebuild_cost_model.py::test_full_population_projection (opt-in, ~40 real rebuild passes, minutes) and the full non-integration suite (devtools verify --all) — this PR's surface is a single new test in an already-green file plus a PR-body-only measurement, not new production plumbing.

AC matrix (against polylogue-2cuv)

AC	status
Bounded-memory producer/consumer pipeline overlapping parse decode with apply	Already satisfied — _ReplaySpillPrefetcher (PR #3478), pre-existing
Wired on both daemon bulk-rebuild route and offline route	Already satisfied — both share rebuild_index_from_source_sync; this PR adds the regression test proving it from that shared entry point
Preserve lineage-aware ordering from #3485	Already satisfied — prefetcher consumes the same ordered_logical_keys the writer loop does
Outcome parity proven by a differential test (accepted_raw_ids/adoption identical)	Already satisfied — pre-existing test_pipelined_decode_matches_serial_archive_state (byte-identical RevisionBackfillResult + full index content manifest, both RAM-miss decode lanes)
Honest before/after measurement on the rebuild-cost harness fixture	Satisfied in this PR body (see above); wall-clock deltas explicitly caveated as unreliable under current host load rather than reported as a speedup claim
Census-stage overlap (the remaining ~9-16% pre-#3478 serial bucket not touched by Lever A)	Not implemented / out of scope — would require overlapping census across page boundaries with the prior page's apply, a materially larger redesign than what this bead's "spill_load 2830s SERIAL" evidence targeted

Ref polylogue-2cuv

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a168a503-3a05-414c-8996-b1cd4ab0b506

📥 Commits

Reviewing files that changed from the base of the PR and between 4ab9dcf and 4ffc848.

📒 Files selected for processing (1)

• tests/unit/maintenance/test_rebuild_parse_apply_split.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}