refactor(devtools): prove lifecycle fast-forwards with raw replay

[Sinity]

Summary

Unify derived-tier fast-forward proof under the lifecycle planner and require retained-raw replay equivalence before activation.

Problem

Version-specific upgrade helpers could establish a schema-compatible candidate without proving that the current parser, lowering, and materialization path reproduces the derived model from retained source evidence.

Solution

The generic lifecycle executor records a deterministic bounded raw sample, parser and materialization fingerprints, and canonical table comparisons. Activation refuses source, fingerprint, schema, clone, or replay-proof drift. The obsolete v37 helper and its test are removed.

Verification

direnv exec . devtools test tests/unit/devtools/test_index_fast_forward.py

Result: 4 passed in 9.18s.

Ref polylogue-9rw0.

Summary by CodeRabbit

• New Features

  • Added a plan-driven index fast-forward workflow with preparation and activation commands.
  • Added replay-based validation to verify schema, source data, and generated index integrity before promotion.
  • Added automatic cleanup of orphaned attachment metadata during index advancement.
  • Added recovery safeguards for interrupted promotions and detected data or processing changes.

• Documentation

  • Updated workspace command documentation and added an index transition ownership guide.

[coderabbitai]

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 59773285-bc83-4783-89c5-7e4dfb59f19e

📥 Commits

Reviewing files that changed from the base of the PR and between 2ba1d71 and 42f4cfd.

📒 Files selected for processing (2)

• devtools/index_fast_forward.py
• tests/unit/devtools/test_index_fast_forward.py

📝 Walkthrough

Walkthrough

The PR replaces the version-specific v36-to-v37 actuator with a generic index-fast-forward workflow. It adds clone preparation, retained-raw replay proofs, integrity receipts, activation checks, lifecycle repair execution, documentation, and unit tests.

Changes

Index fast-forward lifecycle

Layer / File(s)	Summary
Lifecycle operations and source identity polylogue/storage/index_generation.py, polylogue/storage/sqlite/lifecycle.py, polylogue/storage/sqlite/archive_tiers/index_fast_forward_executor.py	Source snapshots now include all raw session columns. Version 37 declares and executes orphan attachment native-ID repair.
Schema, identity, and replay proof devtools/index_fast_forward.py	The new module validates archive state, canonicalizes schemas, fingerprints processing inputs, builds corpus manifests, and compares retained-raw replay with the candidate index.
Receipt preparation and activation devtools/index_fast_forward.py	prepare_forward creates and proves an inactive clone before writing a receipt. activate_forward revalidates the receipt and promotes the generation atomically.
Command integration and validation devtools/command_catalog.py, docs/devtools.md, docs/internals.md, tests/unit/devtools/test_index_fast_forward.py	The command and ownership documentation use index-fast-forward. Tests cover promotion and activation rejection gates. The former version-specific module and tests were removed.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant prepare_forward
  participant RetainedRawSessions
  participant CandidateIndex
  participant Receipt
  participant activate_forward
  CLI->>prepare_forward: prepare archive
  prepare_forward->>RetainedRawSessions: replay sampled sessions
  RetainedRawSessions->>CandidateIndex: produce replay corpus
  CandidateIndex-->>prepare_forward: return equivalence proof
  prepare_forward->>Receipt: write integrity-hashed receipt
  CLI->>activate_forward: activate receipt
  activate_forward->>Receipt: revalidate proof and fingerprints
  activate_forward->>CandidateIndex: promote inactive generation

Loading

Possibly related PRs

• Sinity/polylogue#2788: Introduces the earlier version of the same index fast-forward workflow.
• Sinity/polylogue#2931: Directly relates to replacing and generalizing the v36-to-v37 actuator.
• Sinity/polylogue#3322: Relates to the v37 lifecycle declaration and generic executor changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: proving lifecycle fast-forwards with retained raw replay.
Description check	✅ Passed	The description includes the required Summary, Problem, Solution, and Verification sections and accurately describes the refactor.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/test/source-replay-fast-forward-current

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@devtools/index_fast_forward.py`:
- Around line 408-427: Update _require_complete_proof to directly compare the
validated fast_forward_hashes and canonical_replay_hashes dictionaries,
including their scoped contents, and raise IndexFastForwardError when they
differ. Keep the existing structural validation and mismatch_details checks, but
make activation-time proof validation self-contained rather than relying solely
on the replay-generated mismatch list.
- Around line 289-309: Chunk SQLite IN-list lookups to stay below the
host-variable ceiling while preserving ordering, grouping, and selection
semantics: update _sample_manifest and the message lookup in _canonical_hashes,
including the blocks query that binds message IDs. Apply limit only after
collecting the same logical results. Add coverage in
tests/unit/devtools/test_index_fast_forward.py that exceeds a small
variable-limit threshold, such as 10 parameters per chunk, and verifies these
paths complete without too-many-variables errors.
- Around line 357-361: Update the fts query in the hash construction to order
rows by stable identity columns rather than rowid, using the same deterministic
ordering across replay and clone databases. Keep the selected columns and
session filtering unchanged so multi-row FTS samples produce comparable hashes.
- Around line 228-235: Update _expected_surplus to import
FastForwardOperationKind and compare operation.kind directly with
FastForwardOperationKind.DROP_TABLE instead of comparing operation.kind.value to
a string literal.

In `@tests/unit/devtools/test_index_fast_forward.py`:
- Around line 166-187: Add a test alongside the existing fast-forward integrity
tests that prepares a receipt, loads its JSON, mutates a field such as
source_snapshot, and writes it back without updating the hash. Then call
activate_forward and assert IndexFastForwardError with a “hash mismatch”
message, while verifying the archive’s active generation remains v36; reuse the
existing _archive, running_daemon_pid patching, and _no_corpus_failure helpers.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e6f1190c-e44e-427b-b7aa-4042991f8b84

📥 Commits

Reviewing files that changed from the base of the PR and between 3ad0140 and 2ba1d71.

📒 Files selected for processing (10)

• devtools/command_catalog.py
• devtools/index_fast_forward.py
• devtools/index_v37_fast_forward.py
• docs/devtools.md
• docs/internals.md
• polylogue/storage/index_generation.py
• polylogue/storage/sqlite/archive_tiers/index_fast_forward_executor.py
• polylogue/storage/sqlite/lifecycle.py
• tests/unit/devtools/test_index_fast_forward.py
• tests/unit/devtools/test_index_v37_fast_forward.py

💤 Files with no reviewable changes (2)

• devtools/index_v37_fast_forward.py
• tests/unit/devtools/test_index_v37_fast_forward.py