test(storage): prove raw-authority fair scheduling and CAS-typed retry (hjpx AC3/AC4)

[Sinity]

Summary

Adds regression tests proving polylogue-hjpx AC3 (bounded fair scheduling that doesn't starve large valid components behind cheap ones) and AC4 (transient retries keep the same plan id; CAS conflicts surface as typed/durable/non-mutating outcomes). No production code gap was found in either area — both invariants were already implemented (largely via hjpx.1, PR #2961, plus later hardening #3029/#3031/#3034/#3043) but lacked direct regression coverage for these specific failure modes.

Problem

polylogue-hjpx's 7-item acceptance criteria include AC3 (fair bounded component scheduling) and AC4 (transient-interruption retry / CAS-conflict typing). Continuing this bead's evidence-first investigation (see bead notes: AC1 was fixed this session via #3337, AC2/AC5 contributed to via the same fix), I read the full history and the closed hjpx.1/hjpx.2 children to avoid duplicating landed work, then investigated whether AC3/AC4 already hold in production or represent a genuine gap.

• AC3: _raw_materialization_ordered_components in polylogue/storage/repair.py orders components by (never-attempted-first, then oldest-attempt-time, then acquisition order) — size is only used as a same-component tie-break, never a cross-component priority. No existing test proved this size-agnostic behavior explicitly, or that a size-preferring order would recreate the exact starvation AC3 names.
• AC4: The generic exception handler around backfill_historical_revision_evidence (repair.py ~L6517-6560) already classifies any RuntimeError — including the CAS-conflict class raised by revision_application.py ("CAS rejected a conflicting accepted head" / "older accepted frontier" / "incomparable accepted frontier") — into a typed, durably-recorded (raw_authority_census_plans in source.db), non-mutating RETRYABLE outcome carrying the same plan_id. No existing test proved plan-id stability across a fail→retry→succeed cycle, or that a genuine CAS-conflict message specifically produces this typed/durable/non-mutating outcome (existing CAS tests only exercise the low-level revision_application.py function raising, not the full reconciler pipeline's classification of that error).

Solution

Three new regression tests in tests/unit/storage/test_repair.py:

1. test_raw_materialization_ordering_is_size_agnostic_and_does_not_starve_large_work — one large-but-executable component (oldest by acquisition) plus five small ones; fair ordering selects the large one first; a cheap-first mutation of _raw_materialization_ordered_components recreates starvation.
2. test_raw_materialization_transient_failure_retries_with_same_plan_id_then_succeeds — a transient lock-style failure produces a retryable outcome with no parse residue; removing the injected failure and retrying executes with the same plan_id.
3. test_raw_materialization_cas_conflict_outcome_is_typed_durable_and_non_mutating — injects the exact CAS-conflict RuntimeError message; asserts the outcome is retryable, durably recorded in raw_authority_census_plans, and leaves zero residue in both source.db (parsed_at_ms) and index.db (raw_revision_applications, raw_revision_heads).

All three were validated anti-vacuously: I temporarily mutated the production code under test (cheap-first ordering; corrupted plan_id in the RETRYABLE branch) and confirmed each test fails for the expected reason before reverting (clean git diff on production code — this PR touches only the test file).

AC5/AC6/AC7 status (not re-litigated by this PR, recorded here for continuity): AC5 is code-complete via hjpx.1 (closed). AC6 (July-15 scale proof) remains open in hjpx.2, blocked on host I/O pressure during corpus generation across 4 documented self-aborts — out of scope to duplicate here. AC7's named commands were run as part of this PR's own verification (below).

Verification

• devtools test tests/unit/storage/test_repair.py tests/unit/sources/test_revision_backfill.py → 109 passed
• devtools test tests/unit/sources/test_revision_backfill.py → 44 passed
• devtools test -k raw_materialization → 118 passed
• devtools test -k raw_authority → 83 passed
• devtools verify --quick → 17/17 steps green, exit 0
• Not run: full devtools verify/--seed-testmon — per repo convention a focused test-only change uses the narrow gate (devtools test <files> + --quick), not a full-suite seed run.

Ref polylogue-hjpx

Summary by CodeRabbit

• Tests
  • Expanded coverage for raw materialization scheduling to ensure larger valid work is not starved by cheaper tasks.
  • Added validation that transient failures can retry successfully while preserving the same operation plan.
  • Added coverage for conflict handling, confirming failures are recorded reliably without leaving partial changes behind.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds unit coverage for raw materialization candidate ordering, transient retry handling, durable CAS-conflict outcomes, and non-mutating failure behavior.

Changes

Raw materialization repair tests

Layer / File(s)	Summary
Candidate selection ordering tests/unit/storage/test_repair.py	Tests fairness- and age-based selection of a large valid component and contrasts it with a monkeypatched cheap-first ordering.
Transient failure retry tests/unit/storage/test_repair.py	Verifies a database-lock failure records a retryable outcome without parsing, then succeeds on retry while preserving the plan ID.
CAS conflict outcome tests/unit/storage/test_repair.py	Verifies typed durable conflict recording and confirms source parsing and index-tier application remain empty.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• Sinity/polylogue#2387: Introduces the raw materialization repair path covered by these tests.
• Sinity/polylogue#2961: Adds the durable raw-authority replay-plan and outcome ledger validated here.
• Sinity/polylogue#3046: Hardens raw-authority repair behavior and related candidate and retry handling.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title is concise and accurately summarizes the new regression tests for fair scheduling and CAS retry behavior.
Description check	✅ Passed	Description covers Summary, Problem, Solution, and Verification, and missing Changelog/Risks is acceptable for a test-only PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/fix/hjpx-ac3-ac4-progress

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unit/storage/test_repair.py`:
- Around line 2444-2446: Update the test stub raise_cas_conflict to record
selected_raw_ids instead of asserting inside the patched call, then assert the
recorded value equals [raw_id] after repair_raw_materialization returns.
Preserve the RuntimeError(cas_message) behavior so the production
exception-to-RETRYABLE path remains exercised.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b3f544a7-545d-4f07-8999-de652bdbc33d

📥 Commits

Reviewing files that changed from the base of the PR and between 2eb2753 and 0f4f15e.

📒 Files selected for processing (1)

• tests/unit/storage/test_repair.py

[coderabbitai]

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm the per-component backfill_historical_revision_evidence call in repair_raw_materialization
# is wrapped in a broad except that stringifies exceptions into the outcome reason.
rg -n 'backfill_historical_revision_evidence\(' -B5 -A25 polylogue/storage/repair.py

Repository: Sinity/polylogue

Length of output: 1936

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the test and implementation around the relevant scope without running repository code.
printf '--- test_repair outline matches ---\n'
fd -a 'test_repair.py' . | while read -r f; do
  wc -l "$f"
  ast-grep outline "$f" --match 'test_raw_materialization_transient_failure_retries_with_same_plan_id_then_succeeds' --view compact 2>/dev/null || true
done

printf '\n--- tests/unit/storage/test_repair.py lines 2360-2465 ---\n'
sed -n '2360,2465p' tests/unit/storage/test_repair.py | nl -ba -v2360

printf '\n--- polylogue/storage/repair.py lines 6460-6545 ---\n'
sed -n '6460,6545p' polylogue/storage/repair.py | nl -ba -v6460

printf '\n--- deterministic except shape probe from source ---\n'
python3 - <<'PY'
from pathlib import Path
p = Path('polylogue/storage/repair.py')
s = p.read_text()
start = s.index('for plan, raw_id in zip(executable_plans, executable_raw_ids, strict=True):')
end = s.index('return RawReplayPlanResults', start)
chunk = s[start:end]
print('contains bare try:', 'try:' in chunk)
print('except Exception or base Exception:', 'except Exception' in chunk or 'except BaseException' in chunk)
print('contains AttributeError specifically before second except:')
for i, idx in enumerate(chunk.find('except ') for _ in range(10)):
    pass
# Locate except bodies after the critical try
except_lines = []
for m in __import__('re').finditer(r'except[^\n]*:(?P<body>(?:\n\s+.*)*)', chunk):
    body = m.group('body').splitlines()
    if body and body[0].strip():
        first = body[0].strip()
        if len(first) < 160 and first.startswith('outcome'):
            except_lines.append((m.group().splitlines()[0].strip(), first))
print('outcome assignments in except bodies:')
for line, first in except_lines:
    print(line)
    print(first)
PY

Repository: Sinity/polylogue

Length of output: 245

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- test_repair outline matches ---'
fd -a 'test_repair.py' . | while read -r f; do
  wc -l "$f"
  ast-grep outline "$f" --match 'test_raw_materialization_transient_failure_retries_with_same_plan_id_then_succeeds' --view compact 2>/dev/null || true
done

printf '%s\n' ''
printf '%s\n' '--- tests/unit/storage/test_repair.py lines 2360-2465 ---'
sed -n '2360,2465p' tests/unit/storage/test_repair.py | nl -ba -v2360

printf '%s\n' ''
printf '%s\n' '--- polylogue/storage/repair.py lines 6460-6545 ---'
sed -n '6460,6545p' polylogue/storage/repair.py | nl -ba -v6460

printf '%s\n' ''
printf '%s\n' '--- deterministic except shape probe from source ---'
python3 - <<'PY'
from pathlib import Path
import re

p = Path('polylogue/storage/repair.py')
s = p.read_text()
start = s.index('for plan, raw_id in zip(executable_plans, executable_raw_ids, strict=True):')
end = s.index('return RawReplayPlanResults', start)
chunk = s[start:end]

print('contains bare try:', 'try:' in chunk)
print('contains except Exception:', 'except Exception' in chunk)
print('contains except BaseException:', 'except BaseException' in chunk)

except_lines = []
for m in re.finditer(r'except[^\n]*:(?P<body>(?:\n\s+.*)*)', chunk):
    body = m.group('body').splitlines()
    if body and body[0].strip().startswith('outcome = RawReplayPlanOutcome'):
        first = body[0].strip()
        except_lines.append((m.group().splitlines()[0].strip(), first))

print('outcome assignments in except bodies:')
for line, first in except_lines:
    print(line)
    print(first)
PY

Repository: Sinity/polylogue

Length of output: 348

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- tests/unit/storage/test_repair.py lines 2360-2465 ---'
sed -n '2360,2465p' tests/unit/storage/test_repair.py

printf '%s\n' ''
printf '%s\n' '--- polylogue/storage/repair.py lines 6460-6545 ---'
sed -n '6460,6545p' polylogue/storage/repair.py

printf '%s\n' ''
printf '%s\n' '--- deterministic except shape probe from source ---'
python3 - <<'PY'
from pathlib import Path
import re

p = Path('polylogue/storage/repair.py')
s = p.read_text()
start = s.index('for plan, raw_id in zip(executable_plans, executable_raw_ids, strict=True):')
end = s.index('return RawReplayPlanResults', start)
chunk = s[start:end]

print('contains bare try:', 'try:' in chunk)
print('contains except Exception:', 'except Exception' in chunk)
print('contains except BaseException:', 'except BaseException' in chunk)

except_lines = []
for m in re.finditer(r'except[^\n]*:(?P<body>(?:\n\s+.*)*)', chunk):
    body = m.group('body').splitlines()
    if body and body[0].strip().startswith('outcome = RawReplayPlanOutcome'):
        first = body[0].strip()
        except_lines.append((m.group().splitlines()[0].strip(), first))

print('outcome assignments in except bodies:')
for line, first in except_lines:
    print(line)
    print(first)
PY

Repository: Sinity/polylogue

Length of output: 10222

---

Move the selected_raw_ids invariant outside the patched call

repair_raw_materialization wraps backfill_historical_revision_evidence() in except Exception and converts runtime failures to a RETRYABLE outcome with reason=f"component execution raised {type(exc).__name__}: {exc}". If assert selected_raw_ids == [raw_id] fails, the test surfaces as a failed retryable run instead of a clear assertion failure. Record the call arg and assert after repair_raw_materialization returns.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/storage/test_repair.py` around lines 2444 - 2446, Update the test
stub raise_cas_conflict to record selected_raw_ids instead of asserting inside
the patched call, then assert the recorded value equals [raw_id] after
repair_raw_materialization returns. Preserve the RuntimeError(cas_message)
behavior so the production exception-to-RETRYABLE path remains exercised.