test: replayable loop fixtures and evals (extends #2499) #2582

Goal

Test the controller deterministically and offline against recorded milestone, issue, PR, and check states.

Background

#2499 already scopes replayable AI-harness fixtures for the release train. The same capture-and-replay approach should serve the loop, so controller changes are tested without GitHub or a model.

Next-release planning review of 2026-10-14/15 against main at 7faf6dfb. Tracking: #2555.

Prerequisites: #2577.

Scope

• Fixture format and a recorder for issue, PR, check, and review states.
• A replay harness driving the controller with fixtures and asserting ledger output.
• Seed fixtures from real sequences: a superseded PR, a CI-red repair, a budget stop.

Acceptance Criteria

• Controller tests run with no network and no model.
• At least five fixtures cover success, repair, and each stop condition.
• Fixtures share format with release: publish the stable release-train contract and AI harness in Base 1.11.0 #2499 where possible.

Validation

BASE_CLI_SOURCE_DIR=../base-cli/lib/python \
PYTHONPATH=../base-cli/lib/python:lib/python:cli/python \
python -m pytest <new replay tests>
git diff --check

Non-Goals

• Do not require live GitHub in CI.

Project Fields

• Status: Backlog
• Priority: P2
• Area: CI
• Initiative: Agentic Coding Platform
• Size: M
• Milestone: v1.12.0
• Category: ci

Agent Assignment

Assignee: codeforester. Blocked on #2577. Coordinate with #2499.

Plan revision (2026-10-16)

Move this issue to v1.12.0 and make it a prerequisite for #2578, #2580, and #2581. Add #2576 and #2499 to the prerequisite set alongside #2577. Replay fixtures must cover claim/lease ambiguity, authority denial, redaction failure, stale evidence, interruption/resume, superseded PR, CI-red repair, budget exhaustion, and non-retryable failure. The harness must run without GitHub or a model and must assert ledger events and stop reasons, not just exit codes.