Phase A: correctness closure from the 2026-07-07 audit

.planning/notes/2026-07-07-execution-loop.md

@@ -0,0 +1,63 @@
+# Execution Loop — 2026-07-07 Improvement Plan
+
+**Plan:** `.planning/notes/2026-07-07-audit-improvement-plan.md` (phases A–F)
+**Orchestrator:** Fable session (this file is its working memory; any fresh session resumes from here)
+**Authority:** Alan approved autonomous execution incl. Phase E spend (2026-07-07). Hard stops: public publishing (F.2/F.3), anything under `runners/codex-ps/**`, and any git operation that rewrites master history.
+
+## GOAL
+
+All six phases (A–F) merged to master with green 3-OS CI, every phase's done-means checklist independently verified, and the Phase E measurement numbers (κ judge-vs-gold, inter-judge κ, position-bias flip rate, self-consistency rate, canary catch rate, A/B verdict vs its pre-registered criterion) recorded in this file and in `evals/` docs. Terminal state: Phase F publishing artifacts staged and presented to Alan for the one remaining gate.
+
+## LOOP
+
+Repeat until GOAL or a hard stop:
+
+1. **Pick** the next phase whose dependencies are merged (order: A → B → C ∥ D → E → F).
+2. **Build wave** — decompose the phase into worker briefs (Goal / Scope / Contract / Done-means, per repo Token Discipline); fan out parallel agents with disjoint file ownership; workers on opus for bash surgery, sonnet for docs/tests, `codex exec` for mechanical sweeps.
+3. **Verify wave** (independent of builders — no agent verifies its own work):
+   a. `bash tests/run-all.sh` in an isolated subagent (summary only reaches the loop).
+   b. Claude adversarial review of the phase diff (adversarial-reviewer agent).
+   c. gpt-5.5 cross-vendor review of the same diff (`codex exec -s read-only`) — the repo's own philosophy applied to its own PRs; reviewer disagreements are surfaced, not averaged.
+   d. Done-means checklist from the plan, checked item-by-item by a non-builder agent.
+4. **Fix loop** — findings from (3) go back to a build agent; one retry with a tighter brief, then escalate to the orchestrator itself. Max 3 fix cycles per phase before the phase is marked BLOCKED here and the loop moves to any non-dependent phase.
+5. **Land** — commit (imperative, <72 chars), push branch, open PR citing this file's SHA, wait for 3-OS CI (background monitor, no polling), **merge on green** (pre-authorized), delete branch.
+6. **Record** — update the Progress and Measurements sections below; capture lessons to ExoCortex; if context is near compaction or ~2h elapsed, write handoff notes here and continue in a fresh session reading this file.
+7. **Regression watch** — after each merge, rerun the full suite on master once and append the result to the trend table. A red master halts the loop and fixes forward immediately.
+
+Loop mechanics: background agents re-invoke the orchestrator on completion (no polling); a ScheduleWakeup heartbeat (~25 min) survives hangs. This file is the single source of truth — sessions are disposable, the loop is not.
+
+## Verification additions (beyond the plan's per-phase done-means)
+
+- **V-1 Cross-vendor PR review** on every phase (loop step 3c) — added because the audit's headline defect (C-1) was a single-reviewer blind spot on a "finished" fix.
+- **V-2 Live smoke per protocol-touching phase** (A, C, D): one real, minimal, non-stubbed run (`--bounces 1`, tiny doc; or one dev-review verify on a 5-line diff) — stub-fidelity gaps have bitten this repo twice; hermetic green is necessary, not sufficient. 💰-tiny, codex-guard-capped.
+- **V-3 Self-referential dogfood gate** (Phase C done-means, kept prominent): the tool must bounce its own BOUNCE-PROTOCOL.md without tripping the honesty gate; the run becomes a permanent sim fixture.
+- **V-4 Sneaky-canary verifier calibration** (Phase E.4): planted plausible-but-wrong diffs; catch rate is the standing rubber-stamp metric.
+- **V-5 Pre-registration** (Phase E.3): the A/B success criterion is written into the run manifest before any run executes.
+- **V-6 Master trend table** (below): scorer values per merge, so drift is visible across the whole campaign, not just within a phase.
+
+## Progress
+
+| Phase | Status | Branch / PR | Verify (suite / adv / codex / done-means) | Notes |
+|-------|--------|-------------|-------------------------------------------|-------|
+| A — Correctness closure | PR #47 open, suite 32/32, awaiting CI → merge | claude/nervous-hodgkin-bcf03d → PR #47 | ✓32/32 / ✓(F1 fixed) / ✓(H1,H2,L1 fixed) / ✓ | Cross-vendor review earned its keep: codex found the partial-failure→converged gap (H1) and both vendors independently flagged the bare-banner auth gap (H2→`output_is_auth_failure` in lib, 3 call sites). Claude reviewer caught the Scenario-F grep regression (F1) + missing guard scenario (→Scenario G). Bonus find-along: bounce-scorer-verification.sh had a Windows jq-CRLF bug (5/7→7/7, fixed) before wiring into run-all (C-5). Accepted residual: none remaining — F2/H2 fixed. Sims: auth-gate 28/28, marker-lifecycle 41/41 (byte-parity intact), audit-hardening 18/18, worktree-mgmt green, reliability 17/17. |
+| B — Robustness/injection | pending | | | |
+| C — Protocol v0.2 | pending | | | includes docs sweep + STACK.md re-check |
+| D — Signal quality | pending | | | can start once C's marker changes are stable |
+| E — Measurement | pending | | | panel-labeled gold set; spend approved |
+| F — Learning loop + distribution | pending | | | F.2/F.3 publishing = HARD STOP for Alan |
+
+## Measurements
+
+| Metric | Value | Date | Source |
+|--------|-------|------|--------|
+| Inter-judge κ (fable-5 vs gpt-5.5, gold set) | – | | E.1 |
+| Judge-vs-gold κ (judge-bounce.sh) | – | | E.2 |
+| Position-bias flip rate | – | | E.2 |
+| Self-consistency (3-run agreement) | – | | E.2 |
+| Verifier canary catch rate (n=3) | – | | E.4 |
+| A/B: cross- vs same-vendor (pre-registered criterion) | – | | E.3 |
+| Master suite trend | baseline: 27 sims + scorer gate green @ 05d151e | 2026-07-07 | V-6 |
+
+## Handoff notes
+
+(none yet)

co-evolve-bouncer.sh

@@ -150,10 +150,18 @@ while [[ $# -gt 0 ]]; do
       shift 2
       ;;
     --agents)
+      # Must be exactly two non-empty, comma-separated agent names. A value with
+      # NO comma (e.g. `--agents claude`) previously self-paired silently:
+      # ${2%%,*} and ${2#*,} both return the whole string, so AGENT_A==AGENT_B
+      # and the "bounce" ran an agent against itself with no error. Require a
+      # comma so that mistake dies loudly. The three checks together — comma
+      # present, not two commas, both names non-empty — mean exactly one comma
+      # separating two non-empty names.
+      [[ "${2:-}" == *","* ]] || die "--agents requires two comma-separated agents (e.g., claude,codex), got: ${2:-<missing>}"
+      [[ "$2" == *","*","* ]] && die "--agents requires exactly two agents (e.g., claude,codex)"
       AGENT_A="${2%%,*}"
       AGENT_B="${2#*,}"
       AGENT_B="${AGENT_B%%,*}"
-      [[ "$2" == *","*","* ]] && die "--agents requires exactly two agents (e.g., claude,codex)"
       [[ -z "$AGENT_A" || -z "$AGENT_B" ]] && die "--agents requires exactly two agents separated by comma (e.g., claude,codex)"
       shift 2
       ;;
@@ -735,6 +743,13 @@ ${CONTEXT_BLOCK}${INPUT_CONTENT}"
 RUN_CONVERGED_NATURALLY="false"
 RUN_FINAL_MARKERS=0
 RUN_FINAL_MARKERS_RAW=0
+# C-2: count bounce passes that produced usable output AND were applied to
+# WORKING_FILE (i.e. reached append_bounce_pass). A loop that breaks on empty
+# agent output (call + retry both empty) never applies a pass, so this stays 0
+# and the post-loop guard refuses to launder the un-reviewed compose draft as a
+# converged final. Healthy runs apply >= 1 pass, so the guard never fires on
+# them and byte-parity is untouched.
+RUN_PASSES_APPLIED=0
 run_bounce_phase() {
   local pass
   local role
@@ -837,8 +852,15 @@ $(cat "$PROTOCOL_TEMPLATE")"
     fi
 
     if [[ ! -s "$output_file" ]]; then
-      log " ERROR: ${current_agent} returned empty output on retry. Stopping."
-      break
+      # C-2/C-8: die, don't break. A `break` here would hand whatever passes
+      # already applied to the post-loop convergence block, which happily
+      # finalizes "converged" on a run the protocol never finished — pass 1
+      # applied, pass 2's agent died, and the half-bounced document launders
+      # into a clean final. An empty retry only ever happens when another pass
+      # was still REQUIRED (markers open, or chain stages pending), so the
+      # honest terminal is the same aborted path as the zero-pass case: the
+      # EXIT trap finalizes status=aborted and a chained --execute never runs.
+      die "bounce pass $pass: ${current_agent} returned empty output on call and retry — the bounce did not complete; run ABORTED. See run.log."
     fi
 
     cp "$output_file" "$RUN_DIR/pass-${pass}-${role}-${current_agent}-raw.md"
@@ -865,6 +887,8 @@ $(cat "$PROTOCOL_TEMPLATE")"
     append_bounce_pass "$STATE_FILE" "$pass" "$role" "$current_agent" \
       "pass-${pass}-${role}-${current_agent}-raw.md" "pass-${pass}-clean.md" \
       "$contested" "$clarify" "$word_count"
+    # C-2: this pass produced usable output and is now recorded in state.passes.
+    RUN_PASSES_APPLIED=$((RUN_PASSES_APPLIED + 1))
 
     # Human check
     if [[ "$AUTO" == "false" ]]; then
@@ -893,6 +917,25 @@ $(cat "$PROTOCOL_TEMPLATE")"
     fi
   done
 
+  # C-2: a bounce that applied ZERO usable passes never reviewed the document,
+  # and WORKING_FILE still holds the un-reviewed compose draft. That draft is
+  # marker-free, so the convergence honesty block below would read 0 raw
+  # markers and finalize "converged" — laundering a failed bounce into a clean
+  # final that --execute would run. Empty agent output now dies IN the loop
+  # (C-8, above), so this guard is the belt-and-suspenders invariant for any
+  # other way of arriving here passless (e.g. --bounces 0). Refuse it: die
+  # non-zero. The EXIT trap (_finalize_bounce_state_on_exit) then finalizes
+  # status=aborted with convergence_status left null, exactly like the
+  # auth-failure abort (bounce-state-simulation.sh S4); the scorer gate fails
+  # on status=aborted, and a chained --execute never runs because the process
+  # died before the hand-off. This is the honest terminal state for "the agent
+  # produced nothing usable" — distinct from `stuck` (passes ran, but markers
+  # could not be resolved). Healthy runs apply >= 1 pass, so this never fires
+  # and the byte-parity path below is unchanged.
+  if (( RUN_PASSES_APPLIED == 0 )); then
+    die "bounce produced zero usable passes; the document was never reviewed — run ABORTED. See run.log."
+  fi
+
   # A-5: convergence is decided by the marker count after the last pass, NOT by
   # mode. Any run — standard OR chain — that ends with 0 live markers converged
   # naturally and takes the byte-parity path (no adjudication). This also covers

dev-review/codex/README.md

@@ -43,6 +43,7 @@ bash dev-review/codex/dev-review.sh --skip-plan --plan .planning/phases/04-docs-
 | `--plan FILE` | Plan file used with `--skip-plan` |
 | `--model MODEL` | Override the Codex model for Codex-backed passes |
 | `--workdir DIR` | Execute against a target working directory |
+| `--run-dir PATH` | Write run artifacts (`state.json`, plan, logs) to `PATH` instead of the default `runs/dev-review-<timestamp>/`. Path-traversal guarded: any `..` segment is rejected (`--run-dir must not contain '..'`) — defense in depth over the harness-side sanitizer |
 | `--lab MODE` | Route to `lab/<MODE>/entry.sh` (opt-in beta channel; fails fast with `unknown --lab mode: <MODE>. Available: <list>` if `lab/<MODE>/` does not exist). See [`lab/README.md`](../../lab/README.md) for boundary, graduation criteria, and sandbox guarantee. |
 
 ### Lab routing

dev-review/codex/dev-review.sh

@@ -113,6 +113,7 @@ Options:
   --live                   Launch visible Windows terminal tailing each phase's stderr (Windows-only; warns + falls back on other OS)
   --branch auto|NAME       Create a feature branch off HEAD before execute (auto = dev-review/auto-<timestamp>-<slug>); mutually exclusive with --worktree
   --worktree auto|PATH     Create a git worktree for isolation before execute (auto = sibling dir); mutually exclusive with --branch
+  --run-dir PATH           Write run artifacts (state.json, plan, logs) to PATH instead of the default runs/dev-review-<timestamp>/ (path-traversal guarded: rejects any '..' segment)
   --parent-run RUN_ID      Lineage tag: record the orchestrator's parent run id in state.orchestration.parent_run_id (re-kicks always get a fresh run dir; no behavior change)
   --lab MODE               Route to lab/<MODE>/entry.sh (opt-in beta channel; see lab/README.md)
   --target FILE            PEL-only: file to mutate (used with --lab pel-proposer; must be repo-relative forward-slash path, e.g. lib/co-evolution.sh — NOT absolute or WSL/Windows-style)
@@ -242,6 +243,13 @@ abort_on_timeout() {
     if [[ -n "${STATE_JSON:-}" ]]; then
       write_state_phase "$STATE_JSON" "$phase_name" "timeout" 124 "$phase_start" "$phase_end"
       write_state_field "$STATE_JSON" ".completed_at" "string" "$phase_end"
+      # A timeout abort is a terminal exit-1 run: mirror the EOF terminal block
+      # so a status reader sees a failed run, not one stuck "pending" mid-phase
+      # (the phase[].exit_code=124 above stays for observability). Without these,
+      # .status kept its "pending" init and .current_phase stayed non-null,
+      # which the status reader treats as "still in <phase>".
+      write_state_field "$STATE_JSON" ".status" "string" "failed"
+      write_state_field "$STATE_JSON" ".current_phase" "null"
     fi
     log "ERROR: ${phase_name} phase timed out after ${PHASE_TIMEOUT}s - aborting run"
     cleanup_runtime_artifacts
@@ -349,7 +357,7 @@ agent_auth_failed() {
   local agent="$1"
   local output_file="${2:-}"
   local stderr_file="${3:-}"
-  local cli_name words
+  local cli_name
 
   cli_name=$(agent_cli_name "$agent")
 
@@ -358,15 +366,19 @@ agent_auth_failed() {
   # a substantial work product that merely echoes auth strings — e.g. plan text,
   # or the auth-detection source itself — is never misread as an auth failure.
   #
-  # (1) Auth banner IN THE OUTPUT, but only when the output is short (< 50
-  #     words). A long output that mentions "Unauthorized"/"Not logged in" is
-  #     real work, not the CLI's own banner.
-  if [[ -n "$output_file" ]] && file_contains_auth_failure "$output_file"; then
-    words=$(wc -w < "$output_file" | tr -d '\r\n ')
-    if (( words < 50 )); then
-      log "WARNING: ${cli_name} authentication failed. Refresh the ${cli_name} CLI session and rerun."
-      return 0
-    fi
+  # (1) Auth banner IN THE OUTPUT: route through lib's anchored
+  #     output_contains_auth_banner (A-2 / C-1). A real CLI auth error prints a
+  #     short banner that STANDS ALONE at the top of its output before doing any
+  #     work, so the strict head-scan catches an auth-error PAGE of any length
+  #     while still letting a long legitimate document that merely echoes
+  #     "Unauthorized"/"Not logged in" mid-body pass. This replaces the old
+  #     loose file_contains_auth_failure + whole-file <50-word ceiling, which
+  #     accepted an auth page longer than 50 words as work product. The
+  #     output_is_auth_failure wrapper keeps the short+loose catch for a bare
+  #     "Unauthorized"-style banner the anchor deliberately excludes (C-8).
+  if [[ -n "$output_file" ]] && output_is_auth_failure "$output_file"; then
+    log "WARNING: ${cli_name} authentication failed. Refresh the ${cli_name} CLI session and rerun."
+    return 0
   fi
 
   # (2) Auth banner in STDERR counts only when the agent produced NO output. A
@@ -505,7 +517,12 @@ inspect_plan_output() {
   PLAN_OUTPUT_REASON=""
   cli_name=$(agent_cli_name "$agent")
 
-  if file_contains_auth_failure "$output_file" || file_contains_auth_failure "$stderr_file"; then
+  # C-6: anchor the OUTPUT-path auth check to lib's strict head-scan so a
+  # legitimate plan that discusses "401 Unauthorized" or "npm login required"
+  # mid-body is not routed to manual review; the loose matcher stays on stderr,
+  # where any auth string is the CLI's own banner (mirrors validate_agent_artifact).
+  # output_is_auth_failure adds back the short+loose catch for a bare banner (C-8).
+  if output_is_auth_failure "$output_file" || file_contains_auth_failure "$stderr_file"; then
     PLAN_OUTPUT_STATUS="review"
     PLAN_OUTPUT_REASON="${cli_name} authentication failed"
     return 1
@@ -1694,12 +1711,16 @@ fi
 # `--branch auto --plan-only` is a silent no-op on the branching side because
 # plan artifacts intentionally stay on the parent branch.
 # Mutually exclusive: parser already rejected both-set; only one path fires.
-if [[ -n "$BRANCH_SPEC" ]]; then
+# PLAN_EXIT==0 guard: a failed compose/bounce phase must not spawn a stray
+# branch/worktree — the run is about to abort with the plan's non-zero exit
+# (the revise loop returns PLAN_EXIT immediately), so any branch created here
+# would be an orphan the executor never touches.
+if [[ "${PLAN_EXIT:-0}" -eq 0 && -n "$BRANCH_SPEC" ]]; then
   BRANCH_CREATED=$(maybe_setup_branch "$WORKDIR" "$BRANCH_SPEC" "$TASK")
   if [[ -n "$BRANCH_CREATED" ]]; then
     write_state_field "$STATE_JSON" ".branch_created" "string" "$BRANCH_CREATED"
   fi
-elif [[ -n "$WORKTREE_SPEC" ]]; then
+elif [[ "${PLAN_EXIT:-0}" -eq 0 && -n "$WORKTREE_SPEC" ]]; then
   _new_wt=$(maybe_setup_worktree "$WORKDIR" "$WORKTREE_SPEC" "$TASK")
   if [[ -n "$_new_wt" ]]; then
     WORKTREE_PATH="$_new_wt"

evals/tests/bounce-scorer-verification.sh

@@ -56,6 +56,9 @@ for fixture_dir in "$FIXTURES"/*/; do
 
   if [[ "$ok" == true ]] && jq -e 'has("failing_checks_include")' "$expected" >/dev/null; then
     while IFS= read -r check; do
+      # Windows-native jq emits CRLF line endings; strip the trailing \r or
+      # $check never matches the clean scorer check name.
+      check=${check%$'\r'}
       jq -e --arg c "$check" '[.dimensions[].checks[]? | select(.ok == false) | .name] | index($c) != null' "$out" >/dev/null || ok=false
     done < <(jq -r '.failing_checks_include[]' "$expected")
   fi