feat(evals): retire fixture-only execution

README.md

@@ -72,7 +72,7 @@ eval-magic init
 
 `init` creates `evals/evals.json` with one valid seed case and the pinned Weeknight example
 codebase. Use `eval-magic init --help` to select another URL, local path, or the current directory,
-and use `eval-magic docs codebase` for fixture selection and provenance details. Edit the prompt
+and use `eval-magic docs codebase` for project selection and provenance details. Edit the prompt
 and expected behavior to describe a realistic task, add concrete assertions as the eval matures,
 then check the file:
 
@@ -114,14 +114,14 @@ eval-magic run --mode revision
 The command help and generated runbook describe baseline selection and the rest of the workflow.
 
 An eval can treat coordinated skills as one treatment by setting `skill_name` to an ordered list.
-Pass one listed member with `--skill`; it remains the eval owner and supplies fixtures. See
+Pass one listed member with `--skill`; it remains the eval owner and supplies overlay files. See
 `eval-magic docs isolation` for the complete configuration, Mode A/B behavior, and provenance.
 
 ## How it works
 
 Each eval case runs once per condition and repetition in its own clean Git repository. The two arms
-receive the same task and fixtures; only the condition under test changes. Assertions can combine
-LLM judgment with runner-owned command checks, transcript checks, and final diff limits. Scripted
+receive the same codebase, task, and overlays; only the condition under test changes. Assertions can
+combine LLM judgment with runner-owned command checks, transcript checks, and final diff limits.
 Multi-turn evals resume one native harness session so follow-up answers remain part of the same
 conversation, whether the turns are scripted or derived by a responder (`eval-magic docs
 conversations`).

docs/cline-notes.md

@@ -19,7 +19,8 @@
   (676-line NDJSON capture exercising `read_files`, `run_commands`, and two `skills`
   invocations), `cline history --json`, several `--id` resume attempts, and
   `eval-magic harness lint harnesses/cline.toml --as-builtin --probe --yes` (the live probe
-  rendered the exec template, dispatched it, and recovered a non-empty `final-message.md`).
+  rendered the exec template, dispatched it, parsed the event capture, and recovered a non-empty
+  final response).
   Against 3.0.53: a second dispatch exercising `editor`, `run_commands`, `read_files`, and
   `skills` (arg/result shapes below), and a guard spike — a hand-staged `.cline/plugins/` project
   plugin whose `beforeTool` hook blocked calls, proving headless plugin auto-load, the
@@ -67,8 +68,7 @@ the descriptor references. "Probe capture" refers to the observed dispatches des
 | `guard.armed_message` | see descriptor | prose authored for eval-magic output (same structure as the other built-ins) |
 | Plugin hook contract | `beforeTool({snapshot, tool, toolCall, input})`; block with `{skip: true, reason}`; 3000ms default hook budget (plugin spawns with a 2s timeout so a hung arbiter fails open); `spawnSync` works from the plugin sandbox | 3.0.53 spike capture + the binary's runtime hook loop; the docs' `tool_call_before`/`fail_closed` vocabulary lags the binary |
 | `shadow.preflight` | `cline-skills` | 3.0.53 root probe (one uniquely-named skill per candidate root): dispatch cwd's `.cline/skills` read, ancestor's NOT (no project walk), `~/.agents/skills` IS read (and receives `cline skill install` global installs); `$CLINE_DIR` overrides the `~/.cline` default (3.0.53 binary) |
-| `dispatch.capture_prefix` | `cline` | chosen name (judge capture files `$response_base.cline-events.jsonl`) |
-| `dispatch.exec_template` | see descriptor | flags from `cline --help` (`--act` from the 3.0.52 binary’s hidden option registration + behavioral write test); `--json` NDJSON stdout and `</dev/null` stdin detach from the docs CLI overview (piped stdin becomes prompt context); final-message jq recovery verified by the live probe. `eval-magic dispatch --judges` reuses this same template, bound to the iteration directory and the judge prompt |
+| `dispatch.exec_template` | see descriptor | flags from `cline --help` (`--act` from the 3.0.52 binary’s hidden option registration + behavioral write test); `--json` NDJSON stdout and `</dev/null` stdin detach from the docs CLI overview (piped stdin becomes prompt context); final-text extraction verified by the live probe. `eval-magic dispatch --judges` reuses this same template, bound to the iteration directory and the judge prompt |
 | `dispatch.next_steps_template` / `manifest_template` | see descriptor | prose authored for eval-magic artifacts (same structure as the other built-ins) |
 
 ## Dispatch quirks
@@ -89,10 +89,9 @@ the descriptor references. "Probe capture" refers to the observed dispatches des
   silent-degradation trap as Claude Code's `acceptEdits`, so the flag stays explicit.
 - Piped stdin is appended to the prompt context (docs CLI overview), so every recipe detaches
   with `</dev/null>`, same as codex.
-- There is no `--output-last-message`: the terminal `run_result` NDJSON event carries the final
-  text, and the exec template's trailing jq step writes `final-message.md` from the captured
-  events (the harness probe's final-message contract checks that file; ingest recovers the text
-  from the events file via `extract.final_text`).
+- The terminal `run_result` NDJSON event carries the final text. The harness probe and ingest both
+  recover it from the event file through the configured transcript parser; no separate completion
+  file or jq post-pass is needed.
 - Stream shape (3.0.52): `agent_event` wrappers (`content_start` per streaming chunk for
   text/reasoning — 634 in the probe — but once per tool call; `content_end` with complete
   blocks; `usage`; `iteration_start`/`iteration_end`; `done`), plus `hook_event` lifecycle
@@ -145,8 +144,8 @@ the descriptor references. "Probe capture" refers to the observed dispatches des
   write/shell classification works. Known blind spot: `read_files`' `files:[{path}]` stays
   nested, so the live-source-read path branch doesn't fire for it (shell-based read detection
   still covers `cat`-style reads).
-- **Riding documented fallbacks** (the `run` preflight names each): no `[conversation]`
-  (scripted `turns` evals are rejected).
+- **No native conversation resume**: no `[conversation]`, so scripted `turns` and responder evals
+  are rejected.
 - **Write guard** (`cline-plugin` engine): `run` auto-arms a staged project plugin at
   `.cline/plugins/slow-powers-eval-guard/index.js` whose `beforeTool` hook forwards every tool
   call to `eval-magic guard-hook --harness cline` (`run_commands`' `commands` array joined into

docs/codex-notes.md

@@ -29,13 +29,13 @@ alias.
 - `--sandbox workspace-write` bounds writes to the env.
 - `--json` streams events to stdout — captured as `outputs/codex-events.jsonl`; stderr goes to
   `codex-stderr.log` so progress/status text (e.g. stdin notices) stays out of the JSONL.
-- `--output-last-message <outputs_dir>/final-message.md` writes the final-message file the
-  pipeline reads.
+- `--json` is the sole completion source; the configured transcript reader extracts the final
+  agent message from the captured events.
 - `</dev/null` matters when dispatching in parallel from a pipe (e.g. `xargs -P`): without it,
   Codex treats piped stdin as additional prompt context.
 - Scripted follow-ups run from `<eval-root>` through `codex exec resume <SESSION_ID> <PROMPT>`;
-  `thread.started.thread_id` supplies the id and each round keeps `--json` plus its own
-  `--output-last-message` capture. Verified against `codex exec resume --help` on 2026-07-24.
+  `thread.started.thread_id` supplies the id and each round keeps `--json` with its own event
+  capture. Verified against `codex exec resume --help` on 2026-07-24.
 
 ## Model flag
 

docs/developer_overview.md

@@ -12,8 +12,8 @@ focused internal notes instead of duplicating their details.
 ## How an evaluation moves through the system
 
 1. `eval-magic init` scaffolds an eval workspace next to a skill. Eval definitions describe the
-   task, fixtures, assertions, conditions, run count, and — for a multi-turn eval — either scripted
-   follow-up turns or a responder policy that derives them.
+   codebase, task, overlay files, assertions, conditions, run count, and — for a multi-turn eval —
+   either scripted follow-up turns or a responder policy that derives them.
 2. `eval-magic run` validates the configuration, resolves and copies the skill under test into the
    iteration, creates isolated task roots, stages the requested skill condition from that copy,
    snapshots the starting state, and writes `RUNBOOK.md`, `dispatch.json`, and related campaign

docs/guides/byoh.md

@@ -21,14 +21,14 @@ eval-magic run --harness cool-custom-harness
 `harness init` writes two files:
 
 - `.eval-magic/harnesses/cool-custom-harness.toml` is a commented descriptor with only `label`
-  enabled.
+  enabled, ready for you to declare its runner contract.
 - `.eval-magic/harnesses/cool-custom-harness-notes.md` records the source and harness version for
   every value you enable.
 
-The label-only descriptor is usable. It falls back to `--no-stage`, inlines each `SKILL.md`, uses
-`llm_judge` and runner-owned assertions for grading, and audits writes after dispatch. Read the
-warnings from `run`; each warning names the lower-fidelity fallback carrying an undeclared
-capability.
+The label-only scaffold is not runner-ready until it declares `[dispatch]`, `[transcript]`, and
+`[tools]`. Once those are valid, optional omissions have narrower fallbacks: no `skills_dir` forces
+`--no-stage` and inlines each `SKILL.md`, while no built-in guard leaves the post-dispatch
+`detect-stray-writes` audit as the filesystem safety net. `run` names those tradeoffs.
 
 For a descriptor that should not live in the project, pass it directly:
 
@@ -50,15 +50,30 @@ label = "cool-custom-harness"
 exec_template = '''
 cool-cli run --cd <eval-root>{model_arg} \
   "Read the file at <dispatch_prompt_path> and follow its instructions exactly." \
-  > <outputs_dir>/final-message.md'''
+  > <outputs_dir>/cool-events.jsonl'''
+
+[tools]
+write = ["file_change"]
+patch = []
+shell = ["command_execution"]
+read = ["file_read"]
+
+[transcript]
+events_filename = "cool-events.jsonl"
+surfaces_skill_invocation = false
+
+[transcript.extract.final_text]
+where = { type = "agent_message" }
+field = "text"
 ```
 
-The command has two requirements:
+The runner-ready descriptor has two requirements:
 
 1. Run the agent from the supplied `<eval-root>`. Each condition and repetition owns a private task
    repository there.
-2. Recover the final reply at `<outputs_dir>/final-message.md`. Redirect stdout or copy the native
-   output there when the CLI cannot write the file itself.
+2. Capture the native event stream at `<outputs_dir>/<transcript.events_filename>` and declare a
+   transcript reader that normalizes a non-empty final response. Use `[transcript.extract]` for a
+   flat JSONL stream or a named parser for a supported non-flat shape.
 
 Prove both requirements before a real eval:
 
@@ -67,9 +82,9 @@ eval-magic harness lint .eval-magic/harnesses/cool-custom-harness.toml --probe
 ```
 
 The probe renders the real command, asks for confirmation, invokes the harness CLI in a temporary
-directory, and checks that the final-message file is nonempty. It can spend tokens and use network
-services. Static lint runs first, and non-interactive use defaults to no; `--yes` explicitly accepts
-the dispatch and `--probe-timeout SECONDS` bounds it.
+directory, parses the configured event capture, and checks that the normalized final response is
+nonempty. It can spend tokens and use network services. Static lint runs first, and non-interactive
+use defaults to no; `--yes` explicitly accepts the dispatch and `--probe-timeout SECONDS` bounds it.
 
 ## Use the generated field reference
 

docs/guides/codebase.md

@@ -2,16 +2,16 @@
 
 > **Audience:** Eval authors choosing, pinning, and verifying a project for coding tasks.
 
-An eval's environment can be a real project rather than a handful of fixture files. Declare a
-`codebase` in `evals.json` and every `(eval, condition, run)` environment is built from a checkout
-of it — with history, on a branch, ready for the agent under test to work in.
+Every eval runs against a project codebase. Declare a `codebase` in `evals.json` and every
+`(eval, condition, run)` environment is built from a checkout of it — with history, on a branch,
+ready for the agent under test to work in.
 
 This matters for anything you cannot judge from a toy problem. Whether a skill makes an agent's
 code *better* is not answerable when the task is small enough that any model succeeds.
 
 ## Choose a source during `init`
 
-With no codebase option, `eval-magic init` uses the Weeknight example fixture at its pinned
+With no codebase option, `eval-magic init` uses the Weeknight example project at its pinned
 baseline:
 
 ```sh
@@ -51,9 +51,9 @@ inputs and `--codebase-cwd` are written relative to the generated `evals/` direc
 portability limits described under "A `path` source is not reproducible elsewhere."
 
 The URL/ref, local path, and current-directory modes are mutually exclusive. The chosen source is
-written into the eval file, so the committed configuration records which fixture the suite uses.
+written into the eval file, so the committed configuration records which project the suite uses.
 
-## Choose the fixture scale
+## Choose the project scale
 
 ### Start with Weeknight
 
@@ -64,14 +64,14 @@ tasks where a compact project makes the agent's decisions easy to inspect.
 
 Suitable tasks include changing planner validation, extending the recipe filters, migrating stored
 state, fixing shopping-list aggregation, or improving an interaction with focused tests. Pin the
-fixture commit in the eval file even when a later fixture revision exists; change the ref as a
+project commit in the eval file even when a later revision exists; change the ref as a
 deliberate eval-suite revision.
 
-### Use eval-magic as a complex fixture
+### Use eval-magic as a complex project
 
 Use [eval-magic](https://github.com/slowdini/eval-magic) when the skill needs a larger codebase with
 cross-module Rust behavior, schemas, generated artifacts, integration tests, and repository-level
-contributor instructions. This fixture is appropriate when navigating and preserving those
+contributor instructions. This project is appropriate when navigating and preserving those
 contracts is part of what the eval should measure.
 
 This command scaffolds a pinned eval-magic source:
@@ -84,7 +84,7 @@ eval-magic init \
 
 The larger repository increases preparation, dispatch, and review work. Prefer Weeknight unless the
 task genuinely needs the extra architectural surface. Project instructions and project-local skill
-sources remain part of either fixture unless the eval opts out as described below.
+sources remain part of either project unless the eval opts out as described below.
 
 ## Declare one
 
@@ -103,7 +103,7 @@ A git repository, with an explicit ref:
 Or a directory on this machine:
 
 ```json
-{ "codebase": { "path": "../../fixtures/legacy-service" } }
+{ "codebase": { "path": "../../projects/legacy-service" } }
 ```
 
 A relative `path` resolves against the directory holding `evals.json`, so a committed config means
@@ -183,8 +183,8 @@ Each dispatch gets its own private environment holding:
   branch when the ref is a tag or a SHA
 - `refs/eval-magic/baseline`, marking the state the agent started from
 
-An eval that declares no `codebase` still gets a Git repository, initialized on `work`, exactly as
-it always has.
+Every selected eval must resolve an effective codebase, either from the top-level default or an
+eval-level override. Validation rejects a configuration that supplies neither.
 
 ## The baseline ref is what the run is measured against
 
@@ -207,8 +207,8 @@ What counts is what Git counts, under the same rules the baseline commit was bui
 
 - The codebase's own `.gitignore` holds, so a run that compiles does not report its build output as
   thousands of touched files.
-- Fixtures and staged skills count even when the codebase ignores their paths — they are committed
-  into the baseline regardless, so a change to one is always visible.
+- Overlay files and staged skills count even when the codebase ignores their paths — they are
+  committed into the baseline regardless, so a change to one is always visible.
 - Framework artifacts under `.eval-magic-outputs/` never count.
 - A nested repository's internals never count: Git tracks no path with a `.git` component.
 - A rename counts as two touched files, one created and one deleted.
@@ -249,15 +249,15 @@ paths. Seeding a task-specific file into a real project is the common case:
 }
 ```
 
-A fixture overwrites a codebase file of the same path.
+An overlay file replaces a codebase file at the same path.
 
 The baseline the runner commits respects the codebase's `.gitignore`, so ignored build output stays
-out of it. Fixtures and staged skills are committed regardless of what the codebase ignores — which
-is also what keeps them inside every later measurement.
+out of it. Overlay files and staged skills are committed regardless of what the codebase ignores —
+which is also what keeps them inside every later measurement.
 
 ## A `path` source is not reproducible elsewhere
 
-Someone reading your published results cannot resolve `../../fixtures/legacy-service`. Their machine
+Someone reading your published results cannot resolve `../../projects/legacy-service`. Their machine
 has that directory somewhere else, or not at all. Nothing can fix that, so the artifacts label it:
 the record carries `host_local: true`, the run prints a warning, and the `BASELINE.md` row says so.
 

docs/guides/isolation.md

@@ -163,9 +163,9 @@ non-empty list:
 ```
 
 With a list, `--skill` selects the eval owner: the member whose `evals/` directory supplies the
-definitions and fixtures, and whose name owns the workspace and promotion destination. The owner
-must appear in the list. `--stage-name` is unavailable because one override cannot name several
-staged skills.
+definitions and overlay files, and whose name owns the workspace and promotion destination. The
+owner must appear in the list. `--stage-name` is unavailable because one override cannot name
+several staged skills.
 
 Every treatment member is copied into the eval home before any dispatch runs, and each condition
 stages from those copies. Mode A stages all treatment members in `with_skill` and none in
@@ -207,8 +207,8 @@ copy is carried there: the arm may not be comparing what it claims to.
 Skill-source isolation is about what a dispatch can *load*. The task repository is about what it can
 *reach*: every dispatch runs in its own private environment, a Git repository with no remotes and
 hooks disabled, marked with `refs/eval-magic/baseline` at the state the agent started from. That
-holds whether the environment was built from fixture files or from a sourced codebase — see
-`eval-magic docs codebase`.
+codebase-backed boundary holds for every eval; task-specific `files` are overlays on the sourced
+tree. See `eval-magic docs codebase`.
 
 The two are independent. An environment can be a faithfully isolated repository while the dispatch
 still loads a live skill source, and a shadowed skill is not made safe by the repository boundary.