diff --git a/README.md b/README.md index 63b583e..a383bb7 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ eval-magic init `init` creates `evals/evals.json` with one valid seed case and the pinned Weeknight example codebase. Use `eval-magic init --help` to select another URL, local path, or the current directory, -and use `eval-magic docs codebase` for fixture selection and provenance details. Edit the prompt +and use `eval-magic docs codebase` for project selection and provenance details. Edit the prompt and expected behavior to describe a realistic task, add concrete assertions as the eval matures, then check the file: @@ -114,14 +114,14 @@ eval-magic run --mode revision The command help and generated runbook describe baseline selection and the rest of the workflow. An eval can treat coordinated skills as one treatment by setting `skill_name` to an ordered list. -Pass one listed member with `--skill`; it remains the eval owner and supplies fixtures. See +Pass one listed member with `--skill`; it remains the eval owner and supplies overlay files. See `eval-magic docs isolation` for the complete configuration, Mode A/B behavior, and provenance. ## How it works Each eval case runs once per condition and repetition in its own clean Git repository. The two arms -receive the same task and fixtures; only the condition under test changes. Assertions can combine -LLM judgment with runner-owned command checks, transcript checks, and final diff limits. Scripted +receive the same codebase, task, and overlays; only the condition under test changes. Assertions can +combine LLM judgment with runner-owned command checks, transcript checks, and final diff limits. Multi-turn evals resume one native harness session so follow-up answers remain part of the same conversation, whether the turns are scripted or derived by a responder (`eval-magic docs conversations`). diff --git a/docs/cline-notes.md b/docs/cline-notes.md index b9a1525..8996ca9 100644 --- a/docs/cline-notes.md +++ b/docs/cline-notes.md @@ -19,7 +19,8 @@ (676-line NDJSON capture exercising `read_files`, `run_commands`, and two `skills` invocations), `cline history --json`, several `--id` resume attempts, and `eval-magic harness lint harnesses/cline.toml --as-builtin --probe --yes` (the live probe - rendered the exec template, dispatched it, and recovered a non-empty `final-message.md`). + rendered the exec template, dispatched it, parsed the event capture, and recovered a non-empty + final response). Against 3.0.53: a second dispatch exercising `editor`, `run_commands`, `read_files`, and `skills` (arg/result shapes below), and a guard spike — a hand-staged `.cline/plugins/` project plugin whose `beforeTool` hook blocked calls, proving headless plugin auto-load, the @@ -67,8 +68,7 @@ the descriptor references. "Probe capture" refers to the observed dispatches des | `guard.armed_message` | see descriptor | prose authored for eval-magic output (same structure as the other built-ins) | | Plugin hook contract | `beforeTool({snapshot, tool, toolCall, input})`; block with `{skip: true, reason}`; 3000ms default hook budget (plugin spawns with a 2s timeout so a hung arbiter fails open); `spawnSync` works from the plugin sandbox | 3.0.53 spike capture + the binary's runtime hook loop; the docs' `tool_call_before`/`fail_closed` vocabulary lags the binary | | `shadow.preflight` | `cline-skills` | 3.0.53 root probe (one uniquely-named skill per candidate root): dispatch cwd's `.cline/skills` read, ancestor's NOT (no project walk), `~/.agents/skills` IS read (and receives `cline skill install` global installs); `$CLINE_DIR` overrides the `~/.cline` default (3.0.53 binary) | -| `dispatch.capture_prefix` | `cline` | chosen name (judge capture files `$response_base.cline-events.jsonl`) | -| `dispatch.exec_template` | see descriptor | flags from `cline --help` (`--act` from the 3.0.52 binary’s hidden option registration + behavioral write test); `--json` NDJSON stdout and ``, same as codex. -- There is no `--output-last-message`: the terminal `run_result` NDJSON event carries the final - text, and the exec template's trailing jq step writes `final-message.md` from the captured - events (the harness probe's final-message contract checks that file; ingest recovers the text - from the events file via `extract.final_text`). +- The terminal `run_result` NDJSON event carries the final text. The harness probe and ingest both + recover it from the event file through the configured transcript parser; no separate completion + file or jq post-pass is needed. - Stream shape (3.0.52): `agent_event` wrappers (`content_start` per streaming chunk for text/reasoning — 634 in the probe — but once per tool call; `content_end` with complete blocks; `usage`; `iteration_start`/`iteration_end`; `done`), plus `hook_event` lifecycle @@ -145,8 +144,8 @@ the descriptor references. "Probe capture" refers to the observed dispatches des write/shell classification works. Known blind spot: `read_files`' `files:[{path}]` stays nested, so the live-source-read path branch doesn't fire for it (shell-based read detection still covers `cat`-style reads). -- **Riding documented fallbacks** (the `run` preflight names each): no `[conversation]` - (scripted `turns` evals are rejected). +- **No native conversation resume**: no `[conversation]`, so scripted `turns` and responder evals + are rejected. - **Write guard** (`cline-plugin` engine): `run` auto-arms a staged project plugin at `.cline/plugins/slow-powers-eval-guard/index.js` whose `beforeTool` hook forwards every tool call to `eval-magic guard-hook --harness cline` (`run_commands`' `commands` array joined into diff --git a/docs/codex-notes.md b/docs/codex-notes.md index 1aaa79d..0657b8c 100644 --- a/docs/codex-notes.md +++ b/docs/codex-notes.md @@ -29,13 +29,13 @@ alias. - `--sandbox workspace-write` bounds writes to the env. - `--json` streams events to stdout — captured as `outputs/codex-events.jsonl`; stderr goes to `codex-stderr.log` so progress/status text (e.g. stdin notices) stays out of the JSONL. -- `--output-last-message /final-message.md` writes the final-message file the - pipeline reads. +- `--json` is the sole completion source; the configured transcript reader extracts the final + agent message from the captured events. - `` through `codex exec resume `; - `thread.started.thread_id` supplies the id and each round keeps `--json` plus its own - `--output-last-message` capture. Verified against `codex exec resume --help` on 2026-07-24. + `thread.started.thread_id` supplies the id and each round keeps `--json` with its own event + capture. Verified against `codex exec resume --help` on 2026-07-24. ## Model flag diff --git a/docs/developer_overview.md b/docs/developer_overview.md index 3ed3a29..aee2ff2 100644 --- a/docs/developer_overview.md +++ b/docs/developer_overview.md @@ -12,8 +12,8 @@ focused internal notes instead of duplicating their details. ## How an evaluation moves through the system 1. `eval-magic init` scaffolds an eval workspace next to a skill. Eval definitions describe the - task, fixtures, assertions, conditions, run count, and — for a multi-turn eval — either scripted - follow-up turns or a responder policy that derives them. + codebase, task, overlay files, assertions, conditions, run count, and — for a multi-turn eval — + either scripted follow-up turns or a responder policy that derives them. 2. `eval-magic run` validates the configuration, resolves and copies the skill under test into the iteration, creates isolated task roots, stages the requested skill condition from that copy, snapshots the starting state, and writes `RUNBOOK.md`, `dispatch.json`, and related campaign diff --git a/docs/guides/byoh.md b/docs/guides/byoh.md index 4c2eabf..15ecfa7 100644 --- a/docs/guides/byoh.md +++ b/docs/guides/byoh.md @@ -21,14 +21,14 @@ eval-magic run --harness cool-custom-harness `harness init` writes two files: - `.eval-magic/harnesses/cool-custom-harness.toml` is a commented descriptor with only `label` - enabled. + enabled, ready for you to declare its runner contract. - `.eval-magic/harnesses/cool-custom-harness-notes.md` records the source and harness version for every value you enable. -The label-only descriptor is usable. It falls back to `--no-stage`, inlines each `SKILL.md`, uses -`llm_judge` and runner-owned assertions for grading, and audits writes after dispatch. Read the -warnings from `run`; each warning names the lower-fidelity fallback carrying an undeclared -capability. +The label-only scaffold is not runner-ready until it declares `[dispatch]`, `[transcript]`, and +`[tools]`. Once those are valid, optional omissions have narrower fallbacks: no `skills_dir` forces +`--no-stage` and inlines each `SKILL.md`, while no built-in guard leaves the post-dispatch +`detect-stray-writes` audit as the filesystem safety net. `run` names those tradeoffs. For a descriptor that should not live in the project, pass it directly: @@ -50,15 +50,30 @@ label = "cool-custom-harness" exec_template = ''' cool-cli run --cd {model_arg} \ "Read the file at and follow its instructions exactly." \ - > /final-message.md''' + > /cool-events.jsonl''' + +[tools] +write = ["file_change"] +patch = [] +shell = ["command_execution"] +read = ["file_read"] + +[transcript] +events_filename = "cool-events.jsonl" +surfaces_skill_invocation = false + +[transcript.extract.final_text] +where = { type = "agent_message" } +field = "text" ``` -The command has two requirements: +The runner-ready descriptor has two requirements: 1. Run the agent from the supplied ``. Each condition and repetition owns a private task repository there. -2. Recover the final reply at `/final-message.md`. Redirect stdout or copy the native - output there when the CLI cannot write the file itself. +2. Capture the native event stream at `/` and declare a + transcript reader that normalizes a non-empty final response. Use `[transcript.extract]` for a + flat JSONL stream or a named parser for a supported non-flat shape. Prove both requirements before a real eval: @@ -67,9 +82,9 @@ eval-magic harness lint .eval-magic/harnesses/cool-custom-harness.toml --probe ``` The probe renders the real command, asks for confirmation, invokes the harness CLI in a temporary -directory, and checks that the final-message file is nonempty. It can spend tokens and use network -services. Static lint runs first, and non-interactive use defaults to no; `--yes` explicitly accepts -the dispatch and `--probe-timeout SECONDS` bounds it. +directory, parses the configured event capture, and checks that the normalized final response is +nonempty. It can spend tokens and use network services. Static lint runs first, and non-interactive +use defaults to no; `--yes` explicitly accepts the dispatch and `--probe-timeout SECONDS` bounds it. ## Use the generated field reference diff --git a/docs/guides/codebase.md b/docs/guides/codebase.md index 92836c6..32e1c8d 100644 --- a/docs/guides/codebase.md +++ b/docs/guides/codebase.md @@ -2,16 +2,16 @@ > **Audience:** Eval authors choosing, pinning, and verifying a project for coding tasks. -An eval's environment can be a real project rather than a handful of fixture files. Declare a -`codebase` in `evals.json` and every `(eval, condition, run)` environment is built from a checkout -of it — with history, on a branch, ready for the agent under test to work in. +Every eval runs against a project codebase. Declare a `codebase` in `evals.json` and every +`(eval, condition, run)` environment is built from a checkout of it — with history, on a branch, +ready for the agent under test to work in. This matters for anything you cannot judge from a toy problem. Whether a skill makes an agent's code *better* is not answerable when the task is small enough that any model succeeds. ## Choose a source during `init` -With no codebase option, `eval-magic init` uses the Weeknight example fixture at its pinned +With no codebase option, `eval-magic init` uses the Weeknight example project at its pinned baseline: ```sh @@ -51,9 +51,9 @@ inputs and `--codebase-cwd` are written relative to the generated `evals/` direc portability limits described under "A `path` source is not reproducible elsewhere." The URL/ref, local path, and current-directory modes are mutually exclusive. The chosen source is -written into the eval file, so the committed configuration records which fixture the suite uses. +written into the eval file, so the committed configuration records which project the suite uses. -## Choose the fixture scale +## Choose the project scale ### Start with Weeknight @@ -64,14 +64,14 @@ tasks where a compact project makes the agent's decisions easy to inspect. Suitable tasks include changing planner validation, extending the recipe filters, migrating stored state, fixing shopping-list aggregation, or improving an interaction with focused tests. Pin the -fixture commit in the eval file even when a later fixture revision exists; change the ref as a +project commit in the eval file even when a later revision exists; change the ref as a deliberate eval-suite revision. -### Use eval-magic as a complex fixture +### Use eval-magic as a complex project Use [eval-magic](https://github.com/slowdini/eval-magic) when the skill needs a larger codebase with cross-module Rust behavior, schemas, generated artifacts, integration tests, and repository-level -contributor instructions. This fixture is appropriate when navigating and preserving those +contributor instructions. This project is appropriate when navigating and preserving those contracts is part of what the eval should measure. This command scaffolds a pinned eval-magic source: @@ -84,7 +84,7 @@ eval-magic init \ The larger repository increases preparation, dispatch, and review work. Prefer Weeknight unless the task genuinely needs the extra architectural surface. Project instructions and project-local skill -sources remain part of either fixture unless the eval opts out as described below. +sources remain part of either project unless the eval opts out as described below. ## Declare one @@ -103,7 +103,7 @@ A git repository, with an explicit ref: Or a directory on this machine: ```json -{ "codebase": { "path": "../../fixtures/legacy-service" } } +{ "codebase": { "path": "../../projects/legacy-service" } } ``` A relative `path` resolves against the directory holding `evals.json`, so a committed config means @@ -183,8 +183,8 @@ Each dispatch gets its own private environment holding: branch when the ref is a tag or a SHA - `refs/eval-magic/baseline`, marking the state the agent started from -An eval that declares no `codebase` still gets a Git repository, initialized on `work`, exactly as -it always has. +Every selected eval must resolve an effective codebase, either from the top-level default or an +eval-level override. Validation rejects a configuration that supplies neither. ## The baseline ref is what the run is measured against @@ -207,8 +207,8 @@ What counts is what Git counts, under the same rules the baseline commit was bui - The codebase's own `.gitignore` holds, so a run that compiles does not report its build output as thousands of touched files. -- Fixtures and staged skills count even when the codebase ignores their paths — they are committed - into the baseline regardless, so a change to one is always visible. +- Overlay files and staged skills count even when the codebase ignores their paths — they are + committed into the baseline regardless, so a change to one is always visible. - Framework artifacts under `.eval-magic-outputs/` never count. - A nested repository's internals never count: Git tracks no path with a `.git` component. - A rename counts as two touched files, one created and one deleted. @@ -249,15 +249,15 @@ paths. Seeding a task-specific file into a real project is the common case: } ``` -A fixture overwrites a codebase file of the same path. +An overlay file replaces a codebase file at the same path. The baseline the runner commits respects the codebase's `.gitignore`, so ignored build output stays -out of it. Fixtures and staged skills are committed regardless of what the codebase ignores — which -is also what keeps them inside every later measurement. +out of it. Overlay files and staged skills are committed regardless of what the codebase ignores — +which is also what keeps them inside every later measurement. ## A `path` source is not reproducible elsewhere -Someone reading your published results cannot resolve `../../fixtures/legacy-service`. Their machine +Someone reading your published results cannot resolve `../../projects/legacy-service`. Their machine has that directory somewhere else, or not at all. Nothing can fix that, so the artifacts label it: the record carries `host_local: true`, the run prints a warning, and the `BASELINE.md` row says so. diff --git a/docs/guides/isolation.md b/docs/guides/isolation.md index 07a8d09..a8140f7 100644 --- a/docs/guides/isolation.md +++ b/docs/guides/isolation.md @@ -163,9 +163,9 @@ non-empty list: ``` With a list, `--skill` selects the eval owner: the member whose `evals/` directory supplies the -definitions and fixtures, and whose name owns the workspace and promotion destination. The owner -must appear in the list. `--stage-name` is unavailable because one override cannot name several -staged skills. +definitions and overlay files, and whose name owns the workspace and promotion destination. The +owner must appear in the list. `--stage-name` is unavailable because one override cannot name +several staged skills. Every treatment member is copied into the eval home before any dispatch runs, and each condition stages from those copies. Mode A stages all treatment members in `with_skill` and none in @@ -207,8 +207,8 @@ copy is carried there: the arm may not be comparing what it claims to. Skill-source isolation is about what a dispatch can *load*. The task repository is about what it can *reach*: every dispatch runs in its own private environment, a Git repository with no remotes and hooks disabled, marked with `refs/eval-magic/baseline` at the state the agent started from. That -holds whether the environment was built from fixture files or from a sourced codebase — see -`eval-magic docs codebase`. +codebase-backed boundary holds for every eval; task-specific `files` are overlays on the sourced +tree. See `eval-magic docs codebase`. The two are independent. An environment can be a faithfully isolated repository while the dispatch still loads a live skill source, and a shadowed skill is not made safe by the repository boundary. diff --git a/docs/opencode-notes.md b/docs/opencode-notes.md index 26109d4..6590f38 100644 --- a/docs/opencode-notes.md +++ b/docs/opencode-notes.md @@ -48,15 +48,12 @@ rules (a regex + length cap) — is the descriptor file `harnesses/opencode.toml - Headless permission asks are auto-**rejected** unless `--auto` is passed; explicit `deny` rules are still enforced under `--auto`. - `--format json` streams raw JSON events (`tool_use` / `text` / `step_finish` / `error`) to - stdout; there is no `--output-last-message`, so the final message comes from the `text` - events via transcript ingest (the dispatch prompt still asks for `outputs/final-message.md`, - which wins when the agent writes it). + stdout; the final message comes from the `text` events via transcript ingest. - `-m` takes models in `provider/model` format; the value passes through verbatim. - Live-verified on v1.18.3 (one `opencode run` per the recipe + `ingest`, #153): the staged skill is discovered and invoked under its slug via the `skill` tool, and the events file - ingests to a full run record. An operator config with an explicit `edit: deny` rule blocks - the `final-message.md` write even under `--auto` (deny rules stay enforced) — record-runs - then falls back to the transcript's last `text` part, so the run still records cleanly. + ingests to a full run record. An operator config with an explicit `edit: deny` rule remains + enforced under `--auto`; transcript-owned completion does not require an agent-authored file. - **Shadow preflight:** the `opencode-skills` capability scans every root OpenCode discovers skills from and warns at build time when a staged logical skill is also live there — see "Isolating from live skills" below. diff --git a/docs/progressive-enhancements.md b/docs/progressive-enhancements.md index 0ac2fa6..c3e7cc8 100644 --- a/docs/progressive-enhancements.md +++ b/docs/progressive-enhancements.md @@ -7,11 +7,11 @@ > at a harness it doesn't know — user-supplied descriptor files, layering, `harness > list`/`show`/`lint` — is the user-facing [BYOH guide](guides/byoh.md). -Harness compatibility is not a parity checklist to audit — it is **a minimal baseline every harness -satisfies, plus optional enhancements** a harness's adapter opts into. Most missing enhancements -have a documented lower-fidelity fallback. Native conversation resume is the deliberate exception: -an eval that declares scripted `turns` or a `responder` is rejected when the harness cannot preserve -one session. +Harness compatibility is not a parity checklist to audit — it is **a runner-ready baseline every +harness satisfies, plus optional enhancements** a harness's adapter opts into. Dispatch and +transcript recovery are mandatory because the runner owns execution and run-record assembly. +Native conversation resume remains optional; an eval that declares scripted `turns` or a +`responder` is rejected when the harness cannot preserve one session. ## One dispatch mechanism @@ -27,12 +27,12 @@ runner-owned `command_check` subprocesses remain separate. ## The baseline contract -A harness qualifies at baseline with no harness-specific code beyond naming itself: +A harness qualifies at baseline when its descriptor provides: 1. **A headless exec command** — some way to invoke the harness with a prompt from a chosen cwd and let it run to completion. -2. **A recoverable final message** — the agent writes `outputs/final-message.md` (the dispatch - prompt asks for this), or transcript ingest recovers it where it is wired. +2. **A parseable transcript** — the command captures the harness's event stream under the task's + round output directory, and a named or declarative reader normalizes a non-empty final response. 3. **`--no-stage` when native staging isn't wired** — each `SKILL.md` is inlined into its dispatch prompt instead of staged for native discovery. @@ -41,18 +41,17 @@ runner-owned `command_check` assertions can inject held-out files and execute de runner-owned final-environment metrics land in `diff-scope.json` with the diff itself in `diff.patch`, `diff_scope` assertions gate files/lines deterministically, and the `detect-stray-writes` post-pass (folded into `ingest`) audits -writes that leave the private task environment. Run records without transcript ingest are assembled -from `outputs/final-message.md` or by hand per `schema/run-record.schema.json`. - -In descriptor terms the baseline is one required field: `label`. Everything else in a harness -descriptor is optional for one-shot evals — an absent field or table gets a working generic fallback, and the `run` -preflight *warns* naming that fallback rather than rejecting (a harness without `skills_dir` -forces `--no-stage`; without a declared guard the run continues unguarded behind the -`detect-stray-writes` audit; requested models without a model flag are recorded as provenance -only). Supported enhancements are provided automatically — the write guard auto-arms wherever a -harness declares one and staging is active (`--no-guard` opts out). Only genuinely contradictory -flag combinations stay errors. A selected eval with `turns` or a `responder` also requires -`[conversation]`; no generic fresh-session fallback can preserve the meaning of a follow-up reply. +writes that leave the private task environment. `record-runs` assembles every `run.json` from the +runner-owned dispatch metadata, completion artifact, and per-round transcripts. + +In descriptor terms the baseline is `label`, `[dispatch].exec_template`, `[transcript]` with one +primary reader, and `[tools]` beside that reader. `run` rejects a harness that cannot dispatch or +recover transcripts. A harness without `skills_dir` forces `--no-stage`; without a declared guard +the run continues unguarded behind the `detect-stray-writes` audit; requested models without a +model flag are recorded as provenance only. Supported enhancements are provided automatically — +the write guard auto-arms wherever a harness declares one and staging is active (`--no-guard` opts +out). A selected eval with `turns` or a `responder` also requires `[conversation]`; no generic +fresh-session fallback can preserve the meaning of a follow-up reply. ## Where this lives in code @@ -83,21 +82,19 @@ flag combinations stay errors. A selected eval with `turns` or a `responder` als - `run_capabilities()` (descriptor table `[run]`) + `harness_run_preflight()` (`src/cli/run/util.rs`) — the `run` preflight: it resolves the guard tri-state (auto-arm when the harness declares a guard and staging is active; `--guard`/`--no-guard` make it explicit), - and undeclared enhancements warn naming their fallback and adjust the options (guard forced - off, missing `skills_dir` forces `--no-stage`, the no-transcript-parser warning scoped to eval - configs that actually use `transcript_check`, missing dispatch commands noted); only + rejects descriptors without a dispatch command or transcript reader, and adjusts optional + capabilities (guard forced off, missing `skills_dir` forces `--no-stage`); only contradictory flag combinations (`--bootstrap`/`--stage-name` where the descriptor declares them incompatible with `--no-stage`) and an explicit `--guard` on a user-descriptor-only harness reject. ## Runner-owned environment checks are baseline -Every canonical `(eval, condition, run)` gets a distinct `eval_root`. After fixtures, staging, and -guard installation, `run` establishes a runner-owned Git repository at that root, commits the task -state, marks it with `refs/eval-magic/baseline`, and runs shadow preflight at the resulting -repository boundary. Git is therefore a runtime prerequisite; each task starts clean and has no -remotes. Nothing writes into an environment after the ref is written, so it names exactly what the -agent started from. +Every canonical `(eval, condition, run)` gets a distinct `eval_root` provisioned from its effective +codebase. After overlays, staging, and guard installation, `run` commits the task state, marks it +with `refs/eval-magic/baseline`, and runs shadow preflight at the resulting repository boundary. +Git is therefore a runtime prerequisite; each task starts clean and has no remotes. Nothing writes +into an environment after the ref is written, so it names exactly what the agent started from. During `ingest`, before any held-out setup is injected, Git measures the final environment against that ref. The runner seeds a scratch index from the baseline, brings it up to the working tree with @@ -112,7 +109,7 @@ may gate `max_files_touched`, `max_lines_changed` (added plus removed), or both. was built under: the codebase's own `.gitignore` holds, so a run that compiles does not report its build output as thousands of touched files, and the `.git/info/exclude` entry keeps framework artifacts under `.eval-magic-outputs/` out. Paths the runner force-added despite those rules — the -harness config directories and the declared fixture overlay — are tracked in the baseline and stay +harness config directories and the declared file overlay — are tracked in the baseline and stay measured. Git indexes no path with a `.git` component, so a nested repository's internals are invisible, not just the runner-owned root `.git`. Renames are switched off deliberately: a rename is two touched files, one created and one deleted, which is what the metric has always meant. A binary @@ -126,16 +123,16 @@ that gap — it is what a judge reads to answer whether the work was any good. dispatch so it can validate held-out sources before building. After diff-scope capture, `ingest` copies the assertion's held-out `setup_files` from the skill's `evals/` directory into that root and executes the trusted command through the platform shell. Root `.git` paths are reserved for both -visible and held-out fixtures, while nested repositories remain valid. The runner clears inherited -Git routing variables before optional `env` values override the environment; optional `matrix` -values execute every Cartesian-product cell and persist per-cell results. The files are never staged -or mentioned to the agent, and therefore never inflate scope metrics. +visible overlays and held-out setup files, while nested repositories remain valid. The runner clears +inherited Git routing variables before optional `env` values override the environment; optional +`matrix` values execute every Cartesian-product cell and persist per-cell results. The files are +never staged or mentioned to the agent, and therefore never inflate scope metrics. -This path needs no transcript parser, tool vocabulary, or model flag, so it behaves -the same for built-ins and descriptor-only harnesses. It also does not use harness tools: an armed -agent write guard can remain installed while the runner executes the command. `finalize` converts -the schema-gated intermediate result into an ordinary grading result, leaving aggregation -harness-agnostic. +Beyond the runner-ready dispatch and transcript baseline, this path needs no additional parser +signals, tool vocabulary, or model flag, so it behaves the same for built-ins and descriptor-only +harnesses. It also does not use harness tools: an armed agent write guard can remain installed while +the runner executes the command. `finalize` converts the schema-gated intermediate result into an +ordinary grading result, leaving aggregation harness-agnostic. ## The enhancements @@ -182,10 +179,10 @@ write guard denies through the same permission mechanism, so its blocks land in they are attributed by the `eval guard: ` reason prefix and excluded from the warning so one denial is not reported twice. -*Fallback:* `transcript_check` grades as *unverifiable*, `llm_judge` and runner-owned -`command_check` carry the grading (bias suites toward those for such a harness), tokens/duration go -unrecorded, records are assembled from `outputs/final-message.md` or by hand, and the meta-check -uses the LLM-judge fallback. +There is no no-transcript fallback for a runner-ready harness. A descriptor without a primary +transcript reader is rejected before the workspace is built. A valid reader may still omit optional +token, duration, denial, or skill-invocation evidence; those individual signals remain unavailable +or use their documented grading fallback. *Descriptor fields:* the `[transcript]` table — `events_filename` (gate: an absent table means the ingest pipeline never reads a transcript), one primary summary reader, and @@ -433,12 +430,11 @@ descriptor surface; a harness with a real native plan mode would grow one. *What it unlocks:* `eval-magic dispatch` itself. Without an `exec_template` there is nothing for the runner to run, and `dispatch` fails for that harness. -*Fallback:* none — this is the one enhancement dispatch cannot work around. The `run` preflight -warns at prep time when the descriptor declares no `exec_template` (`has_dispatch_recipes()`), so -the gap surfaces before a workspace is built. +*Fallback:* none. The `run` preflight rejects a descriptor without `exec_template`, so the gap +surfaces before a workspace is built. *Descriptor fields:* the `[dispatch]` table — `env`, `exec_template`, `next_steps_template`, -`manifest_template`, `capture_prefix`, `guard_args`, `model_note`. Templates carry +`manifest_template`, `guard_args`, `model_note`. Templates carry `{model_arg}`/`{guard_args}` slots the renderer fills for eval-agent dispatches; a judge dispatch reuses `exec_template` with `guard_args` deliberately empty, because judges run from the iteration directory outside every guarded task env. `env` contains non-secret eval-agent defaults, applied diff --git a/harnesses/claude-code.toml b/harnesses/claude-code.toml index 15562cf..cec0043 100644 --- a/harnesses/claude-code.toml +++ b/harnesses/claude-code.toml @@ -72,7 +72,6 @@ armed_message = ''' preflight = "claude-plugins" [dispatch] -capture_prefix = "claude" next_steps_template = ''' Next: eval-magic dispatch{target_args} --iteration {iteration} --harness claude-code diff --git a/harnesses/cline.toml b/harnesses/cline.toml index d170810..5dc91d8 100644 --- a/harnesses/cline.toml +++ b/harnesses/cline.toml @@ -114,16 +114,10 @@ preflight = "cline-skills" # (a hidden flag) pins act mode: the global `planActMode` setting otherwise # applies to headless dispatches and converts them to read-only plan sessions # that refuse writes. Piped stdin is treated as extra prompt context, so -# every recipe detaches it with --act --json --auto-approve true{model_arg} \ "Read the file at and follow its instructions exactly. When you finish, make your final response your closing summary." \ /cline-events.jsonl \ - 2> /cline-stderr.log; \ - jq -rj 'select(.type == "run_result") | .text' /cline-events.jsonl \ - > /final-message.md''' + 2> /cline-stderr.log''' manifest_template = ''' Harness dispatch (Cline): -`eval-magic dispatch` runs one fresh `cline --cwd --act --json --auto-approve true` per task. Detach stdin with `` so piped task data cannot become extra prompt context; capture stdout as `outputs/turn-/cline-events.jsonl` and stderr as `outputs/turn-/cline-stderr.log`. `eval-magic dispatch` writes `outputs/final-message.md` itself from the parsed transcript; the template's trailing jq step is a belt-and-braces copy of the terminal `run_result` event. +`eval-magic dispatch` runs one fresh `cline --cwd --act --json --auto-approve true` per task. Detach stdin with `` so piped task data cannot become extra prompt context; capture stdout as `outputs/turn-/cline-events.jsonl` and stderr as `outputs/turn-/cline-stderr.log`. Ingest recovers the final response directly from the terminal `run_result` event. ```bash {exec_command} diff --git a/harnesses/codex.toml b/harnesses/codex.toml index adf8630..6cd15aa 100644 --- a/harnesses/codex.toml +++ b/harnesses/codex.toml @@ -99,7 +99,6 @@ armed_message = ''' preflight = "codex-skills" [dispatch] -capture_prefix = "codex" guard_args = " --dangerously-bypass-hook-trust" next_steps_template = ''' @@ -109,8 +108,7 @@ Then run `eval-magic ingest{target_args} --iteration {iteration} --harness codex # prompt context. exec_template = ''' codex --ask-for-approval never exec --cd --sandbox workspace-write{guard_args}{model_arg} --json \ - --output-last-message /final-message.md \ - "Read the file at and follow its instructions exactly. When you finish, make your final response exactly the same text you wrote to /final-message.md." \ + "Read the file at and follow its instructions exactly. When you finish, make your final response your closing summary." \ /codex-events.jsonl \ 2> /codex-stderr.log''' @@ -130,7 +128,6 @@ Then run `eval-magic ingest --harness codex`; Codex transcript ingest reads each token_usage_aggregation = "last" resume_exec_template = ''' cd && codex --ask-for-approval never exec resume{guard_args}{model_arg} --json \ - --output-last-message /final-message.md \ {session_arg} {prompt_arg} \ /codex-events.jsonl \ diff --git a/harnesses/opencode.toml b/harnesses/opencode.toml index e2e4497..791fb9f 100644 --- a/harnesses/opencode.toml +++ b/harnesses/opencode.toml @@ -87,10 +87,8 @@ preflight = "opencode-skills" # Headless permission asks are auto-rejected unless `--auto` is passed # (explicit deny rules are still enforced), and piped stdin is *appended* to # the message, so every recipe detaches stdin with , , are prose placeholders whoever ## dispatches fills per task (the runbook explains each). ## Two hard requirements: the command runs from the task's (the env dir is the -## subprocess cwd), and the agent's final reply must land in /final-message.md — -## if the CLI cannot write it directly, capture it like the redirect below. -## VERIFY: run the command once by hand with a trivial prompt and confirm the final message -## actually lands in the redirect before declaring this. Multi-line templates are fine in TOML +## subprocess cwd), and it captures the native event stream at the [transcript].events_filename +## path under . VERIFY with `eval-magic harness lint --probe`; the probe parses that +## capture and requires a non-empty final response. Multi-line templates are fine in TOML ## '''...''' strings — see `eval-magic harness show claude-code`. # [dispatch] -# exec_template = '{label} run --cd {model_arg} "Read the file at and follow its instructions exactly." > /final-message.md' - -## capture_prefix names this harness's transcript file (-events.jsonl), which -## `eval-magic dispatch` reads back after each round. -# capture_prefix = "{label}" +# exec_template = '{label} run --cd {model_arg} "Read the file at and follow its instructions exactly." > /{label}-events.jsonl' ## next_steps_template is the post-run handoff text ({exec_command}, {target_args}, {iteration}, ## {model_note} placeholders; referencing {model_note} requires the model_note field). @@ -242,10 +237,11 @@ label = "{label}" ## Before opening a PR (`eval-magic docs byoh` "Upstreaming your descriptor"): ## 1. eval-magic harness lint .eval-magic/harnesses/{label}.toml — all checks pass. ## 2. A real smoke eval: eval-magic run --harness {label}, dispatch the tasks, then ingest and -## finalize — the preflight warnings name the fallback carrying anything not declared. +## finalize — the preflight warnings name optional capabilities using a fallback. ## For the exec template specifically, `eval-magic harness lint .eval-magic/harnesses/{label}.toml -## --probe --yes` renders it with a trivial prompt and verifies final-message recovery — opt-in, -## invokes the real harness CLI, never part of standard CI. See `harness lint --help`. +## --probe --yes` renders it with a trivial prompt, parses the captured transcript, and verifies +## final-response recovery — opt-in, invokes the real harness CLI, never part of standard CI. +## See `harness lint --help`. ## 3. .eval-magic/harnesses/{label}-notes.md records every verified value, its source, and the ## harness version. ## 4. Open the PR with .github/PULL_REQUEST_TEMPLATE/harness-descriptor.md — a data-only PR: diff --git a/schema/benchmark.schema.json b/schema/benchmark.schema.json index 6f318ed..18230da 100644 --- a/schema/benchmark.schema.json +++ b/schema/benchmark.schema.json @@ -55,7 +55,7 @@ }, "codebases": { "type": "array", - "description": "Codebases the compared conditions ran against, echoed from conditions.json. Absent for fixture-only iterations.", + "description": "Codebases the compared conditions ran against, echoed from conditions.json. Absent only in historical benchmarks that predate codebase provenance.", "items": { "$ref": "#/definitions/codebaseUse" } }, "skill_source": { diff --git a/schema/conversation.schema.json b/schema/conversation.schema.json index 9f82a1e..619c190 100644 --- a/schema/conversation.schema.json +++ b/schema/conversation.schema.json @@ -2,7 +2,7 @@ "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://slow-powers.dev/schemas/conversation.schema.json", "title": "Task Conversation Completion", - "description": "Runner-owned completion artifact for one dispatched eval task.", + "description": "Runner-owned completion artifact for one dispatched eval task. Canonical artifacts record the delivered user turns; assistant and tool events remain accepted for compatibility with historical artifacts.", "type": "object", "required": ["status", "delivered_followups", "events"], "additionalProperties": false, @@ -31,6 +31,7 @@ "events": { "type": "array", "minItems": 1, + "description": "Delivered conversation turns. Canonical runner output contains user_message events only; historical artifacts may also contain assistant_message and tool_invocation events.", "items": { "oneOf": [ { "$ref": "#/definitions/userMessage" }, @@ -81,13 +82,6 @@ ] } } - }, - { - "if": { - "properties": { "status": { "enum": ["completed", "stopped"] } }, - "required": ["status"] - }, - "then": { "properties": { "events": { "minItems": 2 } } } } ], "definitions": { diff --git a/schema/evals.schema.json b/schema/evals.schema.json index 56eae83..ca73b7f 100644 --- a/schema/evals.schema.json +++ b/schema/evals.schema.json @@ -33,6 +33,16 @@ "items": { "$ref": "#/definitions/eval" } } }, + "anyOf": [ + { "required": ["codebase"] }, + { + "properties": { + "evals": { + "items": { "required": ["codebase"] } + } + } + } + ], "definitions": { "codebase": { "oneOf": [ @@ -48,7 +58,7 @@ "path": { "type": "string", "minLength": 1, - "description": "Directory on this host to build the task environment from, resolved relative to this evals.json when relative. Unlike files_root it may be absolute or escape the skill tree, because it deliberately points outside it. A path source is host-local: another machine has the directory elsewhere or not at all, so a run recorded against one is not reproducible from this config alone. When the directory is a Git repository the runner also records its origin URL and resolved SHA, which are." + "description": "Directory on this host to build the task environment from, resolved relative to this evals.json when relative. Unlike files_root it may be absolute or escape the skill tree, because it deliberately points outside it. A path source is host-local: another machine has the directory elsewhere or not at all, so a run recorded against one is not reproducible from this config alone. When the directory is a Git repository, the runner also records its origin URL and resolved SHA so another machine can resolve the same tree." }, "exclude_skill_sources": { "type": "boolean", @@ -112,12 +122,12 @@ "files": { "type": "array", "items": { "type": "string" }, - "description": "Fixture destination paths relative to the task repository. Sources use the same paths beneath the skill's evals/ directory, or beneath files_root when set. Copied before dispatch. The normalized first component may not be .git (case-insensitive) because root Git metadata is runner-owned; nested .git paths are allowed." + "description": "Codebase overlay destination paths relative to the task repository. Sources use the same paths beneath the skill's evals/ directory, or beneath files_root when set. Applied after the codebase and condition skills are staged. The normalized first component may not be .git (case-insensitive) because root Git metadata is runner-owned; nested .git paths are allowed." }, "files_root": { "type": "string", "minLength": 1, - "description": "Optional source directory relative to the skill's evals/ directory. When set, files are resolved beneath this directory but retain their declared paths in the task repository. Must be relative and may not contain a parent-directory component." + "description": "Optional source directory for codebase overlay files, relative to the skill's evals/ directory. Files resolve beneath this directory but retain their declared paths in the task repository. Must be relative and may not contain a parent-directory component." }, "runs": { "type": "integer", @@ -132,12 +142,6 @@ "$ref": "#/definitions/guardPolicy", "description": "Shell-command policy for this eval. Presence replaces the config-level guard policy and disables automatic profile detection." }, - "isolation": { - "type": "string", - "enum": ["shared", "isolated"], - "default": "shared", - "description": "Legacy isolation hint retained for config compatibility. Canonical runs already give every (eval, condition, run) dispatch a private environment for diff-scope capture, so 'shared' and 'isolated' currently have the same effective isolation." - }, "skill_should_trigger": { "type": "boolean", "default": true, diff --git a/schema/harness-descriptor.schema.json b/schema/harness-descriptor.schema.json index f2135ce..885b5a1 100644 --- a/schema/harness-descriptor.schema.json +++ b/schema/harness-descriptor.schema.json @@ -355,11 +355,6 @@ }, "additionalProperties": { "type": "string" } }, - "capture_prefix": { - "type": "string", - "minLength": 1, - "description": "Prefix naming the judge recipe's per-task capture files ($response_base.-events.jsonl)." - }, "guard_args": { "type": "string", "description": "Extra args spliced into command templates at {guard_args} when the run is guarded (leading space included)." diff --git a/schema/run-record.schema.json b/schema/run-record.schema.json index 441540c..27691b7 100644 --- a/schema/run-record.schema.json +++ b/schema/run-record.schema.json @@ -92,7 +92,7 @@ }, "codebase": { "$ref": "#/definitions/codebase", - "description": "The codebase this run's environment was built from. Absent for a fixture-only run." + "description": "The codebase this run's environment was built from. Absent only in historical records that predate codebase provenance." }, "skill_source": { "$ref": "#/definitions/skillSource", diff --git a/src/adapters/claude_code/transcript.rs b/src/adapters/claude_code/transcript.rs index 462b241..38a6acb 100644 --- a/src/adapters/claude_code/transcript.rs +++ b/src/adapters/claude_code/transcript.rs @@ -185,7 +185,7 @@ pub(crate) fn extract_events(records: &[TranscriptRecord]) -> Vec Option { let mut final_text: Option = None; diff --git a/src/adapters/descriptor.rs b/src/adapters/descriptor.rs index 1af679f..21c5e01 100644 --- a/src/adapters/descriptor.rs +++ b/src/adapters/descriptor.rs @@ -270,8 +270,6 @@ pub struct DispatchSection { #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub env: BTreeMap, #[serde(skip_serializing_if = "Option::is_none")] - pub capture_prefix: Option, - #[serde(skip_serializing_if = "Option::is_none")] pub guard_args: Option, #[serde(skip_serializing_if = "Option::is_none")] pub model_note: Option, @@ -304,7 +302,6 @@ impl DispatchSection { /// True when no dispatch field is set. pub fn is_empty(&self) -> bool { self.env.is_empty() - && self.capture_prefix.is_none() && self.guard_args.is_none() && self.model_note.is_none() && self.next_steps_template.is_none() @@ -842,6 +839,15 @@ timestamp_spread = "timestamp" assert!(err.contains("harness-descriptor schema"), "{err}"); } + #[test] + fn rejects_the_retired_capture_prefix_field() { + let err = err_of(&format!( + "{MINIMAL}\n[dispatch]\ncapture_prefix = \"demo\"\n" + )); + assert!(err.contains("capture_prefix"), "{err}"); + assert!(err.contains("harness-descriptor schema"), "{err}"); + } + #[test] fn merge_deep_merges_tables_and_replaces_scalars_and_arrays() { let mut base: serde_json::Value = toml::from_str( @@ -853,7 +859,7 @@ config_dirs = [".demo"] flag = "--model" [dispatch] -capture_prefix = "demo" +exec_template = "demo run" [dispatch.env] KEEP = "base" @@ -879,7 +885,7 @@ TZ = "America/Los_Angeles" // Nested-table scalar replaced; sibling table untouched; array replaced // wholesale (no element-wise merge). assert_eq!(base["model"]["flag"], "--model-x"); - assert_eq!(base["dispatch"]["capture_prefix"], "demo"); + assert_eq!(base["dispatch"]["exec_template"], "demo run"); assert_eq!(base["dispatch"]["env"]["KEEP"], "base"); assert_eq!(base["dispatch"]["env"]["ADDED"], "overlay"); assert_eq!( diff --git a/src/adapters/descriptor/validation/tests/transcript.rs b/src/adapters/descriptor/validation/tests/transcript.rs index fc3a6e1..a6e8d90 100644 --- a/src/adapters/descriptor/validation/tests/transcript.rs +++ b/src/adapters/descriptor/validation/tests/transcript.rs @@ -132,6 +132,16 @@ fn rejects_empty_extract_block() { assert!(err.contains("at least one"), "{err}"); } +#[test] +fn rejects_declarative_transcript_without_assistant_or_final_text() { + let err = err_of(&format!( + "{TOOLED}\n[transcript]\nevents_filename = \"demo-events.jsonl\"\n\n\ + [transcript.extract.tools]\nname_field = \"name\"\n" + )); + assert!(err.contains("final response"), "{err}"); + assert!(err.contains("final_text"), "{err}"); +} + #[test] fn rejects_duration_with_both_variants_or_neither() { for duration_block in [ diff --git a/src/adapters/descriptor/validation/transcript.rs b/src/adapters/descriptor/validation/transcript.rs index 70862f1..f25505b 100644 --- a/src/adapters/descriptor/validation/transcript.rs +++ b/src/adapters/descriptor/validation/transcript.rs @@ -109,6 +109,17 @@ pub(super) fn check_tiers(d: &HarnessDescriptor) -> Result<(), String> { .into(), ); } + if transcript.parser.is_none() + && extract.final_text.is_none() + && extract.assistant_messages.is_none() + { + return Err( + "[transcript.extract] cannot recover the agent's final response; declare \ + final_text or assistant_messages so record-runs has transcript-owned text to \ + grade" + .into(), + ); + } } Ok(()) } diff --git a/src/adapters/descriptor_adapter.rs b/src/adapters/descriptor_adapter.rs index 8ebbb44..6636572 100644 --- a/src/adapters/descriptor_adapter.rs +++ b/src/adapters/descriptor_adapter.rs @@ -420,9 +420,9 @@ impl HarnessAdapter for DescriptorAdapter { ctx.target_args, ctx.iteration, self.descriptor.label ); let Some(template) = &self.descriptor.dispatch.next_steps_template else { - // Generic fallbacks: a descriptor with just an exec template still - // earns a copy-pasteable recipe; a baseline descriptor gets the - // harness-agnostic handoff. + // Generic fallbacks: a runner-ready descriptor with an exec + // template still earns a copy-pasteable recipe. The no-template + // branch is diagnostic only because run preflight rejects it. return match &self.descriptor.dispatch.exec_template { Some(_) => format!( "\nNext: iterate the tasks[] array in dispatch.json and dispatch each task \ @@ -430,9 +430,8 @@ impl HarnessAdapter for DescriptorAdapter { self.render_exec_command(ctx.guard, ctx.agent_model, ctx.agent_env) ), None => format!( - "\nNext: read dispatch-manifest.md and dispatch each task through your \ - harness's one-shot CLI from the task's eval_root, saving the agent's \ - final reply to outputs/final-message.md.\nThen run `{ingest_line}`." + "\nThis descriptor is not runner-ready: add `[dispatch].exec_template` and \ + a `[transcript]` parser before running evals.\nThen run `{ingest_line}`." ), }; }; @@ -464,9 +463,8 @@ impl HarnessAdapter for DescriptorAdapter { return Some( format!( "## Dispatch recipe\n\nFrom each task's `eval_root`, dispatch with:\n\ - {exec_command}\n\nEnsure the agent's final reply lands in the task's \ - `outputs/final-message.md` (capture it yourself if the command does not \ - write it).\n" + {exec_command}\n\nThe command must capture the configured transcript under \ + `outputs/turn-/`; ingest recovers the final response from that stream.\n" ) .split('\n') .map(String::from) @@ -540,7 +538,7 @@ mod tests { .expect("an exec template earns a generic manifest recipe") .join("\n"); assert!(manifest.contains("cool-cli run"), "{manifest}"); - assert!(manifest.contains("final-message.md"), "{manifest}"); + assert!(manifest.contains("configured transcript"), "{manifest}"); } #[test] @@ -639,8 +637,8 @@ mod tests { agent_model: None, agent_env: empty_env(), }); - assert!(next.contains("one-shot CLI"), "{next}"); - assert!(next.contains("outputs/final-message.md"), "{next}"); + assert!(next.contains("not runner-ready"), "{next}"); + assert!(next.contains("[transcript]"), "{next}"); assert!( next.contains("ingest --skill x --iteration 1 --harness cool-custom-harness"), "{next}" diff --git a/src/adapters/extract.rs b/src/adapters/extract.rs index 9026d9c..97e8aa0 100644 --- a/src/adapters/extract.rs +++ b/src/adapters/extract.rs @@ -151,7 +151,12 @@ pub(crate) fn parse_full(spec: &ExtractSpec, path: &Path) -> io::Result final-message.md" } + "tool_input": { "command": "printf '%s\n' done > notes.md" } }"#; assert_eq!(verdict("codex", payload, Some(marker())), None); } @@ -221,14 +221,14 @@ fn codex_redirection_scanner_allows_literal_targets_inside_hook_cwd() { "printf done > /work/.eval-magic/absolute.txt", "printf done > fixtures/output.txt", "printf done > ./fixtures/output.txt", - "printf done > .eval-magic-outputs/final-message.md", - "printf done > final-message.md", + "printf done > .eval-magic-outputs/harness-events.jsonl", + "printf done > notes.md", "printf done 2>>\"quoted path.log\"", "printf done >| overwritten.txt", "printf done > stdout.txt 2>> stderr.txt", "printf done | tee \"quoted path.log\"", "printf done | tee -a one.txt two.txt", - "printf done | sudo tee final-message.md", + "printf done | sudo tee notes.md", ] { assert_eq!( codex_bash_verdict(command), @@ -247,8 +247,8 @@ fn codex_redirection_scanner_allows_file_descriptor_duplication() { "printf done >&2", "printf done 1>&2", "printf done >&-", - "printf done > final-message.md 2>&1", - "printf done | tee final-message.md 2>&1", + "printf done > notes.md 2>&1", + "printf done | tee notes.md 2>&1", ] { assert_eq!( codex_bash_verdict(command), @@ -351,7 +351,7 @@ fn guard_denials_append_privacy_safe_sorted_metadata_and_allows_do_not_log() { "hook_event_name": "PreToolUse", "cwd": eval_root, "tool_name": "Bash", - "tool_input": { "command": "printf done > final-message.md" }, + "tool_input": { "command": "printf done > notes.md" }, }); assert_eq!( verdict("codex", &allowed.to_string(), Some(marker.clone())), diff --git a/src/adapters/harness.rs b/src/adapters/harness.rs index 2768145..2f347ef 100644 --- a/src/adapters/harness.rs +++ b/src/adapters/harness.rs @@ -1,20 +1,15 @@ //! The harness adapter API — the single seam between generic dispatch code and //! harness-specific behavior. //! -//! The trait is tiered into a **baseline** every harness must implement and -//! **enhancements** that raise fidelity when a harness has the native support: +//! The trait is tiered into a **runner requirement** every selected harness must +//! satisfy and **enhancements** that raise fidelity when native support exists: //! -//! - **Baseline (required):** [`label`](HarnessAdapter::label) and -//! [`skills_dir`](HarnessAdapter::skills_dir). A new harness compiles with -//! just these two methods; dispatched through its one-shot CLI (with -//! `--no-stage` inlining the skill when native staging isn't wired), it -//! already supports `llm_judge` grading and the `detect-stray-writes` -//! post-pass. -//! - **Enhancements (defaulted):** every other method has a default — either a -//! working generic fallback (e.g. the plain available-skills block) or an -//! `Unsupported` error naming the enhancement it belongs to (e.g. transcript -//! ingest, the write guard). Override the methods of an enhancement to wire -//! it for a harness. +//! - **Runner requirement:** the adapter identifies itself and exposes a +//! dispatch command plus a transcript event filename/reader. `run` rejects a +//! selected harness missing either execution or transcript recovery. +//! - **Enhancements (defaulted):** native staging, guards, model flags, +//! conversations, shadow scans, and richer transcript signals may fall back +//! or reject only the evals/options that require them. //! //! Generic code resolves an adapter with [`adapter_for`](super::registry::adapter_for) //! and then calls the trait — so the [`registry`](super::registry) is the one @@ -197,16 +192,13 @@ pub trait HarnessAdapter { "If the staged skill cannot be resolved".to_string() } - // ── Enhancement: transcript ingest (defaulted) ─────────────────────────── - // Fallback without it: `transcript_check` assertions grade as - // unverifiable, `llm_judge` carries the grading, token/cost/duration go - // unrecorded, and run records are assembled by hand (or from - // `outputs/final-message.md`) instead of auto-ingested. + // ── Runner requirement: transcript ingest (defaulted) ─────────────────── + // Run preflight rejects a harness without this capability. The default + // remains unwired so a partial descriptor can still be linted and shown. - /// **Enhancement: transcript ingest.** The filename (under a task's - /// `outputs/` dir) this harness's one-shot CLI writes the captured - /// transcript to. `None` when no transcript ingest is wired — the ingest - /// pipeline then never calls the readers below. + /// **Runner requirement: transcript ingest.** The filename (under a task's + /// `outputs/turn-N/` dir) this harness's CLI writes the captured transcript + /// to. `None` means the descriptor is not runner-ready. fn cli_events_filename(&self) -> Option { None } @@ -406,14 +398,14 @@ pub trait HarnessAdapter { format!("\n{trimmed}\n") } - // ── Enhancement: dispatch commands (defaulted) ─────────────────────────── + // ── Runner requirement: dispatch commands (defaulted) ──────────────────── // There is no fallback: without an exec template the runner has nothing to - // spawn, so `dispatch` fails for that harness and `run` warns at prep time. + // spawn, so `run` rejects that harness during preflight. - /// **Enhancement: dispatch commands.** Whether a per-task exec command is + /// **Runner requirement: dispatch commands.** Whether a per-task exec command is /// wired (the descriptor's `[dispatch] exec_template`). `false` means /// `eval-magic dispatch` has nothing to run for this harness, and the `run` - /// preflight warns naming that. + /// preflight rejects it. fn has_dispatch_recipes(&self) -> bool { false } diff --git a/src/cli/args.rs b/src/cli/args.rs index c0544cb..1bd7bd7 100644 --- a/src/cli/args.rs +++ b/src/cli/args.rs @@ -214,8 +214,8 @@ pub(crate) enum HarnessCommands { /// With `--probe`, and only after every static check passes, also exercises /// the descriptor end-to-end: renders `dispatch.exec_template` with a /// trivial prompt in a throwaway temp dir, runs it via `sh -c` from - /// the temp `eval_root`, and verifies `outputs/final-message.md` is - /// recovered (non-empty). + /// the temp `eval_root`, parses the configured transcript capture, and + /// verifies it contains a non-empty final response. /// /// `--probe` invokes the real harness CLI (network, tokens, usage /// limits), so it is opt-in and never runs as part of standard CI checks @@ -239,7 +239,7 @@ pub(crate) enum HarnessCommands { #[arg(long, requires = "target")] as_builtin: bool, /// Execute the dispatch exec template with a trivial prompt and verify - /// final-message recovery (opt-in; costs real CLI usage). See the + /// transcript-owned final-response recovery (opt-in; costs real CLI usage). See the /// subcommand description above for the full contract. #[arg(long)] probe: bool, @@ -642,8 +642,8 @@ pub(crate) enum Commands { TeardownGuard(CommonArgs), /// Ingest recorded transcripts into run records. /// - /// Fixed-order chain: record-runs → fill-transcripts → detect-stray-writes → - /// grade. Assembles each task's `run.json` + `timing.json`, scans for stray + /// Fixed-order chain: record-runs → detect-stray-writes → grade. Assembles + /// each task's `run.json` + `timing.json`, scans for stray /// writes, and maps raw per-env guard logs through `dispatch.json` into /// `guard-denials.json` (including tasks without `run.json`). Malformed raw /// records fail with their source path and line number. It measures the @@ -684,14 +684,15 @@ pub(crate) enum Commands { /// Assemble run records from a dispatch and its transcripts. /// /// Assembles a schema-valid `run.json` and backfills `timing.json` for every - /// task in a runner-built iteration, from `dispatch.json` + - /// `outputs/final-message.md` + each task's `outputs/-events.jsonl`. + /// task in a runner-built iteration, from `dispatch.json`, + /// `conversation.json`, and each task's + /// `outputs/turn-/-events.jsonl`. /// Never clobbers existing records without `--overwrite`; transcript-derived /// timing carries `"source": "transcript"`. Use `--overwrite` to regenerate /// records and timing after extractor accounting changes. Folded into `ingest`. /// - /// For harnesses whose captures identify a refused tool call (Claude Code - /// and Codex today), it also writes `permission-denials.json` and warns on + /// For harnesses whose captures identify a refused tool call, it also writes + /// `permission-denials.json` and warns on /// stderr: the dispatch can exit 0 either way, so a run the harness refused /// — and which therefore fell back to static reasoning — is otherwise /// invisible. `aggregate` lifts one validity warning per affected task from @@ -699,7 +700,7 @@ pub(crate) enum Commands { /// so its absence never reads as "nothing was refused". /// /// For harnesses whose captures report the session's discoverable skills and - /// plugins (Claude Code today), it also writes `session-surface.json` — one + /// plugins, it also writes `session-surface.json` — one /// entry per dispatch and per resumed turn — and uses it to resolve the /// build-time shadow preflight's findings, writing `resolved_severity` back /// into `plugin-shadow.json`. A finding refuted in every expected cell @@ -709,13 +710,6 @@ pub(crate) enum Commands { /// surface, so its absence never reads as "nothing loaded". See /// `eval-magic docs isolation`. RecordRuns(CommonArgs), - /// Populate tool invocations from persisted transcripts. - /// - /// Reads each task's `outputs/-events.jsonl` and populates - /// `tool_invocations` in `run.json`. Subsumed by `record-runs` for - /// runner-built iterations; still the tool for filling a pre-existing (hand- or - /// agent-written) `run.json`. - FillTranscripts(CommonArgs), /// Detect writes outside each private task environment. /// /// Scans each run's `tool_invocations` and writes `stray-writes.json`: write @@ -813,8 +807,9 @@ pub(crate) enum Commands { /// /// Extend the seed in `evals/evals.json`: `turns` scripts same-session /// follow-ups and `responder` derives them instead (see - /// `eval-magic docs conversations`), `files_root` mounts fixture sources at - /// the task root, and a per-eval `runs` value overrides `run --runs`. Add + /// `eval-magic docs conversations`), `files_root` resolves overlay sources + /// applied at the codebase root, and a per-eval `runs` value overrides + /// `run --runs`. Add /// assertions after the first iteration, then check the file with /// `eval-magic validate`. Init(InitArgs), diff --git a/src/cli/commands/harness/probe.rs b/src/cli/commands/harness/probe.rs index 9ea1079..cbac61f 100644 --- a/src/cli/commands/harness/probe.rs +++ b/src/cli/commands/harness/probe.rs @@ -1,7 +1,8 @@ //! The `harness lint --probe` live dispatch check: render //! `dispatch.exec_template` with a trivial prompt in a throwaway temp dir, -//! execute it, and verify `outputs/final-message.md` is recovered. Invokes the -//! real harness CLI, so it is opt-in and never part of standard CI checks. +//! execute it, and verify its configured transcript parser recovers a final +//! response. Invokes the real harness CLI, so it is opt-in and never part of +//! standard CI checks. use std::collections::BTreeMap; use std::io::{self, BufRead, Write}; @@ -48,19 +49,21 @@ pub(crate) enum ProbeError { ExecFailed(ExitStatus), #[error("exec template timed out after {0:?}")] Timeout(Duration), - #[error("outputs/final-message.md is missing")] - FinalMessageMissing, - #[error("outputs/final-message.md is empty")] - FinalMessageEmpty, + #[error("the descriptor declares no transcript parser")] + TranscriptUnavailable, + #[error("{0} is missing")] + TranscriptMissing(String), + #[error("{path} could not be parsed: {message}")] + TranscriptUnreadable { path: String, message: String }, + #[error("{0} contains no non-empty final response")] + FinalResponseMissing(String), } /// Render the exec template with the angle placeholders (``, /// ``, ``, ``) shell-quoted and the /// machine placeholders (`{model_arg}`, `{guard_args}`) filled. Mirrors the -/// conversation driver at `src/cli/run/conversation.rs:317` — single -/// left-to-right pass, unknown braces pass through verbatim. `` is -/// intentionally not substituted: the probe is single-shot (turn 1 only), so -/// the exec template never sees rounds and any `` survives verbatim. +/// conversation driver's single left-to-right pass; unknown braces pass +/// through verbatim. The probe is a single turn, so `` resolves to `1`. #[allow(clippy::needless_pass_by_value)] fn render_probe_exec( template: &str, @@ -77,21 +80,37 @@ fn render_probe_exec( &template .replace("", "ed_eval_root) .replace("", "ed_prompt_path) - .replace("", "ed_outputs_dir), + .replace("", "ed_outputs_dir) + .replace("", "1"), &[("model_arg", model_arg), ("guard_args", guard_args)], ) } -/// Verify the final-message recovery contract: `outputs_dir/final-message.md` -/// exists and is non-empty after trimming. -fn verify_final_message(outputs_dir: &Path) -> Result<(), ProbeError> { - let path = outputs_dir.join("final-message.md"); +/// Verify the runner-readiness contract: the configured events file exists, +/// parses successfully, and yields a non-empty final response. +fn verify_transcript(descriptor: &HarnessDescriptor, outputs_dir: &Path) -> Result<(), ProbeError> { + let transcript = descriptor + .transcript + .as_ref() + .ok_or(ProbeError::TranscriptUnavailable)?; + let path = outputs_dir.join(&transcript.events_filename); + let display = path.display().to_string(); if !path.exists() { - return Err(ProbeError::FinalMessageMissing); + return Err(ProbeError::TranscriptMissing(display)); } - let contents = std::fs::read_to_string(&path).map_err(|_| ProbeError::FinalMessageMissing)?; - if contents.trim().is_empty() { - return Err(ProbeError::FinalMessageEmpty); + let summary = + transcript + .parse_full(&path) + .map_err(|error| ProbeError::TranscriptUnreadable { + path: display.clone(), + message: error.to_string(), + })?; + if summary + .final_text + .as_deref() + .is_none_or(|text| text.trim().is_empty()) + { + return Err(ProbeError::FinalResponseMissing(display)); } Ok(()) } @@ -170,8 +189,10 @@ pub(crate) fn run_probe( .map_err(ProbeError::SpawnFailed); match probed { Ok(ShellOutcome::Exited(status)) if status.success() => { - match verify_final_message(&outputs_dir) { - Ok(()) => println!("✓ live exec template: final-message recovered"), + match verify_transcript(&descriptor, &outputs_dir) { + Ok(()) => { + println!("✓ live exec template: transcript final response recovered") + } Err(e) => { eprintln!("✗ {e}"); failed += 1; @@ -201,7 +222,6 @@ pub(crate) fn run_probe( #[cfg(test)] mod tests { use super::*; - use std::fs; use std::path::PathBuf; fn dir(p: &str) -> PathBuf { @@ -224,9 +244,7 @@ mod tests { assert!(rendered.contains("--root '/path with space/eval'")); assert!(rendered.contains("--prompt '/path with space/probe-prompt.md'")); assert!(rendered.contains("--out /var/tmp/out")); - // The probe is single-shot — `` is not a probe placeholder and - // must survive verbatim (the exec template never sees rounds). - assert!(rendered.contains("--round ")); + assert!(rendered.contains("--round 1")); // Machine placeholders substituted in place. assert!(rendered.contains("--model-X gpt-x")); assert!(rendered.contains("--guard on")); @@ -245,29 +263,4 @@ mod tests { assert!(rendered.contains("'echo ${JOBS:-4} m'")); assert!(!rendered.contains("{model_arg}")); } - - #[test] - fn verify_final_message_accepts_a_non_empty_file() { - let tmp = tempfile::TempDir::new().unwrap(); - fs::write(tmp.path().join("final-message.md"), "ok\n").unwrap(); - verify_final_message(tmp.path()).expect("non-empty file should pass"); - } - - #[test] - fn verify_final_message_rejects_a_missing_file() { - let tmp = tempfile::TempDir::new().unwrap(); - let err = verify_final_message(tmp.path()).expect_err("missing should fail"); - assert!( - matches!(err, ProbeError::FinalMessageMissing), - "got {err:?}" - ); - } - - #[test] - fn verify_final_message_rejects_a_blank_file() { - let tmp = tempfile::TempDir::new().unwrap(); - fs::write(tmp.path().join("final-message.md"), " \n\t \n").unwrap(); - let err = verify_final_message(tmp.path()).expect_err("blank should fail"); - assert!(matches!(err, ProbeError::FinalMessageEmpty), "got {err:?}"); - } } diff --git a/src/cli/commands/init.rs b/src/cli/commands/init.rs index 94c10d2..ba3d191 100644 --- a/src/cli/commands/init.rs +++ b/src/cli/commands/init.rs @@ -65,10 +65,8 @@ pub(crate) fn run_init(args: InitArgs) -> anyhow::Result<()> { println!(); println!("Next:"); let target_args = command_target_args(&ctx); - println!(" eval-magic run{} --guard", target_args); - println!(" eval-magic ingest{}", target_args); - println!(" eval-magic finalize{}", target_args); - println!(" eval-magic promote-baseline{}", target_args); + println!(" eval-magic run{}", target_args); + println!("Then follow the generated RUNBOOK.md."); Ok(()) } diff --git a/src/cli/commands/mod.rs b/src/cli/commands/mod.rs index 1425db5..92ce7b7 100644 --- a/src/cli/commands/mod.rs +++ b/src/cli/commands/mod.rs @@ -22,8 +22,7 @@ pub(crate) use guard::{run_guard, run_guard_codex, run_guard_hook, run_teardown_ pub(crate) use harness::run_harness; pub(crate) use init::run_init; pub(crate) use pipeline::{ - run_aggregate, run_detect_stray_writes, run_fill_transcripts, run_finalize, run_grade, - run_ingest, run_record_runs, + run_aggregate, run_detect_stray_writes, run_finalize, run_grade, run_ingest, run_record_runs, }; pub(crate) use run::{run_dispatch, run_run}; pub(crate) use validate::run_validate; diff --git a/src/cli/commands/pipeline.rs b/src/cli/commands/pipeline.rs index 50d4eca..8098283 100644 --- a/src/cli/commands/pipeline.rs +++ b/src/cli/commands/pipeline.rs @@ -1,6 +1,6 @@ //! The post-dispatch / post-judge pipeline command handlers: the `ingest` and //! `finalize` chains and each individual stage (`record-runs`, -//! `fill-transcripts`, `detect-stray-writes`, `grade`, `aggregate`). +//! `detect-stray-writes`, `grade`, `aggregate`). use anyhow::bail; @@ -44,7 +44,6 @@ fn run_step(step: &run::steps::StepCommand) -> anyhow::Result<()> { }; let result = match step.kind { StepKind::RecordRuns => run_record_runs(common), - StepKind::FillTranscripts => run_fill_transcripts(common), StepKind::DetectStrayWrites => run_detect_stray_writes(common), StepKind::Grade { finalize } => run_grade(GradeArgs { common, finalize }), StepKind::Aggregate => run_aggregate(common), @@ -55,22 +54,12 @@ fn run_step(step: &run::steps::StepCommand) -> anyhow::Result<()> { result } -/// Run the post-dispatch chain (record-runs → fill-transcripts → -/// detect-stray-writes → grade) and stop at the judge hand-off. +/// Run the post-dispatch chain (record-runs → detect-stray-writes → grade) and +/// stop at the judge hand-off. pub(crate) fn run_ingest(args: CommonArgs) -> anyhow::Result<()> { let ctx = run_context_from(&args)?; let iteration = resolve_iteration(&ctx, args.iteration)?; - let adapter = crate::adapters::adapter_for(ctx.harness); - if adapter.cli_events_filename().is_none() { - eprintln!( - "ℹ --harness {}: no transcript parser — records come from outputs/final-message.md \ - only; steps/tokens/duration go unrecorded and transcript_check assertions grade \ - as unverifiable (llm_judge carries the grading).", - adapter.label() - ); - } - let steps = run::steps::build_ingest_commands(&run::steps::StepParams { skill_dir: args.skill_dir.as_deref(), skill: args.skill.as_deref(), @@ -154,10 +143,10 @@ pub(crate) fn run_record_runs(args: CommonArgs) -> anyhow::Result<()> { let result = pipeline::record_runs(&dir, iteration, ctx.harness, args.overwrite)?; println!( - "\nRecorded: {}, skipped (existing run.json): {}, skipped (no final message): {}, skipped (prompt unread): {}, skipped (incomplete conversation): {}, missing transcript: {}", + "\nRecorded: {}, skipped (existing run.json): {}, skipped (no final response): {}, skipped (prompt unread): {}, skipped (missing completion artifact): {}, missing transcript: {}", result.recorded, result.skipped_existing, - result.skipped_no_final_message, + result.skipped_no_final_response, result.skipped_prompt_unread, result.skipped_incomplete_conversation, result.missing_transcript @@ -177,20 +166,6 @@ pub(crate) fn run_record_runs(args: CommonArgs) -> anyhow::Result<()> { Ok(()) } -/// Populate `tool_invocations` from persisted transcripts for every `run.json` in -/// the iteration. -pub(crate) fn run_fill_transcripts(args: CommonArgs) -> anyhow::Result<()> { - let ctx = run_context_from(&args)?; - let dir = iteration_dir(&ctx, args.iteration)?; - let result = pipeline::fill_transcripts(&dir, ctx.harness, args.overwrite)?; - - println!( - "\nFilled: {}, skipped (already populated): {}, missing transcript: {}", - result.filled, result.skipped, result.missing - ); - Ok(()) -} - /// Report writes outside each private task environment (and live-source reads) for /// every run in the iteration. pub(crate) fn run_detect_stray_writes(args: CommonArgs) -> anyhow::Result<()> { diff --git a/src/cli/help.rs b/src/cli/help.rs index de6fb3a..df1f8fc 100644 --- a/src/cli/help.rs +++ b/src/cli/help.rs @@ -31,10 +31,10 @@ EXAMPLES: # Reduce cost while iterating on the suite eval-magic run --only case-a,case-b - # Scaffold against the current project instead of the default example fixture; + # Scaffold against the current project instead of the default example project; # init records the selected source in evals.json eval-magic init --codebase-cwd - # Source selection, fixture scale, pinning, and recorded provenance + # Source selection, project scale, pinning, and recorded provenance eval-magic docs codebase # Pair both conditions for exploratory review before writing assertions diff --git a/src/cli/init_args.rs b/src/cli/init_args.rs index ee142da..aae8139 100644 --- a/src/cli/init_args.rs +++ b/src/cli/init_args.rs @@ -44,7 +44,7 @@ pub(crate) struct InitArgs { /// negative evals where correct behavior is not invoking the skill. #[arg(long)] pub skill_should_trigger: Option, - /// Git repository URL to use as the eval fixture codebase. + /// Git repository URL to use as the eval codebase. /// /// Requires `--codebase-ref`. `init` records both values without contacting /// the remote, so use a full commit SHA when the scaffold must be reproducible. @@ -67,7 +67,7 @@ pub(crate) struct InitArgs { conflicts_with_all = ["codebase_path", "codebase_cwd"] )] pub codebase_ref: Option, - /// Local directory to use as the eval fixture codebase. + /// Local directory to use as the eval codebase. /// /// A relative path resolves from the invocation directory and is written /// relative to the generated `evals/` directory. An absolute path remains @@ -78,7 +78,7 @@ pub(crate) struct InitArgs { conflicts_with_all = ["codebase_url", "codebase_ref", "codebase_cwd"] )] pub codebase_path: Option, - /// Use the invocation directory as the eval fixture codebase. + /// Use the invocation directory as the eval codebase. /// /// Writes a path relative to the generated `evals/` directory. Conflicts /// with `--codebase-url`, `--codebase-ref`, and `--codebase-path`. diff --git a/src/cli/mod.rs b/src/cli/mod.rs index 9e7f591..e30fcbc 100644 --- a/src/cli/mod.rs +++ b/src/cli/mod.rs @@ -120,7 +120,6 @@ fn dispatch(command: Option, harness_file: Option<&str>) -> anyhow::Re Commands::GuardHook { harness, marker } => run_guard_hook(&harness, marker), Commands::Fixture(args) => run_fixture(args), Commands::RecordRuns(args) => run_record_runs(args), - Commands::FillTranscripts(args) => run_fill_transcripts(args), Commands::DetectStrayWrites(args) => run_detect_stray_writes(args), Commands::Grade(args) => run_grade(args), Commands::Aggregate(args) => run_aggregate(args), diff --git a/src/cli/run/conversation.rs b/src/cli/run/conversation.rs index d9a694e..64a47eb 100644 --- a/src/cli/run/conversation.rs +++ b/src/cli/run/conversation.rs @@ -18,7 +18,7 @@ use crate::adapters::cli_command::shell_quote_arg; use crate::adapters::descriptor::subst; use crate::adapters::descriptor_adapter::DescriptorAdapter; use crate::adapters::harness::HarnessAdapter; -use crate::adapters::transcript::{TranscriptEvent, TranscriptSummary}; +use crate::adapters::transcript::TranscriptSummary; use crate::core::{ ConversationEvent, ConversationRecord, ConversationStatus, ConversationStopReason, ResponderOutcome, ResponderStopCause, ShellOutcome, run_in_posix_shell, @@ -219,8 +219,6 @@ pub fn run_task( text: task.user_prompt.clone(), origin: None, }]; - let mut next_ordinal = 1_u32; - let first_outputs = base_outputs.join("turn-1"); let initial_command = render_command( &initial_template, @@ -244,7 +242,6 @@ pub fn run_task( // session to resume. The seeded user message is the whole record. return write_conversation( &conversation_path, - base_outputs, ConversationRecord { status: ConversationStatus::TimedOut, delivered_followups: 0, @@ -254,22 +251,23 @@ pub fn run_task( events, responder_outcome: None, }, - None, plan.source(), ); } let first_summary = parse_round(adapter, &first_outputs, &events_filename, 1)?; - let session_id = first_summary - .session_id - .clone() - .filter(|id| !id.trim().is_empty()) - .ok_or_else(|| anyhow!("turn 1 transcript did not expose a native session id"))?; - let mut preceding_assistant = - append_summary_events(&mut events, &mut next_ordinal, 1, &first_summary)?; - let mut final_message = first_summary - .final_text - .clone() - .expect("append_summary_events requires final_text"); + let session_id = if plan.delivers_followups() { + Some( + first_summary + .session_id + .clone() + .filter(|id| !id.trim().is_empty()) + .ok_or_else(|| anyhow!("turn 1 transcript did not expose a native session id"))?, + ) + } else { + None + }; + let mut preceding_assistant = final_text_for_round(1, &first_summary)?; + let mut final_message = preceding_assistant.clone(); let mut delivered_followups = 0_u32; let mut stop_reason = None; @@ -313,12 +311,11 @@ pub fn run_task( let round = followup.saturating_add(1); events.push(ConversationEvent::UserMessage { - ordinal: next_ordinal, + ordinal: events.len() as u32, round, text: prompt.clone(), origin, }); - next_ordinal = next_ordinal.saturating_add(1); delivered_followups = delivered_followups.saturating_add(1); let round_outputs = base_outputs.join(format!("turn-{round}")); @@ -330,7 +327,11 @@ pub fn run_task( eval_root, &task.dispatch_prompt_path, &round_outputs, - Some(&session_id), + Some( + session_id + .as_deref() + .expect("a follow-up task resolved a session id above"), + ), Some(&prompt), round, ); @@ -348,16 +349,15 @@ pub fn run_task( } let summary = parse_round(adapter, &round_outputs, &events_filename, round)?; if let Some(observed) = summary.session_id.as_deref() - && observed != session_id + && Some(observed) != session_id.as_deref() { - bail!("turn {round} resumed session {observed:?}, expected {session_id:?}"); + bail!( + "turn {round} resumed session {observed:?}, expected {:?}", + session_id.as_deref().unwrap_or_default() + ); } - preceding_assistant = - append_summary_events(&mut events, &mut next_ordinal, round, &summary)?; - final_message = summary - .final_text - .clone() - .expect("append_summary_events requires final_text"); + preceding_assistant = final_text_for_round(round, &summary)?; + final_message = preceding_assistant.clone(); } // A timeout outranks a gate stop: the conversation was cut short, so what @@ -369,7 +369,6 @@ pub fn run_task( }; write_conversation( &conversation_path, - base_outputs, ConversationRecord { status, delivered_followups, @@ -381,19 +380,14 @@ pub fn run_task( // outranks a gate stop: the round it judged never finished. responder_outcome: timed_out_in_round.map_or(responder_outcome, |_| None), }, - Some(final_message), plan.source(), ) } -/// Validate, commit, and report one task's completion artifact. `final_message` -/// is absent when no round produced one, which is only possible for a task that -/// timed out before its first answer. +/// Validate, commit, and report one task's completion artifact. fn write_conversation( conversation_path: &Path, - base_outputs: &Path, conversation: ConversationRecord, - final_message: Option, source: TurnSource, ) -> anyhow::Result { let _: ConversationRecord = validate_against_schema( @@ -402,13 +396,6 @@ fn write_conversation( &conversation_path.to_string_lossy(), )?; write_json_atomic(conversation_path, &conversation)?; - fs::create_dir_all(base_outputs)?; - if let Some(final_message) = final_message { - fs::write( - base_outputs.join("final-message.md"), - format!("{}\n", final_message.trim_end()), - )?; - } Ok(match conversation.status { ConversationStatus::Completed => TaskOutcome::Completed { @@ -441,56 +428,13 @@ fn parse_round( .with_context(|| format!("failed to parse turn {round} transcript {}", path.display())) } -fn append_summary_events( - out: &mut Vec, - next_ordinal: &mut u32, - round: u32, - summary: &TranscriptSummary, -) -> anyhow::Result { - let final_text = summary +fn final_text_for_round(round: u32, summary: &TranscriptSummary) -> anyhow::Result { + summary .final_text .as_deref() .filter(|text| !text.trim().is_empty()) - .ok_or_else(|| { - anyhow!("turn {round} transcript did not contain a final assistant message") - })?; - let mut assistant_messages = Vec::new(); - for event in &summary.events { - match event { - TranscriptEvent::AssistantMessage { text, .. } => { - assistant_messages.push(text.clone()); - out.push(ConversationEvent::AssistantMessage { - ordinal: *next_ordinal, - round, - text: text.clone(), - }); - } - TranscriptEvent::ToolInvocation { - name, args, result, .. - } => out.push(ConversationEvent::ToolInvocation { - ordinal: *next_ordinal, - round, - name: name.clone(), - args: args.clone(), - result: result.clone(), - }), - } - *next_ordinal = next_ordinal.saturating_add(1); - } - let final_already_present = assistant_messages - .iter() - .enumerate() - .any(|(start, _)| assistant_messages[start..].join("\n") == final_text); - if !final_already_present { - assistant_messages.push(final_text.to_string()); - out.push(ConversationEvent::AssistantMessage { - ordinal: *next_ordinal, - round, - text: final_text.to_string(), - }); - *next_ordinal = next_ordinal.saturating_add(1); - } - Ok(assistant_messages.join("\n")) + .map(str::to_string) + .ok_or_else(|| anyhow!("turn {round} transcript did not contain a final assistant message")) } /// Render a one-shot dispatch command: the exec template with its task @@ -598,10 +542,9 @@ fn write_json_atomic(path: &Path, value: &impl serde::Serialize) -> anyhow::Resu mod tests { use std::collections::BTreeMap; - use super::{append_summary_events, execute_round}; + use super::{execute_round, final_text_for_round}; use crate::adapters::TranscriptSummary; use crate::adapters::cli_command::shell_quote_arg; - use crate::adapters::transcript::TranscriptEvent; #[test] fn execute_round_creates_the_round_output_directory_before_shell_redirection() { @@ -622,34 +565,17 @@ mod tests { } #[test] - fn joined_final_text_does_not_duplicate_assistant_text_blocks() { + fn final_text_for_round_returns_the_parser_result() { let summary = TranscriptSummary { tool_invocations: Vec::new(), - events: vec![ - TranscriptEvent::AssistantMessage { - ordinal: 0, - text: "Which timezone?".into(), - }, - TranscriptEvent::AssistantMessage { - ordinal: 1, - text: "Please include the locale.".into(), - }, - ], + events: Vec::new(), session_id: Some("session-1".into()), total_tokens: None, duration_ms: None, final_text: Some("Which timezone?\nPlease include the locale.".into()), }; - let mut events = Vec::new(); - let mut next_ordinal = 0; - - let assistant = append_summary_events(&mut events, &mut next_ordinal, 1, &summary).unwrap(); + let assistant = final_text_for_round(1, &summary).unwrap(); assert_eq!(assistant, "Which timezone?\nPlease include the locale."); - assert_eq!( - events.len(), - 2, - "joined final text must not be appended again" - ); } } diff --git a/src/cli/run/dispatch.rs b/src/cli/run/dispatch.rs index 54b73fd..3ea4133 100644 --- a/src/cli/run/dispatch.rs +++ b/src/cli/run/dispatch.rs @@ -23,7 +23,7 @@ use super::RunError; mod prompt_components; -use prompt_components::{effective_bootstrap, render_fixtures_block, render_skill_block}; +use prompt_components::{effective_bootstrap, render_overlay_files_block, render_skill_block}; /// One dispatchable task: the metadata the orchestrator persists per /// `(eval, condition)`. `dispatch_prompt` is held in memory (for manifest @@ -43,7 +43,8 @@ pub struct DispatchTask { #[serde(default, skip_serializing_if = "Option::is_none")] pub available_skills: Option>, pub user_prompt: String, - pub fixtures: Vec, + #[serde(alias = "fixtures")] + pub files: Vec, pub outputs_dir: String, pub run_record_path: String, pub timing_path: String, @@ -107,7 +108,7 @@ pub struct DispatchTaskOpts<'a> { /// treatment reference from optional bootstrap content. pub treatment_names: Option<&'a [String]>, pub user_prompt: &'a str, - pub fixtures: Vec, + pub files: Vec, pub turns: Option<&'a [ScriptedTurn]>, pub outputs_dir: &'a str, pub cond_dir: &'a str, @@ -169,7 +170,7 @@ pub fn build_dispatch_task(opts: &DispatchTaskOpts) -> Result Result Result Result::to_vec), available_skills: opts.skills.map(|_| staged_skills), user_prompt: opts.user_prompt.to_string(), - fixtures: opts.fixtures.clone(), + files: opts.files.clone(), run_record_path: artifact_path(&cond_dir.join("run.json")), timing_path: artifact_path(&cond_dir.join("timing.json")), turns: opts.turns.map(<[ScriptedTurn]>::to_vec), @@ -461,11 +456,9 @@ pub fn build_manifest( header.extend([ "After all dispatches:".to_string(), String::new(), - "1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json` + the task's own `outputs/final-message.md` + the events file the harness CLI wrote under `outputs/turn-/`, and backfills `timing.json` with transcript-derived tokens/duration; never clobbers an existing record), fill-transcripts, detect-stray-writes, and grade. Optional higher-fidelity timing: write `{ \"total_tokens\": , \"duration_ms\": , \"source\": \"completion-event\" }` from the task completion event to `timing.json` right after a dispatch — completion-event numbers always win over the backfill.".to_string(), + "1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json`, `conversation.json`, and the harness events under `outputs/turn-/`, and backfills `timing.json`; never clobbers an existing record), detect-stray-writes, and grade.".to_string(), "2. Run `eval-magic dispatch --judges --harness ` to grade the judge tasks ingest listed, then `eval-magic finalize` for the benchmark.".to_string(), String::new(), - "On a harness without persisted transcripts, instead write each task's `run.json` (matching `skills/evaluating-skills/schema/run-record.schema.json`, enforced at runtime by grade/fill-transcripts/detect-stray-writes) and `timing.json` by hand when its subagent returns: carry over `eval_id`, `condition`, `skill_path` (`null` on the without_skill arm), `prompt`, and `files` from the task; populate `final_message` from the subagent's reply; leave `tool_invocations` as `[]`; capture `total_tokens`/`duration_ms` from the task completion event immediately — they may not be persisted anywhere else.".to_string(), - String::new(), "## Dispatches".to_string(), String::new(), ]); @@ -520,7 +513,6 @@ mod tests { assertions: None, skill_should_trigger: None, runs: None, - isolation: None, turns: None, codebase: None, responder: None, @@ -641,7 +633,7 @@ mod tests { // ── build_dispatch_task: bootstrap injection ────────────────────────── #[test] - fn prompt_allows_task_edits_but_reserves_outputs_for_framework_artifacts() { + fn prompt_allows_task_edits_without_requesting_agent_authored_framework_artifacts() { let task = build_dispatch_task(&DispatchTaskOpts { eval_root: Some("/tmp/env"), ..base_opts() @@ -651,8 +643,10 @@ mod tests { assert!(prompt.contains("Task environment: /tmp/env")); assert!(prompt.contains("edit existing files and create new files inside")); - assert!(prompt.contains("framework artifacts")); assert!(prompt.contains("Do not write outside the task environment.")); + assert!(!prompt.contains("Framework output directory:")); + assert!(!prompt.contains("framework artifacts")); + assert!(!prompt.contains("final-message.md")); assert!(!prompt.contains("Write any files you produce into the output directory.")); assert!(!prompt.contains("Do not write outside the output directory.")); } diff --git a/src/cli/run/dispatch/prompt_components.rs b/src/cli/run/dispatch/prompt_components.rs index 4b49830..de01432 100644 --- a/src/cli/run/dispatch/prompt_components.rs +++ b/src/cli/run/dispatch/prompt_components.rs @@ -93,15 +93,15 @@ pub(super) fn render_skill_block( } } -pub(super) fn render_fixtures_block(fixtures: &[String]) -> String { - if fixtures.is_empty() { - "Available fixture files: none".to_string() +pub(super) fn render_overlay_files_block(files: &[String]) -> String { + if files.is_empty() { + "Codebase overlay files: none".to_string() } else { format!( - "Available fixture files:\n{}", - fixtures + "Codebase overlay files:\n{}", + files .iter() - .map(|fixture| format!(" - {fixture}")) + .map(|file| format!(" - {file}")) .collect::>() .join("\n") ) diff --git a/src/cli/run/fixtures.rs b/src/cli/run/fixtures.rs deleted file mode 100644 index 8d369c1..0000000 --- a/src/cli/run/fixtures.rs +++ /dev/null @@ -1,401 +0,0 @@ -//! Copy an eval's fixtures into its isolated task env, laid out like a real repo -//! so the agent-under-test reads them at natural project-relative paths. -//! [`FixtureClaims`] retains the legacy shared-group collision checks used by the -//! environment planner. - -use std::fs; -use std::path::Path; - -use crate::core::{AssertionCommandCheck, Eval}; - -use super::RunError; -use crate::core::fs::copy_entry_materialized; - -/// Cross-eval claims on env-relative fixture destinations: `dest → (eval_id, source)`. -/// Used when a planned group contains multiple evals; canonical diff-scope runs -/// currently task-scope groups, but legacy plans remain readable. -pub type FixtureClaims = std::collections::HashMap; - -/// Record that `eval_id` provides the fixture at env-relative `dest` from `source`. -/// Returns `Ok(true)` when the dest was already claimed from the same source (idempotent -/// share — skip the re-copy), `Ok(false)` on the first claim, and `Err` when a *different* -/// source already claimed the same dest (an order-dependent cross-eval clobber). -fn claim_fixture_dest( - claims: &mut FixtureClaims, - eval_id: &str, - dest: &str, - source: &str, -) -> Result { - if let Some((prev_eval, prev_source)) = claims.get(dest) { - if prev_source != source { - return Err(RunError::msg(format!( - "fixture conflict: evals '{prev_eval}' and '{eval_id}' both place a fixture at env path '{dest}' from different sources ('{prev_source}' vs '{source}'). Give them distinct paths." - ))); - } - return Ok(true); - } - claims.insert(dest.to_string(), (eval_id.to_string(), source.to_string())); - Ok(false) -} - -/// True for a path that is absolute under *either* platform's rules. -/// -/// `Path::is_absolute` answers for the host only: Windows has no root without a -/// drive, so `/etc/passwd` reads as relative there and slips past a bare -/// `is_absolute` check — then `Path::join`'s root-replacing behavior lands it -/// outside the env entirely. Eval configs are committed and run on every -/// platform, so the verdict must not depend on which host reads them. -/// -/// A drive-relative Windows path (`C:fixture.txt`) is only detectable where the -/// parser produces a `Prefix` component, so it is caught on Windows and treated -/// as an ordinary relative name elsewhere. -fn is_absolute_on_any_platform(raw: &str) -> bool { - let path = Path::new(raw); - path.has_root() - || raw.starts_with('\\') - || matches!( - path.components().next(), - Some(std::path::Component::Prefix(_)) - ) -} - -/// Reject a fixture path that is absolute or escapes `env/` via `..`, so a fixture -/// always lands inside the isolated env. -fn validate_fixture_rel(f: &str) -> Result<(), RunError> { - let p = Path::new(f); - let escapes = is_absolute_on_any_platform(f) - || p.components() - .any(|c| matches!(c, std::path::Component::ParentDir)); - if escapes { - return Err(RunError::msg(format!( - "fixture path must be relative and stay within env: {f}" - ))); - } - let first_normal = p.components().find_map(|component| match component { - std::path::Component::Normal(value) => Some(value.to_string_lossy()), - _ => None, - }); - if first_normal.is_some_and(|component| component.eq_ignore_ascii_case(".git")) { - return Err(RunError::msg(format!( - "fixture path uses reserved runner-owned Git metadata at task root: {f}" - ))); - } - Ok(()) -} - -/// Reject a fixture source root that is absolute or escapes `/evals/`. -/// Unlike a fixture destination, this path never lands in the task repository, -/// so a leading `.git` component is not runner-owned metadata. -fn validate_files_root_rel(root: &str) -> Result<(), RunError> { - let path = Path::new(root); - let escapes = is_absolute_on_any_platform(root) - || path - .components() - .any(|component| matches!(component, std::path::Component::ParentDir)); - if escapes { - return Err(RunError::msg(format!( - "files_root must be relative and stay within the skill's evals directory: {root}" - ))); - } - Ok(()) -} - -/// Resolve an eval's declared fixtures to `(env-relative dest, source path)` pairs, -/// validating each path stays within the env and that the source exists — without -/// copying anything. [`super::grouping`] consumes these pairs to detect cross-eval -/// clobbers before any env is built, and [`copy_fixtures`] reuses them, so fixture -/// path resolution lives in exactly one place. -pub fn fixture_pairs(ev: &Eval, skill_dir: &Path) -> Result, RunError> { - let mut source_root = skill_dir.join("evals"); - if let Some(root) = ev.files_root.as_deref() { - validate_files_root_rel(root)?; - source_root = source_root.join(root); - } - let Some(files) = ev.files.as_ref().filter(|f| !f.is_empty()) else { - return Ok(Vec::new()); - }; - let mut pairs = Vec::with_capacity(files.len()); - for f in files { - validate_fixture_rel(f)?; - let src = source_root.join(f); - if !src.exists() { - return Err(RunError::msg(format!( - "fixture not found: {}", - src.display() - ))); - } - pairs.push((f.clone(), src.to_string_lossy().into_owned())); - } - Ok(pairs) -} - -/// Resolve a command check's held-out setup paths without copying them. This is -/// called during read-only run resolution so a missing source fails before any -/// workspace or staged environment is created. -pub fn setup_file_pairs( - check: &AssertionCommandCheck, - skill_dir: &Path, -) -> Result, RunError> { - let Some(files) = check.setup_files.as_ref().filter(|files| !files.is_empty()) else { - return Ok(Vec::new()); - }; - let mut pairs = Vec::with_capacity(files.len()); - for file in files { - validate_fixture_rel(file)?; - let source = skill_dir.join("evals").join(file); - if !source.exists() { - return Err(RunError::msg(format!( - "command-check setup file not found: {}", - source.display() - ))); - } - pairs.push((file.clone(), source.to_string_lossy().into_owned())); - } - Ok(pairs) -} - -/// Copy an eval's fixture files into `env_root`, preserving each declared relative path -/// so the env reads like a real repo (`files: ["src/main.rs"]` → `env/src/main.rs`), and -/// returning the env-relative paths (the agent-under-test's cwd is `env/`). Fixtures are -/// shared across conditions and runs within one env; `claims` dedups idempotent -/// re-declarations and rejects cross-eval clobbers. Cross-eval clobbers are routed into -/// separate isolation groups by [`super::grouping`] before this is called per group, so -/// within a single group's env a clobber should never reach the `claims` rejection. -pub fn copy_fixtures( - ev: &Eval, - skill_dir: &Path, - env_root: &Path, - claims: &mut FixtureClaims, -) -> Result, RunError> { - let pairs = fixture_pairs(ev, skill_dir)?; - let mut copied = Vec::with_capacity(pairs.len()); - for (dest, source) in &pairs { - let already = claim_fixture_dest(claims, &ev.id, dest, source)?; - if !already { - let dst = env_root.join(dest); - if let Some(parent) = dst.parent() { - fs::create_dir_all(parent)?; - } - copy_entry_materialized(Path::new(source), &dst)?; - } - copied.push(dest.clone()); - } - Ok(copied) -} - -#[cfg(test)] -mod tests { - use super::*; - - fn eval_with_files(id: &str, files: &[&str]) -> Eval { - Eval { - id: id.to_string(), - prompt: "p".to_string(), - expected_output: "o".to_string(), - files: Some(files.iter().map(|f| (*f).to_string()).collect()), - files_root: None, - assertions: None, - skill_should_trigger: None, - runs: None, - isolation: None, - turns: None, - codebase: None, - responder: None, - guard: None, - } - } - - #[test] - fn fixture_pairs_resolves_dest_and_source_without_copying() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - let evals = skill_dir.join("evals"); - fs::create_dir_all(evals.join("data")).unwrap(); - fs::write(evals.join("config.json"), "cfg").unwrap(); - fs::write(evals.join("data/x.json"), "xx").unwrap(); - - let ev = eval_with_files("e1", &["config.json", "data/x.json"]); - let pairs = fixture_pairs(&ev, &skill_dir).unwrap(); - - assert_eq!( - pairs, - vec![ - ( - "config.json".to_string(), - evals.join("config.json").to_string_lossy().into_owned() - ), - ( - "data/x.json".to_string(), - evals.join("data/x.json").to_string_lossy().into_owned() - ), - ] - ); - // Pure: it resolves paths but copies nothing. - assert!(!tmp.path().join("env").exists()); - } - - #[test] - fn fixture_pairs_empty_when_no_files() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - fs::create_dir_all(skill_dir.join("evals")).unwrap(); - let ev = eval_with_files("e1", &[]); - assert!(fixture_pairs(&ev, &skill_dir).unwrap().is_empty()); - } - - #[test] - fn fixture_pairs_rejects_escapes_and_missing_sources() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - fs::create_dir_all(skill_dir.join("evals")).unwrap(); - - let escaping = eval_with_files("e1", &["../escape.txt"]); - assert!( - fixture_pairs(&escaping, &skill_dir) - .unwrap_err() - .to_string() - .contains("relative") - ); - - let missing = eval_with_files("e1", &["nope.json"]); - assert!( - fixture_pairs(&missing, &skill_dir) - .unwrap_err() - .to_string() - .contains("fixture not found") - ); - } - - #[test] - fn fixture_pairs_rejects_files_root_escapes_even_when_sources_exist() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - let evals = skill_dir.join("evals"); - fs::create_dir_all(&evals).unwrap(); - - let outside = skill_dir.join("outside"); - fs::create_dir_all(&outside).unwrap(); - fs::write(outside.join("secret.txt"), "secret").unwrap(); - - for root in [ - "../outside".to_string(), - outside.to_string_lossy().into_owned(), - ] { - let mut ev = eval_with_files("e1", &["secret.txt"]); - ev.files_root = Some(root); - let error = fixture_pairs(&ev, &skill_dir).unwrap_err().to_string(); - assert!(error.contains("files_root"), "error was: {error}"); - assert!(error.contains("relative"), "error was: {error}"); - } - } - - #[test] - fn fixture_pairs_validates_files_root_without_declared_files() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - fs::create_dir_all(skill_dir.join("evals")).unwrap(); - let mut ev = eval_with_files("e1", &[]); - ev.files_root = Some("../outside".to_string()); - - let error = fixture_pairs(&ev, &skill_dir).unwrap_err().to_string(); - - assert!(error.contains("files_root"), "error was: {error}"); - assert!(error.contains("relative"), "error was: {error}"); - } - - #[test] - fn fixture_paths_reserve_only_a_root_dot_git_component_case_insensitively() { - for bad in [ - ".git", - ".git/config", - "./.GIT/config", - ".GiT/hooks/pre-commit", - ] { - let error = validate_fixture_rel(bad).unwrap_err(); - assert!( - error.to_string().contains("reserved"), - "expected root Git metadata rejection for {bad}, got: {error}" - ); - } - - validate_fixture_rel("vendor/.git/config").unwrap(); - } - - #[test] - fn copy_fixtures_preserves_declared_relative_paths_in_env() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - let evals = skill_dir.join("evals"); - fs::create_dir_all(evals.join("data")).unwrap(); - fs::write(evals.join("config.json"), "cfg").unwrap(); - fs::write(evals.join("data/x.json"), "xx").unwrap(); - let env_root = tmp.path().join("env"); - - let ev = eval_with_files("e1", &["config.json", "data/x.json"]); - let mut claims = FixtureClaims::new(); - let copied = copy_fixtures(&ev, &skill_dir, &env_root, &mut claims).unwrap(); - - // Structure preserved under env/, not flattened into an inputs/ bucket. - assert_eq!( - fs::read_to_string(env_root.join("config.json")).unwrap(), - "cfg" - ); - assert_eq!( - fs::read_to_string(env_root.join("data/x.json")).unwrap(), - "xx" - ); - assert!(!env_root.join("inputs").exists()); - // Returns env-relative declared paths (the agent's cwd is env). - assert_eq!( - copied, - vec!["config.json".to_string(), "data/x.json".to_string()] - ); - } - - #[test] - fn copy_fixtures_rejects_parent_escaping_and_absolute_paths() { - let tmp = tempfile::TempDir::new().unwrap(); - let skill_dir = tmp.path().join("skill"); - fs::create_dir_all(skill_dir.join("evals")).unwrap(); - let env_root = tmp.path().join("env"); - - // `\etc\passwd` is absolute on Windows and a legal filename on Unix. - // Eval configs are committed and run on every platform, so the verdict - // must not depend on which host reads them — a fixture that validates - // on Linux and escapes the env on Windows is the worst of both. - for bad in [ - "../escape.txt", - "/etc/passwd", - "a/../../b.txt", - r"\etc\passwd", - ] { - let ev = eval_with_files("e1", &[bad]); - let mut claims = FixtureClaims::new(); - let err = copy_fixtures(&ev, &skill_dir, &env_root, &mut claims).unwrap_err(); - assert!( - err.to_string().contains("relative"), - "expected a path-traversal rejection for {bad}, got: {err}" - ); - } - } - - #[test] - fn claim_fixture_dest_allows_idempotent_share_errors_on_different_source() { - let mut claims = FixtureClaims::new(); - // First eval claims the dest. - assert!( - !claim_fixture_dest(&mut claims, "e1", "config.json", "/a/evals/config.json").unwrap() - ); - // A second eval declaring the same dest from the same source is an idempotent share. - assert!( - claim_fixture_dest(&mut claims, "e2", "config.json", "/a/evals/config.json").unwrap() - ); - // The same dest from a *different* source is an ambiguous cross-eval conflict. - let err = claim_fixture_dest(&mut claims, "e3", "config.json", "/b/evals/config.json") - .unwrap_err(); - let msg = err.to_string(); - assert!(msg.contains("e1"), "names the first claimer: {msg}"); - assert!(msg.contains("e3"), "names the conflicting eval: {msg}"); - assert!(msg.contains("config.json"), "names the path: {msg}"); - } -} diff --git a/src/cli/run/golden_tests.rs b/src/cli/run/golden_tests.rs index 87b570a..d00c1f7 100644 --- a/src/cli/run/golden_tests.rs +++ b/src/cli/run/golden_tests.rs @@ -70,7 +70,7 @@ fn staged_task(harness: Harness) -> DispatchTaskOpts<'static> { condition: "with_skill", staged_skill_path: Some("/work/staged/widget-skill/SKILL.md"), user_prompt: "Build me a widget.", - fixtures: vec!["/work/fixtures/input.txt".to_string()], + files: vec!["/work/overlays/input.txt".to_string()], outputs_dir: "/work/outputs", eval_root: Some("/work/task"), cond_dir: "/work/cond", @@ -88,7 +88,7 @@ fn bare_task(harness: Harness) -> DispatchTaskOpts<'static> { eval_id: "demo-eval", condition: "without_skill", user_prompt: "Build me a widget.", - fixtures: vec!["/work/fixtures/input.txt".to_string()], + files: vec!["/work/overlays/input.txt".to_string()], outputs_dir: "/work/outputs-b", eval_root: Some("/work/task-b"), cond_dir: "/work/cond-b", diff --git a/src/cli/run/grouping.rs b/src/cli/run/grouping.rs index e9931a4..b3c0065 100644 --- a/src/cli/run/grouping.rs +++ b/src/cli/run/grouping.rs @@ -1,161 +1,35 @@ -//! Setup-time task-environment planning, before a run dispatches anything. +//! Deterministic private task-environment planning. //! -//! Canonical runs task-scope every eval/run so final-environment diff metrics have -//! an unambiguous baseline. The older fixture-compatibility and -//! [`Isolation::Isolated`] paths remain available to the pure planner for artifact -//! compatibility, while production inputs set `task_scoped`. -//! -//! The conflict rule is identical to the per-env fixture-claim rule in -//! [`super::fixtures`]: same dest + same source is an idempotent share (evals may -//! co-group); same dest + different source is a clobber (they must not). - -use std::collections::HashMap; - -use crate::core::Isolation; +//! Every eval has its own group, and every run within that group receives its +//! own environment per condition. Group ids remain stable artifact join keys. /// One eval's inputs to grouping. pub struct GroupInput<'a> { pub eval_id: &'a str, - pub isolation: Option, - /// `(env-relative dest, source)` fixture pairs this eval declares. - pub fixtures: &'a [(String, String)], - /// Always-on final-environment metrics require a task-scoped environment. - pub task_scoped: bool, - /// Effective run count, retained for task-scoped per-run env planning. + /// Effective run count, retained for per-run environment planning. pub runs: u32, } -/// A computed isolation group: the evals that share one environment, plus a -/// human-readable reason the group exists (surfaced in `dispatch.json`). +/// A private eval group surfaced in `dispatch.json`. #[derive(Debug, Clone, PartialEq, Eq)] pub struct Group { pub id: String, pub eval_ids: Vec, pub rationale: String, - /// Present for task-scoped groups. Multi-run values fan the group out into - /// one environment per run. - pub task_runs: Option, + /// Multi-run values fan the group out into one environment per run. + pub runs: u32, } -/// Group `evals` (in config order) by fixture compatibility and explicit hints. -/// -/// Deterministic greedy first-fit: each eval joins the first existing group it -/// does not conflict with; an [`Isolation::Isolated`] eval always gets a fresh, -/// sealed singleton; otherwise a new group is started. Group ids are `g1, g2, …` -/// in creation order. With no conflicts and no `isolated` hints this returns a -/// single `g1` containing every eval — the common case. -pub fn compute_groups(evals: &[GroupInput]) -> Vec { - /// A group under construction: its accumulated `dest -> (source, eval_id)` - /// claims plus whether it is sealed against new members (isolated singletons). - struct Building { - id: String, - eval_ids: Vec, - claims: HashMap, - sealed: bool, - rationale: String, - task_runs: Option, - } - - fn claims_of(ev: &GroupInput) -> HashMap { - ev.fixtures - .iter() - .map(|(dest, source)| (dest.clone(), (source.clone(), ev.eval_id.to_string()))) - .collect() - } - - let mut groups: Vec = Vec::new(); - - for ev in evals { - if ev.task_scoped { - let id = format!("g{}", groups.len() + 1); - let rationale = if ev.isolation == Some(Isolation::Isolated) { - "isolation: isolated; metric: diff_scope" - } else { - "metric: diff_scope" - }; - groups.push(Building { - id, - eval_ids: vec![ev.eval_id.to_string()], - claims: claims_of(ev), - sealed: true, - rationale: rationale.to_string(), - task_runs: Some(ev.runs), - }); - continue; - } - - // An `isolated` eval always gets a fresh, sealed singleton — nothing else - // may join it, and it joins nothing else. - if ev.isolation == Some(Isolation::Isolated) { - let id = format!("g{}", groups.len() + 1); - groups.push(Building { - id, - eval_ids: vec![ev.eval_id.to_string()], - claims: claims_of(ev), - sealed: true, - rationale: "isolation: isolated".to_string(), - task_runs: None, - }); - continue; - } - - // Greedy first-fit over the non-sealed groups, in creation order. - let mut joined = false; - let mut conflict_note: Option = None; - for g in groups.iter_mut().filter(|g| !g.sealed) { - // The eval conflicts with this group iff it claims a dest the group - // already holds from a *different* source (an order-dependent clobber). - // Same dest + same source is an idempotent share — not a conflict. - let mut conflict: Option<(String, String)> = None; - for (dest, source) in ev.fixtures { - if let Some((prev_source, prev_eval)) = g.claims.get(dest) - && prev_source != source - { - conflict = Some((dest.clone(), prev_eval.clone())); - break; - } - } - match conflict { - None => { - for (dest, source) in ev.fixtures { - g.claims - .entry(dest.clone()) - .or_insert_with(|| (source.clone(), ev.eval_id.to_string())); - } - g.eval_ids.push(ev.eval_id.to_string()); - joined = true; - break; - } - Some((dest, other_eval)) => { - // Record the first conflict as the new group's rationale, but - // keep scanning — a later group may still accept this eval. - conflict_note.get_or_insert_with(|| { - format!("fixture-conflict: {} vs {other_eval} at {dest}", ev.eval_id) - }); - } - } - } - - if !joined { - let id = format!("g{}", groups.len() + 1); - groups.push(Building { - id, - eval_ids: vec![ev.eval_id.to_string()], - claims: claims_of(ev), - sealed: false, - rationale: conflict_note.unwrap_or_else(|| "default".to_string()), - task_runs: None, - }); - } - } - - groups - .into_iter() - .map(|b| Group { - id: b.id, - eval_ids: b.eval_ids, - rationale: b.rationale, - task_runs: b.task_runs, +/// Assign one deterministic group to each eval in config order. +pub fn compute_groups(evals: &[GroupInput<'_>]) -> Vec { + evals + .iter() + .enumerate() + .map(|(index, eval)| Group { + id: format!("g{}", index + 1), + eval_ids: vec![eval.eval_id.to_string()], + rationale: "private codebase".to_string(), + runs: eval.runs, }) .collect() } @@ -164,192 +38,33 @@ pub fn compute_groups(evals: &[GroupInput]) -> Vec { mod tests { use super::*; - fn input<'a>( - id: &'a str, - isolation: Option, - fixtures: &'a [(String, String)], - ) -> GroupInput<'a> { - GroupInput { - eval_id: id, - isolation, - fixtures, - task_scoped: false, - runs: 1, - } - } - - fn pair(dest: &str, source: &str) -> (String, String) { - (dest.to_string(), source.to_string()) - } - #[test] - fn single_group_when_no_conflicts_or_hints() { - let f1 = [pair("a.txt", "/s/a.txt")]; - let f2 = [pair("b.txt", "/s/b.txt")]; - let evals = [ - input("e1", None, &f1), - input("e2", None, &f2), - input("e3", None, &[]), - ]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 1); - assert_eq!(groups[0].id, "g1"); - assert_eq!(groups[0].eval_ids, vec!["e1", "e2", "e3"]); - assert_eq!(groups[0].rationale, "default"); - } - - #[test] - fn conflicting_fixtures_split_into_two_groups() { - let f1 = [pair("config.json", "/a/config.json")]; - let f2 = [pair("config.json", "/b/config.json")]; - let evals = [input("e1", None, &f1), input("e2", None, &f2)]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 2); - assert_eq!(groups[0].eval_ids, vec!["e1"]); - assert_eq!(groups[1].id, "g2"); - assert_eq!(groups[1].eval_ids, vec!["e2"]); - assert!( - groups[1].rationale.contains("fixture-conflict"), - "rationale: {}", - groups[1].rationale - ); - assert!( - groups[1].rationale.contains("config.json"), - "rationale: {}", - groups[1].rationale - ); - } - - #[test] - fn idempotent_same_source_share_stays_one_group() { - let f = [pair("config.json", "/a/config.json")]; - let evals = [input("e1", None, &f), input("e2", None, &f)]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 1); - assert_eq!(groups[0].eval_ids, vec!["e1", "e2"]); - } - - #[test] - fn isolated_hint_forces_singleton_and_seals_it() { - let evals = [ - input("e1", Some(Isolation::Isolated), &[]), - input("e2", None, &[]), - input("e3", None, &[]), - ]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 2); - assert_eq!(groups[0].eval_ids, vec!["e1"]); - assert_eq!(groups[0].rationale, "isolation: isolated"); - // The shared evals never join the sealed singleton. - assert_eq!(groups[1].eval_ids, vec!["e2", "e3"]); - assert_eq!(groups[1].rationale, "default"); - } - - #[test] - fn isolated_eval_with_fixtures_is_still_a_singleton() { - let f = [pair("x.txt", "/s/x.txt")]; - let evals = [input("e1", Some(Isolation::Isolated), &f)]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 1); - assert_eq!(groups[0].eval_ids, vec!["e1"]); - assert_eq!(groups[0].rationale, "isolation: isolated"); - } - - #[test] - fn ids_are_deterministic_in_creation_order() { - let f1 = [pair("c.json", "/a/c.json")]; - let f2 = [pair("c.json", "/b/c.json")]; - let f3 = [pair("c.json", "/d/c.json")]; - let evals = [ - input("e1", None, &f1), - input("e2", None, &f2), - input("e3", None, &f3), - ]; - let groups = compute_groups(&evals); - assert_eq!( - groups.iter().map(|g| g.id.as_str()).collect::>(), - vec!["g1", "g2", "g3"] - ); - } - - #[test] - fn eval_joins_first_non_conflicting_group() { - // e1 -> g1 (claims `a` from /s1). e2 conflicts on `a` -> g2. e3 shares `a` - // from /s1 (same source as g1) -> rejoins g1, not g2. - let f1 = [pair("a", "/s1/a")]; - let f2 = [pair("a", "/s2/a")]; - let f3 = [pair("a", "/s1/a")]; - let evals = [ - input("e1", None, &f1), - input("e2", None, &f2), - input("e3", None, &f3), - ]; - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 2); - assert_eq!(groups[0].eval_ids, vec!["e1", "e3"]); - assert_eq!(groups[1].eval_ids, vec!["e2"]); - } - - #[test] - fn task_scoped_eval_gets_a_singleton_group_and_per_run_environments() { - let evals = [ - GroupInput { - eval_id: "ordinary-1", - isolation: None, - fixtures: &[], - task_scoped: false, - runs: 1, - }, - GroupInput { - eval_id: "held-out", - isolation: None, - fixtures: &[], - task_scoped: true, - runs: 3, - }, - GroupInput { - eval_id: "ordinary-2", - isolation: None, - fixtures: &[], - task_scoped: false, - runs: 1, - }, - ]; - - let groups = compute_groups(&evals); - assert_eq!(groups.len(), 2); - assert_eq!(groups[0].eval_ids, vec!["ordinary-1", "ordinary-2"]); - assert_eq!(groups[0].task_runs, None); - assert_eq!(groups[1].eval_ids, vec!["held-out"]); - assert_eq!(groups[1].rationale, "metric: diff_scope"); - assert_eq!(groups[1].task_runs, Some(3)); - } - - #[test] - fn diff_metrics_force_every_eval_into_a_task_scoped_group() { + fn every_eval_gets_a_deterministic_private_group() { let evals = [ GroupInput { eval_id: "first", - isolation: None, - fixtures: &[], - task_scoped: true, runs: 1, }, GroupInput { eval_id: "second", - isolation: None, - fixtures: &[], - task_scoped: true, runs: 2, }, ]; let groups = compute_groups(&evals); + assert_eq!(groups.len(), 2); + assert_eq!(groups[0].id, "g1"); assert_eq!(groups[0].eval_ids, vec!["first"]); - assert_eq!(groups[0].rationale, "metric: diff_scope"); - assert_eq!(groups[0].task_runs, Some(1)); + assert_eq!(groups[0].rationale, "private codebase"); + assert_eq!(groups[0].runs, 1); + assert_eq!(groups[1].id, "g2"); assert_eq!(groups[1].eval_ids, vec!["second"]); - assert_eq!(groups[1].task_runs, Some(2)); + assert_eq!(groups[1].runs, 2); + } + + #[test] + fn empty_selection_produces_no_groups() { + assert!(compute_groups(&[]).is_empty()); } } diff --git a/src/cli/run/mod.rs b/src/cli/run/mod.rs index e9ecc85..80133ce 100644 --- a/src/cli/run/mod.rs +++ b/src/cli/run/mod.rs @@ -13,11 +13,11 @@ pub mod conversation; pub mod dispatch; pub mod drive; -pub mod fixtures; #[cfg(test)] mod golden_tests; pub mod grouping; pub mod orchestrate; +pub mod overlays; pub mod runbook; mod scratch; pub mod staging; diff --git a/src/cli/run/orchestrate/build.rs b/src/cli/run/orchestrate/build.rs index b7493c4..8fe2a7f 100644 --- a/src/cli/run/orchestrate/build.rs +++ b/src/cli/run/orchestrate/build.rs @@ -17,7 +17,7 @@ use super::super::RunError; use super::super::dispatch::{ DispatchTaskOpts, ManifestContext, build_dispatch_task, build_manifest, get_skill_description, }; -use super::super::fixtures::fixture_pairs; +use super::super::overlays::overlay_file_pairs; use super::super::runbook::{RunbookContext, build_runbook}; use super::super::staging::skills_dir_for_harness; use super::super::util::unguarded_notice; @@ -156,16 +156,15 @@ pub(super) fn write_dispatch( skills }; - // Each eval's env-relative fixture dests (for the task's `fixtures` field and - // the prompt's fixtures block). The copies themselves are made per env by + // Each eval's task-relative overlay destinations. The copies are made per env by // `stage_conditions`; resolution here is read-only (and re-validated in resolve). - let mut fixtures_by_eval: HashMap<&str, Vec> = HashMap::new(); + let mut overlay_files_by_eval: HashMap<&str, Vec> = HashMap::new(); for ev in &r.selected_evals { - let dests = fixture_pairs(ev, &ctx.skill_subdir)? + let dests = overlay_file_pairs(ev, &ctx.skill_subdir)? .into_iter() .map(|(dest, _source)| dest) .collect(); - fixtures_by_eval.insert(ev.id.as_str(), dests); + overlay_files_by_eval.insert(ev.id.as_str(), dests); } // A single group keeps the `group` key off each task (>1 group tags them); @@ -200,9 +199,7 @@ pub(super) fn write_dispatch( .iteration_dir .join(format!("eval-{}", ev.id)) .join(cond_name); - let codebase_record = r - .codebase_for(std::slice::from_ref(&ev.id))? - .map(super::RunCodebase::record); + let codebase_record = Some(r.codebase_for(std::slice::from_ref(&ev.id))?.record()); let runs = ev.runs.unwrap_or(opts.runs); for run_idx in 1..=runs { @@ -213,10 +210,7 @@ pub(super) fn write_dispatch( } else { (cond_dir.join(format!("run-{run_idx}")), Some(run_idx)) }; - let env_run_index = group - .task_runs - .is_some_and(|task_runs| task_runs > 1) - .then_some(run_idx); + let env_run_index = (group.runs > 1).then_some(run_idx); let env_root = task_env_root_for_run( &r.iteration_dir, &group.id, @@ -245,7 +239,7 @@ pub(super) fn write_dispatch( let outputs_dir = env_root.join(".eval-magic-outputs").join(outputs_rel); fs::create_dir_all(&outputs_dir)?; - let fixtures = fixtures_by_eval + let files = overlay_files_by_eval .get(ev.id.as_str()) .cloned() .unwrap_or_default(); @@ -261,7 +255,7 @@ pub(super) fn write_dispatch( skills: multi_skill.then_some(condition_roster), treatment_names: multi_skill.then_some(treatment_names.as_slice()), user_prompt: &ev.prompt, - fixtures, + files, turns: ev.turns.as_deref(), responder: ev.responder.as_ref(), outputs_dir: &outputs_dir_str, diff --git a/src/cli/run/orchestrate/envs.rs b/src/cli/run/orchestrate/envs.rs index d1fdb28..8f751ea 100644 --- a/src/cli/run/orchestrate/envs.rs +++ b/src/cli/run/orchestrate/envs.rs @@ -4,7 +4,7 @@ //! A canonical run gives every `(eval, condition, run)` dispatch a private //! `iteration-N/env--[-run-]/`. Each subprocess `cd`s into //! its own env, which holds only that condition's skill (or none) and that eval's -//! fixtures. +//! overlay files. use std::path::{Path, PathBuf}; @@ -18,7 +18,7 @@ pub(super) struct EnvTarget { /// `(condition name, that condition's skill path)` staged into this env — /// exactly one per env. pub conditions: Vec<(&'static str, Option)>, - /// Eval ids whose fixtures populate this env (its group's evals). + /// The single eval whose overlay files populate this environment. pub eval_ids: Vec, } @@ -53,9 +53,10 @@ pub(super) fn task_env_root_for_run( } pub(super) fn task_run_indices(group: &Group) -> Vec> { - match group.task_runs { - Some(runs) if runs > 1 => (1..=runs).map(Some).collect(), - _ => vec![None], + if group.runs > 1 { + (1..=group.runs).map(Some).collect() + } else { + vec![None] } } @@ -97,14 +98,14 @@ mod tests { Group { id: "g1".into(), eval_ids: vec!["e1".into()], - rationale: "default".into(), - task_runs: None, + rationale: "private codebase".into(), + runs: 1, }, Group { id: "g2".into(), eval_ids: vec!["e2".into()], - rationale: "fixture-conflict: e2 vs e1 at c.json".into(), - task_runs: None, + rationale: "private codebase".into(), + runs: 1, }, ] } @@ -162,8 +163,8 @@ mod tests { let groups = vec![Group { id: "g1".into(), eval_ids: vec!["held-out".into()], - rationale: "metric: diff_scope".into(), - task_runs: Some(2), + rationale: "private codebase".into(), + runs: 2, }]; let targets = env_targets(&EnvLayoutInput { iteration_dir: iter, diff --git a/src/cli/run/orchestrate/git.rs b/src/cli/run/orchestrate/git.rs index b46a106..d33dfa5 100644 --- a/src/cli/run/orchestrate/git.rs +++ b/src/cli/run/orchestrate/git.rs @@ -3,15 +3,13 @@ use std::fs; use std::path::{Path, PathBuf}; -use crate::adapters::registry::all_config_dir_names; -use crate::core::{BASELINE_REF, GitOutput, IsolatedGit, run_git}; -use crate::source::INITIALIZED_BRANCH; - use super::super::RunError; -use super::super::fixtures::fixture_pairs; +use super::super::overlays::overlay_file_pairs; use super::Resolved; use super::envs::{EnvLayoutInput, env_targets}; +use crate::adapters::registry::all_config_dir_names; use crate::core::RunContext; +use crate::core::{BASELINE_REF, GitOutput, IsolatedGit, run_git}; const BASELINE_MESSAGE: &str = "eval-magic task baseline"; const BASELINE_NAME: &str = "eval-magic"; @@ -42,16 +40,9 @@ pub(super) fn initialize_task_repositories( skill_path_b: resolved.skill_path_b.as_deref(), }); for target in targets { - let codebase = resolved.codebase_for(&target.eval_ids)?; + resolved.codebase_for(&target.eval_ids)?; let plan = TaskRepository { root: target.root.clone(), - // A sourced environment already *is* a repository, carrying the - // history the clone brought with it. - sourced: codebase.is_some(), - branch: codebase.map_or_else( - || INITIALIZED_BRANCH.to_string(), - |codebase| codebase.source.branch.clone(), - ), forced_paths: runner_placed_paths(ctx, resolved, &target)?, }; initialize_task_repository(&plan).map_err(|error| { @@ -70,7 +61,7 @@ pub(super) fn initialize_task_repositories( /// A real repository ignores its build output, and a blanket forced add would /// sweep `target/` or `node_modules/` into the baseline. So the baseline add /// respects `.gitignore` and these paths — the harness config directories, and -/// the declared fixture overlay — are forced on top of it. +/// the declared file overlay — are forced on top of it. fn runner_placed_paths( ctx: &RunContext, resolved: &Resolved, @@ -88,7 +79,7 @@ fn runner_placed_paths( else { continue; }; - for (dest, _source) in fixture_pairs(eval, &ctx.skill_subdir)? { + for (dest, _source) in overlay_file_pairs(eval, &ctx.skill_subdir)? { if target.root.join(&dest).exists() { paths.push(dest); } @@ -102,10 +93,6 @@ fn runner_placed_paths( /// One task repository to establish. struct TaskRepository { root: PathBuf, - /// Whether a codebase already put a repository here. A sourced environment - /// keeps its `.git`; a fixture-only one is initialized from nothing. - sourced: bool, - branch: String, forced_paths: Vec, } @@ -113,28 +100,9 @@ fn initialize_task_repository(plan: &TaskRepository) -> Result<(), String> { let root = plan.root.as_path(); let git = IsolatedGit::new()?; - if plan.sourced { - // The clone's history is the point of sourcing a codebase, so this is - // the one case that must not reset `.git`. - strip_remotes(root, &git)?; - } else { - remove_existing_git_dir(root)?; - let template = git.template_dir().to_string_lossy().into_owned(); - run_checked( - &git, - root, - &[ - "init", - "--quiet", - "--initial-branch", - &plan.branch, - "--template", - &template, - ".", - ], - &[], - )?; - } + // Materialization wraps even a historyless directory in a repository. + // Preserve that source history while severing every route back to it. + strip_remotes(root, &git)?; let hooks_dir = root.join(".git/eval-magic-disabled-hooks"); fs::create_dir_all(root.join(".git/info")) @@ -220,26 +188,6 @@ fn strip_remotes(root: &Path, git: &IsolatedGit) -> Result<(), String> { Ok(()) } -fn remove_existing_git_dir(root: &Path) -> Result<(), String> { - let git_dir = root.join(".git"); - let metadata = match fs::symlink_metadata(&git_dir) { - Ok(metadata) => metadata, - Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()), - Err(error) => return Err(format!("could not inspect {}: {error}", git_dir.display())), - }; - if metadata.is_dir() && !metadata.file_type().is_symlink() { - fs::remove_dir_all(&git_dir) - } else { - fs::remove_file(&git_dir) - } - .map_err(|error| { - format!( - "could not reset runner-owned {}: {error}", - git_dir.display() - ) - }) -} - fn verify_task_repository(root: &Path, git: &IsolatedGit) -> Result<(), String> { let top_level = run_checked(git, root, &["rev-parse", "--show-toplevel"], &[])?; let reported = PathBuf::from(String::from_utf8_lossy(&top_level.stdout).trim()); diff --git a/src/cli/run/orchestrate/mod.rs b/src/cli/run/orchestrate/mod.rs index 45d30fc..48a75e9 100644 --- a/src/cli/run/orchestrate/mod.rs +++ b/src/cli/run/orchestrate/mod.rs @@ -85,8 +85,7 @@ impl RunOptions<'_> { struct Resolved { mode: Mode, baseline: Option, - /// Distinct codebases the selection declares, already resolved to a commit. - /// Empty for a fixture-only run, which is what keeps that path unchanged. + /// Distinct codebases backing the selection, already resolved. codebases: Vec, /// The skill under test, resolved but not yet copied. skill: RunSkill, @@ -164,32 +163,18 @@ impl RunCodebase { } impl Resolved { - /// The codebase backing an environment, given the evals sharing it. - /// - /// Production always task-scopes, so an environment carries exactly one - /// eval and the question is trivial. The error covers the planner's older - /// multi-eval grouping, where two evals with different codebases could not - /// share one working tree even in principle. - fn codebase_for(&self, eval_ids: &[String]) -> Result, RunError> { - let mut found: Option<&RunCodebase> = None; - for eval_id in eval_ids { - let codebase = self - .codebases - .iter() - .find(|candidate| candidate.eval_ids.contains(eval_id)); - match (found, codebase) { - (None, next) => found = next, - (Some(previous), Some(next)) if !std::ptr::eq(previous, next) => { - return Err(RunError::msg(format!( - "evals {} share an environment but declare different codebases; \ - give them distinct environments", - eval_ids.join(", ") - ))); - } - _ => {} - } - } - Ok(found) + /// The codebase backing a private eval environment. + fn codebase_for(&self, eval_ids: &[String]) -> Result<&RunCodebase, RunError> { + let [eval_id] = eval_ids else { + return Err(RunError::msg(format!( + "private task environment must contain exactly one eval, found: {}", + eval_ids.join(", ") + ))); + }; + self.codebases + .iter() + .find(|candidate| candidate.eval_ids.contains(eval_id)) + .ok_or_else(|| RunError::msg(format!("eval '{eval_id}' has no resolved codebase"))) } } @@ -263,15 +248,10 @@ pub fn command_run(ctx: &RunContext, opts: &RunOptions) -> Result<(), RunError> } } - // The harness preflight provides supported enhancements automatically (the - // write guard auto-arms), warns about undeclared ones (naming each - // fallback), and adjusts the options — it only rejects genuinely - // contradictory flag combinations. - let preflight = super::util::harness_run_preflight( - opts, - ctx, - super::util::evals_use_transcript_check(&resolved.selected_evals), - )?; + // The harness preflight enforces the runner-ready dispatch/transcript + // contract, provides supported enhancements automatically (the write guard + // auto-arms), and adjusts optional capabilities such as native staging. + let preflight = super::util::harness_run_preflight(opts, ctx)?; for warning in &preflight.warnings { eprintln!("⚠ {warning}"); } diff --git a/src/cli/run/orchestrate/resolve.rs b/src/cli/run/orchestrate/resolve.rs index 4272da9..3ddd5f6 100644 --- a/src/cli/run/orchestrate/resolve.rs +++ b/src/cli/run/orchestrate/resolve.rs @@ -13,8 +13,8 @@ use crate::validation::validate_evals_config; use super::super::RunError; use super::super::dispatch::select_evals; -use super::super::fixtures::{fixture_pairs, setup_file_pairs}; use super::super::grouping::{GroupInput, compute_groups}; +use super::super::overlays::{overlay_file_pairs, setup_file_pairs}; use super::super::util::{condition_names_for, make_run_nonce, next_iteration}; use super::{Resolved, RunCodebase, RunOptions, RunSkill, skills_copy_root}; @@ -36,9 +36,11 @@ fn resolve_codebases( let mut codebases: Vec = Vec::new(); for eval in selected { - let Some(declared) = eval.codebase.as_ref().or(config.codebase.as_ref()) else { - continue; - }; + let declared = eval + .codebase + .as_ref() + .or(config.codebase.as_ref()) + .expect("validated eval has an effective codebase"); if let Some(existing) = codebases .iter_mut() .find(|candidate| &candidate.declared == declared) @@ -207,23 +209,15 @@ pub(super) fn resolve_request(ctx: &RunContext, opts: &RunOptions) -> Result, _>>()?; + // Resolve overlay sources before any environment is staged, so a missing + // declared file fails without leaving a partial iteration behind. + for eval in &selected_evals { + overlay_file_pairs(eval, &ctx.skill_subdir)?; + } let group_inputs: Vec = selected_evals .iter() - .zip(&fixture_pairs_by_eval) - .map(|(ev, fixtures)| GroupInput { + .map(|ev| GroupInput { eval_id: &ev.id, - isolation: ev.isolation, - fixtures, - // Every canonical run publishes final-environment diff metrics, so - // each eval/run needs a private environment. - task_scoped: true, runs: ev.runs.unwrap_or(opts.runs), }) .collect(); diff --git a/src/cli/run/orchestrate/stage.rs b/src/cli/run/orchestrate/stage.rs index c9e8c18..591940e 100644 --- a/src/cli/run/orchestrate/stage.rs +++ b/src/cli/run/orchestrate/stage.rs @@ -13,7 +13,7 @@ use crate::sandbox::teardown_guard; use super::super::RunError; use super::super::dispatch::get_skill_description; -use super::super::fixtures::{FixtureClaims, copy_fixtures}; +use super::super::overlays::copy_overlay_files; use super::super::staging::{ StageSiblingOpts, StageSkillOpts, cleanup_staged_skills, exclude_codebase_skill_sources, register_staged_skill_for_cleanup, skills_dir_for_harness, stage_sibling_skills, @@ -92,7 +92,7 @@ pub(super) fn stage_conditions( } // The environments to stage: one per (group, condition), each with only its - // condition's skill + its group's fixtures. + // condition's skill and its eval's overlay files. let targets = env_targets(&EnvLayoutInput { iteration_dir: &r.iteration_dir, groups: &r.groups, @@ -122,7 +122,7 @@ pub(super) fn stage_conditions( for target in &targets { // Disarm a prior run's guard before re-staging, so a crashed run can't leave // the write-blocking hook armed across runs. Created unconditionally — even - // under --no-stage, each env's fixtures still land here. + // under --no-stage, each env's overlays still land here. teardown_guard(&target.root); let codebase = r.codebase_for(&target.eval_ids)?; @@ -133,7 +133,7 @@ pub(super) fn stage_conditions( // a directory whose name uses the historical staging prefix. cleanup_staged_skills(&target.root, ctx.harness)?; } - if codebase.is_some() && target.root.exists() { + if target.root.exists() { // An explicit `--iteration N` rebuild would otherwise lay a fresh // codebase over the last run's tree, including whatever the previous // agent left behind. Start from nothing instead. @@ -144,25 +144,19 @@ pub(super) fn stage_conditions( // iteration's single cached materialization of it — a local clone while // the host allows the hard link, a plain copy otherwise — so `--runs 10` // against a real repository costs one checkout, not ten copies. - if let Some(codebase) = codebase { - let source_tree = materialize_codebase(&r.iteration_dir, codebase, &mut materialized)?; - crate::source::provision_env(&codebase.source, &source_tree, &target.root) - .map_err(|error| RunError::msg(error.to_string()))?; - } else { - fs::create_dir_all(&target.root)?; - } + let source_tree = materialize_codebase(&r.iteration_dir, codebase, &mut materialized)?; + crate::source::provision_env(&codebase.source, &source_tree, &target.root) + .map_err(|error| RunError::msg(error.to_string()))?; - if let Some(codebase) = codebase { - let inventoried = super::shadow_preflight::scan_codebase_skill_sources( - &target.root, - ctx.harness, - &evaluated_names, - ); - if codebase.declared.exclude_skill_sources() { - exclude_codebase_skill_sources(&target.root, &ctx.skill_name, ctx.harness)?; - } else if !inventoried.is_empty() { - codebase_shadow_sources.insert(target.root.clone(), inventoried); - } + let inventoried = super::shadow_preflight::scan_codebase_skill_sources( + &target.root, + ctx.harness, + &evaluated_names, + ); + if codebase.declared.exclude_skill_sources() { + exclude_codebase_skill_sources(&target.root, &ctx.skill_name, ctx.harness)?; + } else if !inventoried.is_empty() { + codebase_shadow_sources.insert(target.root.clone(), inventoried); } if !opts.no_stage && ctx.stage_siblings { @@ -241,13 +235,10 @@ pub(super) fn stage_conditions( } } - // Copy this env's group's fixtures. Claims are per env (each env is - // independent); grouping has already routed clobbering evals into separate - // groups, so within one env the same-source/idempotent rule never trips. - let mut claims = FixtureClaims::new(); + // Apply this eval's authored files on top of the staged codebase. for eval_id in &target.eval_ids { if let Some(ev) = r.selected_evals.iter().find(|e| &e.id == eval_id) { - copy_fixtures(ev, &skills.join(&ctx.skill_name), &target.root, &mut claims)?; + copy_overlay_files(ev, &skills.join(&ctx.skill_name), &target.root)?; } } diff --git a/src/cli/run/overlays.rs b/src/cli/run/overlays.rs new file mode 100644 index 0000000..bd5d3d1 --- /dev/null +++ b/src/cli/run/overlays.rs @@ -0,0 +1,248 @@ +//! Resolve and apply eval-authored files on top of a staged codebase. + +use std::fs; +use std::path::Path; + +use crate::core::fs::copy_entry_materialized; +use crate::core::{AssertionCommandCheck, Eval}; + +use super::RunError; + +/// True for a path that is absolute under either supported portable-data shape. +fn is_absolute_on_any_platform(raw: &str) -> bool { + let path = Path::new(raw); + path.has_root() + || raw.starts_with('\\') + || matches!( + path.components().next(), + Some(std::path::Component::Prefix(_)) + ) +} + +/// Reject a task-relative path that can escape the private environment or +/// replace runner-owned root Git metadata. +fn validate_task_relative_path(raw: &str) -> Result<(), RunError> { + let path = Path::new(raw); + let escapes = is_absolute_on_any_platform(raw) + || path + .components() + .any(|component| matches!(component, std::path::Component::ParentDir)); + if escapes { + return Err(RunError::msg(format!( + "task-relative path must be relative and stay within the environment: {raw}" + ))); + } + let first_normal = path.components().find_map(|component| match component { + std::path::Component::Normal(value) => Some(value.to_string_lossy()), + _ => None, + }); + if first_normal.is_some_and(|component| component.eq_ignore_ascii_case(".git")) { + return Err(RunError::msg(format!( + "task-relative path uses reserved runner-owned Git metadata at task root: {raw}" + ))); + } + Ok(()) +} + +/// Reject an overlay source root that escapes `/evals/`. +fn validate_files_root_rel(root: &str) -> Result<(), RunError> { + let path = Path::new(root); + let escapes = is_absolute_on_any_platform(root) + || path + .components() + .any(|component| matches!(component, std::path::Component::ParentDir)); + if escapes { + return Err(RunError::msg(format!( + "files_root must be relative and stay within the skill's evals directory: {root}" + ))); + } + Ok(()) +} + +/// Resolve an eval's overlay files without copying them. +pub fn overlay_file_pairs( + eval: &Eval, + skill_dir: &Path, +) -> Result, RunError> { + let mut source_root = skill_dir.join("evals"); + if let Some(root) = eval.files_root.as_deref() { + validate_files_root_rel(root)?; + source_root = source_root.join(root); + } + let Some(files) = eval.files.as_ref().filter(|files| !files.is_empty()) else { + return Ok(Vec::new()); + }; + let mut pairs = Vec::with_capacity(files.len()); + for file in files { + validate_task_relative_path(file)?; + let source = source_root.join(file); + if !source.exists() { + return Err(RunError::msg(format!( + "overlay file not found: {}", + source.display() + ))); + } + pairs.push((file.clone(), source.to_string_lossy().into_owned())); + } + Ok(pairs) +} + +/// Resolve a command check's held-out setup paths without copying them. +pub fn setup_file_pairs( + check: &AssertionCommandCheck, + skill_dir: &Path, +) -> Result, RunError> { + let Some(files) = check.setup_files.as_ref().filter(|files| !files.is_empty()) else { + return Ok(Vec::new()); + }; + let mut pairs = Vec::with_capacity(files.len()); + for file in files { + validate_task_relative_path(file)?; + let source = skill_dir.join("evals").join(file); + if !source.exists() { + return Err(RunError::msg(format!( + "command-check setup file not found: {}", + source.display() + ))); + } + pairs.push((file.clone(), source.to_string_lossy().into_owned())); + } + Ok(pairs) +} + +/// Copy an eval's overlay files into its private task environment. +pub fn copy_overlay_files( + eval: &Eval, + skill_dir: &Path, + env_root: &Path, +) -> Result, RunError> { + let pairs = overlay_file_pairs(eval, skill_dir)?; + let mut copied = Vec::with_capacity(pairs.len()); + for (dest, source) in &pairs { + let destination = env_root.join(dest); + if let Some(parent) = destination.parent() { + fs::create_dir_all(parent)?; + } + copy_entry_materialized(Path::new(source), &destination)?; + copied.push(dest.clone()); + } + Ok(copied) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn eval_with_files(files: &[&str]) -> Eval { + Eval { + id: "e1".to_string(), + prompt: "p".to_string(), + expected_output: "o".to_string(), + files: Some(files.iter().map(|file| (*file).to_string()).collect()), + files_root: None, + assertions: None, + skill_should_trigger: None, + runs: None, + turns: None, + codebase: None, + responder: None, + guard: None, + } + } + + #[test] + fn overlay_pairs_resolve_without_copying() { + let tmp = tempfile::TempDir::new().unwrap(); + let skill_dir = tmp.path().join("skill"); + let evals = skill_dir.join("evals"); + fs::create_dir_all(evals.join("data")).unwrap(); + fs::write(evals.join("config.json"), "cfg").unwrap(); + fs::write(evals.join("data/x.json"), "xx").unwrap(); + + let pairs = overlay_file_pairs( + &eval_with_files(&["config.json", "data/x.json"]), + &skill_dir, + ) + .unwrap(); + + assert_eq!( + pairs, + vec![ + ( + "config.json".to_string(), + evals.join("config.json").to_string_lossy().into_owned() + ), + ( + "data/x.json".to_string(), + evals.join("data/x.json").to_string_lossy().into_owned() + ), + ] + ); + assert!(!tmp.path().join("env").exists()); + } + + #[test] + fn overlay_pairs_validate_source_root_and_files() { + let tmp = tempfile::TempDir::new().unwrap(); + let skill_dir = tmp.path().join("skill"); + fs::create_dir_all(skill_dir.join("evals")).unwrap(); + + let mut escaping_root = eval_with_files(&[]); + escaping_root.files_root = Some("../outside".to_string()); + let error = overlay_file_pairs(&escaping_root, &skill_dir) + .unwrap_err() + .to_string(); + assert!(error.contains("files_root"), "error was: {error}"); + + let missing = overlay_file_pairs(&eval_with_files(&["missing.txt"]), &skill_dir) + .unwrap_err() + .to_string(); + assert!(missing.contains("overlay file not found"), "{missing}"); + } + + #[test] + fn task_paths_reject_escapes_and_root_git_metadata() { + for bad in [ + "../escape.txt", + "/etc/passwd", + "a/../../b.txt", + r"\etc\passwd", + ] { + let error = validate_task_relative_path(bad).unwrap_err().to_string(); + assert!(error.contains("relative"), "{bad}: {error}"); + } + for bad in [".git", ".git/config", "./.GIT/config"] { + let error = validate_task_relative_path(bad).unwrap_err().to_string(); + assert!(error.contains("reserved"), "{bad}: {error}"); + } + validate_task_relative_path("vendor/.git/config").unwrap(); + } + + #[test] + fn copy_overlay_files_preserves_task_relative_paths() { + let tmp = tempfile::TempDir::new().unwrap(); + let skill_dir = tmp.path().join("skill"); + let evals = skill_dir.join("evals"); + fs::create_dir_all(evals.join("data")).unwrap(); + fs::write(evals.join("config.json"), "cfg").unwrap(); + fs::write(evals.join("data/x.json"), "xx").unwrap(); + let env_root = tmp.path().join("env"); + + let copied = copy_overlay_files( + &eval_with_files(&["config.json", "data/x.json"]), + &skill_dir, + &env_root, + ) + .unwrap(); + + assert_eq!( + fs::read_to_string(env_root.join("config.json")).unwrap(), + "cfg" + ); + assert_eq!( + fs::read_to_string(env_root.join("data/x.json")).unwrap(), + "xx" + ); + assert_eq!(copied, vec!["config.json", "data/x.json"]); + } +} diff --git a/src/cli/run/steps.rs b/src/cli/run/steps.rs index 1d708bc..58ac8df 100644 --- a/src/cli/run/steps.rs +++ b/src/cli/run/steps.rs @@ -20,7 +20,6 @@ use crate::core::Harness; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum StepKind { RecordRuns, - FillTranscripts, DetectStrayWrites, Grade { finalize: bool }, Aggregate, @@ -64,13 +63,11 @@ impl StepParams<'_> { } } -/// The ingest chain: record-runs → fill-transcripts → detect-stray-writes → -/// grade. Each stage reads its transcript from each task's `outputs/` events -/// file. +/// The ingest chain: record-runs → detect-stray-writes → grade. Each stage +/// reads the artifacts written by its predecessor. pub fn build_ingest_commands(p: &StepParams) -> Vec { vec![ p.step("record-runs", StepKind::RecordRuns), - p.step("fill-transcripts", StepKind::FillTranscripts), p.step("detect-stray-writes", StepKind::DetectStrayWrites), p.step("grade", StepKind::Grade { finalize: false }), ] @@ -116,22 +113,16 @@ mod tests { } #[test] - fn ingest_runs_record_fill_stray_grade_in_order() { + fn ingest_runs_record_stray_grade_in_order() { let steps = build_ingest_commands(¶ms()); assert_eq!( steps.iter().map(|s| s.label).collect::>(), - vec![ - "record-runs", - "fill-transcripts", - "detect-stray-writes", - "grade" - ] + vec!["record-runs", "detect-stray-writes", "grade"] ); assert_eq!( steps.iter().map(|s| s.kind).collect::>(), vec![ StepKind::RecordRuns, - StepKind::FillTranscripts, StepKind::DetectStrayWrites, StepKind::Grade { finalize: false }, ] @@ -155,12 +146,7 @@ mod tests { }); assert_eq!( steps.iter().map(|s| s.label).collect::>(), - vec![ - "record-runs", - "fill-transcripts", - "detect-stray-writes", - "grade" - ] + vec!["record-runs", "detect-stray-writes", "grade"] ); assert!( steps diff --git a/src/cli/run/util.rs b/src/cli/run/util.rs index f387c19..12cc079 100644 --- a/src/cli/run/util.rs +++ b/src/cli/run/util.rs @@ -9,7 +9,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; use crate::adapters::adapter_for; use crate::adapters::registry::has_embedded_layer; -use crate::core::{Assertion, Eval, Harness, Mode, RunContext}; +use crate::core::{Harness, Mode, RunContext}; use super::RunError; use super::orchestrate::RunOptions; @@ -65,20 +65,18 @@ pub(crate) fn resolve_plan_mode_profile() -> &'static str { } /// The harness preflight verdict: possibly-adjusted run options plus the -/// warnings to print, each naming the fallback that carries the run. +/// warnings to print for optional capabilities using a fallback. pub(crate) struct HarnessPreflight<'a> { pub opts: RunOptions<'a>, pub warnings: Vec, } -/// Check the run options against the selected harness's declared enhancements -/// — the #126 model: each supported enhancement is provided automatically -/// (the write guard auto-arms when the harness declares one and staging is -/// active), a missing enhancement *warns* naming its fallback, and the run -/// continues degraded. Only genuinely contradictory flag combinations -/// (options the harness declares incompatible with `--no-stage`) and an -/// explicit `--guard` on a harness defined by user-supplied descriptors alone -/// (guards are embedded-only) stay errors. +/// Check the run options against the selected harness's runner contract and +/// optional enhancements. Dispatch and transcript recovery are mandatory. +/// Other supported enhancements are provided automatically (the write guard +/// auto-arms when declared and staging is active), while optional omissions +/// warn when a lower-fidelity fallback is used. Contradictory flag combinations +/// and an explicit `--guard` on a user-descriptor-only harness remain errors. /// /// Adjustments: `opts.guard` arrives tri-state (`None` = auto) and leaves /// resolved to `Some`; a harness without a `skills_dir` forces `--no-stage` @@ -86,12 +84,25 @@ pub(crate) struct HarnessPreflight<'a> { pub(crate) fn harness_run_preflight<'a>( opts: &RunOptions<'a>, ctx: &RunContext, - uses_transcript_check: bool, ) -> Result, RunError> { let adapter = adapter_for(ctx.harness); let capabilities = adapter.run_capabilities(); let label = harness_label(ctx.harness); + if !adapter.has_dispatch_recipes() { + return Err(RunError::msg(format!( + "--harness {label} declares no dispatch exec template, so it is not runner-ready. \ + Add `[dispatch].exec_template` to the descriptor (see `eval-magic docs byoh`)." + ))); + } + if adapter.cli_events_filename().is_none() { + return Err(RunError::msg(format!( + "--harness {label} declares no transcript parser, so it is not runner-ready. Add a \ + `[transcript]` parser or extract mapping that recovers the final response (see \ + `eval-magic docs byoh`)." + ))); + } + // Contradictory-flag declarations stay hard errors: the harness's staging // mechanism conflicts with these options, so no fallback can honor them. let mut unsupported: Vec<&str> = Vec::new(); @@ -180,23 +191,7 @@ pub(crate) fn harness_run_preflight<'a>( } } - if adapter.cli_events_filename().is_none() { - warnings.push(if uses_transcript_check { - format!( - "--harness {label} declares no transcript parser — transcript_check assertions \ - will grade as unverifiable and llm_judge carries the grading; tokens/duration \ - go unrecorded. Recover each final message into outputs/final-message.md \ - (see RUNBOOK.md)." - ) - } else { - format!( - "--harness {label} declares no transcript parser — tokens/duration go \ - unrecorded and run records are assembled from each task's \ - outputs/final-message.md (see RUNBOOK.md)." - ) - }); - } - if adapter.cli_events_filename().is_some() && !adapter.surfaces_permission_denials() { + if !adapter.surfaces_permission_denials() { warnings.push(format!( "--harness {label} cannot tell a permission-denied tool result from an ordinary tool \ error — a dispatch whose calls were refused (and so fell back to static reasoning) \ @@ -213,28 +208,9 @@ pub(crate) fn harness_run_preflight<'a>( default model." )); } - if !adapter.has_dispatch_recipes() { - warnings.push(format!( - "--harness {label} declares no dispatch exec template — `eval-magic dispatch` \ - has no command to run for these tasks and will fail. Add \ - `[dispatch].exec_template` to the descriptor (see `eval-magic docs byoh`)." - )); - } Ok(HarnessPreflight { opts, warnings }) } -/// Whether any selected eval declares a `transcript_check` assertion — scopes -/// the no-transcript-parser preflight warning to the eval configs it actually -/// affects. -pub(crate) fn evals_use_transcript_check(evals: &[Eval]) -> bool { - evals.iter().any(|e| { - e.assertions - .iter() - .flatten() - .any(|a| matches!(a, Assertion::TranscriptCheck(_))) - }) -} - /// A per-run nonce (`-<6 hex>`) that namespaces dispatch /// descriptions so they stay unique across iterations of the same skill. With no /// RNG crate, the low bits of the sub-millisecond clock supply the entropy — @@ -311,7 +287,7 @@ mod tests { guard: Some(true), ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(true)); assert!(preflight.warnings.is_empty(), "{:?}", preflight.warnings); } @@ -321,7 +297,7 @@ mod tests { // No guard flag at all: the enhancement is detected and provided // automatically (#126), with no warning to acknowledge. let (_t, ctx) = ctx_for(Harness::resolve("claude-code").unwrap()); - let preflight = harness_run_preflight(&RunOptions::default(), &ctx, false).unwrap(); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(true), "auto-arm resolves to on"); assert!(preflight.warnings.is_empty(), "{:?}", preflight.warnings); } @@ -335,7 +311,7 @@ mod tests { no_stage: true, ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(false)); assert!(preflight.warnings.is_empty(), "{:?}", preflight.warnings); } @@ -348,7 +324,7 @@ mod tests { // reachable on user-only harnesses now — pinned in tests/run/byoh.rs. for name in ["claude-code", "codex", "opencode"] { let (_t, ctx) = ctx_for(Harness::resolve(name).unwrap()); - let preflight = harness_run_preflight(&RunOptions::default(), &ctx, false).unwrap(); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(true), "{name} auto-arms"); assert!( !preflight.warnings.iter().any(|w| w.contains("write guard")), @@ -366,7 +342,7 @@ mod tests { guard: Some(false), ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(false)); assert!( !preflight.warnings.iter().any(|w| w.contains("write guard")), @@ -384,7 +360,7 @@ mod tests { no_stage: true, ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(false)); let warning = preflight .warnings @@ -408,7 +384,7 @@ mod tests { guard: Some(true), ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert_eq!(preflight.opts.guard, Some(true)); assert!( !preflight.warnings.iter().any(|w| w.contains("--guard")), @@ -418,25 +394,17 @@ mod tests { } #[test] - fn opencode_declares_a_transcript_parser_so_no_transcript_warning_fires() { - // The transcript-less warning's content and transcript_check scoping - // are pinned by the byoh integration tests (a user descriptor without - // a parser); at the unit level every built-in is transcript-wired, so - // this pins that wiring: no transcript warning for opencode, whatever - // the eval config uses. - for uses_transcript_check in [true, false] { - let (_t, ctx) = ctx_for(Harness::resolve("opencode").unwrap()); - let preflight = - harness_run_preflight(&RunOptions::default(), &ctx, uses_transcript_check).unwrap(); - assert!( - !preflight - .warnings - .iter() - .any(|w| w.contains("transcript parser")), - "no transcript-parser warning: {:?}", - preflight.warnings - ); - } + fn opencode_is_runner_ready() { + let (_t, ctx) = ctx_for(Harness::resolve("opencode").unwrap()); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); + assert!( + !preflight + .warnings + .iter() + .any(|w| w.contains("transcript parser")), + "runner-ready built-ins do not warn: {:?}", + preflight.warnings + ); } #[test] @@ -447,7 +415,7 @@ mod tests { // from an ordinary tool error. for name in ["claude-code", "codex", "opencode"] { let (_t, ctx) = ctx_for(Harness::resolve(name).unwrap()); - let preflight = harness_run_preflight(&RunOptions::default(), &ctx, false).unwrap(); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); assert!( !preflight .warnings @@ -459,59 +427,13 @@ mod tests { } } - #[test] - fn evals_use_transcript_check_detects_the_assertion_type() { - use crate::core::{Assertion, AssertionLlmJudge, AssertionTranscriptCheck, Eval}; - - fn eval_with(assertions: Option>) -> Eval { - Eval { - id: "e1".into(), - prompt: "p".into(), - expected_output: "o".into(), - files: None, - files_root: None, - assertions, - skill_should_trigger: None, - runs: None, - isolation: None, - turns: None, - codebase: None, - responder: None, - guard: None, - } - } - - let transcript = Assertion::TranscriptCheck(AssertionTranscriptCheck { - id: "a1".into(), - check: "ran tests".into(), - pattern: None, - must_precede: None, - }); - let judge = Assertion::LlmJudge(AssertionLlmJudge { - id: "a2".into(), - rubric: "r".into(), - model: None, - samples: None, - }); - - assert!(evals_use_transcript_check(&[eval_with(Some(vec![ - judge.clone(), - transcript - ]))])); - assert!(!evals_use_transcript_check(&[ - eval_with(Some(vec![judge])), - eval_with(None) - ])); - assert!(!evals_use_transcript_check(&[])); - } - #[test] fn wired_built_ins_do_not_warn_about_dispatch_recipes() { // The dispatch-recipe warning for a dispatchless harness is pinned on // a user descriptor in tests/run/byoh.rs; every built-in wires recipes. for name in ["claude-code", "cline", "codex", "opencode"] { let (_t, ctx) = ctx_for(Harness::resolve(name).unwrap()); - let preflight = harness_run_preflight(&RunOptions::default(), &ctx, false).unwrap(); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); assert!( !preflight .warnings @@ -533,7 +455,7 @@ mod tests { agent_model: Some("some-model"), ..Default::default() }; - let preflight = harness_run_preflight(&opts, &ctx, false).unwrap(); + let preflight = harness_run_preflight(&opts, &ctx).unwrap(); assert!( !preflight.warnings.iter().any(|w| w.contains("model flag")), "{:?}", @@ -544,7 +466,7 @@ mod tests { #[test] fn no_model_warning_when_no_models_are_requested() { let (_t, ctx) = ctx_for(Harness::resolve("opencode").unwrap()); - let preflight = harness_run_preflight(&RunOptions::default(), &ctx, false).unwrap(); + let preflight = harness_run_preflight(&RunOptions::default(), &ctx).unwrap(); assert!( !preflight.warnings.iter().any(|w| w.contains("model flag")), "{:?}", diff --git a/src/core/fs.rs b/src/core/fs.rs index 213cf40..028ef8c 100644 --- a/src/core/fs.rs +++ b/src/core/fs.rs @@ -8,7 +8,7 @@ //! //! [`copy_entry_materialized`] is the one way to copy here, and it resolves //! symlinks into their target's content rather than mirroring them. Every -//! destination in this tree wants that: staging and fixtures copy *into* an +//! destination in this tree wants that: staging and overlays copy *into* an //! isolated task env, where a preserved link would point back out of the //! sandbox, and a snapshot must freeze content so a later run compares against //! what was captured rather than whatever the link now points at. diff --git a/src/core/types.rs b/src/core/types.rs index d503bec..c54e97d 100644 --- a/src/core/types.rs +++ b/src/core/types.rs @@ -102,7 +102,7 @@ pub struct Eval { pub expected_output: String, #[serde(skip_serializing_if = "Option::is_none")] pub files: Option>, - /// Optional source base under `/evals/`. Fixture destinations remain + /// Optional source base under `/evals/`. Overlay destinations remain /// the task-relative paths declared in [`Self::files`]. #[serde(skip_serializing_if = "Option::is_none")] pub files_root: Option, @@ -116,11 +116,6 @@ pub struct Eval { /// to the flag's value (1 unless raised). #[serde(skip_serializing_if = "Option::is_none")] pub runs: Option, - /// Legacy isolation hint retained for config compatibility. Canonical runs - /// already give every dispatch a private environment for diff-scope capture, - /// so `shared` and `isolated` currently have the same effective isolation. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub isolation: Option, /// Ordered scripted user follow-ups. Absence preserves one-shot dispatch. #[serde(skip_serializing_if = "Option::is_none")] pub turns: Option>, @@ -194,14 +189,6 @@ impl ResponderPolicy { } } -/// Legacy per-eval isolation hint. Every new run is task-scoped regardless. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum Isolation { - Shared, - Isolated, -} - /// Where a task environment's contents come from: a Git repository at an /// explicit ref, or a directory on this host. /// @@ -489,8 +476,8 @@ pub struct ConditionsRecord { /// Operator-declared provenance label, surfaced in `BASELINE.md` on promote. #[serde(skip_serializing_if = "Option::is_none")] pub label: Option, - /// Codebases the iteration's environments were built from. Empty for a - /// fixture-only iteration, which keeps its `conditions.json` unchanged. + /// Codebases the iteration's environments were built from. Empty only when + /// reading a historical iteration that predates codebase provenance. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub codebases: Vec, /// The skill under test, as the run resolved and copied it. Appended last, @@ -547,7 +534,7 @@ pub struct RunRecord { /// The codebase this run's environment was built from. Grading reads /// `run.json` and nothing else, so a result can only be tied to a tree if /// the record names one. Appended last, and omitted when absent, so a - /// fixture-only record serializes as it always did. + /// historical records that predate codebase provenance still deserialize. #[serde(default, skip_serializing_if = "Option::is_none")] pub codebase: Option, /// The skill under test this run staged. Grading reads `run.json` and nothing @@ -801,7 +788,6 @@ mod tests { assertions: None, skill_should_trigger: None, runs: None, - isolation: None, turns: None, codebase: None, responder: None, @@ -816,32 +802,6 @@ mod tests { assert!(out.get("isolation").is_none()); } - #[test] - fn isolation_round_trips_snake_case() { - let eval = Eval { - id: "e1".into(), - prompt: "p".into(), - expected_output: "o".into(), - files: None, - files_root: None, - assertions: None, - skill_should_trigger: None, - runs: None, - isolation: Some(Isolation::Isolated), - turns: None, - codebase: None, - responder: None, - guard: None, - }; - let out = serde_json::to_value(&eval).unwrap(); - assert_eq!( - out.get("isolation"), - Some(&Value::String("isolated".into())) - ); - let back: Eval = serde_json::from_value(out).unwrap(); - assert_eq!(back.isolation, Some(Isolation::Isolated)); - } - #[test] fn run_record_skill_path_null_emitted() { let rec = RunRecord { @@ -948,7 +908,7 @@ mod tests { fn codebase_use_records_effective_skill_source_exclusion() { let value = serde_json::json!({ "kind": "path", - "source": "../fixture", + "source": "../project", "branch": "work", "exclude_skill_sources": true, "evals": ["e1"] diff --git a/src/pipeline/aggregate.rs b/src/pipeline/aggregate.rs index 244dcfd..f5dce92 100644 --- a/src/pipeline/aggregate.rs +++ b/src/pipeline/aggregate.rs @@ -138,8 +138,8 @@ pub struct Benchmark { pub assertions: Value, /// Codebases the compared conditions ran against, echoed from /// `conditions.json` so a published benchmark names the trees it measured - /// without a reader having to hold two artifacts side by side. Empty for a - /// fixture-only iteration, which keeps its benchmark unchanged. + /// without a reader having to hold two artifacts side by side. Empty only + /// when reading a historical iteration that predates codebase provenance. #[serde(skip_serializing_if = "Vec::is_empty")] pub codebases: Vec, /// The skill under test, echoed from `conditions.json` for the same reason diff --git a/src/pipeline/detect_stray_writes.rs b/src/pipeline/detect_stray_writes.rs index 67e7939..41c65a9 100644 --- a/src/pipeline/detect_stray_writes.rs +++ b/src/pipeline/detect_stray_writes.rs @@ -608,11 +608,7 @@ mod tests { #[test] fn a_relative_redirection_resolves_from_the_task_environment() { let f = detect_stray_writes( - &[inv( - "Bash", - json!({"command": "printf done > final-message.md"}), - 0, - )], + &[inv("Bash", json!({"command": "printf done > notes.md"}), 0)], ALLOWED_ROOT, Path::new(ALLOWED_ROOT), ); diff --git a/src/pipeline/diff_scope/tests.rs b/src/pipeline/diff_scope/tests.rs index adbf3ca..44e70bf 100644 --- a/src/pipeline/diff_scope/tests.rs +++ b/src/pipeline/diff_scope/tests.rs @@ -101,7 +101,7 @@ fn measurement_counts_all_task_changes_except_framework_outputs() { fs::remove_file(eval_root.join("src/deleted.txt")).unwrap(); fs::write(eval_root.join("framework.txt"), "after\n").unwrap(); fs::write(eval_root.join("notes.txt"), "one\ntwo\n").unwrap(); - fs::write(outputs_dir.join("final-message.md"), "ignored\n").unwrap(); + fs::write(outputs_dir.join("harness-events.jsonl"), "ignored\n").unwrap(); fs::write( eval_root.join(".eval-magic-outputs/agent-created.txt"), "also ignored\n", diff --git a/src/pipeline/fill_transcripts.rs b/src/pipeline/fill_transcripts.rs deleted file mode 100644 index 1d3c12f..0000000 --- a/src/pipeline/fill_transcripts.rs +++ /dev/null @@ -1,343 +0,0 @@ -//! Stage 2 — `fill-transcripts`. -//! -//! Walks the iteration's `eval-*` -//! directories and, for each `(eval, condition)` `run.json`, populates -//! `tool_invocations` from the events file the harness CLI wrote under the task's -//! `outputs_dir` (e.g. Codex's `codex-events.jsonl`, Claude Code's -//! `claude-events.jsonl`). Records that already carry invocations are skipped -//! unless `overwrite`. - -use std::collections::HashMap; -use std::fs; -use std::path::Path; - -use serde::Deserialize; - -use crate::adapters::adapter_for; -use crate::core::fs::write_json; -use crate::core::{ConditionsRecord, Harness, RunRecord, ToolInvocation}; -use crate::pipeline::error::PipelineError; -use crate::pipeline::slots::{run_key, run_slots}; -use crate::validation::{SchemaName, validate_against_schema}; - -/// Tally of what fill-transcripts did across the iteration's runs. -#[derive(Debug, Default, Clone, PartialEq, Eq)] -pub struct FillTranscriptsResult { - pub filled: usize, - pub skipped: usize, - pub missing: usize, -} - -/// The `dispatch.json` fields fill-transcripts reads back. -#[derive(Debug, Deserialize)] -struct DispatchEnvelope { - tasks: Option>, -} - -#[derive(Debug, Deserialize)] -struct DispatchRef { - eval_id: String, - condition: String, - #[serde(default)] - run_index: Option, - #[serde(default)] - outputs_dir: Option, -} - -/// Populate `tool_invocations` for every `run.json` under `iteration_dir`. See -/// the module docs for the transcript sources and overwrite semantics. -pub fn fill_transcripts( - iteration_dir: &Path, - harness: Harness, - overwrite: bool, -) -> Result { - let conditions_path = iteration_dir.join("conditions.json"); - if !conditions_path.exists() { - return Err(PipelineError::Message(format!( - "missing: {}", - conditions_path.display() - ))); - } - let conditions: ConditionsRecord = - serde_json::from_str(&fs::read_to_string(&conditions_path)?)?; - let condition_names: Vec = conditions - .conditions - .iter() - .map(|c| c.name.clone()) - .collect(); - - let outputs_by_key = outputs_dirs_by_key(iteration_dir); - - let mut result = FillTranscriptsResult::default(); - for entry in fs::read_dir(iteration_dir)? { - let entry = entry?; - let dir_name = entry.file_name().to_string_lossy().into_owned(); - let Some(eval_id) = dir_name.strip_prefix("eval-") else { - continue; - }; - - for cond in &condition_names { - let cond_dir = iteration_dir.join(&dir_name).join(cond); - for slot in run_slots(&cond_dir) { - let run_path = slot.dir.join("run.json"); - if !run_path.exists() { - continue; - } - - let source = run_path.to_string_lossy(); - let mut run: RunRecord = validate_against_schema( - SchemaName::RunRecord, - &serde_json::from_str(&fs::read_to_string(&run_path)?)?, - &source, - )?; - - if run.conversation.is_some() { - // record-runs already populated the authoritative global - // tool order from conversation.json. An empty list is a - // legitimate tool-free conversation, not a missing fill. - result.skipped += 1; - continue; - } - - if !run.tool_invocations.is_empty() && !overwrite { - result.skipped += 1; - continue; - } - - let outputs_dir = outputs_by_key - .get(&run_key(eval_id, cond, slot.run_index)) - .cloned() - .unwrap_or_else(|| slot.dir.join("outputs").to_string_lossy().into_owned()); - - let Some(invocations) = invocations_for_run(harness, Path::new(&outputs_dir)) - else { - result.missing += 1; - continue; - }; - - run.tool_invocations = invocations; - write_json(&run_path, &run)?; - result.filled += 1; - } - } - } - - Ok(result) -} - -/// Map `":[:r]"` → the task's `outputs_dir` from -/// `dispatch.json`. Empty when the file is absent or malformed (callers fall -/// back to convention). -fn outputs_dirs_by_key(iteration_dir: &Path) -> HashMap { - let mut out = HashMap::new(); - if let Ok(raw) = fs::read_to_string(iteration_dir.join("dispatch.json")) - && let Ok(env) = serde_json::from_str::(&raw) - { - for t in env.tasks.unwrap_or_default() { - if let Some(dir) = t.outputs_dir { - out.insert(run_key(&t.eval_id, &t.condition, t.run_index), dir); - } - } - } - out -} - -/// Parse the invocations for one run: read the events file the harness CLI wrote -/// under `outputs_dir` (e.g. Codex's `codex-events.jsonl`, Claude Code's -/// `claude-events.jsonl`). Returns `None` when no events file is found. -fn invocations_for_run(harness: Harness, outputs_dir: &Path) -> Option> { - let events_path = outputs_dir.join(adapter_for(harness).cli_events_filename()?); - if !events_path.exists() { - return None; - } - adapter_for(harness).parse_cli_events(&events_path).ok() -} - -#[cfg(test)] -mod tests { - use super::*; - use serde_json::{Value, json}; - use std::path::PathBuf; - use tempfile::TempDir; - - fn write_dispatch(iteration_dir: &Path, tasks: Value) { - fs::create_dir_all(iteration_dir).unwrap(); - fs::write( - iteration_dir.join("dispatch.json"), - serde_json::to_string_pretty(&json!({"run_nonce": "abc123", "tasks": tasks})).unwrap(), - ) - .unwrap(); - } - - fn jsonl(lines: &[Value]) -> String { - let body = lines - .iter() - .map(|l| l.to_string()) - .collect::>() - .join("\n"); - format!("{body}\n") - } - - fn write_run_record(path: &Path, tool_invocations: Value) { - let record = json!({ - "eval_id": "crash", - "condition": "with_skill", - "skill_path": "/skill/SKILL.md", - "prompt": "Fix it", - "files": [], - "final_message": "Done.", - "tool_invocations": tool_invocations, - "total_tokens": Value::Null, - "duration_ms": Value::Null, - }); - fs::write(path, serde_json::to_string_pretty(&record).unwrap()).unwrap(); - } - - // --- fillTranscripts --- - - #[test] - fn fills_a_claude_run_record_from_outputs_events() { - let root = TempDir::new().unwrap(); - let iteration_dir: PathBuf = root.path().join("iter-claude-fill"); - let cond_dir = iteration_dir.join("eval-crash").join("with_skill"); - let outputs_dir = cond_dir.join("outputs"); - fs::create_dir_all(&outputs_dir).unwrap(); - let run_path = cond_dir.join("run.json"); - write_run_record(&run_path, json!([])); - fs::write( - iteration_dir.join("conditions.json"), - json!({ - "mode": "new-skill", - "conditions": [{"name": "with_skill", "skill_path": "/skill/SKILL.md"}], - "timestamp": "2026-06-07T00:00:00.000Z", - "harness": "claude-code" - }) - .to_string(), - ) - .unwrap(); - write_dispatch( - &iteration_dir, - json!([{"eval_id": "crash", "condition": "with_skill", "outputs_dir": outputs_dir.to_string_lossy()}]), - ); - // `claude -p` stream-json: assistant tool_use + user tool_result + result. - fs::write( - outputs_dir.join("claude-events.jsonl"), - jsonl(&[ - json!({"type": "assistant", "message": {"id": "msg_1", "role": "assistant", "content": [{"type": "tool_use", "id": "toolu_1", "name": "Bash", "input": {"command": "bun test"}}]}}), - json!({"type": "user", "message": {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": "ok"}]}}), - json!({"type": "result", "subtype": "success", "is_error": false, "result": "Done", "duration_ms": 10, "usage": {"input_tokens": 1, "output_tokens": 1, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}}), - ]), - ) - .unwrap(); - - let result = fill_transcripts( - &iteration_dir, - Harness::resolve("claude-code").unwrap(), - false, - ) - .unwrap(); - assert_eq!(result.filled, 1); - assert_eq!(result.missing, 0); - - let updated: RunRecord = - serde_json::from_str(&fs::read_to_string(&run_path).unwrap()).unwrap(); - assert_eq!( - serde_json::to_value(&updated.tool_invocations).unwrap(), - json!([{"name": "Bash", "ordinal": 0, "args": {"command": "bun test"}, "result": "ok"}]) - ); - } - - #[test] - fn fills_a_codex_run_record_from_outputs_events() { - let root = TempDir::new().unwrap(); - let iteration_dir: PathBuf = root.path().join("iter-codex-fill"); - let cond_dir = iteration_dir.join("eval-crash").join("with_skill"); - let outputs_dir = cond_dir.join("outputs"); - fs::create_dir_all(&outputs_dir).unwrap(); - let run_path = cond_dir.join("run.json"); - write_run_record(&run_path, json!([])); - fs::write( - iteration_dir.join("conditions.json"), - json!({ - "mode": "new-skill", - "conditions": [{"name": "with_skill", "skill_path": "/skill/SKILL.md"}], - "timestamp": "2026-06-07T00:00:00.000Z", - "harness": "codex" - }) - .to_string(), - ) - .unwrap(); - write_dispatch( - &iteration_dir, - json!([{"eval_id": "crash", "condition": "with_skill", "outputs_dir": outputs_dir.to_string_lossy()}]), - ); - fs::write( - outputs_dir.join("codex-events.jsonl"), - jsonl(&[ - json!({"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "bun test", "aggregated_output": "ok"}}), - ]), - ) - .unwrap(); - - let result = - fill_transcripts(&iteration_dir, Harness::resolve("codex").unwrap(), false).unwrap(); - assert_eq!(result.filled, 1); - assert_eq!(result.missing, 0); - - let updated: RunRecord = - serde_json::from_str(&fs::read_to_string(&run_path).unwrap()).unwrap(); - assert_eq!( - serde_json::to_value(&updated.tool_invocations).unwrap(), - json!([{"name": "command_execution", "ordinal": 0, "args": {"command": "bun test"}, "result": "ok"}]) - ); - } - - #[test] - fn fills_codex_run_records_in_nested_run_dirs() { - let root = TempDir::new().unwrap(); - let iteration_dir: PathBuf = root.path().join("iter-codex-multi"); - let cond_dir = iteration_dir.join("eval-crash").join("with_skill"); - fs::create_dir_all(&iteration_dir).unwrap(); - fs::write( - iteration_dir.join("conditions.json"), - json!({ - "mode": "new-skill", - "conditions": [{"name": "with_skill", "skill_path": "/skill/SKILL.md"}], - "timestamp": "2026-06-07T00:00:00.000Z", - "harness": "codex" - }) - .to_string(), - ) - .unwrap(); - for (k, command) in [(1, "bun test"), (2, "bun lint")] { - let run_dir = cond_dir.join(format!("run-{k}")); - let outputs_dir = run_dir.join("outputs"); - fs::create_dir_all(&outputs_dir).unwrap(); - write_run_record(&run_dir.join("run.json"), json!([])); - fs::write( - outputs_dir.join("codex-events.jsonl"), - jsonl(&[ - json!({"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": command, "aggregated_output": "ok"}}), - ]), - ) - .unwrap(); - } - - let result = - fill_transcripts(&iteration_dir, Harness::resolve("codex").unwrap(), false).unwrap(); - assert_eq!(result.filled, 2); - assert_eq!(result.missing, 0); - - for (k, command) in [(1, "bun test"), (2, "bun lint")] { - let updated: RunRecord = serde_json::from_str( - &fs::read_to_string(cond_dir.join(format!("run-{k}")).join("run.json")).unwrap(), - ) - .unwrap(); - assert_eq!( - serde_json::to_value(&updated.tool_invocations).unwrap(), - json!([{"name": "command_execution", "ordinal": 0, "args": {"command": command}, "result": "ok"}]), - "wrong invocations for run-{k}" - ); - } - } -} diff --git a/src/pipeline/grade/transcript_check.rs b/src/pipeline/grade/transcript_check.rs index dde7fb5..f1d2bfe 100644 --- a/src/pipeline/grade/transcript_check.rs +++ b/src/pipeline/grade/transcript_check.rs @@ -68,10 +68,9 @@ pub fn grade_transcript_check_with_context( if assertion.check == "tool_invocation_matches" && invocations.is_empty() { return fail( &assertion.id, - "tool_invocations is empty — run record was not filled by a transcript adapter. \ - Run `eval-magic fill-transcripts` for Claude Code, or `eval-magic fill-transcripts \ - --harness codex` when outputs/codex-events.jsonl is present; otherwise rely on \ - `llm_judge` assertions for harnesses without an adapter." + "tool_invocations is empty — the task transcript contained no recorded tool call. \ + Re-dispatch the task if its transcript is missing; otherwise verify the agent \ + actually invoked the expected tool." .to_string(), ); } diff --git a/src/pipeline/mod.rs b/src/pipeline/mod.rs index 0479c8c..cbcedba 100644 --- a/src/pipeline/mod.rs +++ b/src/pipeline/mod.rs @@ -1,8 +1,8 @@ //! The post-dispatch processing chain: stateless JSON-in/JSON-out stages. //! //! Chain order: -//! `record-runs` → `fill-transcripts` → `detect-stray-writes` → `grade` → -//! `aggregate`. Each stage reads JSON/JSONL artifacts from an iteration directory +//! `record-runs` → `detect-stray-writes` → `grade` → `aggregate`. Each stage +//! reads JSON/JSONL artifacts from an iteration directory //! and writes JSON back; no stage pipes to another in-memory, so any stage can //! be run (and re-run) standalone. @@ -11,7 +11,6 @@ pub mod compare; pub mod detect_stray_writes; pub mod diff_scope; pub mod error; -pub mod fill_transcripts; mod git_isolation; pub mod grade; pub(crate) mod guard_denials; @@ -32,7 +31,6 @@ pub use diff_scope::{ DiffScopeMetrics, DiffScopeRecord, DiffScopeSummary, PatchRecord, measure_iteration_diff_scopes, }; pub use error::PipelineError; -pub use fill_transcripts::{FillTranscriptsResult, fill_transcripts}; pub use grade::{GradeContext, emit_judge_tasks, finalize, grade_command_checks}; pub use record_runs::{RecordRunsResult, record_runs}; pub use slots::{RunSlot, run_slots}; diff --git a/src/pipeline/record_runs.rs b/src/pipeline/record_runs.rs index 55165bf..405dadc 100644 --- a/src/pipeline/record_runs.rs +++ b/src/pipeline/record_runs.rs @@ -2,17 +2,15 @@ //! //! Assembles a schema-valid `run.json` (and //! backfills `timing.json`) for every task in the iteration's `dispatch.json`, -//! from sources already on disk: carry-over fields from the dispatch task, the -//! `final_message` (from `/final-message.md`, falling back to the -//! transcript's last assistant text), and `tool_invocations`/tokens/duration from -//! each task's events file. Scripted tasks take ordered messages/tools from the -//! runner-owned `conversation.json` and combine raw timing across +//! from sources already on disk: carry-over fields from the dispatch task, +//! runner-owned completion metadata from `conversation.json`, and assistant +//! messages, tools, final text, tokens, and duration from transcripts under //! `outputs/turn-N/-events.jsonl` according to the harness descriptor. //! -//! Existing records always win: an agent/operator-written `run.json` is skipped -//! without `overwrite`, and `timing.json` is backfill-only — completion-event -//! numbers captured at dispatch time are never replaced by transcript-derived -//! ones (whose accounting is harness-specific and may not be comparable 1:1). +//! Existing records always win: a previously assembled `run.json` is skipped +//! without `overwrite`, and `timing.json` is backfill-only — higher-fidelity +//! timing captured during dispatch is never replaced by transcript-derived +//! values whose accounting may not be comparable 1:1. //! //! Harnesses whose captures identify refused tool calls also get the //! iteration-level `permission-denials.json` written here (see @@ -20,7 +18,7 @@ //! refusal get no file at all, so its absence never reads as "nothing refused". //! //! Two sub-concerns live beside this module: [`conversation`] assembles a -//! scripted task's ordered rounds, and [`prompt_read`] decides whether a +//! task's ordered rounds, and [`prompt_read`] decides whether a //! dispatch ever received its instructions. use std::fs; @@ -31,8 +29,8 @@ use serde::Deserialize; use crate::adapters::{PermissionDenial, TranscriptSummary, adapter_for}; use crate::core::fs::write_json; use crate::core::{ - CodebaseRecord, ConditionSkill, ConversationEvent, ConversationRecord, Harness, RunRecord, - SkillSource, TimingRecord, TimingSource, + CodebaseRecord, ConditionSkill, ConversationRecord, Harness, RunRecord, SkillSource, + TimingRecord, TimingSource, }; use crate::pipeline::error::PipelineError; use crate::pipeline::permission_denials::{self, TaskPermissionDenials}; @@ -62,7 +60,8 @@ struct DispatchTask { #[serde(default)] skills: Option>, user_prompt: String, - fixtures: Vec, + #[serde(alias = "fixtures")] + files: Vec, outputs_dir: String, run_record_path: String, timing_path: String, @@ -70,15 +69,6 @@ struct DispatchTask { dispatch_prompt_path: String, #[serde(default)] conversation_path: Option, - /// Present only for a scripted task. Every task carries a - /// `conversation_path`, so this and `responder` are what tell a task whose - /// rounds are unknown-without-the-artifact from a one-shot task. - #[serde(default)] - turns: Option, - /// Present only for a responder-driven task — the other way a task's rounds - /// become unknown without its completion artifact. - #[serde(default)] - responder: Option, /// Group this task belongs to; absent for a single-group run. Carried so the /// session-surface report can be joined back to the comparison cells a /// shadow finding names. @@ -96,7 +86,7 @@ struct DispatchTask { pub struct RecordRunsResult { pub recorded: usize, pub skipped_existing: usize, - pub skipped_no_final_message: usize, + pub skipped_no_final_response: usize, pub missing_transcript: usize, pub skipped_prompt_unread: usize, pub skipped_incomplete_conversation: usize, @@ -116,30 +106,25 @@ pub struct RecordRunsResult { } impl RecordRunsResult { - /// A loud, actionable warning when one-shot events are missing or a - /// scripted conversation lacks one or more raw round transcripts. Scripted - /// runs retain their ordered conversation evidence, but timing is omitted - /// unless every round can be combined. + /// A loud, actionable warning when a task lacks one or more raw round + /// transcripts. A task with no recoverable final response is skipped; + /// partial multi-round evidence can still be recorded without complete + /// timing or tool evidence. pub fn transcript_warning(&self, harness: Harness) -> Option { if self.missing_transcript == 0 { return None; } let n = self.missing_transcript; let plural = if n == 1 { "" } else { "s" }; - let all = self.recorded > 0 && self.missing_transcript >= self.recorded; - let lead = if all { - format!("⚠ {n} run{plural} recorded but NONE matched a transcript") - } else { - format!("⚠ {n} run{plural} missing a transcript") - }; let file = adapter_for(harness) .cli_events_filename() .unwrap_or_else(|| "the events file".to_string()); let cause = format!("expected `{file}` transcript file(s) were not found under task outputs"); Some(format!( - "{lead} — {cause}; one-shot runs lack tool evidence and timing, while scripted runs \ - retain conversation evidence but omit incomplete timing." + "⚠ {n} task{plural} missing transcript evidence — {cause}; a task with no final \ + response was skipped, while partial multi-round evidence is recorded without \ + complete timing or tool evidence. Re-dispatch the affected task{plural}." )) } @@ -181,9 +166,9 @@ impl RecordRunsResult { )) } - /// Warn when a multi-turn task never produced its runner-owned completion - /// artifact. Raw per-turn transcripts are intentionally not ingested - /// without it because the driver may have failed between turns. + /// Warn when a task never produced its runner-owned completion artifact. + /// Raw per-turn transcripts are intentionally not ingested without it + /// because the driver may have failed between rounds. pub fn incomplete_conversation_warning(&self) -> Option { let n = self.skipped_incomplete_conversation; if n == 0 { @@ -191,7 +176,7 @@ impl RecordRunsResult { } let plural = if n == 1 { "" } else { "s" }; Some(format!( - "⚠ {n} multi-turn conversation{plural} skipped — conversation.json is missing, so \ + "⚠ {n} task{plural} skipped — conversation.json is missing, so \ eval-magic cannot distinguish a completed/stopped scenario from an interrupted \ dispatch. Re-run `eval-magic dispatch` — it retries exactly the tasks with no \ completion artifact." @@ -212,8 +197,7 @@ pub fn record_runs( if !dispatch_path.exists() { return Err(PipelineError::Message(format!( "{} not found — record-runs assembles records from dispatch.json and only \ - supports runner-built iterations. For hand-authored runs, write run.json + \ - timing.json manually (see schema/run-record.schema.json).", + supports runner-built iterations. Re-run `eval-magic run` to prepare the campaign.", dispatch_path.display() ))); } @@ -226,20 +210,13 @@ pub fn record_runs( let mut denial_tasks: Vec = Vec::new(); let mut surface_tasks: Vec = Vec::new(); for task in &tasks { - let conversation = conversation::for_task(task)?; - // Keyed on what drives the turns, not on `conversation_path`: every task - // declares a conversation artifact, so its presence does not distinguish - // a multi-turn one. A scripted or responder-driven task without the - // artifact is genuinely incomplete — which rounds ran is unknown. - if (task.turns.is_some() || task.responder.is_some()) && conversation.is_none() { + let Some(completion) = conversation::for_task(task)? else { result.skipped_incomplete_conversation += 1; continue; - } - let (summary, transcripts_complete) = match &conversation { - Some(conversation) => conversation::summary_for_task(harness, task, conversation), - None => (transcript_summary_for_task(harness, task), true), }; - if summary.is_none() || !transcripts_complete { + let evidence = conversation::evidence_for_task(harness, task, &completion); + let summary = evidence.summary.as_ref(); + if summary.is_none() || !evidence.transcripts_complete { result.missing_transcript += 1; } @@ -252,7 +229,7 @@ pub fn record_runs( task.eval_id.clone(), task.condition.clone(), task.run_index, - permission_denials_for_task(harness, task, conversation.as_ref()), + permission_denials_for_task(harness, task, &completion), ); if denials.harness_denial_count() > 0 { result.permission_denials += denials.harness_denial_count(); @@ -270,15 +247,15 @@ pub fn record_runs( condition: task.condition.clone(), run_index: task.run_index, group: task.group.clone(), - rounds: session_surfaces_for_task(harness, task, conversation.as_ref()), + rounds: session_surfaces_for_task(harness, task, &completion), }); } let run_record_path = Path::new(&task.run_record_path); if run_record_path.exists() && !overwrite { - // An agent/operator already wrote this run.json — leave it untouched. + // A prior ingest already wrote this run.json — leave it untouched. result.skipped_existing += 1; - } else if let Some(summary) = &summary + } else if let Some(summary) = summary && prompt_read_failed( summary, &task.dispatch_prompt_path, @@ -292,24 +269,10 @@ pub fn record_runs( result.skipped_prompt_unread += 1; continue; } else { - let final_message_path = Path::new(&task.outputs_dir).join("final-message.md"); - let final_message = if let Some(conversation) = &conversation { - conversation - .events - .iter() - .rev() - .find_map(|event| match event { - ConversationEvent::AssistantMessage { text, .. } => Some(text.clone()), - _ => None, - }) - } else if final_message_path.exists() { - Some(fs::read_to_string(&final_message_path)?.trim().to_string()) - } else { - summary.as_ref().and_then(|s| s.final_text.clone()) - }; + let final_message = summary.and_then(|summary| summary.final_text.clone()); let Some(final_message) = final_message else { - // No final-message.md and no transcript text — don't write a blank record. - result.skipped_no_final_message += 1; + // No transcript text means there is no final response to grade. + result.skipped_no_final_response += 1; continue; }; @@ -319,22 +282,16 @@ pub fn record_runs( skill_path: task.skill_path.clone(), skills: task.skills.clone(), prompt: task.user_prompt.clone(), - files: task.fixtures.clone(), + files: task.files.clone(), final_message, - tool_invocations: conversation.as_ref().map_or_else( - || { - summary - .as_ref() - .map(|s| s.tool_invocations.clone()) - .unwrap_or_default() - }, - conversation::tool_invocations, - ), + tool_invocations: summary + .map(|summary| summary.tool_invocations.clone()) + .unwrap_or_default(), // Timing lives in timing.json; run.json never carries it. total_tokens: None, duration_ms: None, run_index: task.run_index, - conversation: conversation.clone(), + conversation: Some(evidence.conversation.clone()), codebase: task.codebase.clone(), skill_source: task.skill_source.clone(), }; @@ -349,8 +306,8 @@ pub fn record_runs( // timing.json — backfill only; completion-event numbers always win. let timing_path = Path::new(&task.timing_path); - if transcripts_complete - && let Some(summary) = &summary + if evidence.transcripts_complete + && let Some(summary) = summary && (!timing_path.exists() || overwrite) { let timing = TimingRecord { @@ -382,20 +339,6 @@ pub fn record_runs( Ok(result) } -/// Resolve a task's transcript summary: read the events file the harness CLI -/// wrote under the task's outputs dir (e.g. Codex's `codex-events.jsonl`, Claude -/// Code's `claude-events.jsonl`). Returns `None` when no transcript is found. -fn transcript_summary_for_task(harness: Harness, task: &DispatchTask) -> Option { - let events_path = - Path::new(&task.outputs_dir).join(adapter_for(harness).cli_events_filename()?); - if !events_path.exists() { - return None; - } - adapter_for(harness) - .parse_cli_events_full(&events_path) - .ok() -} - /// The skill/plugin surface each of a task's rounds reported. Unlike refusals, /// these are kept per round rather than flattened: isolation has to hold for the /// initial dispatch and every resumed turn, so a round whose transcript is @@ -403,19 +346,16 @@ fn transcript_summary_for_task(harness: Harness, task: &DispatchTask) -> Option< fn session_surfaces_for_task( harness: Harness, task: &DispatchTask, - conversation: Option<&ConversationRecord>, + conversation: &ConversationRecord, ) -> Vec { let adapter = adapter_for(harness); let Some(filename) = adapter.cli_events_filename() else { return Vec::new(); }; let outputs_dir = Path::new(&task.outputs_dir); - let paths: Vec = match conversation { - Some(conversation) => (1..=conversation.delivered_followups.saturating_add(1)) - .map(|round| outputs_dir.join(format!("turn-{round}")).join(&filename)) - .collect(), - None => vec![outputs_dir.join(&filename)], - }; + let paths: Vec = (1..=conversation.delivered_followups.saturating_add(1)) + .map(|round| outputs_dir.join(format!("turn-{round}")).join(&filename)) + .collect(); paths .iter() .enumerate() @@ -430,25 +370,21 @@ fn session_surfaces_for_task( } /// The tool calls the harness refused across a task's transcript(s): the -/// one-shot events file, or every scripted round's, since each round is its own -/// CLI invocation with its own refusals. A missing or unparseable transcript +/// per-round events files. A missing or unparseable transcript /// contributes nothing — absence of evidence is not a denial. fn permission_denials_for_task( harness: Harness, task: &DispatchTask, - conversation: Option<&ConversationRecord>, + conversation: &ConversationRecord, ) -> Vec { let adapter = adapter_for(harness); let Some(filename) = adapter.cli_events_filename() else { return Vec::new(); }; let outputs_dir = Path::new(&task.outputs_dir); - let paths: Vec = match conversation { - Some(conversation) => (1..=conversation.delivered_followups.saturating_add(1)) - .map(|round| outputs_dir.join(format!("turn-{round}")).join(&filename)) - .collect(), - None => vec![outputs_dir.join(&filename)], - }; + let paths: Vec = (1..=conversation.delivered_followups.saturating_add(1)) + .map(|round| outputs_dir.join(format!("turn-{round}")).join(&filename)) + .collect(); paths .iter() .filter(|path| path.exists()) diff --git a/src/pipeline/record_runs/conversation.rs b/src/pipeline/record_runs/conversation.rs index 0d961a6..71a418e 100644 --- a/src/pipeline/record_runs/conversation.rs +++ b/src/pipeline/record_runs/conversation.rs @@ -1,6 +1,13 @@ use super::*; use crate::adapters::TokenUsageAggregation; -use crate::core::{ConversationRecord, ToolInvocation}; +use crate::adapters::transcript::TranscriptEvent; +use crate::core::{ConversationEvent, ConversationRecord, ToolInvocation}; + +pub(super) struct TaskEvidence { + pub summary: Option, + pub conversation: ConversationRecord, + pub transcripts_complete: bool, +} pub(super) fn for_task(task: &DispatchTask) -> Result, PipelineError> { let Some(path) = task.conversation_path.as_deref() else { @@ -18,18 +25,54 @@ pub(super) fn for_task(task: &DispatchTask) -> Result )?)) } -pub(super) fn summary_for_task( +/// Combine runner-owned user-turn metadata with transcript-owned assistant and +/// tool events. Historical assistant/tool events in the completion artifact are +/// deliberately ignored so one source owns their bytes and ordering. +pub(super) fn evidence_for_task( harness: Harness, task: &DispatchTask, - conversation: &ConversationRecord, -) -> (Option, bool) { - let rounds = conversation.delivered_followups.saturating_add(1); + completion: &ConversationRecord, +) -> TaskEvidence { + let rounds = completion.delivered_followups.saturating_add(1); let Some(filename) = adapter_for(harness).cli_events_filename() else { - return (None, false); + let mut conversation = completion.clone(); + conversation + .events + .retain(|event| matches!(event, ConversationEvent::UserMessage { .. })); + renumber(&mut conversation.events); + return TaskEvidence { + summary: None, + conversation, + transcripts_complete: false, + }; }; + + let mut conversation = completion.clone(); + conversation.events.clear(); + let mut transcript_events = Vec::new(); let mut summaries = Vec::new(); let mut complete = true; + for round in 1..=rounds { + for event in completion.events.iter().filter(|event| { + matches!(event, ConversationEvent::UserMessage { round: event_round, .. } if *event_round == round) + }) { + let ConversationEvent::UserMessage { + text, + origin, + .. + } = event + else { + unreachable!("filter retains only user messages"); + }; + conversation.events.push(ConversationEvent::UserMessage { + ordinal: conversation.events.len() as u32, + round, + text: text.clone(), + origin: origin.clone(), + }); + } + let path = Path::new(&task.outputs_dir) .join(format!("turn-{round}")) .join(&filename); @@ -37,32 +80,109 @@ pub(super) fn summary_for_task( complete = false; continue; } - match adapter_for(harness).parse_cli_events_full(&path) { - Ok(summary) => summaries.push(summary), - Err(_) => complete = false, + let summary = match adapter_for(harness).parse_cli_events_full(&path) { + Ok(summary) => summary, + Err(_) => { + complete = false; + continue; + } + }; + + let mut final_text_present = false; + for event in &summary.events { + let ordinal = conversation.events.len() as u32; + match event { + TranscriptEvent::AssistantMessage { text, .. } => { + final_text_present |= summary.final_text.as_deref() == Some(text); + conversation + .events + .push(ConversationEvent::AssistantMessage { + ordinal, + round, + text: text.clone(), + }); + transcript_events.push(TranscriptEvent::AssistantMessage { + ordinal, + text: text.clone(), + }); + } + TranscriptEvent::ToolInvocation { + name, args, result, .. + } => { + conversation.events.push(ConversationEvent::ToolInvocation { + ordinal, + round, + name: name.clone(), + args: args.clone(), + result: result.clone(), + }); + transcript_events.push(TranscriptEvent::ToolInvocation { + ordinal, + name: name.clone(), + args: args.clone(), + result: result.clone(), + }); + } + } } - } - if summaries.is_empty() { - return (None, complete); - } - let total_tokens = match adapter_for(harness).conversation_token_usage_aggregation() { - TokenUsageAggregation::Sum => { - sum_some(summaries.iter().map(|summary| summary.total_tokens)) + if let Some(text) = &summary.final_text + && !final_text_present + { + let ordinal = conversation.events.len() as u32; + conversation + .events + .push(ConversationEvent::AssistantMessage { + ordinal, + round, + text: text.clone(), + }); + transcript_events.push(TranscriptEvent::AssistantMessage { + ordinal, + text: text.clone(), + }); } - TokenUsageAggregation::Last => summaries.last().and_then(|summary| summary.total_tokens), - }; - let duration_ms = sum_some(summaries.iter().map(|summary| summary.duration_ms)); - ( - Some(TranscriptSummary { - tool_invocations: tool_invocations(conversation), - events: Vec::new(), + summaries.push(summary); + } + + let summary = (!summaries.is_empty()).then(|| { + let total_tokens = match adapter_for(harness).conversation_token_usage_aggregation() { + TokenUsageAggregation::Sum => { + sum_some(summaries.iter().map(|summary| summary.total_tokens)) + } + TokenUsageAggregation::Last => { + summaries.last().and_then(|summary| summary.total_tokens) + } + }; + let duration_ms = sum_some(summaries.iter().map(|summary| summary.duration_ms)); + let final_text = summaries + .iter() + .rev() + .find_map(|summary| summary.final_text.clone()); + TranscriptSummary { + tool_invocations: tool_invocations(&conversation), + events: transcript_events, session_id: None, total_tokens, duration_ms, - final_text: None, - }), - complete, - ) + final_text, + } + }); + + TaskEvidence { + summary, + conversation, + transcripts_complete: complete, + } +} + +fn renumber(events: &mut [ConversationEvent]) { + for (ordinal, event) in events.iter_mut().enumerate() { + match event { + ConversationEvent::UserMessage { ordinal: value, .. } + | ConversationEvent::AssistantMessage { ordinal: value, .. } + | ConversationEvent::ToolInvocation { ordinal: value, .. } => *value = ordinal as u32, + } + } } fn sum_some(values: impl Iterator>) -> Option { diff --git a/src/pipeline/record_runs/tests/assembly.rs b/src/pipeline/record_runs/tests/assembly.rs index dbd93f5..600d848 100644 --- a/src/pipeline/record_runs/tests/assembly.rs +++ b/src/pipeline/record_runs/tests/assembly.rs @@ -1,6 +1,6 @@ //! Assembling `run.json` and `timing.json` from what each harness left on disk, -//! including the per-harness final-message fallbacks and the `dispatch.json` -//! input contract. +//! including transcript-owned final responses and the `dispatch.json` input +//! contract. use super::*; @@ -14,17 +14,15 @@ fn assembles_run_and_timing_for_every_task_from_disk() { FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: Some("Fixed it."), }, FixtureTask { eval_id: "crash", condition: "without_skill", - final_message: Some("Done, I think."), }, ], ); - write_claude_events(&paths[0].outputs_dir, "unused"); - write_claude_events(&paths[1].outputs_dir, "unused"); + write_claude_events(&paths[0].outputs_dir, "Fixed it."); + write_claude_events(&paths[1].outputs_dir, "Done, I think."); let result = record_runs(&iter, 1, Harness::resolve("claude-code").unwrap(), false).unwrap(); assert_eq!(result.recorded, 2); @@ -39,7 +37,7 @@ fn assembles_run_and_timing_for_every_task_from_disk() { assert_eq!(run.final_message, "Fixed it."); assert_eq!(run.tool_invocations.len(), 1); assert_eq!(run.tool_invocations[0].name, "Bash"); - assert_eq!(run.tool_invocations[0].ordinal, 0); + assert_eq!(run.tool_invocations[0].ordinal, 1); assert!( read_run(&iter, "crash", "without_skill") @@ -63,8 +61,9 @@ fn carries_the_codebase_from_dispatch_task_into_each_run_record() { let cond_dir = iter.join("eval-crash").join("with_skill"); let outputs_dir = cond_dir.join("outputs"); fs::create_dir_all(&outputs_dir).unwrap(); - fs::write(outputs_dir.join("final-message.md"), "Fixed it.").unwrap(); - write_codex_events(&outputs_dir, "unused"); + let conversation_path = outputs_dir.join("conversation.json"); + write_one_shot_completion(&conversation_path, "Do the crash task"); + write_codex_events(&outputs_dir, "Fixed it."); let codebase = json!({ "kind": "git", "source": "https://example.com/project.git", @@ -82,8 +81,9 @@ fn carries_the_codebase_from_dispatch_task_into_each_run_record() { "condition": "with_skill", "skill_path": "/staged/skill/SKILL.md", "user_prompt": "Do the crash task", - "fixtures": [], + "files": [], "outputs_dir": outputs_dir.to_string_lossy(), + "conversation_path": conversation_path.to_string_lossy(), "run_record_path": cond_dir.join("run.json").to_string_lossy(), "timing_path": cond_dir.join("timing.json").to_string_lossy(), "agent_description": "crash:with_skill:i1-nonce1", @@ -110,8 +110,9 @@ fn carries_the_skill_source_from_dispatch_task_into_each_run_record() { let cond_dir = iter.join("eval-crash").join("with_skill"); let outputs_dir = cond_dir.join("outputs"); fs::create_dir_all(&outputs_dir).unwrap(); - fs::write(outputs_dir.join("final-message.md"), "Fixed it.").unwrap(); - write_codex_events(&outputs_dir, "unused"); + let conversation_path = outputs_dir.join("conversation.json"); + write_one_shot_completion(&conversation_path, "Do the crash task"); + write_codex_events(&outputs_dir, "Fixed it."); let skill_source = json!({ "kind": "path", "source": "/work/skills/mr-review", @@ -131,8 +132,9 @@ fn carries_the_skill_source_from_dispatch_task_into_each_run_record() { "condition": "with_skill", "skill_path": "/staged/skill/SKILL.md", "user_prompt": "Do the crash task", - "fixtures": [], + "files": [], "outputs_dir": outputs_dir.to_string_lossy(), + "conversation_path": conversation_path.to_string_lossy(), "run_record_path": cond_dir.join("run.json").to_string_lossy(), "timing_path": cond_dir.join("timing.json").to_string_lossy(), "agent_description": "crash:with_skill:i1-nonce1", @@ -161,7 +163,6 @@ fn omits_the_codebase_key_when_a_task_declares_none() { &[FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: Some("Fixed it."), }], ); write_claude_events(&paths[0].outputs_dir, "unused"); @@ -185,20 +186,18 @@ fn carries_run_index_from_dispatch_task_into_each_run_record() { let run_dir = cond_dir.join(format!("run-{k}")); let outputs_dir = run_dir.join("outputs"); fs::create_dir_all(&outputs_dir).unwrap(); - fs::write( - outputs_dir.join("final-message.md"), - format!("Fixed it in run {k}."), - ) - .unwrap(); - write_codex_events(&outputs_dir, "unused"); + let conversation_path = outputs_dir.join("conversation.json"); + write_one_shot_completion(&conversation_path, "Do the crash task"); + write_codex_events(&outputs_dir, &format!("Fixed it in run {k}.")); serialized.push(json!({ "eval_id": "crash", "condition": "with_skill", "run_index": k, "skill_path": "/staged/skill/SKILL.md", "user_prompt": "Do the crash task", - "fixtures": [], + "files": [], "outputs_dir": outputs_dir.to_string_lossy(), + "conversation_path": conversation_path.to_string_lossy(), "run_record_path": run_dir.join("run.json").to_string_lossy(), "timing_path": run_dir.join("timing.json").to_string_lossy(), "agent_description": format!("crash:with_skill:r{k}:i1-nonce1"), @@ -230,7 +229,6 @@ fn assembles_codex_records_from_each_tasks_events() { &[FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: Some("Fixed it."), }], ); write_codex_events(&paths[0].outputs_dir, "Codex final."); @@ -240,10 +238,10 @@ fn assembles_codex_records_from_each_tasks_events() { assert_eq!(result.missing_transcript, 0); let run = read_run(&iter, "crash", "with_skill"); - assert_eq!(run.final_message, "Fixed it."); + assert_eq!(run.final_message, "Codex final."); assert_eq!( serde_json::to_value(&run.tool_invocations).unwrap(), - json!([{"name": "command_execution", "ordinal": 0, "args": {"command": "bun test"}, "result": "ok"}]) + json!([{"name": "command_execution", "ordinal": 1, "args": {"command": "bun test"}, "result": "ok"}]) ); let timing = read_timing_value(&iter, "crash", "with_skill"); @@ -254,7 +252,7 @@ fn assembles_codex_records_from_each_tasks_events() { } #[test] -fn falls_back_to_codex_final_agent_message_when_final_message_md_missing() { +fn records_codex_final_agent_message_from_the_transcript() { let root = TempDir::new().unwrap(); let iter = dirs(&root); let paths = write_iteration( @@ -262,7 +260,6 @@ fn falls_back_to_codex_final_agent_message_when_final_message_md_missing() { &[FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: None, }], ); write_codex_events(&paths[0].outputs_dir, "Closing summary from Codex."); @@ -284,7 +281,6 @@ fn assembles_claude_records_from_each_tasks_events() { &[FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: Some("Fixed it."), }], ); write_claude_events(&paths[0].outputs_dir, "Closing summary."); @@ -294,11 +290,10 @@ fn assembles_claude_records_from_each_tasks_events() { assert_eq!(result.missing_transcript, 0); let run = read_run(&iter, "crash", "with_skill"); - // final-message.md wins when present. - assert_eq!(run.final_message, "Fixed it."); + assert_eq!(run.final_message, "Closing summary."); assert_eq!( serde_json::to_value(&run.tool_invocations).unwrap(), - json!([{"name": "Bash", "ordinal": 0, "args": {"command": "bun test"}, "result": "ok"}]) + json!([{"name": "Bash", "ordinal": 1, "args": {"command": "bun test"}, "result": "ok"}]) ); let timing = read_timing_value(&iter, "crash", "with_skill"); assert_eq!( @@ -308,9 +303,7 @@ fn assembles_claude_records_from_each_tasks_events() { } #[test] -fn falls_back_to_claude_result_final_text_when_final_message_md_missing() { - // Claude `-p` has no --output-last-message, so the result event's text is - // the primary final-message source. +fn records_claude_result_final_text_from_the_transcript() { let root = TempDir::new().unwrap(); let iter = dirs(&root); let paths = write_iteration( @@ -318,7 +311,6 @@ fn falls_back_to_claude_result_final_text_when_final_message_md_missing() { &[FixtureTask { eval_id: "crash", condition: "with_skill", - final_message: None, }], ); write_claude_events(&paths[0].outputs_dir, "Closing summary from claude -p."); diff --git a/src/pipeline/record_runs/tests/conversation.rs b/src/pipeline/record_runs/tests/conversation.rs index 3cfab6d..0838c70 100644 --- a/src/pipeline/record_runs/tests/conversation.rs +++ b/src/pipeline/record_runs/tests/conversation.rs @@ -10,7 +10,6 @@ fn assembles_multi_turn_run_using_last_cumulative_codex_tokens_and_summed_durati &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: None, }], ); let conversation_path = iter @@ -24,30 +23,16 @@ fn assembles_multi_turn_run_using_last_cumulative_codex_tokens_and_summed_durati "delivered_followups": 1, "events": [ {"type": "user_message", "ordinal": 0, "round": 1, "text": "Fix it."}, - {"type": "assistant_message", "ordinal": 1, "round": 1, "text": "Which timezone?"}, - {"type": "user_message", "ordinal": 2, "round": 2, "text": "US timezones."}, - { - "type": "tool_invocation", - "ordinal": 3, - "round": 2, - "name": "file_change", - "args": {"path": "src/date.rs"} - }, - { - "type": "assistant_message", - "ordinal": 4, - "round": 2, - "text": "Updated the date handling." - } + {"type": "user_message", "ordinal": 1, "round": 2, "text": "US timezones."} ] })) .unwrap(), ) .unwrap(); - for round in [1, 2] { + for (round, final_text) in [(1, "Which timezone?"), (2, "Updated the date handling.")] { let round_dir = paths[0].outputs_dir.join(format!("turn-{round}")); fs::create_dir_all(&round_dir).unwrap(); - write_codex_events(&round_dir, "unused"); + write_codex_events(&round_dir, final_text); } let dispatch_path = iter.join("dispatch.json"); let mut dispatch: Value = @@ -70,13 +55,32 @@ fn assembles_multi_turn_run_using_last_cumulative_codex_tokens_and_summed_durati assert_eq!(run.final_message, "Updated the date handling."); assert_eq!( serde_json::to_value(&run.tool_invocations).unwrap(), - json!([{ - "name": "file_change", - "args": {"path": "src/date.rs"}, - "ordinal": 3 - }]) + json!([ + { + "name": "command_execution", + "args": {"command": "bun test"}, + "ordinal": 1, + "result": "ok" + }, + { + "name": "command_execution", + "args": {"command": "bun test"}, + "ordinal": 4, + "result": "ok" + } + ]) + ); + assert_eq!( + serde_json::to_value(run.conversation.unwrap()).unwrap()["events"], + json!([ + {"type": "user_message", "ordinal": 0, "round": 1, "text": "Fix it."}, + {"type": "tool_invocation", "ordinal": 1, "round": 1, "name": "command_execution", "args": {"command": "bun test"}, "result": "ok"}, + {"type": "assistant_message", "ordinal": 2, "round": 1, "text": "Which timezone?"}, + {"type": "user_message", "ordinal": 3, "round": 2, "text": "US timezones."}, + {"type": "tool_invocation", "ordinal": 4, "round": 2, "name": "command_execution", "args": {"command": "bun test"}, "result": "ok"}, + {"type": "assistant_message", "ordinal": 5, "round": 2, "text": "Updated the date handling."} + ]) ); - assert_eq!(run.conversation.unwrap().delivered_followups, 1); let timing = read_timing_value(&iter, "clarify", "with_skill"); assert_eq!(timing["total_tokens"], 40); @@ -84,6 +88,48 @@ fn assembles_multi_turn_run_using_last_cumulative_codex_tokens_and_summed_durati assert_eq!(timing["source"], "transcript"); } +#[test] +fn one_shot_task_without_its_completion_artifact_is_skipped_as_incomplete() { + let root = TempDir::new().unwrap(); + let iter = dirs(&root); + let paths = write_iteration( + &iter, + &[FixtureTask { + eval_id: "one-shot", + condition: "with_skill", + }], + ); + let round_dir = paths[0].outputs_dir.join("turn-1"); + fs::create_dir_all(&round_dir).unwrap(); + write_claude_events(&round_dir, "Done."); + + let dispatch_path = iter.join("dispatch.json"); + let mut dispatch: Value = + serde_json::from_str(&fs::read_to_string(&dispatch_path).unwrap()).unwrap(); + dispatch["tasks"][0]["conversation_path"] = json!( + iter.join("eval-one-shot") + .join("with_skill") + .join("conversation.json") + .to_string_lossy() + ); + fs::write( + &dispatch_path, + serde_json::to_string_pretty(&dispatch).unwrap(), + ) + .unwrap(); + + let result = record_runs(&iter, 1, Harness::resolve("claude-code").unwrap(), false).unwrap(); + + assert_eq!(result.skipped_incomplete_conversation, 1); + assert_eq!(result.recorded, 0); + let warning = result + .incomplete_conversation_warning() + .expect("missing completion artifact is reported"); + assert!(warning.contains("1 task skipped"), "{warning}"); + assert!(!warning.contains("multi-turn"), "{warning}"); + assert!(!paths[0].run_record_path.exists()); +} + #[test] fn assembles_multi_turn_run_by_summing_independent_claude_round_timing() { let root = TempDir::new().unwrap(); @@ -93,7 +139,6 @@ fn assembles_multi_turn_run_by_summing_independent_claude_round_timing() { &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: None, }], ); let conversation_path = iter @@ -150,7 +195,6 @@ fn skips_multi_turn_run_when_conversation_shows_failed_prompt_read() { &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: None, }], ); let prompt_path = iter @@ -172,21 +216,7 @@ fn skips_multi_turn_run_when_conversation_shows_failed_prompt_read() { "status": "completed", "delivered_followups": 0, "events": [ - {"type": "user_message", "ordinal": 0, "round": 1, "text": "Fix it."}, - { - "type": "tool_invocation", - "ordinal": 1, - "round": 1, - "name": "Read", - "args": {"file_path": prompt_path.to_string_lossy()}, - "result": "File is outside the allowed working directory." - }, - { - "type": "assistant_message", - "ordinal": 2, - "round": 1, - "text": "I could not read the prompt file." - } + {"type": "user_message", "ordinal": 0, "round": 1, "text": "Fix it."} ] })) .unwrap(), @@ -194,7 +224,12 @@ fn skips_multi_turn_run_when_conversation_shows_failed_prompt_read() { .unwrap(); let round_dir = paths[0].outputs_dir.join("turn-1"); fs::create_dir_all(&round_dir).unwrap(); - write_claude_events(&round_dir, "unused"); + write_claude_events_prompt_read( + &round_dir, + &prompt_path.to_string_lossy(), + "File is outside the allowed working directory.", + "I could not read the prompt file.", + ); let dispatch_path = iter.join("dispatch.json"); let mut dispatch: Value = @@ -225,7 +260,6 @@ fn does_not_record_partial_timing_when_a_conversation_round_transcript_is_missin &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: None, }], ); let conversation_path = iter @@ -283,7 +317,6 @@ fn a_responder_task_without_its_completion_artifact_is_skipped_as_incomplete() { &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: Some("Which cache?"), }], ); let round_dir = paths[0].outputs_dir.join("turn-1"); @@ -326,7 +359,6 @@ fn records_a_run_whose_conversation_timed_out_in_a_later_round() { &[FixtureTask { eval_id: "clarify", condition: "with_skill", - final_message: None, }], ); let conversation_path = iter diff --git a/src/pipeline/record_runs/tests/mod.rs b/src/pipeline/record_runs/tests/mod.rs index e19fdd2..6dad494 100644 --- a/src/pipeline/record_runs/tests/mod.rs +++ b/src/pipeline/record_runs/tests/mod.rs @@ -23,6 +23,42 @@ fn jsonl(lines: &[Value]) -> String { format!("{body}\n") } +fn transcript_dir(outputs_dir: &Path) -> PathBuf { + let is_round_dir = outputs_dir + .file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| name.starts_with("turn-")); + let dir = if is_round_dir { + outputs_dir.to_path_buf() + } else { + outputs_dir.join("turn-1") + }; + fs::create_dir_all(&dir).unwrap(); + dir +} + +fn write_transcript_file(outputs_dir: &Path, filename: &str, contents: impl AsRef<[u8]>) { + fs::write(transcript_dir(outputs_dir).join(filename), contents).unwrap(); +} + +fn write_one_shot_completion(path: &Path, prompt: &str) { + fs::write( + path, + serde_json::to_string_pretty(&json!({ + "status": "completed", + "delivered_followups": 0, + "events": [{ + "type": "user_message", + "ordinal": 0, + "round": 1, + "text": prompt + }] + })) + .unwrap(), + ) + .unwrap(); +} + fn write_codex_events(outputs_dir: &Path, final_text: &str) { let lines = vec![ json!({"type": "thread.started", "timestamp": "2026-06-04T10:00:00.000Z"}), @@ -30,7 +66,7 @@ fn write_codex_events(outputs_dir: &Path, final_text: &str) { json!({"type": "item.completed", "timestamp": "2026-06-04T10:00:20.000Z", "item": {"id": "item_2", "type": "agent_message", "text": final_text}}), json!({"type": "turn.completed", "timestamp": "2026-06-04T10:00:30.000Z", "usage": {"input_tokens": 100, "cached_input_tokens": 80, "output_tokens": 20, "reasoning_output_tokens": 5}}), ]; - fs::write(outputs_dir.join("codex-events.jsonl"), jsonl(&lines)).unwrap(); + write_transcript_file(outputs_dir, "codex-events.jsonl", jsonl(&lines)); } /// A `claude -p --output-format stream-json` events fixture: a `system/init` @@ -44,7 +80,7 @@ fn write_claude_events(outputs_dir: &Path, final_text: &str) { json!({"type": "user", "message": {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": "ok"}]}}), json!({"type": "result", "subtype": "success", "is_error": false, "result": final_text, "duration_ms": 30_000, "usage": {"input_tokens": 100, "output_tokens": 20, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5}}), ]; - fs::write(outputs_dir.join("claude-events.jsonl"), jsonl(&lines)).unwrap(); + write_transcript_file(outputs_dir, "claude-events.jsonl", jsonl(&lines)); } /// An `opencode run --format json` events fixture: a final `text` part and a @@ -55,7 +91,7 @@ fn write_opencode_events(outputs_dir: &Path, final_text: &str) { json!({"type": "text", "timestamp": 1_000, "sessionID": "ses_1", "part": {"id": "p1", "type": "text", "text": final_text}}), json!({"type": "step_finish", "timestamp": 2_000, "sessionID": "ses_1", "part": {"id": "p2", "type": "step-finish", "reason": "stop", "tokens": {"input": 1, "output": 1, "reasoning": 0, "cache": {"read": 0, "write": 0}}}}), ]; - fs::write(outputs_dir.join("opencode-events.jsonl"), jsonl(&lines)).unwrap(); + write_transcript_file(outputs_dir, "opencode-events.jsonl", jsonl(&lines)); } /// A `claude -p` events fixture where the agent reads its dispatch prompt: @@ -74,7 +110,7 @@ fn write_claude_events_prompt_read( json!({"type": "user", "message": {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": read_result}]}}), json!({"type": "result", "subtype": "success", "is_error": false, "result": final_text, "duration_ms": 30_000, "usage": {"input_tokens": 100, "output_tokens": 20, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5}}), ]; - fs::write(outputs_dir.join("claude-events.jsonl"), jsonl(&lines)).unwrap(); + write_transcript_file(outputs_dir, "claude-events.jsonl", jsonl(&lines)); } const PROMPT_SENTINEL: &str = @@ -83,8 +119,6 @@ const PROMPT_SENTINEL: &str = struct FixtureTask { eval_id: &'static str, condition: &'static str, - /// Written to `outputs/final-message.md` when `Some`. - final_message: Option<&'static str>, } /// Paths the tests reach into after building the iteration. @@ -104,9 +138,8 @@ fn write_iteration(iteration_dir: &Path, tasks: &[FixtureTask]) -> Vec Vec//evals/evals.json` with the given contents. diff --git a/src/validation/evals.rs b/src/validation/evals.rs index f849af8..d16a1e0 100644 --- a/src/validation/evals.rs +++ b/src/validation/evals.rs @@ -17,7 +17,9 @@ use crate::validation::schema::{SchemaName, validate_against_schema}; /// supplemental duplicate-`id`, command environment, and held-out path guards, /// returning the typed config on success. pub fn validate_evals_config(config: &Value, source: &str) -> Result { + validate_retired_fields(config, source)?; validate_codebase_declarations(config, source)?; + validate_effective_codebases(config, source)?; validate_turn_source_declarations(config, source)?; let validated: EvalsConfig = validate_against_schema(SchemaName::Evals, config, source)?; @@ -123,16 +125,16 @@ pub fn validate_evals_config(config: &Value, source: &str) -> Result Result Result<(), ValidationError> { + let evals = config.get("evals").and_then(Value::as_array); + for (index, eval) in evals.into_iter().flatten().enumerate() { + if eval.get("isolation").is_none() { + continue; + } + let id = eval + .get("id") + .and_then(Value::as_str) + .map_or_else(|| format!("evals[{index}]"), str::to_string); + return Err(ValidationError::InvalidConfig { + path: source.to_string(), + message: format!( + "eval '{id}': field 'isolation' is no longer supported or needed; every eval run already uses a private environment" + ), + }); + } + Ok(()) +} + +/// Every eval runs against a codebase. A config-level declaration supplies the +/// default; without one, each eval must carry its own declaration. +fn validate_effective_codebases(config: &Value, source: &str) -> Result<(), ValidationError> { + if config.get("codebase").is_some() { + return Ok(()); + } + let evals = config.get("evals").and_then(Value::as_array); + for (index, eval) in evals.into_iter().flatten().enumerate() { + if eval.get("codebase").is_some() { + continue; + } + let id = eval + .get("id") + .and_then(Value::as_str) + .map_or_else(|| format!("evals[{index}]"), str::to_string); + return Err(ValidationError::InvalidConfig { + path: source.to_string(), + message: format!( + "eval '{id}': no effective codebase; set top-level 'codebase' or this eval's 'codebase'" + ), + }); + } + Ok(()) +} + fn validate_guard_policy( policy: &crate::core::GuardPolicyConfig, source: &str, @@ -305,13 +355,14 @@ fn paths_overlap(left: &Path, right: &Path) -> bool { #[cfg(test)] mod tests { use super::validate_evals_config; - use crate::core::{Assertion, CodebaseSource}; + use crate::core::Assertion; use serde_json::{Value, json}; /// The minimal valid config the cases below mutate. fn base() -> Value { json!({ "skill_name": "demo", + "codebase": { "path": "." }, "evals": [ { "id": "e1", @@ -393,45 +444,6 @@ mod tests { assert!(err.contains("skill_should_trigger"), "error was: {err}"); } - #[test] - fn accepts_isolation_isolated() { - let mut config = base(); - config["evals"][0]["isolation"] = json!("isolated"); - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - assert_eq!( - parsed.evals[0].isolation, - Some(crate::core::Isolation::Isolated) - ); - } - - #[test] - fn accepts_isolation_shared() { - let mut config = base(); - config["evals"][0]["isolation"] = json!("shared"); - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - assert_eq!( - parsed.evals[0].isolation, - Some(crate::core::Isolation::Shared) - ); - } - - #[test] - fn defaults_isolation_to_none_when_absent() { - let config = base(); - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - assert_eq!(parsed.evals[0].isolation, None); - } - - #[test] - fn rejects_an_unknown_isolation_value() { - let mut config = base(); - config["evals"][0]["isolation"] = json!("sometimes"); - let err = validate_evals_config(&config, "evals.json") - .unwrap_err() - .to_string(); - assert!(err.contains("isolation"), "error was: {err}"); - } - #[test] fn rejects_a_non_kebab_case_id() { let mut config = base(); @@ -839,130 +851,10 @@ mod tests { let config = with_command_check(&["src/main.rs"], &["holdout/test.txt"]); validate_evals_config(&config, "evals.json").unwrap(); } - - #[test] - fn accepts_a_top_level_git_codebase_as_the_default() { - let mut config = base(); - config["codebase"] = json!({ "url": "https://example.com/project.git", "ref": "main" }); - - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - - assert_eq!( - parsed.codebase, - Some(CodebaseSource::Git { - url: "https://example.com/project.git".to_string(), - reference: "main".to_string(), - exclude_skill_sources: false, - }) - ); - } - - #[test] - fn accepts_a_per_eval_path_codebase_overriding_the_default() { - let mut config = base(); - config["codebase"] = json!({ "url": "https://example.com/project.git", "ref": "main" }); - config["evals"][0]["codebase"] = json!({ "path": "../fixtures/legacy-service" }); - - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - - assert_eq!( - parsed.evals[0].codebase, - Some(CodebaseSource::Path { - path: "../fixtures/legacy-service".to_string(), - exclude_skill_sources: false, - }) - ); - } - - #[test] - fn accepts_a_top_level_path_codebase() { - let mut config = base(); - config["codebase"] = json!({ "path": "/srv/projects/legacy-service" }); - - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - - assert_eq!( - parsed.codebase, - Some(CodebaseSource::Path { - path: "/srv/projects/legacy-service".to_string(), - exclude_skill_sources: false, - }) - ); - } - - #[test] - fn accepts_codebase_skill_source_exclusion() { - let mut config = base(); - config["codebase"] = json!({ - "path": "/srv/projects/legacy-service", - "exclude_skill_sources": true - }); - - let parsed = validate_evals_config(&config, "evals.json").unwrap(); - let declared = serde_json::to_value(parsed.codebase.unwrap()).unwrap(); - - assert_eq!(declared["exclude_skill_sources"], true); - } - - /// `minLength: 1` admits `" "`, so the schema cannot carry this on its own. - #[test] - fn rejects_whitespace_only_codebase_values() { - for (field, codebase) in [ - ("url", json!({ "url": " ", "ref": "main" })), - ( - "ref", - json!({ "url": "https://example.com/p.git", "ref": "\t" }), - ), - ("path", json!({ "path": " " })), - ] { - let mut config = base(); - config["codebase"] = codebase.clone(); - let error = validate_evals_config(&config, "evals.json") - .unwrap_err() - .to_string(); - assert!(error.contains("codebase"), "{field}: error was: {error}"); - assert!(error.contains(field), "{field}: error was: {error}"); - - // The per-eval override runs through the same guard, and names the eval. - let mut config = base(); - config["evals"][0]["codebase"] = codebase; - let error = validate_evals_config(&config, "evals.json") - .unwrap_err() - .to_string(); - assert!(error.contains("e1"), "{field}: error was: {error}"); - assert!(error.contains(field), "{field}: error was: {error}"); - } - } - - /// A source is one thing or the other. The schema's `oneOf` plus - /// `additionalProperties: false` on each branch is what rejects the hybrid; - /// this pins that so a later schema edit cannot quietly admit it. - #[test] - fn rejects_a_codebase_that_is_both_git_and_path() { - let mut config = base(); - config["codebase"] = json!({ - "url": "https://example.com/p.git", - "ref": "main", - "path": "/srv/p" - }); - - assert!(validate_evals_config(&config, "evals.json").is_err()); - } - - /// #244 decision 5: the runner records the resolved SHA, so a git source - /// without an explicit ref could not be re-run against what it measured. - #[test] - fn rejects_a_git_codebase_without_a_ref() { - let mut config = base(); - config["codebase"] = json!({ "url": "https://example.com/p.git" }); - - let error = validate_evals_config(&config, "evals.json") - .unwrap_err() - .to_string(); - - assert!(error.contains("ref"), "error was: {error}"); - } } +#[cfg(test)] +mod codebase_tests; + #[cfg(test)] mod multi_skill_tests; diff --git a/src/validation/evals/codebase_tests.rs b/src/validation/evals/codebase_tests.rs new file mode 100644 index 0000000..bb3c39f --- /dev/null +++ b/src/validation/evals/codebase_tests.rs @@ -0,0 +1,206 @@ +use super::validate_evals_config; +use crate::core::{CodebaseSource, EvalsConfig}; +use crate::validation::schema::{SchemaName, validate_against_schema}; +use serde_json::{Value, json}; + +fn base() -> Value { + json!({ + "skill_name": "demo", + "codebase": { "path": "." }, + "evals": [ + { + "id": "e1", + "prompt": "do the thing", + "expected_output": "the thing is done" + } + ] + }) +} + +#[test] +fn rejects_isolation_with_the_retirement_message() { + let mut config = base(); + config["evals"][0]["isolation"] = json!("isolated"); + let err = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + assert!( + err.contains( + "eval 'e1': field 'isolation' is no longer supported or needed; every eval run already uses a private environment" + ), + "error was: {err}" + ); +} + +#[test] +fn isolation_error_wins_when_the_config_also_lacks_a_codebase() { + let mut config = base(); + config.as_object_mut().unwrap().remove("codebase"); + config["evals"][0]["isolation"] = json!("shared"); + + let err = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + + assert!(err.contains("field 'isolation'"), "error was: {err}"); + assert!(!err.contains("no effective codebase"), "error was: {err}"); +} + +#[test] +fn rejects_an_eval_without_an_effective_codebase() { + let mut config = base(); + config.as_object_mut().unwrap().remove("codebase"); + + let err = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + + assert!( + err.contains( + "eval 'e1': no effective codebase; set top-level 'codebase' or this eval's 'codebase'" + ), + "error was: {err}" + ); +} + +#[test] +fn schema_requires_a_default_or_per_eval_codebase() { + let mut config = base(); + config.as_object_mut().unwrap().remove("codebase"); + + let result: Result = + validate_against_schema(SchemaName::Evals, &config, "evals.json"); + + assert!(result.is_err()); +} + +#[test] +fn accepts_per_eval_codebases_without_a_default() { + let mut config = base(); + config.as_object_mut().unwrap().remove("codebase"); + config["evals"][0]["codebase"] = json!({ "path": "../project" }); + + let parsed = validate_evals_config(&config, "evals.json").unwrap(); + + assert!(parsed.codebase.is_none()); + assert!(parsed.evals[0].codebase.is_some()); +} + +#[test] +fn accepts_a_top_level_git_codebase_as_the_default() { + let mut config = base(); + config["codebase"] = json!({ "url": "https://example.com/project.git", "ref": "main" }); + + let parsed = validate_evals_config(&config, "evals.json").unwrap(); + + assert_eq!( + parsed.codebase, + Some(CodebaseSource::Git { + url: "https://example.com/project.git".to_string(), + reference: "main".to_string(), + exclude_skill_sources: false, + }) + ); +} + +#[test] +fn accepts_a_per_eval_path_codebase_overriding_the_default() { + let mut config = base(); + config["codebase"] = json!({ "url": "https://example.com/project.git", "ref": "main" }); + config["evals"][0]["codebase"] = json!({ "path": "../projects/legacy-service" }); + + let parsed = validate_evals_config(&config, "evals.json").unwrap(); + + assert_eq!( + parsed.evals[0].codebase, + Some(CodebaseSource::Path { + path: "../projects/legacy-service".to_string(), + exclude_skill_sources: false, + }) + ); +} + +#[test] +fn accepts_a_top_level_path_codebase() { + let mut config = base(); + config["codebase"] = json!({ "path": "/srv/projects/legacy-service" }); + + let parsed = validate_evals_config(&config, "evals.json").unwrap(); + + assert_eq!( + parsed.codebase, + Some(CodebaseSource::Path { + path: "/srv/projects/legacy-service".to_string(), + exclude_skill_sources: false, + }) + ); +} + +#[test] +fn accepts_codebase_skill_source_exclusion() { + let mut config = base(); + config["codebase"] = json!({ + "path": "/srv/projects/legacy-service", + "exclude_skill_sources": true + }); + + let parsed = validate_evals_config(&config, "evals.json").unwrap(); + let declared = serde_json::to_value(parsed.codebase.unwrap()).unwrap(); + + assert_eq!(declared["exclude_skill_sources"], true); +} + +/// `minLength: 1` admits `" "`, so the schema cannot carry this on its own. +#[test] +fn rejects_whitespace_only_codebase_values() { + for (field, codebase) in [ + ("url", json!({ "url": " ", "ref": "main" })), + ( + "ref", + json!({ "url": "https://example.com/p.git", "ref": "\t" }), + ), + ("path", json!({ "path": " " })), + ] { + let mut config = base(); + config["codebase"] = codebase.clone(); + let error = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + assert!(error.contains("codebase"), "{field}: error was: {error}"); + assert!(error.contains(field), "{field}: error was: {error}"); + + let mut config = base(); + config["evals"][0]["codebase"] = codebase; + let error = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + assert!(error.contains("e1"), "{field}: error was: {error}"); + assert!(error.contains(field), "{field}: error was: {error}"); + } +} + +/// A source is either Git or a local path; hybrid declarations are invalid. +#[test] +fn rejects_a_codebase_that_is_both_git_and_path() { + let mut config = base(); + config["codebase"] = json!({ + "url": "https://example.com/p.git", + "ref": "main", + "path": "/srv/p" + }); + + assert!(validate_evals_config(&config, "evals.json").is_err()); +} + +/// A Git source needs an explicit ref so a run can be reproduced from its resolved SHA. +#[test] +fn rejects_a_git_codebase_without_a_ref() { + let mut config = base(); + config["codebase"] = json!({ "url": "https://example.com/p.git" }); + + let error = validate_evals_config(&config, "evals.json") + .unwrap_err() + .to_string(); + + assert!(error.contains("ref"), "error was: {error}"); +} diff --git a/src/validation/evals/multi_skill_tests.rs b/src/validation/evals/multi_skill_tests.rs index 4c3ac13..a51a670 100644 --- a/src/validation/evals/multi_skill_tests.rs +++ b/src/validation/evals/multi_skill_tests.rs @@ -5,6 +5,7 @@ use super::validate_evals_config; fn base() -> Value { json!({ "skill_name": "demo", + "codebase": { "path": "." }, "evals": [{ "id": "e1", "prompt": "do the thing", diff --git a/src/validation/evals_guard_tests.rs b/src/validation/evals_guard_tests.rs index 5f94bd9..ab6968e 100644 --- a/src/validation/evals_guard_tests.rs +++ b/src/validation/evals_guard_tests.rs @@ -5,6 +5,7 @@ use super::evals::validate_evals_config; fn base() -> Value { json!({ "skill_name": "demo", + "codebase": { "path": "." }, "evals": [{ "id": "e1", "prompt": "do the thing", diff --git a/tests/cli/aggregate/shadow.rs b/tests/cli/aggregate/shadow.rs index db6465c..b75a9d3 100644 --- a/tests/cli/aggregate/shadow.rs +++ b/tests/cli/aggregate/shadow.rs @@ -399,6 +399,10 @@ fn aggregate_echoes_the_resolved_codebases_into_the_benchmark() { agg_cmd(&cwd, &skill_dir).assert().success(); let b = read_benchmark(&iteration_dir); + assert_eq!( + b["codebases"][0]["source"], + "https://example.com/project.git" + ); assert_eq!( b["codebases"][0]["revision"], "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678" diff --git a/tests/cli/basics.rs b/tests/cli/basics.rs index 19aae19..55d5fa1 100644 --- a/tests/cli/basics.rs +++ b/tests/cli/basics.rs @@ -7,7 +7,7 @@ use std::fs; use tempfile::TempDir; /// A minimal valid `evals.json` body. -const VALID_EVALS: &str = r#"{ "skill_name": "demo", "evals": [ +const VALID_EVALS: &str = r#"{ "skill_name": "demo", "codebase": { "path": "." }, "evals": [ { "id": "e1", "prompt": "p", "expected_output": "o" } ] }"#; /// Build `//evals/evals.json` with the given contents. @@ -95,7 +95,6 @@ fn every_visible_command_and_harness_subcommand_renders_help() { "compare --help", "finalize --help", "record-runs --help", - "fill-transcripts --help", "detect-stray-writes --help", "grade --help", "aggregate --help", @@ -117,6 +116,15 @@ fn every_visible_command_and_harness_subcommand_renders_help() { } } +#[test] +fn retired_fill_transcripts_command_is_not_exposed() { + skill_eval() + .args(["fill-transcripts", "--help"]) + .assert() + .failure() + .stderr(contains("unrecognized subcommand 'fill-transcripts'")); +} + #[test] fn init_help_documents_extended_eval_authoring() { skill_eval() @@ -125,6 +133,7 @@ fn init_help_documents_extended_eval_authoring() { .success() .stdout(contains("turns")) .stdout(contains("files_root")) + .stdout(contains("overlay sources")) .stdout(contains("per-eval `runs`")) .stdout(contains("eval-magic validate")); } diff --git a/tests/cli/command_check.rs b/tests/cli/command_check.rs index f501da8..2f28a1a 100644 --- a/tests/cli/command_check.rs +++ b/tests/cli/command_check.rs @@ -19,6 +19,7 @@ fn finalize_folds_command_check_result_into_normal_pass_rate() { skill_sub.join("evals/evals.json"), serde_json::to_string_pretty(&json!({ "skill_name": "mr-review", + "codebase": { "path": "." }, "evals": [{ "id": "pos-eval", "prompt": "Fix it.", diff --git a/tests/cli/docs.rs b/tests/cli/docs.rs index 352308f..3f56744 100644 --- a/tests/cli/docs.rs +++ b/tests/cli/docs.rs @@ -281,7 +281,6 @@ fn every_guide_reference_in_shipped_help_resolves() { "ingest --help", "finalize --help", "record-runs --help", - "fill-transcripts --help", "detect-stray-writes --help", "grade --help", "aggregate --help", diff --git a/tests/cli/docs/codebase.rs b/tests/cli/docs/codebase.rs index 6dc9a45..1512f63 100644 --- a/tests/cli/docs/codebase.rs +++ b/tests/cli/docs/codebase.rs @@ -4,10 +4,10 @@ use crate::helpers::skill_eval; use predicates::str::contains; /// The parts a config author cannot infer have to survive an edit: init's -/// source modes and fixture choices, the required git ref, overlay semantics, +/// source modes and project choices, the required git ref, overlay semantics, /// local-path portability, cache provisioning, and the measured baseline. #[test] -fn keeps_declaration_rules_fixture_choices_caveat_and_provisioning_contract() { +fn keeps_declaration_rules_project_choices_caveat_and_provisioning_contract() { skill_eval() .args(["docs", "codebase"]) .assert() @@ -19,7 +19,7 @@ fn keeps_declaration_rules_fixture_choices_caveat_and_provisioning_contract() { .stdout(contains("--codebase-path")) .stdout(contains("--codebase-cwd")) .stdout(contains("Weeknight")) - .stdout(contains("eval-magic as a complex fixture")) + .stdout(contains("eval-magic as a complex project")) .stdout(contains("\"ref\"")) .stdout(contains("`ref` is required")) .stdout(contains("overlay")) diff --git a/tests/cli/grade.rs b/tests/cli/grade.rs index 8d28d61..790d9dd 100644 --- a/tests/cli/grade.rs +++ b/tests/cli/grade.rs @@ -1,6 +1,6 @@ //! The `grade` subcommand — judge-task emission and `--finalize` folding. -use crate::helpers::{canonical_root, skill_eval}; +use crate::helpers::{canonical_root, skill_eval, with_default_codebase}; use assert_cmd::Command; use predicates::str::contains; use std::fs; @@ -13,7 +13,7 @@ fn write_skill(skill_sub: &std::path::Path, skill_md: &str, evals: &serde_json:: fs::write(skill_sub.join("SKILL.md"), skill_md).unwrap(); fs::write( skill_sub.join("evals").join("evals.json"), - serde_json::to_string_pretty(evals).unwrap(), + serde_json::to_string_pretty(&with_default_codebase(evals)).unwrap(), ) .unwrap(); } diff --git a/tests/cli/grade_models.rs b/tests/cli/grade_models.rs index e5f244e..ffa55e0 100644 --- a/tests/cli/grade_models.rs +++ b/tests/cli/grade_models.rs @@ -1,6 +1,6 @@ //! Model-selection behavior for emitted judge tasks. -use crate::helpers::{canonical_root, skill_eval}; +use crate::helpers::{canonical_root, skill_eval, with_default_codebase}; use assert_cmd::Command; use std::fs; @@ -9,7 +9,7 @@ fn write_skill(skill_sub: &std::path::Path, skill_md: &str, evals: &serde_json:: fs::write(skill_sub.join("SKILL.md"), skill_md).unwrap(); fs::write( skill_sub.join("evals").join("evals.json"), - serde_json::to_string_pretty(evals).unwrap(), + serde_json::to_string_pretty(&with_default_codebase(evals)).unwrap(), ) .unwrap(); } diff --git a/tests/cli/harness.rs b/tests/cli/harness.rs index c10cee3..ee83473 100644 --- a/tests/cli/harness.rs +++ b/tests/cli/harness.rs @@ -10,6 +10,7 @@ use std::path::Path; use tempfile::TempDir; mod lint_modes; +mod probe; /// Write `/.eval-magic/harnesses/` with the given TOML. fn write_project_descriptor(root: &Path, file: &str, contents: &str) { @@ -428,168 +429,3 @@ fn missing_harness_file_is_fatal() { .failure() .stderr(contains("--harness-file").and(contains("missing.toml"))); } - -// --------------------------------------------------------------------------- -// `harness lint --probe`: live dispatch probe for the exec template. -// --------------------------------------------------------------------------- -// -// The probe invokes the real harness CLI (network/tokens) and is opt-in, never -// part of standard CI. The tests below use fake shell `exec_template`s that -// write the final-message file themselves — so no agent CLI is ever invoked -// and the suite stays hermetic. - -/// Descriptor with a fake `exec_template` that writes the final message the -/// probe looks for. Used as the "probe succeeds" baseline. -const PROBE_OK_TOML: &str = "label = \"probe-ok\"\n\n\ - [dispatch]\n\ - exec_template = 'printf \"ok\\n\" > /final-message.md'\n"; - -#[test] -fn harness_lint_probe_recovers_final_message_for_fake_exec_template() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("probe-ok.toml"); - fs::write(&file, PROBE_OK_TOML).unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes"]) - .assert() - .success() - .stdout(contains("✓ live exec template")); -} - -#[test] -fn harness_lint_probe_applies_descriptor_agent_environment() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("probe-env.toml"); - fs::write( - &file, - r#" -label = "probe-env" - -[dispatch] -exec_template = '[ "$PROBE_ENV" = "visible" ] && printf "ok\n" > /final-message.md' - -[dispatch.env] -PROBE_ENV = "visible" -"#, - ) - .unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes"]) - .assert() - .success() - .stdout(contains("✓ live exec template")); -} - -#[test] -fn harness_lint_probe_runs_against_a_registered_name() { - let tmp = TempDir::new().unwrap(); - write_project_descriptor(tmp.path(), "probe-ok.toml", PROBE_OK_TOML); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint", "probe-ok", "--probe", "--yes"]) - .assert() - .success() - .stdout(contains("✓ live exec template")); -} - -#[test] -fn harness_lint_probe_fails_when_final_message_missing() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("probe-bad.toml"); - fs::write( - &file, - "label = \"probe-bad\"\n\n[dispatch]\nexec_template = 'true'\n", - ) - .unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes"]) - .assert() - .failure() - .stderr(contains("✗").and(contains("final-message.md"))); -} - -#[test] -fn harness_lint_probe_aborts_without_yes_on_non_yes_stdin() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("probe-ok.toml"); - fs::write(&file, PROBE_OK_TOML).unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe"]) - .write_stdin("n\n") - .assert() - .failure() - .stderr( - contains("About to execute") - .and(contains("aborted")) - .and(contains("✓ live exec template").not()), - ); -} - -#[test] -fn harness_lint_probe_timeout_kills_a_long_command() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("probe-slow.toml"); - fs::write( - &file, - "label = \"probe-slow\"\n\n[dispatch]\nexec_template = 'sleep 3'\n", - ) - .unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes", "--probe-timeout", "1"]) - .assert() - .failure() - .stderr(contains("✗").and(contains("timed out"))); -} - -#[test] -fn harness_lint_probe_without_exec_template_reports_nothing_to_run() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("no-dispatch.toml"); - fs::write(&file, "label = \"no-dispatch\"\n").unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes"]) - .assert() - .failure() - .stderr(contains("✗").and(contains("dispatch.exec_template"))); -} - -#[test] -fn harness_lint_probe_does_not_run_after_static_checks_fail() { - let tmp = TempDir::new().unwrap(); - let file = tmp.path().join("broken.toml"); - fs::write(&file, "label = \"broken\"\nmystery = 1\n").unwrap(); - - skill_eval() - .current_dir(tmp.path()) - .args(["harness", "lint"]) - .arg(&file) - .args(["--probe", "--yes"]) - .assert() - .failure() - .stderr(contains("mystery").and(contains("About to execute").not())); -} diff --git a/tests/cli/harness/probe.rs b/tests/cli/harness/probe.rs new file mode 100644 index 0000000..ccca498 --- /dev/null +++ b/tests/cli/harness/probe.rs @@ -0,0 +1,181 @@ +//! Hermetic coverage for the opt-in live dispatch probe. + +use super::*; + +/// Descriptor with a fake `exec_template` that writes a parseable transcript. +const PROBE_OK_TOML: &str = r#"label = "probe-ok" + +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "probe-events.jsonl" +parser = "codex-items" + +[dispatch] +exec_template = '''printf '%s\n' '{"type":"item.completed","item":{"id":"m1","type":"agent_message","text":"ok"}}' > /probe-events.jsonl''' +"#; + +#[test] +fn harness_lint_probe_recovers_final_response_from_fake_transcript() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("probe-ok.toml"); + fs::write(&file, PROBE_OK_TOML).unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes"]) + .assert() + .success() + .stdout(contains( + "✓ live exec template: transcript final response recovered", + )); +} + +#[test] +fn harness_lint_probe_applies_descriptor_agent_environment() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("probe-env.toml"); + fs::write( + &file, + r#" +label = "probe-env" + +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "probe-events.jsonl" +parser = "codex-items" + +[dispatch] +exec_template = '''[ "$PROBE_ENV" = "visible" ] && printf '%s\n' '{"type":"item.completed","item":{"id":"m1","type":"agent_message","text":"ok"}}' > /probe-events.jsonl''' + +[dispatch.env] +PROBE_ENV = "visible" +"#, + ) + .unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes"]) + .assert() + .success() + .stdout(contains("✓ live exec template")); +} + +#[test] +fn harness_lint_probe_runs_against_a_registered_name() { + let tmp = TempDir::new().unwrap(); + write_project_descriptor(tmp.path(), "probe-ok.toml", PROBE_OK_TOML); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint", "probe-ok", "--probe", "--yes"]) + .assert() + .success() + .stdout(contains("✓ live exec template")); +} + +#[test] +fn harness_lint_probe_fails_when_transcript_is_missing() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("probe-bad.toml"); + fs::write( + &file, + PROBE_OK_TOML.replace( + "exec_template = '''printf '%s\\n' '{\"type\":\"item.completed\",\"item\":{\"id\":\"m1\",\"type\":\"agent_message\",\"text\":\"ok\"}}' > /probe-events.jsonl'''", + "exec_template = 'true'", + ), + ) + .unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes"]) + .assert() + .failure() + .stderr(contains("✗").and(contains("probe-events.jsonl"))); +} + +#[test] +fn harness_lint_probe_aborts_without_yes_on_non_yes_stdin() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("probe-ok.toml"); + fs::write(&file, PROBE_OK_TOML).unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe"]) + .write_stdin("n\n") + .assert() + .failure() + .stderr( + contains("About to execute") + .and(contains("aborted")) + .and(contains("✓ live exec template").not()), + ); +} + +#[test] +fn harness_lint_probe_timeout_kills_a_long_command() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("probe-slow.toml"); + fs::write( + &file, + "label = \"probe-slow\"\n\n[dispatch]\nexec_template = 'sleep 3'\n", + ) + .unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes", "--probe-timeout", "1"]) + .assert() + .failure() + .stderr(contains("✗").and(contains("timed out"))); +} + +#[test] +fn harness_lint_probe_without_exec_template_reports_nothing_to_run() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("no-dispatch.toml"); + fs::write(&file, "label = \"no-dispatch\"\n").unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes"]) + .assert() + .failure() + .stderr(contains("✗").and(contains("dispatch.exec_template"))); +} + +#[test] +fn harness_lint_probe_does_not_run_after_static_checks_fail() { + let tmp = TempDir::new().unwrap(); + let file = tmp.path().join("broken.toml"); + fs::write(&file, "label = \"broken\"\nmystery = 1\n").unwrap(); + + skill_eval() + .current_dir(tmp.path()) + .args(["harness", "lint"]) + .arg(&file) + .args(["--probe", "--yes"]) + .assert() + .failure() + .stderr(contains("mystery").and(contains("About to execute").not())); +} diff --git a/tests/cli/helpers.rs b/tests/cli/helpers.rs index 2188d74..c300514 100644 --- a/tests/cli/helpers.rs +++ b/tests/cli/helpers.rs @@ -1,6 +1,7 @@ //! Shared helpers for the `cli` integration tests. use assert_cmd::Command; +use serde_json::{Value, json}; use std::fs; use std::path::{Path, PathBuf}; use tempfile::TempDir; @@ -33,3 +34,13 @@ pub fn canonical_root() -> (TempDir, PathBuf) { let root = resolved(tmp.path()); (tmp, root) } + +/// Give a test-authored eval config the mandatory local codebase when its +/// subject is a later pipeline stage rather than environment provisioning. +pub fn with_default_codebase(evals: &Value) -> Value { + let mut evals = evals.clone(); + if evals.get("codebase").is_none() { + evals["codebase"] = json!({ "path": "." }); + } + evals +} diff --git a/tests/cli/init.rs b/tests/cli/init.rs index bfd0d94..e807c6a 100644 --- a/tests/cli/init.rs +++ b/tests/cli/init.rs @@ -62,7 +62,10 @@ fn init_with_flags_writes_valid_seed_evals() { .stderr("") .stdout(contains("Initialized evals for mr-review")) .stdout(contains("eval-magic run --skill-dir")) - .stdout(contains("eval-magic promote-baseline")); + .stdout(contains("follow the generated RUNBOOK.md")) + .stdout(contains("eval-magic ingest").not()) + .stdout(contains("eval-magic finalize").not()) + .stdout(contains("eval-magic promote-baseline").not()); let written = fs::read_to_string(skill_sub.join("evals/evals.json")).unwrap(); let parsed: serde_json::Value = serde_json::from_str(&written).unwrap(); @@ -268,12 +271,11 @@ fn init_from_skill_dir_prints_copy_pasteable_next_steps() { .success() .stdout(contains(" eval-magic run --skill-dir")) .stdout(contains("--skill mr-review --workspace-dir")) - .stdout(contains("--guard")) - // ingest auto-resolves the subagents dir now, so the placeholder is gone. - .stdout(contains(" eval-magic ingest --skill-dir")) - .stdout(contains("--subagents-dir ").not()) - .stdout(contains(" eval-magic finalize --skill-dir")) - .stdout(contains(" eval-magic promote-baseline --skill-dir")); + .stdout(contains("--guard").not()) + .stdout(contains("follow the generated RUNBOOK.md")) + .stdout(contains(" eval-magic ingest --skill-dir").not()) + .stdout(contains(" eval-magic finalize --skill-dir").not()) + .stdout(contains(" eval-magic promote-baseline --skill-dir").not()); } #[test] diff --git a/tests/cli/stray_writes.rs b/tests/cli/stray_writes.rs index 55353a0..0f81af5 100644 --- a/tests/cli/stray_writes.rs +++ b/tests/cli/stray_writes.rs @@ -140,8 +140,8 @@ fn detect_stray_writes_flags_unverifiable_when_nothing_was_inspected() { ) .unwrap(); - // A recorded run whose transcript never linked: final message present, - // tool_invocations empty. + // A historical recorded run whose transcript never linked: normalized final + // response present, tool_invocations empty. fs::write( cond_dir.join("run.json"), serde_json::to_string(&json!({ diff --git a/tests/golden/claude-code/dispatch-prompt.golden.txt b/tests/golden/claude-code/dispatch-prompt.golden.txt index 05f795a..85bc51c 100644 --- a/tests/golden/claude-code/dispatch-prompt.golden.txt +++ b/tests/golden/claude-code/dispatch-prompt.golden.txt @@ -19,17 +19,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable via the Skill tool. If you invoke it, use that identifier. If the Skill tool cannot resolve that identifier, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/claude-code/manifest.golden.md b/tests/golden/claude-code/manifest.golden.md index 51b1937..2e863dd 100644 --- a/tests/golden/claude-code/manifest.golden.md +++ b/tests/golden/claude-code/manifest.golden.md @@ -35,11 +35,9 @@ Then run `eval-magic ingest --harness claude-code`; ingest reads each task's `ou After all dispatches: -1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json` + the task's own `outputs/final-message.md` + the events file the harness CLI wrote under `outputs/turn-/`, and backfills `timing.json` with transcript-derived tokens/duration; never clobbers an existing record), fill-transcripts, detect-stray-writes, and grade. Optional higher-fidelity timing: write `{ "total_tokens": , "duration_ms": , "source": "completion-event" }` from the task completion event to `timing.json` right after a dispatch — completion-event numbers always win over the backfill. +1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json`, `conversation.json`, and the harness events under `outputs/turn-/`, and backfills `timing.json`; never clobbers an existing record), detect-stray-writes, and grade. 2. Run `eval-magic dispatch --judges --harness ` to grade the judge tasks ingest listed, then `eval-magic finalize` for the benchmark. -On a harness without persisted transcripts, instead write each task's `run.json` (matching `skills/evaluating-skills/schema/run-record.schema.json`, enforced at runtime by grade/fill-transcripts/detect-stray-writes) and `timing.json` by hand when its subagent returns: carry over `eval_id`, `condition`, `skill_path` (`null` on the without_skill arm), `prompt`, and `files` from the task; populate `final_message` from the subagent's reply; leave `tool_invocations` as `[]`; capture `total_tokens`/`duration_ms` from the task completion event immediately — they may not be persisted anywhere else. - ## Dispatches ### demo-eval / with_skill @@ -69,17 +67,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable via the Skill tool. If you invoke it, use that identifier. If the Skill tool cannot resolve that identifier, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: @@ -98,17 +93,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. No skill is loaded. Respond as you naturally would. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task-b Task-local scratch directory: /work/task-b/tmp -Framework output directory: /work/outputs-b Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs-b/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/cline/dispatch-prompt.golden.txt b/tests/golden/cline/dispatch-prompt.golden.txt index b744ccd..d2fa865 100644 --- a/tests/golden/cline/dispatch-prompt.golden.txt +++ b/tests/golden/cline/dispatch-prompt.golden.txt @@ -19,17 +19,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable as a Cline skill. If you invoke it, use that identifier. If it does not load as a Cline skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/cline/manifest.golden.md b/tests/golden/cline/manifest.golden.md index 528b643..ecae026 100644 --- a/tests/golden/cline/manifest.golden.md +++ b/tests/golden/cline/manifest.golden.md @@ -20,7 +20,7 @@ It runs `--jobs` tasks at a time, each in its own private environment, and write Harness dispatch (Cline): -`eval-magic dispatch` runs one fresh `cline --cwd --act --json --auto-approve true` per task. Detach stdin with `` so piped task data cannot become extra prompt context; capture stdout as `outputs/turn-/cline-events.jsonl` and stderr as `outputs/turn-/cline-stderr.log`. `eval-magic dispatch` writes `outputs/final-message.md` itself from the parsed transcript; the template's trailing jq step is a belt-and-braces copy of the terminal `run_result` event. +`eval-magic dispatch` runs one fresh `cline --cwd --act --json --auto-approve true` per task. Detach stdin with `` so piped task data cannot become extra prompt context; capture stdout as `outputs/turn-/cline-events.jsonl` and stderr as `outputs/turn-/cline-stderr.log`. Ingest recovers the final response directly from the terminal `run_result` event. ```bash unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_COMMON_DIR GIT_CEILING_DIRECTORIES @@ -28,20 +28,16 @@ cline --cwd --act --json --auto-approve true -m model-x \ "Read the file at and follow its instructions exactly. When you finish, make your final response your closing summary." \ /cline-events.jsonl \ - 2> /cline-stderr.log; \ - jq -rj 'select(.type == "run_result") | .text' /cline-events.jsonl \ - > /final-message.md + 2> /cline-stderr.log ``` Then run `eval-magic ingest --harness cline`; ingest reads each task's `outputs/turn-/cline-events.jsonl`. After all dispatches: -1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json` + the task's own `outputs/final-message.md` + the events file the harness CLI wrote under `outputs/turn-/`, and backfills `timing.json` with transcript-derived tokens/duration; never clobbers an existing record), fill-transcripts, detect-stray-writes, and grade. Optional higher-fidelity timing: write `{ "total_tokens": , "duration_ms": , "source": "completion-event" }` from the task completion event to `timing.json` right after a dispatch — completion-event numbers always win over the backfill. +1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json`, `conversation.json`, and the harness events under `outputs/turn-/`, and backfills `timing.json`; never clobbers an existing record), detect-stray-writes, and grade. 2. Run `eval-magic dispatch --judges --harness ` to grade the judge tasks ingest listed, then `eval-magic finalize` for the benchmark. -On a harness without persisted transcripts, instead write each task's `run.json` (matching `skills/evaluating-skills/schema/run-record.schema.json`, enforced at runtime by grade/fill-transcripts/detect-stray-writes) and `timing.json` by hand when its subagent returns: carry over `eval_id`, `condition`, `skill_path` (`null` on the without_skill arm), `prompt`, and `files` from the task; populate `final_message` from the subagent's reply; leave `tool_invocations` as `[]`; capture `total_tokens`/`duration_ms` from the task completion event immediately — they may not be persisted anywhere else. - ## Dispatches ### demo-eval / with_skill @@ -71,17 +67,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable as a Cline skill. If you invoke it, use that identifier. If it does not load as a Cline skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: @@ -100,17 +93,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. No skill is loaded. Respond as you naturally would. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task-b Task-local scratch directory: /work/task-b/tmp -Framework output directory: /work/outputs-b Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs-b/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/codex/dispatch-prompt.golden.txt b/tests/golden/codex/dispatch-prompt.golden.txt index d014e37..2eeae1b 100644 --- a/tests/golden/codex/dispatch-prompt.golden.txt +++ b/tests/golden/codex/dispatch-prompt.golden.txt @@ -19,17 +19,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable as a Codex skill. If you invoke it, use that identifier. If it does not load as a Codex skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/codex/manifest.golden.md b/tests/golden/codex/manifest.golden.md index e262baf..9d96163 100644 --- a/tests/golden/codex/manifest.golden.md +++ b/tests/golden/codex/manifest.golden.md @@ -25,8 +25,7 @@ Harness dispatch (Codex): ```bash unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_COMMON_DIR GIT_CEILING_DIRECTORIES codex --ask-for-approval never exec --cd --sandbox workspace-write --dangerously-bypass-hook-trust -m model-x --json \ - --output-last-message /final-message.md \ - "Read the file at and follow its instructions exactly. When you finish, make your final response exactly the same text you wrote to /final-message.md." \ + "Read the file at and follow its instructions exactly. When you finish, make your final response your closing summary." \ /codex-events.jsonl \ 2> /codex-stderr.log @@ -36,11 +35,9 @@ Then run `eval-magic ingest --harness codex`; Codex transcript ingest reads each After all dispatches: -1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json` + the task's own `outputs/final-message.md` + the events file the harness CLI wrote under `outputs/turn-/`, and backfills `timing.json` with transcript-derived tokens/duration; never clobbers an existing record), fill-transcripts, detect-stray-writes, and grade. Optional higher-fidelity timing: write `{ "total_tokens": , "duration_ms": , "source": "completion-event" }` from the task completion event to `timing.json` right after a dispatch — completion-event numbers always win over the backfill. +1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json`, `conversation.json`, and the harness events under `outputs/turn-/`, and backfills `timing.json`; never clobbers an existing record), detect-stray-writes, and grade. 2. Run `eval-magic dispatch --judges --harness ` to grade the judge tasks ingest listed, then `eval-magic finalize` for the benchmark. -On a harness without persisted transcripts, instead write each task's `run.json` (matching `skills/evaluating-skills/schema/run-record.schema.json`, enforced at runtime by grade/fill-transcripts/detect-stray-writes) and `timing.json` by hand when its subagent returns: carry over `eval_id`, `condition`, `skill_path` (`null` on the without_skill arm), `prompt`, and `files` from the task; populate `final_message` from the subagent's reply; leave `tool_invocations` as `[]`; capture `total_tokens`/`duration_ms` from the task completion event immediately — they may not be persisted anywhere else. - ## Dispatches ### demo-eval / with_skill @@ -70,17 +67,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with_skill__widget-skill` and is discoverable as a Codex skill. If you invoke it, use that identifier. If it does not load as a Codex skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: @@ -99,17 +93,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. No skill is loaded. Respond as you naturally would. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task-b Task-local scratch directory: /work/task-b/tmp -Framework output directory: /work/outputs-b Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs-b/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/opencode/dispatch-prompt.golden.txt b/tests/golden/opencode/dispatch-prompt.golden.txt index 7351375..ee68809 100644 --- a/tests/golden/opencode/dispatch-prompt.golden.txt +++ b/tests/golden/opencode/dispatch-prompt.golden.txt @@ -25,17 +25,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with-skill-widget-skill` and is discoverable as an OpenCode skill. If you invoke it, use that identifier. If it does not load as an OpenCode skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/golden/opencode/manifest.golden.md b/tests/golden/opencode/manifest.golden.md index a6d67c1..2c61b07 100644 --- a/tests/golden/opencode/manifest.golden.md +++ b/tests/golden/opencode/manifest.golden.md @@ -35,11 +35,9 @@ Then run `eval-magic ingest --harness opencode`; OpenCode transcript ingest read After all dispatches: -1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json` + the task's own `outputs/final-message.md` + the events file the harness CLI wrote under `outputs/turn-/`, and backfills `timing.json` with transcript-derived tokens/duration; never clobbers an existing record), fill-transcripts, detect-stray-writes, and grade. Optional higher-fidelity timing: write `{ "total_tokens": , "duration_ms": , "source": "completion-event" }` from the task completion event to `timing.json` right after a dispatch — completion-event numbers always win over the backfill. +1. Run `eval-magic ingest --harness ` — a fixed-order chain of record-runs (assembles every task's `run.json` from `dispatch.json`, `conversation.json`, and the harness events under `outputs/turn-/`, and backfills `timing.json`; never clobbers an existing record), detect-stray-writes, and grade. 2. Run `eval-magic dispatch --judges --harness ` to grade the judge tasks ingest listed, then `eval-magic finalize` for the benchmark. -On a harness without persisted transcripts, instead write each task's `run.json` (matching `skills/evaluating-skills/schema/run-record.schema.json`, enforced at runtime by grade/fill-transcripts/detect-stray-writes) and `timing.json` by hand when its subagent returns: carry over `eval_id`, `condition`, `skill_path` (`null` on the without_skill arm), `prompt`, and `files` from the task; populate `final_message` from the subagent's reply; leave `tool_invocations` as `[]`; capture `total_tokens`/`duration_ms` from the task completion event immediately — they may not be persisted anywhere else. - ## Dispatches ### demo-eval / with_skill @@ -75,17 +73,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. The `widget-skill` skill is registered under the identifier `slow-powers-eval-2-with-skill-widget-skill` and is discoverable as an OpenCode skill. If you invoke it, use that identifier. If it does not load as an OpenCode skill, read the skill from `/work/staged/widget-skill/SKILL.md` instead. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task Task-local scratch directory: /work/task/tmp -Framework output directory: /work/outputs Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs/final-message.md. - Do not write outside the task environment. User request: @@ -104,17 +99,14 @@ Treat this as a real user request — do NOT optimize behavior for the eval. No skill is loaded. Respond as you naturally would. -Available fixture files: - - /work/fixtures/input.txt +Codebase overlay files: + - /work/overlays/input.txt Task environment: /work/task-b Task-local scratch directory: /work/task-b/tmp -Framework output directory: /work/outputs-b Instructions: - Work normally on the task: you may edit existing files and create new files inside the task environment. - Keep temporary and scratch files in the task-local scratch directory, not in a host temp directory. -- Use the framework output directory only for framework artifacts. -- After completing the task, write your final user-facing response to /work/outputs-b/final-message.md. - Do not write outside the task environment. User request: diff --git a/tests/run/byoh.rs b/tests/run/byoh.rs index f456fa5..b8d5115 100644 --- a/tests/run/byoh.rs +++ b/tests/run/byoh.rs @@ -13,11 +13,19 @@ use std::path::{Path, PathBuf}; mod extract; -/// A minimal BYOH descriptor: label + exec template, nothing else. +/// A runner-ready BYOH descriptor using the named Codex transcript capability. const COOL_DESCRIPTOR: &str = r#"label = "cool-custom-harness" +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "cool-events.jsonl" +parser = "codex-items" + [dispatch] -exec_template = "cool-cli run --cd {model_arg} > /final-message.md" +exec_template = "cool-cli run --cd {model_arg} > /cool-events.jsonl" "#; /// Write `/.eval-magic/harnesses/cool.toml`. @@ -45,6 +53,40 @@ fn dispatch_tasks(cwd: &Path) -> Vec { .clone() } +fn write_completed_task(cwd: &Path, task: &Value, final_text: &str) { + let outputs = resolve(cwd, task["outputs_dir"].as_str().unwrap()); + let turn = outputs.join("turn-1"); + fs::create_dir_all(&turn).unwrap(); + fs::write( + turn.join("cool-events.jsonl"), + format!( + "{{\"type\":\"item.completed\",\"item\":{{\"id\":\"item_1\",\"type\":\"agent_message\",\"text\":{}}}}}\n", + serde_json::to_string(final_text).unwrap() + ), + ) + .unwrap(); + write_completion(cwd, task); +} + +fn write_completion(cwd: &Path, task: &Value) { + let conversation_path = resolve(cwd, task["conversation_path"].as_str().unwrap()); + fs::write( + conversation_path, + serde_json::to_string_pretty(&serde_json::json!({ + "status": "completed", + "delivered_followups": 0, + "events": [{ + "type": "user_message", + "ordinal": 0, + "round": 1, + "text": task["user_prompt"] + }] + })) + .unwrap(), + ) + .unwrap(); +} + /// Criterion: a descriptor file alone produces a complete llm_judge-graded /// run with the stray-writes audit — warnings name their fallbacks, and the /// exec recipe lands in RUNBOOK.md and dispatch-manifest.md. @@ -54,7 +96,7 @@ fn descriptor_alone_carries_a_complete_run() { let (skill_dir, cwd) = setup(tmp.path(), DEFAULT_EVALS); write_project_descriptor(&cwd, COOL_DESCRIPTOR); - // Build the run: every undeclared enhancement warns naming its fallback. + // Build the run: optional undeclared enhancements warn naming their fallback. skill_eval() .current_dir(&cwd) .args(["run", "--skill-dir"]) @@ -74,10 +116,6 @@ fn descriptor_alone_carries_a_complete_run() { .stderr( contains("declares no skills_dir") .and(contains("--no-stage")) - .and(contains("declares no transcript parser")) - .and(contains("tokens/duration")) - .and(contains("unverifiable").not()) - .and(contains("final-message.md")) .and(contains("declares no model flag")) .and(contains("provenance")), ); @@ -98,14 +136,12 @@ fn descriptor_alone_carries_a_complete_run() { "no-stage run stages nothing" ); - // Simulate the dispatches: recover each final message by hand. + // Simulate runner-owned completion metadata and per-round transcripts. for task in &tasks { - let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "I reviewed the MR.\n").unwrap(); + write_completed_task(&cwd, task, "I reviewed the MR."); } - // Ingest: record-runs from final messages, the stray-writes audit, and + // Ingest: record-runs from transcripts, the stray-writes audit, and // grade's llm_judge hand-off all run without any harness code. skill_eval() .current_dir(&cwd) @@ -120,8 +156,7 @@ fn descriptor_alone_carries_a_complete_run() { "1", ]) .assert() - .success() - .stderr(contains("no transcript parser")); + .success(); assert!( iteration_dir(&cwd).join("stray-writes.json").exists(), @@ -138,16 +173,19 @@ fn descriptor_alone_carries_a_complete_run() { } } -/// A descriptor without an exec_template warns at prep time, because the runner -/// will have nothing to spawn: `eval-magic dispatch` fails outright for such a -/// harness, so the gap is worth naming before the workspace is built. (The -/// built-in-harness half of this pin — wired harnesses stay quiet — lives in -/// src/cli/run/util.rs.) +/// A descriptor without an exec_template is rejected before a workspace is +/// built because the runner has no command to spawn. #[test] -fn dispatchless_descriptor_warns_that_dispatch_has_nothing_to_run() { +fn dispatchless_descriptor_is_rejected_before_build() { let tmp = tempfile::TempDir::new().unwrap(); let (skill_dir, cwd) = setup(tmp.path(), DEFAULT_EVALS); - write_project_descriptor(&cwd, "label = \"cool-custom-harness\"\n"); + write_project_descriptor( + &cwd, + &COOL_DESCRIPTOR.replace( + "\n[dispatch]\nexec_template = \"cool-cli run --cd {model_arg} > /cool-events.jsonl\"\n", + "\n", + ), + ); skill_eval() .current_dir(&cwd) @@ -162,12 +200,14 @@ fn dispatchless_descriptor_warns_that_dispatch_has_nothing_to_run() { "cool-custom-harness", ]) .assert() - .success() + .failure() .stderr( contains("declares no dispatch exec template") - .and(contains("eval-magic dispatch")) + .and(contains("runner-ready")) .and(contains("eval-magic docs byoh")), ); + + assert!(!iteration_dir(&cwd).join("dispatch.json").exists()); } /// `--guard` with a harness that exists only in user-supplied descriptors is @@ -238,17 +278,24 @@ fn auto_guard_stays_off_without_error_on_user_only_harness() { ); } -/// The transcript_check clause of the no-transcript-parser warning is scoped -/// to eval configs that actually use the assertion type. +/// A transcript parser is a runner-readiness requirement regardless of which +/// assertion types the eval declares. #[test] -fn transcript_check_warning_fires_only_when_evals_use_it() { +fn transcriptless_descriptor_is_rejected_before_build() { let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review this MR", "expected_output": "a review", "assertions": [ { "id": "a1", "type": "transcript_check", "check": "ran tests", "pattern": "cargo test" } ] } ] }"#; let tmp = tempfile::TempDir::new().unwrap(); let (skill_dir, cwd) = setup(tmp.path(), evals); - write_project_descriptor(&cwd, COOL_DESCRIPTOR); + write_project_descriptor( + &cwd, + r#"label = "cool-custom-harness" + +[dispatch] +exec_template = "cool-cli run --cd " +"#, + ); skill_eval() .current_dir(&cwd) @@ -263,13 +310,14 @@ fn transcript_check_warning_fires_only_when_evals_use_it() { "cool-custom-harness", ]) .assert() - .success() + .failure() .stderr( contains("declares no transcript parser") - .and(contains("unverifiable")) - .and(contains("llm_judge")) - .and(contains("final-message.md")), + .and(contains("runner-ready")) + .and(contains("eval-magic docs byoh")), ); + + assert!(!iteration_dir(&cwd).join("dispatch.json").exists()); } /// A `--harness-file` descriptor becomes the invocation's default harness @@ -340,10 +388,10 @@ exec_template = "cool-cli run --cd > > = fs::read_dir(iteration.join(".codebase")).unwrap().collect(); - assert_eq!( - cached.len(), - 1, - "both arms of the comparison share one cached materialization" - ); - - for condition in ["old_skill", "new_skill"] { - let env = iteration.join(format!("env-g1-{condition}")); - assert_eq!( - fs::read_to_string(env.join("src/main.rs")).unwrap(), - "fn main() {} -", - "{condition}: the codebase's files must be present" - ); - assert!( - git(&env, &["rev-list", "--count", "HEAD"]) - .parse::() - .unwrap() - >= 2, - "{condition}: the history must survive provisioning" - ); - assert_eq!( - git(&env, &["remote"]), - "", - "{condition}: no env may retain a remote" - ); - assert_eq!( - git(&env, &["rev-parse", "refs/eval-magic/baseline"]), - git(&env, &["rev-parse", "HEAD"]), - "{condition}: the baseline still names the start state" - ); - } -} diff --git a/tests/run/codebase/revision.rs b/tests/run/codebase/revision.rs new file mode 100644 index 0000000..b202152 --- /dev/null +++ b/tests/run/codebase/revision.rs @@ -0,0 +1,96 @@ +//! Revision-mode provisioning and promotion for codebase-backed iterations. + +use super::*; + +#[test] +fn revision_mode_provisions_both_arms_from_the_cached_codebase() { + let tmp = tempfile::TempDir::new().unwrap(); + let origin = codebase_repo(tmp.path(), "origin", "main"); + let source = format!(r#"{{ "url": "{}", "ref": "main" }}"#, wire_path(&origin)); + let (skill_dir, cwd) = setup(tmp.path(), &evals_with_codebase(&source)); + fs::write(skill_dir.join("mr-review/evals/TASK.md"), "task\n").unwrap(); + + skill_eval() + .current_dir(&cwd) + .args(["snapshot", "--skill-dir"]) + .arg(&skill_dir) + .args(["--skill", "mr-review", "--label", "baseline"]) + .assert() + .success(); + + skill_eval() + .current_dir(&cwd) + .args(["run", "--skill-dir"]) + .arg(&skill_dir) + .args([ + "--skill", + "mr-review", + "--mode", + "revision", + "--judge-samples", + "3", + "--dry-run", + ]) + .assert() + .success(); + + let iteration = iteration_dir(&cwd); + let conditions = read_json(&iteration.join("conditions.json")); + assert_eq!(conditions["mode"], "revision"); + assert_eq!(conditions["judge_samples"], 3); + assert_eq!(conditions["codebases"][0]["source"], wire_path(&origin)); + let revision = conditions["codebases"][0]["revision"] + .as_str() + .expect("revision mode records the resolved codebase SHA") + .to_string(); + let cached: Vec<_> = fs::read_dir(iteration.join(".codebase")).unwrap().collect(); + assert_eq!( + cached.len(), + 1, + "both arms of the comparison share one cached materialization" + ); + + for condition in ["old_skill", "new_skill"] { + let env = iteration.join(format!("env-g1-{condition}")); + assert_eq!( + fs::read_to_string(env.join("src/main.rs")).unwrap(), + "fn main() {}\n", + "{condition}: the codebase's files must be present" + ); + assert!( + git(&env, &["rev-list", "--count", "HEAD"]) + .parse::() + .unwrap() + >= 2, + "{condition}: the history must survive provisioning" + ); + assert_eq!( + git(&env, &["remote"]), + "", + "{condition}: no env may retain a remote" + ); + assert_eq!( + git(&env, &["rev-parse", "refs/eval-magic/baseline"]), + git(&env, &["rev-parse", "HEAD"]), + "{condition}: the baseline still names the start state" + ); + } + + // Promotion carries provenance from the revision-mode conditions record + // into the durable baseline report. + fs::write( + iteration.join("benchmark.json"), + r#"{"delta":{"pass_rate":0}}"#, + ) + .unwrap(); + skill_eval() + .current_dir(&cwd) + .args(["promote-baseline", "--skill-dir"]) + .arg(&skill_dir) + .args(["--skill", "mr-review", "--iteration", "1"]) + .assert() + .success(); + let baseline = read_str(&skill_dir.join("mr-review/evals/baseline/BASELINE.md")); + assert!(baseline.contains(&wire_path(&origin)), "{baseline}"); + assert!(baseline.contains(&revision[..7]), "{baseline}"); +} diff --git a/tests/run/codebase_compat.rs b/tests/run/codebase_compat.rs deleted file mode 100644 index c1c8d32..0000000 --- a/tests/run/codebase_compat.rs +++ /dev/null @@ -1,28 +0,0 @@ -//! Compatibility behavior for evals that do not declare a sourced codebase. - -use crate::codebase_support::git; -use crate::helpers::*; - -#[test] -fn a_fixture_only_eval_still_gets_the_repository_it_always_had() { - let tmp = tempfile::TempDir::new().unwrap(); - let (skill_dir, cwd) = setup(tmp.path(), DEFAULT_EVALS); - - skill_eval() - .current_dir(&cwd) - .args(["run", "--skill-dir"]) - .arg(&skill_dir) - .args(["--skill", "mr-review", "--mode", "new-skill", "--dry-run"]) - .assert() - .success(); - - let env = cli_env_dir(&cwd, "g1", "with_skill"); - assert_eq!(git(&env, &["symbolic-ref", "--short", "HEAD"]), "work"); - assert_eq!(git(&env, &["rev-list", "--count", "HEAD"]), "1"); - assert_eq!(git(&env, &["remote"]), ""); - assert_eq!(git(&env, &["status", "--porcelain"]), ""); - assert_eq!( - git(&env, &["rev-parse", "refs/eval-magic/baseline"]), - git(&env, &["rev-parse", "HEAD"]) - ); -} diff --git a/tests/run/codebase_harness_config.rs b/tests/run/codebase_harness_config.rs index 5ab45f3..007c937 100644 --- a/tests/run/codebase_harness_config.rs +++ b/tests/run/codebase_harness_config.rs @@ -376,7 +376,19 @@ fn exclusion_is_a_noop_for_a_byoh_harness_without_project_skill_roots() { fs::create_dir_all(&harness_dir).unwrap(); fs::write( harness_dir.join("cool.toml"), - "label = \"cool-custom-harness\"\n", + r#"label = "cool-custom-harness" + +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "cool-events.jsonl" +parser = "codex-items" + +[dispatch] +exec_template = "cool-cli run --cd > /cool-events.jsonl" +"#, ) .unwrap(); diff --git a/tests/run/codex.rs b/tests/run/codex.rs index 10d0665..f6764df 100644 --- a/tests/run/codex.rs +++ b/tests/run/codex.rs @@ -424,8 +424,7 @@ fn codex_warns_when_user_skill_shadows_staged_skill() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "p1", "expected_output": "o", "files": ["a.txt"] }, - { "id": "e2", "prompt": "p2", "expected_output": "o", "files": ["b.txt"], - "isolation": "isolated" } + { "id": "e2", "prompt": "p2", "expected_output": "o", "files": ["b.txt"] } ] }"#; let (skill_dir, cwd) = setup(tmp.path(), evals); fs::write(skill_dir.join("mr-review/evals/a.txt"), "a").unwrap(); diff --git a/tests/run/codex_permission_denials.rs b/tests/run/codex_permission_denials.rs index 956a22b..68f4470 100644 --- a/tests/run/codex_permission_denials.rs +++ b/tests/run/codex_permission_denials.rs @@ -33,10 +33,10 @@ fn codex_record_runs_reports_permission_denials_from_stderr() { assert_eq!(tasks.len(), 2, "{tasks:?}"); for task in &tasks { let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "Reviewed.\n").unwrap(); + let turn = outputs.join("turn-1"); + fs::create_dir_all(&turn).unwrap(); fs::write( - outputs.join("codex-events.jsonl"), + turn.join("codex-events.jsonl"), concat!( r#"{"type":"thread.started","thread_id":"thread_1","timestamp":"2026-07-30T06:00:00Z"}"#, "\n", @@ -49,7 +49,7 @@ fn codex_record_runs_reports_permission_denials_from_stderr() { .unwrap(); if task["condition"] == "with_skill" { fs::write( - outputs.join("codex-stderr.log"), + turn.join("codex-stderr.log"), concat!( "2026-07-30T06:00:01Z ERROR codex_core::tools::router: ", "error=exec_command failed for `/bin/zsh -lc pwd`: CreateProcess { ", @@ -59,6 +59,7 @@ fn codex_record_runs_reports_permission_denials_from_stderr() { ) .unwrap(); } + write_task_completion(&cwd, task); } skill_eval() diff --git a/tests/run/command_check.rs b/tests/run/command_check.rs index a33861f..2ada5c2 100644 --- a/tests/run/command_check.rs +++ b/tests/run/command_check.rs @@ -8,8 +8,16 @@ use std::path::{Path, PathBuf}; const COOL_DESCRIPTOR: &str = r#"label = "cool-custom-harness" +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "cool-events.jsonl" +parser = "codex-items" + [dispatch] -exec_template = "cool-cli run --cd {model_arg} > /final-message.md" +exec_template = "cool-cli run --cd {model_arg} > /cool-events.jsonl" "#; fn write_project_descriptor(cwd: &Path) { @@ -34,6 +42,15 @@ fn dispatch_tasks(cwd: &Path) -> Vec { .clone() } +fn write_cool_task_result(cwd: &Path, task: &Value) { + write_task_transcript( + cwd, + task, + "cool-events.jsonl", + "{\"type\":\"item.completed\",\"item\":{\"id\":\"m1\",\"type\":\"agent_message\",\"text\":\"done\"}}\n", + ); +} + fn setup_exists_command() -> String { fixture(&["--require-file", "holdout/secret.txt"]) } @@ -74,7 +91,7 @@ fn auto_isolates_and_multi_run_tasks_get_distinct_hidden_envs() { assert_eq!(groups[0]["evals"], json!(["ordinary-1"])); assert_eq!(groups[1]["evals"], json!(["held-out"])); assert_eq!(groups[2]["evals"], json!(["ordinary-2"])); - assert_eq!(groups[1]["rationale"], "metric: diff_scope"); + assert_eq!(groups[1]["rationale"], "private codebase"); let command_tasks: Vec<_> = dispatch["tasks"] .as_array() @@ -102,7 +119,7 @@ fn auto_isolates_and_multi_run_tasks_get_distinct_hidden_envs() { !root.join("holdout/secret.txt").exists(), "held-out setup is absent before ingest" ); - assert_eq!(task["fixtures"], json!([])); + assert_eq!(task["files"], json!([])); let prompt = read_str(Path::new(task["dispatch_prompt_path"].as_str().unwrap())); assert!(!prompt.contains("holdout/secret.txt")); } @@ -169,7 +186,7 @@ fn root_git_setup_path_is_rejected_before_staging() { } #[test] -fn descriptor_without_transcript_support_runs_command_checks() { +fn runner_ready_descriptor_runs_command_checks() { let tmp = tempfile::TempDir::new().unwrap(); let evals = json!({ "skill_name": "mr-review", @@ -213,9 +230,7 @@ fn descriptor_without_transcript_support_runs_command_checks() { for task in &tasks { let eval_root = resolve(&cwd, task["eval_root"].as_str().unwrap()); assert!(!eval_root.join("holdout/secret.txt").exists()); - let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "done").unwrap(); + write_cool_task_result(&cwd, task); } skill_eval() @@ -231,8 +246,7 @@ fn descriptor_without_transcript_support_runs_command_checks() { "1", ]) .assert() - .success() - .stderr(contains("no transcript parser")); + .success(); for condition in ["with_skill", "without_skill"] { let result = read_json( @@ -310,9 +324,7 @@ fn ingests_without_transcripts_while_guard_is_armed_and_aggregates() { }, ) .unwrap(); - let outputs_dir = Path::new(task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(outputs_dir).unwrap(); - fs::write(outputs_dir.join("final-message.md"), "done").unwrap(); + write_default_task_result(&cwd, task, "done"); } skill_eval() diff --git a/tests/run/command_check/matrix.rs b/tests/run/command_check/matrix.rs index 469c38a..6dc5f10 100644 --- a/tests/run/command_check/matrix.rs +++ b/tests/run/command_check/matrix.rs @@ -55,9 +55,7 @@ fn reports_cells_and_any_failure_fails_finalized_grading() { .assert() .success(); for task in dispatch_tasks(&cwd) { - let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "done").unwrap(); + write_cool_task_result(&cwd, &task); } skill_eval() diff --git a/tests/run/config_model.rs b/tests/run/config_model.rs new file mode 100644 index 0000000..4b78087 --- /dev/null +++ b/tests/run/config_model.rs @@ -0,0 +1,45 @@ +//! Breaking eval-configuration contracts for the single codebase-backed model. + +use crate::helpers::*; +use predicates::str::contains; + +#[test] +fn missing_effective_codebase_fails_before_creating_an_iteration() { + let tmp = tempfile::TempDir::new().unwrap(); + let (skill_dir, cwd) = setup_raw(tmp.path(), DEFAULT_EVALS); + + skill_eval() + .current_dir(&cwd) + .args(["run", "--skill-dir"]) + .arg(&skill_dir) + .args(["--skill", "mr-review", "--mode", "new-skill", "--dry-run"]) + .assert() + .failure() + .stderr(contains( + "eval 'e1': no effective codebase; set top-level 'codebase' or this eval's 'codebase'", + )); + + assert!(!iteration_dir(&cwd).exists()); +} + +#[test] +fn retired_isolation_field_fails_with_migration_guidance() { + let tmp = tempfile::TempDir::new().unwrap(); + let evals = r#"{ "skill_name": "mr-review", "evals": [ + { "id": "e1", "prompt": "review", "expected_output": "a review", "isolation": "isolated" } + ] }"#; + let (skill_dir, cwd) = setup_raw(tmp.path(), evals); + + skill_eval() + .current_dir(&cwd) + .args(["run", "--skill-dir"]) + .arg(&skill_dir) + .args(["--skill", "mr-review", "--mode", "new-skill", "--dry-run"]) + .assert() + .failure() + .stderr(contains( + "field 'isolation' is no longer supported or needed; every eval run already uses a private environment", + )); + + assert!(!iteration_dir(&cwd).exists()); +} diff --git a/tests/run/conversation.rs b/tests/run/conversation.rs index fdf87fe..4e06ab9 100644 --- a/tests/run/conversation.rs +++ b/tests/run/conversation.rs @@ -166,16 +166,22 @@ fn the_driver_runs_a_task_that_declares_no_scripted_turns() { let conversation = read_json(Path::new(task["conversation_path"].as_str().unwrap())); assert_eq!(conversation["status"], "completed"); assert_eq!(conversation["delivered_followups"], 0); + assert_eq!( + conversation["events"], + serde_json::json!([{ + "type": "user_message", + "ordinal": 0, + "round": 1, + "text": "Fix the date." + }]) + ); let outputs = Path::new(task["outputs_dir"].as_str().unwrap()); assert!( outputs.join("turn-1").join("codex-events.jsonl").is_file(), "a one-shot transcript belongs under turn-1 like every other round" ); - assert_eq!( - fs::read_to_string(outputs.join("final-message.md")).unwrap(), - "Updated the date handling.\n" - ); + assert!(!outputs.join("final-message.md").exists()); } const ONE_SHOT_EVALS: &str = r#"{ @@ -249,8 +255,8 @@ fn stub_exec_template(cwd: &Path, template: &str) { .unwrap(); } -/// A one-shot harness stub: emits a session id, one agent message, and a usage -/// event, which is the minimum a transcript needs to parse. Written as a POSIX +/// A one-shot harness stub: emits one agent message and a usage event without a +/// resumable session id. Written as a POSIX /// script and invoked through `sh` for the same reason the scripted stub below /// is — that is the shape of a real `exec_template`, and it needs no executable /// bit on any host. @@ -261,8 +267,7 @@ fn one_shot_stub(dir: &Path, message: &str) -> String { r#"#!/bin/sh outputs=$1 message=$2 -printf '%s\n' '{"type":"thread.started","thread_id":"session-1"}' > "$outputs/codex-events.jsonl" -printf '%s\n' "{\"type\":\"item.completed\",\"item\":{\"id\":\"m1\",\"type\":\"agent_message\",\"text\":\"$message\"}}" >> "$outputs/codex-events.jsonl" +printf '%s\n' "{\"type\":\"item.completed\",\"item\":{\"id\":\"m1\",\"type\":\"agent_message\",\"text\":\"$message\"}}" > "$outputs/codex-events.jsonl" printf '%s\n' '{"type":"turn.completed","usage":{"input_tokens":2,"output_tokens":3}}' >> "$outputs/codex-events.jsonl" "#, ) @@ -375,20 +380,33 @@ printf '%s\n' '{"type":"turn.completed","usage":{"input_tokens":2,"output_tokens assert_eq!(conversation["delivered_followups"], 2); assert_eq!(conversation["stop_reason"], Value::Null); assert_eq!( - conversation["events"] - .as_array() - .unwrap() - .iter() - .filter(|event| event["type"] == "user_message") - .count(), - 3 + conversation["events"], + serde_json::json!([ + { + "type": "user_message", + "ordinal": 0, + "round": 1, + "text": "Fix the date." + }, + { + "type": "user_message", + "ordinal": 1, + "round": 2, + "text": "Affected users are in US timezones." + }, + { + "type": "user_message", + "ordinal": 2, + "round": 3, + "text": "It is a date-only field." + } + ]) ); - assert_eq!( - fs::read_to_string( - Path::new(task["outputs_dir"].as_str().unwrap()).join("final-message.md") - ) - .unwrap(), - "Updated the date handling.\n" + assert!( + !Path::new(task["outputs_dir"].as_str().unwrap()) + .join("final-message.md") + .exists(), + "the final assistant response comes from the last round transcript" ); dispatch_one(&skill_dir, &cwd, "codex", 1, false) @@ -446,11 +464,7 @@ fn multi_turn_eval_rejects_a_harness_without_native_resume_support() { fs::create_dir_all(&descriptor_dir).unwrap(); fs::write( descriptor_dir.join("cool.toml"), - r#"label = "cool-custom-harness" - -[dispatch] -exec_template = "cool-cli run --cd {model_arg} > /final-message.md" -"#, + RUNNER_READY_NO_RESUME_DESCRIPTOR, ) .unwrap(); diff --git a/tests/run/conversation/responder.rs b/tests/run/conversation/responder.rs index a802c82..44de76f 100644 --- a/tests/run/conversation/responder.rs +++ b/tests/run/conversation/responder.rs @@ -279,11 +279,7 @@ fn a_responder_eval_is_rejected_on_a_harness_without_native_resume() { fs::create_dir_all(&descriptor_dir).unwrap(); fs::write( descriptor_dir.join("cool.toml"), - r#"label = "cool-custom-harness" - -[dispatch] -exec_template = "cool-cli run --cd {model_arg} > /final-message.md" -"#, + RUNNER_READY_NO_RESUME_DESCRIPTOR, ) .unwrap(); diff --git a/tests/run/diff_scope.rs b/tests/run/diff_scope.rs index f66d37e..301460d 100644 --- a/tests/run/diff_scope.rs +++ b/tests/run/diff_scope.rs @@ -26,7 +26,7 @@ fn ingest_writes_diff_scope_for_every_run_without_an_assertion() { for task in dispatch["tasks"].as_array().unwrap() { let eval_root = Path::new(task["eval_root"].as_str().unwrap()); let outputs_dir = Path::new(task["outputs_dir"].as_str().unwrap()); - fs::write(outputs_dir.join("final-message.md"), "done").unwrap(); + write_default_task_result(&cwd, task, "done"); if task["condition"] == "with_skill" { fs::write(eval_root.join("source.txt"), "new\n").unwrap(); fs::write(eval_root.join("notes.txt"), "one\n").unwrap(); @@ -161,9 +161,8 @@ fn diff_scope_is_captured_before_command_check_setup_and_reused() { let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); for task in dispatch["tasks"].as_array().unwrap() { let eval_root = Path::new(task["eval_root"].as_str().unwrap()); - let outputs_dir = Path::new(task["outputs_dir"].as_str().unwrap()); fs::write(eval_root.join("source.txt"), "new\n").unwrap(); - fs::write(outputs_dir.join("final-message.md"), "done").unwrap(); + write_default_task_result(&cwd, task, "done"); } skill_eval() @@ -240,8 +239,7 @@ fn finalize_grades_diff_scope_thresholds_from_the_persisted_measurement() { let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); for task in dispatch["tasks"].as_array().unwrap() { let eval_root = Path::new(task["eval_root"].as_str().unwrap()); - let outputs_dir = Path::new(task["outputs_dir"].as_str().unwrap()); - fs::write(outputs_dir.join("final-message.md"), "done").unwrap(); + write_default_task_result(&cwd, task, "done"); if task["condition"] == "with_skill" { fs::write(eval_root.join("source.txt"), "new\n").unwrap(); } @@ -364,11 +362,7 @@ fn benchmark_diff_scope_is_ordered_by_eval_id_then_run_index() { let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); for task in dispatch["tasks"].as_array().unwrap() { - fs::write( - Path::new(task["outputs_dir"].as_str().unwrap()).join("final-message.md"), - "done", - ) - .unwrap(); + write_default_task_result(&cwd, task, "done"); } skill_eval() @@ -453,8 +447,7 @@ fn revision_mode_measures_and_captures_the_diff_for_both_arms() { let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); for task in dispatch["tasks"].as_array().unwrap() { let eval_root = Path::new(task["eval_root"].as_str().unwrap()); - let outputs_dir = Path::new(task["outputs_dir"].as_str().unwrap()); - fs::write(outputs_dir.join("final-message.md"), "done").unwrap(); + write_default_task_result(&cwd, task, "done"); fs::write(eval_root.join("source.txt"), "new\n").unwrap(); } diff --git a/tests/run/env_layout.rs b/tests/run/env_layout.rs index 410fda9..9a3f421 100644 --- a/tests/run/env_layout.rs +++ b/tests/run/env_layout.rs @@ -1,7 +1,7 @@ //! Isolated-run env builder: staging redirects into the per-`(group, condition)` -//! `env--/` dirs, fixtures are copied into each like a real repo, -//! and `RUNBOOK.md` lives above them in `iteration-N/`. eval-magic meta stays above -//! the envs in `iteration-N/`. +//! `env--/` dirs, overlays are applied to each codebase, and +//! `RUNBOOK.md` lives above them in `iteration-N/`. eval-magic meta stays above the +//! envs in `iteration-N/`. use crate::helpers::*; use serde_json::json; @@ -60,13 +60,13 @@ fn env_dir_created_even_with_no_stage() { .success(); // Even with staging disabled, each per-(group, condition) env must exist for - // fixtures + the per-env guard. + // the codebase + the per-env guard. assert!(cli_env_dir(&cwd, "g1", "with_skill").is_dir()); assert!(cli_env_dir(&cwd, "g1", "without_skill").is_dir()); } #[test] -fn fixtures_copied_into_env_like_a_real_repo() { +fn overlay_files_are_applied_to_each_private_codebase() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review", "expected_output": "a review", @@ -87,7 +87,7 @@ fn fixtures_copied_into_env_like_a_real_repo() { .success(); // Structure preserved under each per-condition env, not flattened into an - // inputs/ bucket. Fixtures are copied into every relevant env (per its group). + // inputs/ bucket. Overlay files are copied into every relevant environment. for cond in ["with_skill", "without_skill"] { let env = cli_env_dir(&cwd, "g1", cond); assert_eq!(read_str(&env.join("src/main.rs")), "fn main() {}"); @@ -95,7 +95,7 @@ fn fixtures_copied_into_env_like_a_real_repo() { assert!(!env.join("inputs").exists()); } - // The dispatch prompt lists fixtures env-relative — the agent's cwd is env. + // The dispatch prompt and task record list overlays relative to the codebase. let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); let task = dispatch["tasks"] .as_array() @@ -104,20 +104,22 @@ fn fixtures_copied_into_env_like_a_real_repo() { .find(|t| t["condition"] == "with_skill") .unwrap(); let prompt = read_str(Path::new(task["dispatch_prompt_path"].as_str().unwrap())); + assert_eq!(task["files"], json!(["src/main.rs", "data/x.json"])); + assert!(task.get("fixtures").is_none()); assert!(prompt.contains("- src/main.rs")); assert!(prompt.contains("- data/x.json")); assert!(!prompt.contains("inputs/")); } #[test] -fn files_root_mounts_nested_fixture_sources_at_task_root() { +fn files_root_resolves_nested_overlay_sources_at_codebase_root() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review", "expected_output": "a review", - "files_root": "fixtures/todo-app", + "files_root": "overlays/todo-app", "files": ["package.json", "src/hooks/useDebounce.ts"] } ] }"#; let (skill_dir, cwd) = setup(tmp.path(), evals); - let source_root = skill_dir.join("mr-review/evals/fixtures/todo-app"); + let source_root = skill_dir.join("mr-review/evals/overlays/todo-app"); fs::create_dir_all(source_root.join("src/hooks")).unwrap(); fs::write(source_root.join("package.json"), "{}").unwrap(); fs::write( @@ -141,7 +143,7 @@ fn files_root_mounts_nested_fixture_sources_at_task_root() { read_str(&env.join("src/hooks/useDebounce.ts")), "export function useDebounce() {}" ); - assert!(!env.join("fixtures").exists()); + assert!(!env.join("overlays").exists()); } let dispatch = read_json(&iteration_dir(&cwd).join("dispatch.json")); @@ -152,13 +154,13 @@ fn files_root_mounts_nested_fixture_sources_at_task_root() { .find(|task| task["condition"] == "with_skill") .unwrap(); assert_eq!( - task["fixtures"], + task["files"], json!(["package.json", "src/hooks/useDebounce.ts"]) ); let prompt = read_str(Path::new(task["dispatch_prompt_path"].as_str().unwrap())); assert!(prompt.contains("- package.json")); assert!(prompt.contains("- src/hooks/useDebounce.ts")); - assert!(!prompt.contains("fixtures/todo-app")); + assert!(!prompt.contains("overlays/todo-app")); } #[test] @@ -291,13 +293,13 @@ fn dispatch_outputs_live_under_env() { } #[test] -fn fixture_is_copied_into_each_private_run_environment() { +fn overlay_is_applied_to_each_private_run_environment() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review", "expected_output": "a review", - "files": ["fixture.txt"] } ] }"#; + "files": ["TASK.md"] } ] }"#; let (skill_dir, cwd) = setup(tmp.path(), evals); - fs::write(skill_dir.join("mr-review/evals/fixture.txt"), "DATA").unwrap(); + fs::write(skill_dir.join("mr-review/evals/TASK.md"), "DATA").unwrap(); skill_eval() .current_dir(&cwd) @@ -320,16 +322,13 @@ fn fixture_is_copied_into_each_private_run_environment() { assert_eq!(tasks.len(), 4, "1 eval × 2 conditions × 2 runs"); for task in tasks { let eval_root = Path::new(task["eval_root"].as_str().unwrap()); - assert_eq!(read_str(&eval_root.join("fixture.txt")), "DATA"); - assert_eq!( - task["fixtures"].as_array().unwrap(), - &vec![json!("fixture.txt")] - ); + assert_eq!(read_str(&eval_root.join("TASK.md")), "DATA"); + assert_eq!(task["files"].as_array().unwrap(), &vec![json!("TASK.md")]); } } #[test] -fn two_evals_sharing_a_fixture_declaration_succeeds() { +fn two_evals_sharing_an_overlay_declaration_succeeds() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "p1", "expected_output": "o", "files": ["shared.txt"] }, @@ -355,7 +354,7 @@ fn two_evals_sharing_a_fixture_declaration_succeeds() { .find(|t| t["eval_id"] == id && t["condition"] == "with_skill") .unwrap(); assert_eq!( - task["fixtures"].as_array().unwrap(), + task["files"].as_array().unwrap(), &vec![json!("shared.txt")] ); assert_eq!( diff --git a/tests/run/git_isolation.rs b/tests/run/git_isolation.rs index 8d9b486..5d8b777 100644 --- a/tests/run/git_isolation.rs +++ b/tests/run/git_isolation.rs @@ -157,7 +157,7 @@ fn explicit_iteration_rebuild_resets_task_git_history_branch_and_remotes() { } #[test] -fn nested_git_repository_fixture_is_preserved_inside_the_task_repo() { +fn nested_git_repository_overlay_is_preserved_inside_the_task_repo() { let tmp = tempfile::TempDir::new().unwrap(); let evals = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review", "expected_output": "a review", diff --git a/tests/run/grouping.rs b/tests/run/grouping.rs index 672a2e3..fba5c2f 100644 --- a/tests/run/grouping.rs +++ b/tests/run/grouping.rs @@ -1,18 +1,15 @@ -//! Isolation-group batching during `run`: how the setup phase groups evals into -//! environments and records the plan in `dispatch.json`. Covers the per-(group, -//! condition) env split that closes the condition-isolation gap — emitted for every -//! run now, including the bare default invocation — and the explicit -//! `isolation: isolated` hint that fans a second group out into its own envs. +//! Private eval-group planning during `run`: every eval receives a deterministic +//! group, and every condition and run receives a private codebase environment. use crate::helpers::*; use serde_json::json; use std::fs; -const TWO_EVALS_ONE_ISOLATED: &str = r#"{ "skill_name": "mr-review", "evals": [ +const TWO_EVALS: &str = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "p1", "expected_output": "o", "files": ["a.txt"] }, - { "id": "e2", "prompt": "p2", "expected_output": "o", "files": ["b.txt"], "isolation": "isolated" } ] }"#; + { "id": "e2", "prompt": "p2", "expected_output": "o", "files": ["b.txt"] } ] }"#; -fn write_fixtures(skill_dir: &std::path::Path) { +fn write_overlays(skill_dir: &std::path::Path) { fs::write(skill_dir.join("mr-review/evals/a.txt"), "AAA").unwrap(); fs::write(skill_dir.join("mr-review/evals/b.txt"), "BBB").unwrap(); } @@ -114,10 +111,10 @@ fn cli_single_group_emits_groups_and_splits_env_per_condition() { } #[test] -fn isolated_hint_splits_into_two_groups() { +fn every_eval_gets_a_private_group_and_overlay() { let tmp = tempfile::TempDir::new().unwrap(); - let (skill_dir, cwd) = setup(tmp.path(), TWO_EVALS_ONE_ISOLATED); - write_fixtures(&skill_dir); + let (skill_dir, cwd) = setup(tmp.path(), TWO_EVALS); + write_overlays(&skill_dir); skill_eval() .current_dir(&cwd) .args(["run", "--skill-dir"]) @@ -130,17 +127,10 @@ fn isolated_hint_splits_into_two_groups() { let groups = dispatch["groups"] .as_array() .expect("groups summary present"); - assert_eq!(groups.len(), 2, "the isolated eval forms its own group"); + assert_eq!(groups.len(), 2, "each eval forms its own group"); assert_eq!(groups[0]["evals"], json!(["e1"])); assert_eq!(groups[1]["evals"], json!(["e2"])); - assert!( - groups[1]["rationale"] - .as_str() - .unwrap() - .contains("isolated"), - "second group's rationale names the hint: {}", - groups[1]["rationale"] - ); + assert_eq!(groups[1]["rationale"], "private codebase"); // With two groups, tasks are tagged with their group. let e2_task = dispatch["tasks"] @@ -151,7 +141,7 @@ fn isolated_hint_splits_into_two_groups() { .unwrap(); assert_eq!(e2_task["group"], "g2"); - // Each group gets its own per-condition envs, holding only that group's fixtures — + // Each group gets its own per-condition envs, holding only that eval's overlays — // g1's a.txt never leaks into g2's env and vice versa. assert_eq!( read_str(&cli_env_dir(&cwd, "g1", "with_skill").join("a.txt")), @@ -159,7 +149,7 @@ fn isolated_hint_splits_into_two_groups() { ); assert!( !cli_env_dir(&cwd, "g1", "with_skill").join("b.txt").exists(), - "the isolated group's fixture is not staged into g1's env" + "the second eval's overlay is not staged into g1's env" ); assert_eq!( read_str(&cli_env_dir(&cwd, "g2", "with_skill").join("b.txt")), @@ -167,15 +157,15 @@ fn isolated_hint_splits_into_two_groups() { ); assert!( !cli_env_dir(&cwd, "g2", "with_skill").join("a.txt").exists(), - "g1's fixture is not staged into the isolated group's env" + "g1's overlay is not staged into g2's env" ); } #[test] -fn isolated_hint_splits_into_separate_envs_cli() { +fn private_groups_use_separate_environments_for_codex() { let tmp = tempfile::TempDir::new().unwrap(); - let (skill_dir, cwd) = setup(tmp.path(), TWO_EVALS_ONE_ISOLATED); - write_fixtures(&skill_dir); + let (skill_dir, cwd) = setup(tmp.path(), TWO_EVALS); + write_overlays(&skill_dir); skill_eval() .current_dir(&cwd) .args(["run", "--skill-dir"]) @@ -184,7 +174,7 @@ fn isolated_hint_splits_into_separate_envs_cli() { .assert() .success(); - // Each group gets its own per-condition envs, holding only that group's fixtures. + // Each group gets its own per-condition envs, holding only that eval's overlays. assert_eq!( read_str(&cli_env_dir(&cwd, "g1", "with_skill").join("a.txt")), "AAA" diff --git a/tests/run/helpers.rs b/tests/run/helpers.rs index 14f2bbf..bfbbfa9 100644 --- a/tests/run/helpers.rs +++ b/tests/run/helpers.rs @@ -4,10 +4,23 @@ use std::fs; use std::path::{Path, PathBuf}; use assert_cmd::Command; -use serde_json::Value; +use serde_json::{Value, json}; pub const STAGED_MANIFEST: &str = ".slow-powers-eval-manifest.json"; pub const DEFAULT_EVALS: &str = r#"{ "skill_name": "mr-review", "evals": [ { "id": "e1", "prompt": "review this MR", "expected_output": "a review" } ] }"#; +pub const RUNNER_READY_NO_RESUME_DESCRIPTOR: &str = r#"label = "cool-custom-harness" + +[dispatch] +exec_template = "cool-cli run --cd {model_arg} > /cool-events.jsonl" + +[tools] +write = ["file_change"] +shell = ["command_execution"] + +[transcript] +events_filename = "cool-events.jsonl" +parser = "codex-items" +"#; pub fn skill_eval() -> Command { let mut cmd = Command::cargo_bin("eval-magic").expect("binary `eval-magic` should build"); @@ -25,6 +38,16 @@ pub fn skill_eval() -> Command { /// Build `/skill-dir/mr-review/{SKILL.md,evals/evals.json}` and a `work` /// cwd; returns `(skill_dir, cwd)`. pub fn setup(root: &Path, evals_json: &str) -> (PathBuf, PathBuf) { + setup_inner(root, evals_json, true) +} + +/// Write an authored config byte-for-byte for tests that exercise invalid +/// configuration before any environment can be built. +pub fn setup_raw(root: &Path, evals_json: &str) -> (PathBuf, PathBuf) { + setup_inner(root, evals_json, false) +} + +fn setup_inner(root: &Path, evals_json: &str, add_default_codebase: bool) -> (PathBuf, PathBuf) { let skill_dir = root.join("skill-dir"); let skill_sub = skill_dir.join("mr-review"); fs::create_dir_all(skill_sub.join("evals")).unwrap(); @@ -33,7 +56,19 @@ pub fn setup(root: &Path, evals_json: &str) -> (PathBuf, PathBuf) { "---\nname: mr-review\ndescription: review merge requests\n---\n\nbody\n", ) .unwrap(); - fs::write(skill_sub.join("evals").join("evals.json"), evals_json).unwrap(); + let rendered = if add_default_codebase { + let codebase = root.join("codebase"); + fs::create_dir_all(&codebase).unwrap(); + fs::write(codebase.join("README.md"), "# Test codebase\n").unwrap(); + let mut config: Value = serde_json::from_str(evals_json).unwrap(); + if config.get("codebase").is_none() { + config["codebase"] = json!({ "path": codebase.to_string_lossy() }); + } + serde_json::to_string_pretty(&config).unwrap() + } else { + evals_json.to_string() + }; + fs::write(skill_sub.join("evals").join("evals.json"), rendered).unwrap(); let cwd = root.join("work"); fs::create_dir_all(&cwd).unwrap(); (skill_dir, cwd) @@ -47,8 +82,8 @@ pub fn iteration_dir(cwd: &Path) -> PathBuf { /// A per-`(group, condition)` env dir — the cwd each `claude -p`/`codex exec` /// subprocess runs from: `iteration-N/env--/`. Each holds only -/// that condition's skill (or none, for the control arm) and its group's fixtures. -/// Staging, fixtures, and the guard marker all land under here, below +/// that condition's skill (or none, for the control arm) and its eval's overlays. +/// Staging, overlays, and the guard marker all land under here, below /// `iteration_dir`; `RUNBOOK.md` lives above it in `iteration_dir`. pub fn cli_env_dir(cwd: &Path, group: &str, condition: &str) -> PathBuf { iteration_dir(cwd).join(format!("env-{group}-{condition}")) @@ -116,6 +151,59 @@ pub fn read_json(path: &Path) -> Value { serde_json::from_str(&fs::read_to_string(path).unwrap()).unwrap() } +pub fn resolve_task_path(cwd: &Path, value: &Value) -> PathBuf { + let path = PathBuf::from(value.as_str().expect("dispatch path is a string")); + if path.is_absolute() { + path + } else { + cwd.join(path) + } +} + +/// Write the runner-owned one-shot completion artifact for a dispatch task. +pub fn write_task_completion(cwd: &Path, task: &Value) { + let path = resolve_task_path(cwd, &task["conversation_path"]); + fs::create_dir_all(path.parent().unwrap()).unwrap(); + fs::write( + path, + serde_json::to_string_pretty(&serde_json::json!({ + "status": "completed", + "delivered_followups": 0, + "events": [{ + "type": "user_message", + "ordinal": 0, + "round": 1, + "text": task["user_prompt"] + }] + })) + .unwrap(), + ) + .unwrap(); +} + +/// Write one canonical turn-1 transcript and its matching completion artifact. +pub fn write_task_transcript(cwd: &Path, task: &Value, filename: &str, contents: &str) { + let outputs = resolve_task_path(cwd, &task["outputs_dir"]); + let turn = outputs.join("turn-1"); + fs::create_dir_all(&turn).unwrap(); + fs::write(turn.join(filename), contents).unwrap(); + write_task_completion(cwd, task); +} + +/// Minimal Claude transcript for integration tests whose subject is a later +/// pipeline stage rather than transcript parsing. +pub fn write_default_task_result(cwd: &Path, task: &Value, final_text: &str) { + write_task_transcript( + cwd, + task, + "claude-events.jsonl", + &format!( + "{{\"type\":\"result\",\"subtype\":\"success\",\"is_error\":false,\"result\":{},\"duration_ms\":1,\"usage\":{{\"input_tokens\":1,\"output_tokens\":1}}}}\n", + serde_json::to_string(final_text).unwrap() + ), + ); +} + /// Every path under `root`, directories included. For asserting that /// something is absent from an artifact tree, where a targeted `exists()` check /// would only cover the one place it was expected. diff --git a/tests/run/main.rs b/tests/run/main.rs index 6e17049..862f68b 100644 --- a/tests/run/main.rs +++ b/tests/run/main.rs @@ -15,13 +15,13 @@ mod claude_cli; mod cline; mod cline_permission_denials; mod codebase; -mod codebase_compat; mod codebase_harness_config; mod codebase_support; mod codex; mod codex_guard; mod codex_permission_denials; mod command_check; +mod config_model; mod conversation; mod diff_scope; mod env_layout; diff --git a/tests/run/multi_skill.rs b/tests/run/multi_skill.rs index e78864e..41240d6 100644 --- a/tests/run/multi_skill.rs +++ b/tests/run/multi_skill.rs @@ -112,9 +112,7 @@ fn conditions_and_dispatch_record_the_ordered_treatment_roster() { ); for task in dispatch["tasks"].as_array().unwrap() { - let outputs = Path::new(task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "done\n").unwrap(); + write_default_task_result(&cwd, task, "done"); } skill_eval() .current_dir(&cwd) @@ -174,7 +172,7 @@ fn a_one_member_list_still_uses_list_artifacts_and_indexed_meta_files() { "skill_path": task["skill_path"], "skills": task["skills"], "prompt": task["user_prompt"], - "files": task["fixtures"], + "files": task["files"], "final_message": "done", "tool_invocations": [], "total_tokens": null, @@ -388,7 +386,7 @@ fn deterministic_grading_reports_each_skill_and_suite_invocation_is_any_member() "skill_path": task["skill_path"], "skills": task["skills"], "prompt": task["user_prompt"], - "files": task["fixtures"], + "files": task["files"], "final_message": "done", "tool_invocations": invocations, "total_tokens": null, @@ -479,7 +477,7 @@ fn provenance_names_the_source_and_revision_of_every_treatment_member() { "--quiet", "--no-gpg-sign", "-m", - "fixture", + "test setup", ]) .current_dir(&skill_dir) .status() @@ -573,7 +571,7 @@ fn codex_fallback_emits_one_named_llm_meta_task_per_treatment_member() { "skill_path": task["skill_path"], "skills": task["skills"], "prompt": task["user_prompt"], - "files": task["fixtures"], + "files": task["files"], "final_message": "done", "tool_invocations": [], "total_tokens": null, @@ -626,7 +624,7 @@ fn live_source_detection_checks_every_treatment_member() { "skill_path": task["skill_path"], "skills": task["skills"], "prompt": task["user_prompt"], - "files": task["fixtures"], + "files": task["files"], "final_message": "done", "tool_invocations": [{ "name": "Read", diff --git a/tests/run/opencode.rs b/tests/run/opencode.rs index 327169f..0e957d5 100644 --- a/tests/run/opencode.rs +++ b/tests/run/opencode.rs @@ -419,17 +419,12 @@ fn opencode_ingest_parses_events_and_code_checks_the_skill_invocation() { // Simulate `opencode run --format json` dispatches: a bash call, the staged // skill loaded through the native `skill` tool, two text parts, and a - // step_finish token report. No final-message.md — the transcript's last - // text is the final-message fallback. + // step_finish token report. The transcript's last text is authoritative. for task in dispatch_tasks(&cwd) { - let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); let slug_line = format!( r#"{{"type":"tool_use","timestamp":3000,"sessionID":"ses_1","part":{{"id":"p3","type":"tool","tool":"skill","state":{{"status":"completed","input":{{"name":"{OPENCODE_SLUG}"}},"output":"","title":"skill","metadata":{{}},"time":{{"start":2900,"end":3000}}}}}}}}"# ); - fs::write( - outputs.join("opencode-events.jsonl"), - [ + let events = [ r#"{"type":"step_start","timestamp":1000,"sessionID":"ses_1","part":{"id":"p1","type":"step-start"}}"#.to_string(), r#"{"type":"tool_use","timestamp":2000,"sessionID":"ses_1","part":{"id":"p2","type":"tool","tool":"bash","state":{"status":"completed","input":{"command":"ls"},"output":"ok","title":"ls","metadata":{},"time":{"start":1900,"end":2000}}}}"#.to_string(), slug_line, @@ -438,9 +433,8 @@ fn opencode_ingest_parses_events_and_code_checks_the_skill_invocation() { r#"{"type":"step_finish","timestamp":6000,"sessionID":"ses_1","part":{"id":"p6","type":"step-finish","reason":"stop","cost":0.002,"tokens":{"input":100,"output":20,"reasoning":5,"cache":{"read":75,"write":0}}}}"#.to_string(), ] .join("\n") - + "\n", - ) - .unwrap(); + + "\n"; + write_task_transcript(&cwd, &task, "opencode-events.jsonl", &events); } skill_eval() diff --git a/tests/run/opencode_permission_denials.rs b/tests/run/opencode_permission_denials.rs index 8e749f6..4fdbecd 100644 --- a/tests/run/opencode_permission_denials.rs +++ b/tests/run/opencode_permission_denials.rs @@ -78,13 +78,14 @@ fn opencode_record_runs_reports_permission_denials_from_the_event_stream() { assert_eq!(tasks.len(), 2, "{tasks:?}"); for task in &tasks { let outputs = resolve(&cwd, task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); - fs::write(outputs.join("final-message.md"), "Reviewed.\n").unwrap(); + let turn = outputs.join("turn-1"); + fs::create_dir_all(&turn).unwrap(); if task["condition"] == "with_skill" { - write_opencode_events_with_denial(&outputs); + write_opencode_events_with_denial(&turn); } else { - write_opencode_events_without_denials(&outputs, "Reviewed."); + write_opencode_events_without_denials(&turn, "Reviewed."); } + write_task_completion(&cwd, task); } skill_eval() diff --git a/tests/run/shadow_runtime_id.rs b/tests/run/shadow_runtime_id.rs index bb46486..5bf9c3e 100644 --- a/tests/run/shadow_runtime_id.rs +++ b/tests/run/shadow_runtime_id.rs @@ -2,7 +2,7 @@ use crate::helpers::*; use std::fs; -use std::path::{Path, PathBuf}; +use std::path::Path; fn write_live_skill(path: &Path) { fs::create_dir_all(path).unwrap(); @@ -99,8 +99,6 @@ fn claude_refutes_a_live_skill_when_only_the_staged_slug_is_reported() { let iteration = iteration_dir(&cwd); let dispatch = read_json(&iteration.join("dispatch.json")); for task in dispatch["tasks"].as_array().unwrap() { - let outputs = PathBuf::from(task["outputs_dir"].as_str().unwrap()); - fs::create_dir_all(&outputs).unwrap(); let skills = task["staged_skill_slug"] .as_str() .into_iter() @@ -127,7 +125,7 @@ fn claude_refutes_a_live_skill_when_only_the_staged_slug_is_reported() { .map(|event| event.to_string()) .collect::>() .join("\n"); - fs::write(outputs.join("claude-events.jsonl"), format!("{events}\n")).unwrap(); + write_task_transcript(&cwd, task, "claude-events.jsonl", &format!("{events}\n")); } skill_eval() diff --git a/tests/run/skill_source.rs b/tests/run/skill_source.rs index 3dc8667..bf3d23c 100644 --- a/tests/run/skill_source.rs +++ b/tests/run/skill_source.rs @@ -73,7 +73,7 @@ fn the_copy_carries_an_uncommitted_edit() { } /// `evals/` is the eval author's material, not the agent's. It rides into the -/// copy because fixtures are read from there, and must still be filtered out of +/// copy because overlay sources are read from there, and must still be filtered out of /// what the agent can discover. #[test] fn the_copy_keeps_evals_while_the_staged_skill_still_excludes_them() { @@ -89,7 +89,7 @@ fn the_copy_keeps_evals_while_the_staged_skill_still_excludes_them() { .join("evals") .join("evals.json") .exists(), - "the copy dropped evals/, which fixtures are read from" + "the copy dropped evals/, which overlay sources are read from" ); let staged = cli_env_dir(&cwd, "g1", "with_skill") .join(".claude/skills") diff --git a/tests/run/staging.rs b/tests/run/staging.rs index 4836a52..888538d 100644 --- a/tests/run/staging.rs +++ b/tests/run/staging.rs @@ -17,7 +17,16 @@ fn setup_direct_skill(root: &Path) -> (PathBuf, PathBuf, PathBuf) { "---\nname: mr-review\ndescription: review merge requests\n---\n\nbody\n", ) .unwrap(); - fs::write(skill_sub.join("evals").join("evals.json"), DEFAULT_EVALS).unwrap(); + let codebase = root.join("codebase"); + fs::create_dir_all(&codebase).unwrap(); + fs::write(codebase.join("README.md"), "# Test codebase\n").unwrap(); + let mut evals: Value = serde_json::from_str(DEFAULT_EVALS).unwrap(); + evals["codebase"] = serde_json::json!({ "path": codebase.to_string_lossy() }); + fs::write( + skill_sub.join("evals").join("evals.json"), + serde_json::to_string_pretty(&evals).unwrap(), + ) + .unwrap(); fs::write( helper.join("SKILL.md"), "---\nname: helper-skill\ndescription: helper\n---\n\nhelper\n", @@ -273,38 +282,6 @@ fn stage_name_threads_verbatim_name_and_registers_cleanup() { assert!(!prompt.contains("slow-powers-eval-")); } -#[test] -fn stage_name_refuses_to_clobber_preexisting_dir() { - let tmp = tempfile::TempDir::new().unwrap(); - let (skill_dir, cwd) = setup(tmp.path(), DEFAULT_EVALS); - // Staging now lands in env-g1-with_skill/.claude/skills, which is fresh per - // iteration. The clobber guard still matters on a re-run (--iteration 1) where the - // env already holds an untracked skill dir; pre-seed that and confirm it is preserved. - let preexisting = cli_env_dir(&cwd, "g1", "with_skill").join(".claude/skills/my-real-skill"); - fs::create_dir_all(&preexisting).unwrap(); - fs::write(preexisting.join("SKILL.md"), "USER OWNED").unwrap(); - - skill_eval() - .current_dir(&cwd) - .args(["run", "--skill-dir"]) - .arg(&skill_dir) - .args([ - "--skill", - "mr-review", - "--mode", - "new-skill", - "--iteration", - "1", - "--stage-name", - "my-real-skill", - "--dry-run", - ]) - .assert() - .failure(); - - assert_eq!(read_str(&preexisting.join("SKILL.md")), "USER OWNED"); -} - #[test] fn dispatch_prompt_lists_only_sut_without_bootstrap() { let tmp = tempfile::TempDir::new().unwrap();