feat(evals): retire fixture-only execution - #293
Merged
Conversation
Remove the isolation and fixture-only execution paths so every eval resolves into a private codebase environment. Treat authored files as overlays and preserve resolved source provenance through run artifacts. Use runner-owned conversations and harness transcripts as the single completion contract, and retire the fill-transcripts, final-message, and capture-prefix fallbacks.
Migrate integration fixtures and dispatch goldens to mandatory codebase-backed environments and runner-owned completion artifacts. Cover revision-mode provenance and promotion, runner-ready harness probing, transcript recovery, overlays, and the explicit rejection of retired configuration.
Document codebases as mandatory, authored files as overlays, and native transcripts as the completion source across shipped guides and harness notes. Update the BYOH baseline to require dispatch and transcript contracts so operators have one runner-ready workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #266.
What and why
This removes the second, fixture-only execution architecture after codebase-backed evals became the
canonical path. Every selected eval resolves an effective project, every dispatch receives a private
environment built from that project, and task-authored files are overlays on the project tree.
The cleanup also removes completion paths that depended on agent-authored framework files. The
runner owns conversation completion state, and each harness's native transcript owns assistant
messages, tool evidence, timing, and the final response.
Implementation
codebaseor an override on every selected eval. Materialize eachsource once, build a private repository for every condition and repetition, apply
filesandfiles_rootas overlays, and mark the starting tree with the baseline ref.isolationfrom the schema and Rust type. Validation gives old configs a targeted errorthat names the field and explains why it is unnecessary.
run.jsonfromdispatch.json, runner-ownedconversation.json, and per-round nativetranscripts. Remove
fill-transcripts,outputs/final-message.md, and flat-transcript fallbacks.validation requires a usable transcript reader and tool vocabulary;
harness lint --probeexecutes and parses the real capture before accepting the descriptor.
conditions.json, everyrun.json,benchmark.json, and promotedBASELINE.mdoutput in both new-skill and revision modes.coverage, and generated dispatch goldens for the single workflow.
Breaking changes and compatibility
Before, an eval could omit
codebase, carry a no-op isolation hint, and treatfilesas the entiretask environment:
{ "skill_name": "example-skill", "evals": [ { "id": "update-config", "prompt": "Update the project configuration.", "expected_output": "The configuration is updated.", "isolation": "shared", "files": ["config.toml"] } ] }After, the project source is explicit and the same
filesdeclaration overlays that source:{ "skill_name": "example-skill", "codebase": { "url": "https://github.com/example/project.git", "ref": "0123456789abcdef0123456789abcdef01234567" }, "evals": [ { "id": "update-config", "prompt": "Update the project configuration.", "expected_output": "The configuration is updated.", "files": ["config.toml"] } ] }Configs that still contain
isolationfail with migration guidance. BYOH descriptors must declare[dispatch],[transcript], and[tools];dispatch.capture_prefixis rejected. The retiredfill-transcriptscommand and agent-authored final-message contract are unavailable. Historical runrecords that predate codebase provenance remain readable.
Safety and provenance boundaries
baseline ref. Overlay files and staged skills are committed into that baseline even when ignored
by the source project.
persisted native transcripts determine whether a task completed and what response was returned.
deterministic auditing, while runner-owned command checks and diff capture remain outside the
harness process.
promotion instead of being re-resolved later.
Review guide
927d758implements the environment, harness, schema, and pipeline contracts.f4a43d4migrates integration coverage and dispatch goldens.a34bed8updates shipped guidance and internal harness documentation.Verification
cargo fmt --checkcargo buildcargo test— 909 library tests, 210 CLI tests, and 199 run tests passedcargo clippy --all-targets -- -D warningsgit diff --check dev...HEADCommit hooks reran formatting and clippy for each commit. The pre-push hook reran the complete test
suite before publishing the branch.