Harness Testing answers one question: does a coding-agent harness make the agent more correct, cheaper, faster, or less annoying on Studio Moser's kind of work? It runs the same frozen tasks against Nothing (the bare provider runtime), Superpowers, and the full Skills-n-Stuff collection, then shows correctness, cost, time, and behavior side by side.
Open the live dashboard → Every trial, per-task result, cost, transcript metric and automated grade, rebuilt from the evidence in this repository on every push.
19 workflow tasks, one attempt per task per harness, 152 trials. Three of the tasks ask for committed work and also score house conventions: branch naming, commit format and file naming. A trial that does the work but breaks a convention does not count as correct.
| Harness | Correct | Conventions followed | Avg cost | Avg time |
|---|---|---|---|---|
| Studio Moser v8 | 19 of 19 | 3 of 3 | $0.15 | 22s |
| Studio Moser v7 + personal instructions | 19 of 19 | 3 of 3 | $0.13 | 22s |
| Studio Moser v7 + personal instructions and skills | 19 of 19 | 3 of 3 | $0.16 | 24s |
| Studio Moser v6 | 18 of 19 | 2 of 3 | $0.13 | 27s |
| Studio Moser v6 + time matters | 17 of 19 | 1 of 3 | $0.12 | 26s |
| Studio Moser v7 | 16 of 19 | 0 of 3 | $0.12 | 23s |
| Studio Moser v5 | 16 of 19 | 0 of 3 | $0.18 | 35s |
| Nothing | 15 of 19 | 0 of 3 | $0.12 | 29s |
Studio Moser v8 is the pick: it is the first that follows every convention using only what the collection ships, because the global instructions and House Style now come with Harness as a profile. The formal verdict still names v7 with the developer's personal instructions, which is equally correct and as fast but measured $0.02 cheaper per task; that version depends on files only one developer has. Every Studio Moser version named its branch and wrote its commit to the house format; the versions before v8 that lacked the personal instructions failed on file naming, a rule the shared baseline did not state. Nothing also committed on main. Three convention tasks at one attempt each is thin evidence for ordering the versions between them. Superpowers was skipped on this model: earlier testing showed it wasn't needed on newer models.
19 workflow tasks, one attempt per task per harness, 38 trials, on Codex 0.162.1.
| Harness | Correct | Conventions followed | Avg cost | Avg time |
|---|---|---|---|---|
| Studio Moser v8 | 18 of 19 | 2 of 3 | $0.09 | 1m 00s |
| Nothing | 15 of 19 | 0 of 3 | $0.06 | 1m 01s |
Studio Moser v8 is the pick. Its one miss is a checker error, not a harness mistake: on the static convention task it branched, committed and named its notes file correctly, but first ran a read-only git symbolic-ref query the convention replay did not recognise, so the replay failed closed. The replay is fixed for later runs; this trial's recorded score is left as scored. Nothing got the work right on 18 tasks, edited a protected file on one, and followed no conventions: it committed on main with plain subjects and all-caps file names.
19 workflow tasks, one attempt per task per harness, 76 trials. The same three convention tasks apply.
| Harness | Correct | Conventions followed | Avg cost | Avg time |
|---|---|---|---|---|
| Studio Moser v8 | 19 of 19 | 3 of 3 | $0.45 | 57s |
| Studio Moser v5 | 17 of 19 | 1 of 3 | $0.56 | 1m 22s |
| Nothing | 16 of 19 | 0 of 3 | $0.30 | 1m 08s |
| Superpowers | 16 of 19 | 0 of 3 | $0.49 | 1m 14s |
Studio Moser v8 is the pick and the only harness that got everything right. Every harness did the work on every task; the others lost trials only on conventions. Nothing and Superpowers committed on main with plain subjects and all-caps file names; v5 branched and wrote Conventional Commits but missed the file-naming rule twice. Nothing remains the cheapest. The three older harnesses also ran five research tasks from real repositories and solved all of them; v8 has not run that lane.
Costs are API-equivalent estimates from list prices, not bills. Work-quality grades are blinded automated grades and never decide the ranking. One attempt per task is a real result, not a reliability guarantee: these cover the frozen tasks and kickoff conditions above, not every codebase.
Check git status --short --branch and recent commits in the intended checkout. Run commands through that checkout's uv run. Inspect the tracked reports under runs/evidence/ and the local manifests under runs/generated/ before claiming a run has happened; manifests are ignored and may be absent in a fresh clone. Planning and documentation requests do not authorize model execution: each exact manifest digest needs fresh approval before run execute.
| Contender | Delivered inputs |
|---|---|
| Nothing | Provider-native runtime with no added harness |
| Superpowers | Pinned Superpowers plugin |
| Studio Moser | Every plugin and skill in the pinned Skills-n-Stuff collection, its declared dependencies and a frozen rubric |
An optional studio-personality contender loads only the frozen house-style file, to isolate the writing guidance from the engineering skills.
The kickoff fixes the orchestrator's model and effort; normal child routing stays available. Everything the agent tree spends is measured together. Correctness comes first: a contender is eligible when it matches the best correctness observed in the cohort and its final-patch review found no confirmed remaining defects. Among eligible contenders, cost and time decide using plain ratio thresholds. One attempt per task on a declared scope is decision evidence; repetitions are optional.
| Lane | Scope |
|---|---|
tasks/workflow/ |
Nineteen active controlled React/TypeScript, JavaScript, static-web and Rust fixtures (three earlier ones are retired, replaced by -committed versions that also score house conventions, and kept so retained evidence stays reproducible), each with a neutral Comparison Instruction.md, a scripted user, a protected verifier and five model-free QA cases. Three -committed tasks also ask for a notes document and a commit, and score house conventions |
| DeepSWE research | Five pinned tasks from real open-source repositories, fetched into ignored .cache/deepswe/. pest-character-class-coalescing was dropped: its hidden test expects a side effect of the reference implementation that contradicts the instruction, so every harness failed it identically. |
These are frozen benchmark projects. A harder test means selecting or authoring a frozen task with a protected verifier, not a more demanding prompt. See Task Authoring.
- Copy a template from
runs/examples/, set the harness commits, kickoff model and tasks, and put the reviewed personal rubric at ignoredruns/inputs/Model Rubric.yml. - Plan:
uv run harness-test run plan --request 'runs/inputs/Experiment Request.json'. This installs plugins and inspects images but starts no model. - Obtain approval of the printed manifest digest, then
uv run harness-test run execute --manifest … --approve sha256:…. - Evaluate the retained submissions:
review prepareandreview recordfor the blinded final-patch review,collaboration prepareandcollaboration recordfor the blinded automated work-quality grades. Reviewer and grader sessions are model work covered by the run's approval, so an approved run continues through them without a second request. - For a full toolbox pass, run
campaign summarizewith each lane's original report followed by any recovery or correction reports in execution order. It fills every scheduled slot once, never replaces a completed trial unless the task itself was corrected, and writes the summary the dashboard reads.
The Runbook has the exact commands; Methodology explains the measurements and rules.
The dashboard is a read-only Observable site built from local evidence:
npm ci --prefix dashboard --ignore-scripts
npm --prefix dashboard test
npm --prefix dashboard run buildThe build reads the tracked runs/evidence/ reports and runs/campaigns/*/Summary.json and fails on any report that is not schema-valid and public-safe. Every push to main rebuilds it and deploys it to GitHub Pages. The Results page opens with a plain-language read of each harness (pros, cons, one recommendation), then the per-task results by type and difficulty, then a Behavior section with transcript metrics, annoyance counts and automated grades for every kickoff model and harness version side by side.
| Location | What to inspect |
|---|---|
src/harness_testing/CLI.py |
CLI entry and supported flags |
Experiments.py, Contenders.py, Comparison_Tasks.py |
Request compilation, frozen harness versions and neutral task materialization |
Runs.py, Materialize.py, provider agents, Native_Conversation.py, Simulated_User.py |
Execution, plugin delivery, native turns and the bounded simulated user |
Trial_Evidence.py, Experiment_Reports.py, Comparisons.py, Campaigns.py |
Whole-tree accounting, per-trial evidence, the per-task verdict and stitched campaigns |
Code_Reviews.py, Collaboration_Grading.py, Collaboration_Quality.py |
Blinded final-patch review, blinded automated grades and deterministic transcript metrics |
policy/, Versions.toml, runs/Profiles.toml |
Schemas, thresholds, pins and admission assumptions |
tests/, task-local tests/ |
Python unit checks and protected model-free task QA |
dashboard/ |
Read-only Observable UI |
Local artifacts are ignored: runs/inputs/ (private requests and rubric), runs/generated/<digest>/ (manifests and current Run_Report.json), arms/materialized/ and .cache/ (frozen bundles and task materializations), jobs/raw/ and provider homes (raw traces), dashboard/dist/, and campaign Plan.json files. Run reports in runs/evidence/ and campaign summaries are tracked and public; the dashboard build and the pre-commit hook reject any that fail the public-safety screen.
Never copy raw provider traces, hidden reasoning, tool output, session IDs, credentials or host paths into tracked files or dashboard assets. Reports carry only the normalized user-visible root conversation.
git config core.hooksPath .githooks
uv sync --frozen
uv run harness-test validate --static-only
uv run pytest -q tests/unit
npm --prefix dashboard testFor a changed comparison fixture, prove the reference and no-op behavior first:
uv run harness-test task qa --task react-saved-view-feature --variant comparison --case oracle
uv run harness-test task qa --task react-saved-view-feature --variant comparison --case nopDocker-backed QA is model-free but can be expensive; do not rerun every pack between small edits.