Add a source-verified five-app Muse and Jev evaluation workflow

.gitignore

@@ -40,3 +40,11 @@ go.work.sum
 # on disk and the compiler still reads it. `find . -name '* [0-9]*'` finds
 # them when a build fails for no apparent reason.
 * [0-9]*
+
+# Local evaluation dependencies and evidence
+/evals/node_modules/
+/evals/.cache/
+/evals/runs/
+/dist/
+__pycache__/
+*.pyc

README.md

@@ -327,3 +327,7 @@ for and installs updates. Non-interactive output remains animation-free.
 
 Version tags matching `v*` create permanent GitHub releases. Every push to
 `main` refreshes the rolling `latest` release used by the self-updater.
+
+## Reproduce the Muse + Jev benchmark
+
+See [the evaluation guide](evals/README.md) for source-verified builds of original vanilla and the frozen winner, offline calibration, and the five-app comparison. The paid benchmark is a separate explicit step.

evals/.gitignore

@@ -0,0 +1,5 @@
+.cache/
+node_modules/
+__pycache__/
+*.pyc
+results/

evals/HILLCLIMB.md

@@ -0,0 +1,3 @@
+# Validation only
+
+Do not hillclimb on these tasks during the frozen comparison. See [PROTOCOL.md](PROTOCOL.md). Once failures are used for product tuning, this suite is no longer an unseen validation set.

evals/PROTOCOL.md

@@ -0,0 +1,47 @@
+# YoloCoder unseen-app validation 002
+
+This suite was authored after the Round 032 winner was frozen. It is validation, not a new hillclimb. The winner and original vanilla binaries are copied verbatim and identified by SHA-256; neither actor is edited or rebuilt.
+
+| App | Distinctive behaviour | Seeded bug | Feature |
+|---|---|---|---|
+| CSV explorer | Quoted CSV, validation, persistent datasets, row search | Case-sensitive search | Numeric/text sorting composed with search |
+| Quiz builder | Question editor, answer validation, practice and scoring | Choice zero never scores | Persisted explanations and schema migration |
+| Inventory manager | Case-insensitive SKU uniqueness and atomic stock adjustments | Negative stock is clipped instead of refused | Live low-stock filtering |
+| Notes editor | Literal text, partial edits, search and preservation | Title-only updates erase body | Archive/restore with migration and filtering |
+| Application form | Drafts, three-step navigation, validation, immutable submissions | Back navigation clears the statement | Duplicate a submission into an independent draft |
+
+Every app also has a scaffold-to-app creation task and a small heading/button-colour edit. All 20 tasks are independent; three repetitions produce 60 attempts per actor, 120 planned in total. Noncreation tasks start from handwritten reference implementations, with pre-existing records. No generated app becomes the next task's starting point.
+
+## Fixed comparison
+
+Both actors use MindsHub Muse `muse-spark-1-3` for coding. The frozen winner additionally uses its existing `jev-1.13.0` router. The original vanilla binary is the exact one used for the first baseline, not the later context/checklist control.
+
+One actor runs at a time. Within matching task/repetition pairs, A/B and B/A order is balanced; creation tasks run first, followed by features, bugs, then edits. Order inside each block is seeded. Both have a 180-second actor budget and 30 forwarded model calls. Setup and independent grading are excluded from agent time. Timeout cleanup can make observed wall time exceed the nominal budget; raw timings are retained.
+
+An attempt passes only if the actor reports no error and all required typecheck, functional browser/API, protected-file, persistence and mobile-fit checks pass. Graders and reference solutions are outside the project given to the actor. This is process and HOME isolation, not an OS filesystem sandbox.
+
+The winner validates only if it has at least as many passes as vanilla, no paired failures of the two storage-preservation checks when vanilla passes those checks, and strictly lower mean and nearest-rank P50 across all attempts, including failures. The candidate's 60/60 result is reported separately. Five app families cannot establish universal reliability, and repetitions are correlated.
+
+The authoritative executable rules are `evals/yoloeval/unseen.py`. A JSON preregistration, manifests, evaluator/dependency hashes, binary hashes and full schedule are written before the first paid call. Do not use the legacy general-purpose comparator's promotion rule for this validation.
+
+Early stopping occurs only after an attempt and all its upstream calls are drained: an unrecoverable pass-count deficit, a paired storage-check regression, or a mathematically impossible latency result. Unknown future vanilla times have no invented upper bound; latency stopping therefore waits until vanilla's full timing set is known. Early-stopped campaigns are incomplete and are never presented as 60-sample results. Infrastructure gaps stop the campaign without silently retrying or replacing an attempt.
+
+## Offline validation
+
+Each task has a correct positive control and a deliberately incomplete or buggy negative control: 40 controls. Noncreation negatives must fail exactly the intended requirement while other checks pass. Creation negatives must fail the app contract but retain valid TypeScript and protected files. Exploratory calibration exposed exact-label matching problems, migration requirements incorrectly grouped with preservation, an overbroad reference colour change and header rows included in CSV sort assertions. The final `calibration-frozen` directory reruns every control against fixed source fingerprints. Earlier calibration directories are preserved as development evidence and excluded from the final calibration gate.
+
+Unit tests cover recorder behaviour, scoring, pairing, deadline cleanup and new stopping rules. The runtime integration check uses a local fake provider and no paid model calls. No source/scoring changes are allowed once the paid comparison begins. Any future product tuning against these tasks makes them development cases, requiring another fresh holdout.
+
+Visual polish, comprehensive accessibility/security, internet integrations and deployment are outside this functional benchmark. Provider token accounting is preserved as reported; unknown prices and inconsistent token usage are not converted into invented cost or TPS estimates.
+
+## Campaign 002: native search-input correction
+
+Campaign 001 was stopped after 24 completed attempts and one interrupted attempt because the grader incorrectly excluded the browser's `searchbox` role. The generated `notes.create` trial 1 app had a correctly associated label and a native `type="search"` input. Its failed search check was a harness false negative. The original artifacts and interruption were retained in the private working archive; that campaign is invalidated and cannot establish acceptance.
+
+Campaign 002 adds searchbox support and a direct associated-label fallback. Browser unit tests cover text, search, number, email, URL, telephone and password inputs, both wrapping and external labels, populated textareas, selects and checkboxes. Additional offline controls test native search inputs in both CSV and notes, and the exact generated notes source that exposed the false negative.
+
+The original vanilla and winning binaries, public tasks, repetitions, budgets, order seed, pass requirements and acceptance rules are unchanged. No YoloCoder source, prompts or model configuration were tuned against these cases. The entire 120-attempt comparison restarts under a newly frozen evaluator; no previous paid row is substituted into it. This is an infrastructure correction, not a candidate revision.
+
+## Portable replication
+
+This published harness preserves the tasks, fixtures, graders, ordering, budgets and decision function from campaign 002. Its launcher accepts locally rebuilt actor hashes rather than requiring the original macOS binaries or the original account-specific credential prefix. `reproduce.py` verifies every actor source file against the frozen inventory, runs offline controls, and records new binary, evaluator and dependency hashes before a new campaign. The published suite digest therefore differs from the historical receipt. See [README.md](README.md).

evals/README.md

@@ -0,0 +1,41 @@
+# Replicate the Muse + Jev comparison
+
+This reproduces the **frozen Round 032 winner versus original vanilla**, using the five-app unseen-validation suite. It does not substitute current main or the new integration branch for either actor. Source files are checked against [actors.json](actors.json) before building. Build hashes vary across platforms/toolchains and are recorded as a new campaign.
+
+## Requirements
+
+- A full clone of this repository, including `review/frozen-r032` (or fetch that branch).
+- Go 1.24+, Python 3.10+, Node.js 22 and npm; macOS or Linux with `git`, `tar`, and `/bin/sh`. SQLite's native dependency may need local build tools.
+- For the paid step only, your own YoloCoder MindsHub configuration in `~/.config/yolocoder/config.json` and `credentials.json`. Set it up through YoloCoder's `config` command. The harness requires `https://api.mindshub.ai`, pins coding to `muse-spark-1-3`, and allows `jev-1.13.0` only for the candidate. Never commit credentials.
+
+## Workflow
+
+From the repository root:
+
+```sh
+python3 evals/reproduce.py setup
+python3 evals/reproduce.py prepare
+python3 evals/reproduce.py check
+# Paid step: up to 120 attempts; it is not started by setup/prepare/check.
+python3 evals/reproduce.py run
+```
+
+Use `--output /absolute/path/to/new-campaign` on prepare/check/run to choose a different output directory. `GO=/path/to/go` selects a Go executable. Local output and dependencies are ignored by Git. Preparation will not overwrite an existing build; use a fresh directory for another campaign. Setup installs pinned npm dependencies and Playwright Chromium.
+
+The offline check runs unit/browser tests, all **40 positive/negative controls**, and fake-provider timeout/cleanup checks. The paid runner rejects changed actors, graders or dependencies; preregisters the whole schedule; preserves all attempts; and cannot retry an interrupted attempt in place. Resuming a fully recorded boundary reuses its original rows. Do not edit source/dependencies or run multiple campaigns against the same cache while a campaign is active. Eval apps receive a proxy credential, not your provider key. Isolation is per process/project/HOME, not an OS security sandbox.
+
+Results appear in `evals/runs/replication/comparison/`: `decision.json` is the authoritative acceptance result; each actor has a report and every attempt's result, model-call metadata, generated source changes, browser screenshots and logs. Auto-update is disabled for actor processes. Provider errors and timeouts count in the primary outcome; no retries or deletions improve the score.
+
+## Tasks and scoring
+
+Five apps: CSV explorer, quiz builder, inventory manager, notes editor, and a three-step application form. Each has app creation, a small UI edit, a feature, and a seeded bug, repeated three times per actor. Tasks start independently from a scaffold or handwritten reference with existing data. The coding agent does not receive the graders or solutions.
+
+Passing requires every functional browser/API, TypeScript, data-preservation/restart, protected-file and mobile-fit check, plus no actor error. A new candidate is accepted only with at least vanilla's passes, no paired storage regressions, and lower mean **and** nearest-rank P50 agent time over all 60 attempts. Setup/grading time is excluded. Difficult tasks run first, with balanced serial A/B order. Mathematically impossible candidates stop early and are labelled incomplete.
+
+See [the full protocol](PROTOCOL.md), [task definitions](yoloeval/catalog.py), [grading code](browser/checks.mjs), and [acceptance/early-stop rules](yoloeval/unseen.py). Jev routes the candidate; deterministic checks grade the results. There is no LLM judge in this protocol.
+
+## Interpretation
+
+The recorded unseen result was **45/60 → 53/60**, mean **70.58s → 47.37s**, P50 **54.30s → 30.92s**. This is a complete-bundle result, not an isolated Jev ablation. Campaign 001 was invalidated for a native-search-input grader bug; campaign 002 restarted all 120 attempts after correcting it. A provider outage was held between attempts; all scored service errors remained in the final data.
+
+The earlier tuned suite reached 60/60; it is distinct from this holdout. These published tasks are now exposed and must not become evidence of generalization after further tuning. Use fresh apps for the next holdout. This checks a specific React/Express/SQLite stack, not comprehensive visual design, accessibility, security, deployment, or other languages. A replicated run measures current provider conditions, so identical scores and timings are not guaranteed.

evals/RUBRIC.md

@@ -0,0 +1,7 @@
+# Scoring
+
+An attempt passes only with no actor error and all mandatory checks passing. Weighted scores (acceptance 60%, regression 30%, usability 10%) are diagnostic only and cannot turn a failed attempt into a success.
+
+Campaign acceptance requires at least vanilla's pass count, no paired storage-check regression, and lower mean and nearest-rank median agent time across all attempts. The full executable rules are yoloeval/unseen.py and the generated preregistration.json. First-visible-correct-preview latency is not measured.
+
+Five new applications × four independent tasks × three repetitions = 60 attempts per actor. The repeat count does not mean 60 independent applications. No model judge determines functional pass/fail.