Skip to content

Add a source-verified five-app Muse and Jev evaluation workflow - #8

Closed
ianu82 wants to merge 2 commits into
stagingfrom
review/jev-muse-evals
Closed

ianu82 wants to merge 2 commits into
stagingfrom
review/jev-muse-evals

Conversation

@ianu82

@ianu82 ianu82 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Makes the frozen Muse + Jev comparison reproducible from a clone, with independently graded tasks and source-verified actor builds. Product changes and recorded campaign data are reviewed separately.

  • Five apps (CSV, quiz, inventory, notes, application form), each with creation/edit/feature/bug tasks, three repetitions: 60 attempts per actor.
  • Deterministic browser/API, TypeScript, persistence/restart, protected-file and mobile-fit checks; 40 positive/negative controls. Jev routes the candidate, not the grader.
  • Serial balanced A/B, fixed budgets, all failures retained, and mathematical early stopping. Acceptance requires at least vanilla's passes, no paired storage regression, and lower mean and P50.
  • evals/reproduce.py separates setup, verified builds, offline checks and the explicit paid step. It rebuilds original vanilla plus the exact frozen winner source, rather than the newer integration build. Platform-specific hashes and the complete protocol are frozen before model calls. Each colleague uses their own MindsHub credentials.

Start with evals/README.md, then PROTOCOL.md, yoloeval/catalog.py, browser/checks.mjs, and yoloeval/unseen.py. Most added lines are the pinned fixture lockfile.

Validated locally: both source-verified builds, 23 Python tests, two real-browser control tests, all 40 controls, and fake-provider timeout/drain/process-cleanup integration. No paid calls. The published portability changes replace original account/binary assumptions; fixtures, grading and acceptance rules are preserved. Campaign 001's grader correction and the limits of five exposed app families are documented. The historical results and offline audit are in #9, stacked on this PR; the product port is #7. A fresh GitHub clone successfully rebuilt both source-verified actors and reproduced the recorded results audit.

GitHub CI status

Linux/macOS tests, formatting and shellcheck pass. Windows record-isolation tests already fail on the unchanged main revision (TestTheNewestRecordComesFirst, TestNoRecordsIsNotAnError, TestAnUnwritableHomeIsSurvived): their HOME-only setup does not isolate Windows USERPROFILE. This PR reproduces those existing failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant