[XuanL17]

Adds the engine behind the HumanStudy-Hub playground: pick a study, a base model, and the prompt the agent takes into the experiment, then see where the agent behaved like the original participants and where it did not.

How it works

playground/run_playground.py drives each study's own scripts/config.py and scripts/evaluator.py, so a playground run is scored exactly like a benchmark run. The model, the participant prompt, and the participant count are the only things that change. Every model — including Claude and Gemini names — is routed through OpenRouter by passing the OpenRouter base explicitly.

playground/analysis.py normalises the two evaluator output shapes into one comparison table. Eleven studies already enrich their test results with effect sizes; for the rest it derives Cohen's d from the paper's reported F/t/r and from the agent's t-statistic. Tests with no honest single-d equivalent (multi-numerator F, chi-square) are left out of the effect-size comparison rather than given a substituted zero.

Charts and the written reading come from Claude Code through OpenRouter, following the contract in playground/CLAUDE.md. validate_charts.py checks the output on the runner before it is stored — plain JSON only, restricted trace types, no *src/meta fields — and anything invalid or late falls back to the deterministic charts in default_charts.py. A run always ends with charts a researcher can read.

Run budget and researcher keys

Runs on the shared OPENROUTER_API_KEY are capped at 60 participant sessions. A researcher who supplies their own key gets 600. HumanStudy-Hub seals that key with AES-256-GCM under a secret shared with this workflow, so the private jobs repository only ever holds ciphertext, and the workflow drops it once the run ends.

Repository settings this needs

Type	Name	Notes
Actions secret	PLAYGROUND_KEY_SECRET	New. Must match the web app; without it the playground still runs on the shared key and only refuses researcher-supplied keys
Actions secret	OPENROUTER_API_KEY	Already set for Build Study
Actions secret	HUMANSTUDY_PIPELINE_TOKEN	Already set for Build Study
Actions variable	PLAYGROUND_CHARTS_MODEL	Optional; defaults to anthropic/claude-sonnet-5

Separate commit: study_006

studies/study_006/scripts/config.py overrode load_material but resolved it against the study root instead of source/, so every lookup missed and the study could not be loaded at all — by any runner, not just the playground. Kept as its own commit.

Verification

• All 15 studies load, build trials, and score end to end via --simulate (no API calls).
• pytest tests — 127 passed (118 existing, 9 new in tests/test_playground.py).
• The AES seal/open round trip was checked against the web app's implementation, including tamper rejection.

Not yet exercised: a real dispatched run against OpenRouter. That needs this branch on main first, since GitHub can only dispatch a workflow that exists on the default branch.

🤖 Generated with Claude Code