Stage S spread runner + post-run analysis/media toolkit

.claude/skills/run-analysis/SKILL.md

@@ -0,0 +1,117 @@
+---
+name: run-analysis
+description: Full post-run pipeline for RLE benchmark runs — leaderboard analysis vs baseline, failure taxonomy, story mining with verbatim verification, footage indexing, and X-ready video production. Use after any live run or model spread completes.
+---
+
+# RLE Post-Run Pipeline
+
+Process the output of a live run/spread end-to-end. Proven on the Stage S 6-model spread
+(2026-06-10). Governing discipline: **separate harness artifacts from model quality before
+ranking anything**, and **verify every quote verbatim before publishing it**.
+
+Inputs per model: `results/<run>/<name>/` containing `01_*_summary.json`, `01_*.csv`,
+`events.jsonl`, `01_*_deliberations.jsonl`. Read with `PYTHONIOENCODING=utf-8`.
+
+## Phase 0 — Media capture setup (BEFORE launching the run)
+
+Everything here must be running before tick 1 — capture started late on the Stage S spread
+and lost the first model's early tick snapshots + 4.7 min of OBS coverage.
+
+1. **OBS** (user starts it): source must be **Game Capture** — Display/Window capture
+   region-grabs whatever floats over the game (dialogs, browser tabs with API keys).
+   Output to **D:** (~10 GB/hour at the spread's settings; check free space). The default
+   filename encodes the start time (`2026-06-10 20-55-09.mkv`) — that timestamp is the
+   `--obs-start` input for the footage index later.
+2. **Verify OBS is actually recording** — silence is not success. NTFS under-reports size
+   for open files; check the real stream length twice a few seconds apart:
+   `[System.IO.File]::Open($p,'Open','Read','ReadWrite').Length` (PowerShell). It must grow.
+3. **Capture loop** (frames + tick snapshots), in background for the whole run:
+   `bash scripts/capture_run_media.sh results/<run> D:/RLE_media/footage/frames`
+   (set FFMPEG env var if ffmpeg isn't on PATH; winget package Gyan.FFmpeg).
+   Note: ffmpeg gdigrab needs even dimensions — the script's single-frame mode is fine,
+   but any direct video capture needs `-vf "crop=trunc(iw/2)*2:trunc(ih/2)*2"`.
+4. **Game hygiene for clean footage**: disable dev-mode auto-open of the debug log
+   (issue #33) and expect the camera to sit at map center unless #34 is built — plan for
+   caption-driven clips, not action footage.
+5. Don't screen-record the desktop yourself (privacy: anything can float over the game
+   region) — periodic stills are auditable; OBS Game Capture is the continuous source.
+
+## Phase 1 — Analysis (deterministic)
+
+```
+PYTHONIOENCODING=utf-8 python scripts/analyze_spread.py --spread-dir results/<run>
+```
+
+Writes `leaderboard.json` + `analysis.md`. Key rules baked in:
+- **Day-align** against the baseline (`results/baseline/.../seed*/...csv`), never tick-align —
+  the no-agent baseline ticks ~30x/day, agent runs ~2-3x/day.
+- Rank by **mean composite over the run**, not the final tick (endpoint = one bad event).
+- Action success reported raw AND ex-artifact (harness failure markers are listed in the
+  script — keep them current as harness bugs are fixed/found).
+- Always reconcile estimated cost against the OpenRouter dashboard/key endpoint — the
+  tracker has undercounted reasoning tokens before (issue #33).
+
+Then per playbook: check per-type action success for any type at 0/N (suspect harness, not
+model), count repeated identical failures (perseveration — check whether DO-NOT-REPEAT
+feedback was delivered before blaming the model), find best/worst tick and tie each to an
+event.
+
+## Phase 2 — Story mining (parallel agents, then VERIFY)
+
+Fan out one Explore agent per model over `01_*_deliberations.jsonl` asking for 3-5 quotable
+moments (tick, agent role, VERBATIM quote, why it's good content). Give each agent that
+model's stats (rank, quirks) as context.
+
+**Then re-grep every quote against the actual log before using it anywhere.** Miners
+paraphrase: in the Stage S spread, 2 of 6 "verbatim" headline quotes were embellishments.
+Save the verified bank to `results/<run>/stories.md` with ✅/⚠ verification marks.
+
+## Phase 3 — Footage index (if Phase 0 ran)
+
+With the OBS master + capture-loop output from Phase 0:
+
+```
+python scripts/build_footage_index.py --log <console.log> --obs-file <rec.mkv> \
+    --obs-start "YYYY-MM-DD HH:MM:SS" --out footage_index.json
+```
+
+Maps every model + tick to wall clock and OBS file offsets, and catalogs warnings.
+
+## Phase 4 — Videos (EDL → ffmpeg → Remotion)
+
+Deck workflow (thariqs cc-video-editing-deck): edit-decision-list as JSON with rationale →
+ffmpeg executes cuts → Remotion renders data-driven graphics → verify by extracting stills
+of every segment (Read the PNGs — never ship an unviewed render).
+
+- Remotion project: `D:\RLE_media\rle_video\` (timing knobs in `src/anim.ts`, leaderboard
+  data in `src/data.json` regenerated from `leaderboard.json`). Assets must live under
+  `public/` (staticFile) — junctions break the bundler.
+- Timelapse source: `ffmpeg -i master.mkv -vf "setpts=PTS/60,fps=24,crop=trunc(iw/2)*2:trunc(ih/2)*2" -an`.
+- **Dashboard clip rig** (source: `scripts/record_dashboard.mjs`) — headless Playwright
+  recording, never screen capture. Four processes, in order:
+  1. RimWorld + RIMAPI up (the dashboard's connect gate probes :8765; CORS is fine, but
+     the URL field starts EMPTY — the script fills it and clicks Connect).
+  2. `./.venv/Scripts/python scripts/serve_dashboard.py results/replay` (CORS server :9000).
+  3. Dashboard: `PORT=3001 BROWSER=none bun run start` in rimapi-dashboard — NOT :3000
+     (Remotion's render server squats on 3000; craco exits "already running" instead of failing loud).
+  4. `python scripts/replay_ticks.py --model <name> --interval 2 --loop` (feeds
+     results/replay/latest_tick.json from the run's tick_snapshots).
+  Then record: set up a scratch dir once (`bun add playwright && bunx playwright install
+  chromium`), copy record_dashboard.mjs next to it, and run with **node** — Playwright's
+  chromium launch hangs 180s under bun on Windows. The script seeds the 5 RLE widgets via
+  a localStorage `dashboard_presets` preset (fresh browser contexts have no layout, and the
+  RLE widgets aren't in the default). Record 1920x1080 and 1080x1920 passes; trim the
+  connect/load preamble (~8s) when converting webm → mp4
+  (`ffmpeg -ss 8 -i in.webm -c:v libx264 -crf 19 -pix_fmt yuv420p -r 30 out.mp4`).
+- Verify every render before calling it done: extract stills of each segment with ffmpeg
+  and actually Read them — stale ports, empty widgets, and black OffthreadVideo frames all
+  look identical to success in the exit code.
+- Render 16:9 and 9:16 variants (compositions take portrait into account via useVideoConfig).
+- All media outputs to **D:** (`D:\RLE_media\`), never C:.
+
+## Phase 5 — Publish prep
+
+- Update memory (`project_benchmark_spread` or successor) with results + artifact locations.
+- X thread: 3-5 posts max. Lead with the strongest narrative clip (hero or disaster), data
+  (leaderboard) in the middle, method/repo last. Draft to `D:\RLE_media\x_thread.md`.
+- File issues for every harness bug found before the next run.

results/benchmark_history.jsonl

@@ -5,3 +5,9 @@
 {"timestamp": "2026-06-10T01:14:38.139936+00:00", "scoring_version": "1.0", "git_commit": "295f478", "git_branch": "fix/strict-action-schema", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 2, "outcome": "timeout", "final_score": 0.8598, "ticks_run": 2, "cost_snapshot": {"total_prompt_tokens": 28, "total_completion_tokens": 20578, "total_tokens": 20606, "estimated_cost_usd": 1.02918, "wall_time_s": 163.26, "num_calls": 14, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 99, "errors_by_type": {}, "avg_deliberation_ms": 25524.75, "action_success_rate": 0.7273, "total_tokens": 20606, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8598, "outcome": "timeout", "ticks": 2}]}
 {"timestamp": "2026-06-10T01:32:30.462265+00:00", "scoring_version": "1.0", "git_commit": "5d361d0", "git_branch": "master", "git_dirty": false, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7536, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 102045, "total_tokens": 102185, "estimated_cost_usd": 5.10365, "wall_time_s": 897.04, "num_calls": 70, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 633, "errors_by_type": {}, "avg_deliberation_ms": 26282.04, "action_success_rate": 0.7167, "total_tokens": 102185, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7536, "outcome": "timeout", "ticks": 10}]}
 {"timestamp": "2026-06-10T04:12:15.578574+00:00", "scoring_version": "1.1", "git_commit": "98ee8e2", "git_branch": "fix/run-integrity", "git_dirty": false, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 2, "outcome": "timeout", "final_score": 0.8594, "ticks_run": 2, "cost_snapshot": {"total_prompt_tokens": 28, "total_completion_tokens": 19000, "total_tokens": 19028, "estimated_cost_usd": 0.95028, "wall_time_s": 151.53, "num_calls": 14, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 96, "errors_by_type": {}, "avg_deliberation_ms": 24304.49, "action_success_rate": 0.7857, "total_tokens": 19028, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8594, "outcome": "timeout", "ticks": 2}]}
+{"timestamp": "2026-06-11T01:05:39.907207+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.6272, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 104867, "total_tokens": 105007, "estimated_cost_usd": 5.24475, "wall_time_s": 911.47, "num_calls": 70, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 673, "errors_by_type": {}, "avg_deliberation_ms": 27323.28, "action_success_rate": 0.7188, "total_tokens": 105007, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.6272, "outcome": "timeout", "ticks": 10}]}
+{"timestamp": "2026-06-11T01:19:31.805063+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-opus-4-8", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7969, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 81269, "total_tokens": 81409, "estimated_cost_usd": 2.032425, "wall_time_s": 828.72, "num_calls": 70, "prompt_price_per_token": 5e-06, "completion_price_per_token": 2.5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 651, "errors_by_type": {}, "avg_deliberation_ms": 20362.56, "action_success_rate": 0.76, "total_tokens": 81409, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7969, "outcome": "timeout", "ticks": 10}]}
+{"timestamp": "2026-06-11T01:41:06.279109+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "openai/gpt-5.5", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.609, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 195940, "total_completion_tokens": 88922, "total_tokens": 284862, "estimated_cost_usd": 3.64736, "wall_time_s": 1293.37, "num_calls": 69, "prompt_price_per_token": 5e-06, "completion_price_per_token": 3e-05, "pricing_source": "override"}, "event_summary": {"total_events": 988, "errors_by_type": {"provider_error": 1}, "avg_deliberation_ms": 32919.53, "action_success_rate": 0.7077, "total_tokens": 284862, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.609, "outcome": "timeout", "ticks": 10}]}
+{"timestamp": "2026-06-11T01:52:11.998324+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "google/gemini-3.5-flash", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.8108, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 194601, "total_completion_tokens": 166376, "total_tokens": 360977, "estimated_cost_usd": 1.789286, "wall_time_s": 664.45, "num_calls": 69, "prompt_price_per_token": 1.5e-06, "completion_price_per_token": 9e-06, "pricing_source": "override"}, "event_summary": {"total_events": 753, "errors_by_type": {"parse_failure": 1}, "avg_deliberation_ms": 15690.77, "action_success_rate": 0.6957, "total_tokens": 360977, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8108, "outcome": "timeout", "ticks": 10}]}
+{"timestamp": "2026-06-11T02:00:51.207361+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "x-ai/grok-4.3", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7904, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 209815, "total_completion_tokens": 66013, "total_tokens": 275828, "estimated_cost_usd": 0.427301, "wall_time_s": 517.38, "num_calls": 70, "prompt_price_per_token": 1.25e-06, "completion_price_per_token": 2.5e-06, "pricing_source": "override"}, "event_summary": {"total_events": 761, "errors_by_type": {}, "avg_deliberation_ms": 8401.06, "action_success_rate": 0.7647, "total_tokens": 275828, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7904, "outcome": "timeout", "ticks": 10}]}
+{"timestamp": "2026-06-11T02:29:49.825955+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "deepseek/deepseek-v4-pro", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.5959, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 189569, "total_completion_tokens": 121047, "total_tokens": 310616, "estimated_cost_usd": 0.187773, "wall_time_s": 1735.02, "num_calls": 68, "prompt_price_per_token": 4.35e-07, "completion_price_per_token": 8.7e-07, "pricing_source": "override"}, "event_summary": {"total_events": 895, "errors_by_type": {"deliberation_timeout": 1, "parse_failure": 1}, "avg_deliberation_ms": 45005.51, "action_success_rate": 0.7368, "total_tokens": 310616, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.5959, "outcome": "timeout", "ticks": 10}]}