Skip to content

Stage S spread runner + post-run analysis/media toolkit#35

Merged
jkbennitt merged 1 commit into
masterfrom
feat/run-analysis-toolkit
Jun 11, 2026
Merged

Stage S spread runner + post-run analysis/media toolkit#35
jkbennitt merged 1 commit into
masterfrom
feat/run-analysis-toolkit

Conversation

@jkbennitt

Copy link
Copy Markdown
Member

Summary

Codifies everything built during the 2026-06-10 Stage S 6-model spread so the post-run pipeline is repeatable:

  • scripts/run_spread_n1.sh — the spread runner (was untracked since Stage S prep)
  • scripts/analyze_spread.py — cross-model leaderboard: day-aligns against the no-agent baseline (tick-alignment is wrong: baseline ticks ~30x/day vs agent 2-3x/day), ranks by mean composite, reports raw + ex-artifact action success with a harness-failure marker list
  • scripts/build_footage_index.py — maps console-log model markers + ticks to wall clock and OBS recording offsets for frame-accurate video cuts
  • scripts/capture_run_media.sh — during-run loop: game-window stills (gdigrab) + per-model latest_tick.json snapshots (dashboard replay source)
  • scripts/replay_ticks.py / scripts/record_dashboard.mjs — replay a finished run into the React dashboard and record it headless (Playwright; run with node, not bun)
  • .claude/skills/run-analysis — the whole pipeline as a repo skill: Phase 0 capture setup → analysis → story mining with mandatory verbatim re-verification → footage index → EDL/ffmpeg/Remotion videos → publish prep
  • benchmark_history.jsonl — the 6 spread run entries

Spread context

Leaderboard (N=1, mean composite): grok43 0.831 > opus48 0.826 > gemini35 0.825 > fable5 0.803 > gpt55 0.722 > deepseekv4 0.704. Actual OpenRouter spend $7.12. Key findings + queued fixes in #33; capture improvement idea in #34. Detailed artifacts in results/spread/ (gitignored; story bank + analysis also mirrored to the rle-media repo).

Test plan

  • ruff check src/ tests/ scripts/ clean, mypy src/ strict clean, 458 tests pass
  • analyze_spread.py smoke-run post-cleanup reproduces the leaderboard byte-for-byte
  • Capture/replay/record scripts are the exact code used live during the spread (parameterized paths only)

🤖 Generated with Claude Code

- scripts/run_spread_n1.sh: the 6-model N=1 spread runner used 2026-06-10
- scripts/analyze_spread.py: cross-model leaderboard (day-aligned vs no-agent
  baseline, raw + ex-artifact action success, perseveration-friendly failure
  taxonomy)
- scripts/build_footage_index.py: console log -> per-model/per-tick wall clock
  + OBS file offsets for frame-accurate video cuts
- scripts/capture_run_media.sh: during-run capture loop (game-window stills +
  latest_tick.json snapshots for dashboard replay)
- scripts/replay_ticks.py + scripts/record_dashboard.mjs: replay a run into the
  React dashboard and record it headless via Playwright
- .claude/skills/run-analysis: the full post-run pipeline as a repo skill
  (capture setup -> analysis -> verified story mining -> footage index ->
  EDL/ffmpeg/Remotion videos -> publish prep)
- benchmark_history.jsonl: 6 spread run entries

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jkbennitt jkbennitt merged commit 9fa6fac into master Jun 11, 2026
3 checks passed
@jkbennitt jkbennitt deleted the feat/run-analysis-toolkit branch June 11, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant