diff --git a/.claude/skills/run-analysis/SKILL.md b/.claude/skills/run-analysis/SKILL.md new file mode 100644 index 0000000..a6b2bab --- /dev/null +++ b/.claude/skills/run-analysis/SKILL.md @@ -0,0 +1,117 @@ +--- +name: run-analysis +description: Full post-run pipeline for RLE benchmark runs — leaderboard analysis vs baseline, failure taxonomy, story mining with verbatim verification, footage indexing, and X-ready video production. Use after any live run or model spread completes. +--- + +# RLE Post-Run Pipeline + +Process the output of a live run/spread end-to-end. Proven on the Stage S 6-model spread +(2026-06-10). Governing discipline: **separate harness artifacts from model quality before +ranking anything**, and **verify every quote verbatim before publishing it**. + +Inputs per model: `results///` containing `01_*_summary.json`, `01_*.csv`, +`events.jsonl`, `01_*_deliberations.jsonl`. Read with `PYTHONIOENCODING=utf-8`. + +## Phase 0 — Media capture setup (BEFORE launching the run) + +Everything here must be running before tick 1 — capture started late on the Stage S spread +and lost the first model's early tick snapshots + 4.7 min of OBS coverage. + +1. **OBS** (user starts it): source must be **Game Capture** — Display/Window capture + region-grabs whatever floats over the game (dialogs, browser tabs with API keys). + Output to **D:** (~10 GB/hour at the spread's settings; check free space). The default + filename encodes the start time (`2026-06-10 20-55-09.mkv`) — that timestamp is the + `--obs-start` input for the footage index later. +2. **Verify OBS is actually recording** — silence is not success. NTFS under-reports size + for open files; check the real stream length twice a few seconds apart: + `[System.IO.File]::Open($p,'Open','Read','ReadWrite').Length` (PowerShell). It must grow. +3. **Capture loop** (frames + tick snapshots), in background for the whole run: + `bash scripts/capture_run_media.sh results/ D:/RLE_media/footage/frames` + (set FFMPEG env var if ffmpeg isn't on PATH; winget package Gyan.FFmpeg). + Note: ffmpeg gdigrab needs even dimensions — the script's single-frame mode is fine, + but any direct video capture needs `-vf "crop=trunc(iw/2)*2:trunc(ih/2)*2"`. +4. **Game hygiene for clean footage**: disable dev-mode auto-open of the debug log + (issue #33) and expect the camera to sit at map center unless #34 is built — plan for + caption-driven clips, not action footage. +5. Don't screen-record the desktop yourself (privacy: anything can float over the game + region) — periodic stills are auditable; OBS Game Capture is the continuous source. + +## Phase 1 — Analysis (deterministic) + +``` +PYTHONIOENCODING=utf-8 python scripts/analyze_spread.py --spread-dir results/ +``` + +Writes `leaderboard.json` + `analysis.md`. Key rules baked in: +- **Day-align** against the baseline (`results/baseline/.../seed*/...csv`), never tick-align — + the no-agent baseline ticks ~30x/day, agent runs ~2-3x/day. +- Rank by **mean composite over the run**, not the final tick (endpoint = one bad event). +- Action success reported raw AND ex-artifact (harness failure markers are listed in the + script — keep them current as harness bugs are fixed/found). +- Always reconcile estimated cost against the OpenRouter dashboard/key endpoint — the + tracker has undercounted reasoning tokens before (issue #33). + +Then per playbook: check per-type action success for any type at 0/N (suspect harness, not +model), count repeated identical failures (perseveration — check whether DO-NOT-REPEAT +feedback was delivered before blaming the model), find best/worst tick and tie each to an +event. + +## Phase 2 — Story mining (parallel agents, then VERIFY) + +Fan out one Explore agent per model over `01_*_deliberations.jsonl` asking for 3-5 quotable +moments (tick, agent role, VERBATIM quote, why it's good content). Give each agent that +model's stats (rank, quirks) as context. + +**Then re-grep every quote against the actual log before using it anywhere.** Miners +paraphrase: in the Stage S spread, 2 of 6 "verbatim" headline quotes were embellishments. +Save the verified bank to `results//stories.md` with ✅/⚠ verification marks. + +## Phase 3 — Footage index (if Phase 0 ran) + +With the OBS master + capture-loop output from Phase 0: + +``` +python scripts/build_footage_index.py --log --obs-file \ + --obs-start "YYYY-MM-DD HH:MM:SS" --out footage_index.json +``` + +Maps every model + tick to wall clock and OBS file offsets, and catalogs warnings. + +## Phase 4 — Videos (EDL → ffmpeg → Remotion) + +Deck workflow (thariqs cc-video-editing-deck): edit-decision-list as JSON with rationale → +ffmpeg executes cuts → Remotion renders data-driven graphics → verify by extracting stills +of every segment (Read the PNGs — never ship an unviewed render). + +- Remotion project: `D:\RLE_media\rle_video\` (timing knobs in `src/anim.ts`, leaderboard + data in `src/data.json` regenerated from `leaderboard.json`). Assets must live under + `public/` (staticFile) — junctions break the bundler. +- Timelapse source: `ffmpeg -i master.mkv -vf "setpts=PTS/60,fps=24,crop=trunc(iw/2)*2:trunc(ih/2)*2" -an`. +- **Dashboard clip rig** (source: `scripts/record_dashboard.mjs`) — headless Playwright + recording, never screen capture. Four processes, in order: + 1. RimWorld + RIMAPI up (the dashboard's connect gate probes :8765; CORS is fine, but + the URL field starts EMPTY — the script fills it and clicks Connect). + 2. `./.venv/Scripts/python scripts/serve_dashboard.py results/replay` (CORS server :9000). + 3. Dashboard: `PORT=3001 BROWSER=none bun run start` in rimapi-dashboard — NOT :3000 + (Remotion's render server squats on 3000; craco exits "already running" instead of failing loud). + 4. `python scripts/replay_ticks.py --model --interval 2 --loop` (feeds + results/replay/latest_tick.json from the run's tick_snapshots). + Then record: set up a scratch dir once (`bun add playwright && bunx playwright install + chromium`), copy record_dashboard.mjs next to it, and run with **node** — Playwright's + chromium launch hangs 180s under bun on Windows. The script seeds the 5 RLE widgets via + a localStorage `dashboard_presets` preset (fresh browser contexts have no layout, and the + RLE widgets aren't in the default). Record 1920x1080 and 1080x1920 passes; trim the + connect/load preamble (~8s) when converting webm → mp4 + (`ffmpeg -ss 8 -i in.webm -c:v libx264 -crf 19 -pix_fmt yuv420p -r 30 out.mp4`). +- Verify every render before calling it done: extract stills of each segment with ffmpeg + and actually Read them — stale ports, empty widgets, and black OffthreadVideo frames all + look identical to success in the exit code. +- Render 16:9 and 9:16 variants (compositions take portrait into account via useVideoConfig). +- All media outputs to **D:** (`D:\RLE_media\`), never C:. + +## Phase 5 — Publish prep + +- Update memory (`project_benchmark_spread` or successor) with results + artifact locations. +- X thread: 3-5 posts max. Lead with the strongest narrative clip (hero or disaster), data + (leaderboard) in the middle, method/repo last. Draft to `D:\RLE_media\x_thread.md`. +- File issues for every harness bug found before the next run. diff --git a/results/benchmark_history.jsonl b/results/benchmark_history.jsonl index 699a557..b7ea5d9 100644 --- a/results/benchmark_history.jsonl +++ b/results/benchmark_history.jsonl @@ -5,3 +5,9 @@ {"timestamp": "2026-06-10T01:14:38.139936+00:00", "scoring_version": "1.0", "git_commit": "295f478", "git_branch": "fix/strict-action-schema", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 2, "outcome": "timeout", "final_score": 0.8598, "ticks_run": 2, "cost_snapshot": {"total_prompt_tokens": 28, "total_completion_tokens": 20578, "total_tokens": 20606, "estimated_cost_usd": 1.02918, "wall_time_s": 163.26, "num_calls": 14, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 99, "errors_by_type": {}, "avg_deliberation_ms": 25524.75, "action_success_rate": 0.7273, "total_tokens": 20606, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8598, "outcome": "timeout", "ticks": 2}]} {"timestamp": "2026-06-10T01:32:30.462265+00:00", "scoring_version": "1.0", "git_commit": "5d361d0", "git_branch": "master", "git_dirty": false, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7536, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 102045, "total_tokens": 102185, "estimated_cost_usd": 5.10365, "wall_time_s": 897.04, "num_calls": 70, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 633, "errors_by_type": {}, "avg_deliberation_ms": 26282.04, "action_success_rate": 0.7167, "total_tokens": 102185, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7536, "outcome": "timeout", "ticks": 10}]} {"timestamp": "2026-06-10T04:12:15.578574+00:00", "scoring_version": "1.1", "git_commit": "98ee8e2", "git_branch": "fix/run-integrity", "git_dirty": false, "rle_version": "0.1.0", "felix_sdk_version": "0.2.2", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": false, "tick_interval": 30.0, "max_ticks": 2, "outcome": "timeout", "final_score": 0.8594, "ticks_run": 2, "cost_snapshot": {"total_prompt_tokens": 28, "total_completion_tokens": 19000, "total_tokens": 19028, "estimated_cost_usd": 0.95028, "wall_time_s": 151.53, "num_calls": 14, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 96, "errors_by_type": {}, "avg_deliberation_ms": 24304.49, "action_success_rate": 0.7857, "total_tokens": 19028, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8594, "outcome": "timeout", "ticks": 2}]} +{"timestamp": "2026-06-11T01:05:39.907207+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-fable-5", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.6272, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 104867, "total_tokens": 105007, "estimated_cost_usd": 5.24475, "wall_time_s": 911.47, "num_calls": 70, "prompt_price_per_token": 1e-05, "completion_price_per_token": 5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 673, "errors_by_type": {}, "avg_deliberation_ms": 27323.28, "action_success_rate": 0.7188, "total_tokens": 105007, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.6272, "outcome": "timeout", "ticks": 10}]} +{"timestamp": "2026-06-11T01:19:31.805063+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "claude-opus-4-8", "provider": "claude-code", "base_url": null, "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7969, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 140, "total_completion_tokens": 81269, "total_tokens": 81409, "estimated_cost_usd": 2.032425, "wall_time_s": 828.72, "num_calls": 70, "prompt_price_per_token": 5e-06, "completion_price_per_token": 2.5e-05, "pricing_source": "override"}, "event_summary": {"total_events": 651, "errors_by_type": {}, "avg_deliberation_ms": 20362.56, "action_success_rate": 0.76, "total_tokens": 81409, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7969, "outcome": "timeout", "ticks": 10}]} +{"timestamp": "2026-06-11T01:41:06.279109+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "openai/gpt-5.5", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.609, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 195940, "total_completion_tokens": 88922, "total_tokens": 284862, "estimated_cost_usd": 3.64736, "wall_time_s": 1293.37, "num_calls": 69, "prompt_price_per_token": 5e-06, "completion_price_per_token": 3e-05, "pricing_source": "override"}, "event_summary": {"total_events": 988, "errors_by_type": {"provider_error": 1}, "avg_deliberation_ms": 32919.53, "action_success_rate": 0.7077, "total_tokens": 284862, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.609, "outcome": "timeout", "ticks": 10}]} +{"timestamp": "2026-06-11T01:52:11.998324+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "google/gemini-3.5-flash", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.8108, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 194601, "total_completion_tokens": 166376, "total_tokens": 360977, "estimated_cost_usd": 1.789286, "wall_time_s": 664.45, "num_calls": 69, "prompt_price_per_token": 1.5e-06, "completion_price_per_token": 9e-06, "pricing_source": "override"}, "event_summary": {"total_events": 753, "errors_by_type": {"parse_failure": 1}, "avg_deliberation_ms": 15690.77, "action_success_rate": 0.6957, "total_tokens": 360977, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.8108, "outcome": "timeout", "ticks": 10}]} +{"timestamp": "2026-06-11T02:00:51.207361+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "x-ai/grok-4.3", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.7904, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 209815, "total_completion_tokens": 66013, "total_tokens": 275828, "estimated_cost_usd": 0.427301, "wall_time_s": 517.38, "num_calls": 70, "prompt_price_per_token": 1.25e-06, "completion_price_per_token": 2.5e-06, "pricing_source": "override"}, "event_summary": {"total_events": 761, "errors_by_type": {}, "avg_deliberation_ms": 8401.06, "action_success_rate": 0.7647, "total_tokens": 275828, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.7904, "outcome": "timeout", "ticks": 10}]} +{"timestamp": "2026-06-11T02:29:49.825955+00:00", "scoring_version": "1.1", "git_commit": "d0c9a62", "git_branch": "master", "git_dirty": true, "rle_version": "0.1.0", "felix_sdk_version": "0.3.0", "platform": "win32", "python_version": "3.14.0", "docker_mode": false, "random_seed": 42, "rimapi_dll_path": "C:\\Steam\\steamapps\\workshop\\content\\294100\\3593423732\\1.6\\Assemblies\\RIMAPI.dll", "rimapi_dll_sha256": "8b26c3820e37bb21ae7227094a7e84fce52ef1099c913899c39a6c78c8f0f4e1", "rimapi_fork_commit": "", "scenario": "Crashlanded Survival", "scenario_save_name": "rle_crashlanded_v1", "model": "deepseek/deepseek-v4-pro", "provider": "openai", "base_url": "https://openrouter.ai/api/v1", "no_think": false, "parallel": true, "no_agent": false, "no_pause": true, "tick_interval": 30.0, "max_ticks": 10, "outcome": "timeout", "final_score": 0.5959, "ticks_run": 10, "cost_snapshot": {"total_prompt_tokens": 189569, "total_completion_tokens": 121047, "total_tokens": 310616, "estimated_cost_usd": 0.187773, "wall_time_s": 1735.02, "num_calls": 68, "prompt_price_per_token": 4.35e-07, "completion_price_per_token": 8.7e-07, "pricing_source": "override"}, "event_summary": {"total_events": 895, "errors_by_type": {"deliberation_timeout": 1, "parse_failure": 1}, "avg_deliberation_ms": 45005.51, "action_success_rate": 0.7368, "total_tokens": 310616, "estimated_cost_usd": 0.0}, "run_type": "scenario", "scenarios": [{"name": "Crashlanded Survival", "difficulty": "easy", "score": 0.5959, "outcome": "timeout", "ticks": 10}]} diff --git a/scripts/analyze_spread.py b/scripts/analyze_spread.py new file mode 100644 index 0000000..8ca14e1 --- /dev/null +++ b/scripts/analyze_spread.py @@ -0,0 +1,196 @@ +"""Cross-model analysis of the Stage S spread per the run-analysis playbook: +composite vs pinned baseline, action success BY TYPE (harness vs model failures), +parse health, trajectory shape. Writes leaderboard.json + analysis.md. +""" +from __future__ import annotations + +import argparse +import csv +import json +from collections import Counter, defaultdict +from pathlib import Path + +# Failure substrings attributable to the harness / known RIMAPI quirks, not the model. +HARNESS_FAILURE_MARKERS = [ + "Invalid plant definition", # Plant_Rice / Plant_Potato def-name quirk + "invalid literal for int()", # stockpile priority string crash (our executor) + "Object reference not set", # RIMAPI null-ref +] + + +def load_model(d: Path) -> dict: + summary = json.loads((d / "01_crashlanded_survival_summary.json").read_text(encoding="utf-8")) + ticks = [] + with (d / "01_crashlanded_survival.csv").open(encoding="utf-8") as f: + for row in csv.DictReader(f): + ticks.append({ + k: (float(v) if v.replace(".", "", 1).replace("-", "", 1).isdigit() else v) + for k, v in row.items() + }) + by_type: dict[str, Counter] = defaultdict(Counter) + failures = [] + delib = {"count": 0, "zero_action": 0, "latencies": [], "confidences": []} + with (d / "events.jsonl").open(encoding="utf-8") as f: + for ln in f: + e = json.loads(ln) + et, data = e.get("event_type"), e.get("data", {}) + if et == "action_exec": + t = data.get("action_type", "?") + ok = data.get("success") + by_type[t]["ok" if ok else "fail"] += 1 + if not ok: + failures.append({"tick": e.get("tick"), "type": t, + "error": (data.get("error") or "")[:160]}) + elif et == "deliberation": + delib["count"] += 1 + delib["latencies"].append(data.get("latency_ms", 0)) + delib["confidences"].append(data.get("confidence", 0)) + if data.get("num_actions", 0) == 0: + delib["zero_action"] += 1 + + harness_fails = sum(1 for fl in failures + if any(m in fl["error"] for m in HARNESS_FAILURE_MARKERS)) + total_ok = sum(c["ok"] for c in by_type.values()) + total_fail = sum(c["fail"] for c in by_type.values()) + total = total_ok + total_fail + model_fails = total_fail - harness_fails + return { + "summary": summary, + "ticks": ticks, + "by_type": {t: dict(c) for t, c in sorted(by_type.items())}, + "failures": failures, + "deliberation": { + "count": delib["count"], + "zero_action": delib["zero_action"], + "avg_latency_ms": round( + sum(delib["latencies"]) / max(len(delib["latencies"]), 1), 1), + "avg_confidence": round( + sum(delib["confidences"]) / max(len(delib["confidences"]), 1), 3), + }, + "actions": { + "total": total, "ok": total_ok, "fail": total_fail, + "raw_success": round(total_ok / total, 4) if total else None, + "harness_fails": harness_fails, "model_fails": model_fails, + "ex_artifact_success": round(total_ok / (total - harness_fails), 4) + if total - harness_fails else None, + }, + } + + +def load_baseline_by_day(baseline_runs_dir: Path) -> dict[int, float]: + """Day-indexed baseline composite: mean across seeds of per-day mean composite. + + The pinned sidecar trajectory is tick-indexed at ~30 ticks/day while agent runs + tick ~2-3x/day, so tick-alignment compares different game days. Day-align instead. + """ + per_day: dict[int, list[float]] = defaultdict(list) + for csv_path in sorted(baseline_runs_dir.glob("seed*/01_crashlanded_survival.csv")): + seed_days: dict[int, list[float]] = defaultdict(list) + with csv_path.open(encoding="utf-8") as f: + for row in csv.DictReader(f): + seed_days[int(float(row["day"]))].append(float(row["composite"])) + for day, vals in seed_days.items(): + per_day[day].append(sum(vals) / len(vals)) + return {day: sum(v) / len(v) for day, v in sorted(per_day.items())} + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--spread-dir", default="results/spread") + ap.add_argument("--baseline", + default="src/rle/scenarios/definitions/01_crashlanded_survival.baseline.json") + ap.add_argument("--baseline-runs", default="results/baseline/01_crashlanded_survival") + ap.add_argument("--out-json", default="results/spread/leaderboard.json") + ap.add_argument("--out-md", default="results/spread/analysis.md") + args = ap.parse_args() + + baseline = json.loads(Path(args.baseline).read_text(encoding="utf-8")) + base_by_day = load_baseline_by_day(Path(args.baseline_runs)) + + models = {} + for d in sorted(Path(args.spread_dir).iterdir()): + if d.is_dir() and (d / "01_crashlanded_survival_summary.json").exists(): + models[d.name] = load_model(d) + + rows = [] + for name, m in models.items(): + traj = [t["composite"] for t in m["ticks"]] + days = [t["day"] for t in m["ticks"]] + n = len(traj) + max_base_day = max(base_by_day) + base_slice = [base_by_day[min(int(d), max_base_day)] for d in days] + delta = [round(a - b, 4) for a, b in zip(traj, base_slice)] + ticks_above = sum(1 for x in delta if x > 0) + c = m["summary"]["cost_snapshot"] + rows.append({ + "model": m["summary"]["model"], + "name": name, + "final_composite": m["summary"]["final_score"], + "mean_composite": round(sum(traj) / n, 4), + "vs_baseline_mean_delta": round(sum(delta) / n, 4), + "ticks_above_baseline": f"{ticks_above}/{n}", + "end_day": days[-1] if days else None, + "raw_action_success": m["actions"]["raw_success"], + "ex_artifact_success": m["actions"]["ex_artifact_success"], + "harness_fails": m["actions"]["harness_fails"], + "model_fails": m["actions"]["model_fails"], + "zero_action_delibs": m["deliberation"]["zero_action"], + "delib_count": m["deliberation"]["count"], + "avg_latency_s": round(m["deliberation"]["avg_latency_ms"] / 1000, 1), + "avg_confidence": m["deliberation"]["avg_confidence"], + "wall_min": round(c["wall_time_s"] / 60, 1), + "est_cost_usd": round(c["estimated_cost_usd"], 2), + "trajectory": traj, + "days": days, + "baseline_slice": [round(x, 4) for x in base_slice], + }) + rows.sort(key=lambda r: r["mean_composite"], reverse=True) + + Path(args.out_json).write_text(json.dumps( + {"baseline": {"mean_time_to_end_days": baseline["time_to_end_days_mean"], + "n_runs": baseline["n_runs"]}, + "note": ("N=1 content-first spread; not statistically valid. " + "Ranked by mean composite over 10 ticks."), + "rows": rows}, indent=2), encoding="utf-8") + + md = ["# Stage S Spread — N=1 Leaderboard (10 ticks, seed 42)\n", + "Ranked by **mean composite over 10 ticks** (endpoint scores are single-event noisy).", + f"Baseline: no-agent, N={baseline['n_runs']}, mean time-to-end " + f"{baseline['time_to_end_days_mean']}d.\n", + "| # | model | mean comp | final | vs base Δ | ticks>base | end day | act ok (raw) " + "| act ok (ex-artifact) | harness/model fails | 0-act delibs | avg delib s " + "| wall min | $ |", + "|---|-------|-----------|-------|-----------|------------|---------|-------|-------|" + "------|------|------|------|---|"] + for i, r in enumerate(rows, 1): + md.append( + f"| {i} | {r['name']} | {r['mean_composite']:.3f} | {r['final_composite']:.3f} " + f"| {r['vs_baseline_mean_delta']:+.3f} | {r['ticks_above_baseline']} " + f"| {r['end_day']:.0f} " + f"| {r['raw_action_success']:.0%} | {r['ex_artifact_success']:.0%} " + f"| {r['harness_fails']}/{r['model_fails']} | {r['zero_action_delibs']} " + f"| {r['avg_latency_s']} | {r['wall_min']} | {r['est_cost_usd']} |") + md.append("\n## Per-model action success by type\n") + for name, m in models.items(): + md.append(f"### {name}") + for t, c in m["by_type"].items(): + ok, fail = c.get("ok", 0), c.get("fail", 0) + flag = " ⚠ ALL FAILED" if ok == 0 and fail > 0 else "" + md.append(f"- {t}: {ok}/{ok + fail}{flag}") + if m["failures"]: + md.append(f"- failures ({len(m['failures'])}):") + for fl in m["failures"][:8]: + md.append(f" - tick {fl['tick']} {fl['type']}: {fl['error']}") + md.append("") + Path(args.out_md).write_text("\n".join(md), encoding="utf-8") + print(f"wrote {args.out_json} and {args.out_md}") + for i, r in enumerate(rows, 1): + print(f"{i}. {r['name']:<11} mean={r['mean_composite']:.3f} " + f"final={r['final_composite']:.3f} " + f"Δbase={r['vs_baseline_mean_delta']:+.3f} " + f"ex-artifact-ok={r['ex_artifact_success']:.0%} " + f"0act={r['zero_action_delibs']} ${r['est_cost_usd']}") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_footage_index.py b/scripts/build_footage_index.py new file mode 100644 index 0000000..c58d87f --- /dev/null +++ b/scripts/build_footage_index.py @@ -0,0 +1,101 @@ +"""Parse a spread console log into a footage index: +per-model wall-clock ranges, OBS file offsets, per-tick timestamps/scores, warnings. + +Usage: + python build_footage_index.py --log --obs-file \ + --obs-start "2026-06-10 20:55:09" --out footage_index.json +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +from datetime import datetime + +TS = re.compile(r"^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+ (\S+) (\w+) (.*)") +MARKER = re.compile(r"^(>>>|<<<) \[(\w+)\](.*)") +TICK = re.compile(r"Tick (\d+) \(day (\d+)\): (\d+) actions, (\d+) executed \| score=([\d.]+)") + + +def build_index(log_path: str, obs_file: str, obs_start: datetime, msg_limit: int) -> dict: + def obs_offset(dt: datetime) -> float | None: + off = (dt - obs_start).total_seconds() + return round(off, 1) if off >= 0 else None + + models: list[dict] = [] + cur: dict | None = None + last_ts: datetime | None = None + awaiting_start = False + + with open(log_path, encoding="utf-8", errors="replace") as f: + for line in f: + line = line.rstrip("\n") + m = MARKER.match(line) + if m: + kind, name = m.group(1), m.group(2) + if kind == ">>>": + cur = {"model": name, "start": None, "start_obs_offset_s": None, + "end": None, "end_obs_offset_s": None, "status": None, + "ticks": [], "warnings": []} + models.append(cur) + awaiting_start = True + elif kind == "<<<" and cur is not None: + cur["end"] = last_ts.isoformat(sep=" ") if last_ts else None + cur["end_obs_offset_s"] = obs_offset(last_ts) if last_ts else None + cur["status"] = "OK" if "OK" in m.group(3) else m.group(3).strip() + cur = None + continue + t = TS.match(line) + if not t: + continue + dt = datetime.strptime(t.group(1), "%Y-%m-%d %H:%M:%S") + last_ts = dt + if cur is not None and awaiting_start: + cur["start"] = dt.isoformat(sep=" ") + cur["start_obs_offset_s"] = obs_offset(dt) + awaiting_start = False + logger, level, msg = t.group(2), t.group(3), t.group(4) + if cur is None: + continue + tk = TICK.search(msg) + if tk: + cur["ticks"].append({ + "tick": int(tk.group(1)), + "day": int(tk.group(2)), + "actions": int(tk.group(3)), + "executed": int(tk.group(4)), + "score": float(tk.group(5)), + "wall": dt.isoformat(sep=" "), + "obs_offset_s": obs_offset(dt), + }) + elif level in ("WARNING", "ERROR") and "httpx" not in logger: + cur["warnings"].append({"wall": dt.isoformat(sep=" "), "level": level, + "msg": msg[:msg_limit]}) + + return {"obs_file": obs_file, "obs_start": obs_start.isoformat(sep=" "), "models": models} + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--log", required=True, help="spread console log path") + ap.add_argument("--obs-file", required=True, + help="OBS recording path (for reference in output)") + ap.add_argument("--obs-start", required=True, help="OBS recording start, 'YYYY-MM-DD HH:MM:SS'") + ap.add_argument("--out", default="footage_index.json", help="output JSON path") + ap.add_argument("--msg-limit", type=int, default=200, help="warning message truncation length") + args = ap.parse_args() + + obs_start = datetime.strptime(args.obs_start, "%Y-%m-%d %H:%M:%S") + out = build_index(args.log, args.obs_file, obs_start, args.msg_limit) + with open(args.out, "w", encoding="utf-8") as f: + json.dump(out, f, indent=2) + for m in out["models"]: + print(f"{m['model']}: ticks={len(m['ticks'])} status={m['status']} " + f"obs={m['start_obs_offset_s']}-{m['end_obs_offset_s']}s " + f"warnings={len(m['warnings'])}") + print(f"wrote {args.out}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/scripts/capture_run_media.sh b/scripts/capture_run_media.sh new file mode 100644 index 0000000..39337af --- /dev/null +++ b/scripts/capture_run_media.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# During-run media capture loop. Start BEFORE launching the run/spread, stop after. +# - every FRAME_INTERVAL s: one PNG still of the RimWorld window (timestamp-named — +# doubles as a wall-clock index into the OBS recording) +# - every POLL_INTERVAL s: snapshot each model's latest_tick.json when it changes +# (dashboard replay source; the run only keeps the latest tick otherwise) +# +# Usage: +# bash scripts/capture_run_media.sh +# Env overrides: FFMPEG (path to ffmpeg.exe), WINDOW_TITLE, FRAME_INTERVAL, POLL_INTERVAL +set -u +RESULTS_DIR="${1:?usage: capture_run_media.sh }" +FRAMES_DIR="${2:?usage: capture_run_media.sh }" +FFMPEG="${FFMPEG:-ffmpeg}" +WINDOW_TITLE="${WINDOW_TITLE:-RimWorld by Ludeon Studios}" +FRAME_INTERVAL="${FRAME_INTERVAL:-10}" +POLL_INTERVAL="${POLL_INTERVAL:-5}" + +mkdir -p "$FRAMES_DIR" +echo "capturing: window '$WINDOW_TITLE' every ${FRAME_INTERVAL}s -> $FRAMES_DIR" +echo "snapshots: $RESULTS_DIR/*/latest_tick.json every ${POLL_INTERVAL}s" + +last_frame=0 +while true; do + now=$(date +%s) + if [ $((now - last_frame)) -ge "$FRAME_INTERVAL" ]; then + ts=$(date +%Y%m%d_%H%M%S) + "$FFMPEG" -y -loglevel quiet -f gdigrab -framerate 1 -i "title=$WINDOW_TITLE" \ + -frames:v 1 "$FRAMES_DIR/frame_$ts.png" 2>/dev/null + last_frame=$now + fi + for d in "$RESULTS_DIR"/*/; do + f="$d/latest_tick.json" + [ -f "$f" ] || continue + h=$(md5sum "$f" | cut -d' ' -f1) + snap="$d/tick_snapshots" + mkdir -p "$snap" + last="" + [ -f "$snap/.last" ] && last=$(cat "$snap/.last") + if [ "$last" != "$h" ]; then + cp "$f" "$snap/snap_$(date +%s)_$h.json" + echo "$h" > "$snap/.last" + fi + done + sleep "$POLL_INTERVAL" +done diff --git a/scripts/record_dashboard.mjs b/scripts/record_dashboard.mjs new file mode 100644 index 0000000..e2bdca7 --- /dev/null +++ b/scripts/record_dashboard.mjs @@ -0,0 +1,42 @@ +// Record the RLE dashboard replaying a run, via Playwright video capture. +// Usage: node record_dashboard.mjs [width] [height] +import { chromium } from 'playwright'; + +const [url = 'http://localhost:3001', secs = '28', outDir = 'D:/RLE_media/capture/video', + w = '1920', h = '1080'] = process.argv.slice(2); + +const browser = await chromium.launch(); +const ctx = await browser.newContext({ + viewport: { width: Number(w), height: Number(h) }, + recordVideo: { dir: outDir, size: { width: Number(w), height: Number(h) } }, +}); +const page = await ctx.newPage(); +// Seed an RLE-focused widget layout so the 5 RLE widgets render on load. +const rlePreset = [{ + name: 'RLE', + layout: [ + { i: 'rleAgentStatus', x: 0, y: 0, w: 4, h: 2, isBounded: true }, + { i: 'rleScoreTimeline', x: 4, y: 0, w: 8, h: 2, isBounded: true }, + { i: 'rleAgentLog', x: 0, y: 2, w: 6, h: 3, isBounded: true }, + { i: 'rleHelixPhase', x: 6, y: 2, w: 3, h: 3, isBounded: true }, + { i: 'rleConflictResolution', x: 9, y: 2, w: 3, h: 3, isBounded: true }, + ], + cardSettings: {}, +}]; +await page.addInitScript((preset) => { + localStorage.setItem('dashboard_presets', JSON.stringify(preset)); + localStorage.setItem('last_selected_preset', 'RLE'); +}, rlePreset); +await page.goto(url, { waitUntil: 'networkidle' }); +// Connect screen: fill the server URL, then connect. +const urlInput = page.locator('input').first(); +if (await urlInput.isVisible().catch(() => false)) { + await urlInput.fill('http://localhost:8765/api/v1'); + await page.getByRole('button', { name: /^connect$/i }).first().click(); + await page.waitForTimeout(5000); +} +await page.waitForTimeout(Number(secs) * 1000); +const path = await page.video().path(); +await ctx.close(); +await browser.close(); +console.log('saved:', path); diff --git a/scripts/replay_ticks.py b/scripts/replay_ticks.py new file mode 100644 index 0000000..745d0f2 --- /dev/null +++ b/scripts/replay_ticks.py @@ -0,0 +1,53 @@ +"""Replay a spread model's tick snapshots into a directory served by +serve_dashboard.py, so the React dashboard animates the run for screen recording. + +Usage: + python replay_ticks.py --model fable5 [--interval 2.0] [--loop] +""" +from __future__ import annotations + +import argparse +import json +import shutil +import time +from pathlib import Path + + +def snap_sort_key(p: Path) -> int: + try: + return int(json.loads(p.read_text(encoding="utf-8")).get("tick", 0)) + except (json.JSONDecodeError, OSError): + return 0 + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--model", required=True, help="model dir name under --spread-dir") + ap.add_argument("--spread-dir", default=r"c:\Users\redmo\Projects\RLE\results\spread") + ap.add_argument("--replay-dir", default=r"c:\Users\redmo\Projects\RLE\results\replay") + ap.add_argument("--interval", type=float, default=2.0, help="seconds per tick") + ap.add_argument("--loop", action="store_true", help="repeat forever") + args = ap.parse_args() + + snaps_dir = Path(args.spread_dir) / args.model / "tick_snapshots" + snaps = sorted(snaps_dir.glob("snap_*.json"), key=snap_sort_key) + if not snaps: + raise SystemExit(f"no snapshots in {snaps_dir}") + out = Path(args.replay_dir) + out.mkdir(parents=True, exist_ok=True) + target = out / "latest_tick.json" + + print(f"replaying {len(snaps)} ticks of {args.model} -> {target} " + f"@ {args.interval}s/tick{' (loop)' if args.loop else ''}") + while True: + for s in snaps: + shutil.copyfile(s, target) + tick = json.loads(s.read_text(encoding="utf-8")).get("tick") + print(f" tick {tick}") + time.sleep(args.interval) + if not args.loop: + break + + +if __name__ == "__main__": + main() diff --git a/scripts/run_spread_n1.sh b/scripts/run_spread_n1.sh new file mode 100644 index 0000000..480e9e5 --- /dev/null +++ b/scripts/run_spread_n1.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Stage S: N=1 content-first model spread on Crashlanded, 10 ticks, seed 42. +# Sequential (one live game). Continue-on-error so one model can't block the rest. +set -u +cd "$(dirname "$0")/.." +PY=./.venv/Scripts/python +export ROLE_TIMEOUT_S=180 +export PYTHONIOENCODING=utf-8 + +# name | provider | model | prompt$/MTok | completion$/MTok | base-url(optional) +RUNS=( + "fable5|claude-code|claude-fable-5|10|50|" + "opus48|claude-code|claude-opus-4-8|5|25|" + "gpt55|openai|openai/gpt-5.5|5|30|https://openrouter.ai/api/v1" + "gemini35|openai|google/gemini-3.5-flash|1.5|9|https://openrouter.ai/api/v1" + "grok43|openai|x-ai/grok-4.3|1.25|2.5|https://openrouter.ai/api/v1" + "deepseekv4|openai|deepseek/deepseek-v4-pro|0.435|0.87|https://openrouter.ai/api/v1" +) + +echo "=== STAGE S SPREAD START ===" +for entry in "${RUNS[@]}"; do + IFS='|' read -r name provider model pin pout baseurl <<< "$entry" + out="results/spread/${name}" + echo "" + echo ">>> [$name] provider=$provider model=$model" + args=(scripts/run_scenario.py crashlanded + --provider "$provider" --model "$model" + --ticks 10 --seed 42 --tick-interval 30 --no-pause --visualize + --output "$out" + --prompt-price-per-mtok "$pin" --completion-price-per-mtok "$pout") + [ -n "$baseurl" ] && args+=(--base-url "$baseurl") + "$PY" "${args[@]}" + rc=$? + if [ $rc -eq 0 ]; then echo "<<< [$name] OK"; else echo "<<< [$name] FAILED rc=$rc (continuing)"; fi +done +echo "" +echo "=== STAGE S SPREAD DONE ==="