docs(benchmark): Terminal-Bench 2.0 study + the merged-system fifth arm

[OsherElhadad]

What

The second benchmark of the study, after SWE-bench Verified: Terminal-Bench 2.0, 89 open-ended terminal tasks, claude-code on aws/claude-sonnet-5, run live through the harness. Same four arms as SWE — baseline (off passthrough), context-guru (codesmart), headroom (hd-cache), rtk.

The claude-code trajectory parser, the cache-aware cost model and the summarizer are agent-specific rather than benchmark-specific, so every number is computed identically to the SWE arms and the two benchmarks are directly comparable in methodology. Only the Harbor dataset and jobs-root differ.

Headline (89-task raw)

dimension	baseline	context-guru	headroom	rtk
solved / 89	56 (62.9%)	58 (65.2%)	64 (71.9%)	55 (61.8%)
total billed cost	$100.81	$102.55	$114.75	$118.83
cache-read tokens	216.0M	204.6M	198.0M	254.0M
cache-write tokens	4.01M	6.53M	12.37M	7.05M
mean steps (completed)	31.5	34.7	32.0	40.0

The correction, stated up front

Six baseline trials are degenerate — the baseline aborted in 2–6 steps (16–800 s) where the compaction arms ran 50–160 steps, so the cost delta on those six measures the baseline not doing the work. extract-moves-from-video alone is $24.10 of headroom's $24.65 "regression".

Recomputed over the 83 clean tasks: context-guru −9.7%, headroom −16.0%, rtk +6.4%. Both proxies save on Terminal-Bench; only rtk regresses. The comparison page opens with this caveat and the six tasks are queued for re-run at low concurrency; the tables will be regenerated then.

What the study establishes

• The agent is ~98% cached on TB too, so cache-read is again the largest cost term — but cache-write, a rounding error on SWE-bench, is the deciding term on TB's ~1.7M-token contexts.
• Cost tracks agent steps, r = 0.95 across every arm and both benchmarks.
• One cache-write costs 11.5 cache-reads — so mutating already-cached content is the expensive mistake, not sending too many tokens.
• Unique token removal is 0.02–0.13% of the billed total. Content compaction is not where the money is.
• cache_control is metadata, not hashed content — moving a breakpoint is free and lossless.

docs/results/improvement-plan.md carries that synthesis and the prioritized work it implies.

Also in this PR

swebench.py: captures and dumps move from a fixed /tmp path to the run's own jobs-root. start_proxy unlinks the capture path, so launching any new run truncated the capture an earlier analysis was computed from — which is how a 472-request capture was lost mid-analysis. Adds the cacheonly arm, which removes no content tokens at all, so any cost delta vs off is purely breakpoint placement.

Reproduction

docs/results/REPRODUCE.md §7 (baseline: task-list generation, n=24 concurrency feasibility on this box, the timeout/budget methodology, task-metadata extraction) and §7b (the three framework-arm commands). Every credential is read from the environment at run time; nothing is embedded.

Verification

• CGO_ENABLED=1 go build -tags cg_skeleton ./... and go test -tags cg_skeleton ./... green (docs/harness only — no library change).
• All 89 tasks carry a scored outcome; timeouts count as reward-0 failures rather than being dropped.

---

Fifth arm added (2026-08-10) — supersedes #23

This branch is #23 rebased onto main after all fifteen code PRs landed, plus the re-measurement of the merged system as a fifth arm. Branch protection blocks force-push, hence the new branch.

The merged system on Terminal-Bench (81 clean tasks)

	baseline	context-guru (old)	context-guru (merged)
solved	53	55	61
cache-write / cache-read	1.86%	2.86%	1.86%
own LLM cost	$0	$2.97	$0
added latency	—	449.8 ms	38.5 ms
total billed	$94.85	$85.72 (−9.6%)	$79.32 (−16.4%)

Read the median, not the aggregate. −16.4% is dominated by path-tracing alone (−9.1% without it; an independent re-derivation with a stricter degenerate rule gave −13.7% → −2.8% on the same exclusion). The median per-task ratio is −7.8% with 49/81 tasks cheaper. Both are published because they differ by 9 points and answer different questions.

The one result needing no caveat: cache-write/cache-read returns to 1.86%, identical to baseline, where the previous arm ran 2.86%. That is the "cache-write tax" this study named as the deciding term on Terminal-Bench, eliminated — and being a ratio rather than a sum, it holds under every exclusion rule tried.

What could not be verified, stated as such

#40 (freeze TTL) is NOT exercised by this arm. All five frozen_* counters are zero, because its only callers are mask, failed_run and extract_llm — all three excluded by the chosen config. The arm is not evidence for or against #40 in either direction, and none of the cost improvement is credited to it.

cachesplit has zero legal opportunity on TB. The Claude Agent SDK never appends the git/env snapshot the CLI does, so all 73 captured requests carry 3 system blocks and no volatile-tail marker. The −34.1% split figure came from SWE-bench CLI traffic and does not transfer.

Regressions published rather than omitted

system-administration +17.2% cost and −2 solved (the only category losing both ways) · security +25.6% · fresh_input 3.8× baseline · small tasks inflating up to +311% at n=1, so size-gating remains an unclaimed win. And the honest framing: the merged arm's raw model cost nearly ties the old one and its cache-read is higher — it wins mainly by not spending $2.97 on haiku.

The methodological rule this study produced

improvement-plan.md §F-1 now records eight wrong premises, each with what was believed and what was true. Two were the orchestrator's, and one of those was an unfavourable number accepted without checking its mechanism — which reframes the bias as incuriosity rather than optimism. Nine countermeasures, the most useful being: group request lineages by append-only prefix match; instrument "did the component act?" separately from "did the metric improve?"; verify the verifier; and a sum over heterogeneous tasks can be one task.