perf(extract_llm): economic gate, global result cache, and the honest verdict (#28, rebased)

[OsherElhadad]

Closes #28. Supersedes #34 (rebased; force-push blocked by branch protection).

extract_llm is the only component that spends money to save money, and on Terminal-Bench it lost: 271 calls, $3.26, 1,592,467 ms of added latency against 197,548 unique tokens saved — 82× underwater once those tokens are priced at the rate they would actually have been billed. ~93% of its realized value came from the replay cache, not the LLM.

The cause is arithmetic. A request to a caching backend is ~99.95% cached, so a token removed from a cached region saves the cache-read rate ($0.30/MTok), not the fresh-input rate ($3/MTok) — a 10× haircut.

What's implemented

A. Prompt-cache the preamble — and the measurement that matters. The ~1,463-token invariant contract now goes in a stable system block with cache_control (a leading system message on the OpenAI backend, which has no explicit breakpoints). Measured against the gateway, this is INERT on claude-haiku-4-5, whose minimum cacheable prefix is 4096 tokens — a sub-minimum breakpoint is silently ignored:

Prefix	Model	Result
~1.5k	claude-haiku-4-5	write=0 read=0 — inert
~4.5k	claude-haiku-4-5	write=5401 then read=5401
~1.5k	claude-sonnet-5	write=2653 then read=2653

Shipped because it is free and correct where it wins, but /stats exposes prompt_cache_read_tokens so nobody infers a cache win from placement.

B. Reusing the agent's cached prefix — PROTOTYPED AND REJECTED, with numbers. It works mechanically (a 103,019-token prefix read from cache, no write, no invalidation) but cache-read is cheap, not free:

Prefix size	Cost of one extraction	vs a dedicated call ($0.004)
103,019 tok	$0.03398	8.5×
500,000 tok	$0.15307	38.3×
1,700,000 tok	$0.51307	128.3×

At ~1.7M contexts with 4 concurrent calls/turn that is ~$2.04/turn vs ~$0.016. Plus: risks a cache-write on the agent's own prefix (11.5× a read), and couples the compaction model to the agent model.

C. Global content-hash result cache. Re-keyed on sha256(content + prompt version + model + config fingerprint) with no session prefix — 82 of 103 unique contents recurred across sessions. A version/model/config change misses rather than serving stale. Old entries inert once (noted); a session-scoped entry is honored as a migration read. Contrast #27's deliberately session-scoped index: reference vs derived result.

D. Economic gate (on by default). expected_saving = removed × (1 + expected replays) × per-token value vs the analytic size-aware cost of a call. Per-token value from Ctx.CacheAware; ratio learned from results; cost from real pricing (CHEAP_MODEL_PRICE_*), never a constant. Every decision records a reason.

E. Derived triggering. Context pressure + growth rate replace min_tokens, which stays honored when set explicitly. A merely growing context no longer fires every step.

F. Metrics. /stats gains an extract block: calls, calls avoided, calls suppressed, prompt-cache behavior, cost, gross value, and NET value after cost — the honest headline, previously impossible to see because cost lived in a field separate from savings. Purely additive; every existing key keeps its name so deploy/harbor/*.py keeps parsing.

Two deadlocks found by benchmarking

Both the same shape — a pessimistic estimate suppressing every call, so nothing was observed, so the estimate could never correct:

1. Flat per-call cost priced everything at the $0.012 average (~5× true cost on small outputs) and suppressed everything. Actual: $0.0024/call. Cost is now analytic and size-aware, correct on the first call with zero observations.
2. Compression ratio defaulted to 0.45; measured reality is ~0.12 (31–254 tokens removed per call on 400–2,000-token outputs). Corrected — and a bounded 3-call exploration budget now lets the tracker learn whether a workload actually compresses.

Also fixed: /compact hard-coded the context window as unknown, silently disabling every fraction-based trigger and the new pressure logic on that path, so offline eval measured a different component than ships.

Benchmark results

Replay of real captures, aws/claude-haiku-4-5. forced = pre-#28, gated = new default.

Terminal-Bench (20 requests):

Arm	Backend	Calls	Cost	Saved	NET	Avg latency
forced	caching	4	$0.0086	1,220	−$0.0082	9,726 ms
gated	caching	3	$0.0065	0	−$0.0065	6,966 ms
forced	non-caching	6	$0.0109	6,664	+$0.0091	10,287 ms
gated	non-caching	7	$0.0232	17,286	+$0.0287	4,654 ms

SWE-bench (19 requests):

Arm	Backend	Calls	Cost	Saved	NET	Avg latency
forced	caching	2	$0.0098	793	−$0.0095	8,173 ms
gated	caching	2	$0.0092	806	−$0.0089	5,060 ms
forced	non-caching	13	$0.0593	726	−$0.0571	5,150 ms
gated	non-caching	2	$0.0090	2,451	−$0.0016	3,754 ms

The verdict — stated plainly

On a non-caching backend the gate is a strict improvement, but not a profit. The tables above were produced before the H3 attribution fix, when /stats credited the whole pipeline's savings to this one component. With extract-scoped attribution the best available result is break-even, not the reported +$0.0287. What survives the correction is real and worth having: 68% less waste on Terminal-Bench, 26 calls → 1 on SWE-bench, latency roughly halved. A genuine positive result needs outputs above ~1,800 tokens on non-caching traffic, which neither capture contains.

That attribution error is the third benchmark result in this PR traced to a measurement artifact, alongside the flat $0.012/call cost and the 0.45 compression-ratio prior. All three pointed the same direction — making the component look better than it is.

On a caching backend extract_llm is still net-negative even after #28. The gate reduces the loss and cuts latency ~30%, but it cannot make the component pay when saved tokens bill at the cache-read rate and break-even is ~30,500 tokens/output against a largest-observed output of 2,053. Recommendation: disable it by default on caching backends — prefer codesafe (no LLM pass) or drop extract_llm from codesmart for caching traffic. This is documented as the verdict rather than dressed up as a win.

Break-even figures are pinned by test, so drift fails CI instead of silently invalidating the documented verdict.

Update: the external arm re-derived the economics, and it is worse

The final Terminal-Bench arm running on merged main excluded extract_llm entirely and independently re-derived its economics with a harsher result:

Quantity	Value
Unique tokens saved	197,548
Where they sat	the cached prefix — so they bill at the cache-read rate
Their value at cache-read	$0.0395 (197,548 × $0.20/MTok)
Cost	$3.26
Blocking time	1,592,467 ms
Ratio	82× underwater

The improvement plan's ~8× implicitly priced those tokens as fresh input. They were never fresh input: every one of them was inside the cached prefix, so the correct rate is the cache-read rate and the honest ratio is an order of magnitude worse. The 1,592,467 ms of blocking time is not priced at that ratio at all.

Priced instead at the $0.30/MTok sonnet-class rate the gate itself uses, the tokens are worth $0.0593 and the ratio is 55×. Both figures are the same conclusion; neither is within two orders of magnitude of paying for $3.26. The gate reasons with the more generous of the two, so the shipped decline is the conservative one.

This PR's verdict was right, and understated. Nothing in the recommendation changes — the component is already hard-declined on caching backends — but the docs now carry the corrected arithmetic instead of the 8× figure.

Cache-safety: ordering of the two result-cache reads

Restored during the rebase, and the reason it matters:

• A same-session replay is safe at any depth, cached prefix included. This session already sent those compacted bytes, so the provider's cached prefix holds the compacted form and replaying is byte-identical.
• A cross-session global hit has no such guarantee. The receiving session never compacted that content, so its cached prefix holds the original. Splicing another session's result at depth mutates the cached prefix and forces a suffix re-write at 11.5× the read price — exactly the harm fix(cache): slide the store TTL, pin frozen decisions, and repair lost ones #40 removed repairLostResult to prevent.

So the order is: getResult (depth-safe) → tail gate → getResultGlobal, and only the same-session path may bypass the gate. A global hit is treated as a new decision: tail-only, then frozen into the session so later turns replay it from the depth-safe path. TestGlobalCacheHitIsNotSplicedAtDepth fails if the ordering is flattened — verified by flattening it and watching it fail, then restoring. The invariant is named at the call site so the next person does not have to rediscover it.

Rebase onto current main

This branch is a rebase of #34 onto main at 5768b00, replayed across nine merges (#33, #36, #40, #41, #42, #43, #44, #49, #50). Force-pushing the original branch is blocked by branch protection, so this supersedes #34 — same pattern as #37 → #42. Two merges required adaptation:

• fix(cache): slide the store TTL, pin frozen decisions, and repair lost ones #40 unified the result-cache key. getResult/getResultGlobal now return (cachedResult, bool) carrying {Projected, Summary} in one JSON value, and putResult/putResultGlobal take four arguments; getSummary/putSummary/summaryKey and their global twins are gone. Adopted on both the session and global paths. The comments record why: the split cg:res: + cg:sum1: keys had independent TTLs and pin slots, so a replay could hit one and miss the other and emit half a decision — projected text with the summary segment silently gone. Splitting the pair across two global keys would re-create that bug cross-session, so the global namespace keeps them together too.
• feat(proxy): add observe mode — measure what compaction would save, without enforcing it #43's observe-mode /stats block and this PR's extract block both extend the same function; both are additive and both are kept.

Shipping guard, re-verified

AllowOnCachingBackend (allow_on_caching_backend) and the hard decline in evaluateGate survived the rebase intact, and there is now an end-to-end test that fails if either is ever lost: TestNoDefaultConfigRunsExtractLLMOnCachingBackend drives the bare defaults and the codesmart preset config against a ~240k-token output — far above the ~30,500-token cached break-even, so the economics alone would permit the call and only the hard decline can suppress it. Mutation-checked: forcing allowCached = true at construction makes it fail. The escape hatch is asserted too, so the guard is a default and not a wall.

Verification

• CGO_ENABLED=1 go build -tags cg_skeleton ./... ✅
• CGO_ENABLED=1 go test -tags cg_skeleton ./... ✅
• go test -race ✅ (the component runs concurrent per-output calls)
• make lint ✅
• go test -race -count=2 ./components/... ./proxy/... ✅
• gofmt -l clean · go vet ./... ✅
• Tests: cached-system-block shape on both backends, cross-session cache reuse, version bump misses rather than serving stale, gate suppression when cache-aware, permission on non-caching and for recurring content, cost model vs known tokens × known price, documented break-even sizes, trigger not firing every step, and regressions for both deadlocks.
• Added on the rebase: TestNoDefaultConfigRunsExtractLLMOnCachingBackend (the shipping guard, end to end through the presets). TestGlobalCacheHitIsNotSplicedAtDepth confirmed to fail when the read ordering is flattened.

Docs

docs/components/extract_llm.md rewritten (economics, triggering, caching, measured tables, honest verdict), docs/reference/config.md (pricing env), docs/reference/presets.md (codesmart/aggressive behavior change + verdict).

Assisted-By: Claude Opus 5

[OsherElhadad]

Verified independently before merge

Ordering fix confirmed by reading the merged tree, not just the description — components/offload/extract_llm.go: same-session getResult at :343, tail gate at :365, getResultGlobal at :403, i.e. inside the gate. That is the invariant my resolution flattened.

Both guard tests mutation-verified myself. The AllowOnCachingBackend one is the important addition, because it closes the exact gap that let me nearly ship a regression:

allowCached: allowCached || true    // simulate losing the guard in a rebase
--- FAIL: TestNoDefaultConfigRunsExtractLLMOnCachingBackend/codesmart

Your diagnosis of why the old coverage was insufficient is right and worth restating: evaluateGate was unit-tested directly, so the gate logic was proven while the wiring to it was not. I lost AllowOnCachingBackend in my own rebase attempt and nothing failed. Driving the bare defaults and the codesmart preset end to end with a ~240k-token output — above the break-even, so economics alone would permit the call and only the hard decline can suppress it — is exactly the right shape for that test.

Also good catch that the rebase was never actually finished: the branch sat on #42 while main had moved five merges ahead. I'd assumed it was current.

Full gates re-run on the final tree: build, go test -tags cg_skeleton ./... (20/20), gofmt -l clean.

On your two judgement calls — both right

The rate inconsistency: documenting both is the correct choice. 82× at $0.20/MTok, 55× at $0.30/MTok, with the gate reasoning on the more generous rate so the shipped decline is the conservative one. Silently mixing them would have been the failure mode; picking the bigger number for the headline would have been worse. Either figure is an order of magnitude past break-even, so the conclusion doesn't depend on the choice — which is itself worth saying, because a conclusion that survives both plausible parameterisations is stronger than one tuned to a rate.

Replacing the stale "clear win: net +$0.0287" was right, not overreach. That figure was produced by the H3 attribution bug and I'd already retracted it in the fix-round comment; leaving it in the body while the doc said otherwise would have shipped a contradiction. Marking it as the third result traced to a measurement artifact is the honest framing.

That count now matters beyond this PR: three of this component's results were measurement artifacts, and all three pointed the same way — making the component look better than it is. That pattern is why the final Terminal-Bench arm excluded extract_llm outright rather than trusting the gate to suppress it.

What this PR now establishes

On prompt-caching backends extract_llm cannot pay for itself — break-even needs ~30,500 tokens/output against a largest-observed 2,053, a 15× structural gap — and it now makes zero calls and costs nothing there, enforced in code rather than a doc note. On non-caching traffic the gate is strictly better in every arm (68% less waste, 26 calls → 1, latency halved) but still only reaches break-even on available captures.

A negative result, honestly measured, with the component disabled where it loses money and instrumented so the next person can tell. Merging.