[MaxGhenis]

Follow-up to the latency replay audit (#228). That audit found 328 households in the frozen US run that are full disk-cache replays and 33 that are partial replays, and that their rows record the cache lookup as elapsed_seconds. The spend ledger already records cache_hit per call, but prediction rows did not, so analysis could not tell a replay from a call. Now every prediction row records where its responses came from.

The fields

Column	Meaning
live_calls	responses behind the row that the provider answered
cache_hit_calls	responses behind the row that the LiteLLM disk cache replayed
live_elapsed_seconds	the live calls' share of elapsed_seconds

The counts cover the request group whose time the row's elapsed_seconds apportions. For run_no_tools_eval that is the whole household: every chunk, repair round, budget escalation and explanation repair. For single-output runs and row repairs it is the one output, and for batch rows the one request. Every row of a group repeats the counts. live_elapsed_seconds is divided among the rows like elapsed_seconds. All three come from _response_cache_hit, the flag the spend ledger already used. LiteLLM sets _hidden_params["cache_hit"] = True on every disk-cache replay (litellm/caching/caching_handler.py, read for this PR).

Every writer

• Sync (eval_no_tools): _request_predictions_once and _request_explanations_once record each answered request. _aggregate_request_results and the chunked branch of run_single_no_tools sum them. Both row builders write them: household rows divide the live time by batch_size, and single-output rows write it unchanged. Their exception rows record zero calls. Unknown in means unknown out: a result without provenance makes the aggregate empty rather than undercounted.
• Batch (batch_eval.rows_from_unit): cache_hit_calls is 0, because batch requests never read the disk cache. live_calls is 1 when a response answered the row, and the live time is empty, like elapsed_seconds.
• Row repair: repaired rows carry their own request's provenance, and its exception row records zero calls. _append_rows used to append under whatever header the file had. It now widens an older header first and keeps every existing cell's text, so a repair round resumed across this change cannot shift cells. It also replaces a headerless empty file, which it previously appended to without a header. A merge that applies an older repair, written without the columns, empties them on the row it replaces: the source's values described the replaced request and could otherwise exceed the repair's time.
• Retry (retry_eval) and the merges (chunked_eval, fold_board, supervisor.combine) concatenate frames, so the columns pass through and older rows stay empty. Tests cover the retry and row-repair merges.
• Run store: the columns are response-level (_RESPONSE_COLUMNS). Their exact text is stashed for a byte-identical export, and typed copies go in responses.usage_json for SQL queries. A CSV written before the columns existed exports without them, byte for byte.
• Supervisor: _scenario_complete still accepts a scenario file that predates the columns, so a supervised run resumed across this change does not refuse its finished scenarios. A file with only some of the three columns is rejected.
• Analysis: call_provenance.household_call_provenance classifies each household as live, partial_replay, full_replay, no_response or unknown, with its summed live time. It is a new module, so this PR doesn't change analysis.py, whose hash the paper prints as the scoring code. Against the branch's base the file differs only by main's Give a model with no recorded tokens no cost, not $0 #230 cost change, which the merge brings in.

Nothing published changes

latencySeconds, model_cost_latency and usage_summary_by_model are unchanged, and nothing published reads the new fields. A property test checks that adding the columns leaves both functions' output identical. Whether replays leave the median is decision d1542, which is still open, so it is not implemented here. The frozen board predates the fields, so all of its households classify as unknown, and a test reads the snapshot header to confirm it.

Invariants

These hold for every row a writer emits. tests/test_call_provenance.py executes each one:

1. live_calls and cache_hit_calls are non-negative integers. Their sum is every response behind the row's time, live or replayed.
2. 0 <= live_elapsed_seconds <= elapsed_seconds, and the two are empty together.
3. A row whose responses were all replays has live_calls == 0 and live_elapsed_seconds == 0 exactly.
4. A row with no replays has live_elapsed_seconds == elapsed_seconds exactly. Both sums run over the same sequence, and float addition and division are monotone, so 2 to 4 hold exactly, not within a tolerance.
5. The counts and times agree with the requests actually made. The test's oracle is the mock's own call history, and the spend ledger must record every one of those requests. When none of a group's requests failed, the row counts every returned response: its counts equal the replayed and live returns, its live time their live seconds, and the ledger's elapsed times give the same sums. When a request failed, a failed chunk or household keeps none of its time, so the row equals some subset of the returned responses, counts and times together, and never more. That subset check runs on every example. A meet-in-the-middle search over the distinct durations (equal durations are interchangeable) finds the candidate sums in windows of twice the tolerance, and each candidate is then judged by the same pytest.approx comparisons as the brute-force combinations definition. A differential property checks the two agree. It discards inputs where some choice's exact sum is within rounding of a tolerance's edge, because there the order the seconds are added in can decide the comparison, and the two searches add them in different orders; tests pin the edge itself. An example too large for the search is discarded with hypothesis.reject(), never passed; none were in the runs below.
6. Batch rows never record a cache hit and carry no live time.
7. Run-store and CSV round trips are byte-identical with the columns. Without them, the CSV round-trips byte-identical and stays without them.
8. Adding the columns leaves model_cost_latency and usage_summary_by_model identical.

Intended non-invariant: the counts are not additive across rows (each row of a group repeats them). live_elapsed_seconds is additive.

How the end-to-end property runs

test_rows_record_each_responses_provenance_and_agree_with_the_ledger drives run_no_tools_eval and run_no_tools_single_output_eval through the real request path: _request_predictions_once, escalation, repair rounds, explanation repair, retries, partial rounds, chunking, and failed chunks and households. Only litellm.completion, litellm.responses and the clock are mocked. The models cover chat completions with one output per request (Claude Sonnet 4.6) and unchunked (Claude Opus 4.6), and the Responses API unchunked (GPT-5.4) and three outputs per request (GPT-5.5). Hypothesis draws each response's cache flag, duration and shape: an answer to the requested tool (submit_outputs or submit_explanations), values only, nothing, a length cut, or a failure. Over 100 examples it produced partial replays (53%), full replays (35%), live households (21%) and households with no response (2%). A request failed in 43% of examples, 43% used the Responses API, and 13% answered an explanation repair. Each example checks invariants 1 to 5 and 7 on the rows, the written CSV and a run-store round trip. In the store, each prediction is joined to its own response (source_attempt = attempt), and that response's typed live_calls, cache_hit_calls and live_elapsed_seconds in usage_json must equal that output's CSV row exactly, as the store's reader parses it; an empty live time must be left out. test_responses_api_requests_record_replays_and_live_calls pins a Responses API replay followed by a live repair.

Mutation check

Twenty-two mutants, applied one at a time, each break one rule: a replay keeping its time or counting as live, the aggregate counting only one response or ignoring an unknown one, the chunk path dropping provenance, a failed chunk turning provenance unknown, live time not being split, explanation repairs losing provenance or hiding a replay, answer replays going unseen, a failed household turning unknown, batch rows counting a hit, repair rows dropping counts, _append_rows ignoring the header, the legacy-repair merge keeping the source's values, the run store skipping the typed copies, the supervisor requiring the columns or accepting some of them, and the classifier misreading mixes and unknowns. All 22 are killed (~/reviews/policybench-row-cache-hit-calls/mutants-r2.log). Two first survived and led to tests: the supervisor requiring the columns (test_scenario_written_before_call_provenance_is_still_complete) and explanation replays going unseen (test_an_explanation_repair_records_its_own_provenance, which kills it on its own).

Review r2's follow-ups added five more mutants, all killed (~/reviews/policybench-row-cache-hit-calls/r3/mutants-r3.log): live_elapsed_seconds dropped from the run store's typed fields, and the subset oracle ignoring multiplicity, the replay count or the live sum, or passing large inputs again. The live-sum mutant first survived; the differential property then gained targets with the total right but the live time wrong.

Review r3's follow-ups added six more, all killed (~/reviews/policybench-row-cache-hit-calls/r4/mutants-r4.log):

• the subset search trusting its float window for the live time, or for the total;
• its windows cut back to one tolerance;
• every prediction linked to its scenario's first response;
• each response's typed fields taken from its scenario's first response while its exported text stays right. This is the reviewer's case, and the round-3 tests pass it under three seeds (r4/swap-mutant-old-vs-new.txt);
• live_elapsed_seconds dropped from the typed fields again.

The one-tolerance mutant first survived. A search (r4/window_search.py) then found sums it drops: the lower bound for the second of two durations, computed as the window's lower edge less the first, rounds above it, though the two still add up to within tolerance. Two such cases are now pinned.

Review

Round 1 (GPT-6.1 Sol at ce67c35) asked for changes: an older repair file kept the source row's provenance (blocking, fixed above); invariant 5 was overstated and failures were untested; successful explanation repairs and the Responses API were uncovered. All three are addressed at 182decf.

Round 2 (GPT-6.1 Sol at 182decf) approved, with two non-blocking test gaps, both closed at 4c323c4:

• The subset oracle returned True above 200,000 combinations, so 28 one-second live responses kept as "20 calls, 19 seconds" could pass. It is now the subset search described under invariant 5, and test_the_subset_search_checks_what_combinations_could_not pins that case.
• The SQL check skipped the typed live_elapsed_seconds. test_run_store_round_trips_provenance_byte_for_byte now asserts it against the repo's own CSV reader (pandas' default parser, which can land an ulp from the written float) and closely against the drawn value, with explicit non-null examples.

Main is merged in (f21b292). At the merged head the new end-to-end assertion found its own over-strict expectation: the single-output writer stores one response per output, so a scenario can hold a response with no live time next to one with a live time. cdf7975 then let each stored value match any of the scenario's rows, as the count checks did. That was the test, not the writers.

Round 3 (GPT-6.1 Sol at cdf7975) approved, with three non-blocking follow-ups, all closed at d8abc3f:

• The subset search wasn't exactly pytest.approx. A 2.000001-second live response is 1.000000000139778e-6 from a row's 2.0 in binary, past the 1e-6 tolerance, but lay inside the window as floats computed it. The search now checks each candidate with pytest.approx, and the PR no longer calls it exact (invariant 5). test_the_subset_search_decides_the_tolerance_edge_as_approx_does pins both sides of the edge for the live time and the total.
• The SQL check lost which response belonged to which output. Typed times swapped between outputs passed, and one untimed output excused a missing time on another. The check now joins each prediction to its own response (above).
• Evidence wording. The seed runs are now of the whole provenance file, and the analysis.py sentence names the comparison base.

Round 4 (GPT-6.1 Sol at d8abc3f) approved, with three non-blocking wording corrections, all made in b6d13931 (a comment and a docstring in tests/test_call_provenance.py, and this body):

• 2.000001 - 2.0 is 1.000000000139778e-6, not 1.0000000000139778e-6.
• In the two pinned window cases, the lower bound for the second duration rounds above it (7.6204078397967105 > 7.62040783979671, 482.7581355537797 > 482.75813555377965), not below.
• Round 2's entry no longer calls the subset search exact.

Tests

• PYTHONHASHSEED=0 pytest -m "not slow" -n 4 at 182decf: 5169 passed, 58 skipped. Without the pinned seed, xdist workers collect tests/test_reference_audit_pins.py in different orders, because it parametrizes over frozensets; that predates this PR, CI runs without -n, and a separate task fixes it.
• At the merged head cdf7975 (~/reviews/policybench-row-cache-hit-calls/r3/), the slow payload checks give 4 passed, 6 skipped (slow-payload-merged.log). That includes test_the_committed_payload_rebuilds_through_export_and_the_carry; the skips need release stages that aren't on this machine. d8abc3f changes only tests/test_call_provenance.py, which they don't use.
• At d8abc3f (~/reviews/policybench-row-cache-hit-calls/r4/):
  • all 44 tests in tests/test_call_provenance.py pass under each of Hypothesis seeds 1–8 (seeds.log);
  • PYTHONHASHSEED=0 pytest -n 2 -m "not slow" over the provenance, tool-pilot, carried-usage, cost-latency, run-store, eval, row-repair, retry, batch, sweep-timing and supervisor test files gives 664 passed, 5 skipped (related.log).
  • CI: run 40166733294 at d8abc3f9. Lint, app and paper pass. The test job's first attempt stalled at 88%, inside tests/test_runstore.py and tests/test_scenarios.py, and hit its 30-minute timeout; no test failed. Those files are unchanged here and pass locally in 24 seconds. On the rerun, pytest -m "not slow" gave 5480 passed, 79 skipped, 36 deselected: the 5,559 tests a local collection finds, 19 more than at cdf7975e.
  • CI runs the whole not-slow suite. I didn't rerun the full local suite, because a capacity reservation held the host.
• At b6d13931, which changes only a comment and a docstring after round 4: the 44 tests in tests/test_call_provenance.py pass (r4/provenance-after-wording.log). CI run 40176448694: lint, app, paper and test pass, and pytest -m "not slow" gives 5507 passed, 79 skipped, 36 deselected. The run tests the branch merged with current main, which now includes Disclose why 330 frozen households record near-zero latency and two models record none #228.
• ruff format --check and ruff check are clean on the changed files.

axiom: n/a: benchmark harness instrumentation, no policy change.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 25, 2026 8:25am UTC

[MaxGhenis]

Taking this PR over from session local_1bc636d9, which has been idle since 2026-10-14; I've messaged it. Review r2 (Sol) approved at 182decf with two non-blocking test gaps. I'll close them, merge main and get a fresh review. I'll merge it on green CI and an agreeing independent review, then remove its worktree. — session local_a8447b8d ("Finish five stale PolicyBench draft PRs")

[MaxGhenis]

Merged as d507d2b (squash, --match-head-commit b6d13931).

Gates at b6d13931:

• gh pr checks exits 0. CI run 40176448694: lint, app, paper and test pass (pytest -m "not slow": 5507 passed, 79 skipped, 36 deselected), plus Vercel.
• mergeable is MERGEABLE. The PR was marked ready before the merge, and there is no CHANGES_REQUESTED review.
• Independent review: round 4 (GPT-6.1 Sol, at d8abc3f9) returned APPROVE with three non-blocking wording corrections. b6d13931 makes exactly those corrections: a test comment, a test docstring, and the body. Both numbers were checked by computing them: 2.000001 - 2.0 == 1.000000000139778e-06, and the lower bounds 7.6204078397967105 > 7.62040783979671 and 482.7581355537797 > 482.75813555377965. The 44 provenance tests pass after the change. Rounds 1–3 are summarized in the body.

No published number changes: the PR adds the row fields only.

Evidence: ~/reviews/policybench-row-cache-hit-calls/ (r3/, r4/, and the four reviews).