Repository navigation
Measure cold prefill, not a cached replay, and report the network floor - #43
Merged
Merged
Conversation
Two defects made earlier comparisons unsound without looking wrong. First, repeating an identical prompt measures a prefix-cache hit rather than a prefill. One retrieval case read 0.52 s and 0.56 s on two of three samples and 3.14 s on the other; the fast pair was a cache hit, and taking the median turned a 3.1 s cold prefill into 0.56 s. `--cache-bust` now gives every sample a unique one-line system prefix, so each measurement is a cold prefill. Cases declaring `turns` are exempt: their whole purpose is to observe prefix reuse. Second, the first request against a schema or a fresh engine pays compile and warmup costs that later samples do not. `--warmup N` discards N runs per case before sampling. Third, a TTFT measured across a tunnel or a WAN hop contains transport time. The receipt now records `network_floor_ms` from five metadata reads, so a reader can separate "engine was slow" from "the client was far away". On the two hosts used here the floors differed by two orders of magnitude, which changes the conclusion for the longest case from a wide gap to parity. Receipt schema goes to 3 and records `warmup` and `cache_bust` so a result can never be read as cold when it was warm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects made the earlier comparison unsound without looking wrong.
Prefix cache. Repeating an identical prompt measures a cache hit. One retrieval case read 0.52 s / 3.14 s / 0.56 s across three samples; the median reported 0.56 s for what is physically a 3.1 s cold prefill.
--cache-bustgives every sample a unique one-line system prefix. Cases declaringturnsare exempt because observing prefix reuse is their purpose.Warmup. First-request compile and graph-capture costs leak into samples.
--warmup Ndiscards N runs per case.Network floor. TTFT measured across a tunnel contains transport time. The receipt now records
network_floor_msfrom five metadata reads. In the two deployments measured here the floors were 3 ms and 125 ms; subtracting it changed the longest-context result from a wide gap to parity.Receipt schema goes to 3 and records
warmupandcache_bust, so a warm result can never be read as cold.docs/benchmarks.mddocuments all three traps. 243 tests pass.