Repository navigation
Add an agent-shaped case and calibrate the long-context filler - #44
Merged
Merged
Conversation
The published cases topped out at a few thousand input tokens, which is not the shape an agent actually sends. Against a production endpoint the difference is material: the shortest band of real requests decoded far slower than the short-prompt cases suggested, because decode walks the whole KV each step. A suite of short prompts alone therefore overstates throughput for this workload. `agent-turn` fills that gap: roughly 67K input tokens with a few hundred output tokens, i.e. a long-context turn rather than a bulk-copy or retrieval probe. The filler was calibrated at 1.6 characters per token, which undersized every long case by about 45%; `needle-32k` actually produced 17.5K tokens. At 2.92 the same spec measures 31,989 and 127,796 as named. The factor is a tokenizer approximation, so the docs now say the case name is a magnitude and the receipt's `prompt_tokens` is the truth. docs/benchmarks.md gains a section on the gap between synthetic cases and live agent traffic, and advises re-measuring with a long-input case before judging a deployment meant for agent backends.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The published cases topped out at a few thousand input tokens, which is not the shape an agent actually sends. Against a production endpoint the shortest band of real requests decoded far slower than the short-prompt cases suggested, because decode walks the whole KV each step — so a suite of short prompts alone overstates throughput for this workload.
agent-turn: roughly 67K input tokens with a few hundred output tokens, a long-context turn rather than a bulk-copy or retrieval probe.Filler calibration: the generator assumed 1.6 characters per token and undersized every long case by about 45% —
needle-32kreally produced 17.5K tokens. At 2.92 the same spec measures 31,989 and 127,796 as named. The factor is a tokenizer approximation, so the docs now state that the case name is a magnitude and the receiptprompt_tokensis the truth.docs/benchmarks.mdgains a section on the gap between synthetic cases and live agent traffic.Measured with the calibrated cases, 3 repeats, cold prefill, medians:
agent-turnDeepInfer TTFT 11.23 s / TPOT 10.33 ms vs the shared vLLM endpoint 9.67 s / 7.03 ms. As always these are different topologies and no numbers are committed. 243 tests pass.