Repository navigation
Add a reproducible inference latency benchmark with 13 synthetic cases - #40
Merged
Merged
Conversation
Publish the measurement harness used to compare serving configurations on the same weights: TTFT to the first streamed delta, a second clock to the first visible content, and TPOT over the server's own completion token count so prefill never leaks into the per-token number. Thirteen cases cover the shapes that matter for these deployments: tiny requests, long decode, long-form generation, code, single and multi tool calls, single/multi/thumbnail slides, JSON-schema constrained output, 32K and 128K retrieval, and a three-turn prefix-cache series. Every case is synthetic: the prose is original filler, the slides are drawn by make_slide.py, the code snippet is written for this repo, and the calibration codes are invented, so nothing here carries machine identity, credentials or customer data. bench.py keeps host addresses out of receipts (scheme, port and a loopback/private-network tag only) and takes the bearer token from an environment variable, so keys never reach argv or the record. docs/benchmarks.md states the measurement definitions, the case inventory, how to attach an engine configuration to a run, and the caveats that make two numbers incomparable: different serving topology, differing output length, and noise on a shared endpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishes the harness used to compare serving configurations on the same weights.
tools/bench/bench.py: TTFT to first streamed delta, a second clock to first visible content, TPOT over the server-reportedcompletion_tokens, median of repeated samples. Host addresses stay out of receipts; the bearer token comes from an environment variable.tools/bench/cases/: 13 synthetic cases across tiny request, long decode, long-form, code, single/multi tool calls, single/multi/thumbnail slides, JSON-schema constrained output, 32K/128K retrieval, and a three-turn prefix-cache series.tools/bench/make_slide.py: draws the slide assets so images need no download and carry no customer data.docs/benchmarks.md: metric definitions, case inventory, engine-configuration record template, and the caveats that make two numbers incomparable (topology, output length, shared-endpoint noise).No performance numbers are committed: per repo policy the receipt stays private, so the public content stays method-plus-tooling rather than a published result.