Repository navigation
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/large
There was a problem hiding this comment.
🟡 Changes recommended
An unverified remote shell script is executed during setup, and the documented smoke-test configuration uses invalid or ineffective keys that can fail or run the full dataset.
3 open issues
What changed in this PR
Adds MLPerf speech-to-text support for Whisper Large V3 over LibriSpeech, including multipart OpenAI-compatible transcription requests, WER scoring, dataset loading, configuration, reference rules, documentation, and tests.
Changes:
- Adds the
openai_transcriptionsAPI adapter and non-streaming response handling. - Adds the Whisper LibriSpeech dataset loader, cache/download workflow, and MLPerf-compatible WER scorer.
- Adds configuration validation, MLCommons reference metadata, examples, documentation, and unit/integration coverage.
This pull request requires changes.
| File | Summary | Review notes |
|---|---|---|
tests/unit/openai/test_transcriptions_registration.py |
Tests transcription API registration and adapter wiring. | — |
tests/unit/openai/test_transcriptions_adapter.py |
Tests multipart encoding, decoding, transforms, and injection handling. | — |
tests/unit/evaluation/test_wer.py |
Tests WER calculations and MLPerf reference vectors. | — |
tests/unit/evaluation/test_wer_scorer.py |
Tests corpus scoring, partial runs, normalization, and preflight behavior. | — |
tests/unit/endpoint_client/test_http.py |
Tests configurable HTTP content types and binary request bodies. | — |
tests/unit/dataset_manager/test_whisper_librispeech_dataset.py |
Tests manifest parsing, caching, downloading, and validation. | — |
tests/unit/config/test_schema.py |
Tests transcription-specific configuration capabilities and validation. | — |
tests/unit/async_utils/services/metrics_aggregator/test_token_metrics.py |
Tests tokenizer behavior without chat templates. | — |
tests/integration/openai/test_transcriptions_adapter_integration.py |
Exercises audio upload, worker handling, mock-server integration, and WER scoring. | — |
tests/integration/commands/test_probe_command.py |
Verifies unsupported transcription API types are omitted from probe help. | — |
src/inference_endpoint/openai/types.py |
Adds the transcription response type. | — |
src/inference_endpoint/openai/transcriptions_adapter.py |
Implements multipart audio requests and transcription response decoding. | moderate, 1 vote: response_format permits values such as text, srt, or vtt, but decoding only supports JSON responses with a text field. Reject unsupported formats or implement their decoders.moderate, 1 vote: Audio bytes are not checked for the fixed multipart boundary, allowing delimiter injection. Reject the delimiter in audio or generate a per-request boundary. moderate, 1 vote: Missing or unreadable clips can raise before _handle_error(), producing neither an error nor completion. Validate manifest paths during setup or convert request-preparation exceptions into error results. |
src/inference_endpoint/evaluation/wer.py |
Implements MLPerf speech-to-text normalization and WER alignment. | — |
src/inference_endpoint/evaluation/scoring.py |
Adds the corpus-level WERScorer. |
moderate, 1 vote: WER/extractor incompatibility is detected only during finalization after endpoint work completes. Validate the combination during setup. |
src/inference_endpoint/endpoint_client/worker.py |
Passes adapter-declared content types into HTTP templates. | — |
src/inference_endpoint/endpoint_client/http.py |
Supports configurable content types in request prefixes. | — |
src/inference_endpoint/endpoint_client/config.py |
Registers the transcription adapter and non-streaming accumulator. | — |
src/inference_endpoint/endpoint_client/adapter_protocol.py |
Adds the adapter CONTENT_TYPE contract. |
— |
src/inference_endpoint/endpoint_client/accumulator_protocol.py |
Adds a guard accumulator for non-streaming APIs. | — |
src/inference_endpoint/dataset_manager/predefined/whisper_librispeech/__init__.py |
Adds the LibriSpeech manifest, cache, and R2 download loader. | critical, 1 vote: A setup script fetched from raw.githubusercontent.com is executed without digest or signature verification. Verify a known SHA-256/signature or vendor the downloader.moderate, 1 vote: Cache reuse is based only on manifest presence, so partial downloads can be treated as complete. Record or verify download completion, or validate every referenced audio path before reuse. moderate, 1 vote: requests is imported without an explicit project dependency and relies on a transitive package. Declare the runtime dependency or use an already-declared HTTP client. |
src/inference_endpoint/dataset_manager/__init__.py |
Exports the Whisper dataset. | — |
src/inference_endpoint/core/types.py |
Adds the transcription API enum, route, and capability flags. | — |
src/inference_endpoint/config/templates/online_template_full.yaml |
Documents the new scorer and API type in the online template. | — |
src/inference_endpoint/config/templates/offline_template_full.yaml |
Documents the new scorer and API type in the offline template. | — |
src/inference_endpoint/config/templates/concurrency_template_full.yaml |
Documents the new scorer and API type in the concurrency template. | — |
src/inference_endpoint/config/schema.py |
Adds WER configuration and API-specific validation. | — |
src/inference_endpoint/config/rulesets/mlcommons/rules.py |
Adds Whisper throughput rules. | — |
src/inference_endpoint/config/rulesets/mlcommons/models.py |
Registers Whisper reference accuracy and dataset settings. | — |
src/inference_endpoint/config/rulesets/mlcommons/datasets.py |
Registers the 1,633-sample LibriSpeech dataset. | — |
src/inference_endpoint/commands/probe.py |
Rejects transcription API types from text-based probing. | — |
src/inference_endpoint/async_utils/services/metrics_aggregator/token_metrics.py |
Handles tokenizers without chat templates. | — |
examples/12_Whisper_Example/README.md |
Documents serving, dataset preparation, execution, and scoring. | nit, 3 votes: The smoke-test instructions change DatasetConfig.samples, which does not limit this predefined dataset. Instruct users to set generate_params.max_samples: 8 on both entries.nit, 2 votes: generate_params.force is not valid for this loader path and can be passed twice, causing TypeError. Use the supported regeneration option. |
examples/12_Whisper_Example/offline_whisper.yaml |
Provides an offline Whisper benchmark configuration. | — |
docs/evaluation/DESIGN.md |
Documents the WER scorer. | — |
docs/ENDPOINT_CLIENT.md |
Documents configurable request content types. | — |
AGENTS.md |
Updates architecture and component documentation. | — |
.gitignore |
Ignores generated Whisper run logs. | — |
Review effort: Lite
Comment on lines
+129
to
+136
| resp = requests.get( | ||
| "https://raw.githubusercontent.com/mlcommons/r2-downloader/" | ||
| f"{cls.R2_DOWNLOADER_COMMIT}/mlc-r2-downloader.sh", | ||
| timeout=30, | ||
| ) | ||
| resp.raise_for_status() | ||
| script_path.write_bytes(resp.content) | ||
| script_path.chmod(0o755) |
|
|
||
| The parsed-row cache is keyed on the dataset ID alone, so if you change | ||
| `manifest` or `data_dir` after a first run, also pass | ||
| `generate_params.force: true` once — otherwise the rows built from the previous |
Comment on lines
+71
to
+73
| To check the wiring first, set `runtime.n_samples_to_issue` and both `samples` | ||
| values to 8. The dataset truncates in manifest order, so the smoke subset is a | ||
| prefix of the full corpus — but a partial perf phase is not a reportable result. |
Benchmarks whisper-large-v3 over LibriSpeech against an OpenAI-compatible transcription endpoint and scores Word Error Rate, following the framework's existing extension points. The one framework change: `HttpRequestTemplate` folded `Content-Type: application/json` into its two pre-rendered prefixes, and `extra_headers` can only append (a per-request header would duplicate it). The media type now comes from `HttpRequestAdapter.CONTENT_TYPE`, which the worker passes to the template once at setup — still pre-encoded, no hot-path cost. `/v1/audio/transcriptions` needs this because the OpenAI Audio API is a `multipart/form-data` file upload with no JSON variant, and it is the only vLLM route that serves `WhisperForConditionalGeneration`. - `api_type: openai_transcriptions` -> `openai/transcriptions_adapter.py`. The dataset row carries the clip's *path*; the worker reads the bytes in `encode_query`, so the corpus never crosses the issue thread or the ZMQ frame. Non-streaming (OpenAI does not support `stream` for Whisper), served by a shared `NonStreamingAccumulator`. - `whisper_librispeech` consumes the MLCommons-prepared `dev-all-repack.json` manifest. The ~30 s repackaging stays upstream: different clip boundaries change the WER, so a client-side rebuild would not be comparable to the reference. - `eval_method: wer` reports `100 - WER%` corpus-wide. `evaluation/wer.py` ports the benchmark's concatenation-aware metric (a generic WER library does not reproduce the reference number, which credits joined words across spans of up to three) and pairs it with `transformers`' `EnglishTextNormalizer`. Verified against upstream: 20k randomized alignment trials and 3k normalizer cases, zero divergence. No new dependencies. - A tokenizer with no chat template (Whisper and other ASR tokenizers) no longer logs an ERROR stack trace for its zero chat-template baseline. - The corpus is fetched on first run: `download_from_r2` pulls the published MLCommons artifact (pinned downloader commit, host allowlist, MD5-verified) into the dataset cache, so `generate_params.manifest` is now an override for a local copy rather than a prerequisite. - Manifest `fname` entries keep the upstream pipeline's `./data/` prefix, which the hosted artifact's layout does not reproduce. The loader probes the first clip and drops leading components that are absent under the root, so both layouts resolve with no configuration. Reference accuracy is 97.9329% (WER 2.0671%); submissions need 99% of that. See examples/12_Whisper_Example/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enforce in code what the config comments only asserted, and decide the two semantics the review flagged as underspecified. - `language` is now actually sent. `dataset_transforms` seeded only model/response_format/temperature, and the manifest has no `language` column, so every request fell back to the server's per-clip auto-detection rather than the reference's pinned decoder language. `DEFAULT_LANGUAGE` supplies it; `overwrite=False` still lets a dataset column win. - A missing sample is scored as an empty hypothesis instead of being dropped. Scoring iterated the returned rows, so a sample with no COMPLETE event left the denominator and dropping the samples a SUT handles worst would raise the score. `WERScorer` now walks the issued samples and sets `complete = False` when any is missing, so a partial run is machine-detectable rather than needing a reader to notice `missing_samples`. - Three api_type constraints move from YAML comments into `_resolve_and_validate`: `streaming: on` is an error (AUTO resolves OFF even online), `metrics_isl` is forced False, and `warmup.salt` is an error. The unreachable `stream` guard in `encode_query` is gone. - `encode_multipart` escapes the dataset-controlled header parameters and rejects a field value carrying the delimiter, so a row cannot forge a part. - `WERScorer.preflight` resolves the normalizer before the run. It needs HF network access or a warm cache, and scoring runs after the whole performance phase, so a cold cache failed the run at its last step with the endpoint work already spent. A configured `extractor` is now rejected rather than silently ignored. - `download_from_r2` is the single R2 path: BFCLv4 and OpenOrca lose their copies of the pinned commit, host allowlist and subprocess plumbing. It gains a timeout, because dataset generation runs in `setup_benchmark` before the whole-run deadline is checked, so nothing else would interrupt a stalled multi-GB fetch. - Register the workload for submission checking: `Whisper_large_v3` (97.9329 at 0.99) and `LibriSpeechDevAll` (1633), both transcribed from the inference submission checker's constants, plus an Offline-only `PerModelRuleset`. The numbers no longer live only in Markdown. - Pin WER parity with `UPSTREAM_VECTORS`: vectors captured from the MLCommons reference at commit 3fbc3299, covering joins in both directions, a join past the 3-word span, and the empty edges. The lowercasing divergence is documented as a no-op on the official corpus (upstream's preparation pipeline already lowercases) kept as defence for a hand-built manifest. Docstring corrections where the code did not do what the prose claimed: the non-streaming accumulators described a mechanism that does not exist (a body with no `data:` frames never reaches `decode_sse_message`, so nothing raises and nothing accumulates), and the adapter now says the audio read is synchronous on the worker's event loop, which stalls every in-flight response in that process rather than only its own request. `probe.py` folds its two unsupported-api_type blocks into one map, `wer_counts` skips the k=1 concatenation probe the branch above already settled, and `duration` is documented as inspection-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback on the speech2text workload: - Drop tests/integration/openai/__init__.py and fold its single-use conftest.py into the test module. The test is renamed to a unique basename because pytest's prepend import mode would otherwise collide it with tests/unit/openai/test_transcriptions_adapter.py. - Keep the R2 download local to the whisper loader. dataset.py, bfcl_v4 and open_orca return to their prior state; the pinned downloader commit, host allowlist and subprocess timeout now live on WhisperLibriSpeech as class attributes, mirroring BFCLv4. - Stop the probe's --api-type help advertising api_types the probe refuses. show_choices is off because cyclopts otherwise renders the full APIType enum, which includes those entries regardless of the help string. Separately, register the whisper golden accuracy as bfloat16 rather than fp32. The field records the precision the value was obtained at, and the MLCommons reference SUT hardcodes dtype = "bfloat16". A healthy run scores a little above the 97.9329% reference. Scoring is not the cause: evaluation/wer.py reproduces upstream's compute_wer_with_concatenation with zero per-sample disagreement over the full 1633-clip corpus, both candidate EnglishTextNormalizer implementations give the same corpus WER, and the max_duration filter drops no clips. The gap is the serving path -- the reference runs vLLM V0 offline with an explicit decoder prompt and in-process librosa decode, while this example goes over HTTP and lets the server own both. The Whisper example README now records that comparison. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Narrows the review surface of the speech2text workload to the files the feature actually needs, without changing behaviour. - probe: drop UNSUPPORTED_PROBE_API_TYPES and reject the two unsupported api_types inline. The dict generalized over videogen, which the whisper change has no reason to touch; the messages it produced are unchanged. - rulesets: trim the Whisper_large_v3 model and PerModelRuleset comment blocks to the rationale that is not already recoverable from the values themselves. - docs/ENDPOINT_CLIENT.md: restore the abridged APIType snippet, adding only OPENAI_TRANSCRIPTIONS. Filling in OPENAI_COMPLETIONS and VIDEOGEN was unrelated to this workload. - schema/types: condense the has_text_prompt and non-streaming comments. - AGENTS.md: condense the Whisper row to what is not in the code. - ATTRIBUTION: drop the speech2text entry; the upstream notice lives in the wer.py header and docstring. - examples/12_Whisper_Example: replace the illustrative accuracy figures with a measured full-corpus run (score 98.10, WER 1.897%, 105077 reference words) and note that reference_words is corpus-fixed, so a different value there means the wrong manifest. Validated against two full 1633-clip runs (perf + accuracy) on whisper-large-v3: score 98.1195 (WER 1.8805%, 1976/105077), 0 failed, 0 empty, 0 missing. WER recomputed independently from events.jsonl and the dev-all-repack manifest, matching the scorer exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mkankana
force-pushed
the
feat/whisper-speech2text
branch
from
November 5, 2026 21:25
d069119 to
245d992
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Changes are not approval-ready because a critical scoring bug can mark failed requests as complete, while the smoke-test guidance remains ineffective and the API-type help is incomplete.
6 open issues
Review effort: Lite
Comment on lines
+519
to
+524
| hypothesis = outputs.get(sample_uuid) | ||
| if hypothesis is None: | ||
| missing += 1 | ||
| hypothesis = "" | ||
| elif not hypothesis: | ||
| empty += 1 |
Comment on lines
+71
to
+73
| To check the wiring first, set `runtime.n_samples_to_issue` and both `samples` | ||
| values to 8. The dataset truncates in manifest order, so the smoke subset is a | ||
| prefix of the full corpus — but a partial perf phase is not a reportable result. |
Comment on lines
+1071
to
+1072
| alias="--api-type", | ||
| help="API type: openai, openai_transcriptions, sglang, or videogen", |
There was a problem hiding this comment.
🟡 Changes recommended
Critical unresolved issues can permit execution of unverified downloaded code, reuse incomplete datasets, and publish apparently complete WER results after failed requests, while a moderate header-escaping defect also remains.
8 open issues
Review effort: Lite
Comment on lines
+158
to
+160
| if result.returncode != 0: | ||
| raise RuntimeError( | ||
| f"R2 downloader failed with code {result.returncode}: {result.stderr}" |
| #: the percent escapes browsers substitute for them (WHATWG HTML, "multipart/ | ||
| #: form-data encoding algorithm"). Servers decode these, so escaping keeps the | ||
| #: original value recoverable where rejecting the row would not. | ||
| _PARAM_ESCAPES = str.maketrans({'"': "%22", "\r": "%0D", "\n": "%0A"}) |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #563 +/- ##
=======================================
Coverage ? 82.84%
=======================================
Files ? 173
Lines ? 24662
Branches ? 0
=======================================
Hits ? 20431
Misses ? 4231
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
arekay-nv
reviewed
Nov 10, 2026
| #: in ``setup_benchmark``, before ``RunWatchdog`` arms its whole-run | ||
| #: deadline, so nothing else would ever interrupt a stalled fetch. Sized for | ||
| #: several GB of audio over a slow link, not for a healthy transfer. | ||
| R2_DOWNLOAD_TIMEOUT_S = 2 * 60 * 60 |
Collaborator
There was a problem hiding this comment.
The first-run corpus download can exceed this timeout.
_download_from_r2 fetches the entire published artifact. Its whisper-dataset.md5 lists 18,719 files: the raw LibriSpeech dev-clean/dev-other FLACs, the unpacked dev-all WAVs, and the repackaged set. The downloader fetches and verifies them one at a time.
The benchmark only reads 1,634 of those files: dev-all-repack/*.wav plus dev-all-repack.json. On shared/network storage the full sequential fetch was on track to run well past this 2 h limit, so the first run fails at setup. Fetching just the 1,634 files avoids this; they MD5-verify against the published whisper-dataset.md5. Pointing WHISPER_LIBRISPEECH_MANIFEST at that copy then works as documented.
Possible fixes:
- restrict the download to
dev-all-repack/+dev-all-repack.json, or - fetch in parallel, or
- at minimum, have the README say the auto-download pulls the full artifact and recommend pre-staging plus
generate_params.manifest/$WHISPER_LIBRISPEECH_MANIFEST.
| --config examples/12_Whisper_Example/offline_whisper.yaml | ||
| ``` | ||
|
|
||
| To check the wiring first, set `runtime.n_samples_to_issue` and both `samples` |
Collaborator
There was a problem hiding this comment.
This smoke-run tip doesn't shrink the accuracy phase.
With runtime.n_samples_to_issue and both samples values set to 8, the performance phase issues 8 samples, but the loader still returns all 1,633 rows (Loaded 1633 samples), so the accuracy phase transcribes and scores the full corpus. The dataset-level samples: field doesn't appear to be read for predefined datasets; that predates this PR.
What does produce an 8-sample smoke run in both phases is generate_params.max_samples (the loader logs Truncating to the first 8 of 1633 rows):
- name: whisper_librispeech
type: "performance"
generate_params:
max_samples: 8
- name: whisper_librispeech
type: "accuracy"
generate_params:
max_samples: 8
accuracy_config: ...(plus runtime.n_samples_to_issue: 8). Truncation happens after the cached parquet is read, so switching back to the full run needs no force. Suggest updating this tip and the matching comment in offline_whisper.yaml.
| # accuracy, the storage requirement for the WAVs, how to point at a local | ||
| # manifest, and what api_type validation derives rather than reading here. | ||
| # | ||
| # For a smoke run, set runtime.n_samples_to_issue and both `samples` to 8. |
Collaborator
There was a problem hiding this comment.
Same as the README comment: samples: 8 doesn't truncate the accuracy phase for this predefined dataset. generate_params.max_samples: 8 on both dataset entries (plus runtime.n_samples_to_issue: 8) does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Benchmarks whisper-large-v3 over LibriSpeech against an OpenAI-compatible transcription endpoint and scores Word Error Rate, following the framework's existing extension points.
Type of change
Related issues
Testing
Checklist