Skip to content

MLPerf speech2text workload integration #563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mkankana wants to merge 5 commits into
base: main
Choose a base branch
from
Open

MLPerf speech2text workload integration #563

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0be98e
feat(whisper): integrate the MLPerf speech2text workload
mkankana Oct 29, 2026
eaba98e
fix(whisper): address review on the speech2text workload
mkankana Nov 2, 2026
fd1ba14
fix(whisper): address review and correct the golden precision label
mkankana Nov 4, 2026
245d992
fix(whisper): trim the speech2text diff to the feature surface
mkankana Nov 5, 2026
cfc7851
Merge branch 'main' into feat/whisper-speech2text
arekay-nv Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,3 +213,6 @@ CLAUDE.local.md
# Generated dataset cache (created by Dataset.get_dataloader())
dataset_cache/
.agents/devdocs/

# Run artifacts: the example configs' default report_dir
logs/
9 changes: 6 additions & 3 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
| **DeepSeek-R1 (MLPerf)** | `src/inference_endpoint/evaluation/scoring.py` (`LegacyMLPerfDeepSeekR1Scorer`), `examples/07_DeepSeekR1_Example/` | MLPerf DeepSeek-R1 accuracy. TensorRT-LLM is OpenAI-compatible, so it is served via `api_type: openai` / `openai_completions` (no dedicated trtllm adapter). The combined multi-subset eval (`math500`/`aime`/`gpqa`/`mmlu_pro`/`livecodebench`) is the official MLCommons `eval_accuracy.py`, run out-of-process via `uv run --project` against the isolated subproject at `src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/` (a uv subproject excluded from the parent wheel; mirrors the VBench pattern). The example feeds the exact MLPerf prompt via pre-tokenized `input_tokens` to `/v1/completions`. |
| **VideoGen** | `src/inference_endpoint/videogen/` | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader. |
| **SWE-bench** | `src/inference_endpoint/dataset_manager/predefined/swe_bench/`, `src/inference_endpoint/evaluation/swe_bench_scorer.py`, `src/inference_endpoint/evaluation/swebench_service/` | `SWEBench` predefined dataset (HuggingFace `princeton-nlp/SWE-bench_Verified` or `_Lite`; `ACCURACY_ONLY=True`). `SWEBenchScorer` sets `SKIP_ENDPOINT_PHASE=True` and bypasses the built-in accuracy phase entirely: it delegates agent execution and grading to the configured SWE-bench service via `accuracy_config.extras.swebench_service_url`. The service is an isolated `uv` subproject; its host owns Docker/runtime execution, artifacts, and credentials, while the benchmark client remains the report-producing entrypoint. |
| **Whisper (MLPerf speech2text)** | `src/inference_endpoint/openai/transcriptions_adapter.py`, `dataset_manager/predefined/whisper_librispeech/`, `evaluation/wer.py` | MLPerf speech2text (whisper-large-v3 / LibriSpeech). `api_type: openai_transcriptions` posts a `multipart/form-data` upload to `/v1/audio/transcriptions`, the only route serving Whisper on vLLM, so the adapter declares `HttpRequestAdapter.CONTENT_TYPE` and the worker folds it into the request prefix at setup. `Query.data` carries the clip's _path_; the worker reads the bytes in `encode_query`, so the corpus never crosses the ZMQ frame. Non-streaming. `whisper_librispeech` fetches the MLCommons-prepared `dev-all-repack.json` via its own `_download_from_r2`; the repackaging stays upstream because clip boundaries change the WER. `WERScorer` (`eval_method: wer`) reports `100 - WER%` from the ported metric in `evaluation/wer.py`. Reference numbers live in `models.Whisper_large_v3` / `datasets.LibriSpeechDevAll`. |
| **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py` | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`accuracy/accuracy_results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass). |
| **Compliance (audit tests)** | `src/inference_endpoint/compliance/`, `commands/audit.py` | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_<TEST>.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Perf-only by default (a phase may opt into accuracy via `AuditRunSpec.test_mode`, but this is unused today). |
| **Validation** | `src/inference_endpoint/validation/`, `docs/validation/` | Cohort checker for submission uploads. A cohort is a directory of YAML checks (four scope files + `valid_refs.yaml` + a `cohort.yaml` naming the packaged ruleset); `python -m inference_endpoint.validation COHORT SUBMISSION` runs them against an upload root (`results/<system>/<model>/r<N>/`) and reports holds/failures per check. Exit codes: 0 holds, 1 blocking failure, 2 refusal. See docs/validation/README.md. |
Expand Down Expand Up @@ -357,7 +358,7 @@ src/inference_endpoint/
│ ├── output_caching_test.py # OutputCachingAudit (MLPerf TEST04): caching detection (reference vs fixed-sample QPS)
│ └── README.md # TEST04 output-caching audit usage (WAN 2.2 example)
├── core/
│ ├── types.py # APIType, Query, QueryResult, StreamChunk, QueryStatus (msgspec Structs)
│ ├── types.py # APIType (default_route + supports_streaming/has_text_prompt capabilities), Query, QueryResult, StreamChunk, QueryStatus (msgspec Structs)
│ └── record.py # EventRecord — transport record used by event logger and ZMQ transport
├── load_generator/
│ ├── session.py # BenchmarkSession, PhaseIssuer, PhaseConfig, PhaseResult, SessionResult
Expand Down Expand Up @@ -399,7 +400,7 @@ src/inference_endpoint/
│ ├── dataset.py # Dataset base class, DatasetFormat enum
│ ├── factory.py # Dataset factory
│ ├── transforms.py # ColumnRemap and other transforms
│ └── predefined/ # Built-in datasets (aime25, cnndailymail, gpqa, etc.)
│ └── predefined/ # Built-in datasets (aime25, cnndailymail, gpqa, whisper_librispeech, etc.)
├── metrics/
│ ├── report.py # Report.from_snapshot(MetricsSnapshot); display + JSON serialization
│ ├── metric.py # Metric types (Throughput, etc.)
Expand All @@ -415,10 +416,11 @@ src/inference_endpoint/
│ ├── rulesets/mlcommons/ # MLCommons-specific rules, datasets, models
│ └── templates/ # YAML config templates (_template.yaml minimal, _template_full.yaml all defaults)
├── openai/ # OpenAI-compatible API types and adapters
│ ├── types.py # OpenAI response types (chat + text completion)
│ ├── types.py # OpenAI response types (chat + text completion + transcription)
│ ├── openai_adapter.py # Chat completions adapter (/v1/chat/completions)
│ ├── openai_msgspec_adapter.py # msgspec-based chat completions adapter (fast path)
│ ├── completions_adapter.py # Text completions adapter (/v1/completions, pre-tokenized input)
│ ├── transcriptions_adapter.py # Audio transcription adapter (/v1/audio/transcriptions, multipart upload)
│ ├── accumulator.py # Streaming response accumulator (shared by chat + completions)
│ └── harmony.py # openai_harmony integration
├── sglang/ # SGLang API adapter (/generate with input_ids)
Expand All @@ -427,6 +429,7 @@ src/inference_endpoint/
│ ├── types.py # Pydantic: VideoPathRequest, VideoPathResponse, VideoPayloadResponse
│ └── adapter.py # VideoGenAdapter (HttpRequestAdapter) + VideoGenAccumulator (no-op)
├── evaluation/ # Accuracy evaluation (extractor, scoring, livecodebench)
│ ├── wer.py # MLPerf speech2text Word Error Rate (ported metric; used by WERScorer)
│ └── swebench_service/ # Isolated uv service for Docker/Pyxis SWE-bench runs
├── compliance/ # Submission compliance checks (config-lock, accuracy gate, run validity)
│ ├── __init__.py
Expand Down
17 changes: 9 additions & 8 deletions docs/ENDPOINT_CLIENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ model validator.
```python
class APIType(str, Enum):
OPENAI = "openai"
OPENAI_TRANSCRIPTIONS = "openai_transcriptions"
SGLANG = "sglang"

class HTTPClientConfig(BaseModel):
Expand Down Expand Up @@ -960,19 +961,19 @@ concatenating them with a single `b"".join()`.

| Method | Async | Description |
| ------------------------------------------------ | ----- | -------------------------------------------------------------------------- |
| `from_url(host, port, path)` | No | Classmethod: create template with pre-encoded request line + Host header |
| `from_url(host, port, path, content_type?)` | No | Classmethod: create template with pre-encoded request line + Host header |
| `cache_headers(headers)` | No | Pre-encode headers (e.g. Authorization) into bytes; call once during setup |
| `build_request(body, streaming, extra_headers?)` | No | Build complete HTTP/1.1 request bytes; fast path when no extra headers |

**Request byte segments:**

| Segment | When Built | Description |
| -------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Static prefix | Once (init) | `POST /v1/chat/completions HTTP/1.1\r\nHost: localhost:8000\r\n` — pre-encoded at construction via `from_url()` |
| Cached headers | Once (setup) | `Authorization: Bearer sk-...\r\n` — pre-encoded via `cache_headers()`; reused across all requests |
| Content-Type | Once (class) | Two class-level `bytes` constants: `HEADERS_STREAMING` and `HEADERS_NON_STREAMING`. The streaming/non-streaming branch selects between them — no string encoding at request time |
| Content-Length | Per-request | `Content-Length: 1234\r\n\r\n` — the only header that changes per request; built from a single f-string and ASCII-encoded |
| Body | Per-request | JSON bytes produced by the adapter's `encode_query()` ([§9.1](#91-httprequestadapter)) |
| Segment | When Built | Description |
| -------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Static prefix | Once (init) | `POST /v1/chat/completions HTTP/1.1\r\nHost: localhost:8000\r\n` — pre-encoded at construction via `from_url()` |
| Cached headers | Once (setup) | `Authorization: Bearer sk-...\r\n` — pre-encoded via `cache_headers()`; reused across all requests |
| Content-Type | Once (setup) | From the adapter's `CONTENT_TYPE` (default `application/json`; `openai_transcriptions` declares a `multipart/form-data` boundary). Folded into two prefixes with the matching `Accept`, one per streaming branch — no string encoding at request time |
| Content-Length | Per-request | `Content-Length: 1234\r\n\r\n` — the only header that changes per request; built from a single f-string and ASCII-encoded |
| Body | Per-request | Bytes produced by the adapter's `encode_query()` ([§9.1](#91-httprequestadapter)), encoded in the adapter's `CONTENT_TYPE` |

![build_request() fast/slow path](res/endpoint_client/11_build_request_flow.png)

Expand Down
1 change: 1 addition & 0 deletions docs/evaluation/DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ The scorer registry in `evaluation/scoring.py` currently includes:
| `vbench` | VBench video-generation accuracy (WAN 2.2 T2V) |
| `bfcl_v4` | BFCL v4 function-calling accuracy |
| `legacy_mlperf_deepseek_r1` | MLPerf DeepSeek-R1 combined multi-subset accuracy |
| `wer` | MLPerf speech2text Word Error Rate (Whisper / LibriSpeech) |

The registry (`Scorer.PREDEFINED`) is auto-populated from `Scorer` subclasses
via `__init_subclass__`.
Expand Down
Loading
Loading