Skip to content

feat(moss): support vLLM diarized responses #3566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LauraGPT merged 1 commit into from
Aug 29, 2026
Merged

feat(moss): support vLLM diarized responses #3566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 53 additions & 16 deletions docs/moss_transcribe_diarize.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ This guide was verified against:
- source: `OpenMOSS/MOSS-Transcribe-Diarize@cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3`
- model: `OpenMOSS-Team/MOSS-Transcribe-Diarize@e8681d68e7042738ffca8ac8212bc8fcb1131ab8`
- license: Apache-2.0 in the pinned upstream source and model metadata
- vLLM nightly index: `68b4a1d582818e67adc903bf1b8fc5a5447da2fa`
- vLLM release: `v0.27.1@6e448d0ea9bf3d88d898b65449ca6dc2aec170ac`
- vLLM CUDA 12.9 x86_64 wheel SHA-256:
`bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b`

Pin all three revisions. The model uses `trust_remote_code`; do not execute a
floating model revision in a production service.
Expand Down Expand Up @@ -76,36 +78,48 @@ model = AutoModel(
backend="vllm",
vllm_base_url="http://127.0.0.1:8898/v1",
vllm_model="moss-transcribe-diarize",
vllm_response_format="diarized_json",
disable_update=True,
)
result = model.generate("audio.wav")[0]
```

The vLLM adapter sends the documented OpenAI-compatible multipart request and
normalizes the returned raw text. Authentication can be supplied with
`vllm_api_key`; keep it in a secret store rather than source code.
normalizes the official speaker-attributed `segments` directly into
`sentence_info`. Authentication can be supplied with `vllm_api_key`; keep it
in a secret store rather than source code.

`vllm_response_format="diarized_json"` is the structured production path on
the pinned vLLM revision. The compatibility default remains `json` so existing
clients keep the exact tagged generation in `raw_text`; in structured mode,
`raw_text` is the cleaned `text` returned by vLLM and the authoritative speaker
metadata is in `sentence_info`.

## Choose a serving path

| Path | Environment | Response contract | Use when |
|---|---|---|---|
| vLLM | CUDA 12 (`cu129`) or CUDA 13 (`cu130`) | `response_format=json` returns raw `[start][Sxx]text[end]` text | You already operate vLLM or need its scheduler |
| vLLM | CUDA 12 (`cu129`) or CUDA 13 (`cu130`) | `response_format=diarized_json` returns OpenAI-compatible speaker segments; `response_format=json` keeps raw `[start][Sxx]text[end]` text | You already operate vLLM or need its scheduler |
| SGLang Omni | CUDA 13 in the current upstream guide | `response_format=verbose_json` returns parsed segments | You need structured speaker segments directly from the API |
| Transformers | PyTorch process | Python objects and raw tagged text | Evaluation, debugging, or custom preprocessing |

The two HTTP backends share `/v1/audio/transcriptions`, but their documented
response formats are not interchangeable. Do not promise vLLM `segments`
without testing the exact vLLM revision.
response formats are not interchangeable. The pinned vLLM revision supports
`diarized_json`; test the exact server revision before promising `segments`.

## vLLM

Create an isolated environment and install the upstream-pinned nightly build:
Create an isolated environment and install the verified vLLM release:

```bash
uv venv --python 3.12 .venv-moss
uv pip install --python .venv-moss/bin/python -U 'vllm[audio]' \
--torch-backend=auto \
--extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129
curl -fL \
https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl \
-o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
echo "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
| sha256sum -c -
uv pip install --python .venv-moss/bin/python --torch-backend=auto \
"vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
```

The `audio` extra is required. A plain `vllm` install can start the server but
Expand All @@ -124,13 +138,13 @@ CUDA_VISIBLE_DEVICES=0 .venv-moss/bin/vllm serve \
--port 8898
```

Submit audio and validate the speaker-tagged text:
Submit audio and validate the official speaker-attributed response:

```bash
curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions \
-F file=@audio.wav \
-F model=moss-transcribe-diarize \
-F response_format=json \
-F response_format=diarized_json \
-F temperature=0 \
| tee moss-transcription.json

Expand All @@ -140,19 +154,36 @@ import json
with open("moss-transcription.json", encoding="utf-8") as stream:
payload = json.load(stream)
text = payload.get("text", "")
segments = payload.get("segments", [])
assert text.strip(), payload
assert "[S01]" in text, text
print(text)
assert segments, payload
assert all(
isinstance(item.get("speaker"), str)
and item.get("text")
and item.get("start") <= item.get("end")
for item in segments
), payload
print(text, sorted({item["speaker"] for item in segments}))
PY
```

For an audit of the model's compact tagged generation, repeat the request with
`response_format=json` and verify `[S01]` appears in the returned `text`.

For long recordings, test `max_completion_tokens` against the longest expected
meeting. A larger limit can increase memory use and tail latency.

### Reproduced vLLM contract

The FunASR ecosystem validation used vLLM
`0.23.1rc1.dev949+g68b4a1d58`, Torch `2.11.0+cu129`, and one H100 80GB:
The structured-response validation used vLLM 0.27.1, Torch
`2.13.0+cu129`, and one H100 80GB. The 15.1685-second two-speaker probe
(`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`)
returned two `diarized_json` segments labelled `S01` and `S02`; the same
request through the FunASR adapter returned two monotonic `sentence_info`
entries with the same speakers.

Earlier raw-response validation on vLLM
`0.23.1rc1.dev949+g68b4a1d58`, Torch `2.11.0+cu129`, and one H100 80GB found:

- the bundled 6.000-second sample
(`ea03e1f473ad1618a03da3327a545369cb8f6f06cb0f4115535e5a866167d47e`)
Expand All @@ -165,6 +196,12 @@ This proves the pinned API and speaker-return contract on those inputs. It is
not a diarization accuracy result, overlap test, throughput benchmark, or
production capacity claim.

The older nightly commit
`68b4a1d582818e67adc903bf1b8fc5a5447da2fa` predates vLLM
[`#48543`](https://github.com/vllm-project/vllm/pull/48543). It accepts
`response_format=json` but returns HTTP 400 for `diarized_json`; upgrade to the
verified release instead of silently retrying an expensive transcription.

The FunASR adapter was also tested on both `backend="hf"` and
`backend="vllm"` with model revision
`e8681d68e7042738ffca8ac8212bc8fcb1131ab8`. The 15.1685-second two-speaker
Expand Down
65 changes: 50 additions & 15 deletions docs/moss_transcribe_diarize_zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ MOSS-Transcribe-Diarize 会联合生成转写、时间戳和 `[S01]` 等说话
- 源码：`OpenMOSS/MOSS-Transcribe-Diarize@cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3`
- 模型：`OpenMOSS-Team/MOSS-Transcribe-Diarize@e8681d68e7042738ffca8ac8212bc8fcb1131ab8`
- 许可证：固定上游源码和模型元数据中的 Apache-2.0
- vLLM nightly 索引：`68b4a1d582818e67adc903bf1b8fc5a5447da2fa`
- vLLM 发布版：`v0.27.1@6e448d0ea9bf3d88d898b65449ca6dc2aec170ac`
- vLLM CUDA 12.9 x86_64 wheel SHA-256：
`bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b`

三者都要固定。模型依赖 `trust_remote_code`，生产服务不要执行浮动的模型 revision。

Expand Down Expand Up @@ -68,34 +70,46 @@ model = AutoModel(
backend="vllm",
vllm_base_url="http://127.0.0.1:8898/v1",
vllm_model="moss-transcribe-diarize",
vllm_response_format="diarized_json",
disable_update=True,
)
result = model.generate("audio.wav")[0]
```

vLLM 适配器发送官方文档中的 OpenAI-compatible multipart 请求，并统一解析返回文本。
认证可通过 `vllm_api_key` 提供；请放在 secret store，不要写入源码。
vLLM 适配器发送官方文档中的 OpenAI-compatible multipart 请求，并把官方返回的
speaker `segments` 直接映射到 `sentence_info`。认证可通过 `vllm_api_key` 提供；
请放在 secret store，不要写入源码。

固定 vLLM revision 的生产结构化路径是
`vllm_response_format="diarized_json"`。为保持已有客户端兼容，默认值仍是 `json`，
此时 `raw_text` 保留原始标签生成；结构化模式下，`raw_text` 是 vLLM 返回的清理后
`text`，权威说话人信息位于 `sentence_info`。

## 选择服务后端

| 路径 | 环境 | 响应契约 | 适用场景 |
|---|---|---|---|
| vLLM | CUDA 12（`cu129`）或 CUDA 13（`cu130`） | `response_format=json` 返回原始 `[start][Sxx]text[end]` 文本 | 已运行 vLLM，或需要其调度能力 |
| vLLM | CUDA 12（`cu129`）或 CUDA 13（`cu130`） | `response_format=diarized_json` 返回 OpenAI-compatible 说话人 segments；`response_format=json` 保留原始 `[start][Sxx]text[end]` 文本 | 已运行 vLLM，或需要其调度能力 |
| SGLang Omni | 当前上游指南为 CUDA 13 | `response_format=verbose_json` 返回解析后的 segments | API 需要直接返回结构化说话人分段 |
| Transformers | PyTorch 进程 | Python 对象和原始标签文本 | 评测、排障或自定义预处理 |

两个 HTTP 后端都使用 `/v1/audio/transcriptions`，但官方文档中的响应格式不能混为
一谈。没有对 exact vLLM revision 做验证时，不要承诺 vLLM 返回 `segments`。
一谈。固定的 vLLM revision 支持 `diarized_json`；承诺 `segments` 前仍要验证实际
部署的 exact server revision。

## vLLM

创建隔离环境并安装上游固定的 nightly build：
创建隔离环境并安装已验证的 vLLM 发布版：

```bash
uv venv --python 3.12 .venv-moss
uv pip install --python .venv-moss/bin/python -U 'vllm[audio]' \
--torch-backend=auto \
--extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129
curl -fL \
https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl \
-o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
echo "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
| sha256sum -c -
uv pip install --python .venv-moss/bin/python --torch-backend=auto \
"vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
```

必须安装 `audio` extra。只安装 plain `vllm` 时服务可以启动，但因为没有音频解码器，
Expand All @@ -113,13 +127,13 @@ CUDA_VISIBLE_DEVICES=0 .venv-moss/bin/vllm serve \
--port 8898
```

提交音频并验证说话人标签文本：
提交音频并验证官方结构化说话人响应：

```bash
curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions \
-F file=@audio.wav \
-F model=moss-transcribe-diarize \
-F response_format=json \
-F response_format=diarized_json \
-F temperature=0 \
| tee moss-transcription.json

Expand All @@ -129,18 +143,34 @@ import json
with open("moss-transcription.json", encoding="utf-8") as stream:
payload = json.load(stream)
text = payload.get("text", "")
segments = payload.get("segments", [])
assert text.strip(), payload
assert "[S01]" in text, text
print(text)
assert segments, payload
assert all(
isinstance(item.get("speaker"), str)
and item.get("text")
and item.get("start") <= item.get("end")
for item in segments
), payload
print(text, sorted({item["speaker"] for item in segments}))
PY
```

如需审计模型原始紧凑标签生成，请把请求改为 `response_format=json`，并验证返回
`text` 中包含 `[S01]`。

长音频需要按业务最长会议验证 `max_completion_tokens`。增大上限也会影响显存和尾延迟。

### 已复现的 vLLM 契约

FunASR 生态验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch
`2.11.0+cu129` 和一张 H100 80GB：
结构化响应验证使用 vLLM 0.27.1、Torch `2.13.0+cu129` 和一张 H100 80GB。
15.1685 秒双说话人样例
（`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`）
返回两个 `diarized_json` segments，标签为 `S01`、`S02`；同一请求通过 FunASR
适配器后，返回两个时间单调且说话人一致的 `sentence_info`。

较早的 raw-response 验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch
`2.11.0+cu129` 和一张 H100 80GB，结果如下：

- 仓库内 6.000 秒样例
（`ea03e1f473ad1618a03da3327a545369cb8f6f06cb0f4115535e5a866167d47e`）
Expand All @@ -152,6 +182,11 @@ FunASR 生态验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch
这证明固定版本对这些输入满足 API 和说话人返回契约，不是 diarization 精度、重叠语音、
吞吐或生产容量结论。

旧 nightly commit `68b4a1d582818e67adc903bf1b8fc5a5447da2fa` 早于 vLLM
[`#48543`](https://github.com/vllm-project/vllm/pull/48543)：它支持
`response_format=json`，但 `diarized_json` 会返回 HTTP 400。应升级到已验证发布版，
不要静默重试一次成本较高的转写请求。

FunASR 适配器还分别使用 `backend="hf"` 和 `backend="vllm"`，按模型 revision
`e8681d68e7042738ffca8ac8212bc8fcb1131ab8` 做了真实测试。15.1685 秒的双说话人
样例（`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`）通过
Expand Down
Loading
Loading