feat(qwen): add paged greedy MTP with FP8 block weights

[big-hip]

Summary

Add opt-in text-only greedy decoding through Qwen's built-in MTP head. Draft 1–4 candidates, verify with the target model, and commit the accepted Conv/GDN checkpoint without replaying the target after rejection. Reuse the existing scheduler, paged cache, linear layers, weight loader and graph facilities.

• Model code stays in csrc/models/qwen3_5/, shared recurrent logic in qwen3_next, and request orchestration in llm/model_runner/mtp_runner.py.
• Support FP8-block/Marlin loading, vocabulary-parallel projection, independent state-row capacity, packed verification and cancellation/length/EOS/shutdown cleanup through existing service entrypoints.
• Optional extensions: K1 single-request draft graphs and TP1 byte-bounded exact-prompt snapshots. K>1 or batching uses eager; no general Prefill graph compiler.
• Keep three MTP correctness modules and an ordinary-model linear-layout regression. No research scripts, model weights or profiler artifacts enter this PR.

Review boundary: NVIDIA single-request TP1/TP2 results pass. New 5090 controlled cancel/re-admit testing has an unresolved exact-token mismatch despite successful cleanup. Resolve it or explicitly narrow the accepted service scope before merge.

Motivation and ordinary-inference impact

The existing model path discards MTP weights. Hybrid speculation additionally needs Conv and GDN state committed at the accepted token boundary; attention KV rollback alone is insufficient.

MTP is opt-in, but vocabulary-parallel projection, corrected norm/tied-weight loading and checkpoint-configured FP32 recurrent state also affect ordinary Qwen inference. FP32 doubles that state component versus BF16, not total model memory. Ordinary short GDN Prefill retains chunked dispatch; checkpointed verification uses per-token recurrence. Graph recapture preserves active KV; timed-out shutdown retains in-flight resources until cleanup. Ordinary numerical/lifecycle checks passed; performance is not certified unchanged for every model/backend.

Type of Change

• feat — optional model-head speculation and service integration.
• fix — required weight-layout, state-ownership and lifecycle correctness.
• Breaking change.

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Source head remains 86208ae1. Core runtime/graph prerequisites are now together in InfiniTensor/InfiniCore#1566; the former #1560 dependency is included there. No Mamba scan dependency.

Platform / scope	Result
NVIDIA tiny Qwen, TP1 / TP2	65 passed + 1 TP2-only skip / 66 passed; candidate acceptance, checkpoints, ties/uneven vocabulary, recapture, lifecycle
Tiny Qwen2/Llama ordinary inference	Separate upstream/current eager/graph checks match logits and greedy tokens; not a general performance non-regression claim
A6000 real 27B, TP2 K2 service	Previously validated packed generation, cancel/re-admit and active/queued close; exact outputs and complete KV/state reclamation
5090 real 27B, TP2 single request	K1/K2/K4 exactly match ordinary graph; longer 256/2047-input, 128-output checks also pass
5090 dynamic batching	Fixed packed MTP matches packed ordinary; controlled cancel/re-admit differs at token 21. Cleanup passes; numerical/state root cause remains unlocalized
Public entrypoints, tiny fixture	Single inference, offline benchmark and evaluation adapter run; HTTP smoke 64/64 requests. Evaluation score is not an accuracy claim; stream chunks are not token throughput
CPU / other accelerators	Configuration/remapping checks only on CPU; full MTP not validated on Ascend/MetaX/Moore due unavailable usable allocation. @wooway777 please review portability

Raw A6000 checks, source/build details and ordinary-path audit; 5090 conditions, raw results and unresolved failures.

Benchmark / Performance Impact

Current-head 2×5090 (32607 MiB/device), SM120, driver 610.43.02, CUDA 13.3, PyTorch 2.13.0a0+8145d630e8.nv26.06; SYS topology without CUDA P2P. Text, greedy, TP2/PP1, batch=1, prefix reuse off, 80×64-token pages, FP32 recurrent state, FP8 weights/BF16 compute (Marlin W8A16). Prompt Prefill stays eager; 64 output tokens, EOS ignored for fixed-work timing, three repeats after warmup. Median rate (N-1)/(wall-TTFT) includes CPU/scheduler, excludes loading/Prefill.

Prompt tokens	Ordinary Decode graph	K2 eager	K4 eager	K2 gain vs graph
Chinese 63	63.21 tok/s	107.25 tok/s	114.69 tok/s	+69.7%
Code 127	62.43 tok/s	116.50 tok/s	143.02 tok/s	+86.6%
Summary 1023	52.16 tok/s	86.79 tok/s	88.17 tok/s	+66.4%

Against the faster ordinary eager/graph mode per case, K2 gains +69.7%/+85.5%/+65.5%. Ordinary eager/graph differ below 1%; K1 draft graphs show no consistent extra gain. Nsight confirms 62 actual ordinary graph launches for 31 steps ×2 ranks. Warm-cache native FP8 probes do not establish an InfiniLM W8A8 integration or justify replacing Marlin for short verification.

100 ms sampled device peaks: ordinary 23766/23774 MiB; K2 24394/24398; K4 24774/24766. Three K1 graph recaptures during active requests and state-pool rebuild preserve exact continuations, peak 24382/24380 MiB; no observed growth above initial load. Brief peaks may be missed; this is not a long-duration leak test. Separate profiling identifies GDN chunk Prefill (~83% of the 1023-token initial step) and rank-wait imbalance as follow-up targets.

Archived TP2 A6000 (48 GiB/device, PCIe, same quantization/precision, 40×64 pages, two repeats): ordinary graph→K2 eager 33.91→49.57, 33.48→53.29, 30.03→48.61 tok/s for 63/127/1023-token prompts (64/64/55 output, last ends at EOS). Those are pre-cleanup measurements using aggregate rate; current-head 5090 medians above are separate evidence. TP2 draft batching remains disabled after inconsistent benefit.

vLLM ordinary native FP8 measures about 73.3 tok/s on the 5090 setup. Its K2 output differs from its own ordinary baseline at the first token; isolated 143.66 tok/s is failed-validation diagnostic only. No validated vLLM MTP speedup comparison; its different cache budget also prevents a capacity-matched memory claim.

Dependencies, scope and review

• Base: InfiniLM-v0.2.9c. Runtime prerequisite: issue/1565 fix(runtime): support Qwen MTP and recurrent graph replay InfiniCore#1566, including graph ownership/replay. There is no separate #1560 dependency and no dependency on Mamba scan #1562.
• Generic linear loading/capture-state overlap with feat(mamba2): integrate model loading and recurrent inference #575 requires ordinary merge-order resolution; no Mamba model/converter/scan enters this feature.
• Scope: dense greedy text, one shared MTP layer, PP1, NVIDIA TP1/TP2, BF16/aligned E4M3-block weights. Random sampling, multimodal/MoE/PP and multiple MTP layers are not supported claims.
• Graphs require K1/max batch1; target verification stays eager. Exact full-prompt snapshots are opt-in, byte-bounded, request-owned and TP1 only; no partial-prefix or distributed recurrent-state transfer claim.
• Review checkpoint ownership and accepted-length KV rollback first. The 5090 dynamic-batch mismatch remains a merge-scope issue, not waived by single-request speed gains.

CI / ChatOps

Unchanged LM head 86208ae1: fork format and Ruff passed; fork hardware ci was skipped. Upstream CI and Ruff are action_required. Maintainer approval/retest must use the matching Core dependency; formatting success is not hardware coverage. Ready for code review, subject to the stated correctness boundary, required CI and maintainer approval before merge.

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits; one feature commit and one focused compatibility fix, based on target InfiniLM-v0.2.9c (270feb3e).
• No fixup! / squash! / wip commits remain.
• N/A Existing legacy issue format: this new branch uses Conventional Commits.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• No raw new/delete; RAII / smart pointers / existing allocators are used.
• Changed files are formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
• Passed single request test (examples/test_infer.py), or specify the reason for skipping.
• Passed offline performance test (examples/bench.py), or specify the reason for skipping.
• Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for skipping.

Build, CI, and Tooling

• The project builds cleanly from a fresh directory on at least one affected platform.
• CI has been triggered manually (Actions → CI on this branch), or /retest was requested.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A Breaking change: the new mode is opt-in; unsupported MTP combinations fail explicitly.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.