feat(qy): MiniCPM5 FA, longctx MoE/RoPE, CUDA-graph capture safety by Ceng23333 · Pull Request #597 · InfiniTensor/InfiniLM

Ceng23333 · 2026-10-14T02:17:44Z

Summary

Enable MiniCPM5 on QY with FlashAttention, longctx MoE/RoPE GPU promotions, and CUDA-graph capture-safe sync helpers. Targets refactor/adopt-modern-infini-stack (post-refactor tip f94d83d).

Commits (high level)

Allow QY flash-attn + MiniCPM5 MoE stack
CUDA graph-safe FlashAttention for MiniCPM5
Merge longctx MoE/RoPE GPU promotions onto cudagraph
4–8. Capture-unsafe RoPE / avoid host sync under graph capture / sync helper fixes

Dependencies (merge order)

InfiniOps feat(qy): QY ops backend with FA and longctx MoE/RoPE kernels InfiniOps#1002
InfiniRT feat(qy): enable QY CUDA runtime device backend InfiniRT#49
InfiniCCL feat(qy): QY-compatible CPU device header fix InfiniCCL#77
InfiniCore chore(qy): pin InfiniRT/Ops/CCL for MiniCPM5 FA/longctx InfiniCore#1575
This PR

Source pin: ITW release/2026.10.13-qy-minicpm5-16a3-fa-longctx-cudagraph

Validation

python3 scripts/format.py --ref f94d83d --check with clang-format 18.1.8 + ruff: pass
python -m unittest discover -s test/scripts -p "test_build_infini_stack.py" -v: 3 errors (test harness looking for ops.json / local layout; not QY-specific regressions)
python -m unittest discover -s test/static -p "test_*.py" -v: 5 failures + 1 error reflecting intentional QY/MiniCPM5 contract expansions (device qy in BaseConfig, new adapters/count, minicpm5_moe model sources, graph-safe FA provider selection). Contract tests need follow-up update on this branch or a stacked commit.
Full QY native build/serve: not run on PR host (no QY SDK). Integration-validated via the ITW release pin above.
build_infini_stack.py --dry-run: blocked here by local InfiniCore submodule worktree SHAs after Ops/RT rebase (pin PR still references pre-rebase tips).

Notes

Branch: feat/qy-backend-build
Large multi-commit PR; each commit is focused (Conventional Commits).

Made with Cursor

Stack PR index (merge order)

InfiniOps: feat(qy): QY ops backend with FA and longctx MoE/RoPE kernels InfiniOps#1002
InfiniRT: feat(qy): enable QY CUDA runtime device backend InfiniRT#49
InfiniCCL: feat(qy): QY-compatible CPU device header fix InfiniCCL#77
InfiniCore: chore(qy): pin InfiniRT/Ops/CCL for MiniCPM5 FA/longctx InfiniCore#1575
InfiniLM: feat(qy): MiniCPM5 FA, longctx MoE/RoPE, CUDA-graph capture safety #597 → base refactor/adopt-modern-infini-stack

ITW pin: https://github.com/Ceng23333/InfiniTensorWorktree/tree/release/2026.10.13-qy-minicpm5-16a3-fa-longctx-cudagraph

Whitelist DEVICE_TYPE_QY for FA, add denglin MoE adaptors and MiniCPM5 MoE model surface, and ship QY stack build/config helpers used by the post-refactor integration.

Select the QY FA provider under graph capture, allowlist MiniCPM5 decode layouts, and keep MoE ATen/_dl_C ops in eager segments.

Apply C1-C3 longctx dirty sources (aten/dl kernels + sparse MoE block + server) on top of CUDA-graph FA MiniCPM5.

Longctx RoPE uses cudaMemcpy/pack and host sync; keep it in eager graph segments so FA/gemm capture still works with ENABLE_GRAPH=1.

Mark MulScalar capture-unsafe with MoE routing, and skip cudaStreamSynchronize while the stream is capturing so longctx QY kernels can participate in graph-on startup.

…dense

…nd ruff Co-authored-by: Cursor <cursoragent@cursor.com>

Ceng23333 · 2026-10-15T03:21:51Z

QY required smoke (PR head)

Head SHA: 60757bd9e3325113d4df1b5f3e30a5d82af4bcf8
Host: Denglin QY, DENGLIN_DEVICES=3, port 18184
Command: build_infini_stack.py --backend qy --operator-config …/infiniops_ops_qy_m2.json --test + xmake _infinicore/_infinilm; EvalScope mixed-4096 P1/N10
Result: PASS
- Stack m2 build/test OK
- Short greedy fluent (你好 → 好的，您好！…)
- EvalScope: EXIT=0, 10/10 (100%), no OOR (avg TTFT ~40.3s; ≤26s not required)
Log: …/logs/prhead-verify/evalscope-mixed-4096.log
Staging: bench_client_minicpm5-16a3_20261015_104057

Sibling PR heads: RT 95d30fe, Ops ed0b40d, CCL 13e5f61; Core gitlinks retargeted locally for e2e.

Ceng23333 · 2026-10-15T06:30:33Z

QY required smoke (PR head) — detailed

Head SHA: 60757bd9e3325113d4df1b5f3e30a5d82af4bcf8
Host: Denglin QY, DENGLIN_DEVICES=3, port 18184 (graph on, FA+paged)
Sibling heads: RT 95d30fe, Ops ed0b40d, CCL 13e5f61
Overall: PASS
Logs: …/logs/prhead-verify/stack-m2-build-test.log, greedy-hello.json, evalscope-mixed-4096.log
Staging: bench_client_minicpm5-16a3_20261015_104057

Build / stack

Step	Detail	Result
`build_infini_stack.py --backend qy` (m2, 23 ops incl. FA)	`--test`	PASS
InfiniRT ctest (via stack)	12/12	PASS
InfiniOps m2 install	linked FA + paged KV	PASS
InfiniCCL install	TP1 CPU/OMPI	PASS
xmake `_infinicore` + `_infinilm`	against prhead prefix	PASS

Serve / greedy

Check	Result
DUT up `http://127.0.0.1:18184/v1/models`	PASS (`minicpm5-16a3`)
Greedy `你好` (thinking off, max_tokens=64)	PASS — `好的，您好！有什么我可以帮助您的吗？` (11 tok)

EvalScope mixed-4096 (P1 / N10)

Metric	Value	Gate
`EXIT`	`0`	required
Success	10 / 10 (100%)	required
Failed	0	required
OOR	none observed	required
Avg input / output tokens	4096 / 971.7	—
Avg TTFT (ms)	40322	≤26s not required
Output throughput (tok/s)	4.2	—
Test duration (s)	2314.5	—

Promote MoE/RoPE/MulScalar to device-graph capture-safe, add INFINI_CUDAGRAPH_POLICY + LastRunProfile, and trim sync-under-capture on QY paths so decode can form a single InfiniRT segment.

Port PiecewisePrefillCompiler, InferencePhase/faInGraphAllowed (F&P=>FA decode), Llama PiecewiseTextCausalLM, and mha_varlen block_table contig for QY 2B capture.

Port V1 scheduler / chunked prefill so long prompts and mixed concurrency work under full_and_piecewise; scrub denglin→dl comments; assert floor 256; drop scheduling meta keys before forward.

Nested (B,1)/(B,1,1) sample layouts and out-of-range ids were crashing tokenizer.decode and killing the step loop; flatten/clamp and recover.

HF vocab_size excludes added tokens on MiniCPM5 (130072 vs 130560), so valid ids were wrongly clamped and starved decode.

EOS-clamping aborted ignore_eos EvalScope early (~100 tok). Keep CB runs alive for PARALLEL>1 perf until QY batched sample root-cause lands.

QY flattened batch sampling returned garbage token ids under V1 multi-request; force the per-index loop and drop the %vocab soft-map that corrupted decode.

Ceng23333 and others added 9 commits October 10, 2026 09:33

feat(qy): allow QY flash-attn and MiniCPM5 MoE stack

5d6a98e

Whitelist DEVICE_TYPE_QY for FA, add denglin MoE adaptors and MiniCPM5 MoE model surface, and ship QY stack build/config helpers used by the post-refactor integration.

feat(qy): enable CUDA graph-safe FlashAttention for MiniCPM5

d3144ab

Select the QY FA provider under graph capture, allowlist MiniCPM5 decode layouts, and keep MoE ATen/_dl_C ops in eager segments.

feat(qy): merge longctx MoE/RoPE GPU promotions onto cudagraph

40376ae

Apply C1-C3 longctx dirty sources (aten/dl kernels + sparse MoE block + server) on top of CUDA-graph FA MiniCPM5.

fix(qy): mark RoPE capture-unsafe for longctx QY pack path

71709ef

Longctx RoPE uses cudaMemcpy/pack and host sync; keep it in eager graph segments so FA/gemm capture still works with ENABLE_GRAPH=1.

fix(qy): avoid host sync under CUDA graph capture for longctx MoE/RoPE

1491d67

Mark MulScalar capture-unsafe with MoE routing, and skip cudaStreamSynchronize while the stream is capturing so longctx QY kernels can participate in graph-on startup.

fix(qy): correct sync_unless_capturing helper (no recursion)

5a0639f

fix(qy): skip cudaDeviceSynchronize under graph capture in moe_fused_…

012b81e

…dense

fix(qy): correct sync_device_unless_capturing helper (no recursion)

fdc1a40

style(qy): format FA/longctx/cudagraph changes with clang-format 18 a…

60757bd

…nd ruff Co-authored-by: Cursor <cursoragent@cursor.com>

Ceng23333 added 8 commits October 19, 2026 20:19

feat(qy): full_and_piecewise CUDA graph policy and capture-safe MoE/RoPE

ecceba6

Promote MoE/RoPE/MulScalar to device-graph capture-safe, add INFINI_CUDAGRAPH_POLICY + LastRunProfile, and trim sync-under-capture on QY paths so decode can form a single InfiniRT segment.

feat(qy): native piecewise prefill + F&P phase policy for MiniCPM5/Llama

b253606

Port PiecewisePrefillCompiler, InferencePhase/faInGraphAllowed (F&P=>FA decode), Llama PiecewiseTextCausalLM, and mha_varlen block_table contig for QY 2B capture.

feat(qy): continuous batching + chunked prefill for MiniCPM5 F&P

b074fb2

Port V1 scheduler / chunked prefill so long prompts and mixed concurrency work under full_and_piecewise; scrub denglin→dl comments; assert floor 256; drop scheduling meta keys before forward.

feat(qy): continuous batching + chunked prefill for MiniCPM5 F&P

5d5419e

Port V1 scheduler / chunked prefill so long prompts and mixed concurrency work under full_and_piecewise; scrub denglin→dl comments; assert floor 256; drop scheduling meta keys before forward.

fix(qy): sanitize batched sample ids under V1 continuous batching

9aa4fae

Nested (B,1)/(B,1,1) sample layouts and out-of-range ids were crashing tokenizer.decode and killing the step loop; flatten/clamp and recover.

fix(qy): use len(tokenizer) for sample-id bounds under CB

b82ebbf

HF vocab_size excludes added tokens on MiniCPM5 (130072 vs 130560), so valid ids were wrongly clamped and starved decode.

fix(qy): map OOR batched samples into-vocab instead of EOS

4976b59

EOS-clamping aborted ignore_eos EvalScope early (~100 tok). Keep CB runs alive for PARALLEL>1 perf until QY batched sample root-cause lands.

fix(qy): disable batch_greedy when n_out>1; hard-fail OOR samples

56f9b50

QY flattened batch sampling returned garbage token ids under V1 multi-request; force the per-index loop and drop the %vocab soft-map that corrupted decode.

Navigation Menu

Repository navigation

feat(qy): MiniCPM5 FA, longctx MoE/RoPE, CUDA-graph capture safety - #597

feat(qy): MiniCPM5 FA, longctx MoE/RoPE, CUDA-graph capture safety#597
Ceng23333 wants to merge 17 commits into
InfiniTensor:refactor/adopt-modern-infini-stackfrom
Ceng23333:feat/qy-backend-build

Ceng23333 commented Oct 14, 2026 •

edited

Loading

Uh oh!

Ceng23333 commented Oct 15, 2026

Uh oh!

Ceng23333 commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Ceng23333 commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commits (high level)

Dependencies (merge order)

Validation

Notes

Stack PR index (merge order)

Uh oh!

Ceng23333 commented Oct 15, 2026

QY required smoke (PR head)

Uh oh!

Ceng23333 commented Oct 15, 2026

QY required smoke (PR head) — detailed

Build / stack

Serve / greedy

EvalScope mixed-4096 (P1 / N10)

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ceng23333 commented Oct 14, 2026 •

edited

Loading