Conversation
Map E4M3 and BOOL through the existing ATen adaptor and preserve the caller's CUDA device across NCCL communicator destruction. Reuse the existing paged Prefill warp kernel for NVIDIA head size 256, without changing other vendors' default dispatch. Extend existing multi-page/long-context coverage and add finite FP8/mask cast checks. Validation: fresh SM86 build, 88 paged Prefill cases, 2 cast tests, and TP2 communicator teardown from both caller devices. Closes InfiniTensor#1565
Consolidate the allocator ownership, reduction/scalar-power recording and MetaX concatenation fixes from InfiniTensor#1560 into the runtime support PR. Preserve the original implementations and focused regressions without adding a separate compiler or Prefill graph path.
big-hip
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
Closes #1565. Target:
InfiniLM-v0.2.9c.Qwen greedy MTP needs E4M3/BOOL conversions, correct TP communicator teardown, head-256 paged Prefill, and reliable graph storage/replay. This PR supplies the complete shared-runtime prerequisites for InfiniTensor/InfiniLM#584. Its graph fixes are also used by Mamba-2 in InfiniTensor/InfiniLM#575.
Implementation
ncclCommDestroy.The implementation is organized into two reviewable commits: operator/communication support, then graph prerequisites. The latter consolidates #1560 without changing its code or tests. No Mamba scan, Prefill compiler, model weights or experiment artifacts are included. Native W8A8 is not introduced.
Validation
Consolidation audit and new focused logs; A6000 evidence; RTX 5090 evidence. Evidence lives on independent fork documentation branches.
The 5090 runtime's graph production changes are now inside this PR; SM120 build-option enumeration remains a documented validation overlay. Builds/tests above retain their exact recorded revisions. Fresh consolidation checks reuse the matching prebuilt runtime, not a newly compiled library. Project formatting (clang-format 21.1.8/Ruff 0.15.20) and whitespace checks pass.
Performance and limitations
On TP2 RTX 5090, real 27B FP8 weights/BF16 compute, batch=1 greedy, 80×64-token pages, 63/127/1023-input and 64-output tokens, three warm repeats: ordinary Decode graph gives 63.21/62.43/52.16 tok/s; K2 eager MTP gives 107.25/116.50/86.79. These are whole-stack MTP gains, not isolated Core-patch speedups. Sampled K2 device peaks are 24394/24398 MiB (100 ms sampling).
Unresolved integration result: controlled 5090 cancel/re-admit MTP differs at the 21st token despite passing KV/state reclamation. The first divergent layer is unlocalized; evidence does not attribute the failure to this Core patch. Single-request success does not establish general dynamic-batching acceptance. vLLM K2 also fails its own ordinary-output comparison; no validated cross-framework MTP speed claim is made.
Review and CI
Head
a3ac4df4: fork Ruff passed; Linux/Windows build matrix passed all four Ubuntu/Windows build and CPU-test jobs. The existing workflow labels jobs debug/release but does not passmatrix.typeto the build command; these results do not certify distinct build modes or GPU execution. Previous original-head build matrices passed. Upstream external-PR jobs need maintainer approval; no upstream success is implied.Please review allocator/graph ownership separately from numerical dispatch. @PanZezhong1725 and @spike-zhu were identified for review; formal assignment previously failed for lack of upstream permissions. Keep approval and required CI as merge gates. The MetaX code is supported by the linked C500 evidence, not a new vendor-wide validation claim.