Perf: 40 层算完后一次 bulk H2D（替代 PR 1872 逐层 eager H2D） - #7
Open
yanghaoran29 wants to merge 11 commits into
Open
Conversation
…est (hw-native-sys#1838) * Add: DeepSeek-V4 FLASH full 43-layer decode network as an EP2 scene test Port the complete DeepSeek-V4 FLASH decode forward from pypto-lib (models/deepseek_v4_flash_mtp/decode_fwd.py) as the first pypto-harvested distributed example: a level-3 scene test on 2 dies with expert-parallel MoE dispatch/combine and a TP2 LM head through a comm domain. - examples/a2a3/tensormap_and_ringbuffer/deepseek_v4_flash_decode/: harvested chip orchestration (7.8k lines) + 367 incore kernels, verbatim pypto codegen plus license headers, clang-format, and the whole-word renames L2TaskArgs->ChipTaskArgs / L0TaskArgs->CoreTaskArgs / Tensor->ChipTensor (pypto's pinned runtime predates the role-based naming contract). - The scene orch fn transcribes the generated host_orch.py: one comm domain (12 window buffers, ~6.6 MiB), 92 tensors + 13 scalars per rank, per-rank submits. Stacked per-rank slices become independent _r0/_r1 host tensors (the rehost passes whole buffers), and pypto's device-resident weight upload becomes plain host tensor args. - simpler_setup/goldens/deepseek_v4_flash_decode.py: deterministic fixture for the 80 network parameters (~100 GiB at full size), metadata transcribed exactly, weight distributions matching the upstream spec builders. - The case is manual (367-kernel compile takes minutes) and skip_golden (upstream runs the same fixture with golden_fn=None; numeric coverage lives with the standalone kernel harnesses in pypto-lib). - .pre-commit-config.yaml: exclude the harvested orchestration translation unit from check-added-large-files (708 KiB is inherent to the 43-layer program). * Raise check-added-large-files cap to 750 kB instead of excluding the orchestration The harvested whole-network chip orchestration decode_fwd.cpp is 725 350 bytes (~708 kB) — one generated translation unit whose size is inherent to the 43-layer program. Keep it under the hook's scrutiny by raising the global cap from the 500 kB default to 750 kB rather than excluding its directory.
Runs the tensormap_and_ringbuffer DeepSeek-V4 case under host_build_graph: same 43-layer network, same 367 kernels, same fixture, same comm-window protocol, only the runtime differs. decode_fwd_hostbuild.cpp is that case's orchestration with two edits, 51 lines in all, and the runtime untouched: - The ten get_tensor_data reads of recv_count_out stand in a constant. host_build_graph builds the whole graph before the device executes anything, so a read of a task-produced tensor has no value to return. The constant holds the per-expert tile loops at their real trip count, ceil(16/16) == 1, which is what the h_i8 [512, 2048] layout budgets per expert. - The six set_initial_value calls are dropped. Their target is a GM-heap device address the host orchestrator cannot store to; leaving them in segfaults the chip subprocess. The other 31 get_tensor_data reads are external tensors the runtime stages with a host view, and are left alone. Loop structure, submit order, dependencies and scope nesting are byte-identical to the source, so the graph keeps the size and shape of the real one (15971 tasks) but not the fixture's routing — hence skip_golden. manual because the 367-kernel compile takes minutes. Host-side construction completes and the graph uploads. Device execution stalls 12 tasks from the end, in hc_head_linear. README.md records the measurement, the ten causes ruled out one at a time on hardware, and the two threads still open.
simpler_run stops after simpler_prepare_run when the variable is set: orchestration, graph construction, image relocation and the SM H2D all run, the kernel launch and its completion wait do not, and simpler_finalize_run still releases the run's resources. No outputs are produced, so a run under this variable is a timing harness, not a test. The host_build_graph DeepSeek-V4 case builds its graph but stalls partway through device execution, which otherwise makes the host side of that case unmeasurable. This is a temporary handle on that, and goes away with the stall; see that case's README.md.
alloc_tensors marked the recording unsupported, which graph_end turns into "the recorded Graph contains a construct that Graph Execution does not support" — a hard failure, not the fallback its comment described. The stated reason, that runtime-allocated outputs cannot be replayed, does not hold. submit_dummy_task records the identical shape — the same graph_record_submit_node call with INVALID_KERNEL_ID in all three slots — and poisons nothing. Replay already reserves the intermediate heap for every internal node, so an allocation node's outputs land at addresses the replayed Definition derives for itself, exactly as a kernel node's do. Verified on a2a3 hardware and in simulation with an alloc_tensors call inside a Graph body: the Definition records, replays, and the numerics hold. Without this, an orchestration that allocates anywhere inside a Graph body cannot use Graph Execution at all.
A Graph boundary carried at most 32 tensors and 16 scalars because it was a CoreTaskArgs, whose capacity every task's payload inherits. Widening that capacity to hold a larger boundary would have grown PTO2TaskPayload from 4864 to 17152 bytes and, with it, every byte of payload shipped to the device — for a graph spanning a decoder layer, hundreds of megabytes. Nothing forces that coupling. graph_reset_outer_payload zeroes the outer GRAPH task's tensor_count, so the boundary never reaches a payload at all; it is read host-side only, by graph_boundary_matches and graph_build_submission_image. Give it its own type, GraphTaskArgs, holding 128 tensors and 64 scalars. PTO2TaskPayload is unchanged at 4864 bytes, the device sees the same bytes it saw before, and the cost lands on orchestration stack alone: 920 bytes per CoreTaskArgs as before, 2888 for the one Graph boundary. The scalar half is not hypothetical headroom. A Graph body is a free function, so every loop-invariant local the enclosing orchestration holds has to cross the boundary too — it can reach neither the caller's frame nor the loop variable. A decoder layer needs 31: four layer indices, twelve per-layer scales, and fifteen locals that would otherwise be in scope. graph_classify_scalar becomes a template because it takes both an internal node's CoreTaskArgs and the boundary's GraphTaskArgs; its identity test on the two now compares addresses through void, the types no longer matching. The three graph_execution scene tests and the qwen3-14B decode case declare their boundary arguments and Graph body signatures with the new type.
SIMPLER_SKIP_DEVICE_RUN lived in simpler_run, but the L3 multi-chip subprocess drives a run through the split prepare/launch/wait/finalize entry points and never calls simpler_run, so the variable never stopped anything there: the AICPU op launched, executed, and failed on the device a run under this variable was meant to leave untouched. The check now sits at the top of simpler_launch_run, which both paths pass through. It marks the run Complete with rc 0 before any execution claim is taken, so wait returns immediately and finalize walks its not-launched branch (kernel-args release, retire of the unused stream, SM pointer cleared, validate reads a null header and returns 0).
decode_fwd_graph.cpp recasts the 20-iteration decoder layer loop (40 of the 43 layers) of the hostbuild baseline as one rt_submit_graph per iteration. The layer's task set becomes the Graph body, a free function that reads its per-layer weight views, scales, indices and the loop's invariant locals through GraphTaskArgs; the boundary is positional — args.tensor(i) in the body and the i-th add_input/add_inout at the submit site are the same slot, and both lists are emitted from one ordered walk so they cannot drift. The submit-everything form made the host hand 15991 tasks to the device scheduler per run; the Graph form records a 744-node Definition once and boots with 1131 host-submitted tasks (measured on both ranks on a2a3 hardware, ASCEND_GLOBAL_LOG_LEVEL=1, host-orch boot line). The two hostbuild edits carry over unchanged: the ten get_tensor_data(recv_count_out) reads stand in HBG_RECV_ROWS_PER_EXPERT and the six set_initial_value calls are dropped. The non-Graph baseline stays in the tree and the README documents both: the baseline is what the stall investigation measured against, and the Graph variant is what the case now runs. Device-side replay of a Definition this size is not yet exercised — an unskipped run fails in Graph activation before the tail stall — so SIMPLER_SKIP_DEVICE_RUN host-side measurement is the verification path for now.
A chip child that received SHUTDOWN still has to release everything it imported before exiting. On a large-scope run (the dsv4 host_build_graph case: ~80 shm backings incl. a 2 GiB ring heap) that teardown measured ~12 s — past the 10 s _ROLLBACK_GRACEFUL_TIMEOUT_S the close path was sharing, so every teardown ended in "child process(es) did not exit within the close budget" even though the children were healthy and merely draining (py-spy showed them inside multiprocessing's SharedMemory close the whole window; no CANN call was involved). The close path now uses its own 60 s constant. Rollback keeps the tighter 10 s: its graceful wait guards an unlink-only path where exceeding the budget means the child is stuck and should be killed, which is the opposite trade-off from a child doing proportional exit work.
…-port Add: DeepSeek-V4 FLASH decode on host_build_graph, in Graph form
Write each Graph layer into a retained pinned host arena and copy_to_device as soon as the POD is ready, so bind skips a later gather of unpinned images.
Remove the per-layer eager upload from PR 1872. During graph build, layer PODs are only written into the pinned host bump. After entry() completes, wire execution storage into each host POD, then issue a single copy_to_device over [pinned_base, pinned_used) into a device blob reused across rounds; graph_context points into that blob at each layer's pinned offset. Layers that fall back to std::vector keep the per-layer upload path. Co-Authored-By: Claude <noreply@anthropic.com>
yanghaoran29
force-pushed
the
perf/hbg-orch-pinned-h2d
branch
5 times, most recently
from
August 18, 2026 12:16
30ac448 to
67df8ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
PR 1872（
perf/hbg-orch-pinned-h2d）已把每层 Graph POD 就地写进 16MB pinned bump，但构图过程中每写完一层就copy_to_device一次（eager）。本 PR 改为：40 层 POD 全部写完后，对[pinned_base, pinned_used)连续 host 区做 一次 H2D。改动
host/runtime_maker.cppGraphPodH2d::upload_all：entry()后一次 blob H2D，device blob 跨 round 复用；去掉 eager 上传runtime/orchestrator_core/pto_orchestrator.cppruntime/graph_host_state.h核心路径：
entry()只构图、写 pinned bump，不 H2Ddevice_malloc(used)（只拷 used 字节，不拷整 16MB），跨 round 复用copy_to_device(dev, host, used)一次graph_context = dev + (pinned − host)，scheduler 仍按GraphSubmission*语义使用std::vector时，仍走逐层upload_one（qwen 稳态不走此路径）优化效果
50 轮交叉 trim-mean（qwen3_14b_decode，去最高/最低 10，剩 30 取 mean）
拷贝形状核对：HEAD 每次推理 41 次
h2d_graph（40 层 + leftover），NEW 每次推理 1 次。上板泳道（NEW vs 1872，device 侧不受影响）
device 侧执行不因 host H2D 形状变化而变。
测试方法（与 PR 1872 相同）
examples/a2a3/host_build_graph/qwen3_14b_decode，--skip-golden，SIMPLER_SKIP_TENSOR_COPY_BACK=1libhost_runtime.so，同位置 STRACE span（仅测量 overlay，不进本 PR）--enable-chip-swimlane，rounds=1，PASSED🤖 Generated with Claude Code