Skip to content

feat(moe): add a generic heterogeneous stage planner - #577

Draft
davide221 wants to merge 8 commits into
codex/ds4-overlapped-moe-balancefrom
codex/generic-heterogeneous-stage-planner
Draft

feat(moe): add a generic heterogeneous stage planner#577
davide221 wants to merge 8 commits into
codex/ds4-overlapped-moe-balancefrom
codex/generic-heterogeneous-stage-planner

Conversation

@davide221

@davide221 davide221 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • keep the generic, model-neutral two-owner stage planner and owner-local residual primitive
  • add the complete R9700 + Strix Halo ROCm 7.2.4 profile: routing inputs, hipBLASLt table, checksum guards, and a one-command qualifier
  • fix a rebase regression that split heterogeneous 2K prefill into 1,920 + 128 tokens, changed the sparse-prefill state, and ended generation after two tokens
  • keep the repaired path as one wide batch while preserving required snapshot-boundary splits
  • report authoritative model-side DSpark decode separately from the historical client transport diagnostic

Reproduce

Build for both GPU architectures, then run with only the two model paths:

cmake -S server -B server/build-hip-dual \
  -DCMAKE_BUILD_TYPE=Release \
  -DDFLASH27B_GPU_BACKEND=hip \
  -DDFLASH27B_HIP_ARCHITECTURES='gfx1151;gfx1201' \
  -DDFLASH27B_TESTS=ON

cmake --build server/build-hip-dual -j \
  --target dflash_server test_tokenizer_harness test_deepseek4_unit

TARGET_MODEL=/path/to/target.gguf \
DRAFT_MODEL=/path/to/dspark-draft.gguf \
BUILD_DIR="$PWD/server/build-hip-dual" \
server/scripts/qualify_ds4_q5_r9700_strix.sh

The wrapper defaults to 2 warmups + 7 measured 2K/128-token requests and fails on a short or non-exact response. See server/docs/DS4_R9700_STRIX_PROFILE.md.

Validation

  • head: 682e419cc21c9de4585e705b2bb21f7dc3274efa
  • dual-architecture HIP Release build: passed
  • full test_deepseek4_unit: passed, including GPU oracles and the new capture-policy cases
  • test_server_unit heterogeneous_stage: 5 passed, 0 failed
  • server-summary Python tests: 2 passed
  • exact model qualification: 7/7 measured responses produced 128 tokens and SHA-256 0f785a7ffa406498aafb14553966eaed0f52220fed0f7cc016b66921d104d194
  • authoritative server decode: 73.7 tok/s median, 73.5-73.8 range, acceptance 1.00
  • historical client diagnostic: 83.403 tok/s median, 83.122-83.509 range

Review finding about the prior 89 tok/s result

The earlier 89.876 value was not model-side throughput. Its server median was 73.2 tok/s. The client timer began at the first non-empty streamed text event but counted all 128 completion tokens; a slow first rejected speculative step fell outside the timed interval and inflated the result.

The repaired current path has full first-block acceptance and is slightly faster model-side (73.7 versus 73.2), although the legacy client number is lower. The qualifier now writes server-decode-summary.json so future comparisons use the correct metric.

Measured-slower attention and shared-width splits remain disabled.

Stacked on #575.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant