Skip to content

perf(ds4): overlap q5 verification and stabilize 16K - #575

Draft
davide221 wants to merge 1 commit into
codex/ds4-q5-verifierfrom
codex/ds4-overlapped-moe-balance
Draft

perf(ds4): overlap q5 verification and stabilize 16K#575
davide221 wants to merge 1 commit into
codex/ds4-q5-verifierfrom
codex/ds4-overlapped-moe-balance

Conversation

@davide221

@davide221 davide221 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • batch q=5 hyper-connection pre/post/output work and drafter-feature capture across all verifier lanes
  • place profiled experts by the predicted two-owner critical path instead of aggregate hit rate
  • use chunked heterogeneous prefill scratch, proactive graph-cache eviction, and a sticky 1K long-context shape on no-VMM devices
  • add reproducible overlap tracing and exact-context qualification controls

Exact-context qualification

Configuration: q=5, four active routed experts, 14,350 MiB hot-expert budget, main/peer rate 4.4, two warm-ups plus three measured 128-token replies per context.

Context Client decode median
2K initial 75.818 tok/s
4K 74.530 tok/s
8K 69.898 tok/s
16K 62.685 tok/s
2K after 16K 76.703 tok/s

All 25 requests completed with the expected deterministic response hash. Peak main-GPU VRAM was 31.089 GiB. The post-16K 2K measurements were 76.685-76.727 tok/s.

A same-budget placement A/B moved 116 experts to the peer and measured 76.636 tok/s at rate 3.8, within 0.1 tok/s of the qualified rate. A post-batching trace found 16-22% simultaneous kernel-busy time in steady decode windows; target verification remains 63.7 ms of a 74.4 ms speculative step.

Validation

  • test_deepseek4_unit: pass
  • test_server_unit: 322 passed, 0 failed
  • full 2K -> 4K -> 8K -> 16K -> 2K model-backed sweep: pass
  • shell syntax and Python compile checks: pass
  • git diff --check: pass

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant