[Issue]: Investigate MegaMoEV2 end-to-end throughput gap vs V1 on 8×MI355X

[jhinpan]

Summary

MegaMoEV2 landed in #876, followed by synchronization and correctness fixes in #954. Standalone operator benchmarks show a clear improvement over MegaMoEV1, especially for prefill where V2 overlaps communication and computation.

However, preliminary DeepSeek-V4-Pro end-to-end serving tests on a single 8×MI355X node appear slower than our previous V1 results. We have not yet isolated whether this comes from MegaMoEV2 itself, integration/configuration, inter-rank synchronization, or an EPLB/serving interaction.

cc @GwilliamHu — could you help review this investigation and advise which traces or counters would best verify that the expected communication/computation overlap is preserved end to end?

Important comparison caveat

The historical V1 and current V2 results are not yet a clean side-by-side A/B.

Our current 12-run matrix compares the standard and current mega backends under different EPLB policies. It does not directly isolate MegaMoEV1 versus MegaMoEV2. We will add a pinned V1 baseline with the rest of the software stack and serving configuration held constant before concluding that this is a V2 regression.

Test environment

• Single node with 8× AMD Instinct MI355X (gfx950)
• Model: DeepSeek-V4-Pro, A8W4
• 8-rank configuration with DP attention and expert parallelism
• ISL/OSL: 8192/1024
• max-num-batched-tokens=8192
• max-num-seqs=512
• kv_cache_dtype=fp8
• Current integration includes the changes from BugFix for MegaMoE #954
• Exact ATOM, aiter, FlyDSL and image revisions are recorded for every run
• The current aiter integration requires flydsl==0.2.4 because its T.i32() usage is incompatible with FlyDSL 0.3.0. This deviation will remain fixed across comparisons.

Benchmark plan

Current-main characterization consists of 12 runs:

• MoE backends: standard, mega
• EPLB configurations:
  • no redundant experts, naive placement
  • 64 redundant experts, naive placement
  • 64 redundant experts, biased placement
• Workloads:
  • concurrency 512, 5120 prompts, rebalance interval 200
  • concurrency 4096, 8192 prompts, rebalance interval 320
• Each request uses ISL 8192 and OSL 1024.
• Differences within a few percent will be rerun three times and compared by median.

The no-redundancy runs use gpu-memory-utilization=0.85, while the 64-redundant-expert runs require 0.90 to retain KV-cache capacity. Therefore, comparisons between those two groups are not strict single-variable experiments. The r64-naive versus r64-biased comparison is clean.

A second phase will run the same workloads against pinned V1 and V2 revisions while holding all other variables constant.

Data-quality gates

For every run we verify:

• The requested backend is actually active.
• MegaMoE initializes MoriAll2AllManager.
• EPLB uses the expected number of physical experts.
• Rebalancing and migrations actually occur.
• completed == num_prompts and the detailed error list is empty.
• Diagnostic probes that introduce device-to-host synchronization are disabled.
• Server and client file-descriptor limits are raised for high concurrency.

Primary metric: request throughput.

Supporting metrics:

• TTFT for prefill behavior
• TPOT and ITL for decode behavior
• P99 ITL for tail latency
• EPLB migration latency and replicated-expert count

Investigation questions

1. Does serving choose the intended fixed-slot or compact MegaMoEV2 path and an appropriate max_tok_per_rank bucket for each phase?
2. Is communication/computation overlap lost across layers because of stream synchronization, graph capture, or inter-rank skew?
3. Does EPLB migration or changing token distribution interfere with MegaMoEV2's synchronization assumptions?
4. Is the apparent E2E gap decode-driven? With OSL=1024, a decode regression may outweigh the standalone prefill improvement.
5. Are there host-side or device-to-host synchronization points in the serving integration that are absent from standalone kernel benchmarks?

Issue #727 previously reported severe inter-card desynchronization at concurrency 512 and 4096, so rank skew is one hypothesis, but it has not yet been confirmed for this case.

Expected result

Under a controlled V1/V2 comparison, MegaMoEV2 should preserve its standalone gains in end-to-end serving, particularly in TTFT/prefill, without regressing decode throughput or tail latency.

We will attach the complete 12-run results, pinned revisions, and relevant server logs once the matrix finishes.