test(e2e): add opt-in Qwen3.5-122B CUDA coverage

[shanyulu]

Purpose

Add repository-owned, opt-in CUDA E2E coverage for
Qwen/Qwen3.5-122B-A10B using the existing Qwen3.5/3.6 adapter family.

Scope

• CUDA
• vLLM 0.26.0 V1
• BF16
• text-only
• eager execution
• native DP4/TP1/EP4 baseline
• synchronous P2pNcclAFDConnector
• AFD 4A4F
• natural routing
• GSM8K-7 with eight-shot prompting
• eight-GPU hardware profile

The checkpoint declares num_experts=256 and num_experts_per_tok=8.

Test design

The profile is explicitly opt-in and requires:

AFD_E2E_BACKEND=gpu
AFD_E2E_LARGE_MODEL=1
AFD_GPU_E2E_MODEL=/path/to/Qwen3.5-122B-A10B
AFD_E2E_DEVICES=0,2,4,6,1,3,5,7

It never automatically downloads the approximately 234 GiB checkpoint and is
not wired into the default Buildkite PR/merge gate.

Cases:

• baseline-eager: native DP4/TP1/EP4 on devices 0,2,4,6
• afd-eager-4a4f: Attention DP4/TP1 on 0,2,4,6 and FFN DP4/TP1/EP4 on
  1,3,5,7

Correctness child environments remove benchmark-only forced-routing variables.

Validation

Exact candidate SHA: 9120b20b9a707e39d436dc8afce99ebea9c7514f

Hardware:

• 8x NVIDIA RTX PRO 6000 Blackwell Server Edition
• 97,887 MiB/card
• Driver 595.71.05
• CUDA 13.2
• PCIe topology without NVLink

Software:

• Python 3.12.3
• PyTorch 2.11.0+cu130
• vLLM 0.26.0
• Transformers 5.8.1

Results:

• Native GSM8K strict match: 7/7
• AFD GSM8K strict match: 7/7
• Native and AFD runner exit code: 0
• No repository-runner cleanup failure
• No residual AFD/vLLM process
• Ports released
• GPU memory returned to 0 MiB
• Full CPU-safe suite passed
• Complete pre-commit passed

Runtime observation: The AFD run emitted vLLM-internal worker shutdown
escalation and Python resource_tracker warnings during process teardown. The
repository E2E runner did not trigger its forced-SIGKILL failure path, exited
successfully, released all ports and GPU allocations, and left no residual
AFD/vLLM processes.

Performance

Performance was not evaluated; this PR makes no throughput or latency claim.

Limitations

• Requires eight CUDA devices with sufficient memory.
• Manual/opt-in only.
• Eager-only.
• Graph and DBO are out of scope while [Bug]: FFN graph cache replays during eager prefill in FULL_DECODE_ONLY #261/fix: gate GPU FFN graph replay on Attention run mode #270 remain unresolved.
• Async, multi-node, quantized, and multimodal paths are not covered.

Relationship

• [RFC] Experimental Qwen3.5 MoE AFD support: CUDA correctness baseline #179 — early Qwen3.5 CUDA correctness context
• Add Qwen3.6 MoE CUDA AFD implementation support #181 — Qwen3.5/3.6 CUDA implementation
• test(e2e): add Qwen3.6 MoE CUDA coverage #256 — existing Qwen3.6 adapter-family CUDA E2E
• [Bug]: FFN graph cache replays during eager prefill in FULL_DECODE_ONLY #261 — graph correctness issue, outside this eager-only PR
• fix: gate GPU FFN graph replay on Attention run mode #270 — active graph fix, not consumed by this PR

[jiangkuaixue123]

Thanks for adding the Qwen3.5 coverage! Do you have any performance data you could share, such as throughput and latency comparisons between the native baseline and AFD 4A4F?

[shanyulu]

Thanks for adding the Qwen3.5 coverage! Do you have any performance data you could share, such as throughput and latency comparisons between the native baseline and AFD 4A4F?

wait