[gspark-etri]

Background

#89: after #83, OffloadedLayer.forward masked and clamped the routing on every call (about 5 extra kernel launches × 48 layers per decode token), padding ids were counted as hits on expert 0, and opt_grouped_mm turned an inf in hidden_states into NaN. This is PR #98 rebased onto current optimus-rm with the review fixes; #98 was closed only because the GPU A/B had no valid run on a busy host. That A/B has now run on a quiet host.

Changes

• accel/moe/layer.py: the owner/local routing tables get one extra entry owned by no tier, so padding ids -1 and num_experts reach no tier with no per-call ops; the device count bins, the Python fast path and the general split skip it; the routing EMA counts ids mod (E+1) and drops the padding bin. The padded host tables are built once per placement. The id contract ([-1, num_experts]) is documented on _padded_tables.
• platforms/cpu/csrc/dispatch.c / native.py: moe_disp_start takes num_experts and skips any id outside [0, num_experts).
• accel/moe/batched.py: opt_grouped_mm starts from zeros connected to both inputs through empty slices (zero gradients for all-padding calls, no NaN from inf).
• Tests: padding not counted, general split with padding, inf hidden with all padding, native dispatcher drops padding and negative ids; the direct dispatcher test from fix: wait for native dispatch jobs when a MoE layer call raises (#170) #179 passes num_experts.

Verification

• pytest tests/test_moe_review.py tests/test_moe_offload.py: 118 passed, 1 skipped (CPU + one A100). Full suite: CI.
• GPU throughput A/B, Qwen3-30B-A3B-Instruct-2507 BF16 on one A100 80GB, moe_offload_generate.py --gpu-budget-gb 20 --kv-cache --max-new-tokens 128, greedy, torch 2.10.0+cu128, transformers 5.1.0. A = optimus-rm 0e69cc9, B = this change (c8f2de8; the MoE/CPU code is byte-identical on this branch, rebased onto aad0ff2). One warmup per side, then 19 alternating A/B pairs. Runs during which another process used at least one full core were discarded (8 runs).

set	A n	A mean / median tok/s	B n	B mean / median tok/s	B vs A (mean)	~95% CI
clean runs	17	11.72 / 11.76	13	12.48 / 12.62	+6.5%	+3.3% .. +9.6%
clean and continuously monitored (r5–r19)	14	11.82 / 11.85	10	12.42 / 12.54	+5.1%	+1.5% .. +8.7%
all runs, nothing discarded	19	11.45 / 11.74	19	12.10 / 12.49	+5.7%	-0.5% .. +11.8%

B was faster in 11 of 13 paired clean runs. Generated text was identical in all 40 runs, and every run routed the same work (36301 GPU / 15923 CPU expert pairs). The gain is less GPU time waiting on the CPU expert tier (2.70 s → 2.53 s per generate).

Compatibility and limits

• Output unchanged; padding is dropped by the routing tables instead of weight masking. Ids must lie in [-1, num_experts]: other negative ids would wrap to a real expert on the torch path, larger ones raise (the native dispatcher skips both).
• The backward (training) path keeps its explicit mask.
• A/B limits: one prompt, batch 1, one GPU budget; tok/s includes prefill; run-to-run spread is about 10% even on a quiet host.

Related Issue

Closes #89. Supersedes #98.

🤖 Generated with Claude Code