[gspark-etri]

Background

#6's remaining criteria are an RNGD vs CPU gradient check on Qwen3-30B-A3B (npu-lora check, 2 layers) and a step-time split between NPU and host. With the current check, Qwen3-30B fails:

• the loss matches (rel. err 3.6e-4);
• but layer gradients differ by 3.2–8.9% (cos 0.996–0.9995), above the 5% bound.

For comparison, dense Qwen3-0.6B passes (worst 3.7%).

The cause is routing, not numerics. With identical inputs and adapters, the top-8 expert set differs between the RNGD and CPU runs for 12 / 128 tokens in layer 0 and 20 / 128 in layer 1. Each flip changes that token's whole MoE output and its gradient.

Changes

• npu_lora/moe.py: RoutingPins.
  • Records each HostMLP router's top-k ids, or forces recorded ids.
  • When pinned, the weights are the router's own softmax at the pinned ids, renormalized like the router (Qwen3/Qwen2-MoE/Mixtral), so gradients still flow through the router.
  • flips() counts the tokens whose expert set differs.
• npu-lora check. On MoE checkpoints the CPU reference is pinned to the NPU run's experts, and the result also reports routing_flips_unpinned from an extra unpinned CPU pass. --free-routing keeps the old comparison.
• LoRATrainer.train_step. On MoE checkpoints it also reports host_moe_s (HostMLP forward + backward) and device_and_transfer_s.
• Test test_routing_pins_record_and_force_experts.

Verification

RNGD 2026.4.0, furiosa-torch 2026.2.0, Qwen3-30B-A3B-Instruct-2507, --layers 2 --seq-len 128:

check	loss rel. err	worst grad rel. err (cos)	result
free routing (old)	3.6e-4	8.9% (0.996)	fail
pinned routing	7.3e-5	2.5% (0.99969)	pass
dense Qwen3-0.6B (reference)	4.6e-4	3.7% (0.99934)	pass

Step-time split, LoRA, seq 128, 3 steps on RNGD chip 0. The host load average was 35–43 from other users' jobs, so the host share is inflated:

step	total	host MoE	device + transfer
1	196 s	191 s	4.5 s
2	289 s	281 s	7.7 s
3	279 s	274 s	5.1 s

The host MoE blocks dominate (97–98%), so the next lever is the host MoE path or hot experts on the device (#14).

CPU tests: the full suite passes.

Compatibility and limits

• check on dense models is unchanged.
• On MoE checkpoints it now pins by default; --free-routing restores the old comparison.
• The step-time numbers come from a shared, loaded host.

Related Issue

Closes #6: functional equivalence and the real-checkpoint run were already merged (#16), and this adds the RNGD gradient check and the step split.

🤖 Generated with Claude Code

[gspark-etri]

Follow-up: on transformers > 5.1 the router returns logits, and RoutingPins used them as probabilities (fixed in #39). The new test also never ran in CI because it sat below the __main__ runner (fixed in #39). The pinned-routing row here should be re-measured if this run used transformers later than 5.1.