Perf: compiled GeGLU product for Gemma4 MoE experts (remaining piece of mlx-swift-lm#27) #236

Tracks what is still worth taking from the closed-for-now draft SharpAI/mlx-swift-lm#27 (2026-04, 8 commits, conflicting with main). Issues are disabled on mlx-swift-lm, so this lives here.

What #27 set out to do

1. GDN: fp32 recurrent state and asyncEval chunk pipelining in prefill (DFlash: generation ignores EOS and outlives the stream; the next request overlaps it and crashes the server (flaky dflash-speculative-decoding job) #224, fix: stop DFlash at end-of-sequence, and cancel it when the stream ends early #225)
2. Fuse silumul / gelumul into one compiled Metal dispatch (compiledSwiGLU, compiledGeGLU)
3. Review follow-ups: keep SwitchGLU intermediates in the original dtype (a gatherQuantizedMM NaN on pre-M4 GPUs), contiguous-array signatures to dodge an MLX Swift segfault, no unconstrained .float32 promotion in GELU

Already on main (checked against SharpAI/mlx-swift-lm main 4d76a2e and ml-explore/mlx-swift-lm main 48114ee)

• (1) is covered: GatedDelta.swift keeps state in .float32, and LLMModel.swift prefill pipelines chunks with asyncEval.
• The SwiGLU half of (2) is covered: SwitchLayers.swift has compiledSiluProduct and SwitchGLU.activationProduct; upstream also has weightedExpertSum and Gemma4Fusions.swift (compiled softcap).

Still missing

• Gemma4 MoE experts use an uncompiled GeGLU. Gemma4Text.swift builds SwitchGLU(..., activation: geluApproximate), and the SwitchGLU forward paths compute activation(xGate) * xUp as two dispatches. Only the per-layer-input gelu*mul (_geluMul) is compiled. A compiled gelu-product for the MoE path is the remaining piece of test: add speculative decoding E2E test to CI pipeline #27's fusion idea.

Before doing it

• Measure first: tokens/s and prefill time on gemma-4 MoE (e.g. 26b-a4b) with and without the fused product. test: add speculative decoding E2E test to CI pipeline #27 shipped no numbers.
• Re-check the pre-M4 gatherQuantizedMM NaN claim on an M1/M2 machine before copying the dtype downcast; it may no longer reproduce on mlx 0.32.3.
• Do it on current main, not by rebasing the old branch (it is based on a 2026-04-19 tree and touches files that moved, e.g. GatedDelta.swift is now in MLXLMCommon).
• Drop the stray run_test.py and test_gemma4_crash.swift from the old branch.