--stream-experts: intermittent kIOGPUCommandBufferCallbackErrorTimeout (server crash), correlates with cold page cache #216

Summary

With --stream-experts, the server intermittently dies with a Metal GPU watchdog timeout right after prefill starts:

MLX/ErrorHandler.swift:345: Fatal error: [METAL] Command buffer execution failed: Caused GPU Timeout Error (00000002:kIOGPUCommandBufferCallbackErrorTimeout)

It happens right after slot_launch ... prefilling..., and the whole server process exits. The crash is intermittent and correlates with a cold OS page cache.

Environment

• Apple Silicon Mac (M-series), 64 GB RAM, macOS with Xcode 27
• Model: Qwen3.5-35B-A3B-4bit (19 GB, VLM-loaded qwen3_5_moe)
• Command: SwiftLM --model <dir> --port 5413 --stream-experts
• Request: non-streaming /v1/chat/completions, temperature 0, max_tokens 120. The prompt is repeated lines of Item i: the quick brown fox number i jumps over lazy dog i*7. plus a summarise instruction.

Observations (all measured)

• Pre-existing. Reproduced on the pre-bump build (SwiftLM 06bac92) and on current main + mlx-swift 3764829 (mlx core 0.32.3) + mlx-swift-lm with fix: memory auto-cap strategy for SSD MoE streaming + speculative decoding (Issue #72) #77/feat: bump mlx-swift-lm for DeepSeek-V4 support #79. It is not caused by the 0.32 bump or by fix: memory auto-cap strategy for SSD MoE streaming + speculative decoding (Issue #72) #77.
• Streaming-specific. The same 873-token prompt WITHOUT --stream-experts completes in about 10 s with no crash.
• Intermittent and not length-dependent. Crashes at 213, 873 and 104 tokens; passes at 433 and 653 tokens. Once the cache was warm, 7 consecutive runs (213/433/873 tokens) all passed, including the 873-token prompt that had crashed earlier.
• Correlates with a cold page cache.
  • I read a 40 GB scratch file twice to evict the model files. The first 213-token request then crashed after 350 s. The second passed but took 196 s (normal warm: 22-51 s).
  • Earlier crashes were also unusually slow: one 104-token request ran 2m05s before crashing, where warm it takes about 9 s.
  • The controlled cold test is only 1 crash / 1 pass, so this is a strong correlation, not proof.

Repro recipe

1. Start the server with --stream-experts right after evicting the page cache (for example, read a file larger than free RAM twice).
2. Send a ~200-token prompt.
3. The request stalls for minutes and the server then aborts with the GPU timeout.

Root cause

Not identified. Suspects from code reading (all unverified):

1. Lazily mmap'd non-expert weights are first touched by the GPU, so page faults from SSD occur inside a command buffer.
2. LoadSSDExpert::eval_gpu in mlx-swift (Source/Cmlx/mlx/mlx/core/moe_stream_op.cpp) does a blocking synchronous pread (SSDStreamer::load_sync) while the command buffer is being encoded. Prefill touches up to 256 experts per projection per layer, so cold reads could stall long enough to trip the GPU watchdog.
3. The per-expert preadInto / gatherQuantizedMM loop in mlx-swift-lm SwitchLayers.swift (QuantizedSwitchLinear.callAsFunction, ExpertStreamingConfig path).

A follow-up instrumentation experiment (timing load_sync and command-buffer commits) is being run separately. Results will be added as a comment on this issue.

Workarounds

• Warm the page cache before serving, for example cat <model>/*.safetensors > /dev/null.
• Run without --stream-experts if the model fits in RAM.

🤖 Generated with Claude Code