Skip to content

ggml-cuda: skip duplicate decode sibling prefetch - #43

Draft
GenerelSchwerz wants to merge 1 commit into
moe-cache-experimentalfrom
codex/moe-cache-decode
Draft

ggml-cuda: skip duplicate decode sibling prefetch#43
GenerelSchwerz wants to merge 1 commit into
moe-cache-experimentalfrom
codex/moe-cache-decode

Conversation

@GenerelSchwerz

Copy link
Copy Markdown
Owner

Summary

Skip redundant sibling prefetch passes during single-token MoE decode after the routing-ID cache identifies later siblings. Preserve existing prefill behavior.

Validation

  • test-moe-cache: passed
  • Untouched dbeb350d1 and candidate PPL: 1.0018 +/- 0.00011
  • Deterministic raw token IDs and token accounting matched base at 8K and 32K
  • Cache misses, evictions, H2D copies, and VRAM matched base
  • 48-slot decode: +0.24% at 8K, +0.49% at 32K
  • Equal-VRAM decode: neutral at 8K/188 slots and 32K/184 slots
  • Combined with routing-ID reuse at 0c5347236, prefill telemetry matched base and 48-slot decode was +0.31% at 8K and +0.15% at 32K
  • All server lifecycles stopped cleanly

AI assistance: Codex assisted with implementation, profiling, and validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants