sparse-upcycling

Here are 2 public repositories matching this topic...

tardellirs / gemma3moe-builder

Sparse-upcycle a dense Gemma-3-270m into a Mixture-of-Experts (gemma3moe) and serve it on llama.cpp (CPU). Upcycle, train, inspect routing, GGUF.

transformers moe gemma upcycling mixture-of-experts cpu-inference llama-cpp small-language-model gemma3 sparse-upcycling

Updated Jul 27, 2026
Python

outlast85 / moe-upcycle

Star

Turn a small dense LLM into a Mixture-of-Experts model, then specialize the experts by distilling from a teacher. Reproducible toolkit + honest benchmarks (Qwen2.5-0.5B demo).

pytorch moe distillation mixture-of-experts model-efficiency llm qwen sparse-upcycling

Updated Jun 18, 2026
Python

Add this topic to your repo

To associate your repository with the sparse-upcycling topic, visit your repo's landing page and select "manage topics."

Learn more