I'm learning and building around AI infrastructure, with a focus on LLM inference.
Currently exploring inference engines, serving, scheduling, batching, KV cache, routing, GPU kernels, quantization, and performance.
- vllm-project/vllm — ⭐ 91690 · Python · A high-throughput and memory-efficient inference and serving engine for LLMs
- sgl-project/sglang — ⭐ 35928 · Python · SGLang is a high-performance serving framework for large language models and multimodal models.
- FlashML-org/FreeToken — ⭐ 12743 · Python · FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
- LMCache/LMCache — ⭐ 11793 · Python · LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
- xorbitsai/inference — ⭐ 9566 · Python · Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
- oumi-ai/oumi — ⭐ 9384 · Python · Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!
- kvcache-ai/Mooncake — ⭐ 6567 · C++ · Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
- vllm-project/semantic-router — ⭐ 5789 · Go · A programmable Mixture-of-Models router for heterogeneous LLM inference
- gpustack/gpustack — ⭐ 5684 · Python · A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
- OpenCSGs/csghub — ⭐ 4106 · Vue · CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️
- PaddlePaddle/FastDeploy — ⭐ 3715 · Python · High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
- mozilla-ai/any-llm — ⭐ 2192 · Python · Communicate with an LLM provider using a single interface
- RightNow-AI/picolm — ⭐ 2040 · C · Run a 1-billion parameter LLM on a $10 board with 256MB RAM
- theopenco/llmgateway — ⭐ 1633 · TypeScript · Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.
- xLLM-AI/xllm — ⭐ 1568 · C++ · A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
- alibaba/rtp-llm — ⭐ 1334 · Python · RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
- ddalcu/mlx-serve — ⭐ 1256 · Zig · Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.
- NVIDIA/kvpress — ⭐ 1209 · Python · LLM KV cache compression made easy
- sgl-project/sglang-omni — ⭐ 1180 · Python · SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
- jmaczan/tiny-vllm — ⭐ 1109 · C++ · Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Learning in public, one inference system at a time.