I'm learning and building around AI infrastructure, with a focus on LLM inference.

Currently exploring inference engines, serving, scheduling, batching, KV cache, routing, GPU kernels, quantization, and performance.

Open the live GitHub search

• vllm-project/vllm — ⭐ 91477 · Python · A high-throughput and memory-efficient inference and serving engine for LLMs
• sgl-project/sglang — ⭐ 35802 · Python · SGLang is a high-performance serving framework for large language models and multimodal models.
• FlashML-org/FreeToken — ⭐ 12419 · Python · FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
• LMCache/LMCache — ⭐ 11753 · Python · LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
• xorbitsai/inference — ⭐ 9566 · Python · Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
• oumi-ai/oumi — ⭐ 9383 · Python · Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!
• kvcache-ai/Mooncake — ⭐ 6553 · C++ · Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
• vllm-project/semantic-router — ⭐ 5728 · Go · A programmable Mixture-of-Models router for heterogeneous LLM inference
• gpustack/gpustack — ⭐ 5667 · Python · A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
• OpenCSGs/csghub — ⭐ 4106 · Vue · CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️
• PaddlePaddle/FastDeploy — ⭐ 3715 · Python · High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
• mozilla-ai/any-llm — ⭐ 2189 · Python · Communicate with an LLM provider using a single interface
• RightNow-AI/picolm — ⭐ 1933 · C · Run a 1-billion parameter LLM on a $10 board with 256MB RAM
• theopenco/llmgateway — ⭐ 1625 · TypeScript · Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.
• xLLM-AI/xllm — ⭐ 1567 · C++ · A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
• alibaba/rtp-llm — ⭐ 1333 · Python · RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
• ddalcu/mlx-serve — ⭐ 1224 · Zig · Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.
• NVIDIA/kvpress — ⭐ 1207 · Python · LLM KV cache compression made easy
• sgl-project/sglang-omni — ⭐ 1158 · Python · SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
• jmaczan/tiny-vllm — ⭐ 1104 · C++ · Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

_{Learning in public, one inference system at a time.}