Local Inference Lab

Popular repositories Loading

1. rtx6kpro rtx6kpro Public

   RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

   Python 1.1k 87

2. b12x b12x Public

   Python 272 71

3. llm-inference-bench llm-inference-bench Public

   LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.

   Python 113 17

4. blackwell-llm-docker blackwell-llm-docker Public

   Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)

   Python 83 23

5. vllm vllm Public

   Forked from vllm-project/vllm

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 41 34

6. quant-toolkit quant-toolkit Public

   Python 17 9