exllamav3

A local app that rents a GPU in your own Colab account and serves Qwen3.8 on it: 27B on an A100-40G, or Flash-Next (125B-A6B MoE) on an A100-80G. The cost is shown before it starts; chat with images; Codex or any OpenAI client uses a fixed local /v1. Auto-stop, CU ledger, live prefill/decode. Windows, macOS, Linux.

Serve EXL3 (ExLlamaV3 trellis) quantized models on vLLM fork runtimes — any architecture, mixed per-layer bitrates, composable with source-format non-routed weights

Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090

Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on disk; the experts run on CPU.

LLM inference server for ExLlamaV3 / EXL3, with OpenAI- and Anthropic-compatible APIs optimized for Agent workloads.

LLM / AI inference server for Windows + NVIDIA (EXL3/ExLlamaV3). OpenAI-compatible API, multi-GPU, Blazor admin. Ollama-like, no Docker.

Windows dual-16GB GPU experiments for Qwen3.8-Flash-Next EXL3: workload-aware MoE placement, 64K context, MTP, benchmarks and agent integrations.

Serve Qwen3.8-Flash-Next (uncensored EXL3) on one NVIDIA DGX Spark: one Docker image, OpenAI-compatible endpoint, MTP speculative decoding, 262k context. ~80 tok/s.

Run Qwen3.8-Flash-Next EXL3 (turboderp pack) on one AMD Strix Halo / Framework Desktop (gfx1151) via the exllamav3-amd runtime fork. 41 tok/s mean, 47.5 peak, MTP self-speculation. Setup encodes the five ROCm/gfx1151 traps.

A llama-server-compatible HTTP front for ExLlamaV3: /props, /health, /slots and /v1/chat/completions with llama-server's timings object, so llama.cpp tooling drives an EXL3 model unchanged.

L0xRE static expert offload runtime for ExLlamaV3 — part of the L0xRE family (hub: seanyourhighness/L0xRE)

Blackwell (sm120) build of Mia'a AI Lab's Qwen3.8-27B EXL3 deployment kit. Measured on one RTX 5090: 193 tok/s decode, full native 262k context in 25.4 GiB.

Terminal dashboard for TabbyAPI + ExLlamaV3 servers: throughput, KV cache, latency, speculative decoding and GPU stats

Run BLACKFROST 3.8 Flash DERISKED EXL3 3.05 BPW on one 96GB RTX PRO 6000 with vLLM, ExLlamaV3 and MTP.

Fast OrcaSAQ2-27B inference on AMD Radeon RDNA3/RDNA4 (RX 7900 XTX, R9700): exllamav3 fork with rewritten EXL3 kernels, MTP speculative decoding, TabbyAPI setup

GLM-5.3-Flash EXL3 on 2x GB10 (ASUS GX10 + DGX Spark), dual-rail: tuned, measured recipe on MiaAI-Lab's kit

Reproducible ExLlamaV3 recipe and optimization log for Qwen3.8-Flash-Next on 3x RTX 5070 Ti 16GB

Ollama-style CLI for ExLlamaV3 + TabbyAPI: pull, list, fit and rank local models with VRAM-aware auto-fit context for consumer NVIDIA GPUs. Serve and chat commands coming.

FastAPI LLM pool for local and OpenAI-compatible remote inference, with multimodal input, scheduling, replicas, metrics, and admin APIs.