NInfer-all

A C++/CUDA inference engine for the Qwen3.5 model family (Qwen3.6 and Qwen3.8 27B, Qwen3.6-35B-A3B, their fine-tunes and Ternary Bonsai 2), Qwen3.8-Flash-Next and Gemma 4 26B-A4B, on NVIDIA GPUs from Volta to consumer Blackwell, from 8 GB of memory. It serves the OpenAI, Anthropic and llama.cpp APIs from one GPU, or from up to eight as pipeline stages on Linux.

Quick start · Supported GPUs · Models · Highlights · Features · Docker · Building · Documentation

NInfer-all continues NInfer through the NInfer-3090 line and carries the work of several forks; CONTRIBUTORS.md credits each of them.

Quick start

Download an artifact (all models) and serve it from the Docker image. The server speaks the OpenAI and Anthropic APIs on http://localhost:8080/v1, and each card picks up its device profile on its own.

hf download WaveCut/Ternary-Bonsai-2-27B-NInfer-v3 Ternary-Bonsai-2-27B-ninfer-v3.ninfer --local-dir models

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Ternary-Bonsai-2-27B-ninfer-v3.ninfer \
  --model-id bonsai2-27b --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 \
  --gdn-state-fp16 --spec dflash2 --draft-tokens 5

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"bonsai2-27b","messages":[{"role":"user","content":"Hello"}]}'

Flags for every model and memory size, from an 8 GB card to the largest context: launch recipes.

Without Docker — the same flags on a build

ninfer-serve models/Ternary-Bonsai-2-27B-ninfer-v3.ninfer --model-id bonsai2-27b \
  --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec dflash2 --draft-tokens 5

A build listens on 127.0.0.1:8080 unless --host and --port say otherwise. Building shows how to make one.

Supported GPUs

NVIDIA GPUs from Volta (compute capability 7.0) to consumer Blackwell (12.0), from 8 GB of memory: Ternary Bonsai 2 27B runs on an 8 GB RTX 3060 Ti, and Qwen3.8-Flash-Next streams its experts from host memory or disk through 8 GB. The Docker image carries the 75, 86 and 120a builds and starts the one that matches the GPU. Every card by name, with what has run on it: Supported GPUs.

Click a section to expand it.

Architectures and builds — six architectures, one build each

Architecture	Compute capability	Build	CUDA	In the image	Status
Blackwell: GeForce RTX 50, RTX PRO Blackwell	12.0	120a	13.1 or newer	yes	measured; built-in device profiles
Ada Lovelace	8.9	89, or the 86 build	12.8 or newer	the 86 build	measured; built-in device profiles
Ampere	8.6	86 (the default)	12.8 or newer	yes	measured (the reference card); built-in device profiles
Ampere, A100 class	8.0	80	12.8 or newer	no	built-in device profiles; not benchmarked
Turing	7.5	75	12.8 or newer	yes	measured; built-in device profiles
Volta	7.0	70	12.8 or 12.9	no: source build	measured; built-in device profiles
Hopper; data-center Blackwell; GB10; Pascal and older		none			no build

What fits in your GPU memory — measured, from 8 GB to 48 GB

GPU memory	Runs	Measured on
8 GB	Ternary Bonsai 2 27B (24,576 tokens with rk4v4, no MTP); Qwen3.8-Flash-Next with its experts in host memory or on disk	RTX 3060 Ti
11 GB	Ternary Bonsai 2 27B (200,704 tokens with rk4v4; 143,360 with MTP)	RTX 2080 Ti
12 GB	Ternary Bonsai 2 27B (266,240 tokens); Qwen3.8-27B GSQ-RCO IQ3_S (8,192 tokens, no MTP)	RTX 3060
16 GB	Gemma 4 26B-A4B Q4_0 (61,440 tokens); Qwen3.8-27B IQ3_S (241,664 tokens); the official Qwen3.8-27B with --embedding-q4 --lm-head-q4	Tesla V100-SXM2-16GB
24 GB	the official Qwen3.8-27B and its MXFP8 tune; Qwen3.6-35B-A3B groupwise-int on an RTX 3090; Flash-Next with every expert on the GPU over two cards	RTX 3090, RTX 4090, TITAN RTX
32 GB	Qwen3.6-35B-A3B NVFP4 (Blackwell only)	RTX 5090
48 GB	Flash-Next Q2_0 with every expert on one card	L40S, RTX PRO 6000

Device profiles and limits — how a card gets its kernel routes

• Device profiles. A profile picks the kernel route of every operation and width. The binary carries profiles measured on 47 GPU models, from the Tesla V100 and RTX 2060 Super to the RTX PRO 6000 (the Profile column of supported GPUs); a built-in profile applies to the same GPU name, compute capability and SM count. Any other card is calibrated once at first start, which takes about half a minute to three and a half minutes, and the result is stored for later starts; ninfer-calibrate measures again after a driver or clock change (device profiles). The 70 build runs its own routes, so the V100 profiles hold only the GDN prefill switch and, for the 32 GB card, the Q8 MoE prefill frontier.
• Architecture limits. NVFP4 expert weights need a 120a build. The Turing and Volta builds replace instructions their GPUs lack and reshape kernels to their shared memory (Turing, Tesla V100). Configure accepts no other target, so Pascal and older, Hopper and data-center Blackwell have no build.
• Speed per card. Performance lists the measured rates of every card.

Artifacts

Each name links to its Hugging Face card; hf download REPO FILE.ninfer --local-dir models fetches the file. The Bonsai, GSQ-RCO, Flash-Next and Gemma 4 artifacts use formats that only this engine reads. ✓ means the artifact carries the component, — that it does not.

Model	Size, GiB	Weights	Vision	MTP	DFlash2	Proposal head	Notes
Ternary Bonsai 2 27B	8.87	ternary, 2.125 bits	✓	✓	✓	✓	Bonsai-trained MTP head and DFlash2 adapter
Qwen3.8-27B	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the official artifact; the reference tables use it
Qwen3.8-27B GSQ-RCO IQ3_S	13.99	GGUF blocks, 3.5 bits	✓	✓	✓	✓	ISTA-DASLab's GGUF blocks byte for byte, Q6_K MTP head
Qwen3.8-27B, abliterated	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the official recipe
Qwen3.8-27B Uncensored, HauhauCS Aggressive	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the tune's MTP head
Qwen3.8-27B MXFP8-CRACK	16.96	groupwise-int	✓	✓	—	✓
Qwen3.6-35B-A3B NVFP4	20.39	NVFP4 experts, Q8 projections	✓	✓	—	✓	RedHatAI's NVFP4 experts code for code; sm_120a GPUs only
Qwen3.8-Flash-Next GSQ-RCO Q2_0	38.49	GGUF blocks, 2.4 bits	✓	✓	—	—	shared-Q8_0 MTP; reads the n-gram table
Qwen3.8-Flash-Next GSQ-RCO IQ3_S	54.50	GGUF blocks, 3.5 bits	✓	✓	—	—	shared-Q8_0 MTP; reads the n-gram table
Qwen3.8-Flash-Next Coder GSQ-RCO IQ1_M	31.02	GGUF blocks	✓	✓	—	—	expert-pruned coding build, 256 experts per layer; shared-Q8_0 MTP; reads the n-gram table
Gemma 4 26B-A4B it QAT	14.75	GGUF Q4_0 blocks	✓	✓	—	—	Google's QAT GGUF byte for byte; the MTP column is Google's QAT assistant

The three Flash-Next releases read one shared n-gram table, which the model names by digest: the n-gram table (26.92 GiB, unchanged IQ4_NL rows plus an optional broad hot-row profile), passed with --ngram-table.

The official NInfer artifacts (Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-27B, groupwise-int and nvfp4) load too; the documentation map links them. Weight conversion shows how the Bonsai and GSQ-RCO artifacts are built.

Highlights

Ternary Bonsai 2 27B decodes 397 tok/s on an RTX 5090 with DFlash2 and holds a 978,944-token context with every planted code found. Qwen3.8-Flash-Next decodes 138 tok/s with every expert on one RTX PRO 6000, and 74 tok/s with its experts in host memory on one RTX 5090. Every table of every card is on the performance page.

Click a section to expand it.

Ternary Bonsai 2 27B — up to 397 tok/s, a million tokens of context

Measured in September 2026, one card each, greedy, one request unless the column says otherwise.

Card	Short chat, DFlash2 7 drafts (tok/s)	After a 261K-token document (tok/s)	261K-token prompt, to the first token	Largest context, all three codes found	Eight requests at once, MTP 3 (tok/s)
RTX 3090	202	90	215 s	970,752	551
RTX 4090	256	123	102 s	958,464	824
RTX 5090	397	218	82 s	978,944	1,063
RTX PRO 6000	381	218	78 s	1,048,576, two of three codes	1,155

The RTX PRO 6000 (96 GB) starts at the engine's ceiling of 1,048,576 tokens with every KV format and drafter. Filled to it, the model misses the code at 90% (about 943K); up to about 880K it found every code on every card.

Qwen3.8-27B — up to 237 tok/s, 872,448 tokens of context filled

The official groupwise-int artifact, measured in September 2026, one card each, greedy, one request unless the column says otherwise.

Card	Short chat, DFlash2 7 drafts (tok/s)	Largest context, all three codes found	Eight requests at once, MTP 3 (tok/s)
RTX 3090	118	417,792	329
RTX 4090	149	405,504	442
RTX 5090	236	872,448	690
RTX PRO 6000	237	1,048,576, two of three codes	739

• RTX PRO 6000 against RTX 5090. Re-measured in October beside an RTX 5090, both at 600 W, the PRO 6000 came back within 2% of September wherever the drafts accepted the same share. One request's decode is bound by memory bandwidth, and both cards have 1.79 TB/s of GDDR7 (Bonsai 2 without speculation: 167.7 against 170.4 tok/s). The PRO 6000's 188 SMs against 170 show where compute decides: the 261K prompt is 2% faster and eight requests at once 6 to 7% faster (re-check).
• Draft length. DFlash2 with seven drafts is fastest on short answers; after long documents the best count is three to seven. MTP runs up to fifteen drafts and is fastest at three to five.

Qwen3.8-Flash-Next — 138 tok/s with the experts on a GPU, 74 tok/s from host memory

The Q2_0 release (35.9 GiB model and 26.8 GiB n-gram table), one request, greedy, October 2026. Decode of a short answer, in tok/s:

Card	Every expert on the GPUs	Experts in host memory	Experts on disk, in the page cache	Experts on disk, cold
RTX PRO 6000	138 (one card)	55	76	35
RTX 5090	136 (two cards)	74	68	25
RTX 4090	97 (two cards)	53	42	18
RTX 3090	90 (two RTX 3090 Ti)	49	47	17

Prefill of a 4,463-token prompt, in tok/s:

Card	Every expert on the GPUs	Experts in host memory	Experts on disk, in the page cache	Experts on disk, cold
RTX PRO 6000	2,939	1,297	2,024	1,000
RTX 5090	3,809	1,677	1,739	694
RTX 4090	3,608	1,138	723	160
RTX 3090	1,504	842	630	195

• The host matters. Host and disk rows depend on the host as much as on the card. The RTX 5090 host had PCIe 5.0 x16; the others had PCIe 4.0 x16. The RTX 4090 rows ran pinned to the GPUs' NUMA node in a two-socket VM; unpinned, host experts decode there at 40 tok/s and disk experts at 32 to 33.
• Older host and disk rows. They were measured before the October 9 decode work. Host experts on an RTX 3090 now decode 87-94 tok/s in a different workload, 98-100 with a repeated prompt (host experts on one RTX 3090).
• Other releases and cards. IQ3_S, the Coder IQ1_M build and the 8 and 12 GB cards are in the expert placements. Six reasoning requests at once on two RTX 5090s produced 193 tok/s of output with Q2_0.

Features

Click a section to expand it.

Models and weight formats — Qwen3.5 family, GGUF block formats, Ternary Bonsai 2, Flash-Next, Gemma 4

• Qwen3.5 family. Dense and MoE (Qwen3_5ForCausalLM, Qwen3_5MoeForCausalLM) with MTP, DFlash (35B-A3B) and DFlash2 (Qwen3.8-27B) drafters, Vision for images and video, and groupwise-int, FP8, NVFP4 and ternary weights. Native MoE routed experts can stay in pinned host memory or the artifact's files, brought into whole-layer GPU banks with an optional device cache (expert residency). NVFP4 expert banks prefill through W4A4 and need a 120a build.
• GGUF block formats. Releases that pick a ggml type per tensor, such as ISTA-DASLab's GSQ-RCO, convert without requantization and multiply their blocks in place. The 3.5-bit Qwen3.8-27B IQ3_S release holds 10.95 GiB of weights instead of 15.9, scores WikiText-2 perplexity 7.071 (the official artifact: 7.286) and decodes at 59.9 tok/s against the official artifact's 40.3 on an RTX 3090 (GGUF block formats).
• Ternary Bonsai 2 27B. PrismML's ternary Qwen3.8-27B runs from t2_g128_fp16 weights: 2.125 bits per weight, imported without rounding, with the Hadamard rotations fused into the norms and gates and integer activations for its projections. The bonsai2_27b_ternary recipe adds ProCreations' MTP head and DFlash2 adapter and an exact proposal head.
• Qwen3.8-Flash-Next. The 125B-parameter MoE (512 experts, about 6B active) from the GSQ-RCO GGUF releases, with its experts on the GPUs, in pinned host memory with a GPU cache (--expert-residency host) or in the artifact's files read into a GPU cache (--expert-residency disk, under 1 GB of RAM). It serves up to eight requests with prefix reuse, structured output, images and video, and MTP drafting. On two RTX 5090s the Q2_0 release scores 93.3% on AIME 2025 and 86.4% on GPQA-Diamond (Qwen3.8-Flash-Next).
• Gemma 4 26B-A4B. Google's QAT Q4_0 GGUF release, imported block for block (13.4 GiB), or a mixed-format GGUF of it: the sliding-window and full-attention blocks (the latter caching values only), the dense MLP beside 128 routed experts, every --kv-dtype format, up to eight concurrent requests with prefix reuse through host memory and shared system prompts, tool calls and structured output, images and video through the release's vision tower (on the GPU, streamed from pinned memory or on CPU threads), and Google's multi-token-prediction assistant for speculative decoding. On an RTX 3090 it prefills a 10,000-token prompt at 6,045 tok/s and decodes at 215 tok/s, 190 after 10,000 tokens, against llama.cpp's 4,697, 169 and 152 on the same GGUF; with two drafts a round chat answers decode at 291 tok/s instead of 209 (Gemma 4).

Speculative decoding — MTP, DFlash, DFlash2 and n-gram copying

• MTP, DFlash and DFlash2 with 1 to 15 drafts; the artifact's proposal head drafts by default when it stores one. Under MTP and DFlash a sampled request keeps a draft only while it equals the model's own keyed draw, so every emitted token is the model's sample; DFlash2 verifies its proposal distribution by rejection sampling.
• Gemma 4 drafts with Google's multi-token-prediction assistant (--spec mtp, 1 to 7 drafts), four blocks that read the model's own KV cache; its verification rows are bit for bit plain decode rows, so the output is the one plain decoding gives.
• --adaptive-mtp picks each round's width from measured draft survival and round cost.
• N-gram copy drafting beside the neural drafter (on with --spec) verifies up to 15 tokens copied from earlier prompt, tool-result or output text (ngram copy proposals).

Context and memory — nine KV formats, up to 1,048,576 tokens, a context cache with a disk tier

• Nine KV formats, from bf16 to rk2v4-e8 at 216 bytes per token and KV head; rk8v4 is the all-round choice and rk4v4 holds the native 262,144-token window of Qwen3.8-27B on a 24 GB card (context and memory).
• Up to 1,048,576 tokens of context, four times the native window, with plain RoPE or YaRN (--rope-yarn).
• A hybrid prefix cache by default, which shares content-addressed KV blocks and sparse state snapshots across requests on the device and in pinned host memory; or a checkpoint catalog (--use-original-prefix-caching) with an optional disk tier that survives restarts (--disk-kv-path, --disk-kv-restore) and endpoint and branch anchors that let an edited or regenerated turn resume where it left the conversation.
• Precision trades that buy context: --embedding-q4, --lm-head-q6, --gdn-state-fp16 (quality trades).

Serving — OpenAI, Anthropic and llama.cpp APIs, model suspend, several models behind one server

• OpenAI Responses and Chat Completions, Anthropic Messages, llama.cpp's /completion, /tokenize, /detokenize, /apply-template, /v1/rerank, /metrics, /slots and /props, streaming, tool calls, structured output through xgrammar (--structured-output), token log probabilities and prompt grafts (HTTP serving).
• Model suspend. --model-suspend lets an idle server give its device memory back without exiting and take it again on the next request, retained conversations included. Ternary Bonsai 2 27B on an RTX 3090: 7.9 GiB down to 0.3 GiB in 0.33 s, back in 1.1 s (model suspend).
• Several models behind one server. With --models-dir or a llama.cpp-style --models-preset, ninfer-serve is a router with llama.cpp's model API; with --model-suspend a model sleeps instead of unloading: two 27B models on one 24 GB RTX 3090 swap in 1.8 s instead of an 11.6 s cold load (several models).
• GET /stats and a monitoring dashboard, request logs with rotation, and an optional compiled-in WebUI.

Kernels and devices — route profiles per card, pipeline stages, FP8 and NVFP4 tensor-core routes

• Device route profiles. Each card's measured profile picks the kernel route for every operation and width: built in for the cards in supported GPUs, measured once at first start on any other (device profiles).
• Pipeline stages. --devices 0,1,... splits the layers over up to eight GPUs on Linux, each stage owning its weights, KV and state; the cards may differ where the build runs on each, such as RTX 30 with RTX 40 cards on the 86 build (pipeline stages).
• FP8 A8 and NVFP4 W4A4 tensor-core routes on every 120a build, an FP4 tensor-core prompt kernel for an NVFP4 KV cache, and programmatic dependent launches on Blackwell.

Docker

ghcr.io/iamwavecut/ninfer-all:latest is built from the newest master commit that passes CI, on CUDA 13.4 and Ubuntu 26.04, and is 1.6 GB compressed. It carries the sm_75, sm_86 and sm_120a builds and starts the one that matches the GPUs. It needs an NVIDIA driver of the CUDA 13 branch (580 or newer) and the NVIDIA Container Toolkit. ghcr.io/iamwavecut/ninfer-all:multi carries one build of all three architectures instead, for a layer pipeline (--devices) across cards of different architectures, where each card runs its own kernels; given such cards, latest runs the oldest card's build on all of them.

Commands, volumes and tags — what the container runs

• Tags. latest, sha-<commit> and the VERSION. A commit that changes only documentation or tooling the image does not carry keeps the image before it. multi and sha-<commit>-multi are the multi-architecture image, published apart from CI after each master push.
• Volumes. /models holds artifacts, /cache the device profile measured on first start, and /grafts/<model>/ optional prompt grafts.
• Checked on an RTX 3090 and an RTX 5090 with driver 580.159.03.

The container's command chooses what runs:

command	runs
serve, ninfer, perplexity, calibrate [args]	that binary with any artifact and flags; serve listens on 0.0.0.0:8080 unless given --host or --port
run <model> [profile] (the default: run qwen38-27b)	the launcher profiles of scripts/run.sh for the official qwen38-27b and qwen36-35b-a3b, with its NINFER_* overrides (-e NINFER_SPEC=mtp, -e NINFER_CONTEXT=131072, ...)
download <model>	scripts/download-model.sh into /models: qwen38-27b, qwen36-27b or qwen36-35b-a3b

The official Qwen3.8-27B with the measured tuned profile, on http://localhost:8080/v1:

docker run --rm -v "$PWD/models:/models" ghcr.io/iamwavecut/ninfer-all download qwen38-27b
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all run qwen38-27b

compose.yaml wires the GPU, the port and the volumes for the same: docker compose run --rm ninfer download qwen38-27b, then docker compose up -d. NINFER_IMAGE_ARCH=sm75|sm86|sm120a|multi overrides the GPU detection, and docker build -t ninfer . builds the image from source (--build-arg ARCHS=86 for one architecture, --build-arg 'ARCHS=75;86;120a' for the multi-architecture build).

Building

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --target ninfer-serve ninfer-calibrate

CMAKE_CUDA_ARCHITECTURES takes one build from supported GPUs, or a list of them for a pipeline across cards of different architectures (several architectures).

Toolchains and options — CUDA versions per target, guides

120a runs the mma.sync compatibility path, which the ternary route needs. CUDA 12.8 or newer builds 75, 80, 86 and 89, and 70 needs CUDA 12.8 or 12.9. A 120a build needs CUDA 13.1 or newer, since 12.8 and 12.9 miscompile sm_120a kernels and configure refuses them. The Linux and Windows guides cover dependencies, the opt-in build options and the launcher scripts; tests and benchmarks have their own guides.

License

Apache-2.0, as upstream. The Bonsai artifact's weights come from PrismML, ProCreations and Qwen, all Apache-2.0; its card lists the notices. The Qwen3.8-Flash-Next artifacts carry the Qwen Community License 1.0 of their model.