NInfer for Windows and around 16GB VRAM: RTX 5070 Ti / 5080 / 5090, Qwen3.8-27B GSQ-RCO Q3, CUDA 13 Native engine, tray manager, model conversion and measured setup guides.
High-performance single-GPU inference for selected model checkpoints and GPUs.
Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker): 86 tok/s decode, 262K context, C4 aggregate 228 tok/s. Every number a controlled measurement.
RTX 5070 Ti Laptop 12GB 跑 Qwen3.8-27B：NInfer + KVMem、256K 逻辑上下文、RAM KV、MTP、快速缓存续接与 dsh 本地 Agent / 搜索 / 浏览器。含源码、Windows 部署及实测。
monitor for a local LLM runtime (llama-swap, NInfer, Strata, more) -- metrics ported to berth.
NInfer fork with end-user improvements such as model router and jev-alike decisions endpoint. Experimental repo, changes are subject to be wiped without notice based on my own usage observations. Packaged with nix.
Single-GPU NInfer + KVMem integration: multi-lane long contexts, DFlash2 + vision, native prefix reuse and reproducible measurements.
Independent Windows distribution of ninfer-all with RTX 3000, 4000 and 5000 builds, runtime contracts and full upstream attribution.
Wire the 1CatAI Split-D D256 FlashAttention kernel (fishlikeX/sm70-attn, MIT) into NInfer on Tesla V100 sm_70: +36-41% prefill, TTFT -3min, decode unchanged. Measured data + integration guide. Published by the user with AI assistance.
笔记本 RTX 5070 Ti Laptop 12G（12,227 MiB / 140 W TGP）· Bonsai-2-27B 三元量化 · MTP vs DFlash2 同上下文 A/B。Laptop GPU only — NOT the desktop 5070 Ti (16 GB / ~300 W); numbers are not comparable across the two.
Converter toolchain behind the NInfer builds of the Ternary Bonsai 2 family. Two routes: v2 containers for Swift-Bonsai-2 27B, v3 containers for Ternary-Bonsai-2 2B/4B/9B. Byte-for-byte ternary packing, no dequantisation or requantisation.
KVMem + ActKV long-context NInfer for Tesla V100 (sm_70) on Windows: host-archived KV working set, Swift-1.5 27B + MTP, native GGUF loading, OpenAI/Anthropic-compatible serving
NInfer sm_120a 引擎包 · 8 GB RTX 5060 Laptop 部署实践：集显档启动器 / 控制面板 / 桌面应用 / 实测记录
Tesla V100 32GB (sm_70) running Qwen3.8-27B: sm70 decode kernel port plus KV context-cache tuning, measured on a real 53-request agent session. Decode 42.3-89.4 tok/s, TTFT 0.54 s on a cache hit, 200k-token prompts, zero failed requests, raw engine logs included. Published by an AI on the machine owner's behalf. 中文版：README.zh-CN.md
Port NInfer, a single-GPU CUDA inference engine, to the NVIDIA L20 (Ada sm_89, 92 SMs, 48 GB): patch set, build tooling, and measured results
To associate your repository with the ninfer topic, visit your repo's landing page and select "manage topics."