v0.10.1

[nvluxiaoz]

TensorRT Edge-LLM 0.10.1 Release 2026-09-03

We are excited to announce the release of TensorRT Edge-LLM 0.10.1!

TensorRT Edge-LLM 0.10.1 adds experimental TP=2 inference across two NVIDIA DGX Spark systems, JetSpec speculative decoding for Qwen3-8B, Qwen3.8-27B DSpark support, end-to-end FP8 ViT attention and DART visual-token pruning. This release also redesigns the experimental OpenAI-compatible server to achieve a 60% - 99% reduction of overall disk usage, and a 50% reduction of cold launch time.

Key Features

• Added experimental TP=2 tensor-parallel inference across two NVIDIA DGX Spark systems using MPI and NCCL.
• Added JetSpec speculative decoding support for Qwen3-8B.
• Added Qwen3.8-27B DSpark support.
• Added end-to-end FP8 ViT attention and DART visual-token pruning to reduce VLM prefill work.
• Added NVFP4 MoE build and runtime support for NVIDIA Nemotron 3 Super 120B-A12B checkpoints with 512 routed experts and top-32 routing.
• Redesigned the experimental OpenAI-compatible server for faster cold launches and lower memory usage.

Other Important Features

Runtime and Performance

• Expanded context-cache reuse for speculative decoding, including hybrid and MTP models.
• Extended MTP verification trees to depth 16.
• Added logit_bias support on speculative decoding paths.
• Added a media artifact cache that retains ViT and audio encoder embeddings across repeated media prefixes.
• Added Blackwell SM110 dense NVFP4-A16 GEMM and GEMV kernels.
• Improved Blackwell performance with single-launch D80 SSD prefill, faster SM12x fused MoE, fused GDN projections, and one fewer per-layer KV copy.
• Added a CuTe DSL RMSNorm plugin.
• Upgraded CuTe DSL to 4.7.0.
• Enabled FMHA-v2 and XQA on Hopper SM90 with JIT-compiled XQA kernels.
• Added mock engine based unit tests.

Export and Quantization

• Added NVFP4 W4A16 export for official LM heads and Qwen3 A16 MoE.
• Fixed export for Gemma 4 Unified quantized checkpoints, Qwen NVFP4 mixed-precision dispatch, fused-MoE MTP drafts, and thinking-default chat templates.

Server and API

• Redesigned the experimental OpenAI-compatible server to utilize the checkpoint-direct builder, enabling single-command model construction and serving alongside profile-aware engine caching.
• Added one-command server deployment for speculative-decoding checkpoints, including MTP, EAGLE3, DFlash, JetSpec, and DSpark.
• Upgraded reasoning capabilities with automatic and selectable parsers for Qwen3, DeepSeek-R1, and Nemotron, featuring incremental streaming via reasoning_content.
• Enabled Nemotron-Omni video input support for the OpenAI Compatible server.

Platform Support and Documentation

• Added architecture-aware Python wheel packaging for x86, Jetson, DRIVE, and DGX Spark.
• Added QNX cross-compilation support
• Added Hopper support for development
• Added TensorRT-RTX as a backend
• Expanded documentation for multi-device inference, the experimental server, memory monitoring, KV-cache reuse, and visual-token pruning.

Bug Fixes

• Fixed Int4GroupwiseGemmPluginV2 accuracy for QKV concat aliasing, split-K workspace isolation, and LoRA insertion.
• Fixed speculative decoding with paged KV Cache issues
• Fixed context reuse for cross-batch image contamination and zero-budget encoder caches.
• Fixed the Cosmos3 Reasoner visual patch layout and batched audio embedding concatenation.
• Fixed cross-platform GPU memory reporting.
• Fixed #162: InternVL3 AWQ export now supports checkpoints that provide PyTorch .bin weights without safetensors. 8B model is removed due to unsupported tokenizer.
• Fixed #163: Qwen2.5-VL and Qwen3-VL AWQ vision export now accepts the num_attention_heads configuration key.
• Fixed #184: Qwen2.5 and Qwen2.5-VL export now accepts standard Hugging Face cache shard symlinks while retaining path-escape validation.
• Fixed #166: Confirmed that Alpamayo-R1-10B export no longer hits the ONNX dynamic-shape assertion reported against 0.9.1.

NVIDIA Contributors

@nvluxiaoz @nvamberl @xiangg-nv @willg-nv @mahu888 @nv-samcheng @duofant @JCalafato @duanyaqi @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia @ever-wong @Jasper-NV @jhalabi-nv @levichen-nvidia @sunghyunp-nvidia @nvyocox @ruocheng-nv @yuanyao-nv @qikail-ctrl @MadFunMaker @kevinfu1

---

This discussion was created from the release v0.10.1.