diff --git a/README.md b/README.md index dc838483e88..33e306b98f6 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ Model Optimizer is also integrated with [NVIDIA Megatron-Bridge](https://github. ## Latest News +- [2026/10/08] [BLOG: AutoQuant KV: Automatic Mixed-Precision KV Cache Quantization](https://nvidia.github.io/Model-Optimizer/announcements/autoquant-kv.html): Per-layer KV format search, GEMM composition, and a Qwen3-8B accuracy case study. - [2026/09/16] [BLOG: Scale Learning: Training NVFP4 Block Scales to Recover Accuracy](https://nvidia.github.io/Model-Optimizer/announcements/scale-learning.html) - [2026/09/16] [**End-to-end W4A4 NVFP4 + QAD tutorial for Qwen3.6-35B-A3B**](./examples/megatron_bridge/tutorials/Qwen3.6-35B-A3B): NVFP4 W4A4 PTQ plus quantization-aware distillation, reaching up to 1.30x vLLM throughput over BF16 and 3.1x smaller checkpoints while recovering the accuracy W4A4 costs. - [2026/09/09] [BLOG: Improving NVFP4 Accuracy with Local-Hessian Weight Scales](https://nvidia.github.io/Model-Optimizer/announcements/local-hessian.html) diff --git a/docs/source/announcements/assets/autoquant-kv-qwen3-8b-accuracy-memory.svg b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-accuracy-memory.svg new file mode 100644 index 00000000000..acb1a28de5e --- /dev/null +++ b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-accuracy-memory.svg @@ -0,0 +1,621 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.2, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 1 + + + + + + + + + + + + + 2 + + + + + + + + + + + + + 3 + + + + + + + + + + + + + 4 + + + + + + + + + + + + + 5 + + + + KV payload at 32,768 tokens (GiB) + + + + + + + + + + + + + + BF16 + + + + + + + + + + FP8 + + + + + + + + + + NVFP4 + + + + + + + + + + NVFP4 Four-over-Six + + + + + + + + + + Native 6.25 + + + + + + + + + + AutoQuant 5.33 + + + + + + + + + + AutoQuant 5.76 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 4.50 + + + 2.25 + + + 1.27 + + + 1.27 + + + 1.76 + + + 1.50 + + + 1.62 + + + Smaller KV cache + + + + + + + + + + + + + + + + + + −8 + + + + + + + + + + + + + −7 + + + + + + + + + + + + + −6 + + + + + + + + + + + + + −5 + + + + + + + + + + + + + −4 + + + + + + + + + + + + + −3 + + + + + + + + + + + + + −2 + + + + + + + + + + + + + −1 + + + + + + + + + + + + + 0 + + + + Accuracy change: result − BF16 (percentage points) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0.00 + + + -0.62 + + + -6.54 + + + -6.09 + + + -0.98 + + + -2.08 + + + -1.04 + + + RULER-128K: change vs BF16 + + + + Qwen3-8B: smaller cache, accuracy close to BF16 + + + BF16 weights • BF16 KV baseline: 77.30% • default-scale AutoQuant • development runtime + + + Calculated KV payload includes block scales; excludes runtime overhead and weights. Negative accuracy change means a drop. + + + + + + + + + + + diff --git a/docs/source/announcements/assets/autoquant-kv-qwen3-8b-layer-allocation.svg b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-layer-allocation.svg new file mode 100644 index 00000000000..05d1550e236 --- /dev/null +++ b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-layer-allocation.svg @@ -0,0 +1,713 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.2, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + 2 + + + + + + + + + + 4 + + + + + + + + + + 6 + + + + + + + + + + 8 + + + + + + + + + + 10 + + + + + + + + + + 12 + + + + + + + + + + 14 + + + + + + + + + + 16 + + + + + + + + + + 18 + + + + + + + + + + 20 + + + + + + + + + + 22 + + + + + + + + + + 24 + + + + + + + + + + 26 + + + + + + + + + + 28 + + + + + + + + + + 30 + + + + + + + + + + 32 + + + + + + + + + + 34 + + + + Attention layer (zero-based) + + + + + + + + + + + + + + 5.33 bits: 17 FP8 K layers + + + + + + + + + + 5.76 bits: 26 FP8 K layers + + + + + + + + + + + + AutoQuant spends more precision on selected keys + + + Green: FP8 K / NVFP4 V • Gray: NVFP4 K / NVFP4 V • All 36 V caches remain NVFP4 + + + + + + + + diff --git a/docs/source/announcements/assets/autoquant-kv-qwen3-8b-methods.json b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-methods.json new file mode 100644 index 00000000000..6885e14a426 --- /dev/null +++ b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-methods.json @@ -0,0 +1,159 @@ +{ + "model": { + "id": "Qwen/Qwen3-8B", + "revision": "b968826d9c46dd6066d109eabc6255188de91218", + "weight_dtype": "BF16", + "attention_layers": 36, + "kv_heads": 8, + "head_dim": 128 + }, + "runtime": { + "kind": "development vLLM", + "image_sha256": "95f552c4d541fbcf1d4fea060f604f9256591424e6fe12732abbd903f19da486", + "tensor_parallelism": 1, + "data_parallelism": 4, + "cuda_graphs": true, + "prefix_caching": true, + "chunked_prefill": true + }, + "search": { + "dataset": "Nemotron Post-Training Dataset v3", + "packed_rows": 64, + "tokens_per_row": 8192, + "scored_tail_tokens": 1024, + "seed": 1234, + "disjoint_validation_rows": 64, + "benchmark_prompts_separate": true, + "candidates": [ + "FP8 K/V", + "FP8 K / NVFP4 V", + "NVFP4 K/V" + ], + "requested_budgets": [ + 5.375, + 5.8125 + ] + }, + "autoquant_scale_policy": { + "mode": "default", + "kv_scale": 1.0, + "persistent_kv_scale_tensors_removed": 72, + "note": "Layer maps and BF16 weights were preserved; search was not rerun after removing scale tensors." + }, + "accuracy_drop": { + "formula": "recipe accuracy_pct - BF16 accuracy_pct", + "units": "percentage points", + "positive": "better than BF16", + "negative": "worse than BF16", + "rounding": "Differences use unrounded input scores; table rounds half up to two decimal places.", + "uncertainty": "Point-estimate differences. CSV stderr_pp refers to each absolute accuracy, not the difference. No paired significance test is implied." + }, + "uncertainty": { + "general": "Retained evaluator SE unless the CSV uncertainty_basis specifies a reconstruction or approximation. Repeated completions do not add independent benchmark problems.", + "scicode_bf16_nvfp4": "Approximate binomial SE: 100 * sqrt(p * (1-p) / 2704); repeated completions treated as independent.", + "scicode_native_6p25": "Mean of three independent run scores: 543/2704, 519/2704, 524/2704. SE is their sample SD / sqrt(3). CSV response counts sum across the three runs.", + "scicode_other": "Repeat-derived standard errors retained in the CSV." + }, + "coverage": { + "six_non_scicode_tasks": "Complete successful response coverage for every displayed cell.", + "scicode": "Fixed denominator 2704 per run; missing and unsuccessful generations remain failures.", + "withheld": "NVFP4 and NVFP4 Four-over-Six AIME2025 controls used unmatched runtime/topology. Legacy-kernel results and unavailable matched AA-LCR, TauBench2, and TerminalBench results are excluded." + }, + "payload": { + "formula_bytes": "resident_tokens * 36 * 8 * 128 * 2 * effective_kv_bits / 8", + "figure_resident_tokens": 32768, + "includes": "NVFP4 block scales", + "excludes": [ + "allocator padding", + "cache-page rounding", + "tensor-scale metadata", + "weights", + "activations", + "runtime workspaces" + ], + "scope": "Calculated KV payload; not total GPU memory, throughput, or concurrency." + }, + "benchmarks": { + "AIME2025": { + "task": "AIME_2025_aa_v2", + "max_model_len": 40960, + "max_new_tokens": 32768, + "temperature": 0.6, + "top_p": 0.95, + "repeats": 64, + "exact_denominator": 1920, + "reasoning_parser": null + }, + "AIME2026": { + "task": "ns_aime2026", + "max_model_len": 40960, + "max_new_tokens": 32768, + "temperature": 1.0, + "top_p": 0.95, + "repeats": 64, + "exact_denominator": 1920, + "reasoning_parser": null + }, + "GPQA": { + "task": "gpqa_diamond_aa_v3", + "max_model_len": 40960, + "max_new_tokens": 32768, + "temperature": 0.6, + "top_p": 0.95, + "repeats": 64, + "exact_denominator": 12672, + "reasoning_parser": null + }, + "LiveCodeBench": { + "task": "ns_livecodebench", + "max_model_len": 40960, + "max_new_tokens": 32768, + "temperature": 1.0, + "top_p": 0.95, + "repeats": 8, + "exact_denominator": 3632, + "reasoning_parser": null, + "dataset": "test-v6-2408-2505" + }, + "IFBench": { + "task": "ns_ifbench", + "max_model_len": 262144, + "max_new_tokens": 131072, + "temperature": 1.0, + "top_p": 0.95, + "repeats": 5, + "exact_denominator": 1500, + "reasoning_parser": "qwen3", + "tool_call_parser": "qwen3_coder", + "canonical_metric": "pass@1[avg-of-5].prompt_loose_accuracy", + "stderr_method": "sample standard error reconstructed from the five cumulative prompt-loose repeat aggregates", + "yarn_factor": 8 + }, + "SciCode": { + "task": "ns_scicode", + "max_model_len": 262144, + "max_new_tokens": 131072, + "temperature": 0.6, + "top_p": 0.95, + "repeats": 8, + "exact_denominator": 2704, + "reasoning_parser": "qwen3", + "tool_call_parser": "qwen3_coder", + "yarn_factor": 8, + "canonical_metric": "pass@1[avg-of-8].subtask_accuracy" + }, + "RULER-128K": { + "task": "ruler-128k-chat", + "max_model_len": 131072, + "max_new_tokens": 2048, + "temperature": 1.0, + "top_p": 0.95, + "exact_denominator": 6500, + "reasoning_parser": null, + "sequence_tokens": 128000, + "subtasks": 13, + "samples_per_subtask": 500 + } + }, + "availability": "Development runtime and study checkpoints are not distributed with the blog. The pinned merged starter example uses a different candidate menu and search dataset. Local BF16 results have not been independently reproduced under the same evaluation contract." +} diff --git a/docs/source/announcements/assets/autoquant-kv-qwen3-8b-results.csv b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-results.csv new file mode 100644 index 00000000000..508721d7a54 --- /dev/null +++ b/docs/source/announcements/assets/autoquant-kv-qwen3-8b-results.csv @@ -0,0 +1,48 @@ +benchmark,variant,effective_kv_bits,search_budget_bits,scale_mode,accuracy_pct,stderr_pp,expected_responses,observed_responses,successful_responses,evaluated_runs,uncertainty_basis,coverage,run_id,response_statistics_sha256,baseline_accuracy_pct,accuracy_drop_pp +AIME2025,BF16,16,,uniform control,67.03125,1.2431339808,1920,1920,1920,1,retained evaluator standard error,complete,d447ce0880c9093d,bd6f1b7705946f68fb80d5ea5bcb83e9756d3f69cec8689e8a333fcae49ca077,67.03125,0.00000 +AIME2025,FP8,8,,uniform control,66.6145833333,0.912881176,1920,1920,1920,1,retained evaluator standard error,complete,619316a9812fb952,d7730e88649aa46d3c3347c304a4bbeeaa0c5546917508a9716634177dafa770,67.03125,-0.4166666667 +AIME2025,Native 6.25,6.250003814697266,,fixed mixed control,66.1979166667,1.1098804,1920,1920,1920,1,retained evaluator standard error,complete,b104b8c493cdffc6,b2ecdcb026c3ed8ebccf190c047d10c52d839ea9943f0b2140ffc133d7fe137a,67.03125,-0.8333333333 +AIME2025,AutoQuant 5.33,5.326392703586155,5.375,default,65.15625,0.9942983559,1920,1920,1920,1,retained evaluator standard error,complete,c30ba7a6323e2ea5,839f5eda34b1bace455d1a0ba4185be0e58591efeedf454f8681444639d9fd96,67.03125,-1.87500 +AIME2025,AutoQuant 5.76,5.763892703586155,5.8125,default,66.7708333333,1.080282785,1920,1920,1920,1,retained evaluator standard error,complete,ac3f2941703fad8a,b31caae041754c595a8bcd86766065ef014193e36e9743d36a6a4963b3aa7ca8,67.03125,-0.2604166667 +AIME2026,BF16,16,,uniform control,66.1979166667,0.6205405,1920,1920,1920,1,retained evaluator standard error,complete,c5fc78b1ae3ac5b9,0281a2bf25dac614ff5536142f0551bc63caa91dfe03676295fa455fec32084c,66.1979166667,0E-10 +AIME2026,FP8,8,,uniform control,66.9270833333,0.6694086,1920,1920,1920,1,retained evaluator standard error,complete,0af7d0ee93f0656e,a22fab3d010123670f64c53924aba5fad32316891476e9df5408fa9a205d2235,66.1979166667,0.7291666666 +AIME2026,NVFP4,4.5,,uniform control,66.1458333333,0.6812991969,1920,1920,1920,1,retained evaluator standard error,complete,eb803219fd7aa436,641dccdf2e82a54c7bb7642afc0c99f2709c68ee49da7a654b92c38e59823e71,66.1979166667,-0.0520833334 +AIME2026,NVFP4 Four-over-Six,4.5,,uniform control,65.9895833333,0.7967488207,1920,1920,1920,1,retained evaluator standard error,complete,af45102aee11239b,d0ed4d6ed1f7ba5908b92d20c0cd549a281cb29624f336da5af8b3bb3d5b41bd,66.1979166667,-0.2083333334 +AIME2026,Native 6.25,6.250003814697266,,fixed mixed control,67.2916666667,0.6161186,1920,1920,1920,1,retained evaluator standard error,complete,b1466674187d72b9,1df08166db792683ded03d12eed8ab7016579494e606467f08c57bd542892e57,66.1979166667,1.0937500000 +AIME2026,AutoQuant 5.33,5.326392703586155,5.375,default,66.3020833333,0.6645021,1920,1920,1920,1,retained evaluator standard error,complete,43a53ba568f55c00,7fcd521a3e17526ae229ca817beeb281cabb1d750aeeeb4e5c41400b0fa78b27,66.1979166667,0.1041666666 +AIME2026,AutoQuant 5.76,5.763892703586155,5.8125,default,65.5729166667,0.7045092,1920,1920,1920,1,retained evaluator standard error,complete,1db2dea5e96e342a,c501876a61cf148bce29c62f4f702b95b66af185a958fc7b6ae72cd3cfa36f62,66.1979166667,-0.6250000000 +GPQA,BF16,16,,uniform control,59.1461489899,0.4307730663,12672,12672,12672,1,retained evaluator standard error,complete,9df121dd3156cc99,8171805e78e41f9e24d13daad0bcd6ef8686068caf6b5eac8ad5b98cb2a099af,59.1461489899,0E-10 +GPQA,FP8,8,,uniform control,59.7932449495,0.4627207691,12672,12672,12672,1,retained evaluator standard error,complete,34e2b1a38e3cd56a,39a5236f123988513add3a899c1ecc09233ad1d2e41a249e5945cce4980138db,59.1461489899,0.6470959596 +GPQA,NVFP4,4.5,,uniform control,57.9072,0.4696,12672,12672,12672,1,retained evaluator standard error,complete,c2b3e1987915554b,d4e4a40cbd527942a5669f9729da807b784d35c3cf475d3bc8fe32c1a2ac5277,59.1461489899,-1.2389489899 +GPQA,NVFP4 Four-over-Six,4.5,,uniform control,57.6626,0.4132,12672,12672,12672,1,retained evaluator standard error,complete,edb9e83c442812c1,af60b7512d641bc88e2113f69e69a1cf872af6cd5105d49aa2ec246bc1f4321a,59.1461489899,-1.4835489899 +GPQA,Native 6.25,6.250003814697266,,fixed mixed control,58.5621843434,0.4363072794,12672,12672,12672,1,retained evaluator standard error,complete,c2df0a93be8e21fc,493afe5c2fdb1450dd3a3f122e85ea8b7452e77470e96841d2eda1f4d1f2f997,59.1461489899,-0.5839646465 +GPQA,AutoQuant 5.33,5.326392703586155,5.375,default,58.0886994949,0.4368373068,12672,12672,12672,1,retained evaluator standard error,complete,cc240e2e38858572,f34cbf03f19ea5ed85bd299f87cc4fa052576f11a4650c40bb1aeed17e35b892,59.1461489899,-1.0574494950 +GPQA,AutoQuant 5.76,5.763892703586155,5.8125,default,58.9330808081,0.4295660611,12672,12672,12672,1,retained evaluator standard error,complete,330235f6a0906f48,d2d1e0e6bb4eb00015847890fd5a853f1ed8b5888e15c7b7d3acafdd54ec9d1a,59.1461489899,-0.2130681818 +LiveCodeBench,BF16,16,,uniform control,55.2588105727,0.2836609,3632,3632,3632,1,retained evaluator standard error,complete,2852516a68615637,1ee0cd0ac51642021b1362102001ab55c81357a5201275874db48f04631e1ed5,55.2588105727,0E-10 +LiveCodeBench,FP8,8,,uniform control,55.5892070485,0.2663335,3632,3632,3632,1,retained evaluator standard error,complete,8cfb0e90fcb99b51,4c08293d05f418637835db34da93b56dc9901ba33b9acfd6c9815ea8b996e322,55.2588105727,0.3303964758 +LiveCodeBench,NVFP4,4.5,,uniform control,53.359030837,0.2488875396,3632,3632,3632,1,retained evaluator standard error,complete,04dc3f9e9261080b,9a5e547f070243902e08456a7d13de4f9c83bc9aaeb18a99a22548277033438c,55.2588105727,-1.8997797357 +LiveCodeBench,NVFP4 Four-over-Six,4.5,,uniform control,54.2676211454,0.3354385051,3632,3632,3632,1,retained evaluator standard error,complete,0f6e5e140224462b,bd39fa7733a55075978bb40ab3a84f87b6fc284a85454fc05a401055da762aef,55.2588105727,-0.9911894273 +LiveCodeBench,Native 6.25,6.250003814697266,,fixed mixed control,55.4515418502,0.3899318355,3632,3632,3632,1,retained evaluator standard error,complete,cdfebca75ac1538c,31e4880a558943609594a6bcc82f38cd534c3507bd02972cdf08ce6213a02fd5,55.2588105727,0.1927312775 +LiveCodeBench,AutoQuant 5.33,5.326392703586155,5.375,default,54.7081497797,0.3555008,3632,3632,3632,1,retained evaluator standard error,complete,7e81dfdc15831851,9babcadffa70874ad36cd37fbe7744fad2367a59906c9058e325c4835c304c01,55.2588105727,-0.5506607930 +LiveCodeBench,AutoQuant 5.76,5.763892703586155,5.8125,default,54.7081497797,0.3651188,3632,3632,3632,1,retained evaluator standard error,complete,0255a6ea0cb7f9ac,429092e0ade1aad58b371e11b0e98ec8fe02091c48d0d1783be4e0f8e196b603,55.2588105727,-0.5506607930 +IFBench,BF16,16,,uniform control,32.8666666667,0.3645392831,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,0e8e890b388d44a0,956e77237b5f418ef1a2466bc3d9fabf14471b6329358b37758632d0eb24c487,32.8666666667,0E-10 +IFBench,FP8,8,,uniform control,35.4666666667,0.8416914188,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,ea98f540e2c5a09d,bad480451660de9c568c588e9245a6dbab82ecbe554f70c8bb07413a0a032ff5,32.8666666667,2.6000000000 +IFBench,NVFP4,4.5,,uniform control,31.8666666667,0.6828046736,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,cb45f5af7c18f625,de7a10608ce3e976e8a23584d5753c8b03f4052873a019354ff2a97997349541,32.8666666667,-1.0000000000 +IFBench,NVFP4 Four-over-Six,4.5,,uniform control,31.4,0.383550663,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,055829139e76fdfe,9d3399ece46717964ed42904fc35842a9a6e79950d0a801a39909764a7f020f4,32.8666666667,-1.4666666667 +IFBench,Native 6.25,6.250003814697266,,fixed mixed control,33.8,0.549343042,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,09af635acac81ac5,3b8cbb86aa2c4d9c1045b8b59291175c88582a3e1ed2e969bec8f48d7b7e65a2,32.8666666667,0.9333333333 +IFBench,AutoQuant 5.33,5.326392703586155,5.375,default,33.5333333333,0.5769652406,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,8ead8183bccd45b1,fe70d1ab3a728b7a0b46e6e31446be9c9d68626e2f789b5ab485aa8297c307cf,32.8666666667,0.6666666666 +IFBench,AutoQuant 5.76,5.763892703586155,5.8125,default,32.8,1.0706176826,1500,1500,1500,1,five prompt-loose repeat aggregates,complete,c5737396d3c3597c,8c05639c5edc22f3cec8ede037a0514b62a4362aedb8a67271afecc9389601c0,32.8666666667,-0.0666666667 +RULER-128K,BF16,16,,uniform control,77.2953846154,0.4808081172,6500,6500,6500,1,retained evaluator standard error,complete,35c745e2bd343f20,a60f70e5ff087d40e620574435a4ee67e065b26ae8ec85405b220e7bceb8a720,77.2953846154,0E-10 +RULER-128K,FP8,8,,uniform control,76.6792307692,0.4849603281,6500,6500,6500,1,retained evaluator standard error,complete,0da669e6b2973e47,ba4072c1c24c6db93d0ac0a016c84c4317e2a265e677ffbcdd8cf353d81b5549,77.2953846154,-0.6161538462 +RULER-128K,NVFP4,4.5,,uniform control,70.7584615385,0.5191439907,6500,6500,6500,1,retained evaluator standard error,complete,b464b6af71dfb21d,9cc6615953e3c648a1a094fbb69dd2ce50d170fd71a855ee8735703593742a9f,77.2953846154,-6.5369230769 +RULER-128K,NVFP4 Four-over-Six,4.5,,uniform control,71.2092307692,0.5216063139,6500,6500,6500,1,retained evaluator standard error,complete,98889c0725a1d596,8bfe80ca3e8eb8cb8568e2bf80a5bdc098d3e1b54257ba2aef7d19f237139547,77.2953846154,-6.0861538462 +RULER-128K,Native 6.25,6.250003814697266,,fixed mixed control,76.32,0.4867454609,6500,6500,6500,1,retained evaluator standard error,complete,77d65ef41ce14c61,ae361f2634a2f4203185f02a9a877ed119294027ae4dbfa728a87a556913bbef,77.2953846154,-0.9753846154 +RULER-128K,AutoQuant 5.33,5.326392703586155,5.375,default,75.2146153846,0.4986442535,6500,6500,6500,1,retained evaluator standard error,complete,122d0eddceaee4c4,7d3918a324ef821788b703fe349c5005ddb2eb6f438da1346b4a4138faea9350,77.2953846154,-2.0807692308 +RULER-128K,AutoQuant 5.76,5.763892703586155,5.8125,default,76.2507692308,0.4893242623,6500,6500,6500,1,retained evaluator standard error,complete,db1747e14b9da97a,bb15105e2bbdde29159b5c23290d3e9970bc3f3d5a4bc6737e7928614f812272,77.2953846154,-1.0446153846 +SciCode,BF16,16,,uniform control,20.3772189349,0.7746183547,2704,2696,2696,1,approximate binomial SE; 2704 subtasks/repeats treated as independent,fixed denominator; missing or unsuccessful generations remain failures,e547522a4e2bbaf8,5d2bb4436ae1b761555246212a6d12fa21544b1c1fc410f9dcf14c0922d44502,20.3772189349,0E-10 +SciCode,FP8,8,,uniform control,19.2307692308,2.2033861927,2704,2696,2696,1,retained evaluator standard error,fixed denominator; missing or unsuccessful generations remain failures,87f3a5b43c71f76a,92de9285fab222919c105d100ee08f71b94fd39e1bf89169472a7deba58a117d,20.3772189349,-1.1464497041 +SciCode,NVFP4,4.5,,uniform control,16.900887574,0.7206918409,2704,2696,2694,1,approximate binomial SE; 2704 subtasks/repeats treated as independent,fixed denominator; missing or unsuccessful generations remain failures,88e4268c578c62ba,f9dc1bd6417dfc223584e1ab8c2d4e2341b271e35df19b2ac05198b17b15b65c,20.3772189349,-3.4763313609 +SciCode,NVFP4 Four-over-Six,4.5,,uniform control,17.6775147929,2.9004117626,2704,2696,2696,1,retained evaluator standard error,fixed denominator; missing or unsuccessful generations remain failures,5352b42e3f30598c,05d00e48f8256237a56f27d0308d7251edcfdcf474b25909c4e7011507a7eb6d,20.3772189349,-2.6997041420 +SciCode,Native 6.25,6.250003814697266,,fixed mixed control,19.55128205128205,0.27036134368172215,8112,8072,8064,3,sample SD of three independent run accuracies / sqrt(3),fixed denominator in each run; all failures retained,ee00b3dfd7dbcdd6;0f9b9be765215a7a;489463b1115e73cb,819b5578160d959cd746c34d0b5835b4a34ca23df918082b53ee6aaa437ea3c7;0134f5e28b67d9584a79876a7541edcc6665538d6539b840fd3b432cdcb89719;17f4e54c414418dbab8caef23ab12c9aa0ec6b8205cb66b8a4b9b3433cb88d1f,20.3772189349,-0.82593688361795 +SciCode,AutoQuant 5.33,5.326392703586155,5.375,default,19.6375739645,1.6010860572,2704,2696,2696,1,retained evaluator standard error,fixed denominator; missing or unsuccessful generations remain failures,05039954415822c5,6f35da30a947ff07c41eef03ba3cdf60ef57fa81e8281659b6246cd4bc6e46fc,20.3772189349,-0.7396449704 +SciCode,AutoQuant 5.76,5.763892703586155,5.8125,default,20.47477744807122,3.151671891914785,2704,2692,2690,1,retained evaluator standard error,fixed denominator; missing or unsuccessful generations remain failures,58c99323c11e8f64,3642e29604c1c24fd3c22f546dca43379a5717085ebfb4161e69c7a0202b1772,20.3772189349,0.09755851317122 diff --git a/docs/source/announcements/assets/autoquant-kv-search.svg b/docs/source/announcements/assets/autoquant-kv-search.svg new file mode 100644 index 00000000000..b523da783dc --- /dev/null +++ b/docs/source/announcements/assets/autoquant-kv-search.svg @@ -0,0 +1,113 @@ + +From layer sensitivity to a mixed-precision KV recipe +Illustrative search with four equal-width layers. GEMM settings stay fixed in the reference and every trial, whether BF16 or already quantized. Compare each isolated layer/candidate trial with a BF16 KV reference. NVFP4 sensitivities are 0.03, 0.32, 0.09, 0.02; FP8 sensitivities are 0.01, 0.02, 0.01, 0.01. Under a 5.375-bit budget, select NVFP4, FP8, NVFP4, NVFP4. Upgrading layer 2 gives the largest sensitivity reduction, 0.30. + + + +From sensitivity to a KV recipe +Illustrative example · four equal-width layers + + +1 +Measure each layer / candidate +GEMM fixed; change one layer’s KV format. +L1 +L2 +L3 +L4 +Reference + +BF16 + + +BF16 + + +BF16 + + +BF16 +Trial + +BF16 + + +NVFP4 + + +BF16 + + +BF16 + + +Restore, then repeat for all choices + +Output KL + + + +2 +Build the sensitivity map +Lower KL = closer to the reference +L1 +L2 +L3 +L4 + +NVFP4 + +0.03 + +0.32 + +0.09 + +0.02 + +FP8 + +0.01 + +0.02 + +0.01 + +0.01 + + +Budget: 5.375 bits + + + +3 +Optimize and assign formats +Minimum total KL within the bit budget +L1 +L2 +L3 +L4 +Selected KV + +NVFP4 + +FP8 + +NVFP4 + +NVFP4 +(4.5 + 8 + 4.5 + 4.5) / 4 = 5.375 bits + diff --git a/docs/source/announcements/autoquant-kv.rst b/docs/source/announcements/autoquant-kv.rst new file mode 100644 index 00000000000..bc9e4097eb9 --- /dev/null +++ b/docs/source/announcements/autoquant-kv.rst @@ -0,0 +1,223 @@ +:orphan: + +AutoQuant KV: Automatic Mixed-Precision KV Cache Quantization +############################################################# + +:Author: Model Optimizer Team +:Date: October 8, 2026 +:Tags: autoquantize, kv-cache, quantization, mixed-precision, modelopt + +Introduction to AutoQuant +************************* + +**AutoQuant automates mixed-precision quantization.** Instead of assigning one +format everywhere, NVIDIA Model Optimizer measures layer sensitivity and +selects formats under a storage budget. Sensitive layers retain more precision; +less sensitive layers use fewer bits. + +GEMM AutoQuant applies this idea to the model's matrix multiplications. +**KV AutoQuant extends it to attention keys and values**, whose storage grows +with context length and concurrent requests. The two searches can be combined, +with independent budgets for weights and the KV cache. + +Below, we explain the search workflow and API, then use Qwen3-8B as a case +study of the accuracy and memory tradeoff. + +How KV AutoQuant works +********************** + +KV AutoQuant searches over K/V format pairs using representative prompts and +an average-bit budget. + +**Prepare candidates.** The starter recipe uses constant scales and offers FP8 +K/V (8 bits) and NVFP4 K/V (4.5 bits including block scales). The case study +also includes FP8 K with NVFP4 V (6.25 bits for equal K/V widths). +NVFP4 K with FP8 V is excluded because the current export path does not +support that pair; this study did not evaluate its accuracy. + +**Score each choice.** GEMM settings stay fixed in the reference and every +trial: BF16 when starting from a BF16 model, or the existing quantized formats +and scales. The reference disables K/V quantization in all eligible layers. +Each trial changes only one layer's K/V pair; KL divergence measures the +change in output predictions. BF16 K/V is the reference, not an implicit +solver choice. + +.. figure:: assets/autoquant-kv-search.svg + :alt: Three-stage illustrative search with GEMM settings fixed in the reference and every trial: score isolated layer/candidate trials against BF16 KV, build a sensitivity map, and optimize under a 5.375-bit budget. NVFP4 scores are 0.03, 0.32, 0.09, and 0.02; FP8 scores are 0.01, 0.02, 0.01, and 0.01. Select FP8 for layer 2 and NVFP4 for the other three layers. + :width: 640px + :align: center + + Layer 2 receives FP8 because its upgrade gives the largest error reduction + within the budget. Values are illustrative, with equal K/V widths across layers. + +**Select one candidate per layer.** An integer program minimizes total +sensitivity under the storage constraint: + +.. math:: + + \begin{gathered} + \min_{\{c_\ell\}}\sum_\ell s_{\ell,c_\ell} \\[4pt] + \text{subject to}\quad + \frac{\sum_\ell w_\ell b_{\ell,c_\ell}}{\sum_\ell w_\ell} \le B. + \end{gathered} + +Here :math:`s` is KL sensitivity, :math:`b` is the width-weighted K/V bit cost +including scales, :math:`w` is the layer's total K/V width, and :math:`B` is +``effective_bits``. Layers matched by ``disabled_layers`` are preserved and +excluded from the budget. + +**Apply and reuse.** Validate the assembled model on held-out tasks: summed +sensitivity is an estimate of combined error. A saved search checkpoint lets +you change the budget without rescoring, provided you restart from the same +pre-search model state and keep scoring data, candidates, and scales unchanged. + +API and examples +**************** + +KV AutoQuant uses ``modelopt.torch.quantization.auto_quantize`` with +``cost_model="kv_cache"`` and ``method="kl_div"``. Given a Hugging Face causal +LM and a batch-size-one calibration loader yielding device-placed +``input_ids`` and ``attention_mask`` tensors: + +.. code-block:: python + + import modelopt.torch.quantization as mtq + from modelopt.recipe import load_recipe + + cfg = load_recipe( + "general/auto_quantize/kv_fp8_nvfp4_cast_kl_div_at_5p4bits" + ).auto_quantize + + def forward_step(model, batch): + logits = model(**batch, use_cache=False).logits + return logits[batch["attention_mask"].bool()] + + model, search_state = mtq.auto_quantize( + model.eval(), + constraints={"cost_model": "kv_cache", "effective_bits": 5.4}, + quantization_formats=[ + c.model_dump(exclude_none=True) for c in cfg.candidate_formats + ], + data_loader=calib_loader, + forward_step=forward_step, + disabled_layers=cfg.disabled_layers, + num_calib_steps=len(calib_loader), + num_score_steps=min(len(calib_loader), cfg.score_size), + method="kl_div", + checkpoint="kv_search.pth", + ) + +For search **and Hugging Face export**, use the shipped example. In a CUDA +environment, install a checkout containing ModelOpt +`#2272 `_ and +`#2273 `_: + +.. code-block:: bash + + git clone https://github.com/NVIDIA/Model-Optimizer.git Model-Optimizer-autoquant-kv + cd Model-Optimizer-autoquant-kv + git checkout 1d392999b45626f9c06ff0bc9797b03529fe24e3 + python -m pip install -e '.[hf]' + python -m pip install -r examples/hf_ptq/requirements.txt + + mkdir -p outputs/qwen3-8b-kv + python examples/hf_ptq/hf_ptq.py \ + --pyt_ckpt_path Qwen/Qwen3-8B \ + --recipe general/auto_quantize/kv_fp8_nvfp4_cast_kl_div_at_5p4bits \ + --dataset cnn_dailymail --calib_size 128 --calib_seq 512 --batch_size 1 \ + --kv_auto_quantize_checkpoint outputs/qwen3-8b-kv/search.pth \ + --export_path outputs/qwen3-8b-kv/hf + +The starter recipe searches full FP8 K/V and full NVFP4 K/V with constant +scales. This small example demonstrates the workflow; use representative data +for an application recipe. Preserve ``hf_quant_config.json`` and exported +scales when serving with a compatible runtime. + +Combining KV and GEMM AutoQuant +******************************* + +**Score KV candidates on the GEMM configuration you will serve.** If GEMMs +will be quantized, apply GEMM AutoQuant or fixed PTQ first. KV-only search is +useful for BF16 deployments, controlled KV studies, or models whose GEMMs are +already quantized. + +A composed recipe runs **GEMM AutoQuant first, then KV AutoQuant**. The shipped +example uses gradient-based NVFP4/FP8 GEMM search, freezes its selected +weight/activation quantizers and calibrated tensors, then runs forward-KL KV +search. Its 5.4-bit weight and 5.4-bit KV targets constrain separate storage +domains; this is a sequential search, not one joint bit budget. + +Use the command above with the following recipe and separate search-state +files, plus a fresh export directory: + +.. code-block:: bash + + --recipe general/auto_quantize/nvfp4_fp8_gradient_then_kv_fp8_nvfp4_cast_kl_div_at_5p4bits \ + --auto_quantize_checkpoint gemm_search.pth \ + --kv_auto_quantize_checkpoint kv_after_gemm_search.pth + +Changing the GEMM configuration requires recomputing KV sensitivity with a new +checkpoint. The +`HF PTQ guide `_ +describes both compositions. Serving support must cover the combined weight +and KV formats; an export marked ``kv_cache_deployment_supported: false`` +requires additional runtime support. + +Mixed-precision results +*********************** + +In our **Qwen3-8B case study**, AQ at **5.76 effective KV bits** reduces modeled cache +payload by **64.0% versus BF16**, with a **−1.04 percentage-point change on +RULER-128K**, compared with −6.54 for uniform NVFP4. These results isolate KV +quantization: all model weights remain BF16. They do not measure the composed +GEMM+KV recipe above. + +.. image:: assets/autoquant-kv-qwen3-8b-accuracy-memory.svg + :alt: AutoQuant 5.76 uses 1.62 GiB of modeled KV payload versus 4.50 GiB for BF16 at 32768 tokens; RULER accuracy changes by minus 1.04 points versus minus 6.54 for NVFP4. + :width: 100% + +**BF16 shows accuracy (%); every recipe column shows result minus BF16 in +percentage points. Negative means an accuracy drop; positive means an +improvement.** Differences use unrounded scores. AutoQuant results use only +default K/V scales of 1.0. + +.. csv-table:: Accuracy change: result − BF16 (percentage points) + :header: "Benchmark","BF16 (%)","FP8","NVFP4","NVFP4 4/6","Native 6.25","AQ 5.33","AQ 5.76" + + "AIME2025","67.03","-0.42","—","—","-0.83","-1.88","-0.26" + "AIME2026","66.20","+0.73","-0.05","-0.21","+1.09","+0.10","-0.63" + "GPQA","59.15","+0.65","-1.24","-1.48","-0.58","-1.06","-0.21" + "LiveCodeBench","55.26","+0.33","-1.90","-0.99","+0.19","-0.55","-0.55" + "IFBench","32.87","+2.60","-1.00","-1.47","+0.93","+0.67","-0.07" + "RULER-128K","77.30","-0.62","-6.54","-6.09","-0.98","-2.08","-1.04" + "SciCode","20.38","-1.15","-3.48","-2.70","-0.83","-0.74","+0.10" + + +**Native 6.25** is a fixed mixed-precision layer map. **AQ 5.33** and +**AQ 5.76** are AutoQuant allocations at their achieved bit costs. +**NVFP4 4/6** uses Four-over-Six block-scale selection. AQ 5.76 uses **7.8% less +KV payload than Native 6.25**, with accuracy within 0.63 points of BF16 on +AIME2025, AIME2026, GPQA, LiveCodeBench, and IFBench. + +The study searched FP8 K/V, FP8 K with NVFP4 V, and NVFP4 K/V. Both AQ maps keep +all values in NVFP4; 17 of 36 layers use FP8 keys at 5.33 bits, and 26 at 5.76 +bits. Their requested budgets were 5.375 and 5.8125 bits. + +.. image:: assets/autoquant-kv-qwen3-8b-layer-allocation.svg + :alt: AutoQuant selects FP8 keys in 17 of 36 layers at 5.33 bits and 26 layers at 5.76 bits; all value caches use NVFP4. + :width: 100% + +The study used a development runtime with asymmetric K/V support, beyond the +starter recipe's candidate menu. See the +`vLLM integration `_ for the +per-layer FP8/NVFP4 path. Payload calculations include block scales but exclude +runtime overhead and weights; they are not total GPU-memory or throughput measurements. + +.. rst-class:: table-note + + All comparisons use the same development vLLM image (TP1/DP4) and matched + settings per benchmark. Dashes withhold unmatched AIME2025 controls. + SciCode counts missing/unsuccessful generations as failures in 2,704 trials + per run; Native 6.25 averages three runs. Differences are point estimates, + not significance claims. Study checkpoints and runtime are not distributed + with this post. diff --git a/docs/source/index.rst b/docs/source/index.rst index 07dbcfb1f49..0c6532c7f39 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -13,6 +13,7 @@ Release notes, technical updates, examples, and deployment stories from the Mode + @@ -30,6 +31,12 @@ Release notes, technical updates, examples, and deployment stories from the Mode
+
+
October 8, 2026 · Model Optimizer Team
+

AutoQuant KV: Automatic Mixed-Precision KV Cache Quantization

+

Learn how AutoQuant selects per-layer KV formats under a storage budget and combines with GEMM quantization, with Qwen3-8B as a case study.

+
autoquantizekv-cachequantizationmixed-precisionmodelopt
+