Repository navigation
docs: add AutoQuant KV mixed-precision quantization blog - #2712
Conversation
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to The announcement discovery changes are mergeable after normal documentation checks. Pre-merge checks ·
 |
| Check name | Status | Explanation |
|---|---|---|
| Description Check |  |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly and concisely describes the main change: adding an AutoQuant KV mixed-precision quantization blog. |
| Docstring Coverage | |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Security Anti-Patterns | |
PASS. The authoritative PR diff changes documentation and data/assets only. It adds no files under modelopt/ or Python example files, and it changes no pyproject.toml or requirements.txt manifes… |
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2712 +/- ##
==========================================
+ Coverage 69.46% 71.61% +2.14%
==========================================
Files 647 649 +2
Lines 71679 71739 +60
==========================================
+ Hits 49794 51377 +1583
+ Misses 21885 20362 -1523
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
|
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
cjluo-nv
left a comment
cjluo-nv
left a comment
There was a problem hiding this comment.
Bot review (bedrock-claude-opus-5-5) — DM the bot to share feedback.
LGTM: this docs-only blog is correct and internally consistent. The size budget doesn't apply because there's no core logic, but a human should sign off on the 1.8k lines of generated figures and the public claims.
Needs action:
- Sign off as a human on the published claims and the rendered preview. The bulk of the diff is generated SVG plus CSV/JSON study data, and the results come from an undistributed development vLLM runtime.
- Confirm that pinned commit
1d392999b45626f9c06ff0bc9797b03529fe24e3in the install snippet is onmainand contains #2272/#2273.
No action needed:
- The design-review gate fired on line count only. The PR adds no new subsystem, abstraction or dependency.
- I verified both recipe paths in
modelopt_recipes/general/auto_quantize/, the--kv_auto_quantize_checkpointflag inhf_ptq.py, andcost_model: kv_cachein the recipe. - The headline numbers match the CSV:
- The 64.0% payload cut (5.764/16).
- 7.8% less payload than Native 6.25.
- 4.50 GiB of BF16 KV payload at 32K tokens.
- The ±0.63-point claim across five benchmarks.
- The half-up rounding, e.g. −1.875 → −1.88.
- There's no licensing change and no test code is affected.
| K/V (8 bits) and NVFP4 K/V (4.5 bits including block scales). The case study | ||
| also includes FP8 K with NVFP4 V (6.25 bits for equal K/V widths). | ||
|
|
||
| **Score each choice.** Compare each isolated layer/candidate trial against a |
There was a problem hiding this comment.
hi @meenchen , shall we refine this section further. At first reading it may not be very clear whether the GEMM are quantized always or not when computing the KL Divergence. For the two cases for fp8 KV vs NVFP4 KV, we will have two choices for layer X KV$.
- output using the bf16 KV$ in layer X- do GEMM stay quantized here?
- output using the fp8 KV$ in layer X- do GEMM stay same quantized here?
- output using the NVFP4 KV$ in layer X- do GEMM stay same quantzied here?
Then we compute KL divergence between 1 and 2 and 1 and 3.
May be helpful to confirm if for #1 GEMM is quantized and stays in same precision as 2 and 3
There was a problem hiding this comment.
Addressed in 6f9d008. The scoring paragraph and diagram now explicitly keep GEMM formats and scales fixed in the reference and every candidate trial. Starting from BF16 preserves BF16 GEMMs; starting from a quantized model preserves that same GEMM configuration throughout. Each trial changes only the selected layer's K/V pair, and both candidate outputs are compared with the same reference.
| K/V (8 bits) and NVFP4 K/V (4.5 bits including block scales). The case study | ||
| also includes FP8 K with NVFP4 V (6.25 bits for equal K/V widths). | ||
|
|
||
| **Score each choice.** Compare each isolated layer/candidate trial against a |
There was a problem hiding this comment.
hi @meenchen , shall we refine this section further. At first reading it may not be very clear whether the GEMM are quantized always or not when computing the KL Divergence. For the two cases for fp8 KV vs NVFP4 KV, we will have two choices for layer X KV$.
- output using the bf16 KV$ in layer X- do GEMM stay quantized here?
- output using the fp8 KV$ in layer X- do GEMM stay same quantized here?
- output using the NVFP4 KV$ in layer X- do GEMM stay same quantzied here?
Then we compute KL divergence between 1 and 2 and 1 and 3.
May be helpful to confirm if for #1 GEMM is quantized and stays in same precision as 2 and 3
There was a problem hiding this comment.
Addressed in 6f9d008. The scoring paragraph and diagram now explicitly keep GEMM formats and scales fixed in the reference and every candidate trial. Starting from BF16 preserves BF16 GEMMs; starting from a quantized model preserves that same GEMM configuration throughout. Each trial changes only the selected layer's K/V pair, and both candidate outputs are compared with the same reference.
|
|
||
| **Prepare candidates.** The starter recipe uses constant scales and offers FP8 | ||
| K/V (8 bits) and NVFP4 K/V (4.5 bits including block scales). The case study | ||
| also includes FP8 K with NVFP4 V (6.25 bits for equal K/V widths). |
There was a problem hiding this comment.
should we mention a line whether the case of NVFP4 K and FP8 V makes sense, or why we excluded that?
There was a problem hiding this comment.
Addressed in 6f9d008. Added a note under candidate preparation explaining that the current export path does not support NVFP4 K with FP8 V. The study did not evaluate that pair's accuracy; the documented exclusion is an export limitation.
| weight/activation quantizers and calibrated tensors, then runs forward-KL KV | ||
| search. Its 5.4-bit weight and 5.4-bit KV targets constrain separate storage | ||
| domains; this is a sequential search, not one joint bit budget. | ||
|
|
There was a problem hiding this comment.
Do we add a recommendation for users to always run the KV$ autoquantize alongside the GEMM scores- I see little value if we run the mixed precision on KV$ alone as we may not get a good signal if the GEMM is not quantized.
Do we actually have cases where we may want to run the KV$ AutoQuantize only?
There was a problem hiding this comment.
Addressed in 6f9d008. The GEMM composition section now recommends scoring KV candidates on the GEMM configuration intended for deployment, applying GEMM AutoQuant or fixed PTQ first when GEMMs will be quantized. It also explains the KV-only use cases: BF16 deployments, controlled KV studies such as this case study, and adding KV compression to an already-quantized model. Changing GEMM settings still requires recomputing KV sensitivity.
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
|
/ok to test 87f27b2 |
4 participants
What does this PR do?
Type of change: documentation
Introduce AutoQuant KV as a general mixed-precision KV cache quantization feature. The blog explains isolated layer scoring and budget-constrained format selection with a diagram and compact equation, demonstrates the Python API and CLI, and shows how KV search follows GEMM AutoQuant.
The scoring explanation makes explicit that GEMM formats and scales stay fixed in the reference and every trial. Deployment guidance recommends scoring on the intended GEMM configuration, covers BF16 and already-quantized KV-only use cases, and explains that NVFP4-K/FP8-V is excluded by the current export path rather than an accuracy comparison.
A Qwen3-8B case study compares default-scale recipes against BF16 across seven benchmarks, including Native 6.25 and AutoQuant 5.33/5.76. At 5.76 effective bits, AutoQuant uses 64% less calculated KV payload than BF16 with a −1.04 percentage-point RULER-128K change. Every recipe difference is result minus BF16.
Three vector SVG figures, the CSV, and evaluation settings accompany the article. The case study uses BF16 weights and a development runtime with asymmetric K/V candidates; the shipped starter example uses a different candidate menu. Payload savings are calculated, accuracy differences are point estimates, and unmatched controls are withheld.
Usage
No new API or flag. The article demonstrates
modelopt.torch.quantization.auto_quantizeand the shippedgeneral/auto_quantize/kv_fp8_nvfp4_cast_kl_div_at_5p4bitsrecipe, plus the composed GEMM/KV recipe.Testing
git diff --checkpassed.1d392999b45626f9c06ff0bc9797b03529fe24e3is the merge commit for [OMNIML-5570] 2/2 Compose GEMM and KV-cache AutoQuant workflows #2273, includes [OMNIML-5570, OMNIML-5569] 1/2 Add layer-wise KV-cache AutoQuant with forward KL #2272, and is an ancestor ofmain.No new GPU evaluation or full API documentation build was run.
Before your PR is "Ready for review"
CONTRIBUTING.md: N/A. No new project dependency or copied implementation.158c3baLGTM; the follow-up documentation clarifications were validated locally.Additional Information
Documents #2272 and #2273. Follows the announcement and asset conventions used by #1979 and #2417. The generated diagram and saved-result artifacts are included with the article so reviewers can check its methodology and claims together.
Summary by CodeRabbit