Repository navigation
[Nemotron 3.5 Super VL] Update the NemotronH NVFP4 PTQ recipe to the Nemotron 3.5 Super VL checkpoint - #2819
Merged
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Docstring Coverage |  |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Security Anti-Patterns | |
PASS. The PR changes only recipe/docs files and renames a Python unit test while changing its recipe path string. The added Python lines introduce no torch.load, numpy.load, trust_remote_code, eval/ex… |
| Description Check | |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly describes the main change: moving the NemotronH NVFP4 PTQ recipe to the Nemotron 3.5 Super VL checkpoint-specific recipe. |
✨ Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2819 +/- ##
==========================================
+ Coverage 69.45% 78.61% +9.15%
==========================================
Files 653 653
Lines 72522 72522
==========================================
+ Hits 50372 57011 +6639
+ Misses 22150 15511 -6639
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
juhi10071998
changed the title
meenchen
approved these changes
Oct 15, 2026
The aggressive-MSE NVFP4 recipe is the PTQ stage for NVIDIA Nemotron 3.5 Super VL, but sat under `model_type/nemotron_h/` and named the model nowhere, so nothing connected it to the release it was written for and the QAD tutorial selected it by a path that named neither. Move it to the released source checkpoint's model-hub path, `models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse`, and select it from there in the QAD tutorial. One recipe, one path: a checkpoint-keyed entry alongside the architecture one would have been two names for the same body. The recipe has not shipped in any release -- it merged after the 0.48.0rc0 cut -- so no published `--recipe` path points at the old location and the move needs no compatibility alias. Scope is stated in the header, the description and ptq.md: running this reproduces the released precision assignment, not the released weights. The published NVFP4 checkpoint is refined by quantization-aware distillation after this stage, which no PTQ recipe performs. A recipe that declares no `$import` has to declare its schema, so it gains the `# modelopt-schema:` comment and drops the deprecated `metadata.recipe_type`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>
juhi10071998
force-pushed
the
nemotron_3_5_super_naming
branch
from
October 15, 2026 17:16
789bc83 to
91e7772
Compare
yeyu-nvidia
approved these changes
Oct 15, 2026
Contributor
|
meenchen
added a commit
that referenced
this pull request
Oct 15, 2026
### What does this PR do? Type of change: New feature Adds one checkpoint-specific AutoQuantize recipe for `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`, following the layout merged in #2819. The recipe lives at `models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/auto_quantize/nvfp4_mse_fp8_mtp_fixed_at_4p9bits`, alongside the checkpoint's PTQ recipe, rather than advertising these tuning choices as defaults for all Nemotron-H models. The previous architecture-tier path was only in this unmerged PR and is not retained as an alias. Gradient-based sensitivity scoring searches decoder modules and `lm_head` over weight-MSE NVFP4, FP8, and BF16 at a 4.9-effective-bit search budget. MTP routed/shared experts stay fixed at MSE-calibrated NVFP4 W4A4; other MTP modules and vision/audio remain BF16. MTP KV uses calibrated FP8 scales collected during fixed-baseline calibration and preserved after search. Backbone KV uses fixed-range FP8 casting (`use_constant_amax`, amax 448), without stored KV scales. The move changes no imports, quantization rules, search settings, or cost exclusions. Static NVFP4 is locally costed at 4.5 bits per weight, including one FP8 scale per 16 weights. Fixed MTP and configured non-language modules are excluded from the search cost, so the target is not an all-parameter checkpoint-size guarantee. This generates a mixed-precision search candidate; it does not reproduce the published QAD-refined NVFP4 weights. ### Usage ```bash python examples/hf_ptq/hf_ptq.py \ --pyt_ckpt_path nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 \ --recipe models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/auto_quantize/nvfp4_mse_fp8_mtp_fixed_at_4p9bits \ --calib_size 128 \ --use_seq_device_map \ --export_path <quantized-checkpoint> ``` Review the checkpoint's modeling code before adding `--trust_remote_code` if required. The example AutoQuantize CLI currently rejects `--calib_with_images`; the sibling PTQ recipe's image-calibration invocation does not apply to this CLI path. ### Testing Rebased onto `f0daaa241`, the merge of #2819. Tested on CPU with Python 3.12.13, PyTorch 2.13.0, Transformers 5.14.0, and pytest 9.1.1: ```bash PYTHONPATH=. HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest \ tests/unit/recipe tests/unit/torch/models/test_nemotron_h_mtp.py \ -q --disable-warnings --tb=short PYTHONPATH=. HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest \ tests/unit/torch/quantization/test_autoquant.py \ -k fixed_ptq_baseline -q --disable-warnings --tb=short ``` Results: **605 passed** in 15.97 seconds, plus **6 passed** in 7.91 seconds. This includes all eight moved AutoQuantize cases, the eleven upstream checkpoint-specific PTQ cases, and the MTP suite. AutoQuantize regressions cover resolved search/fixed groups, NVFP4 costs, routed/shared-expert formats, calibrated MTP KV collection and exported scales, and backbone KV casting without scales. A semantic comparison against the previously published recipe confirms that every non-metadata YAML field is unchanged. The new checkpoint path loads as `ModelOptAutoQuantizeRecipe` using its schema comment, without deprecated `metadata.recipe_type`. Changed-file pre-commit checks passed; the environment-creating recipe hook was skipped and its actual `tools/precommit/check_modelopt_recipes.py` validator passed separately in the dependency-equipped environment. Staged credential audit and diff checks passed. No production-model GPU E2E, accuracy, or speculative-acceptance test was rerun for this path/metadata move. CPU KV tests disable NVFP4 weight/input quantizers. The GPU PTQ E2E results in #2819 are not validation of this AutoQuantize recipe; fresh CI remains required. ### Before your PR is "*Ready for review*" Followed the [contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and [security guidance](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors). Commits are signed and include DCO sign-offs. - Is this change backward compatible?: Yes; the architecture-tier AutoQuantize path never merged or shipped. - If you copied code from other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md?: N/A; no new runtime dependency or copied third-party code. - Did you write any new necessary tests?: Yes; runtime recipe regressions use the checkpoint-specific path. - Did you update Changelog?: Yes. - Did you get Claude approval on this PR?: No new approval claimed. ### Additional Information Uses the existing AutoQuantize fixed-baseline composition and the MTP loading/calibration/export support merged in #2581. No loader, MTP implementation, image-calibration, or expert-parallel changes are included. Naming and placement follow #2819, but the recipe performs a new sensitivity-driven assignment rather than the sibling PTQ recipe's fixed assignment. --------- Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change: Refactor + docs. No behavior change.
Moves the NemotronH aggressive-mse NVFP4 PTQ recipe from the architecture tier to the checkpoint
tier, under the name the model is actually released as:
Why the checkpoint tier. The recipe was written against, and only ever validated on, one
checkpoint:
nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16. Undermodels/<org>/<model_id>/the folder path is the hub path, so the entry states which checkpoint it reproduces instead of
implying it is the recommended setting for every NemotronH hybrid. It also matches the existing
NVIDIA-Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mseprecedent. This is a move, not an alias —one name, so nobody has to work out which of two paths to use.
Scope of the claim. The recipe reproduces the released precision assignment, not the released
weights. The published
nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4is refined byquantization-aware distillation after this stage, which no PTQ recipe performs. The header,
metadata.description,ptq.mdand the changelog all say so explicitly. Flagging it becausemodels/README.mdsays a release is backfilled only when its model card describes a PTQ recipe —this entry is the PTQ stage of that pipeline rather than a claim to reproduce the artefact, and
I am happy to move it back or reword further if reviewers read that rule more strictly.
The 43
quant_cfgrules are unchanged. Only the path, the header comments andmetadata.descriptiondiffer, plus# modelopt-schema: modelopt.recipe.config.ModelOptPTQRecipereplacing the deprecated
metadata.recipe_type: ptq.Also updated:
modelopt_recipes/ptq.md(the architecture-tier entry is folded into acheckpoint-tier entry carrying its full detail), the
NemotronH-Omni-W4A4-QADtutorial README,which now selects the new path, and
CHANGELOG.rst.One consequence worth a reviewer's opinion:
model_type/nemotron_h/no longer exists, so otherNemotronH hybrids fall back to
general/. That was already the situation before #2703 landed, andnothing shipped in between — the recipe is absent from every released tag (0.48.0rc0, 0.47.0,
0.47.0rc2, 0.47.0rc1, 0.46.1), since it merged after the rc0 cut. No published wheel contains the
old path, and all references to it were in-repo and are updated here. If you would rather keep an
architecture-tier recipe for the family, say so and I will restore one that is genuinely generic.
Usage
The runtime notes from #2703 still apply:
--use_seq_device_mapand--gpu_max_mem_percentage 0.5are both required for a ~232 GiB checkpoint on 4x184 GiB, and
--calib_with_imagesmatters on VLcheckpoints because the vision tower must see image batches even though it stays BF16.
Testing
Run on 1 node x 4 x 184 GiB, transformers 5.13.0 / torch 2.12.0a0, against this branch rebased onto
de53aca3a.tests/unit/recipe/test_nemotron_3_5_super_vl_recipe.pytests/examples/hf_ptq/test_nemotron_vl_calib.pytests/unit/recipe/suitetools/precommit/check_modelopt_recipes.pyhf_ptq,--calib_size 32rc=0, 28:22, 0 tracebacksBecause the quantization rules are unchanged, the meaningful check is byte-level rather than a
benchmark. Two exports were produced on the same base commit, differing only in which recipe
they were pointed at:
models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mseorigin/mainatde53aca3a,model_type/nemotron_h/ptq/nvfp4-aggressive-mseHolding the base fixed matters:
de53aca3aitself touchesquant_aware_conversion.py, so acomparison against an older export would have conflated that fix with this rename. Both arms were
verified to carry the identical
quant_aware_conversion.py(24aa840e46…).k_scale+ 1v_scale, both MTP, none on the backboneSo the moved recipe selects the same 43,917 quantizers through the checkpoint tier that the
pre-move recipe selects through the architecture tier, and the resulting checkpoint is identical
byte for byte.
Byte-identity is the right test here and GSM8K is not: two runs over byte-identical artefacts of
this checkpoint differed by ~1.1 pp, so no single benchmark number could have resolved a change
this small.
The run also confirms the moved path resolves at runtime rather than falling back:
Before your PR is "Ready for review"
wheel exposes it, and every in-repo reference is updated here.
N/A; no new dependency or copied code.
tests/unit/recipe/test_nemotron_h_recipe.pyis renamed to
test_nemotron_3_5_super_vl_recipe.pyto match the recipe, and its 11 cases areotherwise unchanged and still green.
🤖 Generated with Claude Code