From 91e77727dacbf73254b3ad6168d726f25797c8c3 Mon Sep 17 00:00:00 2001 From: Juhi Mittal Date: Thu, 15 Oct 2026 04:35:02 +0000 Subject: [PATCH] recipe: key the Nemotron 3.5 Super VL PTQ recipe to its checkpoint The aggressive-MSE NVFP4 recipe is the PTQ stage for NVIDIA Nemotron 3.5 Super VL, but sat under `model_type/nemotron_h/` and named the model nowhere, so nothing connected it to the release it was written for and the QAD tutorial selected it by a path that named neither. Move it to the released source checkpoint's model-hub path, `models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse`, and select it from there in the QAD tutorial. One recipe, one path: a checkpoint-keyed entry alongside the architecture one would have been two names for the same body. The recipe has not shipped in any release -- it merged after the 0.48.0rc0 cut -- so no published `--recipe` path points at the old location and the move needs no compatibility alias. Scope is stated in the header, the description and ptq.md: running this reproduces the released precision assignment, not the released weights. The published NVFP4 checkpoint is refined by quantization-aware distillation after this stage, which no PTQ recipe performs. A recipe that declares no `$import` has to declare its schema, so it gains the `# modelopt-schema:` comment and drops the deprecated `metadata.recipe_type`. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Juhi Mittal --- CHANGELOG.rst | 2 +- .../NemotronH-Omni-W4A4-QAD/README.md | 4 +- .../ptq/nvfp4-mse.yaml} | 27 +++++++---- modelopt_recipes/ptq.md | 48 ++++++++++--------- ...y => test_nemotron_3_5_super_vl_recipe.py} | 4 +- 5 files changed, 48 insertions(+), 37 deletions(-) rename modelopt_recipes/{model_type/nemotron_h/ptq/nvfp4-aggressive-mse.yaml => models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse.yaml} (84%) rename tests/unit/recipe/{test_nemotron_h_recipe.py => test_nemotron_3_5_super_vl_recipe.py} (98%) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index f4e55866e52..f71c87b25b3 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -29,7 +29,7 @@ Changelog - Add Q8_0 weight-only quantization with 32-value GGML blocks, packed unified HF and Megatron export, and a built-in ``q8_0`` PTQ recipe. - Add Hugging Face PTQ calibration and export support for Nemotron-H MTP modules, preserving calibrated expert input scales. -- Add the ``model_type/nemotron_h`` aggressive-MSE NVFP4 PTQ recipe: weight-MSE calibrated NVFP4 routed and shared MoE experts, FP8 attention projections, Mamba ``in_proj``/``out_proj`` and ``lm_head``, and an FP8 KV cache. The latent MoE, router, embeddings and the vision and audio encoders stay in BF16, and the MTP rules apply only to checkpoints that carry an MTP tail. +- Add the ``models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse`` PTQ recipe, the PTQ stage for NVIDIA Nemotron 3.5 Super VL: weight-MSE calibrated NVFP4 routed and shared MoE experts, FP8 attention projections, Mamba ``in_proj``/``out_proj`` and ``lm_head``, and an FP8 KV cache. The latent MoE, router, embeddings and the vision and audio encoders stay in BF16, and the MTP rules apply only to checkpoints that carry an MTP tail. It reproduces the released precision assignment, not the released weights: the published NVFP4 checkpoint adds quantization-aware distillation after this stage. - Backfill checkpoint aliases for three more published NVFP4 releases: ``moonshotai/Kimi-K2.7-Code``, ``google/gemma-4-26B-A4B-it`` and ``google/diffusiongemma-26B-A4B-it``. Each imports an existing general recipe wholesale rather than copying its body. - Backfill checkpoint aliases for nine more published NVFP4 releases, so each is reachable from its source model's hub path: ``zai-org/GLM-5.1`` and ``GLM-5.2``, ``MiniMaxAI/MiniMax-M2.5`` and ``MiniMax-M3``, ``deepseek-ai/DeepSeek-V3.1`` and ``DeepSeek-V3.2``, ``Qwen/Qwen3-235B-A22B-Instruct-2507`` and ``-Thinking-2507``, and ``Qwen/Qwen3.6-27B``. Each imports an existing general or architecture recipe wholesale rather than copying its body. - Add composed Hugging Face AutoQuantize recipes that run fixed PTQ or weight AutoQuantize before diff --git a/examples/megatron_bridge/tutorials/NemotronH-Omni-W4A4-QAD/README.md b/examples/megatron_bridge/tutorials/NemotronH-Omni-W4A4-QAD/README.md index c2036264699..461f2e201c9 100644 --- a/examples/megatron_bridge/tutorials/NemotronH-Omni-W4A4-QAD/README.md +++ b/examples/megatron_bridge/tutorials/NemotronH-Omni-W4A4-QAD/README.md @@ -52,7 +52,7 @@ This writes `train.jsonl` (35,800 conversations) and `validation.jsonl` (1,000) ### 2. Quantization -The recipe is [`model_type/nemotron_h/ptq/nvfp4-aggressive-mse`](../../../../modelopt_recipes/model_type/nemotron_h/ptq/nvfp4-aggressive-mse.yaml), the same file `hf_ptq.py` uses: its rules list both the Hugging Face and the Megatron-Core module names where they differ. +The recipe is [`models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse`](../../../../modelopt_recipes/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse.yaml), the PTQ stage for NVIDIA Nemotron 3.5 Super VL and the same file `hf_ptq.py` uses: its rules list both the Hugging Face and the Megatron-Core module names where they differ. | Component | Precision | | --- | --- | @@ -74,7 +74,7 @@ PTQ runs on **one 4-GPU GB300 node** with 512 calibration samples. torchrun --nproc_per_node 4 /opt/Model-Optimizer/examples/megatron_bridge/quantize.py \ --hf_model_name_or_path \ --trust_remote_code \ - --recipe model_type/nemotron_h/ptq/nvfp4-aggressive-mse \ + --recipe models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse \ --tp_size 1 --ep_size 4 --pp_size 1 \ --calib_dataset_name cnn_nemotron_v2_mix \ --calib_num_samples 512 \ diff --git a/modelopt_recipes/model_type/nemotron_h/ptq/nvfp4-aggressive-mse.yaml b/modelopt_recipes/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse.yaml similarity index 84% rename from modelopt_recipes/model_type/nemotron_h/ptq/nvfp4-aggressive-mse.yaml rename to modelopt_recipes/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse.yaml index e539eb881f9..3007a6b8adf 100644 --- a/modelopt_recipes/model_type/nemotron_h/ptq/nvfp4-aggressive-mse.yaml +++ b/modelopt_recipes/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse.yaml @@ -14,11 +14,17 @@ # limitations under the License. -# NemotronH -- aggressive NVFP4 tier, weight-MSE calibration. +# NVIDIA Nemotron 3.5 Super VL -- NVFP4, weight-MSE calibration. # -# For hybrid Mamba-MoE NemotronH checkpoints, including the omni VL wrappers. Broader FP8 weight -# coverage than the conservative tier; the MTP rules apply only when the checkpoint carries an -# MTP tail and are inert otherwise. +# The PTQ stage for NVIDIA Nemotron 3.5 Super VL, keyed to the BF16 source checkpoint it runs +# on: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16. It reproduces the released precision +# assignment, not the released weights -- the published +# nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 is refined by quantization-aware +# distillation after this stage, which no PTQ recipe performs. +# +# The model is a hybrid Mamba-MoE NemotronH omni checkpoint, and nothing here is specific to +# this release beyond the calibration choices, so the rules apply to NemotronH hybrids +# generally. The MTP rules are inert on a checkpoint with no MTP tail. # # MoE routed + shared experts NVFP4 W4A4, static weight scales # attention q/k/v/o, Mamba in/out_proj, lm_head FP8 @@ -34,18 +40,21 @@ # names. Where the two differ -- attention, lm_head and the MTP block -- both spellings are # listed; each is inert on the other framework's model. +# modelopt-schema: modelopt.recipe.config.ModelOptPTQRecipe imports: nvfp4: configs/numerics/nvfp4 nvfp4_static: configs/numerics/nvfp4_static fp8: configs/numerics/fp8 metadata: - recipe_type: ptq description: >- - NemotronH aggressive tier, weight-MSE calibrated: NVFP4 routed and shared MoE experts with - static weight scales and dynamic NVFP4 activations, the same for the MTP block-1 experts; - FP8 attention projections, Mamba in/out_proj and lm_head; BF16 latent MoE and vision tower. - FP8 KV cache, with the backbone cast at a fixed range and MTP calibrated. + PTQ stage for NVIDIA Nemotron 3.5 Super VL, run on + nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 and weight-MSE calibrated: NVFP4 routed + and shared MoE experts with static weight scales and dynamic NVFP4 activations, the same for + the MTP block-1 experts; FP8 attention projections, Mamba in/out_proj and lm_head; BF16 + latent MoE, router, embeddings and vision stack. FP8 KV cache, with the backbone cast at a + fixed range and MTP calibrated. The released NVFP4 checkpoint adds quantization-aware + distillation on top of this stage. quantize: algorithm: method: mse diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index ee37ec72dcf..6cbab877450 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -308,7 +308,7 @@ that baseline. The deviations come in four kinds: | Kind | What changes vs. the general recipe | Examples | |------|-------------------------------------|----------| -| **Architecture-aware `quant_cfg`** | Per-sub-module format choices a single wildcard scheme can't express | `minimax_m3_vl`, `qwen3_vl`, `qwen3_5`, `qwen3_5_moe`, `qwen3_6_moe`, `vit`, `nemotron_llama`, `nemotron_h` | +| **Architecture-aware `quant_cfg`** | Per-sub-module format choices a single wildcard scheme can't express | `minimax_m3_vl`, `qwen3_vl`, `qwen3_5`, `qwen3_5_moe`, `qwen3_6_moe`, `vit`, `nemotron_llama` | | **Algorithm override** | Same numerics & scope, but the *calibration algorithm* is tweaked because the default breaks or regresses | `gemma`, `gemma4`, `mpt` | | **Extra exclusions** | Adds disabled-quantizer patterns so non-language branches stay full precision | `nemotron_vl`, `diffusion_gemma` | | **Checkpoint mirror** | A mixed-precision map reproducing one published checkpoint exactly | `models/nvidia/NVIDIA-Nemotron-3-*`, `models/mistralai/Mistral-Medium-3.5-128B` | @@ -326,7 +326,7 @@ folder may carry a `README.md` spelling out that delta. > genuinely deviate and aliases the ones that don't; see > [`models/README.md`](models/README.md) for the alias format and when to write one. -### Architecture-aware `quant_cfg` — `minimax_m3_vl`, `qwen3_vl`, `qwen3_5`, `qwen3_5_moe`, `qwen3_6_moe`, `vit`, `nemotron_llama`, `nemotron_h` +### Architecture-aware `quant_cfg` — `minimax_m3_vl`, `qwen3_vl`, `qwen3_5`, `qwen3_5_moe`, `qwen3_6_moe`, `vit`, `nemotron_llama` **`minimax_m3_vl/ptq/mxfp8_nvfp4_experts`** applies MXFP8 to the language-model linear layers and MSE-calibrated NVFP4 to routed experts, with expert @@ -407,27 +407,6 @@ recipes, for Step-3.7 checkpoints; Step-3.5 has its own recipe above. load-bearing: selectors are Megatron-Core leaf names (`mlp.experts.linear_fc1`, `self_attention.linear_qkv`), so under `hf_ptq.py` nothing matches and `mtq.quantize` raises. -- **`nemotron_h/ptq/nvfp4-aggressive-mse`** is the **aggressive** tier for hybrid Mamba-MoE - NemotronH checkpoints, including the omni VL wrappers: - - MoE routed + shared experts → NVFP4 W4A4, `group_size 16`, **static** weight scales - (`nvfp4_static`) with dynamic NVFP4 activations (`nvfp4`) - - attention q/k/v/out, Mamba `in/out_proj`, `lm_head` → FP8 - - MTP block-1 routed + shared experts → identical treatment to the main experts; the MTP rules - are inert on checkpoints without an MTP tail - - latent MoE, embeddings, router and the vision tower → **BF16** - - KV cache → FP8, **asymmetric**: the language-model backbone casts at a fixed FP8 range - (`use_constant_amax`, as in `configs/ptq/units/kv_fp8_cast`, so no `k_scale`/`v_scale` is - exported) while MTP attention stays data-calibrated and keeps its pair - - Calibration is weight-MSE with an FP8-scale sweep, which is why the weight quantizers are - `static`. Use `--calib_with_images` on VL checkpoints: the vision tower must see image batches - even though it stays BF16. - - Rules list both Hugging Face and Megatron-Core spellings where they differ (attention - `linear_qkv`/`linear_proj`, `output_layer`, the MTP `mtp_model_layer.layers`), so the same - recipe applies under `examples/megatron_bridge/quantize.py`; there, static NVFP4 weights need - `--tp_size 1`. - ### Algorithm overrides — `gemma`, `gemma4`, `mpt` These quantize the **same layers** as the general recipes; only the @@ -523,6 +502,29 @@ checkpoint's** quant config verbatim: Four-over-Six NVFP4 W4A16 to routed experts, shared experts, and the language model head; Mamba `in/out_proj` weights and inputs plus the KV cache use FP8, while attention remains BF16. +- **`models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse`** is the PTQ + stage for **NVIDIA Nemotron 3.5 Super VL**, keyed to the BF16 source checkpoint it runs on. It + reproduces the released **precision assignment**, not the released weights — + `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` adds quantization-aware distillation + after this stage. The model is a hybrid Mamba-MoE NemotronH omni checkpoint: + - MoE routed + shared experts → NVFP4 W4A4, `group_size 16`, **static** weight scales + (`nvfp4_static`) with dynamic NVFP4 activations (`nvfp4`) + - attention q/k/v/out, Mamba `in/out_proj`, `lm_head` → FP8 + - MTP block-1 routed + shared experts → identical treatment to the main experts; the MTP rules + are inert on checkpoints without an MTP tail + - latent MoE, embeddings, router and the vision tower → **BF16** + - KV cache → FP8, **asymmetric**: the language-model backbone casts at a fixed FP8 range + (`use_constant_amax`, as in `configs/ptq/units/kv_fp8_cast`, so no `k_scale`/`v_scale` is + exported) while MTP attention stays data-calibrated and keeps its pair + + Calibration is weight-MSE with an FP8-scale sweep, which is why the weight quantizers are + `static`. Use `--calib_with_images` on VL checkpoints: the vision tower must see image batches + even though it stays BF16. + + Rules list both Hugging Face and Megatron-Core spellings where they differ (attention + `linear_qkv`/`linear_proj`, `output_layer`, the MTP `mtp_model_layer.layers`), so the same + recipe applies under `examples/megatron_bridge/quantize.py`; there, static NVFP4 weights need + `--tp_size 1`. - **`models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16`** mirrors the GGUF **Q4_K_M** bit allocation of the Nemotron-H hybrid, mapped onto NVFP4/FP8 **per layer**: Q4_K/Q5_0 linears → NVFP4 W4A4 (attention q/k/v/o kept uniform so export can diff --git a/tests/unit/recipe/test_nemotron_h_recipe.py b/tests/unit/recipe/test_nemotron_3_5_super_vl_recipe.py similarity index 98% rename from tests/unit/recipe/test_nemotron_h_recipe.py rename to tests/unit/recipe/test_nemotron_3_5_super_vl_recipe.py index 2d5d531ab60..c04f1d8f96b 100644 --- a/tests/unit/recipe/test_nemotron_h_recipe.py +++ b/tests/unit/recipe/test_nemotron_3_5_super_vl_recipe.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Effective quantizer selection for ``model_type/nemotron_h/ptq/nvfp4-aggressive-mse``. +"""Effective quantizer selection for ``models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse``. The recipe's ``quant_cfg`` is ordered and last-match-wins, so what a module ends up with depends on rule order as much as on rule content. The KV asymmetry in particular is load @@ -32,7 +32,7 @@ from modelopt.recipe import load_recipe from modelopt.torch.quantization.nn import QuantModule, QuantModuleRegistry, TensorQuantizer -_RECIPE = "model_type/nemotron_h/ptq/nvfp4-aggressive-mse" +_RECIPE = "models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse" _WIDTH = 32