Require transformers>=5.5 and remove transformers 4.x code paths

CHANGELOG.rst

@@ -53,11 +53,11 @@ Changelog
 
 **Backward Breaking Changes**
 
+- Bump minimum transformers version to ``5.5`` instead of ``4.57``; transformers 4.x is no longer supported. Upgrade with ``pip install -U "nvidia-modelopt[hf]"``.
 - ``modelopt.torch.distill.plugins.megatron.TopKLogitsKLLoss`` (``logit_kl_topk`` in ``DistillationConfig``) is renamed to ``TopLogitsKLLoss``, keeping the old name as a deprecated alias. It now normalizes both distributions over the full vocabulary instead of re-normalizing over the Top-K entries, and always appends a "ghost" token holding the probability mass outside the Top-K to both student and teacher (matching Megatron-LM's offline cached-logits KD loss). Loss values change for existing ``logit_kl_topk`` runs.
 - ``LogitsAndIntermediatesLossBalancer`` (Megatron distillation plugin) no longer rescales the distillation loss to the magnitude of the LM loss when the LM loss is included. The total is now the fixed convex combination ``(1 - alpha) * lm_loss + alpha * kd_loss`` with ``DistillationConfig.kd_loss_alpha`` (in [0, 1]), matching Megatron-LM's offline cached-logits KD. The default ``kd_loss_alpha=1.0`` skips the LM loss, as before. ``DistillationConfig.skip_lm_loss`` and ``kd_loss_scale`` are removed and raise ``ValueError`` if passed; set ``kd_loss_alpha`` instead. In ``examples/megatron_bridge/distill.py``, ``--kd_loss_alpha`` replaces ``--no_skip_lm_loss`` and ``--kd_loss_scale``.
 - The Megatron-Core DeepSeek-V4 indexer (``CSAIndexer``) is now a quantization module and persists its quantizer state in the checkpoint as ``indexer._extra_state``. A DeepSeek-V4 model quantized with an earlier release resumes from its ``torch_dist`` checkpoint only with a non-strict load (``--dist-ckpt-strictness log_unexpected`` in Megatron-LM) until it is saved again.
 - ``examples/hf_ptq`` no longer detects MTP layers by name. Weights the loader could not place -- an MTP head, an auxiliary tower -- are identified from Transformers' own accounting: the model is loaded with ``from_pretrained(..., output_loading_info=True)`` and the reported ``unexpected_keys`` (present in the checkpoint, not in the model's architecture) are recorded on the model and carried into the export unchanged. Everything the loader *did* place goes through the normal export path. This removes ``load_mtp_weights``, ``mtp_layer_prefixes_from_checkpoint`` and their support matrix of MTP storage conventions, along with ``_add_mtp_exclusions`` and the pre-quantization ``enable: False`` entries ``hf_ptq`` appended to the recipe's ``quant_cfg``. Two consequences: MTP layers now follow the recipe like any other module instead of being force-excluded by the script -- matching ``examples/megatron_bridge``, which has no MTP-specific code at all -- and ``quantization_config.ignore`` can no longer claim a layer is unquantized that the export in fact quantized. Recipes importing ``configs/ptq/units/default_disabled_quantizers`` still disable ``mtp.*``, so their behaviour is unchanged; a recipe omitting that unit will now quantize an MTP the model actually built.
-
 - ``examples/hf_ptq --vllm_fakequant_export`` now raises ``NotImplementedError`` when the checkpoint holds weights the model has no parameter for and a shard actually provides them (an MTP head, an auxiliary tower). The fake-quant exporter writes only model-backed state, so it would otherwise drop those weights silently -- and a fake-quant checkpoint is evaluated, where a missing head changes the score rather than failing loudly. Use the unified HF export, which carries them through. Buffers Transformers recomputes are not weights to lose: ``*.inv_freq`` is skipped even when a shard provides it, since older Llama/Mistral-lineage conversions do list it in the index and refusing an export over it would reject checkpoints that export correctly today. The check runs immediately after the model loads, not at export time, so an incompatible run fails before calibration rather than after it.
 - The ``modelopt.onnx.quantization.graph_utils`` module has been removed with no
   compatibility shim; update direct imports using this migration map:
@@ -81,7 +81,6 @@ Changelog
     ``get_layer_precision_mapping``, ``get_resize_scales``, ``print_stat``,
     ``remove_partial_input_qdq``, ``should_quantize_to_8bit``, and
     ``validate_8bit_layers``.
-
 - Layerwise calibration now uses prior-layer QDQ activations by default
   (``layerwise.get_qdq_activations_from_prev_layer=True``). Set it to ``False`` to
   preserve full-precision activations for subsequent layers (the default behavior for

examples/alpamayo/qad.py

@@ -337,12 +337,7 @@ def main():
         "per_device_eval_batch_size": 1,
         "gradient_accumulation_steps": args.grad_accum,
         "learning_rate": args.lr,
-        # transformers 5.15 removed warmup_ratio; from 5.0, a warmup_steps below 1 is a ratio.
-        (
-            "warmup_ratio"
-            if "warmup_ratio" in transformers.TrainingArguments.__dataclass_fields__
-            else "warmup_steps"
-        ): args.warmup_ratio,
+        "warmup_steps": args.warmup_ratio,  # a warmup_steps below 1 is a ratio
         "max_steps": args.max_steps,
         "logging_steps": args.logging_steps,
         "save_steps": args.save_steps,

examples/gpt-oss/configs/sft_full.yaml

@@ -16,7 +16,7 @@ per_device_train_batch_size: 2
 per_device_eval_batch_size: 2
 gradient_accumulation_steps: 2
 max_length: 4096
-warmup_steps: 0.03 # use warmup_ratio if using transformers<5.0
+warmup_steps: 0.03
 lr_scheduler_type: cosine_with_min_lr
 lr_scheduler_kwargs:
   min_lr_rate: 0.1

examples/gpt-oss/configs/sft_lora.yaml

@@ -21,7 +21,7 @@ lora_alpha: 16
 lora_dropout: 0.0
 lora_target_modules: all-linear
 max_length: 4096
-warmup_steps: 0.03 # use warmup_ratio if using transformers<5.0
+warmup_steps: 0.03
 lr_scheduler_type: cosine_with_min_lr
 lr_scheduler_kwargs:
   min_lr_rate: 0.1

examples/gpt-oss/qat-finetune-transformers.ipynb

@@ -207,7 +207,7 @@
     "    per_device_eval_batch_size=1,\n",
     "    gradient_accumulation_steps=2,\n",
     "    max_length=4096,\n",
-    "    warmup_steps=0.03,  # use warmup_ratio if using transformers<5.0\n",
+    "    warmup_steps=0.03,\n",
     "    eval_strategy=\"steps\",\n",
     "    eval_on_start=True,\n",
     "    logging_steps=10,\n",

examples/hf_ptq/README.md

@@ -135,7 +135,7 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http
 > *<sup>6.</sup>Some models currently support export to HF format only.* \
 > *<sup>7.</sup>[PTQ for DeepSeek](../deepseek/README.md)* \
 > *<sup>8.</sup>GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \
-> *<sup>9.</sup>Running Whisper model with transformers>=5.0 requires [torchcodec](https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-cuda-enabled-torchcodec) and other system packages (e.g. ffmpeg).* \
+> *<sup>9.</sup>Running Whisper model requires [torchcodec](https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-cuda-enabled-torchcodec) and other system packages (e.g. ffmpeg).* \
 > *<sup>10.</sup>GPT-OSS ships with native MXFP4 weights; NVFP4 export is produced via the closed-form `--cast_mxfp4_to_nvfp4` cast (see [MXFP4 → NVFP4 cast](#mxfp4--nvfp4-cast-for-gpt-oss)).* \
 > *<sup>11.</sup>Vision-language model (VLM): by default, only the language model is quantized while the vision encoder is kept in high precision. Pass `--vlm` to the shell script (see [VLM quantization](#vlm-quantization)).* \
 > *<sup>12.</sup>For VLMs, `int8_smoothquant` only supports TensorRT-LLM checkpoint export and is not compatible with the TensorRT-LLM torch backend.* \

examples/llm_qat/notebooks/QAT_QAD_Walkthrough.ipynb

@@ -274,7 +274,7 @@
     "    per_device_eval_batch_size=1,\n",
     "    gradient_accumulation_steps=2,\n",
     "    max_length=4096,\n",
-    "    warmup_steps=0.03,  # use warmup_ratio if using transformers<5.0\n",
+    "    warmup_steps=0.03,\n",
     "    eval_strategy=\"steps\",\n",
     "    eval_on_start=True,\n",
     "    logging_steps=50,\n",

examples/puzzletron/requirements.txt

@@ -1,4 +1,2 @@
 math-verify
 ray
-# Likely works for transformers v5 also, but we need to test it
-transformers<5.0

modelopt/torch/__init__.py

@@ -16,7 +16,6 @@
 """Model optimization and deployment subpackage for torch."""
 
 import importlib
-import importlib.util
 import warnings as _warnings
 
 from packaging.version import Version as _Version
@@ -49,22 +48,13 @@
 try:
     from transformers import __version__ as _transformers_version
 
-    if _Version(_transformers_version) < _Version("4.57") or _Version(
+    if _Version(_transformers_version) < _Version("5.5") or _Version(
         _transformers_version
     ) >= _Version("5.19"):
         _warnings.warn(
             f"transformers {_transformers_version} is not tested with current version of modelopt and may cause issues."
             " Please install recommended version with `pip install -U nvidia-modelopt[hf]` if working with HF models.",
         )
-
-    # CVE-2026-4372: transformers<5.3 + the optional `kernels` package allows remote code execution
-    # when loading untrusted models via the Hub kernel-download path. Fixed in 5.3.0. Only warn if
-    # `kernels` is actually installed, since ModelOpt's core install never pulls it in.
-    if _Version(_transformers_version) < _Version("5.3") and importlib.util.find_spec("kernels"):
-        _warnings.warn(
-            f"transformers {_transformers_version} with the `kernels` package is affected by"
-            ' CVE-2026-4372; consider `pip install -U "transformers>=5.3"` when loading untrusted models.',
-        )
 except ImportError:
     pass
 

modelopt/torch/export/layerwise_export.py

@@ -98,7 +98,7 @@ def _tied_quantized_modules(model: nn.Module) -> list[str]:
 
     Grouped by name, which survives offload: a ``data_ptr`` grouping sees nothing when the
     weights are on meta and would pass vacuously. Falls back to ``data_ptr`` when the model
-    publishes no map (transformers < 5).
+    publishes no map (e.g. not a ``PreTrainedModel``).
     """
     tied_map = TiedWeightMap(model)
     groups: dict[str, list[str]] = {}

modelopt/torch/export/model_utils.py

@@ -245,11 +245,11 @@ class TiedWeightMap:
     """
 
     def __init__(self, model: nn.Module) -> None:
-        """Source the tie map from HF's ``all_tied_weights_keys`` (transformers >=5.0).
+        """Source the tie map from HF's ``all_tied_weights_keys``.
 
         HF's ``{target: source}`` == our ``{alias: canonical}``, resolved at load, config-gated,
-        ``torch.equal``-pruned, and name-based so it survives FSDP shard / offload. Absent on
-        transformers <5.0 -> empty map (the ``data_ptr`` backstop in postprocess is the net).
+        ``torch.equal``-pruned, and name-based so it survives FSDP shard / offload. Absent on a
+        non-``PreTrainedModel`` -> empty map (the ``data_ptr`` backstop in postprocess is the net).
         """
         all_tied = getattr(model, "all_tied_weights_keys", None)
         # Warn whenever a tie is declared (embedding tie or any ``_tied_weights_keys`` entry, e.g.
@@ -259,12 +259,8 @@ def __init__(self, model: nn.Module) -> None:
         ) or bool(getattr(model, "_tied_weights_keys", None))
         if all_tied is None and declares_tie:
             warnings.warn(
-                "This model may contain tied/shared weights, but deduplicating them on export "
-                "requires transformers>=5.0 (it uses model.all_tied_weights_keys, which is only "
-                "supported in newer versions). On older versions the exported checkpoint may keep "
-                "duplicate copies of the tied weights (larger files), and tied weights may not be "
-                "deduplicated correctly during export. Upgrade to transformers>=5.0 for correct "
-                "tied-weight export."
+                "This model declares tied weights but has no model.all_tied_weights_keys, so the "
+                "exported checkpoint may keep duplicate copies of the tied weights."
             )
         # Drop any self-entry (alias == canonical): HF should not emit one, but a target==source
         # pair would schedule the kept canonical for deletion, so filter it out defensively.

modelopt/torch/export/unified_export_hf_streaming.py

@@ -185,17 +185,9 @@ def name_shards_and_write_index(
 def _parse_shard_size(size: int | str) -> int:
     """Convert a shard-size string (e.g. ``"10GB"``, ``"500MB"``) to bytes.
 
-    Mirrors transformers' ``convert_file_size_to_int``, which reads ``GB``/``MB``/``KB``
-    as decimal and only ``GiB``/``MiB``/``KiB`` as binary. That helper was removed from
-    ``transformers.utils`` in transformers 5.x, so the fallback below is the live path
-    there, not a rarely-taken branch.
+    Mirrors transformers 4.x's ``convert_file_size_to_int`` (removed in 5.x), which reads
+    ``GB``/``MB``/``KB`` as decimal and only ``GiB``/``MiB``/``KiB`` as binary.
     """
-    try:
-        from transformers.utils import convert_file_size_to_int
-
-        return convert_file_size_to_int(size)
-    except ImportError:
-        pass
     if isinstance(size, int):
         return size
     s = size.strip().upper()
@@ -257,9 +249,9 @@ def _undeclared_tied_aliases(model: nn.Module) -> set[str]:
     goes, so the shared address is gone by then. Take the same information up front, off the
     live model, and drop by name instead.
 
-    Needed on transformers <5.0, where ``all_tied_weights_keys`` does not exist and the tie map
-    is empty; without this the alias ships as a second full copy of the same weight. Walk order
-    matches ``state_dict()``, so the tensor kept here is the one the resident path keeps.
+    Covers ties missing from ``all_tied_weights_keys`` (or a model without it); without this the
+    alias ships as a second full copy of the same weight. Walk order matches ``state_dict()``, so
+    the tensor kept here is the one the resident path keeps.
     """
     seen: dict[int, str] = {}
     aliases: set[str] = set()

modelopt/torch/models/README.md

@@ -21,7 +21,7 @@ __all__: list[str] = []
 register(
     ModelSpec(
         model_type="qwen3_moe",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         # modelopt policy, not a model fact: this model is validated for grouped export.
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
@@ -71,7 +71,7 @@ come from, and they must agree:
 | `min_transformers_version` | Earliest `transformers` release whose definitions match this spec; `None` for a `remote_code` model |
 
 Clamp `min_transformers_version` at the repo's minimum supported transformers (`tf_min`
-in `noxfile.py`, currently `4.57`) — a model older than the floor records the floor,
+in `noxfile.py`, currently `5.5`) — a model older than the floor records the floor,
 since nothing older is installed or tested. A model added later records its own release.
 The pair is what lets `test_specs_vs_transformers.py` assert rather than skip: it can
 tell a model that is legitimately absent on an older transformers from a spec that no

modelopt/torch/models/deepseek_v3/specs.py

@@ -28,15 +28,14 @@
 # structural router+experts test in is_moe matches. block_names below is what puts it on
 # the MoE path at all.
 #
-# The experts container changed upstream. transformers 4.57 builds an nn.ModuleList of
-# DeepseekV3MLP, which is the naming recorded here; 5.x replaces it with a fused, 3-D
-# DeepseekV3Experts holding gate_up_proj/down_proj. The fused layout resolves through
-# the structural first-projection check in get_expert_linear_names and never consults
-# this naming -- the same split Mixtral has across the same releases.
+# The naming recorded here is for an nn.ModuleList of DeepseekV3MLP experts (remote code and
+# transformers 4.x). transformers 5.x uses a fused, 3-D DeepseekV3Experts holding
+# gate_up_proj/down_proj, which resolves through the structural first-projection check in
+# get_expert_linear_names and never consults this naming -- the same split as Mixtral.
 register(
     ModelSpec(
         model_type="deepseek_v3",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
             block_names=("DeepseekV3MoE",),

modelopt/torch/models/gemma/specs.py

@@ -24,7 +24,7 @@
 register(
     ModelSpec(
         model_type="gemma",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(weight_plus_one_norm_names=("GemmaRMSNorm",)),
     )
 )

modelopt/torch/models/gemma2/specs.py

@@ -24,7 +24,7 @@
 register(
     ModelSpec(
         model_type="gemma2",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(weight_plus_one_norm_names=("Gemma2RMSNorm",)),
     )
 )

modelopt/torch/models/gemma3/specs.py

@@ -24,7 +24,7 @@
 register(
     ModelSpec(
         model_type="gemma3",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(weight_plus_one_norm_names=("Gemma3RMSNorm",)),
     )
 )

modelopt/torch/models/gpt_oss/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="gpt_oss",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         moe_spec=MoESpec(
             # GPT-OSS fuses gate and up into a single gate_up_proj.
             # transformers names the block GptOssMLP; GptOssMoE is kept for the

modelopt/torch/models/llama/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="llama",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(
             # AWQ pre_quant_scale fusion: fold o_proj into v_proj, down_proj into up_proj.
             pqs_fuse_rules=(

modelopt/torch/models/minimax/specs.py

@@ -30,7 +30,7 @@
 register(
     ModelSpec(
         model_type="minimax",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         moe_spec=MoESpec(
             block_names=("MiniMaxSparseMoeBlock",),
             expert_linear_names=("w1", "w2", "w3"),

modelopt/torch/models/mixtral/specs.py

@@ -24,7 +24,7 @@
 register(
     ModelSpec(
         model_type="mixtral",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
             block_names=("MixtralSparseMoeBlock",),

modelopt/torch/models/nemotron/specs.py

@@ -24,7 +24,7 @@
 register(
     ModelSpec(
         model_type="nemotron",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(
             weight_plus_one_norm_names=("LayerNorm1P", "NemotronLayerNorm1P"),
         ),

modelopt/torch/models/nemotron_h/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="nemotron_h",
-        min_transformers_version="5.3",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
             # NemotronHMOE experts (NemotronHMLP) use up_proj and down_proj only (no gate).

modelopt/torch/models/phimoe/specs.py

@@ -30,7 +30,7 @@
 register(
     ModelSpec(
         model_type="phimoe",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         moe_spec=MoESpec(
             block_names=("PhimoeSparseMoeBlock",),
             expert_linear_names=("w1", "w2", "w3"),

modelopt/torch/models/qwen2_moe/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="qwen2_moe",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
             block_names=("Qwen2MoeSparseMoeBlock",),

modelopt/torch/models/qwen3/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="qwen3",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(
             # AWQ pre_quant_scale fusion: fold o_proj into v_proj, down_proj into up_proj.
             pqs_fuse_rules=(

modelopt/torch/models/qwen3_5_moe/specs.py

@@ -28,7 +28,7 @@
 register(
     ModelSpec(
         model_type="qwen3_5_moe",
-        min_transformers_version="5.2",
+        min_transformers_version="5.5",
         moe_spec=MoESpec(
             block_names=("Qwen3_5MoeSparseMoeBlock",),
             expert_linear_names=("gate_proj", "down_proj", "up_proj"),

modelopt/torch/models/qwen3_moe/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="qwen3_moe",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         moe_spec=MoESpec(
             block_names=("Qwen3MoeSparseMoeBlock",),
             expert_linear_names=("gate_proj", "down_proj", "up_proj"),

modelopt/torch/models/qwen3_next/specs.py

@@ -22,7 +22,7 @@
 register(
     ModelSpec(
         model_type="qwen3_next",
-        min_transformers_version="4.57",
+        min_transformers_version="5.5",
         export_spec=ExportSpec(grouped_expert_export=True),
         moe_spec=MoESpec(
             block_names=("Qwen3NextSparseMoeBlock",),