From 7db0c32d198980fce92732a6130a231db79975a2 Mon Sep 17 00:00:00 2001 From: SID <99672439+SID-6921@users.noreply.github.com> Date: Mon, 28 Sep 2026 12:11:23 -0400 Subject: [PATCH 1/3] fix(onnx): keep the converted weight's type in step with its zero point qdq_to_dq always wrote the converted weight as INT8 and clipped it to [-128, 127]. The DequantizeLinear stays in the graph with its original zero point, so for an unsigned zero point the result is a graph whose weight and zero point disagree on type, and whose values above 127 have been clipped away. onnx.checker accepts that model, so nothing fails until load: Type Error: Type parameter (T1) of Optype (DequantizeLinear) bound to different types (tensor(int8) and tensor(uint8)) Derive the weight type and the clip range from the zero point instead. The module already models unsigned quantization elsewhere -- onnx_bit_dtype_unsigned_map and get_tensor_dtype() hand back UINT8 for a quantizer that has a zero point -- so insert_dq_nodes could produce weights that qdq_to_dq could not round-trip. A zero point of any other integer type now raises instead of being silently written as INT8. Signed weights are unchanged: the INT8 path converts bit-identically. Signed-off-by: SID <99672439+SID-6921@users.noreply.github.com> --- CHANGELOG.rst | 1 + modelopt/onnx/quantization/qdq_utils.py | 31 ++++++++-- .../unit/onnx/quantization/test_qdq_utils.py | 61 +++++++++++++++++++ 3 files changed, 88 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 4f621df5dea..f8c895deab2 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -94,6 +94,7 @@ Changelog - Fix ``megatron_generate`` dropping the VLM vision inputs (``pixel_values`` / ``image_grid_thw`` / ``image_sizes``) after the first generated token when KV-cache decoding is off, including the automatic fallback under sequence parallelism, which made generation silently ignore the image. No other ModelOpt feature is affected. - Fix two issues in the vLLM offline hidden-state dump (``examples/speculative_decoding/collect_hidden_states/compute_hidden_states_vllm.py``) that only surface on large runs. **Resume:** the filter that skips conversations whose ``.pt`` already exists now runs with ``load_from_cache_file=False``. It depends on on-disk state, which is not part of the fingerprint ``datasets`` computes from the function and the dataset, so with a persistent HF cache reused across a resumed or requeued run the cached "keep everything" result from an earlier run was replayed and the dump re-generated and overwrote conversations it had already finished (observed: tens of thousands of ``.pt`` rewritten while the output count stayed flat). **Staging:** generation is now chunked (``--save-chunk-size``, default 256), so each chunk is saved and its staged hidden states freed before the next chunk is generated. Previously the whole dataset was generated before anything was saved, which kept every conversation staged in the connector's ``shared_storage_path`` (``/dev/shm``, i.e. RAM, by default) at once and exhausted it partway through large dumps. Chunking also makes the dump incrementally durable, so an interrupted run keeps its finished conversations and resumes from them. The save path now also frees each conversation's staged hidden states in a ``finally``, so a conversation skipped mid-loop (e.g. a short ``loss_mask``) can no longer leak its staging file, and conversation ids are validated as plain filenames before being used to build output paths. - Hybrid (e.g. Nemotron-H) checkpoints saved by the ``examples/megatron_bridge`` scripts now record their layer spec in ``run_config.yaml`` in a form that reloads, so they can be converted to HuggingFace; a checkpoint saved by an earlier release still needs its ``model.hybrid_stack_spec`` block replaced by hand. +- Fix ``--dq_only`` ONNX export producing a model that fails to load when weights are quantized to an unsigned type: the weight was always written as INT8 while the DequantizeLinear kept its UINT8 zero point, and values above 127 were clipped away. 0.47.0 (2026-09-23) ^^^^^^^^^^^^^^^^^^^ diff --git a/modelopt/onnx/quantization/qdq_utils.py b/modelopt/onnx/quantization/qdq_utils.py index d0010571035..0c8caccf58d 100644 --- a/modelopt/onnx/quantization/qdq_utils.py +++ b/modelopt/onnx/quantization/qdq_utils.py @@ -60,6 +60,13 @@ onnx_bit_dtype_signed_map = {4: "INT4", 8: "INT8"} onnx_bit_dtype_unsigned_map = {4: "UINT4", 8: "UINT8"} +# Integer weight types qdq_to_dq can emit, keyed by the zero point's ONNX type. The +# DequantizeLinear left in the graph keeps its zero point, so the weight must match it. +qdq_weight_np_dtype_map = { + onnx.TensorProto.INT8: np.int8, + onnx.TensorProto.UINT8: np.uint8, +} + np_dtype_map = { "Float": np.float32, "Half": np.float16, @@ -631,7 +638,7 @@ def _convert_weight( ValueError: If scale shape doesn't match weight shape for the operation Note: - - INT8 weights are clipped to [-128, 127] + - Integer weights are clipped to the range of the zero point's type - FP8 weights use float8e4m3fn format """ # Per-op quantization axis mapping (must match ORT config) @@ -672,16 +679,29 @@ def _convert_weight( scale_array = scale_array.reshape(*reshape_dims) zp_array = zp_array.reshape(*reshape_dims) - # Convert to INT8/FP8 + # Convert to integer/FP8 if zp.data_type == onnx_dtype_map["Float8"]: scaled = np.asarray(weight_array / scale_array) + zp_array else: + weight_dtype = _qdq_weight_dtype(zp) + bounds = np.iinfo(weight_dtype) scaled = np.asarray((weight_array / scale_array).round()) - np.clip(scaled + zp_array, -128, 127, out=scaled) + np.clip(scaled + zp_array, bounds.min, bounds.max, out=scaled) return scaled +def _qdq_weight_dtype(zp: onnx.TensorProto) -> type[np.signedinteger | np.unsignedinteger]: + """Returns the numpy type a converted weight must use for this zero point.""" + weight_dtype = qdq_weight_np_dtype_map.get(zp.data_type) + if weight_dtype is None: + raise ValueError( + f"Unsupported zero point type for real weight quantization: " + f"{onnx.TensorProto.DataType.Name(zp.data_type)}" + ) + return weight_dtype + + def _cast_fp8(array: np.ndarray) -> np.ndarray: """Cast a numpy array to FLOAT8E4M3FN using PyTorch.""" array_f32_t = torch.from_numpy(array) @@ -764,8 +784,9 @@ def qdq_to_dq(onnx_model: onnx.ModelProto) -> onnx.ModelProto: new_weight = _create_fp8_tensor(scaled, weight_name) logger.debug(f"Converted {weight_name} to FP8") else: - new_weight = onnx.numpy_helper.from_array(scaled.astype("int8"), weight_name) - logger.debug(f"Converted {weight_name} to INT8") + weight_dtype = _qdq_weight_dtype(zp) + new_weight = onnx.numpy_helper.from_array(scaled.astype(weight_dtype), weight_name) + logger.debug(f"Converted {weight_name} to {weight_dtype.__name__.upper()}") weight.CopyFrom(new_weight) # Track QuantizeLinear node indices for cleanup diff --git a/tests/unit/onnx/quantization/test_qdq_utils.py b/tests/unit/onnx/quantization/test_qdq_utils.py index 95944e1f7ab..d84e98c9865 100644 --- a/tests/unit/onnx/quantization/test_qdq_utils.py +++ b/tests/unit/onnx/quantization/test_qdq_utils.py @@ -1162,6 +1162,67 @@ def test_constant_node_scale_path_still_patched(self): assert (scale_arr > 0).all() +def create_test_model_with_matmul_qdq(zero_point: np.ndarray, scale: np.ndarray, weight): + """Create weight -> Q -> DQ -> MatMul, per-channel on the weight's output axis.""" + graph = helper.make_graph( + nodes=[ + helper.make_node( + "QuantizeLinear", ["weight", "w_scale", "w_zp"], ["w_q"], name="w_q", axis=1 + ), + helper.make_node( + "DequantizeLinear", ["w_q", "w_scale", "w_zp"], ["w_dq"], name="w_dq", axis=1 + ), + helper.make_node("MatMul", ["input", "w_dq"], ["output"], name="matmul"), + ], + name="matmul_qdq", + inputs=[helper.make_tensor_value_info("input", TensorProto.FLOAT, [4, weight.shape[0]])], + outputs=[helper.make_tensor_value_info("output", TensorProto.FLOAT, [4, weight.shape[1]])], + initializer=[ + numpy_helper.from_array(weight, "weight"), + numpy_helper.from_array(scale, "w_scale"), + numpy_helper.from_array(zero_point, "w_zp"), + ], + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 21)]) + model.ir_version = 10 + onnx.checker.check_model(model) + return model + + +class TestQdqToDqWeightType: + """The converted weight must keep the type of the zero point the DQ node still uses.""" + + @pytest.mark.parametrize("unsigned", [False, True]) + def test_converted_weight_matches_zero_point_type(self, unsigned): + weight = np.random.RandomState(3).randn(16, 32).astype(np.float32) + if unsigned: + # Asymmetric UINT8: the zero point is non-zero and the weight spans 0..255. + low, high = weight.min(axis=0), weight.max(axis=0) + scale = ((high - low) / 255.0).astype(np.float32) + zero_point = np.round(-low / scale).astype(np.uint8) + expected_type = TensorProto.UINT8 + else: + scale = (np.abs(weight).max(axis=0) / 127.0).astype(np.float32) + zero_point = np.zeros(32, dtype=np.int8) + expected_type = TensorProto.INT8 + + model = create_test_model_with_matmul_qdq(zero_point, scale, weight) + inputs = {"input": np.random.RandomState(5).randn(4, 16).astype(np.float32)} + + def run(m): + return ort.InferenceSession( + m.SerializeToString(), providers=["CPUExecutionProvider"] + ).run(None, inputs)[0] + + expected = run(model) + converted = qdq_to_dq(onnx.ModelProto.FromString(model.SerializeToString())) + + stored = next(t for t in converted.graph.initializer if t.name == "weight") + assert stored.data_type == expected_type + # onnx.checker does not catch a weight/zero-point type mismatch, so load it. + assert np.array_equal(run(converted), expected) + + class TestQdqToDqValidation: """Regression tests for qdq_to_dq input validation.""" From b62fa06d4a524e7affa333a81e86ebc8b6fd9ed2 Mon Sep 17 00:00:00 2001 From: SID <99672439+SID-6921@users.noreply.github.com> Date: Mon, 28 Sep 2026 14:25:40 -0400 Subject: [PATCH 2/3] Trim the changelog entry to the user-visible effect AGENTS.md keeps root-cause analysis and implementation detail out of CHANGELOG.rst; those belong in the PR description. Signed-off-by: SID <99672439+SID-6921@users.noreply.github.com> --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index f8c895deab2..dd870aac4f5 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -94,7 +94,7 @@ Changelog - Fix ``megatron_generate`` dropping the VLM vision inputs (``pixel_values`` / ``image_grid_thw`` / ``image_sizes``) after the first generated token when KV-cache decoding is off, including the automatic fallback under sequence parallelism, which made generation silently ignore the image. No other ModelOpt feature is affected. - Fix two issues in the vLLM offline hidden-state dump (``examples/speculative_decoding/collect_hidden_states/compute_hidden_states_vllm.py``) that only surface on large runs. **Resume:** the filter that skips conversations whose ``.pt`` already exists now runs with ``load_from_cache_file=False``. It depends on on-disk state, which is not part of the fingerprint ``datasets`` computes from the function and the dataset, so with a persistent HF cache reused across a resumed or requeued run the cached "keep everything" result from an earlier run was replayed and the dump re-generated and overwrote conversations it had already finished (observed: tens of thousands of ``.pt`` rewritten while the output count stayed flat). **Staging:** generation is now chunked (``--save-chunk-size``, default 256), so each chunk is saved and its staged hidden states freed before the next chunk is generated. Previously the whole dataset was generated before anything was saved, which kept every conversation staged in the connector's ``shared_storage_path`` (``/dev/shm``, i.e. RAM, by default) at once and exhausted it partway through large dumps. Chunking also makes the dump incrementally durable, so an interrupted run keeps its finished conversations and resumes from them. The save path now also frees each conversation's staged hidden states in a ``finally``, so a conversation skipped mid-loop (e.g. a short ``loss_mask``) can no longer leak its staging file, and conversation ids are validated as plain filenames before being used to build output paths. - Hybrid (e.g. Nemotron-H) checkpoints saved by the ``examples/megatron_bridge`` scripts now record their layer spec in ``run_config.yaml`` in a form that reloads, so they can be converted to HuggingFace; a checkpoint saved by an earlier release still needs its ``model.hybrid_stack_spec`` block replaced by hand. -- Fix ``--dq_only`` ONNX export producing a model that fails to load when weights are quantized to an unsigned type: the weight was always written as INT8 while the DequantizeLinear kept its UINT8 zero point, and values above 127 were clipped away. +- Fix ``--dq_only`` ONNX export producing a model that ONNX Runtime cannot load when the weights are quantized to an unsigned type. Such exports now load and match the source model. 0.47.0 (2026-09-23) ^^^^^^^^^^^^^^^^^^^ From 260ff60cf936ab5921c5532e4ccc311d21fc4914 Mon Sep 17 00:00:00 2001 From: Siddhardha Nanda <99672439+SID-6921@users.noreply.github.com> Date: Mon, 26 Oct 2026 15:44:57 -0400 Subject: [PATCH 3/3] Move CHANGELOG entry to the 0.49.0 section Signed-off-by: Siddhardha Nanda <99672439+SID-6921@users.noreply.github.com> --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 13ab2ea1f00..618b0b77b1c 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -60,6 +60,7 @@ Changelog - Fix Megatron Top-K/Top-P distillation gradients being multiplied by the tensor-parallel size when tensor parallelism is enabled; reported loss values are unchanged. - Fix the Puzzletron memory objective pricing the KV cache with ``hidden_size // num_attention_heads`` instead of the model's ``head_dim``, which misestimated it for models such as Mistral-Small, GPT-OSS and Gemma-3 and for every candidate with a pruned hidden size. Recompute subblock statistics produced with earlier versions. - Fix ONNX INT4 quantization ending the Python process when the INT4 packing extension fails to build, for example without a C++ compiler. It now falls back to the Python implementation. +- Fix ``--dq_only`` ONNX export producing a model that ONNX Runtime cannot load when the weights are quantized to an unsigned type. Such exports now load and match the source model. 0.48.0 (2026-10-xx) ^^^^^^^^^^^^^^^^^^^ @@ -254,7 +255,6 @@ Changelog - Fix YAML config I/O decoding with the locale codepage instead of UTF-8, which made a config containing any non-ASCII byte fail to load on a machine whose locale is not UTF-8 (notably Windows, where the default is cp1252). ``modelopt/recipe/loader.py``, the two ONNX autotune state files, the two transformers config readers, the distill config and the puzzletron profile now pass ``encoding="utf-8"`` explicitly. Only the YAML config paths are covered: these are the files most likely to carry non-ASCII text in comments, model names or paths, and the only ones read inside a user's process. - Hybrid (e.g. Nemotron-H) checkpoints saved by the ``examples/megatron_bridge`` scripts now record their layer spec in ``run_config.yaml`` in a form that reloads, so they can be converted to HuggingFace; a checkpoint saved by an earlier release still needs its ``model.hybrid_stack_spec`` block replaced by hand. - Fix ONNX export of quantized models failing with ``Unsupported op_type for real weight quantization: Transpose`` when a weight reaches its consumer transposed, as in MaxViT's attention blocks. The same export now also quantizes each weight along the axis its Q/DQ pair declares, correcting silently wrong ``dq_only`` output when that axis differed from the consumer's default. -- Fix ``--dq_only`` ONNX export producing a model that ONNX Runtime cannot load when the weights are quantized to an unsigned type. Such exports now load and match the source model. - Fix the ``evaluation`` agent skill's LiveCodeBench recipe scoring correct solutions that use binary standard I/O (``sys.stdin.buffer`` / ``sys.stdout.buffer``) as failures: the recipe now pins a fixed LiveCodeBench judge in the NeMo-Skills container. LiveCodeBench scores change as a result, so re-run earlier baselines before comparing. - Fix GPTQ with Megatron data parallelism: each data-parallel replica of a module now builds its Hessian from the whole calibration set instead of only its own shard, so all replicas reach the same quantized weights. - Fix ``examples/llm_sparsity/weight_sparsity/finetune.py`` saving a sharded state dict under FSDP2, the default FSDP version in Transformers 5, which made ``export_hf_ckpt.py`` fail to restore it. Re-run finetuning to produce a loadable ``finetuned_modelopt_state.pth``.