fix(ops): dequantize GGUF GGMLTensor weights in cast_bias_weight - #15049
fix(ops): dequantize GGUF GGMLTensor weights in cast_bias_weight#15049Log-Dog012 wants to merge 1 commit into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📥 CommitsReviewing files that changed from the base of the PR and between 93c7435701efa5ae1852b2bb2e9712db1cf3e7d0 and 8e7e69d. 📒 Files selected for processing (1)
📜 Recent review details🧰 Additional context used📓 Path-based instructions (5)**/*📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.py📄 CodeRabbit inference engine (AGENTS.md)
Files:
comfy/**/*.py📄 CodeRabbit inference engine (AGENTS.md)
Files:
**⚙️ CodeRabbit configuration file
Files:
comfy/**⚙️ CodeRabbit configuration file
Files:
🧠 Learnings (2)📚 Learning: 2026-02-21T14:01:41.482ZApplied to files:
📚 Learning: 2026-05-13T12:31:45.069ZApplied to files:
🔇 Additional comments (1)
📝 WalkthroughWalkthroughUpdated 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@comfy/ops.py`:
- Around line 369-370: Update the weight conversion condition in the surrounding
operation to avoid using a generic hasattr(weight, "dequantize") check. Restrict
dequantize() calls to weights identified by the supported dequantization marker
or protocol, such as GGMLTensor, while preserving the existing QuantizedTensor
exclusion and leaving ordinary torch.Tensor weights unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f5fa2ec4-09ac-4601-ad71-8363b542c46f
📥 Commits
Reviewing files that changed from the base of the PR and between a449f5f and 22495f0207510ac2fd504a1b0b78bacd87ee6c05.
📒 Files selected for processing (1)
comfy/ops.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (5)
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Keep changes small, direct, and limited to the narrowest code path and fewest files necessary; prefer practical fixes over broad architectural work.
Prefer fewer dependencies and do not add a ComfyUI dependency unless absolutely necessary.
Remove obsolete code, dead branches, unused options, debug prints, and unnecessary compatibility paths.
Preserve existing APIs, node names, model-loading behavior, file layout, and workflow compatibility unless replacement is explicit.
Keep architectural layers focused; do not leak UI, API, workflow, queue, persistence, telemetry, model-loading, node, or execution concerns into unrelated layers.
Shared core modules should depend only on lower-level primitives and their own domain concepts; higher-level concepts belong at callers, adapters, services, or boundaries.
Pass only the narrowest data needed across boundaries and keep identity mapping, persistence, history, telemetry, response shaping, and UI state in their owning layers.
Before touching many files, identify the smallest owner layer; use caller-side mappings, adapters, events, or narrow interfaces instead of exposing private concepts across layers.
Core ComfyUI code must not make unsolicited internet requests or add uploads, telemetry, analytics, tracking, reporting, update checks, remote configuration, licensing checks, or similar outbound paths.
Model downloading is allowed only when explicitly authorized by the user, limited to the requested artifact, and free of telemetry, tracking, unrelated metadata, or background activity.
Warning and info messages should be short and actionable; documentation and README changes should be concise, factual, and tied to changed behavior.
Use short direct commit subjects such asFix ...,Add ...,Support ...,Remove ..., orUpdate ...; keep PR descriptions short and state the problem, behavior change, and tests.
Prefer one coherent behavioral change per commit and prioritize crashes, incorrect dtype/device behavior, m...
Files:
comfy/ops.py
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Keep state and capability flags on the object that owns the behavior; use explicit parent-owned fields rather than probing children withgetattrfor parent control flow.
Keep public methods aligned with caller contracts; preserve arguments, parameter order, return types, side effects, and error behavior unless all affected interfaces are intentionally updated.
Do not add unused compatibility parameters, flags, attributes, constructor options, or model-specific options to shared helpers.
Normalize third-party and upstream return conventions at integration boundaries so core code receives the expected type and shape.
Avoid caller-side unwrapping such asout = out[0]unless the called interface documents that return structure.
Do not addtorch.no_grad,torch.inference_mode, or inference-mode wrappers; only disable a globally enabled inference mode when a training path requires gradients.
Do not add freeze, unfreeze, or trainability toggles to ComfyUI model classes.
Remove training-only behavior such as dropout from inference models while preserving checkpoint and state-dict compatibility, usingnn.Identitywhen necessary to retain slots.
Keep imports at module scope; use inline imports only for established optional-backend probes or import-cycle avoidance.
Usetry/exceptonly for optional dependency, platform, or backend detection with a useful fallback, and prefer specific exception types.
Do not add code for unsupported pinned library versions or obsolete PyTorch workarounds; unsupported formats, quantization metadata, and bad states should fail clearly.
Match local file style and keep comments sparse, useful, and non-obvious; remove comments that merely restate code.
Files:
comfy/ops.py
comfy/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
comfy/**/*.py: Treat dtype, device placement, VRAM use, offloading, and memory behavior as correctness concerns; use existing ComfyUI quantization, memory, offload, and optimized-operation helpers.
Prefer shared optimized kernels, backend dispatchers, and documented interfaces over duplicate handwritten operations; treat selected backend callables as opaque.
Do not duplicate operations with custom float32-upcasting inference kernels; use generic ComfyUI operations or native PyTorch operations.
If a model constructor has anoperationsparameter, assume it is neverNone; do not add fallback torch operations.
Avoid unnecessary parameters in model, block, and operation constructors or forwards; reuse existing model classes, blocks, operations, and helpers.
Model detectors must inspect only the first dimension of linear weights, guard every dereferenced state-dict key, and order specific signatures before broad fallbacks.
Avoideinopsin core inference code; use native tensor operations such asreshape,view,permute,transpose,flatten, and related methods.
Keep metadata, counters, shape calculations, indices, split boundaries, and control-flow values as Python values rather than tensors.
Avoid unnecessary casts and transfers; preserve intended compute and storage dtypes, tensor shapes, and backend result contracts.
Keep model-native latent layout handling inside the model or latent-format owner rather than reshaping it in nodes or caller-side adapters.
DiT models must pad every patchified target or reference input withcomfy.ldm.common_dit.pad_to_patch_sizeand crop only the target output back to its original dimensions.
Do not add defensive shape, configuration, or dtype casts that merely obscure a clear tensor-operation failure; validate only at meaningful boundaries.
Raw parameters not owned by an operation should be cast at use withcomfy.ops.cast_to_inputorcomfy.model_management.cast_to; model constructors should not contain dtype workaro...
Files:
comfy/ops.py
**
⚙️ CodeRabbit configuration file
**: IMPORTANT: Only comment on issues directly introduced by this PR's code changes.
Treat AGENTS.md as mandatory repository policy, not optional style guidance.
Flag PR changes that violate AGENTS.md even when the code is otherwise functional.
In particular, enforce architecture boundaries, dtype/device/memory rules,
interface contracts, import style, no unnecessary try/except blocks, no inline
imports, no outbound internet paths in core ComfyUI, and narrow scoped fixes.
Prefer direct findings over suggestions when a rule is violated. Only ignore
AGENTS.md when it clearly conflicts with a newer explicit maintainer instruction
in the PR.
Do NOT flag pre-existing issues in code that was merely moved, re-indented,
de-indented, or reformatted without logic changes. If code appears in the diff
only due to whitespace or structural reformatting (e.g., removing awith:block),
treat it as unchanged. Contributors should not feel obligated to address
pre-existing issues outside the scope of their contribution.
Files:
comfy/ops.py
comfy/**
⚙️ CodeRabbit configuration file
comfy/**: Core ML/diffusion engine. Focus on:
- Backward compatibility (breaking changes affect all custom nodes)
- Memory management and GPU resource handling
- Performance implications in hot paths
- Thread safety for concurrent execution
Files:
comfy/ops.py
🧠 Learnings (2)
📚 Learning: 2026-02-21T14:01:41.482Z
Learnt from: pythongosssss
Repo: Comfy-Org/ComfyUI PR: 12555
File: comfy_extras/nodes_glsl.py:719-724
Timestamp: 2026-02-21T14:01:41.482Z
Learning: In PyOpenGL, bare Python scalars can be accepted for 1-element array parameters by NumberHandler. This means you can pass an int/float directly to OpenGL texture deletion (e.g., glDeleteTextures(tex)) without wrapping in a list. Verify function-specific expectations and ensure types match what the OpenGL call expects; use explicit lists only when the API requires an array.
Applied to files:
comfy/ops.py
📚 Learning: 2026-05-13T12:31:45.069Z
Learnt from: rattus128
Repo: Comfy-Org/ComfyUI PR: 13802
File: comfy/pinned_memory.py:19-30
Timestamp: 2026-05-13T12:31:45.069Z
Learning: When reviewing code that uses comfy/pinned_memory.py’s `HostBuffer.extend(size=..., reallocate=...)`: by default (`reallocate` is not True / False), `extend(size=...)` is a *relative increment* that grows the buffer by `size` bytes—so slicing like `[offset:offset+size]` after `hostbuf.extend(size=size)` is correct and the argument should not be rewritten to `offset + size`. Only in the single-segment reallocation mode (`reallocate=True`, e.g., as used by `resize_pin_buffer()` in `comfy/model_management.py`) should `size` be treated as an *absolute target* and the call/arguments should be checked accordingly.
Applied to files:
comfy/ops.py
There was a problem hiding this comment.
Pull request overview
This PR fixes a crash when using GGUF text encoders by ensuring comfy.ops.cast_bias_weight dequantizes GGUF’s packed weight tensors before they reach F.linear, preventing invalid matmul shapes during logits/generation.
Changes:
- Add a duck-typed
dequantize()handling path incast_bias_weightso GGUFGGMLTensorweights are dequantized even though they aren’tQuantizedTensor. - Document why the additional dequantization step is needed (GGUF packed storage vs expected matmul shape).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # GGUF quantized weights are stored as a custom GGMLTensor subclass rather | ||
| # than ComfyUI's QuantizedTensor, so the isinstance check further down misses | ||
| # them and the block-packed storage reaches F.linear with the wrong shape. | ||
| # Dequantize any weight that exposes a dequantize() method (duck-typed) so the | ||
| # real (un-packed) shape is what downstream matmul sees. | ||
| if not isinstance(weight, QuantizedTensor) and hasattr(weight, "dequantize"): | ||
| weight = weight.dequantize() |
22495f0 to
e163ec2
Compare
|
Thanks for the reviews — both points are now addressed in the latest push: Copilot (race condition before sync_stream): moved the duck-typed CodeRabbit (generic if not isinstance(weight, QuantizedTensor) and hasattr(weight, "dequantize") and not weight.is_floating_point():
weight = weight.dequantize()The intent is now explicit: only non-floating-point weights that expose dequantization are touched. (For the record, on current PyTorch a plain Verification: on the GGUF Qwen2.5-VL text encoder, Paired GGUF-side PR: city96/ComfyUI-GGUF#468. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@comfy/ops.py`:
- Around line 378-379: Update the dequantization guard around QuantizedTensor so
it only accepts explicitly supported quantized-weight types, using the existing
GGMLTensor marker/protocol or another defined predicate instead of relying on
hasattr(weight, "dequantize"). Preserve the current handling for QuantizedTensor
and avoid invoking dequantize on unrelated non-floating tensors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b0c62ee8-085e-443e-a1c0-0fa3afe0e81e
📥 Commits
Reviewing files that changed from the base of the PR and between 22495f0207510ac2fd504a1b0b78bacd87ee6c05 and e163ec2dd695ac8a07177b310ed898cc7cc2fa5b.
📒 Files selected for processing (1)
comfy/ops.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (5)
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Keep changes small, direct, and limited to the narrowest code path and fewest files necessary; prefer practical fixes over broad architectural work.
Prefer fewer dependencies and do not add a ComfyUI dependency unless absolutely necessary.
Remove obsolete code, dead branches, unused options, debug prints, and unnecessary compatibility paths.
Preserve existing APIs, node names, model-loading behavior, file layout, and workflow compatibility unless replacement is explicit.
Keep architectural layers focused; do not leak UI, API, workflow, queue, persistence, telemetry, model-loading, node, or execution concerns into unrelated layers.
Shared core modules should depend only on lower-level primitives and their own domain concepts; higher-level concepts belong at callers, adapters, services, or boundaries.
Pass only the narrowest data needed across boundaries and keep identity mapping, persistence, history, telemetry, response shaping, and UI state in their owning layers.
Before touching many files, identify the smallest owner layer; use caller-side mappings, adapters, events, or narrow interfaces instead of exposing private concepts across layers.
Core ComfyUI code must not make unsolicited internet requests or add uploads, telemetry, analytics, tracking, reporting, update checks, remote configuration, licensing checks, or similar outbound paths.
Model downloading is allowed only when explicitly authorized by the user, limited to the requested artifact, and free of telemetry, tracking, unrelated metadata, or background activity.
Warning and info messages should be short and actionable; documentation and README changes should be concise, factual, and tied to changed behavior.
Use short direct commit subjects such asFix ...,Add ...,Support ...,Remove ..., orUpdate ...; keep PR descriptions short and state the problem, behavior change, and tests.
Prefer one coherent behavioral change per commit and prioritize crashes, incorrect dtype/device behavior, m...
Files:
comfy/ops.py
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Keep state and capability flags on the object that owns the behavior; use explicit parent-owned fields rather than probing children withgetattrfor parent control flow.
Keep public methods aligned with caller contracts; preserve arguments, parameter order, return types, side effects, and error behavior unless all affected interfaces are intentionally updated.
Do not add unused compatibility parameters, flags, attributes, constructor options, or model-specific options to shared helpers.
Normalize third-party and upstream return conventions at integration boundaries so core code receives the expected type and shape.
Avoid caller-side unwrapping such asout = out[0]unless the called interface documents that return structure.
Do not addtorch.no_grad,torch.inference_mode, or inference-mode wrappers; only disable a globally enabled inference mode when a training path requires gradients.
Do not add freeze, unfreeze, or trainability toggles to ComfyUI model classes.
Remove training-only behavior such as dropout from inference models while preserving checkpoint and state-dict compatibility, usingnn.Identitywhen necessary to retain slots.
Keep imports at module scope; use inline imports only for established optional-backend probes or import-cycle avoidance.
Usetry/exceptonly for optional dependency, platform, or backend detection with a useful fallback, and prefer specific exception types.
Do not add code for unsupported pinned library versions or obsolete PyTorch workarounds; unsupported formats, quantization metadata, and bad states should fail clearly.
Match local file style and keep comments sparse, useful, and non-obvious; remove comments that merely restate code.
Files:
comfy/ops.py
comfy/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
comfy/**/*.py: Treat dtype, device placement, VRAM use, offloading, and memory behavior as correctness concerns; use existing ComfyUI quantization, memory, offload, and optimized-operation helpers.
Prefer shared optimized kernels, backend dispatchers, and documented interfaces over duplicate handwritten operations; treat selected backend callables as opaque.
Do not duplicate operations with custom float32-upcasting inference kernels; use generic ComfyUI operations or native PyTorch operations.
If a model constructor has anoperationsparameter, assume it is neverNone; do not add fallback torch operations.
Avoid unnecessary parameters in model, block, and operation constructors or forwards; reuse existing model classes, blocks, operations, and helpers.
Model detectors must inspect only the first dimension of linear weights, guard every dereferenced state-dict key, and order specific signatures before broad fallbacks.
Avoideinopsin core inference code; use native tensor operations such asreshape,view,permute,transpose,flatten, and related methods.
Keep metadata, counters, shape calculations, indices, split boundaries, and control-flow values as Python values rather than tensors.
Avoid unnecessary casts and transfers; preserve intended compute and storage dtypes, tensor shapes, and backend result contracts.
Keep model-native latent layout handling inside the model or latent-format owner rather than reshaping it in nodes or caller-side adapters.
DiT models must pad every patchified target or reference input withcomfy.ldm.common_dit.pad_to_patch_sizeand crop only the target output back to its original dimensions.
Do not add defensive shape, configuration, or dtype casts that merely obscure a clear tensor-operation failure; validate only at meaningful boundaries.
Raw parameters not owned by an operation should be cast at use withcomfy.ops.cast_to_inputorcomfy.model_management.cast_to; model constructors should not contain dtype workaro...
Files:
comfy/ops.py
**
⚙️ CodeRabbit configuration file
**: IMPORTANT: Only comment on issues directly introduced by this PR's code changes.
Treat AGENTS.md as mandatory repository policy, not optional style guidance.
Flag PR changes that violate AGENTS.md even when the code is otherwise functional.
In particular, enforce architecture boundaries, dtype/device/memory rules,
interface contracts, import style, no unnecessary try/except blocks, no inline
imports, no outbound internet paths in core ComfyUI, and narrow scoped fixes.
Prefer direct findings over suggestions when a rule is violated. Only ignore
AGENTS.md when it clearly conflicts with a newer explicit maintainer instruction
in the PR.
Do NOT flag pre-existing issues in code that was merely moved, re-indented,
de-indented, or reformatted without logic changes. If code appears in the diff
only due to whitespace or structural reformatting (e.g., removing awith:block),
treat it as unchanged. Contributors should not feel obligated to address
pre-existing issues outside the scope of their contribution.
Files:
comfy/ops.py
comfy/**
⚙️ CodeRabbit configuration file
comfy/**: Core ML/diffusion engine. Focus on:
- Backward compatibility (breaking changes affect all custom nodes)
- Memory management and GPU resource handling
- Performance implications in hot paths
- Thread safety for concurrent execution
Files:
comfy/ops.py
🧠 Learnings (2)
📚 Learning: 2026-02-21T14:01:41.482Z
Learnt from: pythongosssss
Repo: Comfy-Org/ComfyUI PR: 12555
File: comfy_extras/nodes_glsl.py:719-724
Timestamp: 2026-02-21T14:01:41.482Z
Learning: In PyOpenGL, bare Python scalars can be accepted for 1-element array parameters by NumberHandler. This means you can pass an int/float directly to OpenGL texture deletion (e.g., glDeleteTextures(tex)) without wrapping in a list. Verify function-specific expectations and ensure types match what the OpenGL call expects; use explicit lists only when the API requires an array.
Applied to files:
comfy/ops.py
📚 Learning: 2026-05-13T12:31:45.069Z
Learnt from: rattus128
Repo: Comfy-Org/ComfyUI PR: 13802
File: comfy/pinned_memory.py:19-30
Timestamp: 2026-05-13T12:31:45.069Z
Learning: When reviewing code that uses comfy/pinned_memory.py’s `HostBuffer.extend(size=..., reallocate=...)`: by default (`reallocate` is not True / False), `extend(size=...)` is a *relative increment* that grows the buffer by `size` bytes—so slicing like `[offset:offset+size]` after `hostbuf.extend(size=size)` is correct and the argument should not be rewritten to `offset + size`. Only in the single-segment reallocation mode (`reallocate=True`, e.g., as used by `resize_pin_buffer()` in `comfy/model_management.py`) should `size` be treated as an *absolute target* and the call/arguments should be checked accordingly.
Applied to files:
comfy/ops.py
e163ec2 to
93c7435
Compare
|
Re-addressed per the CodeRabbit review (Functional Correctness / Major / Quick win): The previous guard used New version keys off GGUF's own protocol marker instead: if not isinstance(weight, QuantizedTensor) and hasattr(weight, "tensor_type") and hasattr(weight, "dequantize"):
weight = weight.dequantize()
Verification: on the GGUF Qwen2.5-VL text encoder, Paired GGUF-side PR: city96/ComfyUI-GGUF#468. |
comfy.ops.cast_bias_weight only dequantized QuantizedTensor weights. GGUF stores quantized weights as a custom GGMLTensor subclass, so it slipped through un-dequantized and its block-packed storage reached F.linear with the wrong shape, crashing generate() on GGUF text encoders (e.g. Qwen2.5-VL). Dequantize any weight exposing a duck-typed dequantize() method so the real (un-packed) shape is what downstream matmul sees. This lets GGUF weights use the same generic cast path as normal and QuantizedTensor weights without core having to import GGUF types.
93c7435 to
8e7e69d
Compare
|
Addressed the follow-up review (Functional Correctness / Major): the discriminator is now GGUF's own protocol marker, not the presence of a dequantize() method. if not isinstance(weight, QuantizedTensor) and getattr(weight, "tensor_type", None) is not None:
weight = weight.dequantize()Why this satisfies the concern:
Verification: on the GGUF Qwen2.5-VL text encoder, Paired GGUF-side PR: city96/ComfyUI-GGUF#468. |
|
It doesn't really make sense to add a workaround for a custom node to ComfyUI core code. It's also brittle to look for a The right fix is probably doing away with the custom import comfy_kitchen.tensor as cktensor
import gguf
import torch
from . import dequant
ckbase = cktensor.base
gqt = gguf.GGMLQuantizationType
class GGUFTensorConfig(NamedTuple):
qtype: gqt
block_size: int
type_size: int
handler: Callable[[torch.Tensor, int, int, torch.dtype | None], torch.Tensor]
dequant_dtype: torch.dtype | None
class GGUFLayout(ckbase.QuantizedLayout):
@dc.dataclass(frozen=True)
class Params(ckbase.BaseLayoutParams):
scale: None
gguf_config: GGUFTensorConfig
def _tensor_field(self) -> list[str]:
return []
@classmethod
def quantize(
cls,
tensor: torch.Tensor,
**_kwargs: Any,
) -> tuple[torch.Tensor, Params]:
raise NotImplementedError("GGUF doesn't support online quantization")
@classmethod
def dequantize(cls, qdata: torch.Tensor, params: Params) -> torch.Tensor:
gcfg = params.gguf_config
bs, ts = gcfg.block_size, gcfg.type_size
blocks = qdata.view(torch.uint8)
blocks = blocks.reshape(blocks.numel() // ts, ts)
return (
gcfg.handler(blocks, bs, ts, gcfg.dequant_dtype or params.orig_dtype)
.to(dtype=params.orig_dtype)
.contiguous()
)
ckbase.register_layout_class("GGUFLayout", GGUFLayout)This is just a hint for the general direction, not an actual working implementation, and you would need to change the GGUF loader, instantiate params, etc. Vaguely something like: # ...
sd, extra = gguf_sd_loader(unet_path)
for k in sd.keys():
v = sd[k]
if not isinstance(v, GGMLTensor):
continue
qtype = v.tensor_type
qdata = v.data
orig_shape = v.tensor_shape
if qtype in {ck.gqt.F32, ck.gqt.F16}:
sd[k] = qdata.reshape(orig_shape)
continue
block_size, type_size = ck.gguf.GGML_QUANT_SIZES[qtype]
qhandler = dq_handlers[qtype]
params = GGUFLayout.Params(
scale=None,
orig_dtype=dtype,
orig_shape=orig_shape,
gguf_config=ck.GGUFTensorConfig(
qtype=qtype,
block_size=block_size,
type_size=type_size,
handler=qhandler,
dequant_dtype=dqdtype,
),
)
sd[k] = QuantizedTensor(qdata, "GGUFLayout", params)The path of least resistance is to leave the loading logic as it is and convert the edit: All this is obviously a significant amount of work and I don't think the GGUF node maintainer has time to deal with pulls (or at least my own haven't gotten acknowledge). A simpler solution where you intercept direct attribute access for whatever module where the problem is happening (you might need to make a copy of the module type and change the loader slightly to use it) and just dequantize instead of returning the raw quantized tensor might be possible. It would be a lot easier than completely changing how GGUF stores tensors. |
|
Closing per maintainer feedback on the review. The direction of adding a GGUF-specific workaround (probing for a The correct fix belongs on the ComfyUI-GGUF side — making GGUF quantized weights integrate with ComfyUI's existing Thanks for the guidance. |
What
comfy.ops.cast_bias_weightonly dequantizedQuantizedTensorweights(comfy/ops.py). GGUF stores quantized weights as a custom
GGMLTensorsubclass, so it slipped through un-dequantized and its block-packed storage
reached
F.linearwith the wrong shape, crashinggenerate()on GGUF textencoders such as Qwen2.5-VL:
GGUF's own
GGMLLayer.cast_bias_weightdoes dequantize correctly, butBaseGenerate.logits(and any core code calling the genericcomfy.ops.cast_bias_weight) bypasses that module method.Fix
Dequantize any weight that exposes a duck-typed
dequantize()method, so thereal (un-packed) shape is what downstream matmul sees. This is GGUF-agnostic:
core does not import GGUF types, and only weights that opt in with a
dequantize()method are affected. Normal tensors andQuantizedTensorkeeptheir existing behavior unchanged.
Requires city96/ComfyUI-GGUF#468 which adds the
GGMLTensor.dequantize()method.
Verification
Loaded a real Qwen2.5-VL GGUF clip and called the unmodified
transformer.logitsdirectly: before the fix it crashed with the shape error above; after, it returns
(1, 1, 152064)as expected.