Skip to content

Support standalone Qwen3.5 MTP companion GGUFs (-md ... --spec-type mtp) - #43

Merged
Thireus merged 1 commit into
th/fix-qwen35-mtp-companionfrom
th/fix-qwen35-mtp-companion-mainline
Aug 16, 2026
Merged

Support standalone Qwen3.5 MTP companion GGUFs (-md ... --spec-type mtp)#43
Thireus merged 1 commit into
th/fix-qwen35-mtp-companionfrom
th/fix-qwen35-mtp-companion-mainline

Conversation

@Thireus

@Thireus Thireus commented Aug 16, 2026

Copy link
Copy Markdown
Owner

A predictor-only MTP companion for qwen35/qwen35moe declares the full block count (n_main + nextn_predict_layers) but ships only the NextN block. Loading one with -md aborted at

check_tensor_dims: tensor 'blk.0.attn_norm.weight' not found

because create_qwen35{,moe}_tensors() creates every main block as hard-required. Step-3.5 and DeepSeek-V4 already have an "mtp_only" path for this; Qwen3.5 did not, and the surrounding MTP plumbing did not recognise a Qwen3.5 companion either.

  • llama-load-tensors: detect the companion (nextn > 0 and blk.0 attn_norm absent) and mark the absent main blocks TENSOR_SKIP|TENSOR_NOT_REQUIRED, mirroring create_step35_tensors(). Guard the "9B shares q_proj" fixup, which has no trunk to borrow from, and skip empty blocks in the split-mode-graph loop, where split_recurrent_tensors() would assert on the missing ssm_in/wqkv.

  • llama-model: add llama_model_is_qwen35_family() and let Qwen3.5 use the common MTP package contract, so a companion classifies as COMPANION and a plain target as TARGET_ONLY (which is what makes the target export the hidden states the companion consumes). Also accept enorm/hnorm as a predictor tail, since 9B/4B NextN blocks can be built without eh_proj.

  • llama-hparams: map qwen35moe n_layer 49 and 61 to 122B-A10B and 397B-A17B, so a companion of those sizes no longer prints "?B".

  • speculative: recognise the Qwen3.5 companion, validate its arch and head count, keep the embedded-tail + ordinary-draft combination working, and force pooling_type = NONE on the companion context.

  • common: reject a companion passed as -m with an actionable message instead of aborting inside the graph builder.

  • I have read the contributing guidelines

  • Self-reported review complexity:

    • Low
    • Medium
    • High

A predictor-only MTP companion for qwen35/qwen35moe declares the full
block count (n_main + nextn_predict_layers) but ships only the NextN
block. Loading one with -md aborted at

  check_tensor_dims: tensor 'blk.0.attn_norm.weight' not found

because create_qwen35{,moe}_tensors() creates every main block as
hard-required. Step-3.5 and DeepSeek-V4 already have an "mtp_only"
path for this; Qwen3.5 did not, and the surrounding MTP plumbing did
not recognise a Qwen3.5 companion either.

- llama-load-tensors: detect the companion (nextn > 0 and blk.0
  attn_norm absent) and mark the absent main blocks
  TENSOR_SKIP|TENSOR_NOT_REQUIRED, mirroring create_step35_tensors().
  Guard the "9B shares q_proj" fixup, which has no trunk to borrow
  from, and skip empty blocks in the split-mode-graph loop, where
  split_recurrent_tensors() would assert on the missing ssm_in/wqkv.
- llama-model: add llama_model_is_qwen35_family() and let Qwen3.5 use
  the common MTP package contract, so a companion classifies as
  COMPANION and a plain target as TARGET_ONLY (which is what makes the
  target export the hidden states the companion consumes). Also accept
  enorm/hnorm as a predictor tail, since 9B/4B NextN blocks can be
  built without eh_proj.
- llama-hparams: map qwen35moe n_layer 49 and 61 to 122B-A10B and
  397B-A17B, so a companion of those sizes no longer prints "?B".
- speculative: recognise the Qwen3.5 companion, validate its arch and
  head count, keep the embedded-tail + ordinary-draft combination
  working, and force pooling_type = NONE on the companion context.
- common: reject a companion passed as -m with an actionable message
  instead of aborting inside the graph builder.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Thireus
Thireus merged commit c6aa252 into th/fix-qwen35-mtp-companion Aug 16, 2026
1 check passed
Thireus added a commit that referenced this pull request Aug 16, 2026
…on-mainline"

This reverts commit c6aa252, reversing
changes made to 4e49d82.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant