Skip to content

Pass the target as the MTP draft's ctx_other so Gemma 4 sidecars load (#91) - #94

Closed
nyo16 wants to merge 1 commit into
masterfrom
fix/gemma4-mtp-ctx-other
Closed

nyo16 wants to merge 1 commit into
masterfrom
fix/gemma4-mtp-ctx-other

Conversation

@nyo16

@nyo16 nyo16 commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Fixes #91.

Root cause

llama.cpp's gemma4-assistant head builds its draft context over the target's: the draft's llama_kv_cache aliases the target's cell vector (other->v_cells_impl) and the graph reads the target's
tok_embd. Its constructor throws Gemma4Assistant requires ctx_other to be set when llama_context_params.ctx_other is null (llama-context.cpp, LLM_ARCH_GEMMA4_ASSISTANT).

context_create built every context from llama_context_default_params() and never exposed the field. The Qwen 3.8 sidecar path from v0.8.44 worked because qwen35 never reads the pointer; a Gemma 4 pair
could not get past Context.create.

Reproduced pre-fix with Unsloth gemma-4-E4B-it-Q4_K_M.gguf + MTP/mtp-gemma-4-E4B-it-Q8_0.gguf on Metal: validate_head/2 passes (head n_layer_nextn == 4, n_embd_out 2560 on both), then
llama_init_from_model fails with exactly the error in the issue.

Fix

Mirrors upstream's common_speculative_init_result, which sets cparams.ctx_other = ctx_tgt for every draft context.

  • c_src/llama_cpp_ex/llama_nif.cppcontext_create takes std::optional<fine::ResourcePtr<LlamaContext>> ctx_other and forwards it as params.ctx_other.
  • c_src/llama_cpp_ex/llama_nif.hLlamaContext holds the peer's ResourcePtr, so the target cannot be freed under a draft whose KV cells alias it.
  • LlamaCppEx.Context.create/2 — new :ctx_other option (%Context{} or nil), added to the structural keys and documented.
  • LlamaCppEx.MTP.init/2 — passes the target as the draft's :ctx_other unconditionally. llama-context.cpp only reads the field for GEMMA4_ASSISTANT / EAGLE3 / DFLASH, so the in-file and Qwen
    sidecar paths are unchanged.
  • test/mtp_model_test.exstiming_us reports a ckpt bucket asserted ckpt > 0, which pins Qwen 3.8's hybrid-SSM snapshotting rather than the binding. It now branches on
    TestModels.seq_rm_kind(:mtp): :full targets must be billed for snapshots, everything else must not — the same condition as the NIF's needs_ckpt gate.

Creating a Gemma 4 draft context without a target still fails. That is llama.cpp's contract; Context.create/2 documents it.

Verification (Apple M4 Max, Metal source build)

Run Result
Gemma 4 E4B pair, pre-fix {:error, "failed to create context"} / Gemma4Assistant requires ctx_other to be set
Gemma 4 E4B pair, post-fix MTP.init ok; greedy " Paris.", 53% acceptance at n_draft: 3
--include mtp_sidecar on the Gemma pair 6/6, incl. greedy equivalence with plain target decode
--include mtp (Qwen3.5-0.8B-MTP-Q8_0, in-file head) 19 passed
--include smoke --include embeddings 548 passed, 29 excluded
mix test (no model) 428 passed, 149 excluded
mix format --check-formatted, mix credo --strict clean

Not run: the Qwen 3.8 27B sidecar pair (no local copy). Unaffected by construction — qwen35 ignores ctx_other — and the identical code path passed on the Qwen 0.8B in-file head.

Docs

  • CHANGELOG.md## Unreleased → Fixed entry
  • LlamaCppEx.MTP moduledoc — the two sidecar head architectures and what each needs from the target
  • README.md — Gemma 4 E4B pair added to the MTP model list
  • docs/release-guide.md — Gemma mtp_sidecar run added to the test matrix, since a Qwen pair alone cannot exercise the ctx_other wiring

Fixes #91. llama.cpp's gemma4-assistant head builds its draft context over
the target's — the KV cache aliases the target's cells and the graph reads
the target's token embeddings — and its constructor throws "Gemma4Assistant
requires ctx_other to be set" when llama_context_params.ctx_other is null.
The NIF never exposed that field, so MTP.init/2 with a Gemma 4 pair failed
at Context.create while the Qwen 3.8 sidecar path (whose head ignores the
pointer) worked.

Context.create/2 gains :ctx_other (a %Context{}), forwarded as
params.ctx_other; the LlamaContext resource holds the peer's ResourcePtr so
the target outlives a draft that aliases it. MTP.init/2 passes the target as
the draft's :ctx_other unconditionally, as upstream's
common_speculative_init_result does — llama-context.cpp only reads it for
GEMMA4_ASSISTANT / EAGLE3 / DFLASH, so the in-file and Qwen paths are
unchanged.

The :mtp_sidecar test "timing_us reports a ckpt bucket" pinned Qwen 3.8
(hybrid SSM → snapshots) rather than the binding; it now keys off
TestModels.seq_rm_kind(:mtp): :full targets must bill ckpt, others must not.

Verified on M4 Max / Metal:
- pre-fix, unsloth gemma-4-E4B-it-Q4_K_M + MTP/mtp-gemma-4-E4B-it-Q8_0:
  validation passes (nextn=4, n_embd_out 2560 both), then
  llama_init_from_model fails with the ctx_other error
- post-fix, same pair: MTP.init ok, greedy " Paris." with 53% acceptance at
  n_draft 3; --include mtp_sidecar 6/6 (incl. greedy equivalence with plain
  target decode)
- --include mtp (Qwen3.5-0.8B-MTP-Q8_0, in-file head): 19 passed
- --include smoke --include embeddings: 548 passed, 29 excluded
- default suite 428 passed; mix format and credo --strict clean
@nyo16 nyo16 changed the title Pass the target as the MTP draft's ctx_other so Gemma 4 sidecars load Pass the target as the MTP draft's ctx_other so Gemma 4 sidecars load (#91) Sep 14, 2026
@nyo16 nyo16 closed this Sep 14, 2026
@nyo16
nyo16 deleted the fix/gemma4-mtp-ctx-other branch September 14, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemma4 E4B MTP sidecar fails: Context.create leaves ctx_other nullptr

1 participant