Conversation
Fixes #91. llama.cpp's gemma4-assistant head builds its draft context over the target's — the KV cache aliases the target's cells and the graph reads the target's token embeddings — and its constructor throws "Gemma4Assistant requires ctx_other to be set" when llama_context_params.ctx_other is null. The NIF never exposed that field, so MTP.init/2 with a Gemma 4 pair failed at Context.create while the Qwen 3.8 sidecar path (whose head ignores the pointer) worked. Context.create/2 gains :ctx_other (a %Context{}), forwarded as params.ctx_other; the LlamaContext resource holds the peer's ResourcePtr so the target outlives a draft that aliases it. MTP.init/2 passes the target as the draft's :ctx_other unconditionally, as upstream's common_speculative_init_result does — llama-context.cpp only reads it for GEMMA4_ASSISTANT / EAGLE3 / DFLASH, so the in-file and Qwen paths are unchanged. The :mtp_sidecar test "timing_us reports a ckpt bucket" pinned Qwen 3.8 (hybrid SSM → snapshots) rather than the binding; it now keys off TestModels.seq_rm_kind(:mtp): :full targets must bill ckpt, others must not. Verified on M4 Max / Metal: - pre-fix, unsloth gemma-4-E4B-it-Q4_K_M + MTP/mtp-gemma-4-E4B-it-Q8_0: validation passes (nextn=4, n_embd_out 2560 both), then llama_init_from_model fails with the ctx_other error - post-fix, same pair: MTP.init ok, greedy " Paris." with 53% acceptance at n_draft 3; --include mtp_sidecar 6/6 (incl. greedy equivalence with plain target decode) - --include mtp (Qwen3.5-0.8B-MTP-Q8_0, in-file head): 19 passed - --include smoke --include embeddings: 548 passed, 29 excluded - default suite 428 passed; mix format and credo --strict clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #91.
Root cause
llama.cpp's
gemma4-assistanthead builds its draft context over the target's: the draft'sllama_kv_cachealiases the target's cell vector (other->v_cells_impl) and the graph reads the target'stok_embd. Its constructor throwsGemma4Assistant requires ctx_other to be setwhenllama_context_params.ctx_otheris null (llama-context.cpp,LLM_ARCH_GEMMA4_ASSISTANT).context_createbuilt every context fromllama_context_default_params()and never exposed the field. The Qwen 3.8 sidecar path from v0.8.44 worked becauseqwen35never reads the pointer; a Gemma 4 paircould not get past
Context.create.Reproduced pre-fix with Unsloth
gemma-4-E4B-it-Q4_K_M.gguf+MTP/mtp-gemma-4-E4B-it-Q8_0.ggufon Metal:validate_head/2passes (headn_layer_nextn == 4,n_embd_out2560 on both), thenllama_init_from_modelfails with exactly the error in the issue.Fix
Mirrors upstream's
common_speculative_init_result, which setscparams.ctx_other = ctx_tgtfor every draft context.c_src/llama_cpp_ex/llama_nif.cpp—context_createtakesstd::optional<fine::ResourcePtr<LlamaContext>> ctx_otherand forwards it asparams.ctx_other.c_src/llama_cpp_ex/llama_nif.h—LlamaContextholds the peer'sResourcePtr, so the target cannot be freed under a draft whose KV cells alias it.LlamaCppEx.Context.create/2— new:ctx_otheroption (%Context{}ornil), added to the structural keys and documented.LlamaCppEx.MTP.init/2— passes the target as the draft's:ctx_otherunconditionally.llama-context.cpponly reads the field forGEMMA4_ASSISTANT/EAGLE3/DFLASH, so the in-file and Qwensidecar paths are unchanged.
test/mtp_model_test.exs—timing_us reports a ckpt bucketassertedckpt > 0, which pins Qwen 3.8's hybrid-SSM snapshotting rather than the binding. It now branches onTestModels.seq_rm_kind(:mtp)::fulltargets must be billed for snapshots, everything else must not — the same condition as the NIF'sneeds_ckptgate.Creating a Gemma 4 draft context without a target still fails. That is llama.cpp's contract;
Context.create/2documents it.Verification (Apple M4 Max, Metal source build)
{:error, "failed to create context"}/Gemma4Assistant requires ctx_other to be setMTP.initok; greedy" Paris.", 53% acceptance atn_draft: 3--include mtp_sidecaron the Gemma pair--include mtp(Qwen3.5-0.8B-MTP-Q8_0, in-file head)--include smoke --include embeddingsmix test(no model)mix format --check-formatted,mix credo --strictNot run: the Qwen 3.8 27B sidecar pair (no local copy). Unaffected by construction —
qwen35ignoresctx_other— and the identical code path passed on the Qwen 0.8B in-file head.Docs
CHANGELOG.md—## Unreleased→ Fixed entryLlamaCppEx.MTPmoduledoc — the two sidecar head architectures and what each needs from the targetREADME.md— Gemma 4 E4B pair added to the MTP model listdocs/release-guide.md— Gemmamtp_sidecarrun added to the test matrix, since a Qwen pair alone cannot exercise thectx_otherwiring