Summary
Loading the DSpark speculative-decoding drafter (Ternary-Bonsai-27B-dspark-Q4_1.gguf) via --model-draft fails on the Windows Vulkan prebuilt binary with invalid vector subscript. The same command works on the Windows CPU prebuilt binary from the same release.
Environment
- Release:
prism-b9596-9fcaed7
- Binary:
llama-bin-win-vulkan-x64.zip (reports build: b1-9fcaed7, MSVC 19.51)
- OS: Windows x64
- GPU: AMD Radeon RX 6800S (8 GB), Adrenalin driver, Vulkan backend
- Main model:
Ternary-Bonsai-27B-Q2_0.gguf from prism-ml/Ternary-Bonsai-27B-gguf (loads and runs fine on the same Vulkan build without the drafter, ~17 tok/s)
- Draft model:
Ternary-Bonsai-27B-dspark-Q4_1.gguf from the same repo (sha256 c4810091d244eddc61a0cc4966e584b0959f141e3c66c0d371a6652d9f647da9)
Reproduction
llama-cli.exe -m Ternary-Bonsai-27B-Q2_0.gguf ^
--model-draft Ternary-Bonsai-27B-dspark-Q4_1.gguf ^
--spec-type draft-dspark --spec-draft-n-max 4 ^
-p "Hi" -n 8 --temp 0.7 --top-p 0.95 --top-k 20 -c 4096 -ngl 99
Observed (Vulkan build)
0.08.716.940 E llama_model_load: error loading model: invalid vector subscript
0.08.716.949 E llama_model_load_from_file_impl: failed to load model
0.08.717.101 E srv load_model: failed to load draft model, 'Ternary-Bonsai-27B-dspark-Q4_1.gguf'
The failure happens while loading the draft model; the main model loads successfully on this build.
Expected / Control
The identical command using llama-bin-win-cpu-x64.zip from the same release (build b9596-9fcaed763, Clang 20.1.8) loads both models and generates correctly with speculative decoding enabled.
Notes
- The drafter's GGUF metadata reports
general.architecture = dspark with tokenizer.ggml.model = none (tokenizer shared with the target), in case the draft-model load path on the Vulkan build handles this differently.
- Happy to provide full
--verbose logs or test a patched build.
Summary
Loading the DSpark speculative-decoding drafter (
Ternary-Bonsai-27B-dspark-Q4_1.gguf) via--model-draftfails on the Windows Vulkan prebuilt binary withinvalid vector subscript. The same command works on the Windows CPU prebuilt binary from the same release.Environment
prism-b9596-9fcaed7llama-bin-win-vulkan-x64.zip(reportsbuild: b1-9fcaed7, MSVC 19.51)Ternary-Bonsai-27B-Q2_0.gguffromprism-ml/Ternary-Bonsai-27B-gguf(loads and runs fine on the same Vulkan build without the drafter, ~17 tok/s)Ternary-Bonsai-27B-dspark-Q4_1.gguffrom the same repo (sha256c4810091d244eddc61a0cc4966e584b0959f141e3c66c0d371a6652d9f647da9)Reproduction
Observed (Vulkan build)
The failure happens while loading the draft model; the main model loads successfully on this build.
Expected / Control
The identical command using
llama-bin-win-cpu-x64.zipfrom the same release (buildb9596-9fcaed763, Clang 20.1.8) loads both models and generates correctly with speculative decoding enabled.Notes
general.architecture = dsparkwithtokenizer.ggml.model = none(tokenizer shared with the target), in case the draft-model load path on the Vulkan build handles this differently.--verboselogs or test a patched build.