Repository navigation
Extend sharded GGUF reader with IQ2/IQ3/IQ4, Q5_K and Q2_0 dequant - #13
Open
catalinb13 wants to merge 41 commits into
Open
catalinb13 wants to merge 41 commits into
catalinb13 wants to merge 41 commits into
Conversation
… guard OOV logits - Speculative early-exit: gate draft chaining with cumulative confidence threshold in Qwen4ExpTarget.cpp, eliminating doomed GPU forward passes (+3-6% tok/s). - Vectorized Metal buffer bindings: batch compute buffer bindings via native setBuffers:offsets:withRange: in MetalBackend.mm to reduce CPU dispatch overhead. - Driver stall removal: eliminate addScheduledHandler memory sampling in MetalBackend.mm to avoid mid-flight GPU command buffer stalls. - Robust logits guard: reject out-of-vocabulary / non-finite tokens in Engine.cpp cleanly before KV cache publication or output emission. - Warm-up policy pipelines: pre-compile sampling and constrained decoding Metal compute pipelines at startup in Runtime.mm to remove first-token JIT spikes. - Full test suites passing: 21/21 CPU engine tests and 171/171 Python server tests.
…store) CI has failed on every push since the rename; these are the tests and checks that still expected the old names and places: - the engine is build/slipstream-v2 (engine/libslipstream.a, slipstream-v2.metallib), its status thread slipstream-v2-status-refresh, the release data folder Application Support/Slipstream-v2 (thinking.key, the completion catalog cache), the API key variable named in the error is SLIPSTREAM_V2_API_KEY, and the restart message says Slipstream v2; - shell completion lists models from the model store (~/.slipstream/models, SLIPSTREAM_MODELS), which source and release installations share; - architecture-check runs with the first of PYTHON_CANDIDATES (standard library only), not .venv/bin/python, which the production job never creates; - check-source's whole-tree whitespace check: two blank lines at EOF, two trailing spaces in Python, and two Markdown hard breaks now written as `\`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ruff format over the checked folders (27 files); ruff --fix (sorted imports, unused imports); noqa E402 on imports that follow a sys.path change, as the repository's scripts already do; two unused variables removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preparing wrote a ~100 GiB package next to the ~97 GiB of GGUF it came from: it needed twice the model's size on disk, and kept both afterwards. The package is a different encoding (affine Q4 in groups of 64, tiled, permuted heads), so it cannot share bytes with the GGUF, but the GGUF can be given back as it is converted. With --consume-source, each tensor's bytes are freed (F_PUNCHHOLE) once the part that reads it is written, the n-gram table chunk by chunk, and the files are deleted when the package is complete. Without it nothing changes for the source. - Every tensor has one owning part (tensor_owner); a part that reads another part's tensor stops the run rather than free bytes still needed. - Parts are written as .partial, fsynced and renamed, so a part under its final name is whole. prepared/.prepare-journal records the source files (name, size, inode) and each tensor before it is freed. A resume keeps the parts already written; if freed data is needed by a missing part, it says which files to download again. New downloads start afresh. - The n-gram table's scales and biases are written into reserved sections instead of 6 GiB of scratch files (StreamingWeightFile.reserve/write_at). - A free-space check runs before converting: 5 % of the source plus 1.5 GiB per worker in place, 105 % of it otherwise. - A disk that cannot free ranges (not APFS) falls back to deleting the files at the end; an MTP head outside the model's folder is never touched. Tested on Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF (clones of its files): prepared in 219 s (215 s before) with a peak of 102.6 GiB for source plus package, against 197 GiB before; all 64 files byte-identical to the package the converter made before. New tests cover freeing, resuming and refusing a lost part with a synthetic GGUF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it `serve --model <owner/repo>` now prepares a GGUF installation in the model store with --consume-source, so the first start needs a few percent more disk than the model instead of twice it, and the shards are gone once the package is complete. `serve --keep-gguf` keeps them. - An installation prepared before this loses its leftover shards and MTP head on its next serve (unless --keep-gguf): the package replaces them. - A folder of the user's own (`serve --model <dir>`) is never used up. - gguf_downloaded() needs the shards unless the package exists, so removing prepared/ downloads them again. - The download's disk check asks for the shards plus a twentieth and the parts in flight (12 GiB), not the shards twice; with --keep-gguf the converter checks for its own room. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… served
`pull <owner/repo> --check` reports what pull would install, without
downloading it: a package whose manifest the launcher accepts, or
Qwen3.8-Flash-Next GGUF files, with the download size; otherwise why not.
`--json` prints one object for frontends ({"supported": true, "kind",
"bytes", "revision", ...} or {"supported": false, "reason"}); exit 0 or 1.
It uses the bundled huggingface_hub and its token, so private and gated
repositories work as they do for pull.
pull itself now refuses GGUF files the converter cannot take, before writing
anything, instead of downloading 100 GB first:
- the top-level GGUF files must be one model: a single file, or one complete
gguf-split set (<stem>-00001-of-0000N.gguf); several quantisations side by
side, or variants in sub-folders, are named in the error;
- the first file's general.architecture, read from its header with a ranged
request (256 KiB), must be qwen4exp.
Tested against the Hub: both Flash-Next GGUF repositories (3 files each,
104.47 / 104.48 GB with the MTP head) and incoai/Qwen3.8-27B-Splash pass;
QuantFactory/SmolLM-360M-GGUF (14 quantisations) and aladar/TinyLLama-v0-GGUF
(3) are refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explains the two locations introduced with the installer and the Hub-id install, which so far were only in code comments: - releases in ~/.local/share/slipstream/<version>, linked as ~/.local/bin/slipstream: per user, no sudo, the XDG locations Claude Code, pipx and uv use too; versions side by side; - models in ~/.slipstream/models/<owner>/<repo> (SLIPSTREAM_MODELS), shared by every installation and frontend, like ~/.ollama/models and oMLX's ~/.omlx/models, excluded from Time Machine. Also where models were before (Application Support for a release, install/models for a checkout) and how to move them, since nothing is moved automatically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ished it install.sh defaults to npanj/slipstream, so on a fork the notes' one-liner fetched the fork's install.sh but then looked for the tag in npanj's releases. SLIPSTREAM_REPO now names the publishing repository; on npanj/slipstream nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Set ./slipstream as primary executable launcher script - Forward ./slipstream-v2 and ./splash to ./slipstream for backward compatibility - Build targets default to build/slipstream and build/slipstream.metallib - Add Makefile alias rules and symlinks for legacy binary targets - Update server models catalog, SSE keepalives, threads, and UI storage keys - Update test assertions across server and engine test suites
- Document Qwen3.8-Flash-Next V3 (125.7B MoE) and Qwen3.8-27B (Dense Hybrid) - Add download, pull, and serving examples for both model families - Document multi-format support: GGUF, MLX safetensors, and prepared packages - Add side-by-side trade-off matrix to guide model selection - Update architecture documentation and cross-agent status logs
…lution from PR #253
…dices, and vector-sort allocations
…ystemSleep - Replaced std::chrono::steady_clock with AwakeClock (CLOCK_UPTIME_RAW) across runtime and models to prevent false watchdog timeouts and dropped requests after Mac sleep - Enforced zero sleep-counting clocks in production via dev/tools/check_architecture.py - Implemented IdleSleepAssertion (PreventUserIdleSystemSleep) during active request execution in runtime/main.mm - Added --allow-idle-sleep flag to server.py and native command arguments - Added testHoldingRequestsSpansFirstToLastRequest in native_engine_loop_test.cpp and server tests
…ting - Implemented encodePrefillPredictRoute and prefetchPrefillPredicted in Qwen4ExpTarget addPrefill to overlap layer L+2 expert reads with layer L+1 GPU execution - Implemented prefetchLayerMisses with concurrent GCD dispatch_apply reads directly into expertCache slots for exact layer router misses - Ensured single-wave MoE GPU dispatch when all prefetched experts hit in cache - Added wait timing instrumentation to totalStageMs - Backed by environment fallback (SPLASH_NO_PREFILL_PREFETCH)
The reader only covered 6 quant types, so GGUF models mixing in IQ2_XXS/IQ2_XS/IQ2_S/IQ3_XXS/IQ3_S/IQ4_NL/IQ4_XS/Q5_K/Q2_0 failed with 'Unsupported ggml type <n>' at prepare time. - dev/tools/fast_dequant_ext.c: dequant kernels for the 9 types, vendored from llama.cpp (MIT); symbol names follow the fast_dequant.c convention so the reader looks them up uniformly - dev/tools/sharded_gguf_reader.py: build the extended dylib, extend GGML_BLOCKS / DEQUANT_TYPES / _decode - dev/tools/test_fast_dequant_ext.py: known-value + property tests Verified: bit-exact against a standalone ggml-quants.c build on random blocks; live read of every new type in a 125.7B two-shard model (75GB x 2, GGUF v3, 1224 tensors); full conversion pass in 134s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The sharded GGUF reader (
dev/tools/sharded_gguf_reader.py) only dequantized 6 quant types. GGUF models that mix in other types fail at prepare time withUnsupported ggml type <n>. This extends the reader + its dequant dylib to 15 types, adding:Q5_K (13), IQ2_XXS (16), IQ2_XS (17), IQ3_XXS (18), IQ4_NL (20), IQ3_S (21), IQ2_S (22), IQ4_XS (23), Q2_0 (42)
Why
In practice: the ISTA/DASLab
Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXSrelease (125.7B MoE, 2 x 75 GB GGUF v3 shards) uses exactly this mix (IQ4_XS attention, Q2_0/IQ2_* experts, Q5_K output projection). With this PR the full prepare pass completes (134 s on an M1 Max, 69 GB prepared package); without it the converter aborts on the first IQ4_XS tensor.Q2_0is new upstream in llama.cpp, so a reader built from "the common 9 types" still misses it.Changes
dev/tools/fast_dequant_ext.c(new): the 9 kernel bodies, block layouts and lookup tables vendored verbatim from llama.cppggml-quants.c/ggml-common.h(MIT; attribution in the header), exported under thefast_dequant.csymbol convention (dequant_q5_k, …) so the reader looks them up uniformly. Block layouts are pinned withstatic_assertto the upstream byte sizes.dev/tools/sharded_gguf_reader.py: compiles both.cfiles into the dylib, extendsGGML_BLOCKS,DEQUANT_TYPESand the_decodedispatch.dev/tools/test_fast_dequant_ext.py(new): known-value tests (Q2_0 bit patterns, Q5_K scales, IQ4_NL lookup) + determinism/finiteness property checks for all 9 types.python3 -m dev.tools.test_fast_dequant_ext.Verification
ggml-quants.cdirectly.python3 -m dev.tools.test_fast_dequant_ext— 4/4 pass (ruff check + format clean).-O3 -Wall -Wextrabuild offast_dequant_ext.c: zero warnings.Housekeeping note
I read
CONTRIBUTING.md: this repo is a per-release mirror and PRs aren't merged directly — happy to convert this into an issue + patch if that's the preferred route; the fork branchextended-gguf-dequant-typesstays available either way.