Conversation
…ans, scripts Goal a: reproduce the issue #103 M5 Max result (2.06-2.24 tok/s) on M1 Ultra using fmt=2 (per-row int4) weights, which the Metal backend fully supports. - docs/issue_103.md: the reproduction target (full original issue text) - docs/metal_fmt2.md: fmt=4 -> fmt=2 conversion + benchmark plan (Configs A-E) - docs/ultra_benchmark_plan.md: overall M1 Ultra plan (both tracks) - docs/metal_dispatch_gap.md: why the fmt=2 track exists (fmt=4 Metal gap) - docs/metal_issues.txt: master index of Metal issues/PRs - Issue summaries: benchmark datapoints (#47 #87 #107 #180), tuning A/Bs (#387), cache-slowdown diagnosis (#379), OMP spin-wait (#707), prefill GEMM token-exactness caveat (#622) - PR summaries: Metal backend (#72), M5 Max report + methodology (#116), platform-aware cache defaults (#386), tuned Apple defaults (#750), shared-experts GPU technique (#757), GPU prefill attention option (#763) - c/convert_fmt4_to_fmt2.sh: the fmt=4 -> fmt=2 converter invocation - c/x.sh, c/venv.sh: benchmark runner + python env helper
… converter Safety checkpoint before restructuring. Contains the full ephemeral workspace: v1.4.0 run logs + SUMMARY, pre-rebase v1.2.0 logs (fmt2.old), benchmark harness, usage snapshot, context docs, and the staged fmt4->fmt2 converter. None of the docs/fmt2* content is intended for the upstream PR; it will be untracked and excluded in a later commit.
`make clean` removed nothing on Linux and macOS. tools/clean.py globbed
"tests/test_*.exe" -- its own comment says it means "no extension on Unix",
but only the Windows half was implemented, and the engine binaries
(colibri, inkling, kimi_k3) were never in the list at all. `make clean` on
this machine reported "removed 0 files/dirs" with 43 artifacts sitting
there.
That is not untidiness, it silently invalidates verification. Change a
compile flag, run `make clean && make test-c`, and the STALE binaries built
with the old flags are re-run and reported as passing. CONTRIBUTING's
`make check` opens with exactly that sequence.
It is also how this commit nearly shipped as a lie: the first sanitizer run
came back clean, and it was clean because clean.py had left the previous,
unsanitized binaries in place and make had nothing to rebuild. `nm | grep
asan` on the test binary was empty.
So, in order:
tools/clean.py removes Unix test binaries and the four engines. KEEP_EXT
is the safety rail -- a file with a source extension can never match, and
directories (tests/fixtures/) are skipped -- because the failure mode of
getting this wrong is deleting tracked sources.
EXTRA_CFLAGS/EXTRA_LDFLAGS are appended to every platform's flags. The
obvious way to build with sanitizers, `make CFLAGS=...`, REPLACES the
platform's flags and quietly drops its OpenMP and -march settings, which
is a second way to end up with a run that is not what it claims.
`make -C c test-asan` cleans, rebuilds the suite with
-fsanitize=address,undefined -O1 -g, and runs it. Leak detection is off:
the engines allocate the tensor index, parsed config and weight slabs once
and use them until exit -- st.h says so itself ("intentionally leaked ...
one-time startup parsing"). Reporting those 9 sites would bury a
heap-buffer-overflow under noise nobody intends to fix. UBSan halts on
first error rather than scrolling past it.
CI gets a `sanitizers` job running that plus `make fuzz-rans`. The fuzzer
has always been built with ASan+UBSan and asserted byte identity across
every compiled decode arm -- it simply had no job to run in, so nothing
ever ran it.
Result: the suite is clean under ASan+UBSan today, so this lands green and
guards what comes next.
Verified both directions, since a sanitizer job that cannot fail is worse
than none. With a deliberate 4-int buffer written at index 9 in
tests/test_topp.c:
make test-asan -> FAILED: tests/test_topp
make test-c -> passes, silently
Normal builds are unaffected: four engines warning-free, make test-c green.
…rter PR content (the whole intended upstream diff): - docs/METAL-M1ULTRA-FMT2-REPORT.md: fmt=2 on M1 Ultra, best 1.50 tok/s @ --ram 125 vs M5 Max 2.24 (-33%) with near-equal GPU cores (48 vs 40). Disk wait is 55-60% of the serial decode wall and the drive runs at ~93% of its 6.89 GB/s iobench F_NOCACHE ceiling in-decode: the SSD, not the GPU, sets the speed. OMP spin trap absent; PIPE is the only lever (+6.9%); PIPE_WORKERS=8 sweet spot; MTP strict loss at 128 GB. - docs/benchmarks.md: community row (same shape as the M5 Max #103 row) + one Takeaways sentence (M1 Ultra <-> M5 Max as the GPU-core-count control). - c/tools/convert_fmt4_to_fmt2.py: the fmt=4 (g64) -> fmt=2 (per-row) re-quant converter used to build the benchmark container (--selftest passes); the only in-tree path to an fmt=2 container while Metal fmt=4 dispatch is open (#585/#587). One-line c/tools/README.md entry. The ephemeral benchmark workspace (logs, SUMMARY, harness, usage snapshot, plan/context docs) stays on disk under docs/fmt2/ but is untracked and locally excluded (.git/info/exclude) — not part of the PR.
Kimi K3's routed experts are QAT in MXFP4 and streamed un-re-encoded, so
fmt=7 is the format its expert tier runs on. Only the Vulkan shader could
decode it: the CUDA backend understood fmt 0/1/2/3/4/6 and nothing else, and
c/Makefile said so outright --
@echo "*** kimi_k3 has no CUDA backend (#783): building without it."
@echo "*** On NVIDIA, Kimi K3 runs on the Vulkan path"
-- while forcing NOCUDA_CFLAGS to strip -DCOLI_CUDA back out. An NVIDIA host
either went through Vulkan or ran the experts on CPU.
This adds the fmt=7 branch to quant_matmul plus a stateless entry point, and
wires kimi_k3.c's expert_apply to try it first under K3_CUDA=1. Decode only:
at S>1 the CPU kernels amortise over the batch and a per-call upload would
not pay. It returns 0 with the output untouched on any failure, so the caller
falls through to disk+CPU exactly as it does when Vulkan declines -- the same
try-then-fall-back contract vLLM's MXFP4 backends use (FlashInfer/AITER when
available, an emulation path when not).
Two decisions the reference implementation in quant.h dictated:
The ue8m0 exponent is decoded as a bit pattern, (uint32)s << 23 read as
float, not exp2f. That IS 2^(s-127) for s in [1,254] and reproduces the
CPU path's documented edges exactly -- s=0 gives +0, s=255 gives +inf.
exp2f would agree across the normal range and diverge at precisely the two
values where a silent mismatch would hide.
e2m1 is computed arithmetically rather than read from a __constant__ table:
a file-scope __constant__ array with static linkage is initialised per
translation unit, and this file is also compiled into the HIP build and the
Windows DLL.
VERIFIED ON HARDWARE, not just compiled. tests/test_mxfp4_cuda.cu diffs the
kernel against quant.h's matmul_mxfp4 -- the CPU path the engine already
trusts -- on an RTX 4070 (sm_89):
ok all 16 e2m1 codes decode exactly (cpu == gpu == spec)
ok decode + matmul S=1 I=64 O=32 worst rel 2.43e-06
ok multi-row batch S=4 I=128 O=64 worst rel 1.59e-05
ok wide rows (many groups) S=1 I=2048 O=16 worst rel 8.41e-07
ok non-multiple-of-32 columns S=1 I=96 O=8 worst rel 1.23e-07
ok tail group (I%32 != 0) S=1 I=80 O=8 worst rel 6.40e-06
ok exponent 0 -> +0 / 255 -> inf / 127 -> unit scale
The test earned its place immediately: the first run came back with results
off by 1e38 and it took one look to see why. quant_matmul ends with
y[...] = (fmt && fmt != 4 && fmt != 6) ? partial[0] * scales[o] : partial[0];
fmt=7 was not on that exemption list, so the per-group result was multiplied
again by `scales[o]` -- and for MXFP4 that pointer is ue8m0 BYTES, so it did
not merely double-scale, it read garbage as float. A compile check would have
passed. Only a differential run against the CPU could have caught it.
Tolerance is 1e-4 relative, not bit-exactness: the two accumulate in a
different order (CPU serial over columns, CUDA strided across threads then
reduced). Decode errors are not subtle at that scale -- the bug above moved
results by 30+ orders of magnitude.
make test-c passes; all four engines build; kimi_k3.c compiles both with and
without -DCOLI_CUDA.
NOT covered: end-to-end Kimi K3 on CUDA. That needs the 1.6 TB checkpoint,
which a 12 GB card cannot hold. What is proven here is that the kernel
decodes MXFP4 correctly on real hardware; throughput on a real model still
wants a machine that can load one.
tests/test_makefile_cuda_scope.py exists to stop CUDA=1 decorating an engine that has no CUDA backend: the define matches nothing, the cudart link is never called, and with no warning printed the compile line, the libraries and the exit status all say "CUDA build" while the GPU sits idle (#783). It caught the previous commit immediately and correctly -- kimi_k3 now HAS a CUDA path, so three assertions written around "it does not" had to go. Good test. Rewriting it turned up that the guard was only ever half applied. olmoe.c contains zero COLI_CUDA references, and `make olmoe CUDA=1` was emitting: gcc ... -DCOLI_CUDA olmoe.c -o olmoe ... -lcudart -lstdc++ Exactly the failure #783 describes, on the engine nobody checked. olmoe now builds through NOCUDA_CFLAGS/NOCUDA_LDFLAGS, which is where kimi_k3 used to be and no longer needs to be. The file now tests the rule from both sides: an engine WITHOUT the backend must not receive the flag (olmoe), and an engine WITH it must (kimi_k3, colibri) -- because silently dropping -DCOLI_CUDA would compile the new MXFP4 expert path out without a word, which is the same class of bug in the other direction. 289 Python tests pass; olmoe still builds.
`gh release create` errors with "a release with the same tag name already exists" whenever a human published the release before the tag build finished — which is how any release with hand-written notes happens. v1.5.0 hit exactly that. All three platforms built, SHA256SUMS printed, and then the job went red having attached nothing. The release had notes and no downloads; the artifacts had to be pulled out of that run by hand and uploaded, and their checksums verified against the ones the job had already printed. Uploading is the operation that matters and --clobber makes it idempotent, so upload when the release exists and create only when it does not. Notes a human already wrote are never overwritten by the CHANGELOG fallback. Worth knowing separately: CHANGELOG.md stops at 1.1.1, four releases back, so that fallback has been producing "Release vX.Y.Z" for a while. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eclares
`make metal-test` has not compiled on macOS since quant.h gained
static inline int64_t fp8_nblk(int n)
because tests/test_backend_metal.mm defines its own `static int fp8_nblk`
and includes quant.h further down the same translation unit:
error: functions that differ only in their return type cannot be overloaded
Matching the return type alone does not help — it becomes an inline vs
non-inline redefinition instead. The file keeps independent reference
implementations on purpose (its own comment: "NOT quant.h's E4M3_LUT"), so
the local helper takes the ref_ prefix the other independent helpers here
already use, rather than deferring to quant.h and losing the independence
the test exists for.
Diagnosed, fixed and verified by @J-Oe-2 on an M5 Max (macOS 26.5.1,
Homebrew libomp, command-line tools only): metal-test builds and every case
passes, including GEMM, fused attention with grouped-qa, and top-8 select
with E != 256.
Also checked that no other symbol is defined in both files.
Closes #838
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The endpoint accepts none/minimal/low/medium/high/xhigh. render_chat did:
effort = "High" if reasoning_effort == "high" else "Max"
so four of the five levels that enable thinking rendered Max, and the
mapping was not monotonic: a client asking for `minimal` got more
reasoning than one asking for `high`.
On a single machine that is not cosmetic. Unrequested reasoning spends the
token budget before the answer starts -- the same effect behind #814, where
a user concluded the model could not reason at all because THINK was off.
Here the opposite: asking for less gave the most.
GLM-5.2's template takes a word, not a number, so the levels map onto the
ones it understands, in order: minimal/low -> Low, medium -> Medium,
high -> High, xhigh -> Max. `none` cannot reach this branch; it turns
thinking off upstream.
Three tests: the levels are ordered and distinct, minimal/low/medium are
specifically not Max (the reported symptom), and thinking off still emits
no effort line at all.
Reported by @ThefloorMiner.
Closes #809
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(metal-test): rename test-local fp8_nblk, which quant.h now also declares
fix(api): reasoning_effort rendered Max for every level except high
ci(release): attach to an existing release instead of failing on it
`coli doctor` reported a bare
[fail] model.shards [Errno 5] Input/output error
with no indication of which file. An OSError raised by read() on an
already-open stream carries no filename, and doctor.py prints str(error),
so EIO from a bad sector or a dropped network mount looked exactly like a
corrupt download.
In #191 the reporter drew that conclusion and re-downloaded a 372 GB model
to rule it out. It was not the download.
Which shard failed is the whole diagnosis: one file points at storage, all
of them point at the mount. analyze_model now attaches the path.
Errors that already named the file (the ValueErrors in _tensor_sizes for a
short or invalid header) are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(doctor): name the shard that failed to read
Closes the engine half of #810: a clean `systemctl stop` leaves the unit `failed`, so `systemctl is-failed` cannot tell a planned stop from a crash and every restart needs a `reset-failed` first. Only SIGINT was handled. Adding SIGTERM to the same sigaction call is not enough, and the reason is the flag already there: sa.sa_flags = SA_RESTART; /* getline/pread non devono vedere EINTR */ run_serve blocks in getline() waiting for the next request. With SA_RESTART the handler runs, sets the flag, and the read resumes -- the flag is never looked at again and the stop hangs until systemd's TimeoutStopSec turns it into SIGKILL. Measured, rather than assumed: no-SA_RESTART getline=-1 errno=Interrupted system call flag=1 SA_RESTART still blocked after 2 s flag=1 The pread half of that comment no longer holds either: every read path retries EINTR itself (st.h st_pread_full, the mirror loop in this file, uring.h). getline was the only caller SA_RESTART still protected, and it is exactly the one a shutdown has to interrupt. So SIGTERM is installed without it; SIGINT keeps it and is otherwise untouched. The two signals cannot share g_intr. SIGINT is a SOFT stop -- both serve loops clear g_intr and keep serving -- so reusing it would turn Ctrl-C into "quit", a regression. g_shutdown is separate and never cleared, and term_sig sets g_intr as well so an in-flight turn unwinds through its ordinary path (stats, usage_save, KV append, END sentinel) rather than being torn down. Both loops then end and reach the existing if(stats) stats_dump(&m,stats); return 0; Both serve paths are covered: run_serve tests g_shutdown before blocking again, and run_serve_mux breaks AFTER its mux_done sweep so in-flight requests finish first. With no active request that loop blocks in select() with a NULL timeout, and the EINTR from the un-restarted SIGTERM is what wakes it to reach the check; select is otherwise unaffected since it tests `> 0`, so EINTR reads as "no input this round". Named g_shutdown, not g_stop: sample.h already has a g_stop token array. NOT VERIFIED END TO END. The engine needs a quantised SNAP= directory and this box has only raw safetensors, so `systemctl stop` on a live serve is untested. Signal registration is confirmed in the disassembly ($0x2 -> intr_sig, $0xf -> term_sig), the mechanism is measured above, and make test-c passes.
v1.5.0 is titled "DeepSeek V4 Flash" and the Windows archive contains no deepseek_v4.exe. Unzip it, point coli at a DeepSeek checkpoint, and the answer is "deepseek-v4 engine is not built. Run: make -C c deepseek-v4" -- from a release built for that model. The Build engines step loops over `colibri inkling kimi_k3 olmoe`. DeepSeek V4 lives in its own Makefile under a hyphenated target (`deepseek-v4`, producing `deepseek_v4`), so it never joined that loop when the engine landed. Same shape as #720, which is why the loop exists at all. It cannot simply be appended: the engine builds on x86-64 Linux/Windows and aarch64 Linux only (COLI_V4_SUPPORTED in c/Makefile); on macos-arm64 the target deliberately exits 1, and folding it into the loop would turn a platform limitation into a red release build. So it gets a matrix flag and its own step, and the packaging and the archive assertions follow the same flag. The binary is copied under `deepseek_v4` -- underscore, not the hyphen of the make target -- because that is what coli's engine_for() looks for next to itself. The verify step's hardcoded engine tuple is now driven by that one flag. A second hardcoded list that silently omitted an engine is precisely how a green build shipped an unusable archive twice. Checked, not assumed: - `make deepseek-v4 ARCH=x86-64-v3` builds clean here (302 KB binary) - packaging + both archive assertions simulated for v4=1 and v4 unset: the first expects four siblings including deepseek_v4, the second three - negative control with the engine deliberately absent: both the shell loop and the python resolver check fail, so the release job goes red on a regression of exactly this bug rather than publishing silently .gitignore gets the same treatment for the same reason: it lists every engine binary except the one added last, so building it left 26 untracked COLI_V4_UNIT_*.o files in `git status`. Closes #858 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The job is named "All engines" and builds four of the five. deepseek-v4's only coverage anywhere is the Linux v4-tiny job in check.yml, so the MSYS2 build of that engine had no CI job at all -- the release tag was compiling it on Windows for the first time, which is not where you want to discover a build break. Same platform gate as the release matrix, and appended rather than folded into the loop because macos must not run it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handle SIGTERM, and drop the SA_RESTART that would have made it useless
Bugfix/autotune throughput precision
…e-rendering fix(api): render Anthropic prompts per architecture
…index fix(k3-repack): rebuild cumulative index on resume
…est (#856) v1.5.0 halved the expert cache on GLM-5.2. @brad-evony measured 0.7 -> 0.5 tok/s with identical settings, and the two dashboard screenshots name the cause without ambiguity: RAM experts 11,704 -> 5,852 cap/row 154 -> 77 disk experts 7,752 -> 13,604 rows 76 -> 76 disk service 328.4s -> 663.4s 11,704-5,852 = 13,604-7,752. Every expert removed from RAM reappears on disk. Expert matmul and attention are slightly FASTER in v1.5.0; the extra wall time is the lower hit rate, nothing else. Root cause diagnosed by @terrizoaguimor. Verified here independently: the narrow path of expert_bytes_probe() is byte-identical between v1.4.0 and v1.5.0, and the only behavioural difference is the line #793 added: if(m->has_mtp){ int64_t mtp=expert_bytes_layer(m,c->n_layers); if(mtp>eb) eb=mtp; } That is CORRECT for the ws[64] working set, whose slots really are shared across rows -- it fixed a real OOM (#766). It is wrong for the per-layer LRU caches: every row owns its own ecache[layer], so a row costs the width IT holds. cap_for_ram() divided the whole budget by the widest, so on the one container that mixes widths -- int4 routed + int8 MTP, which is how GLM-5.2 ships -- every routed row was charged as int8. One constant served two consumers whose correctness runs in opposite directions: under-estimating kills the process, over-estimating halves the cache. No scalar is right for both. The split was the missing piece. WHY THE PESSIMISM WAS NOT PARANOIA, and why this needs two changes Slabs do not stay in the row that grew them. The LRU promotion swaps ws[q] with a cache slot (colibri.c:4803, "promozione LRU"), so the cache slot's OLD contents travel back into ws[]. A ws slot widened for an int8 MTP expert returns on the next token, loads a narrow int4 expert without resizing, and is then swapped into a MAIN row's cache -- still carrying the MTP width. Up to 64 slabs per token bleed across. Given enough tokens every cache slot in the model really does cost the widest width, so charging it was the true asymptote. So expert_load() now shrinks as well as grows, on both the pread and the io_uring path. 25% relative hysteresis, because experts within a row share a shape and only a slab from a DIFFERENT row can cross it; a 64 KB floor, which has only to clear COLI_METAL's 16 KB alignment rounding. Arena slices (aslab/afslab) are exempt -- they are interior pointers into one per-layer allocation (#419) and are already per-layer width. Stopping the migration is what makes the per-row accounting true rather than optimistic. THE DISPLAYS AGREED WITH THE BUG TIERS, "[PROF] resident experts" and "[PROF] config" all multiplied a COUNT by the widest width, so the engine's self-report confirmed the halving instead of contradicting it. The dashboard claimed a ~221 GB RAM tier while Windows still showed 140 GB free, and a tier figure that disagrees with the operating system teaches people to distrust the panel. All three now use the same per-row sum the cap is computed from. Printing a projection derived differently from the cap is how this survived a release. TESTS: tests/test_cap_mixed_width.c #793 added none, and nothing in the tree had ever called cap_for_ram() -- test_cap_precedence covers which SOURCE wins, never what the number comes out as, so it passes identically whether the cap is 154 or 77. What was missing is an assertion on MAGNITUDE under a controlled A/B. A. GLM-5.2's real geometry, from a real container: 75 MoE rows + MTP = 76, int4 expert 18,915,328 B, int8 MTP head 37,789,696 B -- the 18.9 MB the docs quote and the 37.79 MB measured on the A6000 in #766. Same budget with and without the MTP row, so no slack formula is duplicated: adding one row in 76 costs 3.6%. Before, 51.5%. B. Mixed-width container: the divisor, and that expert_cache_bytes_per_slot (the autopin LRU reserve) agrees with it. D. The migration as moe() performs it. The wide slab reaches a narrow row on the THIRD step, not the first, because the swap hands back the cache slot's old contents -- a two-load test misses it. C. Arena slices are never freed by the shrink. A guard, not a regression test: it passes either way, and it is here because getting it wrong is heap corruption rather than a slow run. Negative control, every section with the fix disabled: A. cap 167 -> 81 on adding one MTP row to 75 (51.5% lost) FAIL B. cost 3.00x instead of 1.51x FAIL D. slab reaching the int4 row 892,928 B, not 450,560 B FAIL make check: 370 tests, 0 failures. NOT MEASURED, and I would rather say so The shrink is new work: a slot alternating between the MTP row and main rows now reallocates instead of staying wide, and at ~19-38 MB glibc serves that with mmap/munmap plus the page faults on first touch. Against a halved cache and doubled disk service it is plainly the right trade, but I have no GLM-5.2 on this machine and have not measured the cost. @brad-evony -- the check is your own repro: same command, same env, 1.5.0 against this branch, and the dashboard's two numbers. If the cap returns to 154 and the tok/s do not, the shrink is the suspect and that is exactly what the number would tell us. Also left alone deliberately: the PIN path still charges resident_bytes at the widest width. It over-counts, which is the safe direction for the RSS guard, and correcting it means touching #403's budget. Separate change. Refs #856, #793, #766 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
release: ship deepseek_v4 in the archives (#858)
plan: price each expert row at its own width, not the container's widest (#856)
chat_attached called sp.stop() before md.close(). Spinner.stop() emits \r\033[K (carriage return + erase-to-end-of-line), and since a reply rarely ends with a newline the cursor still sat on the reply's last partial line — so the spinner stop erased the tail of the answer, which md.close() then never re-printed (the text was already marked printed). The tail of the reply vanished from the screen exactly when the "~N tok · Ns" summary appeared, while the captured reply (and the footer token count) still included it. The GLM serve chat already used the correct order (md.close() before sp.stop()).
The batched block ran the FFN position by position, so a 64-position prefill chunk issued up to 64 x topk expert lookups per layer even when many positions selected the same expert. Measured on the real V4-Flash checkpoint (113-token prompt, #905): 4.37 disk reads per DISTINCT expert -- 42% of prefill bytes were re-reads of experts read moments earlier. v4_moe_batch_union() routes the whole chunk first, then walks the union of selected experts in ascending id order, leasing each expert once and applying it to every (position, rank) that selected it. With the loader pool the union doubles as the issue queue: disk N+lanes overlaps CPU expert N. Exactness is structural, not approximate: moe_token_pipeline() sorts a position's experts ascending and emits one term per MATCHING RANK; the union preserves both (ascending experts outer, ascending item/rank inner), so every position accumulates in the identical order. Verified on the real checkpoint: --record-oracle with the per-position path, --oracle with the union -- 26/26 teacher-forced positions, 8/8 greedy. make deepseek-v4-tiny-check passes token-exact with the union on and off; tests/test_deepseek_v4 green. V4_EXPERT_UNION=0 restores the per-position path. V4_PREFILL_CHUNK (clamped to the batch kernels' documented 64) narrows chunks on RAM-tight boxes; wider stays the default since every chunk boundary re-reads the experts shared with the previous chunk. Closes #905 (manual close on merge -- dev is not the default branch). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v4: expert-major prefill — read each distinct expert once per chunk (#905)
…ting fix: correct CNRE cost and capacity accounting
fix(cli): close the markdown stream before stopping the spinner
Single source of truth for the banner, --version and the packaged artifacts. Bumped before the tag so the released binary does not introduce itself as 1.5.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DeepSeek V4 was gated off macOS by COLI_V4_SUPPORTED, but the stated reasons had quietly expired: the header comment in Makefile.deepseek-v4 claimed a GNU-ld --wrap dependency that nothing uses anymore, and the Linux-isms in the engine already route through compat.h (posix_fadvise -> F_RDADVISE, O_DIRECT -> F_NOCACHE via st.h's direct twins), because every unit includes deepseek_v4_internal.h -> st.h -> compat.h. The one real gap: coli_v4_os_available_memory() read /proc/meminfo with a sysconf(_SC_AVPHYS_PAGES) fallback, and macOS has neither. New __APPLE__ branch uses mach host_statistics64: free + inactive pages, the closest analogue of Linux's MemAvailable. Build: Makefile.deepseek-v4 gains a Darwin arm64 gate and a toolchain branch mirroring the parent Makefile's (clang, Homebrew libomp when its artifacts exist, single-threaded fallback, no -march since NEON is baseline on arm64). Acceptance is behavioural, not "it compiles": ci.yml's macos job now builds the engine (v4: "1") and runs make deepseek-v4-tiny-check — the token-exact oracle — on the macos runner. release.yml packages and asserts deepseek_v4 in the macos-arm64 archive like every other engine. Linux verified unaffected: build clean, tiny-check token-exact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first patch script died on a later assertion without writing the file, so only the toolchain branch landed: the parent Makefile said supported, the internal COLI_V4_OK gate still said no and $(error)ed -- which is exactly what the macos runner reported. Gate, header comment and error text now match the toolchain branch that was already there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d deps make deepseek-v4-tiny-check runs the generator (--force) before the comparison, and the generator needs the pinned torch/transformers the Linux v4-tiny job installs. Round 2 died in generation, not in the oracle: "DeepSeek V4 tiny generation requires PyTorch". Same setup-python + pinned requirements as check.yml's v4-tiny job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix: reject GLM top-k larger than expert count
The pinned torch==2.13.0+cpu resolves on the Linux CPU index but macOS wheels carry the plain version (they are CPU-only by construction), so the new macos oracle step died in pip. Environment markers keep one requirements file and the identical pin on both platforms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fixup marked the +cpu pin sys_platform == "linux", which silently dropped torch for Windows users of the tiny generator (+cpu wheels exist on the CPU index for both Linux and Windows). Negate darwin instead of naming linux. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v4: support arm64 macOS — one mach branch, the rest was already portable
…#908) Engine.generate() only polled the cancelled() callback in the 'data' branch, so a client that disconnected while the engine was still prefilling (no DATA frame yet) never sent CANCEL: the turn ran to its token limit and the generate thread stayed blocked until the engine emitted something. The wait loop now polls cancelled() while idle (events.get with a 50 ms timeout), sending CANCEL pre-first-frame too. Regression test: submit to FakeProcess, flip cancelled() before any ACCEPT/DATA frame, assert CANCEL goes out and the thread unblocks via the engine's CANCELLED error, exactly as the post-first-frame path does.
The hot-thread block set OMP_WAIT_POLICY=active / KMP_BLOCKTIME=200 on every platform, but the re-exec that makes them effective is Linux/FreeBSD only, so on macOS they were inert — currently a good thing, because the #707 measurements show the knobs are a regression on Apple Silicon with LLVM libomp: M1 Max 32 GB, GLM-5.2 int4 — OMP_WAIT_POLICY=active alone +122% decode, KMP_BLOCKTIME=200 alone +115%. Reproduced on an M3 16 GB with the OLMoE engine (same libomp behaviour, smaller scale): OMP_WAIT_POLICY=active alone 2.23 tok/s vs ~3.4 baseline (-34%), KMP_BLOCKTIME=200 alone 2.54 (-25%), the full four-var set 2.87-3.01 (-5% to -25%). Now the spin-wait knobs are compiled out on Apple platforms with an explicit [OMP] note naming #707, so a future Darwin exec path cannot apply a measured regression; OMP_PROC_BIND/OMP_DYNAMIC (noise-level, measured) and the #718 physical-core sizing stay for all platforms. Item 2 of #707 (team size including efficiency cores) is already covered by coli_omp_tune_threads — the M3 sizes to its 4 P cores, not 8 logical.
…ioned-write convert: make multistream shard writes portable on Windows
coli: select python.exe from Windows project venvs
convert_inkling: support output locking on Windows
…-cancel fix(api): cancel a generation that disconnects before the first frame (#908)
fix(omp): skip the spin-wait tuning on macOS — measured slower (#707)
Two silent-failure fixes, both the sibling-of-a-fixed-GLM-defect shape: 1. Temperature carried #509's bug after GLM was cured. olmoe read TEMP raw -- on Windows %TEMP% is a directory path, atof() of it is 0.0, and every chat session silently forced greedy decoding; and 'coli --temp' wrote TEMP=0.7 over the chat child's own %TEMP% directory. olmoe now follows the GLM contract exactly: COLI_TEMP primary, TEMP as a legacy alias only when fully numeric. The launcher stops exporting TEMP for olmoe (COLI_TEMP was already set for every arch). test_v4_cli's olmoe block asserted the poisoning; it now pins the contract instead. 2. expert_get's all-slots-in-flight last resort (lru=0) stole a cache slot whose slab the pilot worker was actively filling outside the lock: two writers racing one buffer, then whichever published last decided which expert id the resident bytes answered to. Every other engine refuses this (kimi_k3 keeps the read pending, V4 skips referenced slots, colibri.c marks reservations never-evictable). Wait for an in-flight publish and rescan instead; the wait always drains, since a load either finishes or the host is already dead. No output-path change in either: sampling defaults and eviction under normal pressure are byte-identical.
fix(olmoe): COLI_TEMP channel, and never evict an in-flight slot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
colibrì v1.6.0 — the regression fixed, and prefill learns to read each expert once
If you are on v1.5.0, update now
v1.5.0 shipped a performance regression that hit GLM-5.2 (#856), left up to
~60 GB of RAM unused with a 13-point expert hit-rate loss (#885), and broke
Kimi K3 outright on some machines (#888). Two independent defects, both fixed:
width, so mixed-width containers (int4 experts + int8 MTP) undercounted how
many rows fit: the cache was silently halved.
of v1.5.0 leaves ~60 GB of RAM unused and loses 13 points of expert hit rate: 18% slower decode than v1.4.0 on a 128 GB single-GPU host #885 confirmed RSS (~105 GB on his box) and 93–97% hit rates restored.
If your symptom was "reinstalled, same commands, can't get above 0.2 tok/s"
(#939) — this release is the fix. No parameter changes needed.
DeepSeek V4: prefill I/O cut nearly in half
by position: measured 4.37 disk reads per distinct expert — 42% of
prefill bytes were re-reads. The MoE now routes the whole chunk first and
reads each distinct expert once. On a 113-token prompt: 12,576 → 7,303
disk reads, 168 → 98 GB moved. Token-exact by construction and verified
against the per-position path on the real checkpoint (26/26 teacher-forced
positions, 8/8 greedy).
V4_EXPERT_UNION=0restores the old path.was never compiled in: every expert load ran at queue depth 1 against
disks that scale nearly linearly to QD8. Pool on by default at depth 3;
V4_LOADER_LANES=<1..16>raises it. Measured ladder on the same prompt,same cache-controlled harness: QD1 157 s → 3 lanes 129 s → 10 lanes 114 s
to first token.
loader workers instead of scheduling compute onto their CPUs. Caveat found
while preparing this release, stated rather than buried: both launchers set
OMP_NUM_THREADSthemselves, which the engine reads as a deliberateoperator choice, so this policy currently applies only when
deepseek_v4is invoked directly. deepseek_v4: let the launchers delegate OpenMP team sizing to the runtime #958 addresses it and is deliberately held back — its
first form reserved from logical CPUs, which would over-subscribe SMT
hosts in exactly the way fix(omp): set OMP_NUM_THREADS from physical cores on every platform #805 fixed everywhere else.
from disk reads.
Formats and kernels
scalar on every x86 CPU without AVX-512 — which is most consumer hardware.
Cherry-picked from MiniMax-M3 support — GQA + MSA block-sparse attention, converter (follow-up to #418) #601 with authorship intact.
CI builds and runs the Vulkan backend headless under Lavapipe (ci: build and run the Vulkan backend, under Lavapipe #895),
so that backend is no longer untested territory.
CLI and web
activated; arrow keys printed escape codes — [Bug]: Pressing the UP button doesn't bring back the previous prompts #922).
when the token-count footer appears ([Bug]: Last line of reply disappears in chat, Kimi-K3 #910): the markdown stream now closes
before the spinner erases the line.
clamps replies to 1024 tokens.
dead port; kimi: a RAM budget, and make --ram mean something (#855) #872 — Kimi K3 gained a real RAM budget and
--ramnowmeans something; Handle SIGTERM, and drop the SA_RESTART that would have made it useless #850 — SIGTERM is handled.
not merely bundle it (coli launcher can't run OLMoE: missing model_arch() branch + CHAT=1 never set #879: OLMoE was named in the banner and routed to
the GLM engine).
Tools
with its cost calibration and capacity allocation corrected in review:
physical misses are no longer conflated with routed requests, and the
dynamic allocator evaluates the full feasible frontier.
Credits
bherald, mohamedmastouri2000-boop (cross-session methodology that kept our
own claims honest), steve-m, Blakeolson21, ZacharyZcR, terrizoaguimor,
dcutugno, Zach and everyone who measured, reported, and re-measured.
Full change list: v1.5.0...v1.6.0