Conversation
- jb[160]->512, jb[1200]->2048, jb[400]->768: prevent truncated JSON in streaming chunks with long model ids or escape-heavy tokens - calloc(c->n_layers,...)->calloc((size_t)c->n_layers,...): silence -Walloc-size-larger-than on all 7 call sites - Add n_layers range guard (1..512) in load_cfg before first calloc - Re-assert n_layers > 0 in model_init_range after validate_cfg - Fix buf/*bn -> buf (*bn) in comment to silence -Wcomment Rebased onto current dev (conflicts with #1215 and #1227 resolved).
docs/benchmarks.md asks people to open an issue with their numbers, and then stopped recording them: the table cited nothing past #949 while sixteen reports sat open in the tracker. A datapoint that is never written down is a datapoint the next reader cannot use, and leaving the issues open was the only thing keeping them findable. Twelve rows added, covering ground the table did not have: RDNA3 under ROCm, Vulkan on a Ryzen AI notebook, DeepSeek V4 Flash on four different CPUs, Thunderbolt 5 external storage, an RTX 3090 VRAM point, and dual NVMe on independent controllers. Two reports became prose instead of rows because neither has a tok/s figure to put in one: 16 GB as a floor below the large models (#923), and the corrected Vulkan-vs-HIP comparison on RDNA4 (#523). Three findings went into Takeaways because they generalise: a second drive is worth +37.5% and more than doubles that under DIRECT=1 (#1249), AVX2-only CPUs bind on kernels rather than storage (#1119), and on Apple Silicon settings moved more than hardware did (#387, #1030).
docs(benchmarks): absorb twelve community hardware datapoints
mir_pread_striped splits a coalesced expert read into one chunk per replica
and pthread_joins all of them, so the read costs whatever the SLOWEST leg
costs. The chunk size was len/nsf regardless of how fast each drive is.
The engine already knows. g_mir_cut[] is the per-drive bandwidth ranking,
from COLI_DISK_WEIGHTS or the startup probe, and the engine prints it:
[MIRROR] probe: primary 5.15 GB/s | mirror 4.92 | mirror 0.54
[MIRROR] 3 drives | read split 48% / 46% / 5% (measured)
That split governed WHICH drive holds an expert and was then ignored for how
much of one each drive reads. A drive correctly weighted down to 5% still got
an equal third of every 19 MB stripe.
MEASURED, 2x NVMe + 1x SATA (990 Pro 5.69, SN850P 5.03, MX500 0.44 GB/s),
19 MB expert, O_DIRECT, one thread per leg as the caller does:
2 legs, equal 9.5 / 9.5 MB join waits 2.0 ms
3 legs, equal 6.33 / 6.33 / 6.33 join waits 12.6 ms
3 legs, weighted 9.69 / 8.56 / 0.75 join waits 1.9 ms
End to end on GLM-5.2 int4, n=5 per arm, interleaved on an idle machine:
2 legs stock 0.900 tok/s sd 0.025
3 legs stock 0.666 tok/s sd 0.010 -26%
3 legs weighted 0.888 tok/s sd 0.029 +33% over stock
So adding a drive the engine already measured at 10x slower cost 26%, and the
same drive weighted is a wash instead. The mechanism is visible in the byte
counters independently of the timing - same read COUNT either way, 6x fewer
bytes to the slow leg:
3 legs stock mirror2 18.30 GB / 3353 reads
3 legs weighted mirror2 3.00 GB / 3196 reads
Matched drives are unaffected: equal weights still produce an equal split,
which the test pins.
The split is factored into mir_stripe_plan() so the arithmetic is testable
without fds or threads. It reads only its arguments plus g_mir_cut.
Verified: gcc and clang (0 warnings in the changed function under -Wall
-Wextra -Wconversion -Wsign-conversion -Wshadow -pedantic), ASan+UBSan clean
over the full C suite, 721 python tests, all five engines build, and
`make colibri CUDA_DLL=1` links.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…small share
Pure arithmetic over mir_stripe_plan() - no fds, no threads, no model. It
reads only its arguments plus g_mir_cut, so the test drives it by setting
that alone. Registered by adding its rule, per the TEST_RULES comment; no
shared list to conflict on.
Five properties, and two of them are negative controls:
- a 4% leg gets under half an equal third, and a 51% leg gets over 1.3x
one. This is the bug.
- sizes sum to exactly len and offsets are contiguous, for every rotation
of rep. A gap or overlap here is silent corruption, not a slow read.
- EQUAL weights still split evenly. Without this the suite would pass on a
plan that always favours replica 0, and every matched-drive user would
quietly get a worse split than before.
- chunk 0 still lands on the routed replica, so the hash keeps spreading
first-chunk load.
- no stripe is negative or larger than len, under a deliberately lopsided
1/1/254 weighting. Guards the remainder branch.
Both bounds in the first check are deliberately far from `equal`. A plain
`> equal` on the fast leg PASSES on the old equal-split code, by 2,731 bytes
of 4K rounding - found by reverting the plan and watching that assertion stay
green while its sibling went red. An assertion that survives the bug is not
testing for the bug.
Negative control run rather than assumed: with the plan reverted to len/nsf,
3 of the 5 fail; restored, all pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(mirror): size stripe chunks by measured bandwidth, not 1/nsf
Every expert cache miss unpacks a packed-int4 expert to int8 in the slot, and the
loop doing it was indexed by ELEMENT:
for (i = 0; i < want_w; i++) {
uint8_t byte = raw[i >> 1];
int8_t v = (i & 1) ? ((byte >> 4) & 0xF) : (byte & 0xF);
if (v & 8) v -= 16;
s->g[i] = v;
}
want_w is 3 * inter * hidden = 6,291,456 for Qwen3.6-35B-A3B, so that is 6.29M
iterations per miss, each reloading raw[i>>1], doing an i&1 select and taking a
branch.
Two changes. Walking BYTES and sign-extending by shifting removes the branch.
Vectorising then needs an INTERLEAVING store -- the two nibble streams are
consecutive in the output -- which is why no compiler reaches it from the scalar
form; checking the disassembly after the branchless rewrite showed zero vector
registers. Written explicitly it emits vst2q on NEON and vpunpcklbw/vpunpckhbw
on AVX2, with the scalar form kept as the tail and the portable fallback.
Bit-exact by construction, not by tolerance: this is integer, so unlike the float
reductions in #442 there is no reassociation question. Verified identical to the
original branching form over all 256 byte values, at every length around a vector
boundary, and on a full-size 6.29M-value random expert, on both AVX2 and NEON.
Standalone kernel, single-threaded: 11.84 -> 0.18 ms on an i9-12900K, 3.19 -> 0.08
ms on an Apple M4 Pro.
End to end, Qwen3.6-35B-A3B int4-gs64, i9-12900K (avx-vnni, no AVX-512), Debian
trixie, gcc 14.2, Samsung 990 Pro ext4, container fully in page cache, cap 32,
N_NEW=48, MemoryMax=40G, three runs per rung, median:
original 1.66 tok/s 28.7s decode TTFT 7.84s
branchless 2.60 tok/s 18.5s TTFT 5.21s
vectorised 3.49 tok/s 13.8s TTFT 3.93s = 2.10x
SNAP=$M TOK=$M/tokenizer.json N_NEW=48 ./qwen36 32 4 prompt.txt
Expert miss count is 6824 in every one of those runs, to the unit, and the hit
rate never moves: placement behaviour is untouched, only what a miss costs.
Generated text is character-identical on the real model, and the tiny fixture
scores the same as before the change.
Located rather than guessed: decode time on this model is a straight line in miss
count, time = 11.1s + 2.62ms * misses, predicting LRU, PILOT and pinned configs
within 1.1%, which put ~60% of decode in the per-miss path. Squeezing the cgroup
from MemoryMax 40G to 15G against a 22 GB container moved it by -0.015 ms/miss,
i.e. not at all, which said that cost was CPU rather than I/O. After the change
the same fit reads 10.9s fixed + 0.43 ms/miss -- fixed compute unchanged, as it
must be, and the per-miss term down 6.1x. The unpack is now 21% of decode instead
of 62%.
Resolves the registration-surface conflicts created by GLM-5.3-Flash landing after this branch started, and three counters git merged without a conflict. Conflicts, all unions of the same engine list: ci.yml, release.yml, Makefile, segment_conformance_fixtures.c (renumbered glm53..qwen38 as 1..6) and the three adapter tests. family_registry.py and openai_server.py were both additive, so both sides are kept. resource_plan.py needed a decision rather than a union. dev's _dense_in_ram accounting and dense_disk_bytes are kept alongside this branch's expert_fixed_bytes; the decisions list keeps THIS branch's version, gated on supports_accelerator, because announcing a VRAM tier to a CPU-only engine writes a plan line nobody can act on. The counters are the part no branch could see alone. Both sides raised the same constant from 6 to 7, so git took the identical edit silently and the merged answer was wrong: with both families registered it is 8. test_segment_conformance.c required_families 7 -> 8 test_segment_adapters_registration.c count == 7 -> 8 test_edge_adapters_registration.c count == 7 -> 8 All seven engines build; the Python suites and the three adapter/conformance tests pass.
The merge inserted qwen38 as fixture index 6, which deepseek_v4 already held. The conflict region covered only the five wrappers both branches touched, so the deepseek_v4 line one below it kept an index that the insertion had just invalidated, and DeepSeek V4 opened Qwen3.8's fixture: Segment conformance failed for DeepSeek V4: cannot open fixture segment engine deepseek_v4 is the eighth entry in g_fixtures and g_adapters, so its index is 7. All eight families now pass. Caught by macOS CI rather than by me: my local check piped the test into tail, so the pipeline reported tail's exit status and the failing run looked like a pass on its last printed line.
The PILOT worker is detached and loops forever; nothing ever stops it. It keeps the Model address in the global pilot_m. That Model was a stack local in main, so when main returns the frame dies while the worker is still dereferencing it: AddressSanitizer: stack-use-after-return READ of size 8 ... thread T4 Matching tokens: 16/16 Reported as #1262. The tokens being right is consistent with the defect: the work is finished by then, and what is left is a thread reading a frame that no longer exists. Fixed by widening the object's lifetime rather than bounding the thread's. Static storage outlives every thread, so the pointer the worker holds stays valid through exit; main runs once and never recurses, so nothing else changes. Bounding the thread instead would mean a stop flag plus a join on every return path in main, which is more risk than this earns. Same defect in three engines, all with a never-stopped worker holding a stack Model: qwen36.c (main), olmoe.c (three branches of main) and colibri.c (main). colibri's other stack Model, in cluster_worker_run, never reaches the pilot path and is left alone. Demonstrated rather than argued, by printing the Model address against /proc/self/maps: before Model=0x7ffebbd911c0 stack=[7ffebbd7a000-7ffebbd9b000] INSIDE after Model=0x5e867a557100 stack=[7ffde5aa2000-7ffde5ac3000] outside I could not reproduce the race itself in 28 attempts, including with detect_stack_use_after_return=1, four OpenMP threads and twelve concurrent processes, on the same GCC 13.3.0 the CI uses. The window is between main returning and the process exiting. So this fix rests on the lifetime argument and the address measurement above, not on a red test turning green. qwen36 stays token-exact at caps 1, 4, 8 and 16; its C tests pass; no new build warnings (the four in qwen36.c are pre-existing, fixed by #1232).
Without this the fix in the previous commit cannot be verified by CI, and neither could the defect be seen. tools/clean.py listed five of the seven engines. For the two it missed, `make clean` removed nothing, and since the engine targets carry no dependency on the build configuration, a rebuild with different EXTRA_CFLAGS reports "up to date" and the caller keeps the OLD binary. That is how the Qwen3.6 oracle job's step named "Same run under ASan + UBSan" re-ran the UN-INSTRUMENTED binary from the step before it, at nm | grep -c __asan == 0, for as long as it existed (#1262). The job was green because the sanitizer had never run. glm53 was missing too, so the same hole was open on the engine added in v1.9.0. test_family_registry now requires every registered engine to appear in FILES. Verified by removing qwen36 again: the suite fails with "tools/clean.py does not remove c/qwen36, so a rebuild with different flags is a silent no-op". clean.py already carries a comment about the same class of bug, when the test globs matched only *.exe and `make clean` removed nothing on Linux and macOS -- noted there as silently invalidating verification. Third time in this file, so it is now a contract rather than a list someone remembers to update.
…gree
The first defect the newly-working sanitizer found.
AddressSanitizer: heap-buffer-overflow
WRITE of size 1 ... load_meta qwen36.c:1186
0 bytes after a 4-byte region allocated in load_cfg qwen36.c:1075
load_cfg sizes is_attn from config.json's layer count. load_meta then
overwrites c->n_layers from qwen36_meta.json and rebuilds is_attn from
layer_types, bounded by the NEW count against the OLD allocation. A
container saying 4 layers in one file and 8 in the other writes past the
end.
The guard already existed and named this exact hazard:
/* is_attn was sized from config.json before meta could override
n_layers. */
CFG_NEED(c->n_layers == n_layers_from_config, ...)
It just ran too late. validate_cfg is called on the line AFTER load_meta
returns, so the check that would have refused the container came one line
after the write it was meant to prevent. Correct check, wrong order.
Moved into load_meta, immediately after the field overrides and before
anything indexes is_attn. That also covers all three call sites
(model_init_range, and the two adapter paths at 2855 and 3228) rather
than only the one, since they share the same load_cfg/load_meta/
validate_cfg ordering.
This is reachable from an ordinary model directory, so it is a hostile-
input path: a mismatched pair of config files in a downloaded container is
enough. Same trust boundary as the safetensors bounds work in #413.
Reproduced locally at the identical address and line, then fixed:
before AddressSanitizer: heap-buffer-overflow ... load_meta:1186
after [cfg] config.json says 4 layers, qwen36_meta.json says 8 -- refusing
exit 1
qwen36 stays token-exact at caps 1, 4, 8, 16 under ASan; its four C tests
pass; colibri, olmoe and glm53 still build.
fix(pilot): the prefetch worker outlives the Model it dereferences
One conflict, in tools/clean.py, and it is a union: this branch added qwen36 and qwen38 to FILES, dev added qwen36 and glm53 with the comment explaining why the list matters. All three engines are now listed. Adding qwen36 there is what made the ASan step build a sanitized binary for the first time, which surfaced two pre-existing defects in qwen36 -- a dangling Model pointer held by the detached PILOT worker, and a heap overflow in load_meta on a container whose two config files disagree. Both are fixed on dev (#1277), so this branch is no longer red for them. Verified here on the merged tree: qwen36 token-exact under ASan with the sanitizer genuinely instrumented (31 __asan symbols), the malformed container refused with the [cfg] message instead of overflowing, and all five engines build.
fix(qwen36): widen jb buffers, add size_t casts, fix nested comment
…4-unpack qwen36: vectorise the int4 expert unpack (2.10x CPU decode, bit-exact)
Feat/qwen38 flash next
Qwen3.8 declares and emits tool calls in an XML-ish form of its own, not the JSON block GLM uses and not DeepSeek's DSML, so it needs both its own renderer and its own parser: <tool_call> <function=NAME> <parameter=KEY> VALUE </parameter> </function> </tool_call> Both sides are transcribed from chat_template.jinja rather than paraphrased. The declaration is what teaches the model the syntax it must emit, so a preamble it has never seen is a different prompt: it does not error, it produces malformed calls. Pinned byte for byte against the official template in tests/test_qwen38_chat_template.py -- 27 cases across the three reasoning levels, covering the separators that are easy to get wrong: a first call attaches directly when there is no preceding text and after a blank line when there is, later calls take a single newline, and consecutive tool results share ONE user turn rather than getting a turn each. The parser has an asymmetry worth stating. The template writes a string argument unquoted, so a value's original type is not recoverable from the text alone. It reads the declared schema and restores numbers and booleans from it, and leaves anything the schema did not describe as a string rather than guessing. Separately, three registry flags were wrong. capabilities.tools is descriptive -- it only feeds the capability dict -- which is exactly why it drifted without anything failing: glm53 and kimi both render and parse tool calls while sharing a COMMON_CAP that said they do not, so a client asking what a family supports was told the opposite of the truth. glm53 is mine, shipped that way in v1.9.0. test_family_registry now asks the renderer instead of trusting the flag, and the two disagreeing is a failure. Verified by putting tools=False back on qwen38: 'registry says tools=False but the renderer accepts them'. Images stay refused: this engine is text-only and a picture must be refused rather than silently dropped. 161 tests in test_openai_server.
The half of vision that decides whether it is correct rather than nearly
correct. The tower is not implemented yet and images are still refused;
this is the piece the tower would otherwise be built on top of blind.
Preprocessing is where vision goes wrong silently. A canvas one patch too
wide, a different patch order, the other model's normalisation constants:
none of that errors. The model answers anyway, and answers worse. So the
reference here is the real Qwen2VLImageProcessor, not a re-reading of the
paper.
It does not need the weights. The reference builds from
preprocessor_config.json alone -- 390 bytes -- so this was developed and
verified without downloading the 185 GB checkpoint.
Measured on eight shapes chosen to exercise every branch of smart_resize
(below the window so it grows, inside it, far above so it shrinks, and
non-square both ways):
geometry and patch order identical on all eight
pixels 0.0000 where no resampling happens
(256x256, 640x480), 0.0157 worst case where
it does
That last number is Pillow's bicubic against torchvision's, and it is
measured and printed rather than declared zero. A wrong patch order would
show up near 1, not 0.01, which is what makes the tolerance meaningful
instead of permissive.
Two differences from GLM-5.3's tower, and the reason this is its own file
rather than a parameter on glm53_image.py:
Resolution is DYNAMIC. GLM-5.3 fits everything onto a 448 canvas and pads.
Qwen keeps the aspect ratio and picks a canvas whose AREA lands inside
[shortest_edge, longest_edge], so there is no padding but the token count
depends on the image. A 1080p photo is 2040 tokens, which on an engine
that streams experts from disk is a prefill nobody will sit through --
preprocess(max_tokens=) is the same lever GLM53_MAX_IMAGE_TOKENS is, and
it shrinks rather than crops.
Normalisation is 0.5/0.5, not the CLIP constants.
An extreme aspect ratio is refused rather than turned into a degenerate
canvas, matching the reference: a 1x5000 image has no sensible
representation in square patches, and failing here is better than failing
further down.
Tower spec for whoever picks it up: 27 blocks, hidden 1152, 16 heads,
patch 16, spatial merge 2, projecting to 2560. The prompt side is already
settled by the template, which splices images as
<|vision_start|><|image_pad|><|vision_end|>.
27 blocks, hidden 1152, 16 heads, patch 16, spatial merge 2, projecting to 2560. Transcribed from Qwen4ExpVisionModel, not deduced from the paper. Verified without the checkpoint. tools/make_qwen38_vision_tiny.py builds a 240 kB tower with random weights from the upstream class and records its forward; the C reads the same fixture. make qwen38-vision-check. Two findings, both invisible until the oracle existed. THE MERGER USES A DIFFERENT GELU FROM THE BLOCKS. The blocks take ACT2FN[hidden_act], which is gelu_pytorch_tanh here; the merger instantiates nn.GELU(), the exact erf one. Using one for both matched to 4.6e-3 -- close enough to look right, far enough to move the image tokens. Found by feeding the merger the REFERENCE's own last_hidden: it still disagreed, which separates 'the merger is wrong' from 'the merger is amplifying an earlier error', and those are indistinguishable from the final number alone. A FIXTURE CAN BE TOO WEAK TO TEST WHAT IT CLAIMS TO. The first version scaled the weights to 0.05, which makes the q.k products so small that softmax comes out essentially uniform: attention degenerates into the mean of the values and stops depending on the scores. A tower with NO RoPE AT ALL passed that fixture -- verified by disabling it and watching the test stay green, not feared in the abstract. At 0.6 the scores have a real range and all four negative controls fail as they should: rope off, wrong GELU, raster patch order, flat position interpolation. The tolerance is measured rather than chosen. The reference in float32 differs from ITSELF in float64 by 1.83e-4 on these activations, so a gap of that order is the arithmetic of what we compare against, not a defect. The threshold sits between that and the 4.6e-3 the real bug produced. Two details a ViT gets wrong silently, and where they are here: pos_embed is 48x48 LEARNED positions bilinearly interpolated with align_corners, and the source indices come from (row, col) decoded in merge-block order, not raster; and RoPE 2D lives ALONGSIDE those learned positions rather than instead of them. Attention is full and quadratic in patches -- a 1080p image is 8160 of them -- so the token ceiling is a requirement here, not a convenience. Wiring the tower into the engine's sequence, and accepting images at the gateway, is the remaining step.
Completes vision. The gateway preprocesses the image, replaces the content part with <|vision_start|> + N x <|image_pad|> + <|vision_end|>, and sends the patches in an IMAGE frame ahead of the SUBMIT they belong to; the engine runs the tower once and substitutes its output for the embedding of each placeholder. N is not a constant, because the resolution is dynamic. The engine REFUSES a request whose prompt and grid disagree instead of guessing which vectors go where -- a mismatch means the template and the preprocessing saw two different images, and proceeding would put the right vectors in the wrong positions. serve_codec.h gains an IMAGE command. Five engines share that file, so the change is additive, and the payload is consumed even by an engine that cannot use it: leaving it in the stream would shift the next frame by megabytes and the gateway would read pixels as a header. The refusal therefore lives in the caller, not the codec. The position map is by ABSOLUTE position, not by offset within a chunk. Prefill arrives in pieces, and a relative index would hand the second piece the wrong image while matching silently. The map is dropped at the end of every turn for the same reason: a stale one would answer the next request about the previous photo, and would match rather than fail. Verified end to end without the checkpoint. tools/make_qwen38_multimodal_tiny.py merges the text tiny model and a tiny tower into one loadable fixture; make qwen38-vision-serve-check. The gate does not check that the model answers -- with random weights it answers regardless, and answers identically while ignoring the picture. It checks that TWO DIFFERENT IMAGES PRODUCE TWO DIFFERENT ANSWERS, which is the only question a random fixture answers honestly, and the one that catches the likeliest defect: patches loaded, tower run, result dropped between the merger and the embeddings. Plus both refusals: a byte count that disagrees with the grid, and a placeholder count that does not match. Regression after touching the shared codec: all eight engines build, qwen38 stays token-exact across all eight prefill/BF16/cap combinations, the serve-framing tests for qwen38 and DeepSeek V4 pass, and the Python suites are green.
make test-c runs every tests/test_* WITHOUT arguments. This one exited 2 with a usage line, which the suite counts as a failure -- so all five CI jobs went red on the same FAILED: tests/test_qwen38_vision. It is the rule this repository applies everywhere else, and that I wrote into the docstring of every fixture-backed test in this branch: a test that did not find its reference has verified nothing, and saying so is the only honest answer. Exiting non-zero fails on every machine that has not generated the fixture yet; exiting zero would be a lie. Now it falls back to ./qwen38_vision_tiny and, when that is absent, prints SKIP with the command that generates it and returns 0. With the fixture present it runs exactly as before -- verified both ways.
feat(qwen38): tool calling, and a tools flag that stops lying
One conflict, in docs/qwen38.md, and only there: this branch was written while tool calling was still an open PR, so it said tools were refused. #1279 has landed since, so the merged text keeps dev's tool-calling section and corrects the opening line -- audio and grammar are still refused, images now are not. Verified after merging that the two features coexist rather than merely compile: qwen38 builds, test_openai_server and test_family_registry pass, the tower still matches the upstream oracle, and the gateway offers the tool block, the image expansion and the tool parser at the same time.
feat(qwen38): image preprocessing, pinned against the official processor
Reported as #1278: with --no-think, GLM-5.3 thinks briefly and then ends without producing an answer. The prompt said two opposite things at once: <|system|>Reasoning Effort: Max ... <|assistant|><think></think> Reason maximally, and you have finished reasoning. The model answers by opening a fresh <think>, the splitter -- which had correctly started in text mode, since the prompt closed the block -- sees it and re-enters reasoning, and everything after that is filed as thought. What the user sees is thinking, then no answer. render_chat (GLM-5.2) has emitted this line under since it was written, and is the one of the two that has never had this report. render_chat_glm53 emitted it unconditionally. Now it does not. The 24 template-pinned cases still render byte-identical: they all have thinking ON, which is the only case chat_template.jinja actually defines. Nothing changed on that path. The test assertion that let this through was mine, and wrong. It required thinking-on and thinking-off to differ ONLY by </think>, which is exactly the mistake -- it treats the effort line as if it belonged in both. It now requires the two differences that must be there and no others: the closed block, and no effort line at all. Honest about what this is: a fix for a verified inconsistency with the sibling renderer, with a mechanism that matches the reported symptom exactly. I could not reproduce on the real checkpoint -- two runs on the 183 GB model timed out at roughly two minutes per token on a machine that is also uploading and training. So it is well-motivated rather than confirmed, and the reporter is asked to check.
fix(glm53): --no-think asked for maximum reasoning and closed the block
glm53, inkling and kimi had interactive_max_output equal to the non-interactive default -- 1024 -- while every other family that reasons had already been given more: glm (5.2) 16384 deepseek_v4 16384 qwen36/38 8192 glm53/inkling/kimi 1024 The token ceiling is a safety net; the real end is decided by the stop tokens. But on a model that reasons before answering, a tight ceiling does not catch you AFTER the answer -- it catches you INSIDE the thinking block, and the turn ends with the user seeing reasoning and no reply. Which is the shape of #1278, whose title is literally 'chat ends on thinking'. GLM-5.3 defaults to Reasoning Effort: Max when thinking is on, so 1024 is easy to spend before </think> arrives. 16384 is not a number I picked. It is what every family with the same 1048576 context ceiling already had; the two at 8192 are the two whose context stops at 262144. All three raised here have the 1048576 ceiling. The registry validated only >= 1, which is why three families could sit at a value that defeats reasoning without anything objecting. test_family_registry now requires a family that declares to have a larger interactive budget than its default, and olmoe -- which does not reason -- is exempt by the same rule rather than by name. Verified by putting glm53 back to 1024: '1024 not greater than 1024 -- reasoning can consume it before the answer starts'. Whether this is the cause of #1278 is not settled: the reporter used --no-think, where there should be no reasoning to spend the budget on. It is a real defect either way.
fix(registry): a reasoning family needs more room than one answer
… true (#1287) Preparing the release surfaced that the four READMEs disagreed with each other and with themselves. Eight places state a family count, two per file, and they said six, seven and eight at once: README.md:21 Eight (with Qwen3.8) README.md:586 Seven (with Qwen3.8, missing GLM-5.3) README.it.md:15 sei (missing GLM-5.3 AND Qwen3.8) README.it.md:339 Sette (with GLM-5.3, missing Qwen3.8) zh-CN, zh-TW seven in both places, both missing Qwen3.8 The English file contradicted itself between line 21 and line 586, and each file omitted a different family. My own #1267 is part of this: it fixed the Italian paragraph that states the count inside a nested quote and missed the opening sentence, because I did not look for a second one. That is the third time today the same shape has bitten: release.yml built six engines and copied seven, the adapter counters asserted 7 against 8 registered families, and now this. A count repeated in two places is a count that will diverge. So the fix is not eight edits, it is the contract. test_family_registry now requires every README -- translations included -- to name every registered family. It checks the NAME rather than the numeral: there are four languages and the numeral is written four ways, while the family name is spelled the same everywhere. Whoever adds a family and forgets the translations finds out here rather than from a reader who cannot find the model in their own language. Verified by removing Qwen3.8 from the Italian, which fails with "README.it.md: does not mention Qwen3.8-Flash-Next (qwen38)". assertTrue rather than assertIn, so the failure names the missing family instead of printing the whole file. Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
…1288) Il numero di versione vive in cinque posti -- c/version.py e i quattro README -- e niente li legava. Il modo in cui questo sbaglia non e' rumoroso: si aggiornano quattro file su cinque, la release esce, e il banner del quinto annuncia la versione precedente a chi legge quella lingua. E' la stessa forma che ha fatto uscire la v1.9.0 senza archivi: una costante, piu' consumatori, nessun controllo. Quindi il bump porta con se' il test che lo terra' vero. Il confronto e' contro version.py, non fra i README fra loro: se divergessero tutti insieme dal codice, un test di sola coerenza reciproca li troverebbe d'accordo e tacerebbe. E il test fallisce anche se il banner cambia forma, invece di smettere di controllare in silenzio. Stessa ragione, secondo punto: test_every_readme_names_every_family teneva a mano l'elenco dei quattro README. Ora li trova con una glob, cosi' un README.fr.md nuovo e' coperto senza che nessuno se ne ricordi. Controlli: allineato passa; un solo README disallineato fallisce ('1.9.0' != '1.10.0'); banner in forma diversa fallisce sul guardrail. Suite 41 test verde. Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.10.0. 41 commits since v1.9.0.
Merge with
--admin(main is protected), then tagv1.10.0and publish the Release so the archives get built.Headline: Qwen3.8-Flash-Next
The eighth family, and the first one that arrives with tools and vision on day one rather than as follow-ups.
The engine (#1250, themorlock). 125B ordinary plus 51B hashed n-gram, 48 layers as 12 blocks of three Gated DeltaNet and one QSA, 512 experts top-10 plus a shared one, four-stream gated residual. Resident 9.2 GiB; with
cap 32, 16.5 GiB total.Tool calling (#1279). Qwen3.8 does not speak the JSON tool dialect the other families use. Its format is XML-ish:
The same PR fixed a
toolscapability flag that was claiming support the renderers did not have, and a contract test now requires the flag to match what the renderer actually does.Vision (#1280). Preprocessing pinned against the official Qwen2-VL processor, then the tower itself: 27 blocks, learned positions bilinearly interpolated plus 2D RoPE, merger grouping four tokens. Verified against the upstream oracle.
One detail worth naming because it cost real time. The reference uses two different GELUs: the blocks take
gelu_pytorch_tanhfromACT2FN[hidden_act], while the merger instantiatesnn.GELU(), the exact one with erf. Using the tanh approximation in both put the tower off by 4.6e-3, which is small enough to look like accumulated float noise and is not.Fixes from the community
Two bugs that were hiding behind a broken build
make cleandid not removeqwen36orglm53, and the ASan target has no build-config dependency, so the "Qwen3.6 tiny oracle" job had been re-running an un-instrumented binary.nm | grep -c __asanon the artifact returned 0.Fixing
clean.pymade ASan actually run, and it immediately found two pre-existing bugs (#1277):Modelthat had already been freedload_metaon a container whose two config files disagreeNeither was introduced this cycle. Both were invisible for as long as the job was green and hollow.
Correctness and limits
--no-thinkon GLM-5.3 was building a prompt that asked for maximum reasoning and closed the reasoning block in the same breath. That is incoherent and is now fixed. It is not a fix for [Bug]: GLM-5.3-Flash chat ends on thinking. #1278, and the notes should not imply it is: I gave that mechanism to the reporter, then measured it on the real checkpoint and it does not hold. [Bug]: GLM-5.3-Flash chat ends on thinking. #1278 stays open.Contracts, because the same defect keeps recurring
Five times this cycle a constant had two consumers and they drifted apart in silence. v1.9.0 itself shipped with no archives because
release.ymlbuilt six engines and copied seven.__version__. Compared againstversion.pyrather than between READMEs, so a version that drifts everywhere at once cannot pass by mutual agreement.Known open
#1278, GLM-5.3
--no-thinkending on thinking, is open and without a confirmed cause. It is not a regression in this release and nothing here makes it worse. Awaiting three diagnostics from the reporter.#1284 (Makefile header prerequisites) is green and correct but covers five of seven engines; glm53 and qwen38 still have the gap. Waiting on the contributor.
Verified before opening this
release.ymlbuilds and copies all seven engines, qwen38 included. That is the v1.9.0 failure and it does not repeat.c/version.pyis 1.10.0 and all four README banners agree.test_family_registrysuite green, 41 tests.