fix(serve/coli): make GPU-vs-fallback counters and chat status actually visible

[aaristov]

Rewritten 2026-08-16. This PR originally enabled the Metal GPU path for
int4-g64 (fmt=4) models — MoE experts + fused attention. dev implemented
that independently and went further, so the merge of dev resolved every one
of those hunks to dev's side. What is left is the part of the branch dev
never picked up: the observability and launcher fixes below. The original
description is preserved at the bottom for history.

What this PR does now

Three fixes that make the GPU tier's behaviour visible. None of them change
what the engine computes.

1. run_serve can report backend counters at all (c/colibri.c, +7)

profile_print only ran on the oracle and generate exit paths, so the
cumulative METAL: / METAL-ATTN: / MIRROR: GPU-vs-fallback counters were
unreachable in serve mode — the mode coli chat / coli web / coli serve
all use. A served session could never show whether the GPU tier engaged or
silently fell back per block.

run_serve now takes a wall base at entry and calls profile_print at exit
under PROF=1. Safe against the byte protocol: stdin is at EOF by that point
and the last END/STAT frame is already out, so the stdout lines cannot
interleave with anything a client is still parsing.

2. coli chat status block was always empty (c/coli)

Two independent bugs, both of which silently rendered nothing:

• The stderr drain used read(), which returns only at EOF. The engine
  stays alive on stdin, so it never returned, the errlog.write() after it
  never ran, and the bounded wait always timed out on an empty file. Now a
  readline() loop with per-line write+flush, so each status line becomes
  visible as it arrives.
• The ready-line regex scanned stderr for a stdout line. loaded in Xs | resident dense: Y MB is a printf (colibri.c:9595, same in inkling.c /
  olmoe.c), so it arrives in the pre-READY stdout preamble — which
  cmd_chat was discarding into lambda b: None. The match could never fire.
  The preamble is now kept (a few hundred bytes) and the regex runs against it.

Also whitelists [METAL] / [CUDA] in the status lines. The splash tagline
(model_banner_line) names the checkpoint, not the backend, so this engine line
is the only confirmation the GPU tier actually engaged.

3. coli bench splash (c/coli)

banner("bench") → banner("bench", model=a.model), so bench names the model
it is about to evaluate like every other command.

Test comment accuracy (c/tests/test_backend_metal.mm, +7/-3)

The fmt=8 fence test documented moe_submit's gate as fmt != 1 && fmt != 2.
On dev the allowlist is {1,2,4,5,6}; comment corrected. No behaviour change.

Merge notes

The mechanical merge of dev left three defects, fixed here:

• moe_gemv ended up with a duplicate fmt == 4 branch — this branch's
  scalar version placed above dev's vectorized one, referencing gsz where
  dev's parameter is named qgs. It would not have compiled. Removed;
  dev's vectorized branch stands.
• run_attn kept an unused KGS/ng pair; dev derives kvng from kvb_gs.
• A coli comment referenced backend_tag, which disappeared with dev's
  banner rewrite.

Not carried over: the original backend_tag() (so the splash would not
claim "CPU" under COLI_METAL=1). dev's model_banner_line rewrote that line
to name the checkpoint and omit the backend entirely, so the false claim now
survives only in dev's no-model fallback string. Re-adding it means
redesigning dev's function — out of scope here, worth a follow-up.

Verification

• make metal-test — all green, including dev's fmt4-g64/g128 MoE cases and
  the grouped-kv_b attention cases.
• make colibri and make METAL=1 colibri — both build warning-clean.

Measured on M4 base / 34 GB. CI reports no checks on this fork branch, so the
CUDA/Vulkan paths dev added are not exercised by the above.

Original description (superseded by dev)

Problem

With an i4 snapshot (int4 weights + per-group scales = fmt=4, e.g.
GLM-5.2-i4), the Metal backend executed no decode work at all:

• moe_submit (backend_metal.mm) gated fmt != 1 && fmt != 2 && fmt != 6 →
  every routed-expert block on every layer silently fell back to CPU.
• Both fused-attention gates in colibri.c required kv_b.fmt==2 exactly →
  attention fell back too.

Since the [METAL] mode: … banner prints regardless, users saw a "GPU-enabled"
build running 100% on CPU.

Fix (as originally proposed)

• moe_gemv fmt=4 branch; moe_submit / coli_metal_moe_block[_begin] take a
  gs4 parameter; allowlist {1,2,4,6}.
• MB_BUILD captures the experts' group size and poisons it on mismatch.
• a_deqrow decodes per-row (fmt=2) or per-group (fmt=4) kv_b scales;
  kvb_gs threaded through AttnW.
• Tests run_moe_g4 and run_attn(kvb_g4=1).

dev landed equivalent work with a wider format matrix (fmt=5/6/8), kv_b grid
chunking, CUDA/Vulkan twins, an mb_gs_compat helper in place of the mgs=-1
poison, a !g_moe_exact refinement on both fused-attention gates, and
c/tests/test_moe_gs_guard.c. The measured numbers below were taken against
this branch's implementation and are kept only as a record of the original
finding.

	before	after
METAL: counters	(all CPU)	blocchi GPU 1248 | fallback CPU 0 | expert su GPU 9973
METAL-ATTN:	—	layer GPU 546 (100% of decode layers fused)
GPU utilization (ioreg)	1–13% (≈idle)	91–94%
decode p50	11.4 s/forward	5.9 s/forward (with PIPE=1 DIRECT=1)

🤖 Generated with Claude Code

[JustVugg]

Welcome, and thank you — this is a real fix for a real defect, and you found the same root cause I did when I went through #813 independently:

-  if (!g_dev || (fmt != 1 && fmt != 2 && fmt != 6)) return nil;
+  if (!g_dev || (fmt != 1 && fmt != 2 && fmt != 4 && fmt != 6)) return nil;

I have to tell you something before you invest more in it, and I would rather you hear it from me now than from a merge conflict later.

#587 is doing the same work, and has been for two weeks. @RDouglasSharp opened it against #585 on 21 July. It touches the same four files this does — backend_metal.h, backend_metal.mm, colibri.c, tests/test_backend_metal.mm — carries 27 comments of review, and on 1 August he wrote "Rebased onto current dev and green… Ready to merge". dev then moved again underneath him.

That is not your fault. Nothing on #813 or #585 pointed at #587 as work-in-progress, and the only reason I know is that I compared the file lists. The gap is mine to close, and I have asked myself the same question about DeepSeek twice this week.

Your PR is not redundant, though, and this is the part worth keeping. #587 fixes the decode path. Yours also fixes why nobody noticed for two weeks:

the METAL: fallback counters are printed by profile_print, which serve mode never called

That is the actual bug behind #813. The banner said "GPU-enabled", the counters that would have contradicted it were never printed, and a user running coli serve on an i4 snapshot had no way to learn their GPU was idle. run_serve calling profile_print under PROF=1 is a small change with more value than it looks, and I do not see it in #587.

There is also MB_BUILD poisoning the group size on a per-expert mismatch so the fallback is clean rather than wrong — that instinct, fail to CPU rather than compute garbage, is exactly right for a mixed-format container.

What I am proposing, and it is a proposal — you two decide, not me:

@RDouglasSharp has seniority here by two weeks and 27 comments, so #587 lands the decode path first. @aaristov, would you be willing to rebase this onto it and keep the observability and the MB_BUILD guard as a focused follow-up? That is a smaller PR, it reviews in one pass, and it closes #813 properly rather than by side effect.

If instead you two look at the two diffs and conclude the reverse — that this one is the better base and #587 should reduce to its delta — say so and I will take that. You are both closer to the Metal code than I am, and the last two overlaps on this repo were settled better by the contributors than by me.

One thing I can promise: whoever ends up rebasing will not be doing it because I let it sit. Both of you have a decision within a day.

Your CI had never run, incidentally — it was held in action_required, which GitHub applies to a first contribution from a fork and shows nowhere on the PR page. I approved it: 13 checks, no failures.

[JustVugg]

@aaristov this is wanted — it is the decode-path half of what #918 now proposes for prefill, and both should share the same expert-view plumbing — but it's marked CONFLICTING against dev. A rebase would put it back in the review queue; ping here if anything in the conflict looks like it came from our side and we'll help untangle it.

[aaristov]

Conflicts resolved against dev

All 37 conflicting hunks across 5 files resolved to dev's side, because
dev had already implemented this branch's feature independently — fmt=4 MoE
experts (as qgs), fused attention kvb_gs, and the kv_b.fmt==2||fmt==4
gates — and carried it further: fmt=5/6/8, kv_b grid chunking, CUDA/Vulkan
twins, an mb_gs_compat helper in place of this branch's mgs=-1 poison, a
!g_moe_exact refinement on both attention gates, and test_moe_gs_guard.c.
Nothing of this branch's compute-path work is kept; dev's is strictly ahead.

Three defects the mechanical merge left behind, fixed in the merge commit:

1. moe_gemv ended up with a duplicate fmt == 4 branch — this branch's
   scalar version above dev's vectorized one, referencing gsz where dev's
   parameter is qgs. It would not have compiled. Removed.
2. run_attn kept an unused KGS/ng pair; dev derives kvng from kvb_gs.
3. A coli comment referenced backend_tag, dropped with dev's banner
   rewrite. (Follow-up: backend_tag is not carried over — see the
   description. dev's no-model fallback string still hardcodes "streaming
   CPU".)

What survives is 37 lines that dev never picked up: the serve-mode
profile_print call and the two coli chat status bugs. Title and description
rewritten to match.

Verified locally (M4 base): make metal-test all green, including dev's
fmt4-g64/g128 MoE and grouped-kv_b attention cases; make colibri and
make METAL=1 colibri both warning-clean.

⚠️ CI has not run. Both CI and check queued on each push and concluded
action_required — the fork-PR approval gate. A maintainer needs to approve the
workflow runs. The CUDA/Vulkan paths dev added are not covered by my local
macOS verification.

🤖 Generated with Claude Code

[JustVugg]

Deferred to the next release (v1.7.1/v1.8.0) — not on merit, on infrastructure. Your PR is 21/22 green; the one job left is CUDA syntax check, whose NVIDIA CDN download has been hanging on GitHub's runners all afternoon. Two reruns changed nothing — the stall is on their side, not in your change.

For the record, the rewrite you did in August (resolving all 37 hunks to dev's side once dev implemented the Metal fmt=4 path independently, leaving only the observability fixes) is exactly the right way to handle being overtaken — and it's why this is now a clean +37/-9 that will go in as soon as the runners recover.