engine: notice when a tensor's format silently disables the fused Metal decode path

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.

A container can mint any fused-bound tensor at a format the fused Metal decode
kernels don't accept — and today that silently pushes the affected layers'
decode attention onto the CPU path with no signal anywhere. We measured the
kv_b_proj case on an M5 Max at +22% end-to-end once cured. This PR makes the
condition visible at load, and makes it impossible for the diagnostic to drift
out of sync with the gates it describes:

• One shared per-layer predicate (metal_fused_layer_fmt_miss, a pure
  bitmask over the 8 fused-bound tensor kinds) now backs BOTH fused-gate call
  sites (attention_rows, layer_forward_rows) and a new one-line-per-kind
  [METAL] stderr notice at model_init. Gate behavior is bit-identical —
  see the matrix below. The notice names the tensor kind, the offending fmt,
  the affected-layer count, the fmt 1/2/3/4 allowlist requirement (kv_b:
  fmt=2 exactly, with the --kvb-bits 4 remedy), and stays silent on clean
  containers and non-Metal builds. sh_* kinds count over the sparse-layer
  population (the only layers that load them); the MTP head is excluded by
  design (commonly INT8, never fused).
• Notice-only: no gate or behavior change. The notice reports a FORMAT
  obstacle; it does not claim the fused path would otherwise engage (the
  gates' dims/batch/config preconditions are deliberately not duplicated).
• docs/FORMATS.md gains a short "Format consumers" note recording these
  requirements with anchors to the predicate, both gates, and the notice.

Requirement	Decisive evidence
Gates decide exactly as before	Exhaustive equivalence sweep, all 86,093,442 fmt×sparse configurations, old conditions vs shipped predicate: 0 mismatches (independently rebuilt twice; harness proven non-vacuous by seeded mutation)
Real-model behavior unchanged	Old-vs-new binary on a real 358 GB GLM-5.2 int4 container, temp-0: generated text byte-identical, hit rates identical, zero notice lines (clean container)
Every notice behavior is pinned and the pins bite	10-part test (test_kvb_notice.c), incl. per-bit gate-mask membership; 7-mutation battery — every mutation caught, incl. a single mask bit dropped
No regressions	make check green at every commit; METAL=1 build zero warnings vs baseline

Commit structure, verification detail, and observations

Commits by origin:

1. 2078199 — the original kv_b-only notice, mechanically rebased onto
   current dev. Only resolution of note: the hand-maintained TEST_BINS
   addition was dropped — dev now auto-derives test binaries from build rules,
   which removes that whole conflict class.
2. 988223d — the widening (all 8 fused-bound kinds) + the shared-predicate
   consolidation (replaces 11 scattered allowlist call sites with 7 inside one
   helper; the gates' comment blocks updated to match).
3. 9fe7894 — review round: sh_* denominators (sparse-layer population),
   per-bit mask-membership test (a dropped mask bit previously passed the
   whole suite), mixed dense/sparse test shape, format-only semantics
   documented in the notice's comment.
4. 353f5ea — the FORMATS.md consumer-requirements note.

Verification: the equivalence sweep enumerated fmt ∈ {0..8} per tensor ×
sparse ∈ {0,1} (all values outside {1,2,3,4} are one behavioral class, so the
domain covers every class with margin) and compared the old gate expressions,
transcribed verbatim from the base revision, against the shipped predicate at
-O3 and -O0. Compiler output at -O3 confirms the helper inlines into both
gates, the attention gate's masked-away sh_* reads are dead-code-eliminated,
and the remaining check vectorizes — it is not slower than the chain it
replaces. Dense layers never load sh_* tensors, so those kinds are checked on
sparse layers only (a literal all-layers check would false-positive on every
real GLM load). Test suite: kv_b miss · each allowlist kind · multi-kind
bounded output (≤8 lines) · all-pass silence · Metal-off silence · MTP
exclusion · dense-exemption · predicate truth table · mixed dense/sparse
counts · per-bit mask membership.

Observations for maintainers (nothing here is changed by this PR):

• make metal-test does not compile on current dev in either
  COLI_METAL_RESSET state: quant.h:483 fp8_nblk (returns int64_t)
  collides with tests/test_backend_metal.mm:99's local static int fp8_nblk. Pre-exists this branch (verified at the untouched base). Happy
  to send the trivial fix separately.
• The metal_fused_fmt_ok comment block said "fmt 1/2/4" while the predicate
  includes fmt==3; the consolidated comment now just names the allowlist.
• attention_rows gained the (S<=4 || g_metal_prefill) prefill scope while
  layer_forward_rows did not — the two gates have diverged on batch scope
  (flagging only; format conditions are what this PR unifies).
• If a future kernel binds a new tensor kind, the predicate, both masks, and
  the notice's kind arrays must move together — the per-bit membership test
  now enforces the mask half of that lockstep.

[michael-denyer]

Second Apple datapoint for this, from the other side of the same wall.

I hit this exact path last week converting a GLM-5.2 REAP-504B container to fmt=6 on an M5
Pro. The routed experts were fmt=6, but the converter defaulted the dense and attention
tensors to grouped int4 (fmt=4), and with no fused Metal fmt=4 attention kernel (#587) the
whole attention path fell to the CPU with no signal. In a 147 s decode that was ~66 s of
attention on the CPU, and it looked like the format itself was slow. It was not. It was this
silent fallback.

Curing it (rebuilding the container per-row so dense matches, --group-size 0) is what let
the real fmt=6 number appear: +24% end-to-end vs int4 on the same box, warmed and quiet
(full run on #732). So your +22% on M5 Max and my +24% on M5 Pro are the same fix measured
on two machines.

The load-time notice would have saved me a full reconversion of diagnosis. Worth having.
One thing that composes with it: the trap has two ends, the engine (this PR, the fused gate)
and the converter (which mints the fmt in the first place). A matching one-line warning at
convert time, "these dense/attention tensors are fmt=4 and the Metal fused path needs fmt=2,
pass --group-size 0", would catch it before the container even exists.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.

Rebased and re-derived onto current dev (base 1658b1f) — head is now 574aac1. Since #587's rebase already carried the narrow kv_b notice onto dev (thanks @RDouglasSharp for preserving authorship), this PR is now exactly the widening that dev still lacks, re-derived rather than merged. Changes sorted by origin:

1. Caused by dev moving. The kv_b gate condition on dev is now two formats with a mode term — fmt==2 || (fmt==4 && !g_moe_exact) — and the notice follows it exactly. The old Makefile hunk is dropped (dev auto-derives TEST_BINS now). docs/FORMATS.md line anchors recomputed. test_fp8_load.c Part F re-pinned (11 → 7 predicate call sites after the refactor). Message text re-derived so the current test's retired-string pins keep passing.

2. A pre-existing defect, found during the re-derive — with run evidence. The narrow notice now on dev has already drifted from the gates it describes: its condition omits the !g_moe_exact term. Concretely — a user who sets COLI_METAL_MOE_EXACT=1 (choosing exactness) with a grouped-int4 (fmt=4) kv_b loses the fused path on every layer and gets no message about it, which is the silent fallback this notice exists to prevent. We demonstrated it on an untouched dev tree (loader-seam test: both gates report closed, notice emits zero bytes), then fixed it structurally: the format condition is now typed once, in a shared predicate that both gates and the notice consume. It cannot drift again, because there is no second copy to go stale.

3. The original widening. One per-layer format-miss bitmask covering kv_b plus the seven allowlist tensors (q_a, q_b, kv_a, o, sh_gate, sh_up, sh_down at fmt {1,2,3,4}; the sh_* trio checked on sparse layers only). Output is one line per offending tensor kind, at most 8 lines total, never per-layer spam. FORMAT-ONLY semantics stated in code (a line names a format obstacle; it does not claim the fused path would otherwise engage). Plus the FORMATS.md consumer note.

The four promises this change makes:

1. Notice-only — zero behavior change outside stderr.
2. The format condition exists in exactly one place.
3. Every miss the gates act on is named, with accurate remedy text in both modes. (The old advice — "requantize with --kvb-bits 4" — was circular under MOE-exact: the converter's default --group-size 64 mints grouped int4, which is exactly what the mode keeps off the fused path. The message now says so instead of suggesting it.)
4. Silent when Metal is off or all layers pass; MTP head excluded by design.

Capstone matrix — the single decisive check per promise, at 574aac1:

promise	the check that would catch us
notice-only	same prompt, old vs new binary, real 358 GB container, temp-0: generated text md5 identical, zero notice lines both arms, identical expert hit rate (run at the pre-rebase pair; carried across the final rebase because the replayed patch is content-identical and the intervening commits touch disjoint regions)
single-source condition	Part F pins the predicate's 7 call sites + per-gate consultation; re-inlining any gate condition fails it
dev's drift defect is real	loader-seam demo on untouched dev: gates closed + notice silent under MOE-exact + fmt=4
the new coverage has teeth	55 of the new test's assertions fail when run against dev's current code
mask integrity	per-bit membership tests for both gate masks; dropping any single bit fails by name
bounded output	mixed 61-layer and 2000-layer fixtures: ≤8 lines, each kind named exactly once

Full requirement → instrument → result matrix

• Notice-only → differential matrix (above) + logical-equivalence review of both gate rewrites → pass
• Sanitizers → ASan+UBSan on both changed tests → pass, zero reports
• Single-source condition → grep enumeration + Part F pins → pass
• kv_b mode term exact → truth-table tests (fmt 2/4 × mode on/off); dropping !g_moe_exact fails 10 assertions → pass
• Superset of dev's current test → every dev scenario preserved, including its negative string pins → pass
• Per-kind coverage → 7 single-kind fixtures + bounded multi-kind case → pass
• sh_* sparse-only, sparse denominators → dense-exemption + denominator pins; inverting the exemption fails 27 assertions → pass
• Remedy accuracy in both modes → exact-phrase pins mode-on and mode-off; collapsing the mode-aware text fails 2/1 assertions → pass
• Mechanical → make check · METAL build zero warnings · metal-test both COLI_METAL_RESSET states · unused-function sweep · POSIX-clean tests → all exit 0

Durable vs current-state. The shared-predicate architecture and the tests are durable. Calibrations, checkable at 574aac1 (macOS/Metal, 2026-08-13): the FORMATS.md line anchors; today's fused tensor set and format allowlist; the kv_b mode term mirroring #587's gates. If a future change adds a fused tensor kind or admits a new format, predicate + masks + notice arrays must move together — the per-bit tests enforce the mask half of that lockstep.

Observations, not changes (pre-existing, disclosed so they're on the record): backend_metal.mm parses COLI_METAL_MOE_EXACT separately with its own static — the same duplicated-condition shape one level down, a plausible follow-up; a near-duplicate mode term exists at the GEMM-eligibility check (c/colibri.c:913), outside this PR's scope, so our single-source claim is scoped to the fused gates + notice; the fused attention gate also evaluates the MTP head's tensors (its INT8 kv_b closes it — carried, correct CPU fallback); and runtime-quantized dense paths (fmt 0) produce accurate per-kind lines, now pinned as accepted behavior.