[gabewillen]

The Metal kernel actor (PR #114) rejects the PersonaPlex LM cache: its op_mul_mat only accepts f32/f16/q8_0 and op_get_rows only accepts f32/f16 codebooks, while the LM cache is q4_k.

This generalizes --quantize q8_0 in tools/bench/moshi_gguf_convert.py to also support the lm component, so the generator can emit an artifact whose dtypes both the aarch64 CPU and Metal kernels dispatch ("right dtypes for both CPU and metal"):

• lm matmul weight classes (the LM_Q4_K_PACKABLE_PATTERNS set) → q8_0, via faithful ggml q4_K/q4_0/q8_0 dequant → q8_0 requant. Rows that don't block (e.g. a 16-wide depformer gate) stay or upcast to a float both backends accept.
• op_get_rows codebooks (text_emb, emb.*, depformer embeddings) → f16 when not already f32/f16.
• Everything else is streamed through unchanged; moshi.lm.quantization + manifest record the rewrite.

Verification (python3 tools/bench/moshi_gguf_convert.py --self-test):

• q4_K decoder pinned to golden ggml decode values (d=1, dmin=80, sc=1/m=0 exact nibble anchors).
• Focused rewrite: q4_K matmul → q8_0 (payload size + dequant round-trip ≈ source), q4_0 codebook → f16 (exact nibble decode), f32 passthrough.
• End-to-end convert() on the tiny f32 LM fixture: transformer weights become q8_0, no matmul tensor left in an unsupported dtype, manifest records q8_0.
• All existing tiny fixtures regenerate byte-identical (no behavior change to default output).

Note: this is the dtype layer only — PersonaPlex-on-Metal still needs rms_norm/rope/mul/argmax kernels beyond the dtype fix, but the weight representation blocker is gone.

---

Note

Low Risk
Bench-only converter changes; default conversion path is unchanged unless --quantize q8_0 is used on an lm GGUF.

Overview
Extends --quantize q8_0 in moshi_gguf_convert.py from mimi-only to lm, so PersonaPlex-style q4_k caches can be rewritten into dtypes aarch64 CPU and Metal both accept.

For lm, matmul weights matching LM_Q4_K_PACKABLE_PATTERNS are dequantized (ggml q4_K/q4_0/q8_0/f16/f32) and requantized to q8_0 when row length aligns to 32; narrow or already-float matmul rows stay f32/f16 or upcast to f16 instead of padded q8_0. Embedding/codebook tensors used by get_rows (text_emb, emb.*, depformer embeddings) are upcast to f16 when quantized, since Metal only allows f32/f16 there. Conversion records moshi.lm.quantization and manifest fields including lm_embedding_upcast_to_f16.

Adds faithful dequant helpers, quantize_lm_tensors, read_enriched_gguf_tensors for verification, --self-test (synthetic q4_k/q4_0 cache plus end-to-end tiny fixture via convert()), and makes --source/--output optional when self-testing.

^{Reviewed by Cursor Bugbot for commit dc3245a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Extends the Moshi GGUF converter’s optional --quantize q8_0 mode to support LM caches by rewriting selected LM matmul weights into q8_0 (dual-dispatch for aarch64 CPU + Metal) and upcasting get_rows embedding/codebook tensors to f16, while preserving non-target tensors unchanged and recording the rewrite in GGUF KV + manifest metadata.

Changes:

• Add LM-specific quantization pipeline (quantize_lm_tensors) including faithful dequant helpers for q4_K/q4_0/q8_0 and f16/f32 passthrough.
• Introduce LM_EMBEDDING_PATTERNS and read_enriched_gguf_tensors() to validate emitted dtypes and implement --self-test covering both synthetic cases and an end-to-end tiny fixture.
• Update CLI/manifest/KV behavior to support LM quantization (moshi.lm.quantization, lm_embedding_upcast_to_f16) and make --source/--output optional for self-test runs.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.