gpu, model, gpu-gates, tools: the common pieces a Nemotron-H card body attaches to

[midagedev]

Part of #156 (Nemotron 3 Nano 4B), step N7a. Head 9ae563e0, parent f13503ce, base origin/main eccb1e69. The common pieces the Nemotron-H card body (N7b) attaches to. No engine caller yet; each piece is an "add" (or, for the store, a "move") in AGENTS.md's table.

What

• gpu::recur holds the recurrent store (RecStore, LayerStore), moved out of arch/qwen3moe/scratch.rs. RecStore::new/stamped take the length pair [state_f32, ring_f32]. LinearShape::lens gives the GDN pair, so Body35, Body38 and the delta path size their stores as before. A Mamba-2 layer will pass its own pair.
• embed_rows_q5_0: the embedding gather over pack_q5_0's resident layout (Nemotron's token_embd is Q5_0). dispatch::embed_into admits Q5_0, and an id past the table raises TokenId and writes a NaN row, as the other embed rows do.
• relu_sqr, beside swiglu: max(x, 0)^2. A non-finite input stays non-finite.
• gqa_flash_seg_g5: the flash GQA decode entry at head 128, group 5 (Nemotron's 40 Q / 8 KV heads). It has its GqaInst row, so select_gqa takes group 5 instead of refusing it.
• Coverage names the rows that are now available. No Program::NemotronHBody yet; N7b adds it.

공통화한 것 (Commonized)

• The recurrent store has one owner outside any model's module. Its only architecture input is the length pair, so a second recurrent kind (Mamba-2) reuses it instead of growing a third LayerStore arm in qwen3moe.
• The Q5_0 embed reads the same packed plane the Q5_0 gemv rows read (N6). No second Q5_0 layout.
• relu² and the group-5 flash entry are math-kind rows in the common elementwise and flash owners. No Nemotron name appears in either.
• Step 0 measured the walk's ripple (dispatch::layer, site_gemv needs Q35Kernels). N7b decides between the common walk with math-kind arms and a body-owned three-arm layer() from that note. This PR lifts no walk.

Gates

Cards: RTX 3090 and RTX A6000, driver 615.71.09, CUDA 13.4. Base origin/main eccb1e6, head 9ae563e.

• gate-gpu-qwen3moe-experts at 9ae563e: exit 0. Q5_0 embed K=1088 (34 blocks, two code windows) bit-identical to dequant_row, out-of-range ids NaN; relu_sqr bit-identical; one captured node each.
• gate-gpu-qwen35moe-e2e at 9ae563e (A6000): exit 0 (the RecStore move keeps the GDN bodies' bits).
• gate-gpu-qwen3moe-flash at f13503c (3090): exit 0; flash group5 head=128 group=5 kv=2 keys=96: measured/bound 2.911e-2 PASS. From f13503c to 9ae563e no kernel changed: SAFETY comments, r * r as the arm's value, a destructure, coverage and the ratchet.
• gate-ptx-spill at f13503c: generate_ds41 305/305, gate_e2e 238/238, gate_qwen3vl_tower 16/16, violations 0. That is the base's entries plus exactly the three new ones.
• check-unsafe: ok after the dated PIN (gpu 1769 → 1786, the three new kernels' blocks, each with its SAFETY line). clippy on bloomery-gpu: 0 errors. tools/check-recipes.sh on Linux: rc 0. Mac static: lint and check-unsafe green; check-recipes red only on main's tools/review/local.py self-test (a MUST_FIX review ends 1 (got 70)), which this PR does not touch.
• just narrow origin/main: not finished; the box connection dropped twice (rc 255).

FAIL-first (through tools/mutant-run.sh, each mutant alone)

• F1, the Q5_0 embed reads the high half from the low word group (word_i = j / 4): embed Q5_0 … FAIL, killed.
• F5, the 32-block window term dropped (wbase = b & 31): embed Q5_0 K=1088 … bit_identical=false … FAIL, killed.
• F2, relu² written as relu: relu_sqr … FAIL, killed.
• F3, group 5 run with group 8's KV head (h0 / 8): the flash group-5 clause FAIL, killed.
• F4, the RecStore lift passing the ring length as the state length for Body35's GDN stores: gate-gpu-qwen35moe-e2e red, killed. The clean run after it is green.
• gate-ptx-spill before pinning: red on exactly embed_rows_q5_0, gqa_flash_seg_g5 and relu_sqr (unpinned, spill 0, jit_local 0) in generate_ds41 and gate_e2e; every other entry pinned, violations=3.

🤖 Generated with Claude Code

[github-actions]

Verdict: MUST_FIX: two additions duplicate existing owners, and a new method has broader visibility than its callers need.

Must fix

1. Share head-128 host launch orchestration

crates/gpu/src/flash_gqa.rs:4631 duplicates crates/gpu/src/flash_gqa.rs:4550: the shape checks, grid conversions and merge are the same facts (R14). Only group width and segment entry differ.
Cost: future bounds fixes need two edits.

Replace both methods. Existing calls use enqueue_pass::<GROUP> (import GROUP in scratch); the group-5 call uses enqueue_pass::<GROUP_5> with a final false argument. Update the GROUP_5 doc link to enqueue_pass.

/// Enqueue head-128 attention; refused counts fault and yield NaNs.
pub fn enqueue_pass<const PACK: usize>(
    &self, stream: &CudaStream, a: GqaArgs<'_>, mma: bool,
) -> Result<(), GpuError> {
    const { assert!(PACK == GROUP || PACK == GROUP_5) };
    let what = "flash_gqa::enqueue";
    if PACK == GROUP_5 && mma {
        return Err(GpuError::shape(what, "group 5 requires the scalar pass"));
    }
    if a.n_kv == 0 || a.ctx == 0 || a.m == 0 {
        return Err(GpuError::shape(what, format!(
            "n_kv={}, ctx={}, m={} must be positive",
            a.n_kv, a.ctx, a.m,
        )));
    }
    let n_head = a.n_kv * PACK;
    let lens = [
        ("q", a.q.len(), a.m * n_head * HEAD),
        ("kc", a.kc.len(), a.n_kv * a.ctx * HEAD),
        ("vc", a.vc.len(), a.n_kv * a.ctx * HEAD),
        ("n_keys", a.n_keys.len(), a.m),
        ("part_v", a.part_v.len(), partials_v_len(a.m, n_head)),
        ("part_ms", a.part_ms.len(), partials_ms_len(a.m, n_head)),
        ("y", a.y.len(), a.m * n_head * HEAD),
    ];
    if let Some((name, got, need)) = lens.iter().find(|(_, got, need)| got < need) {
        return Err(GpuError::shape(what, format!("{name}.len() {got} < {need}")));
    }
    let grid = launch_u32(what, "grid", a.m * a.n_kv * SEGMENTS)?;
    let merge_grid = launch_u32(what, "merge grid", a.m * n_head)?;
    let heads = launch_u32(what, "n_head", n_head)?;
    let n_kv = launch_u32(what, "n_kv", a.n_kv)?;
    let ctx = launch_u32(what, "ctx", a.ctx)?;
    let segs = launch_u32(what, "segs", SEGMENTS)?;
    let seg_keys = launch_u32(what, "seg_keys", SEG_KEYS)?;
    let m = launch_u32(what, "m", a.m)?;
    let cfg = LaunchConfig1D::new(grid, THREADS_U32, 0);
    if PACK == GROUP_5 {
        let prep = self.module.prepare_gqa_flash_seg_g5(cfg)?;
        self.module.gqa_flash_seg_g5(
            stream, &prep, a.q, a.kc, a.vc, a.n_keys, a.scale,
            n_kv, ctx, segs, seg_keys, m, a.part_v, a.part_ms,
        )?;
    } else if mma {
        let prep = self.module.prepare_gqa_flash_seg_mma(cfg)?;
        self.module.gqa_flash_seg_mma(
            stream, &prep, a.q, a.kc, a.vc, a.n_keys, a.scale,
            n_kv, ctx, segs, seg_keys, m, a.part_v, a.part_ms,
        )?;
    } else {
        let prep = self.module.prepare_gqa_flash_seg(cfg)?;
        self.module.gqa_flash_seg(
            stream, &prep, a.q, a.kc, a.vc, a.n_keys, a.scale,
            n_kv, ctx, segs, seg_keys, m, a.part_v, a.part_ms,
        )?;
    }
    let prep = self.module.prepare_gqa_flash_merge(
        LaunchConfig1D::new(merge_grid, MERGE_THREADS, 0),
    )?;
    self.module.gqa_flash_merge(
        stream, &prep, a.part_v, a.part_ms, a.n_keys,
        heads, ctx, segs, seg_keys, m, a.fault, a.y,
    )?;
    Ok(())
}

2. Reuse the synthetic-row generator

crates/gpu-gates/src/bin/gate_qwen3moe_experts.rs:160 adds another raw-row fixture generator beside crates/gpu-gates/src/bin/gate_qwen3moe_experts.rs:470. synthetic_rows already takes the quant type, block count and half-field offsets (R14).
Cost: remove the extra fixture owner now. Delete q5_0_bytes and its f32_to_f16_bits import. Replace the call at line 189 with:

let bytes = synthetic_rows((GgmlType::Q5_0, &[0]), (ROWS, K / 32), 0x2545_f491)?;

This seed still detects both the low-half read and dropped-window mutants.

3. Keep the store-length method within the crate

crates/gpu/src/linear/mod.rs:187 exports lens, but all five callers are inside bloomery-gpu. R13 requires crate visibility here.
Cost: narrow it before external callers depend on it. Replace the declaration:

pub(crate) fn lens(&self, lanes: usize) -> [usize; 2] {

Non-blocking

⚠ Add/move proof: crates/gpu/src/flash_gqa.rs:4631. Attach PTX comparisons against base 13531b6c: existing entries unchanged, three additions. Include gate commands/output and nvidia-smi; the body gives summaries.

Checked

Duplicates: both cited pairs read; the fixes reduce code and owners. The dryer kernel pair shares fold_tile.
Complexity: diagnostics flag three new orchestration functions and one kernel; gather/run gain one arm/call.
Silent failure: Q5_0 invalid ids raise TokenId and write NaNs.
Tests: replacement-fixture host emulation matches dequantization and rejects both addressing mutants.
Gates: dated unsafe PIN; numeric bands unchanged.
Escapes: none.

Notes

GPU gates not run here; card evidence incomplete.

_{Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_qwen3moe_experts.rs, crates/gpu-gates/src/bin/gate_qwen3moe_flash.rs, crates/gpu/src/arch/qwen3moe/body35.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Reuse synthetic_rows with the Q5_0 scale at byte 0; host emulation confirms this seed rejects both addressing mutants.

Suggested change

	let bytes = q5_0_bytes(ROWS, K);
	let bytes = synthetic_rows((GgmlType::Q5_0, &[0]), (ROWS, K / 32), 0x2545_f491)?;

[github-actions]

All callers are inside bloomery-gpu; R13 keeps this method crate-visible.

Suggested change

	pub fn lens(&self, lanes: usize) -> [usize; 2] {
	pub(crate) fn lens(&self, lanes: usize) -> [usize; 2] {

[github-actions]

Verdict: CLEAN: No reachable failure or blocking contract defect remains in the changed code.

Non-blocking

⚠ Refresh evidence for the reviewed head: crates/gpu/src/flash_gqa.rs:4599.
enqueue_pass adds group-5 dispatch. The body reports 9ae563e0/f13503ce; this head is 21e0ea14, and just narrow did not finish.
For the add/move proof required by AGENTS.md, attach completed base/head PTX comparisons showing exactly the two added entries and unchanged existing entries, plus the card line, model file, exact gate commands and output for this head.

Checked

Duplicates: dryer reported 0 candidate pairs. The scalar entries retain separate bodies under R12 and the documented thread-index range constraint; the recurrent store moves to one common owner.
Complexity: diagnostics flag embed_q5_0 (22), its launcher (18), and device core seg_scalar_g (17). enqueue_pass 20→25 adds group dispatch; gather 16→17 and run 67→68 add one case/clause.
Silent failure: the Q5_0 invalid-ID branch raises TokenId and writes NaNs. Absent stamps contribute zero allocated bytes; restamping an unstamped store returns an error.
Tests, static inspection: the Q5_0 gate compares dequantized bits across 34 blocks, eager/graph outputs, positions, and invalid-ID faults. The group-5 gate checks two KV planes against its reference. Shape tests assert group-5 selection and refusal details; coverage tests reject group 5 for Qwen bodies. No mutants were executed during validation.
Performance: enqueue_pass retains one segment launch and one merge launch; the group-5 engine path has no caller in this tree.
Gates: unsafe ratchet increase has a dated PIN and reason; no band or tolerance relaxed.
Escape catalog: C1–C4 do not match this diff.

Notes

GPU kernels and Rust gates were not executed during validation. The supplied summaries lack the complete card evidence required for gpu.json.evidence=true.

_{Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_qwen3moe_experts.rs, crates/gpu-gates/src/bin/gate_qwen3moe_flash.rs, crates/gpu/src/arch/qwen3moe/body35.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}