Skip to content

The five .hip kernel sources are compiled by nothing in the fleet, so a change to them is not merely untested — it is never syntax-checked #95

Description

@forkwright

Finding

crates/kernels/build.rs skips the HIP kernel compile entirely when hipcc is absent from PATH:

    let hipcc = env::var("HIPCC").unwrap_or_else(|_| "hipcc".to_string());
    if which(&hipcc).is_none() {
        println!(
            "cargo:warning=hipcc not found on PATH; skipping HIP kernel compile. \
             Set HIPCC=/path/to/hipcc or install ROCm."
        );
        // Emit a placeholder cfg so Rust code can tell.
        println!("cargo:rustc-cfg=logismos_no_gpu_kernels");
        return Ok(());
    }

No box in this fleet has ROCm — not the workstation, not the build box, and not the GitHub-hosted
runners. So the branch above is the only branch that has ever executed, and these five files have
never been compiled by any verification anyone runs:

crates/kernels/src/matmul/hip/matmul_naive.hip
crates/kernels/src/matmul/hip/matmul_wmma.hip
crates/kernels/src/rms_norm/hip/rms_norm.hip
crates/kernels/src/rope/hip/rope.hip
crates/kernels/src/softmax/hip/softmax.hip

The consequence is stronger than "the GPU path is untested". A change to any of these files is not
compiled, so a syntax error would ship green. There is no instrument between an edit and main.

Evidence

The build script skips on missing hipcc, and returns Ok(()) — so the crate builds and the
workspace is green with the kernels absent. cargo:warning= output does not fail a build and does
not appear in a PR's checks list.

This is not hypothetical, and it bites two open memory-safety issues directly. Both of these are
filed against lines inside the uncompiled sources:

So the two defects most likely to cause an out-of-bounds device write live in the only files nothing
compiles, and any fix for them is unverifiable by every instrument this repo has.

A related symptom already surfaced independently: a toolchain-alignment PR reported "zero new lints
across all crates including kernels" after a seven-minor-version Rust bump. That claim is
structurally unavailable — the GPU kernel code was never fed to a compiler, so no lint could have
fired on it either way. The report was not dishonest; the instrument simply cannot say anything about
those files, and nothing in its output indicates that.

Why this matters

The fleet's rule is that no artifact certifies itself and verification comes from an independent
witness. Here there is no witness at all, and — worse — the absence is silent. A green workspace
build reads as "the kernels are fine" when it means "the kernels were skipped." That is the same
shape as a check that cannot fail, which the fleet has now found several times this week
(forkwright/kanon#3476, #3481, #3486): the output is not wrong, it is about something narrower than
the reader assumes
, and nothing says so.

It also sets a floor on what any audit of this repo can conclude. Findings against .hip sources can
be reasoned about but never demonstrated, so they accumulate without a path to closure.

Desired correction

Two parts, and the first is cheap.

  1. Make the skip visible. A build that silently drops five source files should say so somewhere a
    decision is made — not cargo:warning. Options: fail the build unless an explicit
    LOGISMOS_ALLOW_NO_GPU_KERNELS=1 opts into the degraded mode, or emit a compile-time
    #[deprecated]/compile_error! on the fallback path so a consumer of logismos_no_gpu_kernels
    cannot silently be in it. The point is that "kernels skipped" must be as loud as "kernels broken".
  2. Get them compiled in CI, even without a GPU. Compiling HIP does not require a device — only
    hipcc. A ROCm container image on a hosted runner can compile all five and catch exactly the
    class this issue is about (syntax, type, and index-arithmetic errors), without ever launching a
    kernel. That converts "never checked" into "compiled but not executed", which is a genuine and
    achievable improvement even though it stops short of running them.

Executing them still needs real hardware and stays out of scope here.

Done when: a deliberate syntax error in any .hip file fails CI, and the fallback path cannot be
entered without an explicit opt-in.

Provenance

Found while dispatching fixes for #33 and #32 and checking what could actually verify them.
build.rs read directly; the .hip inventory and both issues' cited paths enumerated from
origin/main.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions