Skip to content

#[unroll] over the clone budget leaves an array-indexing loop rolled and moves the array to .local, with no way to raise the budget #1478

Open

Description

@praveenperera

What problem does this solve?

When #[unroll] on the loop that selects an element of a local array goes over the 65536-operation clone budget (MAX_CLONED_OPS in mir-transforms/src/unroll.rs), the loop stays rolled. The whole array then moves to .local memory. The only signal is a warning, and there's no way to raise the budget for that loop.

The budget counts the outer loop's body after its inner #[unroll] loops have already been expanded. So a typical tensor-core accumulator pattern goes over budget at modest sizes:

#[kernel]
pub fn acc_tiles(a: *const f32, b: *const f32, out: *mut f32) {
    let tid = thread::threadIdx_x() as usize;
    let mut acc = [[0.0f32; 8]; 32];
    let mut t = 0usize;
    #[unroll]
    while t < 32 {
        let mut j = 0usize;
        #[unroll]
        while j < J {
            let mut n = 0usize;
            #[unroll]
            while n < 8 {
                let x = unsafe { *a.add(tid + (j * 32 + t) * 128) };
                let y = unsafe { *b.add(tid + (j * 8 + n) * 128) };
                acc[t][n] += x * y;
                n += 1;
            }
            j += 1;
        }
        t += 1;
    }
    // store acc with the same unrolled t/n loops
}

On main (bafffe45), sm_80, ptxas 13.0.88 -v:

J warning acc registers stack / spill
1 none registers 72 0 / 0
3 none registers 168 0 / 0
8 not unrolled (152672 ops) .local, 792 LDL/STL 96 1024 B / 0
32 not unrolled (609632 ops) .local, 944 LDL/STL 255 1184 B / 160 B
warning: #[unroll] requested but the loop was not unrolled: unrolling would clone 152672 operations
(32 copies x 4771 operations); the safety limit is 65536 cloned operations

At J = 8, ptxas still has registers to spare, but the 1 KiB accumulator array sits in .local. The inner loops unroll correctly; only the loop that selects acc[t] stays rolled. In speakrs, this decided several kernel shapes (crates/speakrs-cuda-kernels/src/segdense.rs, the spk_segdense_conv1_b32_x3 and spk_segdense_embed_b32_x3 instances). We split work across more warps or kernels so the unrolled body fits, even where a larger tile per warp would have been faster.

Proposed behavior

Either or both:

  1. A per-loop override, such as #[unroll(full)] or #[unroll(max_ops = N)], that unrolls even past the default budget. That's like CUDA C's #pragma unroll, which has no size limit.
  2. A more precise diagnostic: name the local array that will move to .local because the loop stays rolled. Optionally, make it an error when the loop index selects an aggregate local, since the result is rarely what the author wants.

Alternatives considered

Restructuring kernels so the expanded body fits, which is what we do now.

Related: #1346 (for range loops not unrolled, same .local consequence), #1375 and #1468 (loop metadata and #[nounroll]).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions