Skip to content

Precompute the Rader's permutation - #178

Merged
ejmahler merged 2 commits into
ejmahler:masterfrom
HEnquist:raders_precompute
Sep 16, 2026
Merged

ejmahler merged 2 commits into
ejmahler:masterfrom
HEnquist:raders_precompute

Conversation

@HEnquist

@HEnquist HEnquist commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Rader's reorders the input by walking g^k mod len, one strength-reduced modular multiply per
element, in a serial chain recomputed on every call. This builds the sequence once at construction
instead.

  • each step is two 64x64->128 widening multiplies. wasm has no high-multiply opcode, so LLVM emits
    two call __multi3 plus a linear-memory round trip per element there, where aarch64 does it in
    seven instructions
  • one Box<[u32]> covers both directions: the inverse walk is that same table read backwards and
    rotated by one, since g^-k == g^(len - 1 - k)
  • costs 4 * (len - 1) bytes per instance, about 400 KB near len 100000, the same order as what
    avx_raders.rs and Bluestein's already store
  • M1, f64, eight smooth primes from 1009 to 100801: 1.24x to 1.61x. Under wasm the per-element
    permutation cost drops from 25.2-26.9 ns to 1.87-3.07 and Rader's comes out 1.9-3.5x faster

Rader's reorders the input by walking g^k mod len, one strength-reduced modular
multiply per element, in a serial chain recomputed on every call. Each step is
two 64x64->128 widening multiplies.

wasm has no high-multiply opcode, so LLVM emits two `call __multi3` plus a
linear-memory round trip per element there, where aarch64 does it in seven
instructions. That, and not the scattered access pattern, is where the backends
diverge: NEON has no gather either.

The sequence depends only on the primitive root and the length, so build it once
at construction into a Box<[u32]> and leave the hot loops as plain indexed
loads. avx_raders.rs already precomputes its output mapping for the same reason.
The inverse walk is that same table read backwards and rotated by one, since
g^-k == g^(len - 1 - k), so one table covers both directions.

Measured on an M1, f64, over eight smooth primes. Rader's runs its inner FFT
twice, so subtracting two inner FFTs isolates the permutation pass:

  len       raders ns  ->  raders ns   speedup   permutation ns/element
  1009          17027       10544        1.61x     8.09 -> 1.62
  2053          39685       26789        1.48x     8.10 -> 1.72
  4051         116000       90233        1.29x     8.33 -> 1.90
  8101         177284      126558        1.40x     8.32 -> 1.99
  15121        426733      337206        1.27x     8.17 -> 2.13
  32401        803375      615661        1.30x     8.12 -> 2.31
  65537       1239810      867489        1.43x     8.08 -> 2.36
  100801      2988425     2416661        1.24x     8.09 -> 2.27

Under wasm, where the chain was costing 25.2-26.9 ns per element, it drops to
1.87-3.07 and Rader's comes out 1.9-3.5x faster (node 24, f64).

The table costs 4 * (len - 1) bytes, about 400 KB for a prime near 100000, the
same order as the twiddles avx_raders.rs and Bluestein's already store at that
size.
@ejmahler

Copy link
Copy Markdown
Owner

one Box<[u32]> covers both directions: the inverse walk is that same table read backwards and
rotated by one, since g^-k == g^(len - 1 - k)

This is what was stopping me before, I was never able to figure out how to reuse the same buffer for both reorders. This is great work, thank you. Perhaps we should reconsider the raders vs bluestein's tuning with this in mind. Not necessarily as a part of this PR.

I'll review next week.


len: StrengthReducedU64,
// Buffer offsets for the input reordering: inner FFT input k reads buffer position
// permutation[k], which is g^(k+1) mod len, less one. The sequence depends only on the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, so twiddles generation could also be:

let (first_half, second_half) = inner_fft_input.split_at_mut((len - 1) / 2);
let mut twiddle_input = 1;
for (input_cell, conjugate_input_cell) in first_half.iter_mut().zip(second_half) {
    let twiddle = twiddles::compute_twiddle(twiddle_input, len, direction);
    *input_cell = twiddle * inner_fft_scale;
    *conjugate_input_cell = input_cell.conj();

    twiddle_input =
        ((twiddle_input as u64 * primitive_root_inverse) % reduced_len) as usize;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, thanks. Implemented this and benchmarked construction time before/after (the process() benches don't touch this since it's all in new()). Solid win across most sizes, roughly 15-28% faster for mid-range primes (149 to 65537), less at the very small end (fixed overhead dominates) and very large end (the inner FFT's own two process() calls inside the constructor start to dominate). No regressions in the existing tests. Committed as 78396b7.

len is an odd prime here, so the multiplicative group mod len is cyclic
of even order len - 1, and primitive_root_inverse^((len-1)/2) == -1 mod
len. That means twiddle_input at position p + (len-1)/2 equals
len - twiddle_input at position p, so the second half of the twiddle
array is just the complex conjugate of the first half. Only the first
half needs an actual compute_twiddle call and a step through the
modular-multiply chain; the rest is filled in with a conjugate.

Idea from ejmahler#178 (comment)
@ejmahler
ejmahler merged commit aeea24f into ejmahler:master Sep 16, 2026
19 checks passed
@ejmahler

Copy link
Copy Markdown
Owner

Looks good. Thanks to @HEnquist for the submission and @awxkee for the feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants