Precompute the Rader's permutation - #178
Conversation
Rader's reorders the input by walking g^k mod len, one strength-reduced modular multiply per element, in a serial chain recomputed on every call. Each step is two 64x64->128 widening multiplies. wasm has no high-multiply opcode, so LLVM emits two `call __multi3` plus a linear-memory round trip per element there, where aarch64 does it in seven instructions. That, and not the scattered access pattern, is where the backends diverge: NEON has no gather either. The sequence depends only on the primitive root and the length, so build it once at construction into a Box<[u32]> and leave the hot loops as plain indexed loads. avx_raders.rs already precomputes its output mapping for the same reason. The inverse walk is that same table read backwards and rotated by one, since g^-k == g^(len - 1 - k), so one table covers both directions. Measured on an M1, f64, over eight smooth primes. Rader's runs its inner FFT twice, so subtracting two inner FFTs isolates the permutation pass: len raders ns -> raders ns speedup permutation ns/element 1009 17027 10544 1.61x 8.09 -> 1.62 2053 39685 26789 1.48x 8.10 -> 1.72 4051 116000 90233 1.29x 8.33 -> 1.90 8101 177284 126558 1.40x 8.32 -> 1.99 15121 426733 337206 1.27x 8.17 -> 2.13 32401 803375 615661 1.30x 8.12 -> 2.31 65537 1239810 867489 1.43x 8.08 -> 2.36 100801 2988425 2416661 1.24x 8.09 -> 2.27 Under wasm, where the chain was costing 25.2-26.9 ns per element, it drops to 1.87-3.07 and Rader's comes out 1.9-3.5x faster (node 24, f64). The table costs 4 * (len - 1) bytes, about 400 KB for a prime near 100000, the same order as the twiddles avx_raders.rs and Bluestein's already store at that size.
This is what was stopping me before, I was never able to figure out how to reuse the same buffer for both reorders. This is great work, thank you. Perhaps we should reconsider the raders vs bluestein's tuning with this in mind. Not necessarily as a part of this PR. I'll review next week. |
|
|
||
| len: StrengthReducedU64, | ||
| // Buffer offsets for the input reordering: inner FFT input k reads buffer position | ||
| // permutation[k], which is g^(k+1) mod len, less one. The sequence depends only on the |
There was a problem hiding this comment.
Interesting, so twiddles generation could also be:
let (first_half, second_half) = inner_fft_input.split_at_mut((len - 1) / 2);
let mut twiddle_input = 1;
for (input_cell, conjugate_input_cell) in first_half.iter_mut().zip(second_half) {
let twiddle = twiddles::compute_twiddle(twiddle_input, len, direction);
*input_cell = twiddle * inner_fft_scale;
*conjugate_input_cell = input_cell.conj();
twiddle_input =
((twiddle_input as u64 * primitive_root_inverse) % reduced_len) as usize;
}There was a problem hiding this comment.
Nice catch, thanks. Implemented this and benchmarked construction time before/after (the process() benches don't touch this since it's all in new()). Solid win across most sizes, roughly 15-28% faster for mid-range primes (149 to 65537), less at the very small end (fixed overhead dominates) and very large end (the inner FFT's own two process() calls inside the constructor start to dominate). No regressions in the existing tests. Committed as 78396b7.
len is an odd prime here, so the multiplicative group mod len is cyclic of even order len - 1, and primitive_root_inverse^((len-1)/2) == -1 mod len. That means twiddle_input at position p + (len-1)/2 equals len - twiddle_input at position p, so the second half of the twiddle array is just the complex conjugate of the first half. Only the first half needs an actual compute_twiddle call and a step through the modular-multiply chain; the rest is filled in with a conjugate. Idea from ejmahler#178 (comment)
Rader's reorders the input by walking g^k mod len, one strength-reduced modular multiply per
element, in a serial chain recomputed on every call. This builds the sequence once at construction
instead.
two
call __multi3plus a linear-memory round trip per element there, where aarch64 does it inseven instructions
Box<[u32]>covers both directions: the inverse walk is that same table read backwards androtated by one, since
g^-k == g^(len - 1 - k)avx_raders.rsand Bluestein's already storepermutation cost drops from 25.2-26.9 ns to 1.87-3.07 and Rader's comes out 1.9-3.5x faster