Add a nightly-only FCMA backend - #177
Conversation
The FCMA instructions from ARMv8.3-A do complex multiply-accumulate directly, but the intrinsics are still unstable in Rust. Rather than hold the Neon code hostage to that, put the FCMA work in its own backend behind a nightly-only "fcma" feature, off by default. This commit is the mechanical part: src/neon copied to src/fcma with everything renamed, the target features widened to "neon,fcma", and runtime detection switched to "fcma". The code is still plain Neon, so the new backend performs exactly like the old one for now. The prime butterflies are regenerated rather than renamed, so the generator gained an fcma architecture. Its target feature list needs more than the single detected feature, so that is now a separate field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both scalar types now do a complex multiply as two fcmla against a zero accumulator, instead of the shuffle-negate-multiply-reverse-fma sequence. Measured on an M1 with the new bench_compare_neon_fcma bench, at sizes 4 to 131072. Everything from 16 up gets faster, and below that the two are equal: f32: 0.94 to 0.76 of the Neon time, best around 4096 f64: 0.92 to 0.84 of the Neon time, fairly flat above 256 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A rotation by 90 degrees is a complex multiplication by i, so an FCMA instruction does the rotation and the accumulation into the next add or subtract at once. Rotation90 now holds the factor to multiply by instead of a sign mask, and column_butterfly4 uses the fused version for its second pair of size-2 FFTs. On top of the complex multiplication change, still on an M1: f32: 0.73 to 0.82 of the Neon time from size 256 and up f64: 0.77 to 0.82 of the Neon time from size 256 and up Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hand written butterflies rotate and then add or subtract in several places, which the FCMA instructions do in one go. Butterfly 4 and 5 use the fused version directly, and butterfly 8 now calls the 45 and 135 degree rotation helpers instead of open coding them, which makes its root2 constants redundant. The new test compares every planned length up to 256, plus a handful of larger ones, against the Neon backend in both directions. The fused instructions round differently, so it compares within a tolerance. The observed difference is around 4e-17 for f64 and 1e-8 for f32, well under the limits used. Measured on an M1: f32: 0.69 to 0.83 of the Neon time from size 16 and up f64: 0.52 at the smallest sizes, settling around 0.70 to 0.74 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The generator had a todo for this: the rotation on the subtraction half of each
size-2 butterfly only feeds a multiply by a real twiddle, and FCMA does the
rotation and that multiply-accumulate in one instruction. So the standalone
rotations are gone, and the imaginary half of each sum is a chain of fcmla.
The generator takes a has_fused_rotate flag for this, which only the fcma
architecture sets, so the sse, neon and wasm_simd files are unchanged. The
vector trait lost mul, nmadd and apply_rotate90, which nothing in the backend
calls any more.
Prime lengths on an M1, as a fraction of the Neon time, before and after:
len f32 f64
7 0.998 -> 0.911 0.765 -> 0.667
11 0.985 -> 0.929 0.816 -> 0.734
13 0.979 -> 0.950 0.831 -> 0.780
23 0.856 -> 0.799 0.938 -> 0.766
29 0.977 -> 0.926 0.900 -> 0.658
31 0.988 -> 0.955 0.894 -> 0.817
961 0.989 -> 0.886 0.951 -> 0.905
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Butterfly 3 rotates the difference of its two inputs and then multiplies it by
a real twiddle twice, once with each sign. Both are single FCMA instructions on
the unrotated difference, one rotating each way, so the standalone rotation and
the negated copy of the twiddle both go away.
Power of three lengths on an M1, as a fraction of the Neon time:
len f32 f64
9 0.911 -> 0.884 0.665 -> 0.621
24 0.867 -> 0.837 0.685 -> 0.609
81 0.925 -> 0.936 0.923 -> 0.888
243 0.955 -> 0.901 0.950 -> 0.918
729 0.961 -> 0.948 0.967 -> 0.936
The single value f32 path is untouched, it packs the sum and the rotated
difference into one vector and does not fit the same rewrite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The job added with the backend was a compile check, based on a wrong assumption of mine that no available runner implements ARMv8.3-A. Both the Linux arm64 runners and the macOS ones do, so drop the cross and QEMU detour and just run the tests. There is one job per target because fcma is in the default target features on aarch64-apple-darwin but not on aarch64-unknown-linux-gnu, and the two detect it in different ways. The tests assert that the feature was detected, so they fail rather than quietly passing if a runner ever stops supporting it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
That was fast! Thanks so much for taking this on. Like I said in the other thread I'll take a look after this weekend. |
|
There wasn't really that much to do, just copying the files and applying the existing FCMA changes to them. Perfect work for Claude, the whole thing took like 15 minutes :) |
ejmahler
left a comment
ejmahler
left a comment
There was a problem hiding this comment.
Looks great. I did a diff between the neon and fcma directories to see what was different, and noticed lots of good optimizations etc. One thing I was thinking about is that our rotate45, rotate135 etc methods that butterfly8 uses probably aren't much faster than a regular complex multiplication with a twiddle anymore. Both are just 2 instructions, excluding load/broadcast/etc. The rotate methods do involve less memory traffic because they don't have to load a twiddle factor vector though, which is worth something.
There was only one thing I noticed, once we make a decision on that we're ready to go.
| @@ -34,7 +34,7 @@ pub const fn prime_butterfly_lens() -> &'static [usize] { | |||
| } | |||
|
|
|||
| /// Safety: The current machine must support the {{arch.cpu_feature_name}} target feature | |||
There was a problem hiding this comment.
Should this also be updated to reference the full list of instruction sets, like the lines down below were?
There was a problem hiding this comment.
Good catch, updated those and the two in new() as well.
|
I posted a comment on the stabilization issue for fcma and it's showing signs of progress. We may not have to wait long. |
Interesting idea, so I tried it. I swapped the rotate_45/135/225 helpers in the fcma backend for plain twiddle multiplies and benchmarked on an M1. As you guessed it doesn't help. Most lengths came out the same, and a few got slightly slower. Nothing got faster, so better to not do this. |
2 participants
Picks up #144 the way you suggested: a copy of the neon folder renamed to fcma, behind a nightly-only feature flag, rather than waiting for the intrinsics to stabilize. Built fresh off current master, since the old branch predates a lot of the neon code.
fcmacargo feature, off by default, needs nightly forstdarch_neon_fcmasrc/fcmaissrc/neonwith the complex arithmetic swapped for FCMA instructionsFftPlannerFcma, detected withis_aarch64_feature_detected!("fcma")and picked ahead of Neonfcmaarchitecture, and thehas_fused_rotateflag finally does the// todo: when we get FCMAin theretests/fcma_matches_neon.rscompares both planners for every length up to 256 plus larger onesbenches/bench_compare_neon_fcma.rsfor the numbers belowOn an M1, as a fraction of the Neon time: 0.69 to 0.80 for f32 and 0.69 to 0.74 for f64 at powers of two from 256 up, and down to 0.52 for small f64 sizes. Prime and power-of-three lengths gain less, 0.89 to 0.97, so there is more to do there.
The rotations matter more than the complex multiply. Swapping just
mul_complexwas worth about 0.80, folding the 90 degree rotations into the surrounding add or subtract got the rest.The tests run on the
ubuntu-24.04-armandmacos-latestrunners, both of which execute the instructions natively. Two jobs becausefcmais a default target feature onaarch64-apple-darwinbut not onaarch64-unknown-linux-gnu.