Skip to content

Improve SVE2 optimizations of SimdGetColSums - #899

Merged
ermig1979 merged 1 commit into
devfrom
cursor/sve2-col-sums-84e4
Aug 20, 2026
Merged

Improve SVE2 optimizations of SimdGetColSums#899
ermig1979 merged 1 commit into
devfrom
cursor/sve2-col-sums-84e4

Conversation

@ermig1979

@ermig1979 ermig1979 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

The SVE2 implementation of SimdGetColSums used svunpklo/svunpkhi plus a separate add, which is more work than NEON vaddw_u8 and was much slower.

This change:

  • Accumulates with SVE2 svaddwb_u16 / svaddwt_u16 (even/odd widening adds) after a single svld1 per vector
  • Keeps that even/odd layout in the 16-bit buffer and reconstructs natural column order with svzip1/svzip2 only when converting to 32-bit
  • Unrolls the inner loop across four vectors for more ILP
  • Applies the same kernel and 8/4-row unrolling to GetAbsDxColSums
  • Does not use structured svld4/svst4 loads/stores, and does not add macros

Release notes for 7.2.165 list the improvement.

The AArch64 object dump of the inner loop is ld1b + uaddwb/uaddwt (no ld4/st4). Unpack and zip appear only in the 16-to-32 conversion.

Correctness:

  • Scalar simulation of addwb/addwt + zip1/zip2 vs Base, for SVE lengths 128/256/512 bits and AutoTest-like shapes (42 cases)
  • Cross-compiled with aarch64-linux-gnu-g++ (-march=armv9-a+sve+sve2, scalable VL)
  • Test::GetColSumsAutoTest and GetAbsDxColSumsAutoTest under QEMU neoverse-n2 and cortex-a710 (SVE2, 128-bit): Base vs Sve2, including [16, 255], [17, 256], [31, 257]. All passed.
Open in Web Open in Cursor 

Replace unpack-and-add with SVE2 widening even/odd adds (svaddwb/svaddwt), keep that layout in the 16-bit buffer, and zip back to natural order only when converting to 32-bit. Unroll the inner loop across four vectors and apply the same approach to GetAbsDxColSums.

Co-authored-by: igor.ermolaev <igor.ermolaev@altezza.org>
@ermig1979
ermig1979 marked this pull request as ready for review August 20, 2026 13:27
@ermig1979
ermig1979 merged commit cb058e0 into dev Aug 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants