chunkers: add AVX-512 / VAES scan kernels

[ThomasWaldmann]

Adds the AVX-512 tier to the chunker scan kernels on x86-64, above the existing AVX2 / AES-NI paths. All kernels stay bit-identical (same cut points, same digests); dispatch is runtime-detected as before.

fastcdc, buzhash64

AVX-512 variants of the 8-lane candidate test: one 512-bit vector instead of two ymm halves, with vptestnmq fusing the AND and the ==0 test directly into a mask register. Only the test narrows — the scalar per-byte work (table loads, prefix tree) is shared with the other kernels and unaffected, so the expected gain is modest (the test is not the bottleneck, especially for buzhash64, which is included mainly to keep the kernel dispatch symmetric).

rabin-aes, goldilocks-aes, toeplitz-aes

A VAES/AVX-512 variant of the shared x86-64 hardware path (kind "vaes"), where the win should be substantial: these chunkers encrypt one AES block per input byte, and VAES does 4 blocks per instruction. The new path processes groups of 32 positions as 8 zmm vectors:

• 4x fewer AES instructions than the 128-bit AES-NI path
• round keys stay register-resident in 11 zmm registers (the xmm path must re-load them every round due to register pressure)
• 8 independent dependency chains hide the vaesenc latency (a naive 2-zmm port would be latency-bound)
• vpexpandq places 4 digests into the block-low qwords in one instruction; a masked vptestnmq tests the 4 ciphertext qwords without any extracts

The VAES path requires GCC >= 11 / clang >= 14 (__builtin_cpu_supports("vaes")); older compilers keep the AES-NI path. Zen 3 (VEX-256 VAES without AVX-512) is not covered and falls back to AES-NI; a 256-bit VAES middle path could be a follow-up if that matters.

Kernel selection for testers

Every kernel level can be selected via env vars, so all pairs can be benchmarked against each other on one machine. The active kernel is reported by the chunker's .kernel property.

fastcdc (buzhash64 works the same with BORG_BUZHASH64_*):

setting	kernel
(default)	best available: avx512 → avx2 → blocked
BORG_FASTCDC_NO_AVX512=1	avx2
BORG_FASTCDC_NO_AVX2=1	blocked
BORG_FASTCDC_FORCE_SCALAR=1	scalar (sequential reference loop)

The NO_* vars are read once per process (like the CPU detection), so set them before the first chunker use; FORCE_SCALAR is per chunker instance.

AES chunkers (all three):

setting	kernel
(default)	best available: vaes → aes-ni (aes-arm64 on arm64) → evp
BORG_PHTE_NO_VAES=1	aes-ni
BORG_RABIN_AES_FORCE_EVP=1 (analogous BORG_GOLDILOCKS_AES_FORCE_EVP, BORG_TOEPLITZ_AES_FORCE_EVP)	evp (OpenSSL batch path)

These are read at chunker creation time.

---

Benchmarks on real AVX-512 hardware

Measured on an AMD Ryzen 5 8500GE (Zen 4/4c: avx512f/dq/bw/vl/vbmi/vbmi2/vnni, vaes, gfni, vpclmulqdq), Debian 13, GCC 14.2, performance governor, pinned to one core. Note Zen 4 implements AVX-512 on a 256-bit datapath (double-pumped), so 512-bit ops have roughly the throughput of two 256-bit ops — width alone buys little here.

What the new tiers bought, as originally written

Isolated chunker benchmark, best of 5, MB/s:

chunker	scalar fallback	AVX2 / AES-NI	AVX-512 / VAES
fastcdc	1006 (blocked)	800	1173
buzhash64	879 (blocked)	1026	1052
toeplitz-aes	—	785 (aes-ni)	799
rabin-aes	—	754	808
goldilocks-aes	—	120	121

Two results contradict the expectations above and are worth recording:

• VAES bought almost nothing (+2% toeplitz, +7% rabin, 0% goldilocks), not the substantial win predicted. These chunkers are not AES-bound: cutting the AES round count from 10 to 2 — a 5x reduction in AES work — only moved toeplitz from 653 to 927 MB/s and goldilocks not at all. The limit is the serial rolling-hash chain, not the PRF.
• fastcdc's AVX2 kernel was slower than its own scalar fallback (800 vs 1006 MB/s). That is a pre-existing bug this PR inherited rather than introduced, but it means most x86 users (no AVX-512) were getting the slowest of the three paths.

Both are addressed by the follow-up commits below.

Follow-up commits

chunkers: build the aligned domain in the vector kernels — the vector kernels computed the per-block prefix work in scalar registers and handed it to the vector unit through the stack. Eight 8-byte stores immediately followed by one 32/64-byte vector load cannot use store-to-load forwarding, so every block paid a stall (confirmed by ls_bad_status2.stli_other, and by the tier ordering: AVX2 has two such loads per block, AVX-512 one, the scalar path none). Both kernels now do the table lookups for the next block while the current one is tested, and build the aligned domain in the vector domain (vpsllvq/vprolvq + valignq prefix scan on AVX-512; vpermq per 4-lane half on AVX2).

goldilocks-aes: keep the field reductions out of the branch predictor — gl_add/gl_mul reduced with plain if(), which GCC turned into 55 conditional jumps in the VAES kernel, missing ~27% of the time. A single gl_mul cost 15.1 cycles against 1.8 for the raw 64x64 multiply it is built on. Mask arithmetic and ternaries do not help — GCC converts both straight back to branches — but __builtin_*_overflow keeps the carry in the flags: 4.4 cycles, 5 conditional jumps.

chunkers: store the AES chunkers' digests pre-spread — the digests are stored with a zero in every odd slot instead of packed, so each AES input is a plain 512-bit load rather than a vpexpandq. (This replaces the vpexpandq bullet in the VAES description above.)

Results

borg benchmark cpu, 1 GB, both states built and measured back to back, pinned, 2 passes each:

chunker	this PR (first 2 commits)	with follow-ups	speedup
goldilocks-aes	8.336 / 8.788 s	2.094 / 2.405 s	3.98x
fastcdc	0.863 / 0.927 s	0.343 / 0.362 s	2.52x
buzhash64	0.888 / 0.912 s	0.668 / 0.669 s	1.33x
rabin-aes	1.254 / 1.259 s	1.108 / 1.167 s	1.13x
toeplitz-aes	1.130 / 1.252 s	1.088 / 1.155 s	~1.1x, see note
buzhash	0.557 / 0.660 s	0.554 / 0.593 s	not modified
fixed	0.056 s	0.056 s	not modified

Both passes are shown because this benchmark is noisy on this machine (untouched code varies by up to ~14% between passes). The first four rows have non-overlapping ranges. toeplitz-aes does not — a dedicated 3-pass A/B of that change measured 1.250 → 1.078 s and the isolated kernel +9.2%, so ~1.1x is the honest estimate.

Per kernel tier afterwards (isolated benchmark, MB/s):

chunker (tier)	before	after
fastcdc — avx512	1173	2689
fastcdc — avx2	800	2583
fastcdc — blocked	1006	1007
buzhash64 — avx512	1052	1509
buzhash64 — avx2	1026	1478
buzhash64 — blocked	879	931
toeplitz-aes — vaes	652	713
rabin-aes — vaes	649	691
goldilocks-aes — all paths	94	347

fastcdc is now the fastest CDC chunker in borg, ahead of legacy buzhash. Note that AVX-512 and AVX2 now perform nearly the same (2689 vs 2583, 1509 vs 1478): on Zen 4 the gains came from removing the stall and vectorizing the prefix work, not from vector width. A machine with a true 512-bit datapath (Intel SPR, Zen 5) should show a wider gap — numbers from such a box would be welcome.

Optimizations tried and rejected

Measured on the same machine, none committed:

• Widening the AES chunkers' rolling hash to 4 lanes at stride 4 — toeplitz 653 → 402, rabin 649 → 458 MB/s. Stride-4 doubles the table footprint (toeplitz 8→16 KiB, rabin 10→18 KiB against a 32 KiB L1d) while the lookups per position stay the same: +33% instructions and 7.8x the L1 misses. A microbenchmark of the rolling step alone caps the ceiling at 16% for rabin and 0% for toeplitz even with no table-pressure penalty.
• Hoisting the chain-independent table lookups into their own pass — 653 → 579 and 650 → 495 MB/s; the delta buffer's traffic costs more than the scheduling gain, i.e. the hardware was already hoisting them.
• A gather-based fastcdc kernel avoiding the stack round-trip entirely — 1102 vs 2778 MB/s; vpgatherqq is too slow on Zen 4.
• Deeper software pipelining (2 blocks ahead instead of 1) — regressed in every variant tried.

Verification

• full chunker test suite passes on arm64 (NEON / aes-arm64 / evp paths)
• cross-compiled clean for x86-64 (-Wall -Wextra); disassembly shows the intended code
• x86-64 build run under Rosetta 2 (has AES-NI, no AVX-512): dispatch correctly selects aes-ni, and cut points + digests are bit-identical to the native aes-arm64 run
• the AVX-512/VAES kernels have now been executed on real hardware (Ryzen 5 8500GE, see above): full chunker suite incl. the fuzz tests (BORG_TESTS_SLOW=1, 182 tests) passes, and the chunk-boundary digests of all six chunkers are identical across the avx512/avx2/blocked and vaes/aes-ni/evp tiers

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.62%. Comparing base (0d6fffb) to head (2020888).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10043      +/-   ##
==========================================
- Coverage   86.63%   86.62%   -0.01%     
==========================================
  Files          97       97              
  Lines       16912    16912              
  Branches     2550     2550              
==========================================
- Hits        14651    14650       -1     
- Misses       1570     1571       +1     
  Partials      691      691              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.