quant: AVX2 kernel for int3-g64 matmul (cherry-pick from #601) - #926
Merged
Conversation
matmul_i3 shipped with a NEON path only; on x86 every 3-bit weight went through the scalar bit-extraction tail. Measured on the Zen2 12-core box (MiniMax-M3 int3-g64 routed experts): routed-CPU effective bandwidth 3.7-3.9 GB/s vs 27 GB/s for the int4 IDOT path — expert-matmul at 88% of decode time, 0.95 tok/s where int4 does 2.8. Low plane reuses matmul_i2's 2-bit unpack (shift/mask + unpacklo into 16 ordered bytes); high plane broadcasts the two mask bytes with pshufb, bit-tests against per-lane masks, and adds 4 where set; the [0,7] byte then drops to [-4,3] at the epi32 widen, two FMAs per 16 weights, one hsum per 64-input group. Ragged tails stay scalar. Verified against a double-precision scalar reference over S=1/S>1, multiple-of-64 and ragged shapes: max rel err 5.4e-6.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of 3ab405d from #601, as suggested in the closing discussion of #452: the fmt=5 int3 matmul had NEON and AVX-512 arms but fell to scalar on x86 CPUs without AVX-512 — which is most consumer hardware (Zen2/Zen3, recent Intel consumer chips).
Authorship preserved: the commit lands exactly as @steve-m wrote it (author field intact); the only conflict against current dev was the doc comment above
matmul_i3, where #661's AVX-512 description and this commit's AVX2 description both live now. Dispatch order after the merge: AVX-512 → NEON → AVX2 → scalar.Verified on an AVX2-only 12-core (no AVX-512, so the new arm is the one exercised, not skipped):
tests/test_int3— numeric check green, including the outlier-rows RMS comparisontests/test_int3_load— container inference green-march=native🤖 Generated with Claude Code