Answer Mod without computing the quotient - #115
Merged
Conversation
x % y opened with y.IsZero, x.IsZero, y.IsOne and then CompareTo. On a machine with AVX-512 that is three 32-byte loads, two vptest, a vpcmpuq/kortestb pair and the vzeroupper they drag in, before any arithmetic happens. Zero and one are one test on the divisor: (y0 >> 1) | y1 | y2 | y3 is zero for both and for nothing else, and y0 tells them apart on the cold path. x == 0 needs no test at all, because y >= 2 by then puts it in the x < y arm. CompareTo cost more than the ordering it was asked for: the JIT compared every limb twice, once for != and once for <, materialised -1/0/1 and then re-tested that against -1 and 0. A descending compare that jumps straight to its answer only pays the second compare on the limb where the two values are equal, and the single-limb dividend check falls out of the limbs already loaded. Mod is 190 bytes of code where it was 266. Measured against the same operands on a Ryzen 9 9950X under Tier-1 with dynamic PGO, both orientations of the A/B run so the slot bias cancels: x < y 0.82, x == y 0.81, one-limb 0.86, power-of-two divisors 0.92-0.94, the benchmark key/value matrix 0.96, a mixed corpus 0.97, wide divisors 0.98-1.00. Geomean 0.947 over twenty shapes, against an A/A control that reads 0.999.
ModFull dispatched on the divisor width to find a power of two, then called DivideImpl, which dispatched on the divisor width a second time through another non-inlined call, and then asked for a full division and threw the quotient away. Mod now dispatches once and calls a kernel that never forms the quotient at all. Keeping the kernels in general registers matters more than the quotient does. The old 256- and 192-bit paths normalised and denormalised through ShiftLeftSmall and ShiftRightSmall, which take and return a UInt256: the 192-bit path assembled its remainder with four 8-byte stores and then read it back with one 32-byte load, a width store-to-load forwarding does not cover, and the 256-bit path did the same at both ends plus a spare 32-byte stack-to-stack copy of the normalised divisor and four vmovq crossings for the shift counts. The kernels do the funnel shifts in general registers and store the result once. A 128-bit divisor needs no multi-limb subtract at all: the 2-by-1 divide already leaves the top of the partial remainder in rhat, so each step is one divide, one product and a two-limb subtract. Normalisation keeps its shift == 0 branch. Folding it away with the (a >> 1) >> (63 - s) funnel, which is what the signed shift uses, costs one extra shift per limb and measured 1.5% worse on both 192-bit shapes even though the branch goes both ways half the time - lzcnt resolves it long before the divides, so the mispredict has nothing to discard. The 256-bit kernel needs no branch either way: ModFull answers y >= 2^255 with a subtract, so its shift is always at least 1. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations so the slot bias cancels, against an A/A control of 0.999: 4x4 0.79, 3x3 0.90, 2x1 0.92, mixed corpus 0.93, key/value matrix 0.93, 4x2 and 2x2 0.94, 4x1 0.96, 4x3 0.97. Geomean 0.962; the shapes that return before reaching a kernel are unchanged. 587,686 tests pass, and 400,000 adversarial dividend/divisor pairs - every bit boundary, every normalising shift, and quotients placed to fire the add-back - agree with BigInteger.
The 192- and 128-bit kernels always ran their full digit count - two and three - however few limbs the dividend actually filled. A dividend that stops short of limb 3 leaves the leading window below the normalised divisor, so that digit is zero: the divide runs, the correction runs, the subtract takes nothing off, and the remainder comes out unchanged. Testing x's top limbs before normalising skips those steps. Same-width operands are the case this rescues, and they are the common one. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations: 3x3 0.66 (16.2 -> 10.7 ns), 2x2 0.66 (10.2 -> 6.7), 3x2 0.93, a mixed corpus 0.91, the benchmark key/value matrix 0.91. Dividends that do fill four limbs pay one predicted test: 4x4 0.998, 4x3 1.004, 4x2 1.001. 587,686 tests pass and 400,000 adversarial pairs agree with BigInteger.
Every digit multiplied the quotient estimate by all of the divisor and subtracted the result from the whole window, including the top limb the divide had just accounted for. The 2-by-1 divide leaves (u[j+n]:u[j+n-1]) - qhat*v[n-1] in rhat by construction, so the window still to reduce is (rhat:u[j+n-2]..u[j]) and only qhat times the low limbs is left to take off it - the same collapse the 128-bit path already used, one limb up. The D3 correction was already carrying qhat*v[n-2], which is exactly that subtrahend's top limb, so keeping it costs nothing and removes a second multiply. A 192-bit divisor goes from four 64x64 products a digit to two, a 256-bit divisor from five to three, and the subtract loses a limb. A correction that pushes rhat past 2^64 supplies the limb the borrow lands in, so the add-back now fires on a borrow out of a zero one rather than on the borrow alone. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations: 4x3 0.84 (19.5 -> 16.3 ns), 3x3 0.84, 4x4 0.86 (11.9 -> 10.3), a mixed corpus 0.94, the key/value matrix 0.92; the 128- and 64-bit paths are untouched at 1.00. 587,686 tests pass and 400,000 adversarial pairs agree with BigInteger.
Int256.Mod took absolute values into two locals before doing anything else, so every call copied both operands - 32 bytes each - and then handed Mod the copies. Neither copy is needed when both operands are already non-negative, which is the common case, and reading the caller's own values also keeps Mod off a load that has to wait on the store that just wrote them. Splitting the sign handling into a fast arm rather than four arms matters: giving each sign combination its own call site grew the method from 260 to 538 bytes for no gain, while the two-arm form lands at 443 and wins. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations, three repeats: 4x4 0.870 / 0.889 / 0.883, 2x2 0.882 / 0.979 / 0.887, y == 1 0.74 / 0.91 / 0.76, one-limb operands 1.01 (unchanged). 587,686 tests pass.
The single-limb kernel skips a divide step whose limb and running remainder are both zero, but only on the arm that uses hardware div; the reciprocal arm, which is what runs without X86Base.X64 and on ARM64, ran all four unconditionally. A two-limb dividend paid four reciprocal divides where the old code paid two. With DOTNET_EnableHWIntrinsic=0 on a Ryzen 9 9950X, 2x1 goes from 1.222 against main to 0.916. The hardware arm is unchanged (1.006 over the single-limb shapes, inside the noise of an A/A control).
MulMod against a single-limb modulus normalised it, shifted a copy of each operand up to match, reduced, and shifted the remainder back - all so the reciprocal 2-by-1 divide would see a divisor with its top bit set. Hardware div has no such requirement: the running remainder starts at zero and stays below the divisor, so it goes straight into rdx and the modulus goes in as it stands. The shifted copy also went through a UInt256, which is four 8-byte stores followed by one 32-byte load. The final reduction needed one divide, not two. Both factors are already below the modulus, so their product is below mod^2 and its upper limb below mod, which is exactly the condition a single div wants. The reciprocal arm still needs the normalised divisor, and it must be shared: deriving it inside the reduction helper instead of once per call cost 1.27 with DOTNET_EnableHWIntrinsic=0, so the two arms are now separate helpers and MulMod passes its own normalisation to the second. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations. Hardware div: 2x1 0.63, 3x1 0.75, 1x1 0.79, 4x1 0.84, mixed corpus 0.95, key/value matrix 0.95, wide moduli unchanged at 0.98-1.00. Without intrinsics: 2x1 0.82, mixed 0.98, and 4x1 at 1.02 - a four-limb dividend never takes the zero-limb skip, so there it only pays for the tests. 587,686 tests pass in each of the default, no-AVX-512, no-AVX2 and no-intrinsics configurations.
Mod tested the divisor for zero and one, compared it against the dividend, and then tail-called ModFull, which read every divisor limb again to pick a kernel and called that. Doing the dispatch where the limbs are already in registers removes a call frame and a round of loads, and it lets the JIT tail-jump straight from Mod into the kernel: one call and one jump where there used to be two calls, two returns and three reads of the divisor. Power-of-two divisors gain most, because masking three limbs was never worth a second call. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations: power-of-two divisors 0.62-0.83, 2x2 0.83, 2x1 0.86, 3x2 0.90, 3x3 0.91, key/value matrix 0.91, mixed corpus 0.93, 4x3 and 4x2 0.93, 4x4 0.97. Geomean 0.887. Shapes that return before the dispatch read 0.99-1.05 across repeats - Mod is 611 bytes of code where it was 190, and they are the ones that pay for it. WideDivideModBenchmark bound ModFull by reflection; it now calls the public entry point, which is the path.
A divisor at or above 2^255 makes the quotient 1, so the remainder is x - y. That went through an inline borrow chain: four sub/cmp/setb/movzx groups and four 8-byte stores, all in general registers. The general subtract loads both operands as vectors, subtracts, and stores the result as a vector without touching a general register at all, and that is worth more than the call it costs. Only where there is a vector unit, though. Fixing the inline chain instead - the two-cycle borrow helper and a 32-byte store - measured 1.068, because assembling that store from general registers costs more than the four limb stores it replaces. So the choice is on Vector128.IsHardwareAccelerated, and a build with no vector unit keeps the chain it had. Ryzen 9 9950X, Tier-1 with dynamic PGO, both A/B orientations, on the shape whose divisor is above 2^255: 0.501 with AVX-512, 0.496 with AVX2 only, 0.727 with SSE alone - which is the same Vector128 path ARM64 takes - and 0.995 with DOTNET_EnableHWIntrinsic=0. The mixed corpus picks up 0.98-0.99 everywhere. 587,686 tests pass in the default, no-AVX2 and no-intrinsics configurations.
Folding it in measured 0.887 on a Ryzen 9 9950X, but on a Neoverse N2 it bought the wide shapes nothing - 0.822 against 0.823 at four-by-four, and the same at every other width - and gave every call the prologue the merged body needs. Mod went from three saved registers to five on x64 and from a 64-byte frame with three on ARM64 to a 96-byte frame with six, which the calls that answer from the entry compare pay in full: a one-limb x % y went from 3.52 to 4.14-4.66 ns across two runs, and a dividend below its divisor from 2.79 to 3.60-3.93. So the dispatch goes back behind a call and Mod keeps its lazy limb reads. That costs the x64 geomean 0.746 -> 0.834, all of it on the power-of-two divisors and the wide shapes, and leaves no shape slower than main on either platform. The x64 gain is real and available if the tradeoff is ever worth making; it is recorded here rather than taken.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The changes substantially rewrite core division/modulo internals in a performance- and correctness-critical numeric type, so they warrant final human review despite no concrete issues found in the diff.
Pull request overview
This PR optimizes UInt256.Mod by computing remainders directly (without constructing/storing the quotient), reducing normalization overhead, and tightening fast paths; it also updates Int256.Mod to avoid unnecessary operand copies and adds a benchmark to measure remainder performance by operand “shape”.
Changes:
- Reworked
UInt256.Mod/ModFullto dispatch once by divisor width and run dedicated remainder-only kernels for 256/192/128/64-bit divisor widths (plus power-of-two and high-divisor shortcuts). - Optimized single-limb modulus flows (notably in
MulModBy64Bits) to avoid normalization when hardware division is available and reduce the number of divides in final reduction. - Simplified
Int256.Modto skip absolute-value copies when both operands are already non-negative, and addedModShapesBenchfor cross-ISA remainder shape benchmarking.
File summaries
| File | Description |
|---|---|
| src/Nethermind.Int256/UInt256.DivideMod.cs | Implements remainder-only kernels and streamlined fast paths to speed up UInt256.Mod and related modular ops without computing/storing the quotient. |
| src/Nethermind.Int256/Int256.cs | Avoids unconditional operand copying in Int256.Mod, taking absolute values only when required and preserving dividend-sign semantics. |
| src/Nethermind.Int256.Benchmark/ModShapesBench.cs | Adds a new BenchmarkDotNet benchmark to measure % performance across operand-width “shapes” and key fast-path scenarios. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
LukaszRozmej
approved these changes
Sep 3, 2026
LukaszRozmej
left a comment
LukaszRozmej
left a comment
Member
There was a problem hiding this comment.
not sure if faster in zkvm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Results
x % yby operand shape, againstmain. Lower PR/main is better: 0.62 means 38% less time.x64 is a Ryzen 9 9950X (Zen 5, AVX-512) under Tier-1 with dynamic PGO, median of three runs;
ARM64 is the
ubuntu-24.04-armrunner (Neoverse N2) throughbenchmark.yml, andModShapesBenchin this PR is what produces that column. The shape is(dividend limbs) x (divisor limbs), which is what decides how many Knuth digits the division
needs.
Geomean over the twenty shapes, by ISA configuration:
DOTNET_EnableHWIntrinsic=0Everything that calls
Modmoves with it, and its neighbours do not:Int256.ModSubtractModAddModMultiplyModDivideInt256.DivideA single-limb divisor is four dependent hardware divides with nothing around them to remove, so
those shapes sit at parity. 3x1 is 1% slower on x64 (7.19 -> 7.27 ns), the only shape anywhere
above the noise band on the wrong side.
What is in it
Modused to be a thin wrapper over the divide. It tested the divisor for zero and one withtwo 256-bit vector compares, asked
CompareTofor a three-way answer, then calledModFull,which dispatched on the divisor width to find a power of two, called
DivideImpl, whichdispatched on the divisor width a second time, and finally asked for a full division and threw
the quotient away.
nothing downstream reads it, so it now stays in a register instead of being assembled into a
UInt256and stored. The width dispatch happens once.ShiftLeftSmallandShiftRightSmall, which take and return aUInt256, meant four 8-byte stores followed by one32-byte load, a width store-to-load forwarding does not cover, plus two
vmovqcrossings forthe shift count. The 192-bit path did that to build its remainder and the 256-bit path did it
at both ends, with a spare 32-byte stack-to-stack copy of the normalised divisor in between.
accounted for. The 2-by-1 divide leaves
(u[j+n]:u[j+n-1]) - qhat*v[n-1]inrhatbyconstruction, so the window still to reduce is
(rhat:u[j+n-2]..u[j])and onlyqhattimesthe low limbs comes off it. D3 was already carrying
qhat*v[n-2], which is that subtrahend'stop limb. A 192-bit divisor went from four 64x64 products a digit to two, a 256-bit divisor
from five to three, and the subtract lost a limb. It is the collapse the 128-bit path already
used, one limb up.
maximum. A dividend that stops short of the top limb leaves the leading window below the
normalised divisor, so that digit is a divide that can only produce zero. This is what rescues
the same-width shapes, and they are the common ones.
(y0 >> 1) | y1 | y2 | y3is zero for both andnothing else.
x == 0needs no test at all, becausey >= 2by then puts it in thex < yarm. The compare jumps straight to its answer rather than materialising -1/0/1 and re-testing
it, and only pays a second compare on the limb where the two values are equal.
x - y. That is nowthe general subtract, which loads both operands as vectors and stores the result as one,
without touching a general register - worth more than the call it costs, but only where there
is a vector unit, so the choice is on
Vector128.IsHardwareAccelerated.MulModby a single-limb modulus normalised for a divide that needs none. Hardware divonly requires the running remainder to stay below the divisor, which it does from zero onward.
Its final reduction also needed one divide rather than two: both factors are already below the
modulus, so the product's upper limb is too.
Int256.Modtook absolute values into two locals first, copying both operands even whenneither was negative.
Not taken
Moditself measured 0.887 on x64 and 1.043 on ARM64, theone place the platforms disagreed. On ARM it bought the wide shapes nothing - 0.822 against
0.823 at 4x4, the same at every width - while giving every call the merged body's prologue:
three saved registers and a 64-byte frame became six and 96 bytes. The calls that answer from
the entry compare paid all of it: one-limb
x % ywent 3.52 -> 4.66 ns and a dividend belowits divisor 2.79 -> 3.93. Reverting it costs the x64 geomean 0.746 -> 0.823 and leaves nothing
slower than
mainanywhere. Worth revisiting only with a way to keepMod's prologue small.kernels showed the test fires about 1% of the time, so the branch predicts perfectly and costs
nothing, while the mask puts about five cycles on the critical path of every digit.
(a >> 1) >> (63 - shift)funnel lost to a plainshift == 0branch by 1.5% on both192-bit shapes, the opposite of the signed shift's result:
lzcntresolves the branch longbefore the divides, so a mispredict has nothing to discard.
& 63does not remove the JIT's ownand reg, 63before shlx/shrx.Validation
587,683 with
-p:EnableZkEvm=true.BigInteger: every bit boundary, everynormalising shift, and quotients placed to fire the add-back. Counters on the rare paths
confirm the corpus reaches them - the add-back 806 times,
rhatoverflowing a limb 9,613, thesaturating quotient digit 6 - so the
borrow & ~rcarrydecision that guards the add-back iscovered rather than assumed.
git diff --checkclean.