Repository navigation
Mark the DecimalType::is_valid_decimal_precision implementations #[inline] #12326
Description
Is your feature request related to a problem or challenge?
DecimalType::is_valid_decimal_precision is a bounds check that callers run once per value. Its implementations for Decimal32Type, Decimal64Type, Decimal128Type, and Decimal256Type are not #[inline] (types.rs L1493, L1525, L1557, L1589). They are not generic, so a caller in another crate depends on LTO to inline them. The arrow-data functions they wrap, such as is_validate_decimal_precision, are already #[inline] (decimal.rs L1031).
In DataFusion Comet the call stays out of line in release builds. Comet's release profile asks for lto = "thin", but Cargo builds libcomet without LTO because native/core is both a cdylib and an rlib (details below). The grouped decimal SUM and AVG accumulators call <Decimal128Type as DecimalType>::is_valid_decimal_precision on every row, both in the published 1.2.0 snapshot builds of libcomet for linux/amd64 and linux/aarch64, and in a macOS aarch64 release build. From SumDecimalGroupsAccumulator::update_single in linux/amd64:
callq *0x4dab27c(%rip) # 0x6d5e9a0, a GOT slot holding <Decimal128Type as DecimalType>::is_valid_decimal_precision
In a TPC-H Q1 profile that function took 11.6% of CPU samples. In a criterion bench, where the call is otherwise inlined, forcing it out of line made those accumulators 10% to 37% slower (apache/datafusion-comet#7660). Comet now looks up the bounds itself (apache/datafusion-comet#7674).
arrow-cast makes the same cross-crate call per value in its decimal casts (cast/decimal.rs L269, L354, L761, L998, L1033, parse.rs L943). The arrow-rs workspace does not set lto, so arrow's own cast_kernels benchmarks show the cost. Adding #[inline] makes eight decimal casts 5% to 37% faster. It also makes three 8% to 88% slower, through LLVM decisions elsewhere in those kernels (measurements below).
Describe the solution you'd like
Add #[inline] to the four is_valid_decimal_precision implementations, and keep the three kernels listed below from regressing in the same change.
Describe alternatives you've considered
A caller can look up MIN_DECIMAL128_FOR_EACH_PRECISION and MAX_DECIMAL128_FOR_EACH_PRECISION once and compare against them, as Comet now does. That works, but every caller repeats the check.
Additional context
Out-of-line calls in arrow-cast
Built from eb62e55 on macOS aarch64 (Apple M5 Max) with rustc 1.99.1 and the default bench profile, which has no LTO:
cargo bench -p arrow --features test_utils --bench cast_kernels --no-run
objdump -d of that binary finds 262 bl instructions to the four is_valid_decimal_precision symbols, in 124 arrow-cast kernel instantiations: apply_decimal_cast (decimal to decimal), cast_floating_point_to_decimal, cast_integer_to_decimal, and parse_decimal_checked. Each symbol exists once, as a global, so rustc's automatic cross-crate inlining of small functions did not apply to them. This is the per-row loop of unary_opt for cast_integer_to_decimal::<Int64Type, Decimal128Type, i128> with safe: true:
mov x0, x25
mov x1, x24
ldr w2, [sp, #0x14] // precision, reloaded from the stack on every row
bl <Decimal128Type as DecimalType>::is_valid_decimal_precision
tbnz w0, #0x0, ...
The callee is 21 instructions: a precision check, two table loads, and two 128-bit compares. With #[inline] on the four implementations, the binary has no calls to them and no out-of-line copies.
cast_kernels with #[inline]
Criterion medians with --warm-up-time 2 --measurement-time 4. The two builds ran in interleaved rounds (main, #[inline], main, #[inline]), and the change compares the means of the two rounds.
| Benchmark | Path | main |
#[inline] |
Change | Round 1 / 2 |
|---|---|---|---|---|---|
cast int64 within 1e6 late overflow to decimal128(18, 2) 512 |
int → Decimal128 |
0.96 µs | 0.69 µs | -28% | -28% / -29% |
cast int32 to decimal32(8, 0) 8192 precision overflow safe |
int → Decimal32 |
10.3 µs | 9.86 µs | -5% | -4% / -5% |
cast decimal128 to decimal128 512 lower precision |
Decimal128 → Decimal128 |
18.1 µs | 15.2 µs | -16% | -16% / -16% |
cast float64 to decimal128(32, 3) |
float → Decimal128 |
16.9 µs | 14.9 µs | -12% | -11% / -13% |
cast float64 to decimal128(20, 3) 8192 valid strict |
float → Decimal128 |
16.9 µs | 15.9 µs | -6% | -3% / -8% |
cast float32 to decimal32(7, 2) 8192 valid safe |
float → Decimal32 |
8.58 µs | 5.80 µs | -32% | -23% / -41% |
cast float32 to decimal32(7, 2) 8192 valid strict |
float → Decimal32 |
9.40 µs | 6.70 µs | -29% | -27% / -31% |
cast float64 to decimal64(3, 0) 8192 precision overflow safe |
float → Decimal64 |
14.4 µs | 9.15 µs | -37% | -38% / -35% |
cast float64 to decimal256(40, -2) 8192 scale down safe |
float → Decimal256 |
171.2 µs | 173.0 µs | +1% | 0% / +2% |
cast decimal256 to decimal128 512 |
Decimal256 → Decimal128 |
47.7 µs | 89.7 µs | +88% | +90% / +86% |
cast decimal256 to decimal256 512 |
Decimal256 → Decimal256 |
39.1 µs | 42.3 µs | +8% | +8% / +9% |
cast int64 to decimal32(9, 0) 512 |
int → Decimal32 |
0.68 µs | 0.84 µs | +24% | +26% / +21% |
The other 23 decimal benchmarks moved by -3% to +2%, except two whose kernels contain the check but rarely reach it: cast int64 to decimal32(9, -1) 512 (-7%) and cast invalid float64 to to decimal128(32, 3) (-9%).
All 35 decimal benchmarks
| Benchmark | main |
#[inline] |
Change | Round 1 / 2 |
|---|---|---|---|---|
cast int64 to decimal32(9, 0) 512 |
0.68 µs | 0.84 µs | +24% | +26% / +21% |
cast int64 to decimal32(9, -1) 512 |
0.69 µs | 0.64 µs | -7% | -4% / -10% |
cast int64 to decimal128(38, 10) 512 |
0.28 µs | 0.28 µs | 0% | 0% / 0% |
cast int64 within 1e6 to decimal128(18, 2) 512 |
0.31 µs | 0.30 µs | -2% | -2% / -1% |
cast int64 within 1e6 late overflow to decimal128(18, 2) 512 |
0.96 µs | 0.69 µs | -28% | -28% / -29% |
cast decimal128 to decimal128 512 |
2.71 µs | 2.70 µs | 0% | 0% / 0% |
cast decimal128 to decimal128 512 lower precision |
18.1 µs | 15.2 µs | -16% | -16% / -16% |
cast decimal128 to decimal256 512 |
11.0 µs | 11.1 µs | 0% | -1% / +2% |
cast decimal256 to decimal128 512 |
47.7 µs | 89.7 µs | +88% | +90% / +86% |
cast decimal256 to decimal256 512 |
39.1 µs | 42.3 µs | +8% | +8% / +9% |
cast decimal128 to decimal128 512 with same scale |
0.02 µs | 0.02 µs | -3% | -2% / -3% |
cast decimal128 to decimal128 512 with lower scale (infallible) |
19.1 µs | 18.5 µs | -3% | -4% / -3% |
cast decimal256 to decimal256 512 with same scale |
0.02 µs | 0.02 µs | -1% | -3% / +1% |
cast string to decimal128(38, 3) |
50.8 µs | 50.9 µs | 0% | -2% / +2% |
cast string to decimal256(76, 3) |
77.5 µs | 75.3 µs | -3% | -5% / -1% |
cast float64 to decimal128(32, 3) |
16.9 µs | 14.9 µs | -12% | -11% / -13% |
cast invalid float64 to to decimal128(32, 3) |
9.34 µs | 8.50 µs | -9% | -10% / -8% |
cast decimal128 to float64 |
12.8 µs | 12.5 µs | -3% | -3% / -3% |
cast decimal128 to int8 |
29.7 µs | 30.0 µs | +1% | +3% / -1% |
cast decimal128 to int64 |
24.5 µs | 24.2 µs | -1% | -3% / 0% |
cast decimal256 to float64 |
24.5 µs | 24.2 µs | -1% | -2% / 0% |
cast decimal256 to int64 |
31.7 µs | 32.2 µs | +1% | +3% / 0% |
cast int32 to decimal32(9, 0) 8192 valid safe |
0.88 µs | 0.86 µs | -3% | -3% / -2% |
cast int32 to decimal32(9, 0) 8192 valid strict |
0.88 µs | 0.86 µs | -3% | -3% / -2% |
cast int64 to decimal128(38, 2) 8192 valid safe |
3.08 µs | 3.16 µs | +2% | +3% / +2% |
cast int64 to decimal128(38, 2) 8192 valid strict |
3.22 µs | 3.18 µs | -1% | -1% / -1% |
cast uint64 to decimal256(76, -1) 8192 scale down safe |
4.07 µs | 4.04 µs | -1% | +3% / -4% |
cast int64 to decimal32(9, -20) 8192 all zero safe |
0.43 µs | 0.44 µs | +1% | 0% / +1% |
cast int32 to decimal32(8, 0) 8192 precision overflow safe |
10.3 µs | 9.86 µs | -5% | -4% / -5% |
cast int64 to decimal32(9, -1) 8192 scale before narrowing safe |
5.17 µs | 5.24 µs | +1% | +3% / -1% |
cast float32 to decimal32(7, 2) 8192 valid safe |
8.58 µs | 5.80 µs | -32% | -23% / -41% |
cast float32 to decimal32(7, 2) 8192 valid strict |
9.40 µs | 6.70 µs | -29% | -27% / -31% |
cast float64 to decimal128(20, 3) 8192 valid strict |
16.9 µs | 15.9 µs | -6% | -3% / -8% |
cast float64 to decimal256(40, -2) 8192 scale down safe |
171.2 µs | 173.0 µs | +1% | 0% / +2% |
cast float64 to decimal64(3, 0) 8192 precision overflow safe |
14.4 µs | 9.15 µs | -37% | -38% / -35% |
The disassembly shows where the three regressions come from:
cast decimal256 to decimal128 512: the per-row closure is out of line in both builds. Once the check is inlined into it, LLVM compiles the rounding step inmake_downscaler(d + 1whenr >= half) as a branch on the remainder instead of acsel. The bench values are random, so that branch goes either way for about half the rows, which likely explains the slowdown.cast decimal256 to decimal256 512andcast int64 to decimal32(9, 0) 512: onmain, LLVM inlines the per-row closure into theunary_optloop. With the check inlined into the closure, it stops doing that, so every row calls the closure out of line instead. In theint64case the closure also contains theformat!of themul_checkedoverflow error, which.ok()then discards.
Why Comet's release build has no LTO
Comet's [profile.release] sets lto = "thin" (native/Cargo.toml L93-L97), and native/core declares crate-type = ["cdylib", "rlib"] (native/core/Cargo.toml L161). Cargo runs LTO for a unit only when all of its crate types support it, and rlib does not (lto.rs L113-L115). With Cargo 1.99.1, a minimal crate with that profile and those crate types builds its library without -C lto=thin. With crate-type = ["cdylib"] it gets -C lto=thin. Its bench executable gets -C lto=thin either way, which is why Comet's criterion benches inline the call.
DataFusion TPC-H Q1
DataFusion's own Q1 does not reach this check. Its SUM and AVG accumulators do not validate precision per row, and the arrow kernels Q1 uses do not call it. DataFusion main (6fb291312) built with fat LTO against arrow-rs eb62e55, with and without the change, ran Q1 at SF10 in the same time: 2290 ms vs 2285 ms with one partition, and 249 ms vs 250 ms with 18 partitions over 8 ABBA-ordered pairs.
validate_decimal_precision
The four validate_decimal_precision implementations next to them are not #[inline] either. arrow-cast calls them per value in the safe: false integer-to-decimal and decimal-to-decimal casts, from 190 call sites in the same binary. I have not measured those.
The arrow-rs disassembly and benchmarks were collected with Claude Code.
Activity
docs: link the arrow-rs issue for the missing `#[inline]`
perf(arrow-cast): inline `is_valid_decimal_precision` in decimal casts
perf(arrow-cast): inline `is_valid_decimal_precision` in decimal casts (