Repository navigation
Conversation
The grouped decimal SUM and AVG accumulators checked the running sum on every row with `Decimal128Type::is_valid_decimal_precision`. arrow-rs does not mark that method `#[inline]`, and release builds of libcomet call it out of line, with its precision guard and two table lookups. Look up the bounds for the sum's precision once per accumulator, and compare against them in update, merge, and SUM's evaluate. The values each path accepts are unchanged, which a new test pins against arrow's check. Add a `decimal_groups` bench that drives both accumulators through `update_batch`. The ungrouped AVG accumulator is left alone, since apache#7116 tracks giving it a wider running sum.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @comphead. Loading the bounds once per accumulator is the right first step, given that the fast path in your comment on #7660 didn't pass the bench gate.
| - Spark 3.4.3 (2026-05-26) | ||
| - Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3 | ||
| - Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only `QueryContext` import path differs. `YearMonthIntervalType` and `DayTimeIntervalType` inputs (supported by Spark) fall back to Spark in Comet. | ||
| - Performance (tuned 2026-11-17): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Bench builds already inline that call: 2% to 5% faster on most shapes, the 4-group shape within 1%. Benchmark: `benches/decimal_groups.rs`. |
There was a problem hiding this comment.
Could we drop the AVG percentages here? DataFusion's sum, which this PR doesn't touch, moved -1.8% and +0.5% in the same runs. The AVG shapes moved between +1.1% and -4.6%, which is within about twice that noise floor on a laptop with a load average of 4. The change this entry can state is the call that release builds no longer make. The other entries in this file also link their PR.
| - Performance (tuned 2026-11-17): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Bench builds already inline that call: 2% to 5% faster on most shapes, the 4-group shape within 1%. Benchmark: `benches/decimal_groups.rs`. | |
| - Performance (tuned 2026-11-17, PR [#7674](https://github.com/apache/datafusion-comet/pull/7674)): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Benchmark: `benches/decimal_groups.rs`. |
There was a problem hiding this comment.
Done. I dropped the AVG percentages and added the PR link, as you suggested.
|
|
||
| ## sum | ||
|
|
||
| - Performance (tuned 2026-11-17): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. 9% to 12% faster on most shapes even in bench builds, which inline that call. Benchmark: `benches/decimal_groups.rs`. |
There was a problem hiding this comment.
The 9% to 12% comes from bench builds on an Apple M-series laptop. Comet deploys on x86-64 servers, and Apple silicon results don't predict those. Could you rerun decimal_groups on an x86-64 machine before putting a number in the audit, or drop it as in the AVG entry? Here is the version without it, with the PR link the other entries carry:
| - Performance (tuned 2026-11-17): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. 9% to 12% faster on most shapes even in bench builds, which inline that call. Benchmark: `benches/decimal_groups.rs`. | |
| - Performance (tuned 2026-11-17, PR [#7674](https://github.com/apache/datafusion-comet/pull/7674)): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Benchmark: `benches/decimal_groups.rs`. |
There was a problem hiding this comment.
I don't have an x86-64 machine to rerun decimal_groups on, so I dropped the number and kept the call that release builds no longer make, with the PR link.
| // A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far | ||
| // from it. Values of 1e27 to 5e28 come within a few batches of it without overflowing. |
There was a problem hiding this comment.
This comment doesn't match what the near limit shape does. The groups are i % 1024 and the sign alternates on i % 2. Because 1024 is even, every row of a group has the same sign, so nothing cancels. Each group adds 8 rows per batch, and after the 10 batches the largest running sum is 2.8e30 units, 2.8% of the 1e32 limit. No row reaches the latch, so in this PR the shape runs the same path as (38,6), 1024 groups.
What the shape does guarantee is that one batch's absolute values sum to 2.1e32 units, past the limit. That makes a bound on the batch's total, like item 1 in the #7660 plan, fall back to the per-row loop, which is the regression gate the plan describes. Could the comment say that?
| // A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far | |
| // from it. Values of 1e27 to 5e28 come within a few batches of it without overflowing. | |
| // A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far | |
| // from it. Values of 1e27 to 5e28 have absolute values that sum to 2.1e32 per batch, past | |
| // the 1e32 limit, so a bound on a batch's total can't skip the per-row check. Each group's | |
| // rows share one sign, and no running sum passes 3% of the limit, so no group latches. |
There was a problem hiding this comment.
You're right. I checked the shape: no group mixes signs, the largest running sum after the 10 batches is 2.8e30 (2.8% of 1e32), and one batch's absolute values sum to 2.09e32. I took your comment.
| target_scale, | ||
| sum_data_type: sum_data_type.clone(), | ||
| sum_precision, | ||
| sum_bounds: Decimal128Bounds::new(sum_precision), |
There was a problem hiding this comment.
Could we add a test that checks the latch below precision 38? Every grouped AVG test uses precision 38 for both the sum and the result, so swapping sum_precision for target_precision here would pass them all. The SQL tests can't cover it either, because inputs within their declared precision need more than 10^10 rows to reach a sum precision below 38. This test fails if the line uses target_precision, and passes as written:
#[test]
fn grouped_latches_at_the_sum_precision_below_38() -> Result<()> {
// A DECIMAL(15,2) input sums as DECIMAL(25,2) and averages as DECIMAL(19,6), so a latch
// that used the result precision would also null group 0.
let max = 10_i128.pow(25) - 1;
let new_acc = || {
AvgDecimalGroupsAccumulator::new(
&DataType::Decimal128(19, 6),
&DataType::Decimal128(25, 2),
19,
6,
25,
2,
EvalMode::Legacy,
None,
crate::create_query_context_map(),
)
};
let sums = || Decimal128Array::from(vec![max, max, 1]).with_precision_and_scale(25, 2);
let mut updated = new_acc();
updated.update_batch(&[Arc::new(sums()?)], &[0, 1, 1], None, 2)?;
let mut merged = new_acc();
merged.merge_batch(
&[Arc::new(sums()?), Arc::new(Int64Array::from(vec![1; 3]))],
&[0, 1, 1],
2,
)?;
for mut acc in [updated, merged] {
let state = acc.state(EmitTo::All)?;
let sums = state[0].as_primitive::<Decimal128Type>();
assert_eq!(sums.iter().collect::<Vec<_>>(), vec![Some(max), None]);
}
Ok(())
}There was a problem hiding this comment.
Added as grouped_latches_at_the_sum_precision_below_38. Every existing grouped AVG test uses precision 38 for the sum and the result, so none of them could catch this without changing what it tests.
| is_empty: Vec::new(), | ||
| result_type, | ||
| precision, | ||
| bounds: Decimal128Bounds::new(precision), |
There was a problem hiding this comment.
Could we add the same check for grouped SUM? Its only grouped tests use Decimal128(10, 2) sums that stay small, and the overflow tests cover the ungrouped accumulator at precision 38. This test runs update and merge in both eval modes, and it fails if the bounds come from another precision:
#[test]
fn grouped_latches_at_the_result_precision_below_38() {
use datafusion::logical_expr::{EmitTo, GroupsAccumulator};
// Inputs within their declared precision need more than 10^10 rows to reach a sum
// precision below 38, so only a unit test can check that the latch uses the result
// precision there.
let data_type = DataType::Decimal128(25, 2);
let max = 10_i128.pow(25) - 1;
let decimals = |values: Vec<i128>| -> ArrayRef {
Arc::new(Decimal128Array::from(values).with_data_type(data_type.clone()))
};
for eval_mode in [EvalMode::Legacy, EvalMode::Ansi] {
let new_acc = || {
SumDecimalGroupsAccumulator::new(
data_type.clone(),
25,
eval_mode,
None,
crate::create_query_context_map(),
)
};
// Group 0 reaches the largest value that fits and group 1 goes one past it.
let mut updated = new_acc();
let update = updated.update_batch(&[decimals(vec![max, max, 1])], &[0, 1, 1], None, 2);
let mut merged = new_acc();
let is_empty: ArrayRef = Arc::new(BooleanArray::from(vec![false; 3]));
let merge = merged.merge_batch(&[decimals(vec![max, max, 1]), is_empty], &[0, 1, 1], 2);
for (result, mut acc) in [(update, updated), (merge, merged)] {
if eval_mode == EvalMode::Ansi {
assert!(result.is_err());
} else {
result.unwrap();
let sums = acc.evaluate(EmitTo::All).unwrap();
let sums = sums.as_primitive::<Decimal128Type>();
assert_eq!(sums.iter().collect::<Vec<_>>(), vec![Some(max), None]);
}
}
}
}There was a problem hiding this comment.
Added as grouped_latches_at_the_result_precision_below_38, next to the other grouped tests. Those use Decimal128(10, 2) sums far from the limit, so I kept this one separate.
| /// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked | ||
| /// up once so that a per-row check is two inline comparisons. Arrow does not mark that method | ||
| /// `#[inline]`, and release builds call it out of line on every row. |
There was a problem hiding this comment.
Could you file an arrow-rs issue to mark is_valid_decimal_precision #[inline], and link it from this comment? The Decimal128Type impl still lacks it on arrow-rs main. The published builds show the same out-of-line call as your macOS table. In the 1.2.0 jar, whose sum_decimal.rs, avg_decimal.rs, and utils.rs match this PR's base, both linux/amd64/libcomet.so (x86-64-v3) and linux/aarch64/libcomet.so (neoverse-n1) call it from update_single once, AVG update_batch twice, both merge_batches once, and SUM evaluate twice. On amd64 the call goes through the GOT:
movzx edx, byte ptr [r14 + 0x61] # precision
mov rdi, r12 # new_sum, low half
mov rsi, r13 # new_sum, high half
call qword ptr [rip + 0x4da27d4] # GOT slot 0x6d64218 -> Decimal128Type::is_valid_decimal_precisionarrow-cast's decimal casts make the same call from another crate (decimal.rs L269), so other arrow-rs users would get the change too. The #7660 plan lists it for later, and an issue keeps it from getting lost.
Decimal128Bounds stays useful after that lands, since your bench builds inline the call and still show SUM faster. The comment can say both reasons, so it doesn't go stale when arrow adds #[inline]:
| /// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked | |
| /// up once so that a per-row check is two inline comparisons. Arrow does not mark that method | |
| /// `#[inline]`, and release builds call it out of line on every row. | |
| /// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked | |
| /// up once so that a per-row check is two comparisons, with no precision guard or table lookup. | |
| /// Arrow does not mark that method `#[inline]` (<arrow-rs issue link>), so release builds also | |
| /// call it out of line on every row. |
There was a problem hiding this comment.
Filed apache/arrow-rs#12326 and linked it from the comment in b0b77cb. The comment keeps both reasons, as you suggested.
Before filing, I checked the claims:
- On arrow-rs
main(eb62e557), none of the fouris_valid_decimal_precisionimpls is#[inline], while the arrow-data functions they wrap are. - arrow-cast calls the method per value in
cast/decimal.rs. - In the 1.2.0 snapshot jar (
52ea712, with the same three files as this PR's base), bothlinux/aarch64/libcomet.soandlinux/amd64/libcomet.socall it from the places you listed. That isupdate_singleonce, AVGupdate_batchtwice, eachmerge_batchonce, and SUMevaluatetwice. On amd64 it goes through GOT slot0x6d5e9a0.
Every grouped AVG test used precision 38 for the sum and the result, and the grouped SUM tests kept their sums far from the limit, so bounds built from another precision would have passed them all. Two tests now latch a group at a sum precision of 25, in update and merge, with SUM in both eval modes. The audit entries drop the bench percentages and link the PR, the `near limit` bench comment describes what the shape does, and the `Decimal128Bounds` comment gives both reasons it skips arrow's call.
The `Decimal128Bounds` comment now points at apache/arrow-rs#12326, which asks arrow-rs to mark the `DecimalType::is_valid_decimal_precision` implementations `#[inline]`.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @comphead for the follow-ups and for filing apache/arrow-rs#12326. LGTM!
Which issue does this PR close?
Part of #7660.
Rationale for this change
The grouped decimal SUM and AVG accumulators check the running sum's precision on every row with
Decimal128Type::is_valid_decimal_precision. arrow-rs does not mark that method#[inline](types.rsL1540 in 59.3.2), and release builds oflibcometcall it out of line, with its precision guard and two table lookups, on every row. In a TPC-H Q1 profile it took 11.6% of CPU samples across its callers (#7660).What changes are included in this PR?
Decimal128Boundsinutils.rslooks up the bounds for one precision once. Itscontainsis two comparisons.SumDecimalGroupsAccumulatorandAvgDecimalGroupsAccumulatorkeep these bounds instead of the precision. They compare against them in update, merge, and SUM's evaluate. Every path accepts exactly the values it accepted before.benches/decimal_groups.rsdrives both accumulators throughupdate_batchover 4 (TPC-H Q1's shape), 1024, and 64K groups, with nulls, a capped precision, and values near the limit.The ungrouped AVG accumulator is left alone, since #7116 tracks giving it a wider running sum.
How are these changes tested?
The existing tests cover overflow, recovery, and ANSI errors for both accumulators. A new test checks
Decimal128Boundsagainstis_valid_decimal_precisionfor precisions 0, 1, 18, 25, 38, and 39, at and around each precision's bound. Two more check that grouped SUM and AVG latch at the sum's precision below 38, in update and merge, and for SUM in both eval modes. The other grouped tests use precision 38 for the sum, or sums far from the limit, so they would pass with the bounds of another precision.Release builds. These are calls to arrow's out-of-line
<Decimal128Type as DecimalType>::is_valid_decimal_precisionin the releaselibcomet.dylib(macOS aarch64,cargo build --release), counted withobjdump --disassemble-symbols:mainSumDecimalGroupsAccumulator::update_single(every row)SumDecimalGroupsAccumulator::merge_batchSumDecimalGroupsAccumulator::evaluateAvgDecimalGroupsAccumulator::update_batch(every row, both loops)AvgDecimalGroupsAccumulator::merge_batchThe
maincolumn comes from the releaselibcomet.dylibof780a860. Itssum_decimal.rs,avg_decimal.rs, andutils.rsmatch this PR's base. The function stays in the library for its other callers, so the zero counts mean the calls are gone, not that the symbol is.Bench builds inline arrow's call already, so criterion shows only the part that comes from loading the bounds once.
cargo bench --bench decimal_groups -- --baseline base, two samples, on a shared Apple M-series laptop (load average about 4):DataFusion's
sumin the same bench, which this PR does not touch, moved −1.8% and +0.5%. The AVG 4-group shape stays within 1.1%. That is criterion's noise threshold, and release builds also remove a call from that loop.For scale, #7660 forced
main's call out of line in the bench, as release builds compile it. That made grouped SUM 22% to 30% slower and grouped AVG 10% to 37% slower than with the call inlined. This PR removes the call. I have not measured a release build end to end.