Skip to content

perf: inline the precision check in grouped decimal SUM and AVG - #7674

Open
comphead wants to merge 3 commits into
apache:mainfrom
comphead:decimal-agg-inline-precision-check
Open

comphead wants to merge 3 commits into
apache:mainfrom
comphead:decimal-agg-inline-precision-check

Conversation

@comphead

@comphead comphead commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #7660.

Rationale for this change

The grouped decimal SUM and AVG accumulators check the running sum's precision on every row with Decimal128Type::is_valid_decimal_precision. arrow-rs does not mark that method #[inline] (types.rs L1540 in 59.3.2), and release builds of libcomet call it out of line, with its precision guard and two table lookups, on every row. In a TPC-H Q1 profile it took 11.6% of CPU samples across its callers (#7660).

What changes are included in this PR?

  • Decimal128Bounds in utils.rs looks up the bounds for one precision once. Its contains is two comparisons.
  • SumDecimalGroupsAccumulator and AvgDecimalGroupsAccumulator keep these bounds instead of the precision. They compare against them in update, merge, and SUM's evaluate. Every path accepts exactly the values it accepted before.
  • benches/decimal_groups.rs drives both accumulators through update_batch over 4 (TPC-H Q1's shape), 1024, and 64K groups, with nulls, a capped precision, and values near the limit.

The ungrouped AVG accumulator is left alone, since #7116 tracks giving it a wider running sum.

How are these changes tested?

The existing tests cover overflow, recovery, and ANSI errors for both accumulators. A new test checks Decimal128Bounds against is_valid_decimal_precision for precisions 0, 1, 18, 25, 38, and 39, at and around each precision's bound. Two more check that grouped SUM and AVG latch at the sum's precision below 38, in update and merge, and for SUM in both eval modes. The other grouped tests use precision 38 for the sum, or sums far from the limit, so they would pass with the bounds of another precision.

Release builds. These are calls to arrow's out-of-line <Decimal128Type as DecimalType>::is_valid_decimal_precision in the release libcomet.dylib (macOS aarch64, cargo build --release), counted with objdump --disassemble-symbols:

Function main This PR
SumDecimalGroupsAccumulator::update_single (every row) 1 0
SumDecimalGroupsAccumulator::merge_batch 1 0
SumDecimalGroupsAccumulator::evaluate 2 0
AvgDecimalGroupsAccumulator::update_batch (every row, both loops) 2 0
AvgDecimalGroupsAccumulator::merge_batch 1 0

The main column comes from the release libcomet.dylib of 780a860. Its sum_decimal.rs, avg_decimal.rs, and utils.rs match this PR's base. The function stays in the library for its other callers, so the zero counts mean the calls are gone, not that the symbol is.

Bench builds inline arrow's call already, so criterion shows only the part that comes from loading the bounds once. cargo bench --bench decimal_groups -- --baseline base, two samples, on a shared Apple M-series laptop (load average about 4):

Shape SUM AVG
(25,2), 4 groups −9.4%, −9.7% +1.1%, +0.6%
(25,2), 1024 groups −12.3%, −11.6% −3.3%, −2.8%
(25,2), 64K groups +2.5%, −3.6% −2.6%, −1.8%
(25,2), 1024 groups, 10% nulls −8.3%, −8.9% −4.6%, −4.3%
(38,6), 1024 groups −12.4%, −11.8% −2.6%, −1.5%
(38,6), 1024 groups, near limit −10.4%, −11.2% +0.5%, −0.4%

DataFusion's sum in the same bench, which this PR does not touch, moved −1.8% and +0.5%. The AVG 4-group shape stays within 1.1%. That is criterion's noise threshold, and release builds also remove a call from that loop.

For scale, #7660 forced main's call out of line in the bench, as release builds compile it. That made grouped SUM 22% to 30% slower and grouped AVG 10% to 37% slower than with the call inlined. This PR removes the call. I have not measured a release build end to end.

The grouped decimal SUM and AVG accumulators checked the running sum on
every row with `Decimal128Type::is_valid_decimal_precision`. arrow-rs
does not mark that method `#[inline]`, and release builds of libcomet
call it out of line, with its precision guard and two table lookups.

Look up the bounds for the sum's precision once per accumulator, and
compare against them in update, merge, and SUM's evaluate. The values
each path accepts are unchanged, which a new test pins against arrow's
check. Add a `decimal_groups` bench that drives both accumulators
through `update_batch`.

The ungrouped AVG accumulator is left alone, since apache#7116 tracks giving
it a wider running sum.
@github-actions github-actions Bot added enhancement New feature or request performance area:aggregation Hash aggregates, aggregate expressions area:expressions Expression evaluation labels Nov 17, 2026
@mbutrovich
mbutrovich self-requested a review November 17, 2026 23:43

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead. Loading the bounds once per accumulator is the right first step, given that the fast path in your comment on #7660 didn't pass the bench gate.

- Spark 3.4.3 (2026-05-26)
- Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3
- Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only `QueryContext` import path differs. `YearMonthIntervalType` and `DayTimeIntervalType` inputs (supported by Spark) fall back to Spark in Comet.
- Performance (tuned 2026-11-17): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Bench builds already inline that call: 2% to 5% faster on most shapes, the 4-group shape within 1%. Benchmark: `benches/decimal_groups.rs`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we drop the AVG percentages here? DataFusion's sum, which this PR doesn't touch, moved -1.8% and +0.5% in the same runs. The AVG shapes moved between +1.1% and -4.6%, which is within about twice that noise floor on a laptop with a load average of 4. The change this entry can state is the call that release builds no longer make. The other entries in this file also link their PR.

Suggested change
- Performance (tuned 2026-11-17): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Bench builds already inline that call: 2% to 5% faster on most shapes, the 4-group shape within 1%. Benchmark: `benches/decimal_groups.rs`.
- Performance (tuned 2026-11-17, PR [#7674](https://github.com/apache/datafusion-comet/pull/7674)): the grouped decimal accumulator loads the sum's precision bounds once and compares against them on every update and merge, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Benchmark: `benches/decimal_groups.rs`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I dropped the AVG percentages and added the PR link, as you suggested.


## sum

- Performance (tuned 2026-11-17): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. 9% to 12% faster on most shapes even in bench builds, which inline that call. Benchmark: `benches/decimal_groups.rs`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 9% to 12% comes from bench builds on an Apple M-series laptop. Comet deploys on x86-64 servers, and Apple silicon results don't predict those. Could you rerun decimal_groups on an x86-64 machine before putting a number in the audit, or drop it as in the AVG entry? Here is the version without it, with the PR link the other entries carry:

Suggested change
- Performance (tuned 2026-11-17): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. 9% to 12% faster on most shapes even in bench builds, which inline that call. Benchmark: `benches/decimal_groups.rs`.
- Performance (tuned 2026-11-17, PR [#7674](https://github.com/apache/datafusion-comet/pull/7674)): the grouped decimal accumulator loads the result's precision bounds once and compares against them on every update, merge, and evaluate, instead of calling `Decimal128Type::is_valid_decimal_precision`, which release builds keep out of line. Benchmark: `benches/decimal_groups.rs`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have an x86-64 machine to rerun decimal_groups on, so I dropped the number and kept the call that release builds no longer make, with the PR link.

Comment on lines +122 to +123
// A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far
// from it. Values of 1e27 to 5e28 come within a few batches of it without overflowing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't match what the near limit shape does. The groups are i % 1024 and the sign alternates on i % 2. Because 1024 is even, every row of a group has the same sign, so nothing cancels. Each group adds 8 rows per batch, and after the 10 batches the largest running sum is 2.8e30 units, 2.8% of the 1e32 limit. No row reaches the latch, so in this PR the shape runs the same path as (38,6), 1024 groups.

What the shape does guarantee is that one batch's absolute values sum to 2.1e32 units, past the limit. That makes a bound on the batch's total, like item 1 in the #7660 plan, fall back to the per-row loop, which is the regression gate the plan describes. Could the comment say that?

Suggested change
// A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far
// from it. Values of 1e27 to 5e28 come within a few batches of it without overflowing.
// A DECIMAL(38,6) input keeps the sum at the capped precision 38. Values near 1e5 stay far
// from it. Values of 1e27 to 5e28 have absolute values that sum to 2.1e32 per batch, past
// the 1e32 limit, so a bound on a batch's total can't skip the per-row check. Each group's
// rows share one sign, and no running sum passes 3% of the limit, so no group latches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I checked the shape: no group mixes signs, the largest running sum after the 10 batches is 2.8e30 (2.8% of 1e32), and one batch's absolute values sum to 2.09e32. I took your comment.

target_scale,
sum_data_type: sum_data_type.clone(),
sum_precision,
sum_bounds: Decimal128Bounds::new(sum_precision),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test that checks the latch below precision 38? Every grouped AVG test uses precision 38 for both the sum and the result, so swapping sum_precision for target_precision here would pass them all. The SQL tests can't cover it either, because inputs within their declared precision need more than 10^10 rows to reach a sum precision below 38. This test fails if the line uses target_precision, and passes as written:

    #[test]
    fn grouped_latches_at_the_sum_precision_below_38() -> Result<()> {
        // A DECIMAL(15,2) input sums as DECIMAL(25,2) and averages as DECIMAL(19,6), so a latch
        // that used the result precision would also null group 0.
        let max = 10_i128.pow(25) - 1;
        let new_acc = || {
            AvgDecimalGroupsAccumulator::new(
                &DataType::Decimal128(19, 6),
                &DataType::Decimal128(25, 2),
                19,
                6,
                25,
                2,
                EvalMode::Legacy,
                None,
                crate::create_query_context_map(),
            )
        };
        let sums = || Decimal128Array::from(vec![max, max, 1]).with_precision_and_scale(25, 2);
        let mut updated = new_acc();
        updated.update_batch(&[Arc::new(sums()?)], &[0, 1, 1], None, 2)?;
        let mut merged = new_acc();
        merged.merge_batch(
            &[Arc::new(sums()?), Arc::new(Int64Array::from(vec![1; 3]))],
            &[0, 1, 1],
            2,
        )?;
        for mut acc in [updated, merged] {
            let state = acc.state(EmitTo::All)?;
            let sums = state[0].as_primitive::<Decimal128Type>();
            assert_eq!(sums.iter().collect::<Vec<_>>(), vec![Some(max), None]);
        }
        Ok(())
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added as grouped_latches_at_the_sum_precision_below_38. Every existing grouped AVG test uses precision 38 for the sum and the result, so none of them could catch this without changing what it tests.

is_empty: Vec::new(),
result_type,
precision,
bounds: Decimal128Bounds::new(precision),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add the same check for grouped SUM? Its only grouped tests use Decimal128(10, 2) sums that stay small, and the overflow tests cover the ungrouped accumulator at precision 38. This test runs update and merge in both eval modes, and it fails if the bounds come from another precision:

    #[test]
    fn grouped_latches_at_the_result_precision_below_38() {
        use datafusion::logical_expr::{EmitTo, GroupsAccumulator};

        // Inputs within their declared precision need more than 10^10 rows to reach a sum
        // precision below 38, so only a unit test can check that the latch uses the result
        // precision there.
        let data_type = DataType::Decimal128(25, 2);
        let max = 10_i128.pow(25) - 1;
        let decimals = |values: Vec<i128>| -> ArrayRef {
            Arc::new(Decimal128Array::from(values).with_data_type(data_type.clone()))
        };
        for eval_mode in [EvalMode::Legacy, EvalMode::Ansi] {
            let new_acc = || {
                SumDecimalGroupsAccumulator::new(
                    data_type.clone(),
                    25,
                    eval_mode,
                    None,
                    crate::create_query_context_map(),
                )
            };
            // Group 0 reaches the largest value that fits and group 1 goes one past it.
            let mut updated = new_acc();
            let update = updated.update_batch(&[decimals(vec![max, max, 1])], &[0, 1, 1], None, 2);
            let mut merged = new_acc();
            let is_empty: ArrayRef = Arc::new(BooleanArray::from(vec![false; 3]));
            let merge = merged.merge_batch(&[decimals(vec![max, max, 1]), is_empty], &[0, 1, 1], 2);
            for (result, mut acc) in [(update, updated), (merge, merged)] {
                if eval_mode == EvalMode::Ansi {
                    assert!(result.is_err());
                } else {
                    result.unwrap();
                    let sums = acc.evaluate(EmitTo::All).unwrap();
                    let sums = sums.as_primitive::<Decimal128Type>();
                    assert_eq!(sums.iter().collect::<Vec<_>>(), vec![Some(max), None]);
                }
            }
        }
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added as grouped_latches_at_the_result_precision_below_38, next to the other grouped tests. Those use Decimal128(10, 2) sums far from the limit, so I kept this one separate.

Comment thread native/spark-expr/src/utils.rs Outdated
Comment on lines +327 to +329
/// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked
/// up once so that a per-row check is two inline comparisons. Arrow does not mark that method
/// `#[inline]`, and release builds call it out of line on every row.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you file an arrow-rs issue to mark is_valid_decimal_precision #[inline], and link it from this comment? The Decimal128Type impl still lacks it on arrow-rs main. The published builds show the same out-of-line call as your macOS table. In the 1.2.0 jar, whose sum_decimal.rs, avg_decimal.rs, and utils.rs match this PR's base, both linux/amd64/libcomet.so (x86-64-v3) and linux/aarch64/libcomet.so (neoverse-n1) call it from update_single once, AVG update_batch twice, both merge_batches once, and SUM evaluate twice. On amd64 the call goes through the GOT:

movzx  edx, byte ptr [r14 + 0x61]    # precision
mov    rdi, r12                      # new_sum, low half
mov    rsi, r13                      # new_sum, high half
call   qword ptr [rip + 0x4da27d4]   # GOT slot 0x6d64218 -> Decimal128Type::is_valid_decimal_precision

arrow-cast's decimal casts make the same call from another crate (decimal.rs L269), so other arrow-rs users would get the change too. The #7660 plan lists it for later, and an issue keeps it from getting lost.

Decimal128Bounds stays useful after that lands, since your bench builds inline the call and still show SUM faster. The comment can say both reasons, so it doesn't go stale when arrow adds #[inline]:

Suggested change
/// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked
/// up once so that a per-row check is two inline comparisons. Arrow does not mark that method
/// `#[inline]`, and release builds call it out of line on every row.
/// The values that `Decimal128Type::is_valid_decimal_precision` accepts for one precision, looked
/// up once so that a per-row check is two comparisons, with no precision guard or table lookup.
/// Arrow does not mark that method `#[inline]` (<arrow-rs issue link>), so release builds also
/// call it out of line on every row.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed apache/arrow-rs#12326 and linked it from the comment in b0b77cb. The comment keeps both reasons, as you suggested.

Before filing, I checked the claims:

  • On arrow-rs main (eb62e557), none of the four is_valid_decimal_precision impls is #[inline], while the arrow-data functions they wrap are.
  • arrow-cast calls the method per value in cast/decimal.rs.
  • In the 1.2.0 snapshot jar (52ea712, with the same three files as this PR's base), both linux/aarch64/libcomet.so and linux/amd64/libcomet.so call it from the places you listed. That is update_single once, AVG update_batch twice, each merge_batch once, and SUM evaluate twice. On amd64 it goes through GOT slot 0x6d5e9a0.

Every grouped AVG test used precision 38 for the sum and the result, and
the grouped SUM tests kept their sums far from the limit, so bounds built
from another precision would have passed them all. Two tests now latch a
group at a sum precision of 25, in update and merge, with SUM in both
eval modes.

The audit entries drop the bench percentages and link the PR, the
`near limit` bench comment describes what the shape does, and the
`Decimal128Bounds` comment gives both reasons it skips arrow's call.
The `Decimal128Bounds` comment now points at apache/arrow-rs#12326, which
asks arrow-rs to mark the `DecimalType::is_valid_decimal_precision`
implementations `#[inline]`.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead for the follow-ups and for filing apache/arrow-rs#12326. LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:aggregation Hash aggregates, aggregate expressions area:expressions Expression evaluation enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants