Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
Internal decimal wrapper emitted around every decimal `+ - * /`, `sum`, and `avg` result to null out (non-ANSI) or raise on (ANSI) values that exceed the declared precision. Native impl: `math_funcs/internal/checkoverflow.rs`.

- Performance (tuned 2026-07-15, PR [#4937](https://github.com/apache/datafusion-comet/pull/4937)): both the ANSI and non-ANSI paths share a no-overflow fast path built on `is_valid_decimal_precision`, a small inlined bounds check scanned with `all` (short-circuits at the first overflow). When nothing overflows (the common shape) the input buffers are reused via `to_data()` (cheap Arc metadata clone) instead of allocating through `null_if_overflow_precision` (non-ANSI) or running the heavier per-value `validate_decimal_precision` (ANSI). The ANSI path only falls back to `validate_decimal_precision` when an overflow is present, to build the precise Spark error. ~10% faster on the no-overflow shape, ~17% with nulls, and ~69% faster for ANSI no-overflow (down to parity with non-ANSI); overflow shapes unchanged. Benchmark: `benches/check_overflow.rs`.
- Performance (tuned 2026-11-16, PR [#7658](https://github.com/apache/datafusion-comet/pull/7658)): the no-overflow scan reads every slot of the values buffer, null or not, with one unsigned comparison per value against the shifted precision bounds, instead of iterating `Option`s through the null bitmap. A null slot can hold any value, so the scan reads a slot's validity bit only at a value that doesn't fit and resumes after a null one. Benchmark: `benches/check_overflow.rs`, which gains dense-null, ANSI-with-nulls, last-row-overflow and out-of-range-null-slot shapes.

## abs

Expand Down
47 changes: 47 additions & 0 deletions native/spark-expr/benches/check_overflow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
// under the License.

use arrow::array::{Array, Decimal128Array};
use arrow::buffer::NullBuffer;
use arrow::compute::concat_batches;
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use criterion::{criterion_group, criterion_main, Criterion};
Expand Down Expand Up @@ -67,6 +69,7 @@ fn criterion_benchmark(c: &mut Criterion) {
// The common TPC-DS shape: decimal arithmetic result that fits the declared precision.
let no_overflow = build(size, 0, 0);
let no_overflow_nulls = build(size, 17, 0);
let no_overflow_dense_nulls = build(size, 2, 0);
let sparse_overflow = build(size, 0, 17);
let dense_overflow = build(size, 0, 2);

Expand All @@ -76,6 +79,15 @@ fn criterion_benchmark(c: &mut Criterion) {
c.bench_function("check_overflow: no overflow, with nulls", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
c.bench_function("check_overflow: no overflow, dense nulls", |b| {
b.iter(|| {
black_box(
legacy
.evaluate(black_box(&no_overflow_dense_nulls))
.unwrap(),
)
})
});
c.bench_function("check_overflow: sparse overflow", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&sparse_overflow)).unwrap()))
});
Expand All @@ -85,6 +97,41 @@ fn criterion_benchmark(c: &mut Criterion) {
c.bench_function("check_overflow: ansi no overflow", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow)).unwrap()))
});
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
Comment on lines +100 to +102

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a shape for the scan's worst case? Both sparse overflow and dense overflow put an overflow at row 0, because build tests i % overflow_every == 0, so the scan stops in the first chunk. The costliest input for this design is one overflow near the end of the batch: the scan reads every chunk, and then null_if_overflow_precision reads the batch again. apache/arrow-rs#12058 and apache/arrow-rs#12062 use the same one-pass-then-fallback design without stopping early, so every overflow case they benchmark pays for the full pass plus the fallback, and they report that cost. Could you add this shape's numbers to the description? It needs use arrow::compute::concat_batches; at the top.

Suggested change
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
// One overflow in the last row, so the scan reads the whole batch before the slow path runs.
// `sparse overflow` and `dense overflow` overflow at row 0 and stop in the first chunk.
let head = build(size - 1, 0, 0);
let last_row_overflow = concat_batches(&head.schema(), [&head, &build(1, 0, 1)]).unwrap();
c.bench_function("check_overflow: last row overflow", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&last_row_overflow)).unwrap()))
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added last row overflow as you wrote it. With the scan in this revision it takes 7.38 µs against 8.72 µs on main, whose scan also reads the whole batch there before null_if_overflow_precision runs. I also added no overflow, null slot out of range, the worst case for a scan that includes null slots: the with nulls batch with a value that doesn't fit in its last slot, which is null. The numbers are in the description.


// One overflow in the last row, so the scan reads the whole batch before the slow path runs.
// `sparse overflow` and `dense overflow` overflow at row 0 and stop in the first chunk.
Comment on lines +104 to +105

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scan no longer works in chunks, so "stop in the first chunk" describes the earlier fold. With the position loop, these two shapes stop at row 0.

Suggested change
// One overflow in the last row, so the scan reads the whole batch before the slow path runs.
// `sparse overflow` and `dense overflow` overflow at row 0 and stop in the first chunk.
// One overflow in the last row, so the scan reads the whole batch before the slow path runs.
// `sparse overflow` and `dense overflow` overflow at row 0, where the scan stops.

let head = build(size - 1, 0, 0);
let last_row_overflow = concat_batches(&head.schema(), [&head, &build(1, 0, 1)]).unwrap();
c.bench_function("check_overflow: last row overflow", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&last_row_overflow)).unwrap()))
});

// The `with nulls` shape, except that its last row is a null slot holding a value that
// doesn't fit. A scan over every slot reaches it at the end of the batch and has to find
// that the slot is null, though nothing overflows.
let null_slot_overflow = {
let nulls = build(size, 17, 0);
let nulls = nulls
.column(0)
.as_any()
.downcast_ref::<Decimal128Array>()
.unwrap();
let mut values = nulls.values().to_vec();
values[size - 1] = OVERFLOW_VALUE;
let mut validity: Vec<bool> = (0..size).map(|i| nulls.is_valid(i)).collect();
validity[size - 1] = false;
let array = Decimal128Array::new(values.into(), Some(NullBuffer::from(validity)))
.with_precision_and_scale(IN_PRECISION, SCALE)
.unwrap();
let schema = Schema::new(vec![Field::new("d", array.data_type().clone(), true)]);
RecordBatch::try_new(Arc::new(schema), vec![Arc::new(array)]).unwrap()
};
c.bench_function("check_overflow: no overflow, null slot out of range", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&null_slot_overflow)).unwrap()))
});
}

criterion_group!(benches, criterion_benchmark);
Expand Down
151 changes: 141 additions & 10 deletions native/spark-expr/src/math_funcs/internal/checkoverflow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@
use arrow::datatypes::{DataType, Schema};
use arrow::{
array::{as_primitive_array, Array, ArrayRef, Decimal128Array},
datatypes::{Decimal128Type, DecimalType},
datatypes::{
Decimal128Type, DecimalType, MAX_DECIMAL128_FOR_EACH_PRECISION,
MIN_DECIMAL128_FOR_EACH_PRECISION,
},
record_batch::RecordBatch,
};
use datafusion::common::{DataFusionError, ScalarValue};
Expand Down Expand Up @@ -119,16 +122,12 @@ impl PhysicalExpr for CheckOverflow {

let decimal_array = as_primitive_array::<Decimal128Type>(&array);

// Fast path shared by both ANSI and non-ANSI: `is_valid_decimal_precision` is a
// small, inlined bounds check and `all` short-circuits at the first overflow. When
// nothing overflows (the common shape for decimal arithmetic in TPC-DS) we reuse the
// input buffers via `to_data()`, which only clones cheap Arc metadata. This avoids
// the heavier per-value `validate_decimal_precision` scan (ANSI) or the allocating
// Fast path shared by both ANSI and non-ANSI. When nothing overflows (the common
// shape for decimal arithmetic in TPC-DS) we reuse the input buffers via
// `to_data()`, which only clones cheap Arc metadata. This avoids the heavier
// per-value `validate_decimal_precision` scan (ANSI) or the allocating
// `null_if_overflow_precision` (non-ANSI) below.
let no_overflow = decimal_array
.iter()
.flatten()
.all(|v| Decimal128Type::is_valid_decimal_precision(v, *precision));
let no_overflow = fits_precision(decimal_array, *precision);

let casted_array = if no_overflow {
Decimal128Array::from(decimal_array.to_data())
Expand Down Expand Up @@ -249,9 +248,46 @@ impl PhysicalExpr for CheckOverflow {
}
}

/// Whether every non-null value of `array` fits `precision`, the check that
/// `Decimal128Type::is_valid_decimal_precision` makes for one value.
#[inline]
fn fits_precision(array: &Decimal128Array, precision: u8) -> bool {
Comment on lines +253 to +254

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this scan also go into arrow-rs, in PrimitiveArray::validate_decimal_precision? It makes the same check with the same per-slot is_valid lookup and early return that this PR removes here. The vectorized arithmetic in apache/arrow-rs#12058 and apache/arrow-rs#12062 detects only i128 overflow and doesn't check precision, so it won't replace this check when it lands.

#5104 lists CheckOverflow as hand-rolled on purpose, because ANSI errors need the offending value. This PR splits the scan, which returns only a bool, from the error path that finds the value, so that reason no longer covers the scan.

I'd land the scan here first and settle its design in this PR, including the null handling and loop form from my other comments. Then the arrow-rs change can port the same code instead of reopening those choices. Could you file the arrow-rs issue now and link it from a comment on fits_precision? Once Comet's arrow version includes the upstream change, fits_precision can become validate_decimal_precision(precision).is_ok(), and DataFusion and other arrow-rs callers get the faster check too.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you file the arrow-rs issue now and link it from a comment on fits_precision?

apache/arrow-rs#12342 adds #[inline] to is_valid_decimal_precision and changes the arrow-cast kernels, but it doesn't touch this scan. PrimitiveArray::validate_decimal_precision on arrow-rs main still calls is_valid on every slot and returns at the first error. Could you file the issue for porting fits_precision there, and add a comment on fits_precision that links it?

let bounds = (
MIN_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
MAX_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
);
let (Some(&min), Some(&max)) = bounds else {
return array
.iter()
.flatten()
.all(|v| Decimal128Type::is_valid_decimal_precision(v, precision));
};
// Scan every slot of the values buffer, null or not, instead of reading the null bitmap per
// slot. Shifting by `min` turns the two bounds checks into one unsigned comparison, so the
// loop is one compare per value and a branch that is only taken at a value that doesn't fit.
let range = max.wrapping_sub(min) as u128;
let values = array.values();
let nulls = array.nulls();
let mut start = 0;
while let Some(offset) = values[start..]
.iter()
.position(|&v| v.wrapping_sub(min) as u128 > range)
{
// A null slot can hold any value, so a value that doesn't fit only counts in a valid
// slot. The null bitmap is read only at such a value, and the scan resumes after it.
let index = start + offset;
if !nulls.is_some_and(|nulls| nulls.is_null(index)) {
return false;
}
start = index + 1;
}
true
}

#[cfg(test)]
mod tests {
use super::*;
use arrow::buffer::NullBuffer;
use arrow::datatypes::{Field, Schema};
use arrow::record_batch::RecordBatch;
use std::fmt::{Display, Formatter};
Expand Down Expand Up @@ -508,5 +544,100 @@ mod tests {
let batch = array_batch(vec![Some(999), Some(9999)], 38, 0);
let out = eval_array(&array_check_overflow(3, 0, false), &batch);
assert_eq!(out.iter().collect::<Vec<_>>(), vec![Some(999), None]);

// The scan shifts by the precision's minimum and compares unsigned, so also check the
// values where that shift wraps, at the widest precision.
let max = MAX_DECIMAL128_FOR_EACH_PRECISION[38];
let min = MIN_DECIMAL128_FOR_EACH_PRECISION[38];
let fits = array_batch(vec![Some(max), Some(min), Some(0)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, true), &fits);
assert_eq!(out.null_count(), 0);
for v in [max + 1, min - 1, i128::MAX, i128::MIN] {
let batch = array_batch(vec![Some(0), Some(v)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, false), &batch);
assert!(out.is_null(1), "{v}");
let ansi = array_check_overflow(38, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{v}");
}
}

#[test]
fn test_array_overflow_at_every_position_and_sliced() {
// The scan stops at the first value that doesn't fit, and a compiled loop can split the
// batch into an unrolled part and a remainder, so move one value that doesn't fit
// through every row. 203 rows are not a multiple of any unroll factor.
for overflow in [1000, -1000] {
for i in 0..203 {
let mut values = vec![Some(5); 203];
values[i] = Some(overflow);
let batch = array_batch(values, 38, 0);
let out = eval_array(&array_check_overflow(3, 0, false), &batch);
assert_eq!(out.null_count(), 1, "{overflow} at {i}");
assert!(out.is_null(i), "{overflow} at {i}");
let ansi = array_check_overflow(3, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{overflow} at {i}");
}
}

// A sliced array must only be checked within its slice. This one starts right after
// the overflowing value, so the slice has a non-zero offset.
let mut values = vec![Some(5); 200];
values[130] = Some(1000);
let sliced = array_batch(values, 38, 0).slice(131, 69);
let out = eval_array(&array_check_overflow(3, 0, true), &sliced);
assert_eq!(out.len(), 69);
assert_eq!(out.null_count(), 0);
}

/// A Decimal128(38, 0) batch whose null slots keep the values given for them.
fn array_batch_with_null_slots(values: Vec<i128>, valid: Vec<bool>) -> RecordBatch {
let arr = Decimal128Array::new(values.into(), Some(NullBuffer::from(valid)))
.with_precision_and_scale(38, 0)
.unwrap();
let schema = Schema::new(vec![Field::new("d", arr.data_type().clone(), true)]);
RecordBatch::try_new(Arc::new(schema), vec![Arc::new(arr)]).unwrap()
}

#[test]
fn test_array_null_slot_out_of_range_is_ignored() {
// A null slot can hold any value. One that doesn't fit the precision must not null out
// valid values or raise an ANSI error, and the scan must still find a value that doesn't
// fit in a valid slot after it, next to it or at the end.
let n = 70;
for i in 0..n - 1 {
let mut values = vec![5_i128; n];
let mut valid = vec![true; n];
values[i] = 1000;
valid[i] = false;
let mut expected = vec![Some(5_i128); n];
expected[i] = None;
let batch = array_batch_with_null_slots(values.clone(), valid.clone());
for fail_on_error in [false, true] {
let out = eval_array(&array_check_overflow(3, 0, fail_on_error), &batch);
assert_eq!(out.iter().collect::<Vec<_>>(), expected, "null slot {i}");
}
for j in [i + 1, n - 1] {
let mut values = values.clone();
values[j] = -1000;
let batch = array_batch_with_null_slots(values, valid.clone());
let out = eval_array(&array_check_overflow(3, 0, false), &batch);
let mut expected = expected.clone();
expected[j] = None;
assert_eq!(out.iter().collect::<Vec<_>>(), expected, "{i}, {j}");
let ansi = array_check_overflow(3, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "null slot {i}, overflow at {j}");
}
}

// In a slice, the values and the null bitmap both count from the slice's first row.
let mut values = vec![5_i128; 100];
let mut valid = vec![true; 100];
values[40] = 1000;
values[60] = 1000;
valid[60] = false;
let sliced = array_batch_with_null_slots(values, valid).slice(41, 59);
let out = eval_array(&array_check_overflow(3, 0, true), &sliced);
assert_eq!(out.null_count(), 1);
assert!(out.is_null(19));
}
}
Comment on lines +590 to 643

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add tests that move the value that doesn't fit through every row, and that cover the edges of the shift? The compiled scan handles each part of the batch with different code: on aarch64 a 32-value vector loop, an 8-value loop, and a one-value loop, and on x86-64 a two-value unrolled loop and a remainder. The new test checks one position, row 130. apache/arrow-rs#12058 sweeps the overflowing row through every lane for the same reason.

The array tests also all use precision 3, so nothing checks precision 38, where v - min wraps for i128::MAX and i128::MIN. The last test covers a null slot that holds a value that doesn't fit, which the suggestion on fits_precision makes reachable.

Suggested change
}
}
}
#[test]
fn test_array_no_nulls_overflow_at_every_position() {
// The compiled scan handles the main loop, the chunk boundaries, and the remainder
// differently, so move one value that doesn't fit through every row. 203 rows leave a
// last chunk of 11 values.
for overflow in [1000, -1000] {
for i in 0..203 {
let mut values = vec![Some(5); 203];
values[i] = Some(overflow);
let batch = array_batch(values, 38, 0);
let out = eval_array(&array_check_overflow(3, 0, false), &batch);
assert_eq!(out.null_count(), 1, "{overflow} at {i}");
assert!(out.is_null(i), "{overflow} at {i}");
let ansi = array_check_overflow(3, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{overflow} at {i}");
}
}
}
#[test]
fn test_array_no_nulls_precision_38_extremes() {
// The scan shifts by the precision's minimum and compares unsigned, so check the values
// where that shift wraps, at the widest precision.
let max = MAX_DECIMAL128_FOR_EACH_PRECISION[38];
let min = MIN_DECIMAL128_FOR_EACH_PRECISION[38];
let fits = array_batch(vec![Some(max), Some(min), Some(0)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, true), &fits);
assert_eq!(out.null_count(), 0);
for v in [max + 1, min - 1, i128::MAX, i128::MIN] {
let batch = array_batch(vec![Some(0), Some(v)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, false), &batch);
assert!(out.is_null(1), "{v}");
let ansi = array_check_overflow(38, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{v}");
}
}
#[test]
fn test_array_null_slot_out_of_range_is_ignored() {
// A null slot can hold any value. One that doesn't fit the precision must not null out
// valid values or raise an ANSI error.
use arrow::buffer::NullBuffer;
let nulls = NullBuffer::from(vec![true, false, true]);
let arr = Decimal128Array::new(vec![5, 1000, 7].into(), Some(nulls))
.with_precision_and_scale(38, 0)
.unwrap();
let schema = Schema::new(vec![Field::new("d", arr.data_type().clone(), true)]);
let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(arr)]).unwrap();
for fail_on_error in [false, true] {
let out = eval_array(&array_check_overflow(3, 0, fail_on_error), &batch);
assert_eq!(out.iter().collect::<Vec<_>>(), vec![Some(5), None, Some(7)]);
}
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I folded the sweep into the test this PR added, now test_array_overflow_at_every_position_and_sliced (both signs, 203 rows, legacy and ANSI, plus the slice), and added precision 38's bounds and i128::MIN/i128::MAX to test_array_boundary_precision_max_passes_and_over_by_one_overflows. test_array_null_slot_out_of_range_is_ignored is new. It puts a null slot that doesn't fit at every position of a 70-row batch and checks that it changes nothing, and that a valid value that doesn't fit right after it or in the last row is still found. It also checks a slice, whose null bitmap has an offset.