You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Skip CheckOverflow around decimal +, -, and * whose result type always fits #7659
DecimalPrecision.promote wraps every decimal Add, Subtract, and Multiply in CheckOverflow. When the operation takes the narrow path in the planner and runs as a DataFusion BinaryExpr, no inputs that fit their declared types can produce a result that exceeds the result precision.
For + and -, the planner takes the narrow path when decimal128_add_sub_width, max(s1, s2) + max(p1 - s1, p2 - s2), is below 38 (condition). Spark's result type is then (max(s1, s2) + max(p1 - s1, p2 - s2) + 1, max(s1, s2)) unchanged, because adjustPrecisionScale and bounded adjust nothing at or below 38 digits. Each operand rescaled to max(s1, s2) is below 10^(max(p1 - s1, p2 - s2) + max(s1, s2)) in absolute value, so their sum or difference is below 10^(result precision).
For *, the planner takes the narrow path when p1 + p2 is below 38. Spark's result type is then (p1 + p2 + 1, s1 + s2), and the product of the unscaled values is below 10^(p1 + p2).
So on this path, CheckOverflow can only null out or raise for an input whose value already exceeds its own declared precision, but it still scans every batch.
In TPC-H Q1, every CheckOverflow is on this path. Spark types the literal 1 as decimal(1,0) by default (coercion, spark.sql.legacy.literal.pickMinimumPrecision). With decimal(15,2) columns, 1 - l_discount and 1 + l_tax are decimal(1,0) with decimal(15,2), and l_extendedprice * (1 - l_discount) is decimal(15,2) * decimal(16,2). The outer * (1 + l_tax) needs 48 digits, so it runs as WideDecimalBinaryExpr, where the planner already drops CheckOverflow. #7658 measured the CheckOverflow scan at about 5% of native CPU samples in Q1 at SF10.
Describe the potential solution
When the child of CheckOverflow is a Decimal128+, -, or * on the narrow path and its type equals the CheckOverflow target type, the planner could return the child without the wrapper, the way it already does for WideDecimalBinaryExpr. The same type check could instead run in DecimalPrecision.promote, so the wrapper is never serialized.
Additional context
The open question is compatibility for inputs that violate their own declared type. Spark's Add, Subtract, and Multiply call checkDecimalOverflow on every result, so Spark nulls out or raises on such a value, and Comet would pass it through. One way to get such a value is a Parquet file whose unscaled values exceed the column's declared precision, if the reader doesn't reject it. I haven't checked whether Spark's or Comet's Parquet readers validate this. Does Comet need to match Spark on this input, or can it be documented as a difference for malformed data?
Willingness to contribute
I cannot contribute a fix for this bug at this time
Activity