Describe the bug
Sliding-window SUM(BIGINT) ignores ANSI and TRY overflow semantics. try_sum returns a wrapped negative value instead of NULL, and ANSI sum returns that value instead of throwing.
Steps to reproduce
With Comet and native shuffle enabled:
SET spark.sql.adaptive.enabled=false;
CREATE TABLE sliding_sum_repro (id INT, v BIGINT) USING parquet;
INSERT INTO sliding_sum_repro VALUES
(1, 9223372036854775807), (2, 1), (3, -1);
SELECT id, try_sum(v) OVER (
ORDER BY id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
) AS s
FROM sliding_sum_repro;
Also run with sum instead of try_sum and spark.sql.ansi.enabled=true.
Expected behavior
For id = 2:
| Expression |
Spark |
Comet |
try_sum, ANSI on or off |
NULL |
-9223372036854775808 |
sum, ANSI on |
ARITHMETIC_OVERFLOW |
-9223372036854775808 |
Both engines return 0 for the third try_sum row. Legacy sum wraps in both engines as expected.
Additional context
Reproduced on main b7f35b6ac, Spark 3.5.9 and 4.1.3, with CometWindowExec asserted and allowIncompatible=false.
process_agg_func uses the mode-aware SumInteger only for ever-expanding frames. Sliding frames use DataFusion's built-in sum, which wraps and does not receive the evaluation mode. #4729 / #4732 guarded the corresponding decimal case, but not integral ANSI/TRY sums.
A focused fix could fall back for these integral sliding frames while retaining native legacy and ever-expanding sums.
Describe the bug
Sliding-window
SUM(BIGINT)ignores ANSI and TRY overflow semantics.try_sumreturns a wrapped negative value instead of NULL, and ANSIsumreturns that value instead of throwing.Steps to reproduce
With Comet and native shuffle enabled:
Also run with
suminstead oftry_sumandspark.sql.ansi.enabled=true.Expected behavior
For
id = 2:try_sum, ANSI on or offsum, ANSI onBoth engines return 0 for the third
try_sumrow. Legacysumwraps in both engines as expected.Additional context
Reproduced on main
b7f35b6ac, Spark 3.5.9 and 4.1.3, withCometWindowExecasserted andallowIncompatible=false.process_agg_funcuses the mode-awareSumIntegeronly for ever-expanding frames. Sliding frames use DataFusion's built-insum, which wraps and does not receive the evaluation mode. #4729 / #4732 guarded the corresponding decimal case, but not integral ANSI/TRY sums.A focused fix could fall back for these integral sliding frames while retaining native legacy and ever-expanding sums.