Skip to content

Sliding-window SUM(BIGINT) ignores ANSI and TRY overflow semantics #6043

Description

@rich7420

Describe the bug

Sliding-window SUM(BIGINT) ignores ANSI and TRY overflow semantics. try_sum returns a wrapped negative value instead of NULL, and ANSI sum returns that value instead of throwing.

Steps to reproduce

With Comet and native shuffle enabled:

SET spark.sql.adaptive.enabled=false;
CREATE TABLE sliding_sum_repro (id INT, v BIGINT) USING parquet;
INSERT INTO sliding_sum_repro VALUES
  (1, 9223372036854775807), (2, 1), (3, -1);

SELECT id, try_sum(v) OVER (
  ORDER BY id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
) AS s
FROM sliding_sum_repro;

Also run with sum instead of try_sum and spark.sql.ansi.enabled=true.

Expected behavior

For id = 2:

Expression Spark Comet
try_sum, ANSI on or off NULL -9223372036854775808
sum, ANSI on ARITHMETIC_OVERFLOW -9223372036854775808

Both engines return 0 for the third try_sum row. Legacy sum wraps in both engines as expected.

Additional context

Reproduced on main b7f35b6ac, Spark 3.5.9 and 4.1.3, with CometWindowExec asserted and allowIncompatible=false.

process_agg_func uses the mode-aware SumInteger only for ever-expanding frames. Sliding frames use DataFusion's built-in sum, which wraps and does not receive the evaluation mode. #4729 / #4732 guarded the corresponding decimal case, but not integral ANSI/TRY sums.

A focused fix could fall back for these integral sliding frames while retaining native legacy and ever-expanding sums.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsbugSomething isn't workingpriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions