Skip to content

Offset overflow - Operators that expand a large string scalar without a batch-size bound #7689

Open

Description

@zhangfengcdt

Describe the bug

#7316 bounds the rows per batch that reach an operator expanding a large string or binary scalar, by putting a CometBatchSplitExec under it. It covers projection, filter, hash aggregate, sort with a limit (TopK), and the hash and range shuffle writers. Other native operators evaluate expressions over a batch and expand a scalar the same way, but they get no split, so a large scalar can still fail with byte array offset overflow.

These were raised in the review of #7316 and are not yet covered:

  • Join filter. The residual join condition is built as a DataFusion JoinFilter in native/core/src/execution/planner/join.rs, and nothing bounds it. It is evaluated over the matched index pairs, so the bound is the batch size, not the input width. A residual that references both sides stays a join condition, for example:

    SELECT count(*) FROM a JOIN b
      ON a.k = b.k AND length(concat(repeat('x', 262144), a.s || b.s)) > 0

    The same query in a plain DataFusion session (default 8192-row batch, a hash join with one matching key) fails with Execution error: byte array offset overflow. It has not yet been run end to end through Comet.

  • ExpandExec. native/operators/src/expand.rs calls into_array(batch.num_rows()) on each projection expression, so a large scalar in a grouping-sets projection is expanded to the full batch length. Not yet reproduced.

  • Window partition and order expressions. These are evaluated over the batch the window receives, which can be a whole partition. Not yet reproduced.

  • Sort without a limit. fix: bound the batch under the operator that expands a large string scalar #7316 puts the split under every sort, but the external sorter concatenates its in-memory input into larger runs before evaluating the keys, so the split under it does not bound what the key expressions see. Not yet reproduced.

Steps to reproduce

See the queries above. For comparison, the cases #7316 fixed are in spark/src/test/resources/sql-tests/operators/large_scalar_string.sql.

Expected behavior

Each query returns the same result as Spark, as the operators covered by #7316 now do.

Additional context

Follow-up to #7247 and #7316. Part of #7267.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions