You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Offset overflow - Operators that expand a large string scalar without a batch-size bound #7689
#7316 bounds the rows per batch that reach an operator expanding a large string or binary scalar, by putting a CometBatchSplitExec under it. It covers projection, filter, hash aggregate, sort with a limit (TopK), and the hash and range shuffle writers. Other native operators evaluate expressions over a batch and expand a scalar the same way, but they get no split, so a large scalar can still fail with byte array offset overflow.
These were raised in the review of #7316 and are not yet covered:
Join filter. The residual join condition is built as a DataFusion JoinFilter in native/core/src/execution/planner/join.rs, and nothing bounds it. It is evaluated over the matched index pairs, so the bound is the batch size, not the input width. A residual that references both sides stays a join condition, for example:
SELECTcount(*) FROM a JOIN b
ONa.k=b.kAND length(concat(repeat('x', 262144), a.s||b.s)) >0
The same query in a plain DataFusion session (default 8192-row batch, a hash join with one matching key) fails with Execution error: byte array offset overflow. It has not yet been run end to end through Comet.
ExpandExec.native/operators/src/expand.rs calls into_array(batch.num_rows()) on each projection expression, so a large scalar in a grouping-sets projection is expanded to the full batch length. Not yet reproduced.
Window partition and order expressions. These are evaluated over the batch the window receives, which can be a whole partition. Not yet reproduced.
Sort without a limit.fix: bound the batch under the operator that expands a large string scalar #7316 puts the split under every sort, but the external sorter concatenates its in-memory input into larger runs before evaluating the keys, so the split under it does not bound what the key expressions see. Not yet reproduced.
Steps to reproduce
See the queries above. For comparison, the cases #7316 fixed are in spark/src/test/resources/sql-tests/operators/large_scalar_string.sql.
Expected behavior
Each query returns the same result as Spark, as the operators covered by #7316 now do.
Activity