Skip to content

fix: bound the batch under the operator that expands a large string scalar - #7316

Open
zhangfengcdt wants to merge 13 commits into
apache:mainfrom
zhangfengcdt:fix/scalar-string-batch-cap
Open

zhangfengcdt wants to merge 13 commits into
apache:mainfrom
zhangfengcdt:fix/scalar-string-batch-cap

Conversation

@zhangfengcdt

@zhangfengcdt zhangfengcdt commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #7247. The operators that are still unbounded are tracked in #7689.

Rationale for this change

A string scalar, from a scalar subquery or a literal, is expanded to the batch length by aggregates, the hash partitioner and string kernels. A 256 KiB string over the default 8192-row batch needs 2 GiB of value bytes, past Arrow's 32-bit Utf8 offsets, so native execution panics with offset overflow or fails with byte array offset overflow. Spark evaluates such scalars row by row and succeeds.

The expansion happens in DataFusion (aggregate argument evaluation, concat) as well as in Comet's shuffle partitioner, and a plan-time fallback cannot see a subquery's value. So the fix bounds the rows per batch that reach the expanding operator.

The first version of this PR lowered the session batch size and sliced batches at the scans. Review showed that does not bound what reaches the operator: a filter coalesces its output to its own batch size, a window emits its whole partition, and a scalar built from a subquery result (repeat(<subquery>, 2)) is wider than the result the planner measured. This version enforces the bound at the operator instead.

What changes are included in this PR?

  • A new CometBatchSplitExec in the operators crate, a thin operator over DataFusion's BatchSplitStream, which emits an input batch in slices of at most batch_size rows. Slices share buffers.
  • The planner wraps the input of a projection, filter, hash aggregate (grouping, aggregate inputs and filters), sort with a limit (its keys) or shuffle writer with hash or range partitioning in a CometBatchSplitExec whenever that operator expands a string or binary scalar. The split size is the smaller of the session batch size and the rows at which the scalars, expanded, reach 1 GiB; for the issue's string that is 4096 rows. No width is assumed for what arrives from below, since a filter coalesces toward DataFusion's default, a window emits its whole partition and a broadcast arrives as a few wide batches. A plan without such a scalar is unchanged.
  • A scalar is any column-free subtree of Utf8 or Binary type. The planner evaluates it once on a one-row batch, so a scalar built by an expression counts at its final width. A scalar that a comparison (=, <>, <, <=> and the other operators DataFusion evaluates as a Datum) or an IN list takes as a constant is compared and never expanded, so it is skipped: payload = 'x' installs no split. An operand computed per row, such as the IN member concat(repeat('x', 262144), CAST(id AS STRING)), counts the scalars it contains. A column, a volatile expression such as uuid(), or a JVM UDF is a per-row node: it is never evaluated at plan time, but the scalars beside it are still counted, so concat(repeat('x', 262144), uuid()) is measured at 256 KiB.
  • The split under a filter is transparent to join runtime filtering: the join planner aligns the Spark and native trees through it, so the filter keeps its pushdown permission, and reader attachment crosses it to reach the Parquet scan, since slicing batches discards no rows.
  • executePlan pulls JVM input only when the plan waits for it, as its first call already did, instead of pulling the next batch up front on every call. A limit can then finish on rows a split still holds without reading another batch.
  • The session batch size, ScanExec, ShuffleScanExec and the range source are unchanged from main.

How are these changes tested?

  • Fixture operators/large_scalar_string.sql with thirteen queries that fail on main with byte array offset overflow: the grouped count, the hash partitioning key, concat with a subquery and with a literal, the literal over a native range, a filter under the projection, a window under the aggregate, a 64 KiB scalar over a window partition wider than the session batch size, a scalar beside uuid(), repeat of the subquery result, an IN list with a member computed per row, and a sort key under a limit. All run in query mode, so the plan is checked to be native. NullPropagation is excluded so the grouped count keeps its literal argument rather than becoming count(1).
  • Fixture operators/large_scalar_string_small_batch.sql runs the filter case with spark.comet.batchSize=256, where the scalar fits the session batch size as is.
  • CometExecIteratorLifecycleSuite: a 10-row JVM batch followed by an input that throws, under concat(s, '!') with batch size 2 and LIMIT 3, returns three rows without reading the input again.
  • CometNativeShuffleSuite: a range shuffle whose ordering expression is a 256 KiB scalar, with no projection between the exchange and the range source.
  • CometJoinSuite: the deterministic residual test also runs a string residual, concat(CAST(p.payload AS STRING), '!') <> '2506!', and checks the reader filter still attaches and prunes row groups with the split under the filter. Rust unit tests cover the filter keeping its permission through the split and reader attachment crossing it.
  • Rust unit tests for CometBatchSplitExec and for the scalar measurement: literals and columns, a scalar built by an expression, a scalar beside a volatile sibling that is not evaluated, IN list members, a scalar compared directly against one computed per row, the planner wrapping the input only when the scalars need a smaller batch, and the split going under a sort with a limit but not under a full sort.
  • CometSqlFileTestSuite for operators/, scalar_subquery and shuffle, CometNativeShuffleSuite and CometJoinSuite, on Spark 4.1, and the core and operators crate test suites.

A string scalar from a literal or a scalar subquery is expanded to the
batch length by aggregates, the hash partitioner and string kernels. A
256 KiB string over an 8192-row batch needs 2 GiB of value bytes, past
Arrow's offsets, and native execution panics where Spark succeeds.

Add up the string and binary scalars while planning and lower the
session batch size so their expansion stays within 1 GiB. Native sources
read the batch size when they run, the range source now included, and
the JVM-fed leaves slice wider batches to it.
@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Oct 29, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowering the task's batch size from the planned scalar bytes reaches every operator that reads the batch size from the TaskContext at execute time, and the arithmetic in scalar_batch_rows is safe for zero and oversized scalars. Three comments, on an operator that regrows batches, on literals that are never expanded, and on slicing at default settings. Nothing here was run.

Comment thread native/core/src/execution/jni_api.rs Resolved
Comment thread native/core/src/execution/planner.rs Resolved

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the new SQL fixture on Spark 4.1.3 against e9b8b0c3. It passed in normal mode. I’m requesting changes because the scan can now pull an unneeded JVM batch and make a previously successful LIMIT fail. The review also covers the nullable-batch slicing cost, the grouped test’s strict-mode failure, and scalar expansions that still overflow. I added the filter and window results to the existing batch-size discussion.

Comment thread native/core/src/execution/planner.rs Resolved
…calar

Rework after review. Capping the input batch size does not bound what
reaches the expanding operator: a filter coalesces its output to its own
batch size, a window emits its whole partition, and a scalar built from
a subquery result can be wider than the result. The scan slicers also
dropped a pending slice under LIMIT and double counted output_rows.

A new CometBatchSplitExec, over DataFusion's BatchSplitStream, is placed
right under a projection, filter, hash aggregate or hash shuffle writer
when that operator's string and binary scalars, expanded to the session
batch size, would exceed 1 GiB. The planner measures each column-free
subtree of those types by evaluating it once on a one-row batch, so that
repeat(<subquery>, n) counts at its final width. IN list members are
compared and never expanded, so they are skipped. The session batch size,
the scan and shuffle scan operators and the range source are unchanged.
@zhangfengcdt zhangfengcdt changed the title fix: cap the rows per batch so a large string scalar fits 32-bit offsets fix: bound the batch under the operator that expands a large string scalar Oct 30, 2026
uuid(0) and randstr(10, 0) are column-free string expressions, so the
planner evaluated them once at plan time to measure their width, which
drew the first value from their seeded generator. Execution then
produced Spark's second value onward and the bit-for-bit tests failed.

A subtree with a volatile expression yields a value per row rather than
a scalar, so it is never measured: Comet's stateful expressions and any
DataFusion scalar function declared volatile.
@zhangfengcdt

Copy link
Copy Markdown
Member Author

Thanks @comphead and @sunchao ! Agreed that capping the input batch size cannot bound what reaches the expanding operator, and it is not a optimal solution and fix. So I reworked the fix to enforce the bound at the operator, with the following changes:

  • A new CometBatchSplitExec over DataFusion's BatchSplitStream is placed directly under a projection, filter, hash aggregate or hash shuffle writer, only when that operator's string scalars would exceed 1 GiB expanded. Plans without a large scalar are unchanged.
  • Scalars are measured at plan time by evaluating each column-free string subtree once, so repeat(<subquery>, 8) counts at its final width. IN list members, volatile expressions and JVM UDFs are skipped.
  • The fixture covers the WHERE, window and repeat cases, all in query mode so it passes strict testing. The grouped count uses a literal; the missingInput issue with a grouped count of a subquery is separate.

Appreciate if you both could take another look and let me know if you'd refine it or go with other directions. Thanks!

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting the splitter below the expanding expression fixes the original placement problem. There is still a case where the splitter disappears.

…he scalar cap

The split was installed only when the scalar cap fell below the session
batch size. With spark.comet.batchSize=256 a 256 KiB scalar allows 4096
rows, so nothing was installed, but DataFusion's filter coalesces toward
its own default of 8192 rows whatever the session says, and the
projection above it overflowed the 32-bit offsets.

The decision now compares the cap against the larger of the session
batch size and DataFusion's default, and splits at the smaller of the
cap and the session size. A new fixture runs the filter case with a
256-row session.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built 9e4b6b05 and ran the new small-session filter fixture successfully. The earlier filter case is covered. I reproduced an offset overflow on the window path and found that Spark simplifies away the wide argument in the grouped-count test.

The split was installed only when the scalar cap fell below the widest
batch assumed to arrive, the larger of the session batch size and
DataFusion's default. A window emits its whole partition and a broadcast
arrives as a few wide batches, so no width can be assumed: a 64 KiB
scalar over a 32768-row window partition skipped the split and
overflowed the 32-bit offsets.

An operator that expands any string or binary scalar now has its input
split at the smaller of the scalar cap and the session batch size. A
plan without such a scalar is unchanged.

The fixture excludes NullPropagation so that the grouped count keeps the
wide literal as its argument instead of being rewritten to count(1), and
adds the window case above.
@zhangfengcdt

Copy link
Copy Markdown
Member Author

I built 9e4b6b05 and ran the new small-session filter fixture successfully. The earlier filter case is covered. I reproduced an offset overflow on the window path and found that Spark simplifies away the wide argument in the grouped-count test.

THanks @sunchao for the comments! Both should be fixed now.

…ch-cap

# Conflicts:
#	native/core/src/execution/planner.rs

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 949f550 across five independent scopes, including the new merge from main. The filter merge preserves both the bounded input and session batch size. The operator-level split and revised fixtures address the previous window/input-size and NullPropagation coverage gaps. Two paths still overflow: a deterministic large scalar beneath a volatile expression, and a large scalar in a range-shuffle key. Both have executed Spark/default-batch/small-batch controls in the inline comments. CI (fork) currently requires contributor action.

…keys

The plan-time measurement stopped at the first volatile expression, so
a column-free subtree such as concat(repeat('x', 262144), uuid()) was
measured as zero bytes and left unbounded. A column, a volatile
function, a JVM UDF or a subquery is now a boundary rather than a
reason to give up: the scalars among its siblings are still counted,
and the volatile node itself is never evaluated.

The shuffle writer only bounded hash partitioning keys. Range
partitioning evaluates its ordering expressions in the exchange too,
so those are bounded the same way.
@zhangfengcdt

Copy link
Copy Markdown
Member Author

Reviewed 949f550 across five independent scopes, including the new merge from main. The filter merge preserves both the bounded input and session batch size. The operator-level split and revised fixtures address the previous window/input-size and NullPropagation coverage gaps. Two paths still overflow: a deterministic large scalar beneath a volatile expression, and a large scalar in a range-shuffle key. Both have executed Spark/default-batch/small-batch controls in the inline comments. CI (fork) currently requires contributor action.

Thanks for the review again, @sunchao. It should be fixed now.

The measurement now treats a column, volatile function or JVM UDF as a boundary and keeps counting the scalars beside it, so concat(repeat('x', 262144), uuid()) is bounded without evaluating uuid() at plan time.

I also added a fixture query and a CometNativeShuffleSuite test similar to the test cases you reported.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 30ee2c7 across five independent scopes. The earlier volatile-sibling and range-shuffle findings are addressed, and all eight committed scalar-expansion unit tests pass. One dynamic-IN path still bypasses the bound. I confirmed it with an added native estimator test, then reproduced the query failure using freshly compiled JVM code and the exact-head native library on Spark 4.1.3. Spark and the smaller-batch native control both succeed. Current-head GitHub build/test jobs were skipped, so the validation here is local.

Comment thread native/core/src/execution/planner.rs Resolved
…o join runtime filters

The measurement only looked at the value an IN list tests, so a list
member computed per row, such as concat(repeat('x', 262144),
CAST(id AS STRING)), expanded its scalar unbounded. The scalars inside
such a member now count; a constant member is still compared without
expansion and skipped.

The split that bounds a filter's input sat between the filter and its
Spark child, which broke the pointer alignment the join planner uses
to carry a filter's runtime-filter pushdown permission, and stopped
reader attachment on its way to the Parquet scan. Both now look through
the split and rebuild it over the prepared or filtered child, since
slicing batches discards no rows.
@github-actions github-actions Bot added the area:joins Join operators and dynamic filter pushdown label Nov 6, 2026
@zhangfengcdt

Copy link
Copy Markdown
Member Author

Reviewed 30ee2c7 across five independent scopes. The earlier volatile-sibling and range-shuffle findings are addressed, and all eight committed scalar-expansion unit tests pass. One dynamic-IN path still bypasses the bound. I confirmed it with an added native estimator test, then reproduced the query failure using freshly compiled JVM code and the exact-head native library on Spark 4.1.3. Spark and the smaller-batch native control both succeed. Current-head GitHub build/test jobs were skipped, so the validation here is local.

Thanks @sunchao @rich7420! These should be fixed now.

…ch-cap

# Conflicts:
#	native/core/src/execution/planner/join.rs

@rich7420 rich7420 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhangfengcdt thanks for the update!

A scalar operand of a comparison is compared as a Datum and never expanded,
so `payload = 'x'` no longer installs a split; an operand computed per row
still counts the scalars it contains. The sort bounds its input with its keys,
which covers the TopK under a limit.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

Moving the bound from the scans to the operator that evaluates the expression is the right shape, and the estimator that measures a scalar at its final width answers the earlier threads well. The remaining concern is coverage. The split is installed for the projection, filter, hash aggregate, sort and the hash and range shuffle writers, but other native operators evaluate user expressions over a batch and expand a scalar the same way. The one I checked is the join filter. Comet builds it as a DataFusion JoinFilter in planner/join.rs and nothing calls bound_scalar_expansion for it. A residual such as ON a.k = b.k AND length(concat(repeat('x', 262144), a.s || b.s)) > 0 references both sides, so Spark keeps it as the join condition and it does not become a Filter that this PR covers. I ran the equivalent query in a DataFusion session with the default 8192-row batch size over a hash join with a single matching key. It failed with Execution error: byte array offset overflow. I did not run it end to end through Comet, and I did not get a passing control at a small batch size. The join filter evaluates over the matched index pairs, so the bound is the batch size and not the input width.

ExpandExec in native/operators/src/expand.rs (it calls into_array(batch.num_rows()) on each projection expression) and the window partition and order expressions look like the same class. Could the PR either bound the join filter (and say which of the other operators were checked), or open a tracking issue that lists the operators still unbounded so #7247 does not read as fully closed? The join filter case could also be added to large_scalar_string.sql once it is covered.

On behavior against branch-1.2: queries with a string scalar of ordinary size now get an extra CometBatchSplitExec at the session batch size under a projection, aggregate, sort or shuffle writer. Narrow batches pass straight through, so I expect the cost to be small. Results do not change. The queries that overflowed on the release now succeed in the covered operators.

…ch-cap

# Conflicts:
#	native/core/src/execution/planner/aggregate.rs
@zhangfengcdt

Copy link
Copy Markdown
Member Author

Could the PR either bound the join filter (and say which of the other operators were checked), or open a tracking issue that lists the operators still unbounded so #7247 does not read as fully closed?

Thanks @andygrove for the review! I opened #7689 to track the operators that are still unbounded: the join filter, ExpandExec, the window partition and order expressions, and the sort without a limit. This PR covers the projection, filter, hash aggregate, sort with a limit, and the hash and range shuffle writers. I changed the description to "Part of #7247" so the issue stays open until those are covered as well.

I also merged main into this branch to resolve the conflict with the new MinMaxReaderFilterExec.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e8eae446e82eff57703b4b9b1044a6d6b022730e across five independent scopes, including the merge with main. The dynamic-IN, comparison, TopK and runtime-filter revisions address the corresponding earlier comments. The remaining expansion limitations are now tracked in #7689.

One inline comment covers a new exposure to the known DataFusion sort-accounting bug. The earlier pending-slice/read-ahead concern also still applies: ScanStream clears its input slot after handing a wide batch to the splitter, but execute_plan calls pull_input_batches before polling the next output. For a projection such as concat(s, '!'), a ten-row first JVM batch and session batch size two, LIMIT 3 can therefore read a failing second input before returning its already-buffered third row. Moving the splitter out of the scan does not make that refill demand-driven. Please retain a regression at the JVM-fed driver boundary; I am referencing the existing thread rather than opening a duplicate.

Compared with branch-1.2, the local expansion bound is intentional, while those two interactions affect previously successful paths. Validation was source inspection, including checksum-verified lockfile-matching DataFusion crates. No local tests, binaries, benchmarks or executable probes ran. At publication, this head had 2 successful checks, 3 queued checks and CI (fork) requiring action because no matching fork run was found. Earlier-head test results are not validation of this commit.

Comment thread native/core/src/execution/planner.rs Outdated
let sort_plan = SortExec::new(
LexOrdering::new(exprs?).unwrap(),
// The sort evaluates its keys on each batch it receives.
let input = self.bound_scalar_expansion(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid exposing ordinary sorts to the shared-buffer reservation bug

Could the ordinary-sort path avoid handing ExternalSorter zero-copy slices of one large allocation until its shared-buffer accounting is fixed? SELECT id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576) SORT BY concat(CAST(id AS STRING), '!') reaches a full-partition WindowAggExec followed by this sort. Even the one-byte '!' installs the splitter at 8192 rows, producing 128 slices of the same roughly 16 MiB buffers. In the pinned DataFusion 55.2.0, insert_batch reserves get_record_batch_memory_size(batch) + batch.get_sliced_size() for each slice independently. That attempts about 2 GiB plus 16 MiB of reservations, versus about 32 MiB for the unsplit batch, and spills when the pool refuses them. The physical input buffers remain shared.

The underlying bug already exists and is tracked in DataFusion #25804, finding 5, with #25820 providing the upstream fix. The regression here is newly exposing a previously unsplit window-to-sort path to it, including tiny scalars with no offset-overflow risk. Please avoid that new exposure, compact/account the shared inputs correctly, or sequence this path after the dependency fix, with coverage for a wide producer and a small scalar sort key.

A full sort concatenates its input before evaluating its keys, so the split
did not bound it, and DataFusion's ExternalSorter reserves each slice of a
shared buffer as if it owned the buffer (apache/datafusion#25804).

executePlan no longer pulls the next JVM batch on every call. It pulls only
when the plan waits for input, so a limit can finish on rows a split still
holds without reading another batch.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at a5708d07ed. Pulling JVM input only from the pending branch fixes the LIMIT read-ahead, and the new CometExecIteratorLifecycleSuite case covers it at the JVM-fed boundary. My three points from e9b8b0c3bc are addressed. @sunchao's shared-buffer point is closed for sort keys but still reaches a full sort through a projection, and one doc comment on the pull path is now stale.

let batch_size = self.session_ctx.copied_config().batch_size();
Arc::new(BatchSplitExec::new(
input,
scalar_rows_cap(bytes).min(batch_size),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sort-key split is gone, but the exposure @sunchao raised at e8eae446e8 still reaches a full sort through a projection. A one-byte scalar gives bytes == 1, so this installs a split at the session batch size, and ProjectionExec passes its column references through as zero-copy slices. I expect SELECT id, cd, concat(CAST(id AS STRING), '!') AS k FROM (SELECT id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576)) SORT BY k to hand ExternalSorter 128 slices of the window's id and cd buffers, which insert_batch reserves one by one as in the earlier report. I have not run it. Splitting at scalar_rows_cap(bytes) alone would still keep every expansion within 1 GiB, and BatchSplitStream passes a batch at or under that size straight through, so a scalar of ordinary size would never slice. The unit tests that assert the session batch size here would change with it.

Comment on lines +1645 to +1647
// JVM input is pulled only when the stream waits for it, in `next_batch` below. An operator
// above a scan, such as a `BatchSplitExec`, can still hold rows of the last batch it took,
// and a limit can finish on them without the JVM producing another batch.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment on pull_input_batches still says "we pull input batches here and insert them into scan operators before polling the stream". After this change it only runs from the pending branch of next_batch, after a poll, so that comment should change with this one. It also still fills every JVM-fed scan whose slot is empty, not only the one the plan waits on, so with two JVM inputs a pending poll on one side still reads the other side ahead. That part is unchanged from main, but "only when the stream waits for it" reads stronger than what happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:joins Join operators and dynamic filter pushdown area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants