Repository navigation
fix: bound the batch under the operator that expands a large string scalar - #7316
zhangfengcdt wants to merge 13 commits into
Conversation
A string scalar from a literal or a scalar subquery is expanded to the batch length by aggregates, the hash partitioner and string kernels. A 256 KiB string over an 8192-row batch needs 2 GiB of value bytes, past Arrow's offsets, and native execution panics where Spark succeeds. Add up the string and binary scalars while planning and lower the session batch size so their expansion stays within 1 GiB. Native sources read the batch size when they run, the range source now included, and the JVM-fed leaves slice wider batches to it.
comphead
left a comment
There was a problem hiding this comment.
Lowering the task's batch size from the planned scalar bytes reaches every operator that reads the batch size from the TaskContext at execute time, and the arithmetic in scalar_batch_rows is safe for zero and oversized scalars. Three comments, on an operator that regrows batches, on literals that are never expanded, and on slicing at default settings. Nothing here was run.
sunchao
left a comment
There was a problem hiding this comment.
I ran the new SQL fixture on Spark 4.1.3 against e9b8b0c3. It passed in normal mode. I’m requesting changes because the scan can now pull an unneeded JVM batch and make a previously successful LIMIT fail. The review also covers the nullable-batch slicing cost, the grouped test’s strict-mode failure, and scalar expansions that still overflow. I added the filter and window results to the existing batch-size discussion.
…calar Rework after review. Capping the input batch size does not bound what reaches the expanding operator: a filter coalesces its output to its own batch size, a window emits its whole partition, and a scalar built from a subquery result can be wider than the result. The scan slicers also dropped a pending slice under LIMIT and double counted output_rows. A new CometBatchSplitExec, over DataFusion's BatchSplitStream, is placed right under a projection, filter, hash aggregate or hash shuffle writer when that operator's string and binary scalars, expanded to the session batch size, would exceed 1 GiB. The planner measures each column-free subtree of those types by evaluating it once on a one-row batch, so that repeat(<subquery>, n) counts at its final width. IN list members are compared and never expanded, so they are skipped. The session batch size, the scan and shuffle scan operators and the range source are unchanged.
uuid(0) and randstr(10, 0) are column-free string expressions, so the planner evaluated them once at plan time to measure their width, which drew the first value from their seeded generator. Execution then produced Spark's second value onward and the bit-for-bit tests failed. A subtree with a volatile expression yields a value per row rather than a scalar, so it is never measured: Comet's stateful expressions and any DataFusion scalar function declared volatile.
|
Thanks @comphead and @sunchao ! Agreed that capping the input batch size cannot bound what reaches the expanding operator, and it is not a optimal solution and fix. So I reworked the fix to enforce the bound at the operator, with the following changes:
Appreciate if you both could take another look and let me know if you'd refine it or go with other directions. Thanks! |
sunchao
left a comment
There was a problem hiding this comment.
Putting the splitter below the expanding expression fixes the original placement problem. There is still a case where the splitter disappears.
…he scalar cap The split was installed only when the scalar cap fell below the session batch size. With spark.comet.batchSize=256 a 256 KiB scalar allows 4096 rows, so nothing was installed, but DataFusion's filter coalesces toward its own default of 8192 rows whatever the session says, and the projection above it overflowed the 32-bit offsets. The decision now compares the cap against the larger of the session batch size and DataFusion's default, and splits at the smaller of the cap and the session size. A new fixture runs the filter case with a 256-row session.
sunchao
left a comment
There was a problem hiding this comment.
I built 9e4b6b05 and ran the new small-session filter fixture successfully. The earlier filter case is covered. I reproduced an offset overflow on the window path and found that Spark simplifies away the wide argument in the grouped-count test.
The split was installed only when the scalar cap fell below the widest batch assumed to arrive, the larger of the session batch size and DataFusion's default. A window emits its whole partition and a broadcast arrives as a few wide batches, so no width can be assumed: a 64 KiB scalar over a 32768-row window partition skipped the split and overflowed the 32-bit offsets. An operator that expands any string or binary scalar now has its input split at the smaller of the scalar cap and the session batch size. A plan without such a scalar is unchanged. The fixture excludes NullPropagation so that the grouped count keeps the wide literal as its argument instead of being rewritten to count(1), and adds the window case above.
THanks @sunchao for the comments! Both should be fixed now. |
…ch-cap # Conflicts: # native/core/src/execution/planner.rs
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 949f550 across five independent scopes, including the new merge from main. The filter merge preserves both the bounded input and session batch size. The operator-level split and revised fixtures address the previous window/input-size and NullPropagation coverage gaps. Two paths still overflow: a deterministic large scalar beneath a volatile expression, and a large scalar in a range-shuffle key. Both have executed Spark/default-batch/small-batch controls in the inline comments. CI (fork) currently requires contributor action.
…keys
The plan-time measurement stopped at the first volatile expression, so
a column-free subtree such as concat(repeat('x', 262144), uuid()) was
measured as zero bytes and left unbounded. A column, a volatile
function, a JVM UDF or a subquery is now a boundary rather than a
reason to give up: the scalars among its siblings are still counted,
and the volatile node itself is never evaluated.
The shuffle writer only bounded hash partitioning keys. Range
partitioning evaluates its ordering expressions in the exchange too,
so those are bounded the same way.
Thanks for the review again, @sunchao. It should be fixed now. The measurement now treats a column, volatile function or JVM UDF as a boundary and keeps counting the scalars beside it, so I also added a fixture query and a CometNativeShuffleSuite test similar to the test cases you reported. |
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 30ee2c7 across five independent scopes. The earlier volatile-sibling and range-shuffle findings are addressed, and all eight committed scalar-expansion unit tests pass. One dynamic-IN path still bypasses the bound. I confirmed it with an added native estimator test, then reproduced the query failure using freshly compiled JVM code and the exact-head native library on Spark 4.1.3. Spark and the smaller-batch native control both succeed. Current-head GitHub build/test jobs were skipped, so the validation here is local.
…o join runtime filters
The measurement only looked at the value an IN list tests, so a list
member computed per row, such as concat(repeat('x', 262144),
CAST(id AS STRING)), expanded its scalar unbounded. The scalars inside
such a member now count; a constant member is still compared without
expansion and skipped.
The split that bounds a filter's input sat between the filter and its
Spark child, which broke the pointer alignment the join planner uses
to carry a filter's runtime-filter pushdown permission, and stopped
reader attachment on its way to the Parquet scan. Both now look through
the split and rebuild it over the prepared or filtered child, since
slicing batches discards no rows.
|
…ch-cap # Conflicts: # native/core/src/execution/planner/join.rs
A scalar operand of a comparison is compared as a Datum and never expanded, so `payload = 'x'` no longer installs a split; an operand computed per row still counts the scalars it contains. The sort bounds its input with its keys, which covers the TopK under a limit.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Moving the bound from the scans to the operator that evaluates the expression is the right shape, and the estimator that measures a scalar at its final width answers the earlier threads well. The remaining concern is coverage. The split is installed for the projection, filter, hash aggregate, sort and the hash and range shuffle writers, but other native operators evaluate user expressions over a batch and expand a scalar the same way. The one I checked is the join filter. Comet builds it as a DataFusion JoinFilter in planner/join.rs and nothing calls bound_scalar_expansion for it. A residual such as ON a.k = b.k AND length(concat(repeat('x', 262144), a.s || b.s)) > 0 references both sides, so Spark keeps it as the join condition and it does not become a Filter that this PR covers. I ran the equivalent query in a DataFusion session with the default 8192-row batch size over a hash join with a single matching key. It failed with Execution error: byte array offset overflow. I did not run it end to end through Comet, and I did not get a passing control at a small batch size. The join filter evaluates over the matched index pairs, so the bound is the batch size and not the input width.
ExpandExec in native/operators/src/expand.rs (it calls into_array(batch.num_rows()) on each projection expression) and the window partition and order expressions look like the same class. Could the PR either bound the join filter (and say which of the other operators were checked), or open a tracking issue that lists the operators still unbounded so #7247 does not read as fully closed? The join filter case could also be added to large_scalar_string.sql once it is covered.
On behavior against branch-1.2: queries with a string scalar of ordinary size now get an extra CometBatchSplitExec at the session batch size under a projection, aggregate, sort or shuffle writer. Narrow batches pass straight through, so I expect the cost to be small. Results do not change. The queries that overflowed on the release now succeed in the covered operators.
…ch-cap # Conflicts: # native/core/src/execution/planner/aggregate.rs
Thanks @andygrove for the review! I opened #7689 to track the operators that are still unbounded: the join filter, I also merged |
sunchao
left a comment
There was a problem hiding this comment.
Reviewed e8eae446e82eff57703b4b9b1044a6d6b022730e across five independent scopes, including the merge with main. The dynamic-IN, comparison, TopK and runtime-filter revisions address the corresponding earlier comments. The remaining expansion limitations are now tracked in #7689.
One inline comment covers a new exposure to the known DataFusion sort-accounting bug. The earlier pending-slice/read-ahead concern also still applies: ScanStream clears its input slot after handing a wide batch to the splitter, but execute_plan calls pull_input_batches before polling the next output. For a projection such as concat(s, '!'), a ten-row first JVM batch and session batch size two, LIMIT 3 can therefore read a failing second input before returning its already-buffered third row. Moving the splitter out of the scan does not make that refill demand-driven. Please retain a regression at the JVM-fed driver boundary; I am referencing the existing thread rather than opening a duplicate.
Compared with branch-1.2, the local expansion bound is intentional, while those two interactions affect previously successful paths. Validation was source inspection, including checksum-verified lockfile-matching DataFusion crates. No local tests, binaries, benchmarks or executable probes ran. At publication, this head had 2 successful checks, 3 queued checks and CI (fork) requiring action because no matching fork run was found. Earlier-head test results are not validation of this commit.
| let sort_plan = SortExec::new( | ||
| LexOrdering::new(exprs?).unwrap(), | ||
| // The sort evaluates its keys on each batch it receives. | ||
| let input = self.bound_scalar_expansion( |
There was a problem hiding this comment.
[P2] Avoid exposing ordinary sorts to the shared-buffer reservation bug
Could the ordinary-sort path avoid handing ExternalSorter zero-copy slices of one large allocation until its shared-buffer accounting is fixed? SELECT id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576) SORT BY concat(CAST(id AS STRING), '!') reaches a full-partition WindowAggExec followed by this sort. Even the one-byte '!' installs the splitter at 8192 rows, producing 128 slices of the same roughly 16 MiB buffers. In the pinned DataFusion 55.2.0, insert_batch reserves get_record_batch_memory_size(batch) + batch.get_sliced_size() for each slice independently. That attempts about 2 GiB plus 16 MiB of reservations, versus about 32 MiB for the unsplit batch, and spills when the pool refuses them. The physical input buffers remain shared.
The underlying bug already exists and is tracked in DataFusion #25804, finding 5, with #25820 providing the upstream fix. The regression here is newly exposing a previously unsplit window-to-sort path to it, including tiny scalars with no offset-overflow risk. Please avoid that new exposure, compact/account the shared inputs correctly, or sequence this path after the dependency fix, with coverage for a wide producer and a small scalar sort key.
A full sort concatenates its input before evaluating its keys, so the split did not bound it, and DataFusion's ExternalSorter reserves each slice of a shared buffer as if it owned the buffer (apache/datafusion#25804). executePlan no longer pulls the next JVM batch on every call. It pulls only when the plan waits for input, so a limit can finish on rows a split still holds without reading another batch.
comphead
left a comment
There was a problem hiding this comment.
Re-reviewed at a5708d07ed. Pulling JVM input only from the pending branch fixes the LIMIT read-ahead, and the new CometExecIteratorLifecycleSuite case covers it at the JVM-fed boundary. My three points from e9b8b0c3bc are addressed. @sunchao's shared-buffer point is closed for sort keys but still reaches a full sort through a projection, and one doc comment on the pull path is now stale.
| let batch_size = self.session_ctx.copied_config().batch_size(); | ||
| Arc::new(BatchSplitExec::new( | ||
| input, | ||
| scalar_rows_cap(bytes).min(batch_size), |
There was a problem hiding this comment.
The sort-key split is gone, but the exposure @sunchao raised at e8eae446e8 still reaches a full sort through a projection. A one-byte scalar gives bytes == 1, so this installs a split at the session batch size, and ProjectionExec passes its column references through as zero-copy slices. I expect SELECT id, cd, concat(CAST(id AS STRING), '!') AS k FROM (SELECT id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576)) SORT BY k to hand ExternalSorter 128 slices of the window's id and cd buffers, which insert_batch reserves one by one as in the earlier report. I have not run it. Splitting at scalar_rows_cap(bytes) alone would still keep every expansion within 1 GiB, and BatchSplitStream passes a batch at or under that size straight through, so a scalar of ordinary size would never slice. The unit tests that assert the session batch size here would change with it.
| // JVM input is pulled only when the stream waits for it, in `next_batch` below. An operator | ||
| // above a scan, such as a `BatchSplitExec`, can still hold rows of the last batch it took, | ||
| // and a limit can finish on them without the JVM producing another batch. |
There was a problem hiding this comment.
The doc comment on pull_input_batches still says "we pull input batches here and insert them into scan operators before polling the stream". After this change it only runs from the pending branch of next_batch, after a poll, so that comment should change with this one. It also still fills every JVM-fed scan whose slot is empty, not only the one the plan waits on, so with two JVM inputs a pending poll on one side still reads the other side ahead. That part is unchanged from main, but "only when the stream waits for it" reads stronger than what happens.
Which issue does this PR close?
Part of #7247. The operators that are still unbounded are tracked in #7689.
Rationale for this change
A string scalar, from a scalar subquery or a literal, is expanded to the batch length by aggregates, the hash partitioner and string kernels. A 256 KiB string over the default 8192-row batch needs 2 GiB of value bytes, past Arrow's 32-bit
Utf8offsets, so native execution panics withoffset overflowor fails withbyte array offset overflow. Spark evaluates such scalars row by row and succeeds.The expansion happens in DataFusion (aggregate argument evaluation,
concat) as well as in Comet's shuffle partitioner, and a plan-time fallback cannot see a subquery's value. So the fix bounds the rows per batch that reach the expanding operator.The first version of this PR lowered the session batch size and sliced batches at the scans. Review showed that does not bound what reaches the operator: a filter coalesces its output to its own batch size, a window emits its whole partition, and a scalar built from a subquery result (
repeat(<subquery>, 2)) is wider than the result the planner measured. This version enforces the bound at the operator instead.What changes are included in this PR?
CometBatchSplitExecin the operators crate, a thin operator over DataFusion'sBatchSplitStream, which emits an input batch in slices of at mostbatch_sizerows. Slices share buffers.CometBatchSplitExecwhenever that operator expands a string or binary scalar. The split size is the smaller of the session batch size and the rows at which the scalars, expanded, reach 1 GiB; for the issue's string that is 4096 rows. No width is assumed for what arrives from below, since a filter coalesces toward DataFusion's default, a window emits its whole partition and a broadcast arrives as a few wide batches. A plan without such a scalar is unchanged.Utf8orBinarytype. The planner evaluates it once on a one-row batch, so a scalar built by an expression counts at its final width. A scalar that a comparison (=,<>,<,<=>and the other operators DataFusion evaluates as aDatum) or anINlist takes as a constant is compared and never expanded, so it is skipped:payload = 'x'installs no split. An operand computed per row, such as theINmemberconcat(repeat('x', 262144), CAST(id AS STRING)), counts the scalars it contains. A column, a volatile expression such asuuid(), or a JVM UDF is a per-row node: it is never evaluated at plan time, but the scalars beside it are still counted, soconcat(repeat('x', 262144), uuid())is measured at 256 KiB.executePlanpulls JVM input only when the plan waits for it, as its first call already did, instead of pulling the next batch up front on every call. A limit can then finish on rows a split still holds without reading another batch.ScanExec,ShuffleScanExecand therangesource are unchanged frommain.How are these changes tested?
operators/large_scalar_string.sqlwith thirteen queries that fail onmainwithbyte array offset overflow: the grouped count, the hash partitioning key,concatwith a subquery and with a literal, the literal over a nativerange, a filter under the projection, a window under the aggregate, a 64 KiB scalar over a window partition wider than the session batch size, a scalar besideuuid(),repeatof the subquery result, anINlist with a member computed per row, and a sort key under a limit. All run inquerymode, so the plan is checked to be native.NullPropagationis excluded so the grouped count keeps its literal argument rather than becomingcount(1).operators/large_scalar_string_small_batch.sqlruns the filter case withspark.comet.batchSize=256, where the scalar fits the session batch size as is.CometExecIteratorLifecycleSuite: a 10-row JVM batch followed by an input that throws, underconcat(s, '!')with batch size 2 andLIMIT 3, returns three rows without reading the input again.CometNativeShuffleSuite: a range shuffle whose ordering expression is a 256 KiB scalar, with no projection between the exchange and therangesource.CometJoinSuite: the deterministic residual test also runs a string residual,concat(CAST(p.payload AS STRING), '!') <> '2506!', and checks the reader filter still attaches and prunes row groups with the split under the filter. Rust unit tests cover the filter keeping its permission through the split and reader attachment crossing it.CometBatchSplitExecand for the scalar measurement: literals and columns, a scalar built by an expression, a scalar beside a volatile sibling that is not evaluated,INlist members, a scalar compared directly against one computed per row, the planner wrapping the input only when the scalars need a smaller batch, and the split going under a sort with a limit but not under a full sort.CometSqlFileTestSuiteforoperators/,scalar_subqueryandshuffle,CometNativeShuffleSuiteandCometJoinSuite, on Spark 4.1, and the core and operators crate test suites.