You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native window operators reserve no memory for the batches they buffer #6253
DataFusion's window operators register no MemoryConsumer (nothing under src/windows/ in datafusion-physical-plan 55.1.0 reserves memory). So whatever Comet's native windows buffer is invisible to Comet's memory pool and to Spark.
The worst case is WindowAggExec. The planner uses it whenever a window expression can't run in bounded memory (planner.rs#L2420-L2446): PERCENT_RANK, CUME_DIST, NTILE, and any aggregate frame that ends at UNBOUNDED FOLLOWING, which includes Spark's default frame for agg(...) OVER (PARTITION BY k). WindowAggExec keeps every input batch of the task's partition (window_agg_exec.rs:606) and concatenates them before evaluating (window_agg_exec.rs:539). A large or skewed partition can then hold a multiple of its input size natively with no reservation behind it. BoundedWindowAggExec buffers only one window partition at a time, but it doesn't reserve either.
Spark's WindowExec buffers through ExternalAppendOnlyUnsafeRowArray, which spills. The native path instead grows until the executor hits its container limit, and the pool never gets a chance to refuse anything. A string column that concatenates past 2 GiB fails the concat outright.
Steps to reproduce
I haven't measured this yet. A traced run (spark.comet.tracing.enabled=true) of PERCENT_RANK() OVER (PARTITION BY k ORDER BY v) over a skewed key should show native_allocated running ahead of the pools' reserved total by about the size of the largest partition.
Expected behavior
Native windows reserve what they buffer. A partition that doesn't fit then fails the task with a memory error, or spills once DataFusion supports it, instead of growing unaccounted.
Additional context
Spilling for WindowAggExec is apache/datafusion#22946. Until that lands, could we reserve the batches WindowAggExec buffers, so an oversized partition fails the task instead of the executor?
I'm closing #6279 without merging. This records what it did, what it measured, and what is left, so the work can be picked up again. The branch is kept at andygrove/datafusion-comet:fix-6253-window-memory (head b87b0dc964, mergeable against main when it was closed).
The gap
DataFusion's window operators register no MemoryConsumer. WindowAggExec, which Comet uses for PERCENT_RANK, CUME_DIST, NTILE and any aggregate whose frame ends at UNBOUNDED FOLLOWING (including Spark's default frame for agg(x) OVER (PARTITION BY k)), buffers the task's whole input and concatenates it without telling the pool. DataFusion 56 (branch-56 at 964f925) is unchanged here: nothing under datafusion/physical-plan/src/windows/ reserves memory or spills. Upstream spilling is apache/datafusion#22946, still open.
A new CometWindowAggExec (native/core/src/execution/operators/window_agg.rs, wired in at planner/window.rs) that evaluates the same WindowExprs one window partition at a time. It emits each window partition as soon as the sorted input moves past it and keeps only the one that may continue. Without PARTITION BY it still buffers everything.
It reserves what it buffers with RecordBatchMemoryCounter (shared buffers counted once) and reserves the concatenated copy by counting only the buffers the copy does not share with its inputs. Per-batch get_slice_memory_size estimates over-charged batches sliced from one list array about tenfold.
It registers with the pool when it buffers its first batch, not in execute(). Registering earlier shrinks the sort's share under fair_unified while the sort is still reading.
A refused reservation says the operator cannot spill, that a window partition needs up to twice its size within the operator's share, and which settings to change (spark.memory.offHeap.size, then greedy_unified, then spark.comet.exec.window.enabled=false).
Reservation alone was not enough. A build that only reserved the buffered batches failed SUM(v) OVER (PARTITION BY k) over 200,000 rows in 5-row window partitions under a 4 MB pool, because WindowAggExec holds the whole task input, not one window partition. The streaming emission is what makes the reservation usable.
Measurements (standalone bench, release, 2M rows in 8192-row batches)
Within 5% of WindowAggExec for 10-row and 100,000-row window partitions and with no PARTITION BY. 6-15% slower with 1,000-row window partitions (about 0.5 ms per 2M rows), because it concatenates per batch.
20M rows (480 MB) in 1,000-row window partitions: WindowAggExec peaked at 1,254 MB RSS, reserved 0 and emitted one 20M-row batch. CometWindowAggExec peaked at 480 MB (the input itself), reserved 609 KB at peak and emitted 9,000-row batches.
Without PARTITION BY both peak at 1,090 MB, and CometWindowAggExec reserves 960 MB of it.
Tests worth reusing
A Rust differential test against WindowAggExec over 20 seeds (random window partition sizes, random batch boundaries including empty batches, null keys, two key columns, with and without PARTITION BY), plus mutation-checked memory tests: no batch reservation, no copy reservation, no release after emitting, early registration, never continuing a window partition across batches, not merging a continued window partition.
windows/window_whole_partition_batches.sql (whole-partition functions over 7-row batches).
Two CometWindowExecSuite tests: many small window partitions under a 4 MB pool pass and match Spark, and a window partition that does not fit fails the task with the CometWindowAggExec cannot spill hint.
Spark 4.1.3 DataFrameWindowFunctionsSuite and DataFrameWindowFramesSuite gave the same result on main and on the branch (74 pass, the same 4 fail, all marked IgnoreComet).
What was not done
The fair_unified TPC-DS window query with a pool small enough to make the sort spill, compared against main. It was asked for in review. The standalone bench does not cover whether the share dilution or the per-batch pool calls cost anything there.
Spilling. A window partition that exceeds the operator's share fails the task where Spark would spill. In on-heap mode the pool is unbounded, so nothing is refused there.
Open question for whoever resumes this
Comet carries its own copy of WindowAggStream and about ten delegated methods through every DataFusion upgrade. The alternatives are to offer the streaming-emission stream upstream as the no-spill first step of apache/datafusion#22946, or to wait for that issue to get a design. Two earlier upstream attempts (apache/datafusion#22947 and apache/datafusion#23207) were closed unmerged, and apache/datafusion#23207 only reserved while still buffering the whole input. As of 2026-11-17 I had not found an upstream design that emits completed window partitions early. Review feedback on #6279 suggested merging the DataFusion 56 upgrade first and then reanalyzing what is still open here.
Activity
andygrove commentedon Nov 19, 2026
I'm closing #6279 without merging. This records what it did, what it measured, and what is left, so the work can be picked up again. The branch is kept at
andygrove/datafusion-comet:fix-6253-window-memory(headb87b0dc964, mergeable against main when it was closed).The gap
DataFusion's window operators register no
MemoryConsumer.WindowAggExec, which Comet uses forPERCENT_RANK,CUME_DIST,NTILEand any aggregate whose frame ends atUNBOUNDED FOLLOWING(including Spark's default frame foragg(x) OVER (PARTITION BY k)), buffers the task's whole input and concatenates it without telling the pool. DataFusion 56 (branch-56at964f925) is unchanged here: nothing underdatafusion/physical-plan/src/windows/reserves memory or spills. Upstream spilling is apache/datafusion#22946, still open.What #6279 did
CometWindowAggExec(native/core/src/execution/operators/window_agg.rs, wired in atplanner/window.rs) that evaluates the sameWindowExprs one window partition at a time. It emits each window partition as soon as the sorted input moves past it and keeps only the one that may continue. WithoutPARTITION BYit still buffers everything.RecordBatchMemoryCounter(shared buffers counted once) and reserves the concatenated copy by counting only the buffers the copy does not share with its inputs. Per-batchget_slice_memory_sizeestimates over-charged batches sliced from one list array about tenfold.execute(). Registering earlier shrinks the sort's share underfair_unifiedwhile the sort is still reading.spark.memory.offHeap.size, thengreedy_unified, thenspark.comet.exec.window.enabled=false).Reservation alone was not enough. A build that only reserved the buffered batches failed
SUM(v) OVER (PARTITION BY k)over 200,000 rows in 5-row window partitions under a 4 MB pool, becauseWindowAggExecholds the whole task input, not one window partition. The streaming emission is what makes the reservation usable.Measurements (standalone bench, release, 2M rows in 8192-row batches)
WindowAggExecfor 10-row and 100,000-row window partitions and with noPARTITION BY. 6-15% slower with 1,000-row window partitions (about 0.5 ms per 2M rows), because it concatenates per batch.WindowAggExecpeaked at 1,254 MB RSS, reserved 0 and emitted one 20M-row batch.CometWindowAggExecpeaked at 480 MB (the input itself), reserved 609 KB at peak and emitted 9,000-row batches.PARTITION BYboth peak at 1,090 MB, andCometWindowAggExecreserves 960 MB of it.Tests worth reusing
WindowAggExecover 20 seeds (random window partition sizes, random batch boundaries including empty batches, null keys, two key columns, with and withoutPARTITION BY), plus mutation-checked memory tests: no batch reservation, no copy reservation, no release after emitting, early registration, never continuing a window partition across batches, not merging a continued window partition.windows/window_whole_partition_batches.sql(whole-partition functions over 7-row batches).CometWindowExecSuitetests: many small window partitions under a 4 MB pool pass and match Spark, and a window partition that does not fit fails the task with theCometWindowAggExec cannot spillhint.DataFrameWindowFunctionsSuiteandDataFrameWindowFramesSuitegave the same result on main and on the branch (74 pass, the same 4 fail, all markedIgnoreComet).What was not done
fair_unifiedTPC-DS window query with a pool small enough to make the sort spill, compared against main. It was asked for in review. The standalone bench does not cover whether the share dilution or the per-batch pool calls cost anything there.BoundedWindowAggExecreserves nothing and is not covered. ARANGE ... CURRENT ROWframe keeps a whole group of peer rows, so a skewedORDER BYvalue can still grow untracked. That is BoundedWindowAggExec reserves no memory for the rows its window frames keep #7092.Open question for whoever resumes this
Comet carries its own copy of
WindowAggStreamand about ten delegated methods through every DataFusion upgrade. The alternatives are to offer the streaming-emission stream upstream as the no-spill first step of apache/datafusion#22946, or to wait for that issue to get a design. Two earlier upstream attempts (apache/datafusion#22947 and apache/datafusion#23207) were closed unmerged, and apache/datafusion#23207 only reserved while still buffering the whole input. As of 2026-11-17 I had not found an upstream design that emits completed window partitions early. Review feedback on #6279 suggested merging the DataFusion 56 upgrade first and then reanalyzing what is still open here.Related DataFusion window work filed by comphead: apache/datafusion#25637 (spilling coverage epic), and apache/datafusion#26537, apache/datafusion#26539 and apache/datafusion#26541 (
BoundedWindowAggExecbuffering and per-partition cost).