fix: spill a native sort's buffered batches in groups when their output merge does not fit

[msafonov]

Problem

merchants_lifecycle (em32 9/12/5) and gcs_cost_insights_v2 fail on 1.1-9/1.1-10/1.1-11 with
Additional allocation failed for ExternalSorterMerge[0] ... consumer holds ~fair limit ... 0 bytes overcommitted, 10/10 task attempts.

The failing path is the in-memory output merge of a sort that never spilled (not the spill-file merge, which bypasses the pool and already degrades). That merge keeps the sorted batches plus the rows RowCursorStream encodes each run's sort key into. With a multi-column key every buffered batch is its own run, so the rows can outgrow what sorting frees; once the sorter has filled its share there is nothing left to spill and the merge fails on a ~0.5-1.3 MB row-buffer grow (exact prod signature reproduced in a Rust test).

Fix

Before the in-memory output merge, estimate its memory (sliced size of each run + sampled row-encoding bytes/row x rows of its current chunk, +25%; nothing extra for single-column field cursors). If the pool cannot grant what exceeds the sorter's and merge's reservations, spill the buffered batches in consecutive groups, each sized so its merge fits its own reservation + merge headroom + what the pool grants, then merge the spill files with the existing multi-level merge (fan-in / read-ahead / split degradation). No infallible growth past the pool is added. Healthy sorts take the same path as before.

merge_uses_row_cursor moved to streaming_merge.rs and shared with multi_level_merge.rs.

Tests

• New output_merge_that_does_not_fit_spills_instead_of_failing: without the fix ResourcesExhausted ... ExternalSorterMerge[0] ... 544 KB; with the fix it spills in groups, peak reserved <= pool limit, output equals the reference sort.
• vendored datafusion-physical-plan: 1772 passed / 0 failed; core sort tests 27 passed; fmt, clippy -D warnings clean.

🤖 Generated with Claude Code