You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Offset overflow - String kernels can produce more than 2 GiB per batch, and some reserve it up front #7500
Several native kernels produce more bytes than they read. Spark evaluates one row at a time, so each row's result only has to fit in a value. In Comet, the batch's result is one Arrow array, so it fails once the batch total passes 2 GiB:
Expression
Kernel
Output size
Failure
replace(col, 'a', 'bbb'), native by default for literal arguments (spark/src/main/scala/org/apache/comet/serde/strings.scala:187-215)
DataFusion replace
grows with replacement length / search length per match
repeat, rpad/lpad, base64 and randstr also reserve the whole batch's output before writing anything (with_capacity on the total). When the allocator refuses that reservation, arrow aborts the process (MutableBuffer::with_capacity calls handle_alloc_error, arrow-buffer 59.3.2 src/buffer/mutable.rs:131-140). So the executor dies instead of the task failing. repeat(s, 1000000) over 8192 rows of 100-byte strings asks for about 819 GB, and rpad(s, 10000000, ' ') for about 82 GB.
Line references are for main at 4e6b122, DataFusion 55.2.0 and arrow 59.3.2.
Workaround
Lower spark.comet.batchSize, which helps when the growth is moderate.
Proposed fix
Check the computed total against i32::MAX before allocating, and return an error that suggests a smaller batch size, in Comet's kernels and upstream in repeat. That removes the abort and the panics.
Call each kernel with a few rows whose total output passes 2 GiB and assert an error instead of a panic. Test the up-front size check with a request far beyond available memory, which today aborts the test process.
Activity