You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi everyone,
I would like to add a functionality enabling to dynamically restack/resize a stream of pa.RecordBatch w.r.t. to minimums and maximums of rows and bytes.
That would means that too large batches would be chunked in smaller ones, and conversely small batches would be concatenated into bigger ones.
That way we would have more predictable resource usage for parallel processing tasks.
Cheers,
Guillaume
As mentioned in the PR, I would first direct you to the documentation to see if a bit more understanding of the design and the concept of Table, RecordBatch and RecordBatch.slice could help with the feature you are looking for. Feel free to re-open the issue if you see the need to do so.
I have re-read the documentation about the design and the concept of Table, RecordBatch and the RecordBatch's slicing capabilities, and I believe that I understand them, moreover I am already using these zero-copy slicing capabilities in the associated PR.
However, I am sorry that I did not sufficiently describe nor motivate the need to re-stack streams of pa.RecordBatch, and I will try to remedy to this.
For conciseness, we define the terms size and mass of a pa.RecordBatch, a pa.Table, or pad.Dataset, as their respective number of rows and bytes.
Let's consider a simple pipeline for data processing.
First, reading batches from a source dataset, whose cumulative mass could be larger than the available memory, say a few terabytes or even petabytes.
Then, applying some arbitrary business logic to the incoming batches.
And finally, writing the outgoing batches to a destination dataset.
Let's consider the following snippet,
importitertoolsimportpyarrowaspaimportpyarrow.datasetaspaddeffoo(batch: pa.RecordBatch) ->pa.RecordBatch:
# Some imaginary business logic is applied to the RecordBatch, altering its content# by adding, removing, combining columns, etc.returnbatchdefmain(src_path: str, dst_path: str, batch_size: int):
# Source -> Iterable[RecordBatch]# We read the input batches from the source dataset.iterable=pad.dataset(source=src_path).to_batches(batch_size=batch_size)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We apply a transformation to the batchesiterable=map(foo, iterable)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We extract the schema from the first output batch and re-join it to the iterablebatch=next(iter(iterable))
schema=batch.schemaiterable=itertools.chain([batch], iterable)
batch=None# Iterable[RecordBatch] -> Sink# We write the output batches to the destination dataset.pad.write_dataset(iterable, dst_path, format="parquet", schema=schema)
if__name__=='__main__':
main(src_path="/path/to/src/dataset", dst_path="/path/to/dst/dataset", batch_size=16)
In the above snippet, let's ignore the fact that the write_dataset() function would currently write to a single Parquet file containing 2^20 rows per group, which might not be optimal for evident reasons.
The pad.Dataset.to_batches() method allows us to set a static batch size to the Iterable of pa.RecordBatch, but that might be suboptimal for controlling the mass of incoming pa.RecordBatch with an arbitrary pa.Schema.
This is exacerbated in the cases where the pa.Schema contains a column with variable-length type, such as pa.ListArray, pa.StringArray, pa.BinaryArray, etc.
And, this is worsened for a variable-length typed column whose rows have a high variance w.r.t. to their number of elements.
In such cases, the pad.Dataset.to_batches(batch_size=batch_size) method could yield an Iterable of pa.RecordBatch with heavily fluctuating masses, i.e. the first batch has a mass of 100 MiB, but the second has a mass of 1 GiB.
This results in hard-to-predict, if not erratic, edge cases w.r.t. the overall memory footprint of the pipeline.
Moreover, we can quantify the expansion factor of the foo() processing function, i.e. let i, an incoming pa.RecordBatch of mass n, and o = foo(i), an outgoing pa.RecordBatch of mass m, the expansion factor would be e = m / n.
If e > 1, it is dilating, and conversely if e < 1 it is contracting.
And akin to Big O notation we can derive its worst-, average- and best-case memory usage scenarios.
Obviously, compounding the fluctuations in mass with a potential dilation worsens the overall stability of the memory footprint.
Thus, resulting in conservative, and inefficient, measures being taken, such as setting an intentionally low batch size, or oversizing the size of the processing machines.
Therefore, I suggest to introduce a functionality enabling the dynamic resizing of Iterable of pa.RecordBatch given minimums and maximums in terms of size and mass, in order to have a better control of the memory footprint.
The snippet would look like,
importitertoolsimportpyarrowaspaimportpyarrow.datasetaspaddeffoo(batch: pa.RecordBatch) ->pa.RecordBatch:
# Some imaginary business logic is applied to the RecordBatch, altering its content# by adding, removing, combining columns, etc.returnbatchdefmain(
src_path: str,
dst_path: str,
min_rows_per_batch: int,
max_rows_per_batch: int,
min_bytes_per_batch: int,
max_bytes_per_batch: int,
):
# Source -> Iterable[RecordBatch]# We read the input batches from the source dataset.iterable=pad.dataset(source=src_path).to_batches()
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We resize the incoming batches to fit constraints in size and mass.iterable=pa.restack_batches(
iterable,
min_rows_per_batch=min_rows_per_batch,
max_rows_per_batch=max_rows_per_batch,
min_bytes_per_batch=min_bytes_per_batch,
max_bytes_per_batch=max_bytes_per_batch,
)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We apply a transformation to the batchesiterable=map(foo, iterable)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We extract the schema from the first output batch and re-join it to the iterablebatch=next(iter(iterable))
schema=batch.schemaiterable=itertools.chain([batch], iterable)
batch=None# Iterable[RecordBatch] -> Sink# We write the output batches to the destination dataset.pad.write_dataset(iterable, dst_path, format="parquet", schema=schema)
if__name__=="__main__":
main(
src_path="/path/to/src/dataset",
dst_path="/path/to/dst/dataset",
min_rows_per_batch=0,
max_rows_per_batch=2** (10+10), # 1M rows per batchmin_bytes_per_batch=0,
max_bytes_per_batch=2** (10+10+9), # 512 MiB per batch
)
Arguably, we could also use the restack_batches() function to resize the Iterable of outgoing pa.RecordBatch and change slightly how we would write the destination dataset in order to write multiple smaller Parquet files, such that,
importitertoolsfromtypingimportIterableimportpyarrowaspaimportpyarrow.datasetaspadimportpyarrow.parquetaspapdeffoo(batch: pa.RecordBatch) ->pa.RecordBatch:
# Some imaginary business logic is applied to the RecordBatch, altering its content# by adding, removing, combining columns, etc.returnbatchdefwrite_dataset(
iterable: Iterable[pa.RecordBatch],
path: str,
):
forindex, batchinenumerate(iterable):
pap.write_table(
table=pa.Table.from_batches(batches=[batch], schema=batch.schema),
where=f"{path}/{index:012d}.parquet",
row_group_size=batch.num_rows,
compression="snappy",
)
defmain(
src_path: str,
dst_path: str,
min_rows_per_batch: int,
max_rows_per_batch: int,
min_bytes_per_batch: int,
max_bytes_per_batch: int,
):
# Source -> Iterable[RecordBatch]# We read the input batches from the source dataset.iterable=pad.dataset(source=src_path).to_batches()
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We resize the incoming batches to fit constraints in size and mass.iterable=pa.restack_batches(
iterable,
min_rows_per_batch=min_rows_per_batch,
max_rows_per_batch=max_rows_per_batch,
min_bytes_per_batch=min_bytes_per_batch,
max_bytes_per_batch=max_bytes_per_batch,
)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We apply a transformation to the batchesiterable=map(foo, iterable)
# Iterable[RecordBatch] -> Iterable[RecordBatch]# We resize the outgoing batches to fit constraints in size and mass.iterable=pa.restack_batches(
iterable,
min_rows_per_batch=min_rows_per_batch,
max_rows_per_batch=max_rows_per_batch,
min_bytes_per_batch=min_bytes_per_batch,
max_bytes_per_batch=max_bytes_per_batch,
)
# Iterable[RecordBatch] -> Sink# We write the output batches to the destination dataset, but this time in smaller files.write_dataset(iterable, path=dst_path)
if__name__=="__main__":
main(
src_path="/path/to/src/dataset",
dst_path="/path/to/dst/dataset",
min_rows_per_batch=0,
max_rows_per_batch=2** (10+10), # 1M rows per batchmin_bytes_per_batch=0,
max_bytes_per_batch=2** (10+10+9), # 512 MiB per batch
)
One fair point of criticism made in the PR is that the restack_batches function is written in Python and not in C++, and most likely sub-par w.r.t. its own efficiency, but that's a starting point rather than the end of a discussion in my opinion.
Another point of criticism would be that it would more efficient to fuse the resizing of the batches while reading, e.g. in the pad.Dataset.to_batches() method, and writing, e.g. in the pad.write_dataset() function.
PS: It seems that I do not have the permission to re-open the issue, could you please re-open it?
Hi @AlenkaF, @raulcd, @rok,
Could you please spare a moment to provide an answer about this potential enhancement request, and its associated PR?
I understand that you do not want to see Python doing the compute but rather C++, but I'd like to know if it is worth it for me to invest in learning C++ for this enhancement.
Thanks in advance,
I would like to add a functionality enabling to dynamically restack/resize a stream of pa.RecordBatch w.r.t. to minimums and maximums of rows and bytes.
Where are your record batches coming from? Parquet reader or some other source? RecordBatch sizes are best controlled at the source e.g. by setting row_group_size at parquet write time.
PyArrow already has concat and slice C++ based methods for manipulating RecordBatch size. So you could use those if you already have RecordBatches of a certain size and have an application level utility function to even out your RecordBatch sizes. I am not convinced yet that you need such a utility in PyArrow. It would be useful to know more about your use case.
Hey @rok,
Yes, at runtime, we are reading them with pyarrow.dataset.dataset().to_batches(), however, we do not have the possibility to control the row_group_size of the Parquet files we are reading from. They come from either exports from BigQuery, or other third-party sources.
And the trouble is that we're handling a large variety of datasets with their respective schemas, and they often have a high variance in the amount of bytes per row, so it's quite difficult to come up with a simple set of rules that fits well.
Hence, me deciding to resize both incoming and outgoing batches with both the amount of bytes and rows.
That way, even if upstream was not neatly packed, we normalize it, and ensure that downstream would receive neatly packed data.
By the way, in the PR, we're making use of the pyarrow.RecordBatch.slice() and concat_batches() capabilities :)
Activity
AlenkaF commentedon Jan 28, 2026
As mentioned in the PR, I would first direct you to the documentation to see if a bit more understanding of the design and the concept of
Table,RecordBatchandRecordBatch.slicecould help with the feature you are looking for. Feel free to re-open the issue if you see the need to do so.guillaume-rochette-oxb commentedon Jan 28, 2026
Hi @AlenkaF,
I have re-read the documentation about the design and the concept of Table, RecordBatch and the RecordBatch's slicing capabilities, and I believe that I understand them, moreover I am already using these zero-copy slicing capabilities in the associated PR.
However, I am sorry that I did not sufficiently describe nor motivate the need to re-stack streams of
pa.RecordBatch, and I will try to remedy to this.For conciseness, we define the terms size and mass of a
pa.RecordBatch, apa.Table, orpad.Dataset, as their respective number of rows and bytes.Let's consider a simple pipeline for data processing.
First, reading batches from a source dataset, whose cumulative mass could be larger than the available memory, say a few terabytes or even petabytes.
Then, applying some arbitrary business logic to the incoming batches.
And finally, writing the outgoing batches to a destination dataset.
Let's consider the following snippet,
In the above snippet, let's ignore the fact that the
write_dataset()function would currently write to a single Parquet file containing 2^20 rows per group, which might not be optimal for evident reasons.The
pad.Dataset.to_batches()method allows us to set a static batch size to theIterableofpa.RecordBatch, but that might be suboptimal for controlling the mass of incomingpa.RecordBatchwith an arbitrarypa.Schema.This is exacerbated in the cases where the
pa.Schemacontains a column with variable-length type, such aspa.ListArray,pa.StringArray,pa.BinaryArray, etc.And, this is worsened for a variable-length typed column whose rows have a high variance w.r.t. to their number of elements.
In such cases, the
pad.Dataset.to_batches(batch_size=batch_size)method could yield anIterableofpa.RecordBatchwith heavily fluctuating masses, i.e. the first batch has a mass of 100 MiB, but the second has a mass of 1 GiB.This results in hard-to-predict, if not erratic, edge cases w.r.t. the overall memory footprint of the pipeline.
Moreover, we can quantify the expansion factor of the
foo()processing function, i.e. leti, an incomingpa.RecordBatchof massn, ando = foo(i), an outgoingpa.RecordBatchof massm, the expansion factor would bee = m / n.If
e > 1, it is dilating, and conversely ife < 1it is contracting.And akin to Big O notation we can derive its worst-, average- and best-case memory usage scenarios.
Obviously, compounding the fluctuations in mass with a potential dilation worsens the overall stability of the memory footprint.
Thus, resulting in conservative, and inefficient, measures being taken, such as setting an intentionally low batch size, or oversizing the size of the processing machines.
Therefore, I suggest to introduce a functionality enabling the dynamic resizing of
Iterableofpa.RecordBatchgiven minimums and maximums in terms of size and mass, in order to have a better control of the memory footprint.The snippet would look like,
Arguably, we could also use the
restack_batches()function to resize theIterableof outgoingpa.RecordBatchand change slightly how we would write the destination dataset in order to write multiple smaller Parquet files, such that,One fair point of criticism made in the PR is that the
restack_batchesfunction is written in Python and not in C++, and most likely sub-par w.r.t. its own efficiency, but that's a starting point rather than the end of a discussion in my opinion.Another point of criticism would be that it would more efficient to fuse the resizing of the batches while reading, e.g. in the
pad.Dataset.to_batches()method, and writing, e.g. in thepad.write_dataset()function.PS: It seems that I do not have the permission to re-open the issue, could you please re-open it?
guillaume-rochette-oxb commentedon Feb 2, 2026
Hi @AlenkaF, what are your thoughts towards this enhancement request, now that I have described and motivated the issue?
guillaume-rochette-oxb commentedon Feb 9, 2026
Hi @AlenkaF, @raulcd, @rok,
Could you please spare a moment to provide an answer about this potential enhancement request, and its associated PR?
I understand that you do not want to see Python doing the compute but rather C++, but I'd like to know if it is worth it for me to invest in learning C++ for this enhancement.
Thanks in advance,
raulcd commentedon Feb 9, 2026
I've answered on the PR.
rok commentedon Feb 9, 2026
Hey @guillaume-rochette-oxb, sorry I'm late to the party.
Where are your record batches coming from? Parquet reader or some other source? RecordBatch sizes are best controlled at the source e.g. by setting row_group_size at parquet write time.
PyArrow already has concat and slice C++ based methods for manipulating RecordBatch size. So you could use those if you already have RecordBatches of a certain size and have an application level utility function to even out your RecordBatch sizes. I am not convinced yet that you need such a utility in PyArrow. It would be useful to know more about your use case.
guillaume-rochette-oxb commentedon Feb 9, 2026
Hey @rok,
Yes, at runtime, we are reading them with
pyarrow.dataset.dataset().to_batches(), however, we do not have the possibility to control therow_group_sizeof the Parquet files we are reading from. They come from either exports from BigQuery, or other third-party sources.And the trouble is that we're handling a large variety of datasets with their respective schemas, and they often have a high variance in the amount of bytes per row, so it's quite difficult to come up with a simple set of rules that fits well.
Hence, me deciding to resize both incoming and outgoing batches with both the amount of bytes and rows.
That way, even if upstream was not neatly packed, we normalize it, and ensure that downstream would receive neatly packed data.
By the way, in the PR, we're making use of the
pyarrow.RecordBatch.slice()andconcat_batches()capabilities :)