Repository navigation
GH-55642: [C++][Python] Make the dataset writer honor min_rows_per_group and add max_rows_queued - #55658
Draft
tolleybot wants to merge 5 commits into
Draft
GH-55642: [C++][Python] Make the dataset writer honor min_rows_per_group and add max_rows_queued#55658tolleybot wants to merge 5 commits into
tolleybot wants to merge 5 commits into
Conversation
…is best effort The dataset writer stops staging rows once 2,097,152 rows are held back across all files being written, after which row groups smaller than min_rows_per_group are written. Describe this in the Python docstring, the C++ option documentation and the Python user guide.
…file and evict the largest staged file When more rows are staged than the dataset writer allows, write out the staged rows of the file that holds the most of them instead of the file that received the latest batch, which could produce very small row groups. Never let the staging limit be smaller than min_rows_per_group, raising the limit on queued rows when needed, so that a single file always reaches min_rows_per_group.
|
|
…aset write options Expose the limit on rows held in memory by the dataset writer as FileSystemDatasetWriteOptions::max_rows_queued and as the max_rows_queued argument of pyarrow.dataset.write_dataset. The limit on staged rows now grows with it without a fixed ceiling, so a partitioned write can reach min_rows_per_group in every partition when given enough room.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The dataset writer stops holding rows back for
min_rows_per_grouponce 2,097,152 rows are staged intotal across all files being written. When that happens it writes out the staged rows of whichever file
received the latest batch, however few they are. As described in #55642 this means:
min_rows_per_groupis (also reported in Is there a max limit to PyArrow write_dataset row-group sizes #45018)
min_rows_per_group, down to a few thousand rowsin the middle of a file
What changes are included in this PR?
Three changes to the dataset writer:
file that received the latest batch. This is what the existing comment on
staged_rows_countalreadydescribed. The staged row count is now updated as rows are staged and delivered, which also removes a
one batch lag in noticing the limit.
min_rows_per_group. A single file therefore alwaysreaches
min_rows_per_group. Staged rows count againstmax_rows_queued, somax_rows_queuedisraised to twice the staging limit when it would otherwise be smaller, to keep the writer from stalling.
max_rows_queuedbecomes an option. It is added toFileSystemDatasetWriteOptionsand topyarrow.dataset.write_dataset, and the two callers infile_base.ccpass it to the writer. Thestaging limit stays at
max_rows_queued / 4but no longer has a fixed ceiling of 8,388,608 rows, soraising
max_rows_queuedlets more partitions reachmin_rows_per_groupat the same time.The documentation for
min_rows_per_groupandmax_rows_queuedis updated infile_base.h, in thewrite_datasetdocstring and in the Python dataset user guide to describe the limit and how the twooptions interact.
Not included: the R bindings do not expose
max_rows_queued.Are these changes tested?
Yes.
dataset_writer_test.cchas three new tests: a single file withmin_rows_per_groupabove the staginglimit, eviction of the file with the most staged rows, and several files that all reach
min_rows_per_groupwhenmax_rows_queuedleaves room. The first two fail without the change.test_dataset.pyhas two new tests covering the single file case and a partitioned write withmax_rows_queuedraised.With the reproducer from #55642 (20,000,000 rows,
min_rows_per_group == max_rows_per_group):max_rows_queuedmax_rows_queued = 4 * partitions * min_rows_per_groupAre there any user-facing changes?
max_rows_queuedinFileSystemDatasetWriteOptionsandpyarrow.dataset.write_dataset.The default is unchanged (8 * 1024 * 1024).
min_rows_per_group. A partitioned write with the defaultmax_rows_queuedstill may not, but theundersized groups are the largest available instead of arbitrary.
min_rows_per_groupabove 2,097,152 now uses more memory than before, because that many rows areheld until the group is complete, and
max_rows_queuedis raised to twicemin_rows_per_groupwhenit is smaller.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: