Skip to content

GH-55642: [C++][Python] Make the dataset writer honor min_rows_per_group and add max_rows_queued - #55658

Draft
tolleybot wants to merge 5 commits into
apache:mainfrom
tolleybot:GH-55642-min-rows-per-group-docs
Draft

tolleybot wants to merge 5 commits into
apache:mainfrom
tolleybot:GH-55642-min-rows-per-group-docs

Conversation

@tolleybot

@tolleybot tolleybot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

The dataset writer stops holding rows back for min_rows_per_group once 2,097,152 rows are staged in
total across all files being written. When that happens it writes out the staged rows of whichever file
received the latest batch, however few they are. As described in #55642 this means:

  • a single file never gets row groups larger than about 2.1 million rows, whatever min_rows_per_group
    is (also reported in Is there a max limit to PyArrow write_dataset row-group sizes #45018)
  • a partitioned write can produce row groups far below min_rows_per_group, down to a few thousand rows
    in the middle of a file
  • there is no option to change the limit

What changes are included in this PR?

Three changes to the dataset writer:

  1. When too many rows are staged, write out the file holding the most staged rows. Before, it was the
    file that received the latest batch. This is what the existing comment on staged_rows_count already
    described. The staged row count is now updated as rows are staged and delivered, which also removes a
    one batch lag in noticing the limit.
  2. The staging limit is never smaller than min_rows_per_group. A single file therefore always
    reaches min_rows_per_group. Staged rows count against max_rows_queued, so max_rows_queued is
    raised to twice the staging limit when it would otherwise be smaller, to keep the writer from stalling.
  3. max_rows_queued becomes an option. It is added to FileSystemDatasetWriteOptions and to
    pyarrow.dataset.write_dataset, and the two callers in file_base.cc pass it to the writer. The
    staging limit stays at max_rows_queued / 4 but no longer has a fixed ceiling of 8,388,608 rows, so
    raising max_rows_queued lets more partitions reach min_rows_per_group at the same time.

The documentation for min_rows_per_group and max_rows_queued is updated in file_base.h, in the
write_dataset docstring and in the Python dataset user guide to describe the limit and how the two
options interact.

Not included: the R bindings do not expose max_rows_queued.

Are these changes tested?

Yes.

  • dataset_writer_test.cc has three new tests: a single file with min_rows_per_group above the staging
    limit, eviction of the file with the most staged rows, and several files that all reach
    min_rows_per_group when max_rows_queued leaves room. The first two fail without the change.
  • test_dataset.py has two new tests covering the single file case and a partitioned write with
    max_rows_queued raised.

With the reproducer from #55642 (20,000,000 rows, min_rows_per_group == max_rows_per_group):

case before after
not partitioned, 3,000,000 rows per group groups of 2,129,920 groups of 3,000,000
3 partitions, 1,000,000 rows per group, default max_rows_queued smallest group inside a file 3,734 smallest group inside a file 699,051
3 or 4 partitions, max_rows_queued = 4 * partitions * min_rows_per_group not possible every group 1,000,000, in 10 of 10 runs with and without threads

Are there any user-facing changes?

  • New option max_rows_queued in FileSystemDatasetWriteOptions and pyarrow.dataset.write_dataset.
    The default is unchanged (8 * 1024 * 1024).
  • Row group sizes change for writes that reach the staging limit. A single file now reaches
    min_rows_per_group. A partitioned write with the default max_rows_queued still may not, but the
    undersized groups are the largest available instead of arbitrary.
  • A min_rows_per_group above 2,097,152 now uses more memory than before, because that many rows are
    held until the group is complete, and max_rows_queued is raised to twice min_rows_per_group when
    it is smaller.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

…is best effort

The dataset writer stops staging rows once 2,097,152 rows are held back across
all files being written, after which row groups smaller than
min_rows_per_group are written. Describe this in the Python docstring, the
C++ option documentation and the Python user guide.
…file and evict the largest staged file

When more rows are staged than the dataset writer allows, write out the staged
rows of the file that holds the most of them instead of the file that received
the latest batch, which could produce very small row groups.

Never let the staging limit be smaller than min_rows_per_group, raising the
limit on queued rows when needed, so that a single file always reaches
min_rows_per_group.
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55642 has been automatically assigned in GitHub to PR creator.

…aset write options

Expose the limit on rows held in memory by the dataset writer as
FileSystemDatasetWriteOptions::max_rows_queued and as the max_rows_queued
argument of pyarrow.dataset.write_dataset. The limit on staged rows now grows
with it without a fixed ceiling, so a partitioned write can reach
min_rows_per_group in every partition when given enough room.
@tolleybot tolleybot changed the title GH-55642: [C++][Python][Docs] Document that min_rows_per_group is best effort GH-55642: [C++][Python] Make the dataset writer honor min_rows_per_group and add max_rows_queued Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant