Skip to content

[C++][Python] write_dataset writes row groups smaller than min_rows_per_group when partitioned #55642

Open

Description

@tolleybot

pyarrow.dataset.write_dataset writes Parquet row groups smaller than min_rows_per_group when the write
is partitioned, including when min_rows_per_group == max_rows_per_group. Without partitioning, a
min_rows_per_group above 2,097,152 is silently capped at about that value, which is what #45018
reported (closed as stale).

The documentation for min_rows_per_group says:

Minimum number of rows per group. When the value is greater than 0, the dataset writer will batch
incoming data and only write the row groups to the disk when sufficient rows have accumulated.

No error or warning is raised in either case.

Reproducer

Deterministic and single-threaded, with inputs of about 100 MB:

import tempfile

import numpy as np
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq


def row_groups(table, **kwargs):
    with tempfile.TemporaryDirectory() as d:
        ds.write_dataset(table, d, format="parquet", **kwargs)
        out = {}
        for f in sorted(ds.dataset(d, format="parquet").files):
            md = pq.ParquetFile(f).metadata
            out[f[len(d):]] = [md.row_group(i).num_rows for i in range(md.num_row_groups)]
        return out


N = 20_000_000
print("pyarrow", pa.__version__)

# 1. Partitioned, partition values interleaved row by row, single-threaded.
#    min_rows_per_group == max_rows_per_group == 1,000,000.
for parts in (2, 3):
    table = pa.table({
        "x": np.zeros(N, dtype=np.uint8),
        "p": np.resize(np.arange(parts, dtype=np.int32), N),
    })
    print(f"\n{parts} partitions")
    for f, sizes in row_groups(
        table,
        partitioning=ds.partitioning(pa.schema([("p", pa.int32())]), flavor="hive"),
        min_rows_per_group=1_000_000,
        max_rows_per_group=1_000_000,
        use_threads=False,
    ).items():
        print(f, sizes)

# 2. Not partitioned, min_rows_per_group == max_rows_per_group == 3,000,000.
print("\nnot partitioned, min = max = 3,000,000")
for f, sizes in row_groups(
    pa.table({"x": np.zeros(N, dtype=np.uint8)}),
    min_rows_per_group=3_000_000,
    max_rows_per_group=3_000_000,
    use_threads=False,
).items():
    print(f, sizes)

Output on 26.0.1 (24.0.0 and 25.0.1 print exactly the same row groups):

pyarrow 26.0.1

2 partitions
/p=0/part-0.parquet [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000]
/p=1/part-0.parquet [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000]

3 partitions
/p=0/part-0.parquet [1000000, 1000000, 894507, 1000000, 507328, 830123, 273066, 764587, 397056]
/p=1/part-0.parquet [939349, 1000000, 714859, 1000000, 1000000, 1000000, 1000000, 12459]
/p=2/part-0.parquet [709973, 884736, 294912, 1000000, 627477, 1000000, 1000000, 1000000, 3734, 145834]

not partitioned, min = max = 3,000,000
/part-0.parquet [2129920, 2129920, 2129920, 2129920, 2129920, 2129920, 2129920, 2129920, 2129920, 830720]

Two partitions give exact groups; three give groups as small as 3,734 rows in the middle of a file.

Where it comes from

In cpp/src/arrow/dataset/dataset_writer.cc (line numbers at apache-arrow-26.0.1, unchanged on main),
DatasetWriterFileQueue::Push flushes a partition's staged batches while
(L217-L219):

while (!staged_batches_.empty() &&
       (writer_state_->StagingFull() ||
        rows_currently_staged_ >= options_.min_rows_per_group)) {

StagingFull() compares one counter shared by every partition's queue against max_rows_staged
(L105).
The comment on that field says the behaviour is deliberate
(L117-L120):

// If too many rows get staged we will end up with poor performance and, if more rows
// are staged than max_rows_queued we will end up with deadlock.  To avoid this, once
// we have too many staged rows we just ignore min_rows_per_group
const uint64_t max_rows_staged;

max_rows_staged is min(1 << 23, max_rows_queued / 4)
(L523-L524),
and max_rows_queued is always the default kDefaultDatasetWriterMaxRowsQueued = 8 * 1024 * 1024, so the
cap is 2,097,152 rows in total across all partitions. It does not depend on the number of partitions.
Once the total staged across partitions reaches it, whichever partition pushes next writes out everything
it holds. The shared counter is updated after the check
(L224),
which is why the unpartitioned groups above are 2,129,920 rather than 2,097,152: the cap is noticed one
batch late.

When it happens

A partition only holds rows while it is below min_rows_per_group, so the total staged cannot exceed
about min_rows_per_group × partitions. On 26.0.1, holding 20,000,000 rows constant, with partition
values in contiguous runs of 1,000,000 (np.resize(np.repeat(np.arange(parts), 1_000_000), 20_000_000))
and use_threads=True, the fraction of runs with any row group below the minimum, not counting the last
group of a file, was:

min_rows_per_group partitions: runs with short groups
1,500,000 1: 0/50, 2: 50/50
1,000,000 2: 0/100, 3: 100/100
500,000 4: 0/50, 5: 28/50, 6: 38/50
400,000 5: 0/50, 6: 0/50, 7: 4/50

No configuration tried with min_rows_per_group × partitions ≤ 2,097,152 produced a short group,
including the unpartitioned runs. Above that
product, short groups are possible but not certain: what matters is how many rows are staged at the
same moment, which depends on how the partition values are ordered and on thread scheduling.

Contiguous input with threads is also affected

The interleaved, single-threaded case above is the easiest to reproduce. With the default
use_threads=True, input whose partition values come in contiguous runs is affected too. With
np.resize(np.repeat(np.arange(4), 1_000_000), 20_000_000) as the partition column and
min_rows_per_group == max_rows_per_group == 1_000_000, five runs wrote 44, 29, 26, 37 and 32 row groups
instead of 20, different on every run, with short groups in the middle of files. The same input with
use_threads=False gave exact groups.

In a larger write (5 billion rows, 13 partitions, one file per partition, so max_open_files is not
involved) this produced about 25,000 row groups where about 5,000 were expected, some of only 64 rows.

No way to raise the cap

max_rows_queued is a parameter of internal::DatasetWriter::Make only. It is not a field of
FileSystemDatasetWriteOptions, and its two callers outside tests, both in file_base.cc
(L422,
L579)
leave it at the default. write_dataset has no corresponding argument. The only check on
min_rows_per_group is that it is not above max_rows_per_group, so a value above the cap is accepted
and then not met.

A workaround that gave exact groups in testing is to write each partition with its own unpartitioned
write_dataset call, keeping min_rows_per_group at or below 2,097,152.

Expected behaviour

Row groups of at least min_rows_per_group rows, as documented, except where a file ends. If the cap is
meant to stay as it is, the documentation for min_rows_per_group should say that it is best effort, that
staging is limited to 2,097,152 rows shared across all partitions being written, and that larger values
are not met.

Versions and platform

Reproduced with pyarrow 24.0.0, 25.0.1 and 26.0.1 from PyPI wheels, Python 3.14.6, numpy 2.5.5, macOS
arm64. The code cited above is the same at all three tags and on main.

Related: #45018, the same cap reached without partitioning.

Component(s)

C++, Python

Activity

  1. tolleybot commented on Nov 17, 2026

    @tolleybot
    ContributorAuthor

    take

  2. added a commit that references this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions