Repository navigation
Populate nan_value_counts when writing parquet files - #4412
AP24110011695 wants to merge 1 commit into
Conversation
data_file_statistics_from_parquet_metadata returned an empty nan_value_counts dict unconditionally, so data files written by the pyarrow writer never reported NaN counts and NaN-aware scan pruning stayed unavailable for them — the metrics evaluator degrades to ROWS_MIGHT_MATCH without nan counts (_InclusiveMetricsEvaluator visit_is_nan / _contains_nans_only). pyarrow's statistics API does not expose NaN counts, so the writer now counts them from the written table itself (pa.compute.is_nan summed per top-level float/double field, excluding nulls per the spec's NaN-count semantics) and passes them through a new optional table_nan_value_counts parameter. The metadata-only caller (parquet_file_to_data_file) keeps its previous behavior. Refs apache#4396. Signed-off-by: Ayush Kumar Saha <225407166+AP24110011695@users.noreply.github.com>
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks @AP24110011695! #4401 merged today with the same change (NaN counts taken from the data before writing, for top-level float and double columns), including tests for repeated write() calls, the none metrics mode and pruning, so this now conflicts with main. Is there a case here that #4401 doesn't cover?
|
Good question, and the answer is no — #4401 covers everything this PR did and a bit more. Same writer scope (top-level float/double, counts from the data before writing, accumulated across repeated For the record, #4396 was opened on Nov 16 (the NaN counts were never accumulated by the writer — the metrics evaluator's nan-based pruning was silently dead for pyiceberg-written files), and #4401 picking it up the next morning and merging is exactly how that should have played out. My branch raced it and lost, and it now conflicts with main as you noted. Closing this one — thanks for checking rather than letting a conflicting duplicate sit. The open thread on #4396 is the |
What's changed
Implements design A from #4396:
data_file_statistics_from_parquet_metadatareturned an emptynan_value_countsdict unconditionally, so data files written by the pyarrow writer never reported NaN counts — and NaN-aware scan pruning stayed silently unavailable for them (_InclusiveMetricsEvaluatordegrades toROWS_MIGHT_MATCHwithout nan counts, foris_nan,not_nanand range predicates via_contains_nans_only).pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no
nan_count/has_nan_count), so the writer now counts them from the written table itself:ParquetFormatWriter.write()accumulatespa.compute.is_nan(...)sums per top-level float/double field (is_nanis null on nulls andsumskips nulls — matching the spec's NaN-count semantics of counting non-null NaN values)table_nan_value_countsparameter ofdata_file_statistics_from_parquet_metadataparquet_file_to_data_file, used byadd_files) keeps its previous behavior — no NaN counts from metadata-only inputNested float fields keep the previous (empty) behavior for now; the metrics plan and evaluator keys make them a natural follow-up once the top-level path is settled.
Test
test_nan_value_counts_are_written(mixed values + null) andtest_nan_value_counts_are_written_across_row_groups(all-NaN, twowrite()calls) — both verified failing-first on main (nan_value_counts == {}) and passing with the change; the second also pins the pruning win (not_nannow prunes an all-NaN file via_contains_nans_only). Fulltests/io/test_pyarrow_stats.py(22) green; the pre-existingazure-fixture failures intest_pyarrow.pyare unrelated (missing optional dep in this environment).AI disclosure: drafted with AI assistance, verified locally by running the failing-first cycle.