Skip to content

feat: report snapshot row count statistics to DataFusion - #114

Merged
mbutrovich merged 4 commits into
apache:mainfrom
mbutrovich:table-statistics
Nov 18, 2026
Merged

mbutrovich merged 4 commits into
apache:mainfrom
mbutrovich:table-statistics

Conversation

@mbutrovich

Copy link
Copy Markdown
Contributor

IcebergDataSource reported unknown statistics, so DataFusion's join selection had nothing to compare for Iceberg scans. The source now reports a row count from the scanned snapshot's total-records summary field, adjusted for position deletes and the scan's fetch. It also reports a byte size when every projected column is fixed-width. A table with no current snapshot reports exactly zero rows.

Row counts stay inexact even when a snapshot has no delete files, so DataFusion doesn't answer COUNT(*) from the summary. That would mean trusting writer-maintained summary fields for query results, which deserves its own discussion.

The exact zero changes one plan in binary_predicate_pushdown.slt. That EXPLAIN runs on an empty table, and DataFusion only adds a round-robin repartition when an exact row count exceeds batch_size (enforce_distribution.rs). So the RepartitionExec above the scan is gone.

The idea comes from @JanKaul's datafusion_iceberg, which reports total-records as an inexact row count. Thanks!

Tests

  • Unit tests in crates/datafusion/src/read/scan.rs call partition_statistics on tables built in memory with chosen snapshot summaries.
  • crates/datafusion/tests/statistics.rs checks the statistics in EXPLAIN output for the taxi fixture through the catalog provider.

AI usage disclosure

I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.

@alexanderbianchi alexanderbianchi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. once #67 lands there will be more to do with stats since we will have the manifest files as well as the snapshot stats.

fn estimated_num_rows(&self) -> Precision<usize> {
let metadata = self.table.metadata();
let snapshot = match self.snapshot_id {
Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This snapshot pattern comes up frequently, I wonder if we should just resolve the snapshot once during data source construction. Not for this PR though,.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to resolving the snapshot once at construction. Could we file an issue for it so it doesn't get lost, and link it here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #118

Comment thread crates/datafusion/src/read/scan.rs Outdated
let count = |key| summary.get(key)?.parse::<usize>().ok();
match count(TOTAL_RECORDS) {
Some(records) => Precision::Inexact(
records.saturating_sub(count(TOTAL_POSITION_DELETES).unwrap_or(0)),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. It's a nice, contained way to give join selection something to work with, and the unit tests cover the summary edge cases well.

I'm a bit worried about subtracting total-position-deletes. After rewrite_data_files without remove-dangling-deletes (the default in Spark), total-records reflects the compacted files while total-position-deletes can still count deletes against data files that are gone. The estimate can then land far below the real row count, down to Inexact(0) with the saturating_sub. Because join_selection uses num_rows against hash_join_single_partition_threshold_rows, a large table could get picked as the CollectLeft build side, which is worse than reporting unknown stats.

Would it be safer to report total-records as-is, as @JanKaul's datafusion_iceberg does? Overestimating seems like the safer failure mode here. If you'd like to keep the subtraction, maybe fall back to total-records (or Absent) when the deletes make up a large share of the records? A unit test for the "deletes exceed records" case could then assert the fallback instead of Inexact(0).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be safer to report total-records as-is, as @JanKaul's datafusion_iceberg does? Overestimating seems like the safer failure mode here.

Agreed. The scan now reports total-records as an upper bound and ignores deletes. The spec lets total-position-deletes count more rows than the deletes remove, in two ways:

  • No rule removes a v2 position delete file when the data files it references are removed. Only deletion vectors must go with their data file (spec). Your RewriteDataFiles case is the common one, since remove-dangling-deletes defaults to false.
  • A deletion vector must contain the deletes of existing position delete files, and readers then ignore those files (spec, spec). Nothing requires removing a partition-scoped delete file that still applies to other data files, so those deletes are counted twice.

supports_collect_by_thresholds compares inexact counts against the thresholds, so an underestimate there picks the large table as the build side. ignores_deletes covers a summary whose position deletes exceed its records.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @mbutrovich!

# Conflicts:
#	crates/datafusion/src/read/scan.rs
@mbutrovich
mbutrovich merged commit 184855b into apache:main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants