Repository navigation
feat: report snapshot row count statistics to DataFusion - #114
Conversation
alexanderbianchi
left a comment
There was a problem hiding this comment.
LGTM. once #67 lands there will be more to do with stats since we will have the manifest files as well as the snapshot stats.
| fn estimated_num_rows(&self) -> Precision<usize> { | ||
| let metadata = self.table.metadata(); | ||
| let snapshot = match self.snapshot_id { | ||
| Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id), |
There was a problem hiding this comment.
This snapshot pattern comes up frequently, I wonder if we should just resolve the snapshot once during data source construction. Not for this PR though,.
There was a problem hiding this comment.
+1 to resolving the snapshot once at construction. Could we file an issue for it so it doesn't get lost, and link it here?
| let count = |key| summary.get(key)?.parse::<usize>().ok(); | ||
| match count(TOTAL_RECORDS) { | ||
| Some(records) => Precision::Inexact( | ||
| records.saturating_sub(count(TOTAL_POSITION_DELETES).unwrap_or(0)), |
There was a problem hiding this comment.
Thanks for this. It's a nice, contained way to give join selection something to work with, and the unit tests cover the summary edge cases well.
I'm a bit worried about subtracting total-position-deletes. After rewrite_data_files without remove-dangling-deletes (the default in Spark), total-records reflects the compacted files while total-position-deletes can still count deletes against data files that are gone. The estimate can then land far below the real row count, down to Inexact(0) with the saturating_sub. Because join_selection uses num_rows against hash_join_single_partition_threshold_rows, a large table could get picked as the CollectLeft build side, which is worse than reporting unknown stats.
Would it be safer to report total-records as-is, as @JanKaul's datafusion_iceberg does? Overestimating seems like the safer failure mode here. If you'd like to keep the subtraction, maybe fall back to total-records (or Absent) when the deletes make up a large share of the records? A unit test for the "deletes exceed records" case could then assert the fallback instead of Inexact(0).
There was a problem hiding this comment.
Would it be safer to report
total-recordsas-is, as @JanKaul'sdatafusion_icebergdoes? Overestimating seems like the safer failure mode here.
Agreed. The scan now reports total-records as an upper bound and ignores deletes. The spec lets total-position-deletes count more rows than the deletes remove, in two ways:
- No rule removes a v2 position delete file when the data files it references are removed. Only deletion vectors must go with their data file (spec). Your
RewriteDataFilescase is the common one, sinceremove-dangling-deletesdefaults tofalse. - A deletion vector must contain the deletes of existing position delete files, and readers then ignore those files (spec, spec). Nothing requires removing a partition-scoped delete file that still applies to other data files, so those deletes are counted twice.
supports_collect_by_thresholds compares inexact counts against the thresholds, so an underestimate there picks the large table as the build side. ignores_deletes covers a summary whose position deletes exceed its records.
# Conflicts: # crates/datafusion/src/read/scan.rs
IcebergDataSourcereported unknown statistics, so DataFusion's join selection had nothing to compare for Iceberg scans. The source now reports a row count from the scanned snapshot'stotal-recordssummary field, adjusted for position deletes and the scan's fetch. It also reports a byte size when every projected column is fixed-width. A table with no current snapshot reports exactly zero rows.Row counts stay inexact even when a snapshot has no delete files, so DataFusion doesn't answer
COUNT(*)from the summary. That would mean trusting writer-maintained summary fields for query results, which deserves its own discussion.The exact zero changes one plan in
binary_predicate_pushdown.slt. ThatEXPLAINruns on an empty table, and DataFusion only adds a round-robin repartition when an exact row count exceedsbatch_size(enforce_distribution.rs). So theRepartitionExecabove the scan is gone.The idea comes from @JanKaul's
datafusion_iceberg, which reportstotal-recordsas an inexact row count. Thanks!Tests
crates/datafusion/src/read/scan.rscallpartition_statisticson tables built in memory with chosen snapshot summaries.crates/datafusion/tests/statistics.rschecks the statistics inEXPLAINoutput for the taxi fixture through the catalog provider.AI usage disclosure
I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.