Repository navigation
feat: report the sort order of sorted Iceberg scans to DataFusion - #55
Open
anuragmantri wants to merge 6 commits into
Open
anuragmantri wants to merge 6 commits into
anuragmantri wants to merge 6 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
FileScanTask) merged in feat(iceberg): thread per-file sort order into FileScanTask iceberg-rust#3128 and is included in the pinned iceberg-rust rev.What changes are included in this PR?
IcebergDataSourcecan now report an output ordering to DataFusion when the data files it reads are already sorted, so DataFusion can drop theSortExecit would otherwise add above the scan for anORDER BYor a sort-merge join.The feature is off by default. It is enabled per session with
SET iceberg.execution.preserve_data_ordering = true, after registering the newIcebergDataFusionConfigextension. The playground and the sqllogictest engine register it. The option name follows Spark'sspark.sql.iceberg.planning.preserve-data-ordering, which apache/iceberg#16750 proposes, and sits in DataFusion'sexecutiongroup. DataFusion lists extension options ininformation_schema.df_settingsby their last name only, andSHOWandRESETdon't support them.When it is on,
TableProvider::scanlists the data files up front. The source reports an ordering only if:there are at most
iceberg.execution.max_merge_files_per_partitionfiles (default 64, matching Comet'ssortMerge.maxFilesPerPartitionin apache/datafusion-comet#5331),sort_order_id, and it resolves to a sorted order, andIt reports the leading fields of that order that are identity transforms of projected top-level columns, stopping at the first float, double, or UUID field. DataFusion sorts floats in the order the Iceberg spec defines, with -0.0 before 0.0 and negative NaN first, but Spark, the usual writer of sorted files, treats the zeros as equal and every NaN as the largest value, so the files it writes may not be sorted in the spec's order. The spec doesn't define UUID ordering (apache/iceberg#14216). In every other case the source is planned exactly as before.
Two files with the same sort order can have overlapping key ranges, so
open()reads each file as its own stream and k-way merges them with DataFusion'sStreamingMergeBuilder, which is the machinerySortPreservingMergeExecuses. A pushed-down limit becomes the merge'sfetch. All the readers are clones of oneArrowReader, so they share its delete-file cache.The scan reports each sort field's null order as the table declares it. Writers commonly sort ascending fields with nulls first, the default of Iceberg Java's
SortOrderBuilder.asc, while DataFusion'sORDER BY cputs them last. On a nullable column sorted that way, onlyORDER BY c NULLS FIRST, or a session withdatafusion.sql_parser.default_null_ordering = 'nulls_min', can skip its sort. Other queries still sort, above the merge.TableProvider::scancan't tell whether anything above the scan needs the order, so it merges whenever it reports one. Merging only when something above uses the order is tracked in #72.The source keeps a single output partition, so this doesn't depend on apache/iceberg-rust#2671, and it composes with it. DataFusion's output ordering is per partition, so once apache/iceberg-rust#2671 groups files into partitions, each group can be merged the same way.
Public API changes:
IcebergDataFusionConfigandIcebergExecutionConfig, the session options.IcebergDataSource::sorted_tasks, the files a sorted source merges, andIcebergDataSourceBuilder::with_sorted_tasks. Since feat: make the plan nodes inspectable and rebuildable #20, a source can be rebuilt from its accessors, for example to ship it to another process. A sorted source rebuilt without its files would read them unordered beneath a plan that already dropped the sort, so a rebuild must pass them towith_sorted_tasks, andbuild()fails if they don't share an order the source can report. The providers build a sorted source through the same builder calls, so the builder stays the only way to construct one.When the option is on but a scan falls back, the files are listed again in
open(), as without the option. The table's manifest cache makes the second listing cheap.Merging opens every file at once. The cap bounds that. Bounding it instead by how many files' key ranges overlap needs per-file column bounds on
FileScanTask, and is tracked in #71.Are these changes tested?
Yes.
tests/sorted_iceberg_scan.rs, over tables with hand-written sorted files:LIMIT;ORDER BYloses itsSortExec, whileORDER BY ... DESCand the option-off case keep it;ORDER BY id NULLS FIRSTloses itsSortExec, whileORDER BY idkeeps it;target_partitions = 4has noSortExec;None;sort_order_reporting.sltchecks theEXPLAINplans over a pre-created sorted table: the option off and on, a descending sort, a table created through SQL, and the file cap.with_sorted_tasksignore its tasks fails 11 of the 14 sorted-scan tests.AI Disclosure
I'm new to Rust. I used Claude Code (Claude Opus 5.5) to write this change and its tests, and I reviewed it manually.