Skip to content

feat: report the sort order of sorted Iceberg scans to DataFusion - #55

Open
anuragmantri wants to merge 6 commits into
apache:mainfrom
anuragmantri:sort-order-reporting-part2-db207e6e
Open

anuragmantri wants to merge 6 commits into
apache:mainfrom
anuragmantri:sort-order-reporting-part2-db207e6e

Conversation

@anuragmantri

@anuragmantri anuragmantri commented Oct 8, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

IcebergDataSource can now report an output ordering to DataFusion when the data files it reads are already sorted, so DataFusion can drop the SortExec it would otherwise add above the scan for an ORDER BY or a sort-merge join.

The feature is off by default. It is enabled per session with SET iceberg.execution.preserve_data_ordering = true, after registering the new IcebergDataFusionConfig extension. The playground and the sqllogictest engine register it. The option name follows Spark's spark.sql.iceberg.planning.preserve-data-ordering, which apache/iceberg#16750 proposes, and sits in DataFusion's execution group. DataFusion lists extension options in information_schema.df_settings by their last name only, and SHOW and RESET don't support them.

When it is on, TableProvider::scan lists the data files up front. The source reports an ordering only if:

there are at most iceberg.execution.max_merge_files_per_partition files (default 64, matching Comet's sortMerge.maxFilesPerPartition in apache/datafusion-comet#5331),

  • every file records the same sort_order_id, and it resolves to a sorted order, and
  • every output column has the same type in the provider's declared schema as in the snapshot schema the files are read with, and is declared nullable wherever the snapshot column is optional. A catalog-backed provider caches its schema, so the two can differ, and DataFusion ignores the null order of a column it believes is non-nullable.

It reports the leading fields of that order that are identity transforms of projected top-level columns, stopping at the first float, double, or UUID field. DataFusion sorts floats in the order the Iceberg spec defines, with -0.0 before 0.0 and negative NaN first, but Spark, the usual writer of sorted files, treats the zeros as equal and every NaN as the largest value, so the files it writes may not be sorted in the spec's order. The spec doesn't define UUID ordering (apache/iceberg#14216). In every other case the source is planned exactly as before.

Two files with the same sort order can have overlapping key ranges, so open() reads each file as its own stream and k-way merges them with DataFusion's StreamingMergeBuilder, which is the machinery SortPreservingMergeExec uses. A pushed-down limit becomes the merge's fetch. All the readers are clones of one ArrowReader, so they share its delete-file cache.

The scan reports each sort field's null order as the table declares it. Writers commonly sort ascending fields with nulls first, the default of Iceberg Java's SortOrderBuilder.asc, while DataFusion's ORDER BY c puts them last. On a nullable column sorted that way, only ORDER BY c NULLS FIRST, or a session with datafusion.sql_parser.default_null_ordering = 'nulls_min', can skip its sort. Other queries still sort, above the merge. TableProvider::scan can't tell whether anything above the scan needs the order, so it merges whenever it reports one. Merging only when something above uses the order is tracked in #72.

The source keeps a single output partition, so this doesn't depend on apache/iceberg-rust#2671, and it composes with it. DataFusion's output ordering is per partition, so once apache/iceberg-rust#2671 groups files into partitions, each group can be merged the same way.

Public API changes:

  • New: IcebergDataFusionConfig and IcebergExecutionConfig, the session options.
  • New: IcebergDataSource::sorted_tasks, the files a sorted source merges, and IcebergDataSourceBuilder::with_sorted_tasks. Since feat: make the plan nodes inspectable and rebuildable #20, a source can be rebuilt from its accessors, for example to ship it to another process. A sorted source rebuilt without its files would read them unordered beneath a plan that already dropped the sort, so a rebuild must pass them to with_sorted_tasks, and build() fails if they don't share an order the source can report. The providers build a sorted source through the same builder calls, so the builder stays the only way to construct one.

When the option is on but a scan falls back, the files are listed again in open(), as without the option. The table's manifest cache makes the second listing cheap.

Merging opens every file at once. The cap bounds that. Bounding it instead by how many files' key ranges overlap needs per-file column bounds on FileScanTask, and is tracked in #71.

Are these changes tested?

Yes.

  • Unit tests for mapping a sort order to a DataFusion ordering:
    • direction and null order,
    • stopping at bucket, truncate, and day transforms,
    • stopping at float, double, UUID, nested, unprojected, and missing fields.
  • Integration tests in tests/sorted_iceberg_scan.rs, over tables with hand-written sorted files:
    • files with overlapping key ranges and duplicate keys merge into sorted order, with struct and list columns carried through;
    • files longer than a reader batch merge correctly across batch boundaries;
    • LIMIT;
    • no ordering when files disagree, have no sort order id, or have the unsorted order, and the plan then matches the one without the option;
    • no ordering when the declared schema is stale, after a column is made optional or its type is promoted;
    • the option is off by default, and the cap is honored;
    • ORDER BY loses its SortExec, while ORDER BY ... DESC and the option-off case keep it;
    • on a nullable sort key, ORDER BY id NULLS FIRST loses its SortExec, while ORDER BY id keeps it;
    • a sort-merge join with target_partitions = 4 has no SortExec;
    • a rebuilt source keeps its ordering when given its sorted tasks, and reports none when given None;
    • positional deletes are applied on the merge path.
    • a sorted scan of one file reports the order and returns its rows, and a sorted table with no files reports no order;
    • a rebuild fails when tasks share a sort order id but record different orders;
  • sqllogictest: sort_order_reporting.slt checks the EXPLAIN plans over a pre-created sorted table: the option off and on, a descending sort, a table created through SQL, and the file cap.
  • Mutation checks:
    • Replacing the merge with concatenation fails 8 of the 14 sorted-scan integration tests.
    • Truncating each file's stream to its first batch fails the batch-boundary test.
    • Dropping the check that every file has the same sort order fails the disagreement test.
    • Dropping the schema check fails both stale-schema tests.
    • Making with_sorted_tasks ignore its tasks fails 11 of the 14 sorted-scan tests.
    • Making the single-file path return no rows fails the one-file test.
    • Comparing sort order ids instead of the orders fails the different-orders test.

AI Disclosure

I'm new to Rust. I used Claude Code (Claude Opus 5.5) to write this change and its tests, and I reviewed it manually.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants