Skip to content

feat: skip hash shuffle for date_bin/date_trunc on Range([timestamp]) - #24501

Open
NGA-TRAN wants to merge 9 commits into
apache:mainfrom
NGA-TRAN:feat/issue-23569-range-date-bin-trunc
Open

feat: skip hash shuffle for date_bin/date_trunc on Range([timestamp])#24501
NGA-TRAN wants to merge 9 commits into
apache:mainfrom
NGA-TRAN:feat/issue-23569-range-date-bin-trunc

Conversation

@NGA-TRAN

@NGA-TRAN NGA-TRAN commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A table that is range-partitioned on timestamp (for example by hour) and grouped by date_bin / date_trunc of that timestamp is already partition-disjoint when bins do not straddle split points. DataFusion still planned Partial → hash RepartitionExec → Final.

After this change, Range([timestamp]) subset-satisfies KeyPartitioned([key, f(timestamp)]) when f is monotonic (date_bin, date_trunc) and evaluating f at each split vs. its predecessor shows the bins are disjoint. Aggregation then runs as one streaming SinglePartitioned step with ordering_mode=Sorted.

Bins that do straddle the split (for example date_trunc('day') on hour-partitioned data) still hash-repartition.

#24500's expected plan changes from:

AggregateExec FinalPartitioned
  RepartitionExec Hash([key, date_bin(...)])
    AggregateExec Partial, ordering_mode=Sorted
      DataSourceExec Range([timestamp]), 2 file groups

to:

AggregateExec SinglePartitioned, ordering_mode=Sorted
  DataSourceExec Range([timestamp]), 2 file groups

What changes are included in this PR?

  • Project Range through monotonic grouping expressions when bins stay disjoint.
  • Treat that case as subset satisfaction so SanityCheckPlan accepts SinglePartitioned over Range([timestamp]).
  • Update range_sorted_time_bin_agg.slt for date_bin(60s) and date_trunc('hour') (no shuffle) and date_trunc('day') (shuffle remains).

Are these changes tested?

  • Unit tests in partitioning.rs for aligned date_bin / date_trunc('hour') vs unaligned split and date_trunc('day').
  • cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt

Are there any user-facing changes?

Queries that group by date_bin / date_trunc of a range-partitioned timestamp may skip a hash shuffle when bins do not cross file-group boundaries. Results are unchanged.

Test plan

  • cargo test -p datafusion-physical-expr --lib -- partitioning
  • cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt
  • CI

Co-author with Cursor

Pin today's Partial + hash RepartitionExec + Final plan for GROUP BY
key, date_bin(timestamp) on a table that is already Range([timestamp])
and sorted on (key, timestamp), so a follow-up can remove the shuffle.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Aug 19, 2026
@codecov-commenter

codecov-commenter commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.57110% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.33%. Comparing base (8332cfa) to head (0c118e0).

Files with missing lines Patch % Lines
datafusion/physical-expr/src/partitioning.rs 95.43% 11 Missing and 7 partials ⚠️
...on/physical-expr/src/equivalence/properties/mod.rs 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24501      +/-   ##
==========================================
+ Coverage   81.32%   81.33%   +0.01%     
==========================================
  Files        1117     1117              
  Lines      396269   396682     +413     
  Branches   396269   396682     +413     
==========================================
+ Hits       322260   322642     +382     
- Misses      55186    55206      +20     
- Partials    18823    18834      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

NGA-TRAN and others added 3 commits August 20, 2026 10:27
Use generic column and table names so the coverage does not expose a
metrics-specific schema.

Co-authored-by: Cursor <cursoragent@cursor.com>
Accept RecordBatches and optional file sort order so the time-bin
table reuses the same registration path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Treat Range([x]) as a subset of KeyPartitioned([..., f(x), ...]) when f
is monotonic and bins do not straddle split points, so GROUP BY
key, date_bin/date_trunc can stream without a hash RepartitionExec.

Co-authored-by: Cursor <cursoragent@cursor.com>
@NGA-TRAN
NGA-TRAN force-pushed the feat/issue-23569-range-date-bin-trunc branch from 05705f6 to b26846d Compare August 20, 2026 15:44
Drop the local order-preserving walk and scalar predecessor so
date_bin/date_trunc range projection uses EquivalenceProperties and
interval_arithmetic instead of duplicating them.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Aug 20, 2026
NGA-TRAN and others added 2 commits August 20, 2026 14:47
Co-authored-by: Cursor <cursoragent@cursor.com>
@NGA-TRAN
NGA-TRAN marked this pull request as ready for review August 20, 2026 19:16
NGA-TRAN and others added 2 commits August 20, 2026 15:34
@NGA-TRAN

Copy link
Copy Markdown
Contributor Author

Tag @gene-bordegaray and @jayshrivastava for review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support functions date_bin and date_trunc in range partition satisfaction

2 participants