Skip to content

Add a plan remainder replanning step - #835

Draft
gabotechs wants to merge 5 commits into
mainfrom
gabrielmusat/remainder-replanning
Draft

gabotechs wants to merge 5 commits into
mainfrom
gabrielmusat/remainder-replanning

Conversation

@gabotechs

@gabotechs gabotechs commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Introduces the concept of plan remainder replanning to the distributed planner, expected to be used during AQE for re-planning the full fragment of the plan that has not yet been injected any network boundaries.

What is a plan remainder?

A plan remainder is what's left of a plan once all the stages in which it depends have been formed. For example, let's imagine this plan:

                         ┌──────┐                 
               ┌────────▶│ Join │◀────────┐       
               │         └──────┘         │       
           ┌──────┐                   ┌──────┐    
           │ Rept │                   │ Rept │    
           └──────┘                   └──────┘    
           ┌──────┐                   ┌──────┐    
    ┌─────▶│ Join │◀───────┐          │ Scan │    
    │      └──────┘        │          └──────┘    
┌──────┐                   │                      
│ Brod │                   │                      
└──────┘                   │                      
┌──────┐                   │                      
│ Scan │               ┌──────┐                   
└──────┘        ┌─────▶│ Join │◀───────┐          
                │      └──────┘        │          
            ┌──────┐               ┌──────┐       
            │ Rept │               │ Rept │       
            └──────┘               └──────┘       
            ┌──────┐               ┌──────┐       
            │ Scan │            ┌─▶│ Join │◀──┐   
            └──────┘            │  └──────┘   │   
                            ┌──────┐          │   
                            │ Brod │          │   
                            └──────┘          │   
                            ┌──────┐      ┌──────┐
                            │ Scan │      │ Scan │
                            └──────┘      └──────┘

Where:

  • Join: a HashJoinExec
  • Rept: a RepartitionExec(Hash)
  • Brod: a BroadcastExec
  • Scan: a DataSourceExec that scans data

If we start walking the plan bottom-up, we can identify those stages which do not depend on any other stage to complete:

                               ┌──────┐                    
                     ┌────────▶│ Join │◀────────┐          
                     │         └──────┘         │          
                 ┌──────┐                   ┌──────┐│      
                 │ Rept │                   │ Rept ││      
                 └──────┘                   └──────┘│4     
                 ┌──────┐                   ┌──────┐│      
          ┌─────▶│ Join │◀───────┐          │ Scan ││      
          │      └──────┘        │          └──────┘▼      
     │┌──────┐                   │                         
     ││ Brod │                   │                         
    1│└──────┘                   │                         
     │┌──────┐                   │                         
     ││ Scan │               ┌──────┐                      
     ▼└──────┘        ┌─────▶│ Join │◀───────┐             
                      │      └──────┘        │             
                 │┌──────┐               ┌──────┐          
                 ││ Rept │               │ Rept │          
                 │└──────┘               └──────┘          
                2│┌──────┐               ┌──────┐          
                 ││ Scan │            ┌─▶│ Join │◀──┐      
                 ▼└──────┘            │  └──────┘   │      
                                 │┌──────┐          │      
                                 ││ Brod │          │      
                                3│└──────┘          │      
                                 │┌──────┐      ┌──────┐   
                                 ││ Scan │      │ Scan │   
                                 ▼└──────┘      └──────┘   

In that plan, we can identify 4 leaf stages as non-dependent on any other stage, therefore, they are good to be scheduled right away for execution. If we separate those stages from the remainder of the plan, we get something like this:

                                                                
      Stage 1       Stage 2        Stage 3        Stage 4       
                                                                
     │┌──────┐     │┌──────┐      │┌──────┐      ┌──────┐│      
     ││ Brod │     ││ Rept │      ││ Brod │      │ Rept ││      
    1│└──────┘     │└──────┘     3│└──────┘      └──────┘│4     
     │┌──────┐    2│┌──────┐      │┌──────┐      ┌──────┐│      
     ││ Scan │     ││ Scan │      ││ Scan │      │ Scan ││      
     ▼└──────┘     ▼└──────┘      ▼└──────┘      └──────┘▼      
                                                                
                                                                
                                                                
                               Remainder                        
                                                                
                               ┌──────┐                         
                     ┌────────▶│ Join │◀──────┐                 
                     │         └──────┘       │                 
                 ┌──────┐                                       
                 │ Rept │                  Stage 4              
                 └──────┘                                       
                 ┌──────┐                                       
           ┌────▶│ Join │◀───────┐                              
           │     └──────┘        │                              
                                 │                              
        Stage 1                  │                              
                                 │                              
                                 │                              
                             ┌──────┐                           
                        ┌───▶│ Join │◀───────┐                  
                        │    └──────┘        │                  
                                         ┌──────┐               
                     Stage 2             │ Rept │               
                                         └──────┘               
                                         ┌──────┐               
                                    ┌───▶│ Join │◀──┐           
                                    │    └──────┘   │           
                                                    │           
                                 Stage 3            │           
                                                    │           
                                                ┌──────┐        
                                                │ Scan │        
                                                └──────┘        

All that subtree that has not yet been walked is the plan remainder, is what's left after doing a first pass to extract all the stages that have no other dependent stage.

During AQE, the plan remainder is a very good place to apply further optimizations, as information for the already sampled stages (1, 2, 3 and 4 in the example) can be used for reordering joins or other optimizations.

This process is recursive, the next pass will produce another independent stage, something like this:

                                            
                    Stage 5                 
                                            
                   ┌──────┐        │        
                   │ Rept │        │        
                   └──────┘        │        
                   ┌──────┐        │        
              ┌───▶│ Join │◀──┐    │        
              │    └──────┘   │    │5       
                              │    │        
           Stage 3            │    │        
                              │    │        
                          ┌──────┐ │        
                          │ Scan │ │        
                          └──────┘ ▼        
                                            
                                            
                                            
                        Remainder           
                                            
                        ┌──────┐            
              ┌────────▶│ Join │◀──────┐    
              │         └──────┘       │    
          ┌──────┐                          
          │ Rept │                  Stage 4 
          └──────┘                          
          ┌──────┐                          
    ┌────▶│ Join │◀───────┐                 
    │     └──────┘        │                 
                          │                 
 Stage 1                  │                 
                          │                 
                          │                 
                      ┌──────┐              
                 ┌───▶│ Join │◀────┐        
                 │    └──────┘     │        
                                            
              Stage 2           Stage 5     
                                            

And yet another pass will return this:

                                               
           Stage 6                             
                                               
          ┌──────┐                      │      
          │ Rept │                      │      
          └──────┘                      │      
          ┌──────┐                      │      
    ┌────▶│ Join │◀───────┐             │      
    │     └──────┘        │             │      
                          │             │      
 Stage 1                  │             │6     
                          │             │      
                          │             │      
                      ┌──────┐          │      
                 ┌───▶│ Join │◀────┐    │      
                 │    └──────┘     │    │      
                                        │      
              Stage 2           Stage 5 ▼      
                                               
                                               
                                               
                                               
                                               
                 Remainder                     
                                               
                 ┌──────┐                      
        ┌───────▶│ Join │◀──────┐              
        │        └──────┘       │              
                                               
     Stage 6                 Stage 4           
                                               

And upon doing another pass, there are no more stages that can be formed. At that moment, distributed planning has finished, and the remainder is treated as the head stage of the plan that runs on the coordinator:

                                 
             Remainder  == head stage         
                                 
             ┌──────┐            
    ┌───────▶│ Join │◀──────┐    
    │        └──────┘       │    
                                 
 Stage 6                 Stage 4 
                                 

This PR still does not apply any optimization to the plan remainder, but it establishes the foundation for re-planning it on the fly based on runtime stats.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@gabotechs
gabotechs added this pull request to stack #837 November 3, 2026 09:02
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch from bfb371a to 4e70ab4 Compare November 3, 2026 09:04
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 2 times, most recently from e0df46c to 99edf75 Compare November 3, 2026 15:33
Base automatically changed from gabrielmusat/fix-planner-annotations to main November 3, 2026 16:26
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 3 times, most recently from 99edf75 to 046e0ce Compare November 6, 2026 11:38
@gabotechs gabotechs changed the title Add a remainder replanner Introduces the concept of plan remainder replanning Nov 9, 2026
@gabotechs gabotechs changed the title Introduces the concept of plan remainder replanning Add a plan remainder replanning step Nov 9, 2026
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch from 046e0ce to 66637dc Compare November 9, 2026 10:16
@gabotechs
gabotechs removed this pull request from stack #837 November 9, 2026 10:17
@gabotechs
gabotechs changed the base branch from main to refactor/single-integration-test-binary November 9, 2026 10:18
@gabotechs
gabotechs added this pull request to stack #867 November 9, 2026 10:18
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 3 times, most recently from de48dad to d6ac55b Compare November 10, 2026 16:24
Base automatically changed from refactor/single-integration-test-binary to main November 10, 2026 16:34
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 2 times, most recently from c7523e9 to 76d5607 Compare November 14, 2026 11:59
@gabotechs
gabotechs removed this pull request from stack #867 November 14, 2026 12:00
@gabotechs
gabotechs changed the base branch from main to gabrielmusat/head-stage-task-finished November 14, 2026 12:00
@gabotechs
gabotechs added this pull request to stack #842 November 14, 2026 12:00
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 2 times, most recently from 89694f0 to fdc4392 Compare November 14, 2026 20:18
@gabotechs
gabotechs removed this pull request from stack #842 November 14, 2026 20:20
@gabotechs
gabotechs changed the base branch from gabrielmusat/head-stage-task-finished to gabrielmusat/plan-dumps November 14, 2026 20:20
@gabotechs
gabotechs added this pull request to stack #888 November 14, 2026 20:20
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch from fdc4392 to 4fb5a96 Compare November 15, 2026 17:17
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 3 times, most recently from 6d80554 to ef3a46a Compare November 16, 2026 08:55
Base automatically changed from gabrielmusat/plan-dumps to main November 16, 2026 09:21
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch 2 times, most recently from 5435e52 to 0002b70 Compare November 16, 2026 10:26
Adds the `dynamic_bytes_ready_sample_percentage` option to `DistributedConfig`, replacing the
hardcoded `BYTES_READY_SAMPLE_PERCENTAGE` used while gathering runtime statistics for dynamic task
count. The default stays at 0.2.

Setting it to 1.0 makes the coordinator wait for every partition of a stage before estimating its
statistics, which makes the estimations exact and deterministic. This is useful for tests that
assert on plans decided from runtime statistics.

# Conflicts:
#	datafusion/distributed/src/metrics/plan_dump/plan_dump_display.rs
A network boundary whose stage is already remote has no children, but DataFusion rules like
EnsureRequirements still call `with_new_children` on it with an empty list. Returning a fresh clone
gave the node a new identity, which dropped the task count annotations that the network boundary
injection keys by node pointer, failing the next planning pass. Remote boundaries now return
themselves, and local ones are rebuilt through `with_input_stage`.
The remainder replanner runs between network boundary injection passes, and replanning decisions
need the same context those passes use: the session and distributed configuration, and the task
counts annotated so far.
@gabotechs
gabotechs force-pushed the gabrielmusat/remainder-replanning branch from 0002b70 to 878ea35 Compare November 18, 2026 10:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant