Repository navigation
Conversation
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
from
November 3, 2026 09:04
bfb371a to
4e70ab4
Compare
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
2 times, most recently
from
November 3, 2026 15:33
e0df46c to
99edf75
Compare
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
3 times, most recently
from
November 6, 2026 11:38
99edf75 to
046e0ce
Compare
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
from
November 9, 2026 10:16
046e0ce to
66637dc
Compare
gabotechs
changed the base branch from
main
to
refactor/single-integration-test-binary
November 9, 2026 10:18
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
3 times, most recently
from
November 10, 2026 16:24
de48dad to
d6ac55b
Compare
Base automatically changed from
refactor/single-integration-test-binary
to
main
November 10, 2026 16:34
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
2 times, most recently
from
November 14, 2026 11:59
c7523e9 to
76d5607
Compare
gabotechs
changed the base branch from
main
to
gabrielmusat/head-stage-task-finished
November 14, 2026 12:00
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
2 times, most recently
from
November 14, 2026 20:18
89694f0 to
fdc4392
Compare
gabotechs
changed the base branch from
gabrielmusat/head-stage-task-finished
to
gabrielmusat/plan-dumps
November 14, 2026 20:20
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
from
November 15, 2026 17:17
fdc4392 to
4fb5a96
Compare
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
3 times, most recently
from
November 16, 2026 08:55
6d80554 to
ef3a46a
Compare
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
2 times, most recently
from
November 16, 2026 10:26
5435e52 to
0002b70
Compare
Adds the `dynamic_bytes_ready_sample_percentage` option to `DistributedConfig`, replacing the hardcoded `BYTES_READY_SAMPLE_PERCENTAGE` used while gathering runtime statistics for dynamic task count. The default stays at 0.2. Setting it to 1.0 makes the coordinator wait for every partition of a stage before estimating its statistics, which makes the estimations exact and deterministic. This is useful for tests that assert on plans decided from runtime statistics. # Conflicts: # datafusion/distributed/src/metrics/plan_dump/plan_dump_display.rs
A network boundary whose stage is already remote has no children, but DataFusion rules like EnsureRequirements still call `with_new_children` on it with an empty list. Returning a fresh clone gave the node a new identity, which dropped the task count annotations that the network boundary injection keys by node pointer, failing the next planning pass. Remote boundaries now return themselves, and local ones are rebuilt through `with_input_stage`.
The remainder replanner runs between network boundary injection passes, and replanning decisions need the same context those passes use: the session and distributed configuration, and the task counts annotated so far.
gabotechs
force-pushed
the
gabrielmusat/remainder-replanning
branch
from
November 18, 2026 10:05
0002b70 to
878ea35
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces the concept of plan remainder replanning to the distributed planner, expected to be used during AQE for re-planning the full fragment of the plan that has not yet been injected any network boundaries.
What is a plan remainder?
A plan remainder is what's left of a plan once all the stages in which it depends have been formed. For example, let's imagine this plan:
Where:
If we start walking the plan bottom-up, we can identify those stages which do not depend on any other stage to complete:
In that plan, we can identify 4 leaf stages as non-dependent on any other stage, therefore, they are good to be scheduled right away for execution. If we separate those stages from the remainder of the plan, we get something like this:
All that subtree that has not yet been walked is the plan remainder, is what's left after doing a first pass to extract all the stages that have no other dependent stage.
During AQE, the plan remainder is a very good place to apply further optimizations, as information for the already sampled stages (1, 2, 3 and 4 in the example) can be used for reordering joins or other optimizations.
This process is recursive, the next pass will produce another independent stage, something like this:
And yet another pass will return this:
And upon doing another pass, there are no more stages that can be formed. At that moment, distributed planning has finished, and the remainder is treated as the head stage of the plan that runs on the coordinator:
This PR still does not apply any optimization to the plan remainder, but it establishes the foundation for re-planning it on the fly based on runtime stats.
Stack created with GitHub Stacks CLI • Give Feedback 💬