Skip to content

perf: overlap ready task execution with setup - #881

Draft
gabotechs wants to merge 1 commit into
gabotechs/coordinator-to-worker-dialerfrom
gabotechs/worker-location
Draft

gabotechs wants to merge 1 commit into
gabotechs/coordinator-to-worker-dialerfrom
gabotechs/worker-location

Conversation

@gabotechs

@gabotechs gabotechs commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #880. An alternative to #852 that needs fewer changes.

🚧 Still under construction 🚧

Breaking changes

RemoteStage::workers changes from Vec<Url> to Vec<WorkerLocation>. Documented in docs/upgrade/5.0.0.md.

@gabotechs
gabotechs force-pushed the gabotechs/worker-location branch from f40c51b to c7f09bb Compare November 12, 2026 13:20
@gabotechs
gabotechs changed the base branch from gabotechs/task-reports-on-drop to gabotechs/coordinator-to-worker-dialer November 12, 2026 13:20
@gabotechs
gabotechs added this pull request to stack #883 November 12, 2026 13:20
@gabotechs
gabotechs force-pushed the gabotechs/worker-location branch 3 times, most recently from 4a0a681 to 2886488 Compare November 19, 2026 11:35
Task routing no longer blocks stage preparation. init_bidirectional_stream
now spawns each task's routing and returns a WorkerLocation right away,
which is pending until the task is routed. RemoteStage::workers becomes a
Vec<WorkerLocation>.

- The coordinator reads a task's output as soon as that task is routed,
  instead of waiting for all its siblings.
- Each task gets KickOffSampling as soon as it is routed, so AQE sampling
  starts per task.
- A task waits for its child stages' locations (ensure_all_resolved)
  before routing, since worker plans embed child stage URLs.
- Routing and work unit feeds stop when the query finishes.
@gabotechs
gabotechs force-pushed the gabotechs/worker-location branch from 2886488 to ce672e8 Compare November 19, 2026 11:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant