Repository navigation
feat: add zero-wait distributed runtime filtering to AQE - #2749
wirybeaver wants to merge 4 commits into
Conversation
ff642c2 to
957f55a
Compare
|
@andygrove The TPC-H benchmark shows ~10% improvement for the Q19. |
andygrove
left a comment
There was a problem hiding this comment.
Disclaimer: this review was generated by an LLM (Claude Code) at my request, against 957f55ae6. File/line references were checked against the code and the planner behaviour below was reproduced with scratch planner tests, but please treat the conclusions as suggestions and push back where it's wrong.
Thanks for this, and for gating it behind a default-off flag and posting TPC-H numbers. I couldn't find a wrong-result path:
- Filtering either input is safe for inner and semi joins.
- NULL-safe equality keeps NULL keys only when the build saw one.
- Filters never go below a fetch or a non-value-preserving operator.
- A payload is bound only when every successful producer task reported.
- The fixed-seed foldhash state hashes the same on every executor.
It also merges cleanly with current main (after #2682 and #2671), and the scheduler lib tests pass on the merged tree.
My main question is where the cross-stage path fires today.
1. The partitioned fallback never gets the filter. With stage_build_side on (the default), a build side that measures too large leaves a pending probe exchange that is created after the build has finished. That is the one case where a complete filter can cut the probe's shuffle with no waiting. But discover_pair returns as soon as the build exchange is shuffle_created() (runtime_filter_planner.rs:113), so the probe stage is planned without an apply and the build tap's work is wasted. Reproduced with MemTable scans, a filter on both sides and broadcast_join_threshold_bytes = 500:
pass 1, stage 0:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
RuntimeFilterBuildExec: id=0, key=id@0
FilterExec: val@1 > 0
DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]
build measured too large, pass 2, stage 1:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
FilterExec: val@1 > 0, projection=[id@0]
DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]
When the build already has a registered filter, binding a not-yet-planned probe exchange to it looks safe, and prepare would find the summary complete at first dispatch.
2. Sibling stages mostly bypass. When both sides are shuffled at once, the apply is planned in the probe stage. It only binds if the build finishes before the probe's first dispatch, which is unusual when slots are free. prepare also freezes the plan when bind_tasks_with merely fetches the stage (cluster/mod.rs:754), even if no task is bound in that round. Until the waiting follow-up lands, would it be worth not tapping builds whose only consumer is a sibling probe?
So today the path that reliably fires is a broadcast join whose probe scan sits in the same task as the build reader (Q19). It would help to say that in the docs section.
3. Bloom sizing. With 8 KiB and k=4, the false-positive rate is about:
- 0.4% at Q19's 4,754 keys
- 4% at 10k
- 50% at ~30k
- 99% at 100k
RuntimeFilterApplyExec only skips at full saturation (~191k distinct keys), so in between every probe row is hashed for little pruning. prepare already holds the merged Bloom, so a cheap guard is to bypass when its fill ratio implies a high false-positive rate. The MinMax bounds aren't used for row filtering either; a range check is cheap and stays selective after the Bloom saturates.
4. Tests. runtime_filters_preserve_adaptive_planning (stage_build_side.rs:171) can't fail. Its StatsExec tables aren't DataSourceExec, so no filter is ever placed; I checked, and there is no tap or apply in either pass. assert_plan! snapshots of the instrumented stage plans over MemTable scans would pin the plan shape the way the other AQE rules' tests do. Cover the broadcast, fallback and sibling flows. That would also cover the EXPLAIN diff I asked for on #2701. Plain EXPLAIN renders the static planner's stages, so these operators only appear under EXPLAIN ANALYZE; an excerpt for Q19 in the PR body would help.
5. Benchmark details. Could you add the cluster setup, the settings used for A and B (AQE, prefer_hash_join, stage_build_side), and the run count? With three runs per side the ±3% rows are hard to read; Q19 looks real. The numbers predate #2671, whose distinct counts feed the join estimates, so a re-run on main would be good. TPC-DS star joins would exercise this more than TPC-H.
Smaller things:
- Each successful build task keeps an 8–9 KB payload per filter in
TaskInfofor the life of the job, so 10k tasks is about 80 MB per filter. Worth a line in the docs, as #2671 did for HLL. instrumentunwraps the boundary's stage id (runtime_filter_planner.rs:182-188). An internal error would be safer in the scheduler.- A consumer stage that re-runs after a rollback keeps the filter it first bound, because
preparedis never cleared. That's fine for deterministic builds; worth a comment, or rebinding on a new attempt. - configs.md asks users to set
prefer_hash_join = true, butDynamicJoinSelectionExec::to_actual_joinsays AQE deliberately ignores it. Is it needed?
|
Thanks for the review. To clarify, each query has 3 measured runs per arm, thus the total measure number is 3*3 = 9. I’ve updated the PR body to make this explicit. Will evaluate in this PR
I previously removed a broader speculative multi-join producer guard after SF10 experiments did not establish a clear benefit. The narrower sibling-only case still needs evaluation. Already deferred to personal PRs |
Attach taps to executable stage copies without changing AQE planning or stage selection. At first dispatch, freeze only summaries with complete current-attempt coverage and bypass unavailable producers. Keep saturated Bloom hashing bounded while continuing NULL and MinMax collection.
TPC-DS SF10 candidate evaluation
D1–D4 queries and query optionsD1 — selective build, partitioned fallback -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 4096;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;D2 — selective build, sibling stages -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;D3 — near-saturated Bloom -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 1500000;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;
SET datafusion.optimizer.default_filter_selectivity = 100;
SET datafusion.optimizer.hash_join_single_partition_threshold = 0;
SET datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE CAST(c.c_birth_month AS VARCHAR) IN ('1', '2')
AND s.ss_quantity BETWEEN 10 AND 90;D4 — broad build, sibling stages -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month <= 11
AND s.ss_quantity BETWEEN 10 AND 90;Per-visit results — all timed standard and directed queries
|
957f55a to
65162ce
Compare
|
@andygrove, could you take another look? The A/B experiments showed benefits from reusing filters for partitioned fallback and bypassing Bloom checks at ≥99% occupancy in their targeted cases, and I've added them in two separate commits. I've also rerun the benchmarks against the upstream base and updated the PR description with the results. |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @wirybeaver, this is a big step forward. Reusing the completed build for the partitioned fallback is exactly the case I hoped would light up, and the D1 result shows it. Thanks also for the much more thorough benchmark write-up. The paired AB/BA rounds and the correctness checks against DataFusion make the numbers easy to trust. Dropping the sibling-only skip after measuring it makes sense to me.
A few things from the last round still look open:
runtime_filters_preserve_adaptive_planningstill can't fail, see inline.instrumentstill unwraps the boundary's stage id, see inline.- Could the docs section mention that each successful build task keeps an ~8–9 KB payload per filter in
TaskInfofor the life of the job? - Could it also say which joins benefit today? That's broadcast joins like Q19 plus the partitioned fallback.
- The docs ask users to set
prefer_hash_join = trueandenable_dynamic_filter_pushdown = false, but I couldn't find anything that checks either one. Are they required? If so, what happens with the defaults? If not, could we drop them so the setup is just the one flag? - A short comment on
preparednoting that a re-run consumer keeps the filter it first bound would help the next reader.
On the Bloom bypass, 99% occupancy is roughly 96% false positives, so there's still a band where we hash every probe row for little pruning. I'm happy for that to be part of the MinMax and sizing work in Task 3.
|
Thanks, @andygrove. The docs now show a single enable flag and describe the supported join paths and per-task memory cost. I also added a comment above |
Motivation
Part of #2700 (Task 2), building on the runtime filter operators and task reports.
Use completed build-side summaries to reduce probe-side row processing without adding a runtime-filter wait. Existing broadcast dependencies, as in Q19, can make summaries available before probe execution. Optional nonzero waiting is scoped to Task 4 (personal PR #13).
Modification
Attach build/apply operators only to executable stage copies after AQE chooses stage boundaries. A job-scoped hub merges reports from complete successful producer coverage through the existing task-status lifecycle.
At stage preparation, bind complete summaries already available or bypass unavailable filters, then freeze the stage plan. Scheduling proceeds without waiting for reports; independently runnable sibling probes may therefore execute without filtering.
The feature is disabled by default and covers the first supported direct-column equality key of inner and semi hash joins, including NULL-safe equality. Fully saturated Blooms bypass hashing while NULL and MinMax collection continue.
Benchmarking Result
A (upstream base) and B (PR HEAD, runtime filtering enabled) were compared on 21 TPC-H SF10 shapes (excluding Q16), 13 TPC-DS SF10 shapes, and two custom shapes using TPC-DS data. On one host with two 4-core/4-GiB executors per arm, each query ran five paired rounds (AB/BA/AB/BA/AB) after preflight and one suite warmup; each timed visit followed one short warmup, with servers kept running. All 72 correctness checks matched DataFusion and all 360 timed visits completed. Most latencies stayed close to base, with pronounced reductions for TPC-H Q19 (−8.93%) and custom D1 (−56.80%); Q3/Q6 measured +3.30%, whose cause remains unconfirmed.
Per-visit results — upstream base vs PR HEAD
Custom D1 — partitioned fallback
Custom D3 — near-saturated Bloom
Non-default query options (main A/B benchmark)
datafusion.optimizer.prefer_hash_jointruetrueballista.planner.runtime_filter.enabledtrueballista.optimizer.broadcast_join_threshold_bytes40964096ballista.optimizer.broadcast_join_threshold_bytes15000001500000ballista.optimizer.stage_build_side_max_estimate_multiple10241024datafusion.optimizer.default_filter_selectivity100100datafusion.optimizer.hash_join_single_partition_threshold00datafusion.optimizer.hash_join_single_partition_threshold_rows00Bloom bypass threshold: 100% vs 99%