Skip to content

feat: add zero-wait distributed runtime filtering to AQE - #2749

Open
wirybeaver wants to merge 4 commits into
apache:mainfrom
wirybeaver:runtime-filter-aqe
Open

wirybeaver wants to merge 4 commits into
apache:mainfrom
wirybeaver:runtime-filter-aqe

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Part of #2700 (Task 2), building on the runtime filter operators and task reports.

Use completed build-side summaries to reduce probe-side row processing without adding a runtime-filter wait. Existing broadcast dependencies, as in Q19, can make summaries available before probe execution. Optional nonzero waiting is scoped to Task 4 (personal PR #13).

Modification

Attach build/apply operators only to executable stage copies after AQE chooses stage boundaries. A job-scoped hub merges reports from complete successful producer coverage through the existing task-status lifecycle.

At stage preparation, bind complete summaries already available or bypass unavailable filters, then freeze the stage plan. Scheduling proceeds without waiting for reports; independently runnable sibling probes may therefore execute without filtering.

build task completion reports -> job-scoped hub
                                      |
probe stage selected -----------------+
                                      v
                         complete summary available?
                           yes              no
                        bind filter     bypass filter
                              \           /
                           freeze plan -> task binding

The feature is disabled by default and covers the first supported direct-column equality key of inner and semi hash joins, including NULL-safe equality. Fully saturated Blooms bypass hashing while NULL and MinMax collection continue.

  • Reuse filters for partitioned fallback: attach existing build-side filters to probe stages created when AQE falls back to a partitioned join.
  • Bypass Bloom at ≥99% occupancy: skip probe-side Bloom checks when at least 99% of bits are set, avoiding low-value hashing while retaining summary metadata.

Benchmarking Result

Query shape A base median (ms) B HEAD median (ms) Median Delta
TPC-H Q1 540.19 542.68 +0.46%
TPC-H Q2 167.70 167.48 -0.13%
TPC-H Q3 674.06 696.31 +3.30%
TPC-H Q4 294.12 281.92 -4.15%
TPC-H Q5 416.84 423.09 +1.50%
TPC-H Q6 182.70 188.73 +3.30%
TPC-H Q7 933.36 948.88 +1.66%
TPC-H Q8 443.92 448.38 +1.01%
TPC-H Q9 1052.17 1024.84 -2.60%
TPC-H Q10 722.29 743.84 +2.98%
TPC-H Q11 78.57 76.08 -3.18%
TPC-H Q12 439.75 441.64 +0.43%
TPC-H Q13 478.74 475.40 -0.70%
TPC-H Q14 213.62 209.46 -1.95%
TPC-H Q15 199.61 197.68 -0.96%
TPC-H Q17 1529.20 1561.05 +2.08%
TPC-H Q18 627.43 625.33 -0.33%
TPC-H Q19 322.94 294.12 -8.93%
TPC-H Q20 521.63 515.52 -1.17%
TPC-H Q21 1847.91 1844.28 -0.20%
TPC-H Q22 208.52 208.95 +0.21%
TPC-DS Q3 152.42 151.42 -0.65%
TPC-DS Q7 468.20 471.95 +0.80%
TPC-DS Q13 556.39 559.18 +0.50%
TPC-DS Q17 279.95 281.63 +0.60%
TPC-DS Q19 218.74 216.64 -0.96%
TPC-DS Q25 361.95 361.06 -0.25%
TPC-DS Q29 290.36 292.88 +0.87%
TPC-DS Q42 148.97 147.31 -1.12%
TPC-DS Q52 156.46 155.35 -0.71%
TPC-DS Q55 150.11 151.87 +1.17%
TPC-DS Q72 1455.07 1474.80 +1.36%
TPC-DS Q82 361.41 365.06 +1.01%
TPC-DS Q96 91.31 92.36 +1.14%
Custom D1 — partitioned fallback 381.07 164.62 -56.80%
Custom D3 — near-saturated Bloom 181.83 184.68 +1.57%

A (upstream base) and B (PR HEAD, runtime filtering enabled) were compared on 21 TPC-H SF10 shapes (excluding Q16), 13 TPC-DS SF10 shapes, and two custom shapes using TPC-DS data. On one host with two 4-core/4-GiB executors per arm, each query ran five paired rounds (AB/BA/AB/BA/AB) after preflight and one suite warmup; each timed visit followed one short warmup, with servers kept running. All 72 correctness checks matched DataFusion and all 360 timed visits completed. Most latencies stayed close to base, with pronounced reductions for TPC-H Q19 (−8.93%) and custom D1 (−56.80%); Q3/Q6 measured +3.30%, whose cause remains unconfirmed.

Per-visit results — upstream base vs PR HEAD
Query shape Visit Arm Latency (ms)
TPC-H Q1 1A Upstream base 541.07
TPC-H Q1 1B PR HEAD 554.07
TPC-H Q2 1A Upstream base 169.86
TPC-H Q2 1B PR HEAD 167.59
TPC-H Q3 1A Upstream base 668.06
TPC-H Q3 1B PR HEAD 692.96
TPC-H Q4 1A Upstream base 294.12
TPC-H Q4 1B PR HEAD 292.60
TPC-H Q5 1A Upstream base 416.84
TPC-H Q5 1B PR HEAD 420.83
TPC-H Q6 1A Upstream base 192.75
TPC-H Q6 1B PR HEAD 192.13
TPC-H Q7 1A Upstream base 963.50
TPC-H Q7 1B PR HEAD 930.54
TPC-H Q8 1A Upstream base 453.12
TPC-H Q8 1B PR HEAD 446.13
TPC-H Q9 1A Upstream base 1077.59
TPC-H Q9 1B PR HEAD 1074.54
TPC-H Q10 1A Upstream base 722.29
TPC-H Q10 1B PR HEAD 763.14
TPC-H Q11 1A Upstream base 80.08
TPC-H Q11 1B PR HEAD 75.31
TPC-H Q12 1A Upstream base 458.28
TPC-H Q12 1B PR HEAD 450.54
TPC-H Q13 1A Upstream base 479.40
TPC-H Q13 1B PR HEAD 475.40
TPC-H Q14 1A Upstream base 213.62
TPC-H Q14 1B PR HEAD 209.46
TPC-H Q15 1A Upstream base 209.39
TPC-H Q15 1B PR HEAD 197.46
TPC-H Q17 1A Upstream base 1547.88
TPC-H Q17 1B PR HEAD 1577.78
TPC-H Q18 1A Upstream base 650.19
TPC-H Q18 1B PR HEAD 625.33
TPC-H Q19 1A Upstream base 322.94
TPC-H Q19 1B PR HEAD 299.53
TPC-H Q20 1A Upstream base 528.45
TPC-H Q20 1B PR HEAD 515.10
TPC-H Q21 1A Upstream base 1866.88
TPC-H Q21 1B PR HEAD 1850.19
TPC-H Q22 1A Upstream base 205.57
TPC-H Q22 1B PR HEAD 204.14
TPC-DS Q3 1A Upstream base 154.26
TPC-DS Q3 1B PR HEAD 153.46
TPC-DS Q7 1A Upstream base 482.38
TPC-DS Q7 1B PR HEAD 485.86
TPC-DS Q13 1A Upstream base 576.17
TPC-DS Q13 1B PR HEAD 566.95
TPC-DS Q17 1A Upstream base 290.93
TPC-DS Q17 1B PR HEAD 282.00
TPC-DS Q19 1A Upstream base 218.74
TPC-DS Q19 1B PR HEAD 216.64
TPC-DS Q25 1A Upstream base 365.03
TPC-DS Q25 1B PR HEAD 363.31
TPC-DS Q29 1A Upstream base 290.36
TPC-DS Q29 1B PR HEAD 295.31
TPC-DS Q42 1A Upstream base 148.01
TPC-DS Q42 1B PR HEAD 148.48
TPC-DS Q52 1A Upstream base 158.76
TPC-DS Q52 1B PR HEAD 155.35
TPC-DS Q55 1A Upstream base 149.41
TPC-DS Q55 1B PR HEAD 145.67
TPC-DS Q72 1A Upstream base 1503.70
TPC-DS Q72 1B PR HEAD 1474.80
TPC-DS Q82 1A Upstream base 368.84
TPC-DS Q82 1B PR HEAD 385.54
TPC-DS Q96 1A Upstream base 92.95
TPC-DS Q96 1B PR HEAD 92.36
Custom D1 — partitioned fallback 1A Upstream base 382.29
Custom D1 — partitioned fallback 1B PR HEAD 164.28
Custom D3 — near-saturated Bloom 1A Upstream base 185.05
Custom D3 — near-saturated Bloom 1B PR HEAD 177.82
TPC-H Q1 2B PR HEAD 542.18
TPC-H Q1 2A Upstream base 544.87
TPC-H Q2 2B PR HEAD 162.01
TPC-H Q2 2A Upstream base 165.46
TPC-H Q3 2B PR HEAD 740.53
TPC-H Q3 2A Upstream base 674.06
TPC-H Q4 2B PR HEAD 282.39
TPC-H Q4 2A Upstream base 318.26
TPC-H Q5 2B PR HEAD 423.09
TPC-H Q5 2A Upstream base 419.53
TPC-H Q6 2B PR HEAD 185.39
TPC-H Q6 2A Upstream base 182.09
TPC-H Q7 2B PR HEAD 948.88
TPC-H Q7 2A Upstream base 1009.84
TPC-H Q8 2B PR HEAD 448.38
TPC-H Q8 2A Upstream base 442.47
TPC-H Q9 2B PR HEAD 1043.22
TPC-H Q9 2A Upstream base 1070.44
TPC-H Q10 2B PR HEAD 744.50
TPC-H Q10 2A Upstream base 748.92
TPC-H Q11 2B PR HEAD 76.08
TPC-H Q11 2A Upstream base 79.77
TPC-H Q12 2B PR HEAD 428.83
TPC-H Q12 2A Upstream base 441.03
TPC-H Q13 2B PR HEAD 477.06
TPC-H Q13 2A Upstream base 467.81
TPC-H Q14 2B PR HEAD 206.69
TPC-H Q14 2A Upstream base 214.67
TPC-H Q15 2B PR HEAD 197.68
TPC-H Q15 2A Upstream base 197.48
TPC-H Q17 2B PR HEAD 1565.78
TPC-H Q17 2A Upstream base 1548.55
TPC-H Q18 2B PR HEAD 612.59
TPC-H Q18 2A Upstream base 619.43
TPC-H Q19 2B PR HEAD 294.12
TPC-H Q19 2A Upstream base 324.54
TPC-H Q20 2B PR HEAD 513.08
TPC-H Q20 2A Upstream base 511.42
TPC-H Q21 2B PR HEAD 1817.93
TPC-H Q21 2A Upstream base 1847.91
TPC-H Q22 2B PR HEAD 205.67
TPC-H Q22 2A Upstream base 208.52
TPC-DS Q3 2B PR HEAD 151.42
TPC-DS Q3 2A Upstream base 149.49
TPC-DS Q7 2B PR HEAD 469.98
TPC-DS Q7 2A Upstream base 467.17
TPC-DS Q13 2B PR HEAD 579.30
TPC-DS Q13 2A Upstream base 569.67
TPC-DS Q17 2B PR HEAD 278.04
TPC-DS Q17 2A Upstream base 283.08
TPC-DS Q19 2B PR HEAD 218.40
TPC-DS Q19 2A Upstream base 215.53
TPC-DS Q25 2B PR HEAD 358.75
TPC-DS Q25 2A Upstream base 359.52
TPC-DS Q29 2B PR HEAD 295.11
TPC-DS Q29 2A Upstream base 290.85
TPC-DS Q42 2B PR HEAD 145.05
TPC-DS Q42 2A Upstream base 148.97
TPC-DS Q52 2B PR HEAD 157.16
TPC-DS Q52 2A Upstream base 157.62
TPC-DS Q55 2B PR HEAD 152.21
TPC-DS Q55 2A Upstream base 152.93
TPC-DS Q72 2B PR HEAD 1477.37
TPC-DS Q72 2A Upstream base 1453.53
TPC-DS Q82 2B PR HEAD 361.42
TPC-DS Q82 2A Upstream base 359.40
TPC-DS Q96 2B PR HEAD 91.07
TPC-DS Q96 2A Upstream base 91.31
Custom D1 — partitioned fallback 2B PR HEAD 160.52
Custom D1 — partitioned fallback 2A Upstream base 381.51
Custom D3 — near-saturated Bloom 2B PR HEAD 185.06
Custom D3 — near-saturated Bloom 2A Upstream base 183.99
TPC-H Q1 3A Upstream base 534.54
TPC-H Q1 3B PR HEAD 538.06
TPC-H Q2 3A Upstream base 163.26
TPC-H Q2 3B PR HEAD 167.48
TPC-H Q3 3A Upstream base 658.83
TPC-H Q3 3B PR HEAD 699.83
TPC-H Q4 3A Upstream base 282.90
TPC-H Q4 3B PR HEAD 281.92
TPC-H Q5 3A Upstream base 412.01
TPC-H Q5 3B PR HEAD 411.13
TPC-H Q6 3A Upstream base 176.61
TPC-H Q6 3B PR HEAD 178.14
TPC-H Q7 3A Upstream base 933.36
TPC-H Q7 3B PR HEAD 954.68
TPC-H Q8 3A Upstream base 443.92
TPC-H Q8 3B PR HEAD 433.66
TPC-H Q9 3A Upstream base 1005.41
TPC-H Q9 3B PR HEAD 1024.84
TPC-H Q10 3A Upstream base 729.77
TPC-H Q10 3B PR HEAD 694.19
TPC-H Q11 3A Upstream base 76.09
TPC-H Q11 3B PR HEAD 74.88
TPC-H Q12 3A Upstream base 429.96
TPC-H Q12 3B PR HEAD 422.44
TPC-H Q13 3A Upstream base 478.74
TPC-H Q13 3B PR HEAD 469.93
TPC-H Q14 3A Upstream base 211.74
TPC-H Q14 3B PR HEAD 204.93
TPC-H Q15 3A Upstream base 197.39
TPC-H Q15 3B PR HEAD 189.57
TPC-H Q17 3A Upstream base 1526.38
TPC-H Q17 3B PR HEAD 1554.94
TPC-H Q18 3A Upstream base 627.43
TPC-H Q18 3B PR HEAD 631.81
TPC-H Q19 3A Upstream base 328.59
TPC-H Q19 3B PR HEAD 278.89
TPC-H Q20 3A Upstream base 501.57
TPC-H Q20 3B PR HEAD 522.16
TPC-H Q21 3A Upstream base 1812.46
TPC-H Q21 3B PR HEAD 1855.09
TPC-H Q22 3A Upstream base 222.18
TPC-H Q22 3B PR HEAD 208.95
TPC-DS Q3 3A Upstream base 155.58
TPC-DS Q3 3B PR HEAD 153.21
TPC-DS Q7 3A Upstream base 468.11
TPC-DS Q7 3B PR HEAD 471.95
TPC-DS Q13 3A Upstream base 556.22
TPC-DS Q13 3B PR HEAD 556.47
TPC-DS Q17 3A Upstream base 279.95
TPC-DS Q17 3B PR HEAD 280.18
TPC-DS Q19 3A Upstream base 213.60
TPC-DS Q19 3B PR HEAD 212.18
TPC-DS Q25 3A Upstream base 361.95
TPC-DS Q25 3B PR HEAD 361.06
TPC-DS Q29 3A Upstream base 289.05
TPC-DS Q29 3B PR HEAD 289.07
TPC-DS Q42 3A Upstream base 148.28
TPC-DS Q42 3B PR HEAD 147.31
TPC-DS Q52 3A Upstream base 150.13
TPC-DS Q52 3B PR HEAD 153.08
TPC-DS Q55 3A Upstream base 156.04
TPC-DS Q55 3B PR HEAD 151.28
TPC-DS Q72 3A Upstream base 1455.07
TPC-DS Q72 3B PR HEAD 1439.25
TPC-DS Q82 3A Upstream base 361.41
TPC-DS Q82 3B PR HEAD 365.06
TPC-DS Q96 3A Upstream base 91.18
TPC-DS Q96 3B PR HEAD 102.15
Custom D1 — partitioned fallback 3A Upstream base 377.14
Custom D1 — partitioned fallback 3B PR HEAD 168.94
Custom D3 — near-saturated Bloom 3A Upstream base 181.83
Custom D3 — near-saturated Bloom 3B PR HEAD 179.64
TPC-H Q1 4B PR HEAD 542.68
TPC-H Q1 4A Upstream base 540.19
TPC-H Q2 4B PR HEAD 168.44
TPC-H Q2 4A Upstream base 167.70
TPC-H Q3 4B PR HEAD 696.31
TPC-H Q3 4A Upstream base 680.48
TPC-H Q4 4B PR HEAD 275.20
TPC-H Q4 4A Upstream base 284.48
TPC-H Q5 4B PR HEAD 423.58
TPC-H Q5 4A Upstream base 426.34
TPC-H Q6 4B PR HEAD 189.41
TPC-H Q6 4A Upstream base 187.27
TPC-H Q7 4B PR HEAD 920.85
TPC-H Q7 4A Upstream base 920.38
TPC-H Q8 4B PR HEAD 459.27
TPC-H Q8 4A Upstream base 443.62
TPC-H Q9 4B PR HEAD 1021.48
TPC-H Q9 4A Upstream base 1017.94
TPC-H Q10 4B PR HEAD 701.19
TPC-H Q10 4A Upstream base 706.59
TPC-H Q11 4B PR HEAD 76.25
TPC-H Q11 4A Upstream base 78.57
TPC-H Q12 4B PR HEAD 441.64
TPC-H Q12 4A Upstream base 438.88
TPC-H Q13 4B PR HEAD 477.33
TPC-H Q13 4A Upstream base 468.68
TPC-H Q14 4B PR HEAD 221.67
TPC-H Q14 4A Upstream base 212.06
TPC-H Q15 4B PR HEAD 200.00
TPC-H Q15 4A Upstream base 199.61
TPC-H Q17 4B PR HEAD 1546.99
TPC-H Q17 4A Upstream base 1529.20
TPC-H Q18 4B PR HEAD 618.49
TPC-H Q18 4A Upstream base 615.09
TPC-H Q19 4B PR HEAD 310.73
TPC-H Q19 4A Upstream base 321.20
TPC-H Q20 4B PR HEAD 515.52
TPC-H Q20 4A Upstream base 521.63
TPC-H Q21 4B PR HEAD 1844.28
TPC-H Q21 4A Upstream base 1833.51
TPC-H Q22 4B PR HEAD 224.28
TPC-H Q22 4A Upstream base 208.04
TPC-DS Q3 4B PR HEAD 149.28
TPC-DS Q3 4A Upstream base 150.87
TPC-DS Q7 4B PR HEAD 476.29
TPC-DS Q7 4A Upstream base 469.54
TPC-DS Q13 4B PR HEAD 559.18
TPC-DS Q13 4A Upstream base 556.39
TPC-DS Q17 4B PR HEAD 281.63
TPC-DS Q17 4A Upstream base 278.88
TPC-DS Q19 4B PR HEAD 214.89
TPC-DS Q19 4A Upstream base 219.27
TPC-DS Q25 4B PR HEAD 355.81
TPC-DS Q25 4A Upstream base 352.14
TPC-DS Q29 4B PR HEAD 286.77
TPC-DS Q29 4A Upstream base 289.68
TPC-DS Q42 4B PR HEAD 145.70
TPC-DS Q42 4A Upstream base 152.22
TPC-DS Q52 4B PR HEAD 158.71
TPC-DS Q52 4A Upstream base 156.46
TPC-DS Q55 4B PR HEAD 151.87
TPC-DS Q55 4A Upstream base 150.11
TPC-DS Q72 4B PR HEAD 1486.46
TPC-DS Q72 4A Upstream base 1405.06
TPC-DS Q82 4B PR HEAD 363.25
TPC-DS Q82 4A Upstream base 352.81
TPC-DS Q96 4B PR HEAD 88.68
TPC-DS Q96 4A Upstream base 89.06
Custom D1 — partitioned fallback 4B PR HEAD 168.68
Custom D1 — partitioned fallback 4A Upstream base 368.69
Custom D3 — near-saturated Bloom 4B PR HEAD 184.68
Custom D3 — near-saturated Bloom 4A Upstream base 176.33
TPC-H Q1 5A Upstream base 530.48
TPC-H Q1 5B PR HEAD 553.09
TPC-H Q2 5A Upstream base 175.15
TPC-H Q2 5B PR HEAD 164.01
TPC-H Q3 5A Upstream base 718.47
TPC-H Q3 5B PR HEAD 692.71
TPC-H Q4 5A Upstream base 298.81
TPC-H Q4 5B PR HEAD 276.81
TPC-H Q5 5A Upstream base 412.04
TPC-H Q5 5B PR HEAD 449.96
TPC-H Q6 5A Upstream base 182.70
TPC-H Q6 5B PR HEAD 188.73
TPC-H Q7 5A Upstream base 900.01
TPC-H Q7 5B PR HEAD 1016.40
TPC-H Q8 5A Upstream base 450.47
TPC-H Q8 5B PR HEAD 450.34
TPC-H Q9 5A Upstream base 1052.17
TPC-H Q9 5B PR HEAD 1007.39
TPC-H Q10 5A Upstream base 703.41
TPC-H Q10 5B PR HEAD 743.84
TPC-H Q11 5A Upstream base 77.88
TPC-H Q11 5B PR HEAD 77.06
TPC-H Q12 5A Upstream base 439.75
TPC-H Q12 5B PR HEAD 452.85
TPC-H Q13 5A Upstream base 505.84
TPC-H Q13 5B PR HEAD 470.51
TPC-H Q14 5A Upstream base 221.99
TPC-H Q14 5B PR HEAD 213.76
TPC-H Q15 5A Upstream base 200.21
TPC-H Q15 5B PR HEAD 207.09
TPC-H Q17 5A Upstream base 1511.23
TPC-H Q17 5B PR HEAD 1561.05
TPC-H Q18 5A Upstream base 634.68
TPC-H Q18 5B PR HEAD 626.11
TPC-H Q19 5A Upstream base 314.24
TPC-H Q19 5B PR HEAD 286.17
TPC-H Q20 5A Upstream base 559.15
TPC-H Q20 5B PR HEAD 542.14
TPC-H Q21 5A Upstream base 1886.51
TPC-H Q21 5B PR HEAD 1841.36
TPC-H Q22 5A Upstream base 217.85
TPC-H Q22 5B PR HEAD 214.70
TPC-DS Q3 5A Upstream base 152.42
TPC-DS Q3 5B PR HEAD 150.17
TPC-DS Q7 5A Upstream base 468.20
TPC-DS Q7 5B PR HEAD 404.90
TPC-DS Q13 5A Upstream base 520.21
TPC-DS Q13 5B PR HEAD 527.70
TPC-DS Q17 5A Upstream base 279.24
TPC-DS Q17 5B PR HEAD 283.58
TPC-DS Q19 5A Upstream base 228.94
TPC-DS Q19 5B PR HEAD 221.14
TPC-DS Q25 5A Upstream base 364.50
TPC-DS Q25 5B PR HEAD 372.18
TPC-DS Q29 5A Upstream base 291.85
TPC-DS Q29 5B PR HEAD 292.88
TPC-DS Q42 5A Upstream base 153.68
TPC-DS Q42 5B PR HEAD 152.70
TPC-DS Q52 5A Upstream base 151.79
TPC-DS Q52 5B PR HEAD 150.02
TPC-DS Q55 5A Upstream base 148.17
TPC-DS Q55 5B PR HEAD 151.99
TPC-DS Q72 5A Upstream base 1458.31
TPC-DS Q72 5B PR HEAD 1462.36
TPC-DS Q82 5A Upstream base 366.82
TPC-DS Q82 5B PR HEAD 431.28
TPC-DS Q96 5A Upstream base 94.99
TPC-DS Q96 5B PR HEAD 103.72
Custom D1 — partitioned fallback 5A Upstream base 381.07
Custom D1 — partitioned fallback 5B PR HEAD 164.62
Custom D3 — near-saturated Bloom 5A Upstream base 177.04
Custom D3 — near-saturated Bloom 5B PR HEAD 185.01

Custom D1 — partitioned fallback

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;

Custom D3 — near-saturated Bloom

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE CAST(c.c_birth_month AS VARCHAR) IN ('1', '2')
AND s.ss_quantity BETWEEN 10 AND 90;

Non-default query options (main A/B benchmark)

Query option Applies to A B
datafusion.optimizer.prefer_hash_join All queries true true
ballista.planner.runtime_filter.enabled All queries Not available true
ballista.optimizer.broadcast_join_threshold_bytes D1 4096 4096
ballista.optimizer.broadcast_join_threshold_bytes D3 1500000 1500000
ballista.optimizer.stage_build_side_max_estimate_multiple D1, D3 1024 1024
datafusion.optimizer.default_filter_selectivity D3 100 100
datafusion.optimizer.hash_join_single_partition_threshold D3 0 0
datafusion.optimizer.hash_join_single_partition_threshold_rows D3 0 0

Bloom bypass threshold: 100% vs 99%

Query shape A: upstream base (ms) B: 100% (ms) C: 99% (ms)
Custom D3 — near-saturated Bloom 175.45 204.09 174.94

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 2, 2026
@wirybeaver
wirybeaver force-pushed the runtime-filter-aqe branch 2 times, most recently from ff642c2 to 957f55a Compare November 3, 2026 06:15
@wirybeaver
wirybeaver marked this pull request as ready for review November 3, 2026 19:16
@wirybeaver

Copy link
Copy Markdown
Contributor Author

@andygrove The TPC-H benchmark shows ~10% improvement for the Q19.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer: this review was generated by an LLM (Claude Code) at my request, against 957f55ae6. File/line references were checked against the code and the planner behaviour below was reproduced with scratch planner tests, but please treat the conclusions as suggestions and push back where it's wrong.

Thanks for this, and for gating it behind a default-off flag and posting TPC-H numbers. I couldn't find a wrong-result path:

  • Filtering either input is safe for inner and semi joins.
  • NULL-safe equality keeps NULL keys only when the build saw one.
  • Filters never go below a fetch or a non-value-preserving operator.
  • A payload is bound only when every successful producer task reported.
  • The fixed-seed foldhash state hashes the same on every executor.

It also merges cleanly with current main (after #2682 and #2671), and the scheduler lib tests pass on the merged tree.

My main question is where the cross-stage path fires today.

1. The partitioned fallback never gets the filter. With stage_build_side on (the default), a build side that measures too large leaves a pending probe exchange that is created after the build has finished. That is the one case where a complete filter can cut the probe's shuffle with no waiting. But discover_pair returns as soon as the build exchange is shuffle_created() (runtime_filter_planner.rs:113), so the probe stage is planned without an apply and the build tap's work is wasted. Reproduced with MemTable scans, a filter on both sides and broadcast_join_threshold_bytes = 500:

pass 1, stage 0:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
  RuntimeFilterBuildExec: id=0, key=id@0
    FilterExec: val@1 > 0
      DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]

build measured too large, pass 2, stage 1:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
  FilterExec: val@1 > 0, projection=[id@0]
    DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]

When the build already has a registered filter, binding a not-yet-planned probe exchange to it looks safe, and prepare would find the summary complete at first dispatch.

2. Sibling stages mostly bypass. When both sides are shuffled at once, the apply is planned in the probe stage. It only binds if the build finishes before the probe's first dispatch, which is unusual when slots are free. prepare also freezes the plan when bind_tasks_with merely fetches the stage (cluster/mod.rs:754), even if no task is bound in that round. Until the waiting follow-up lands, would it be worth not tapping builds whose only consumer is a sibling probe?

So today the path that reliably fires is a broadcast join whose probe scan sits in the same task as the build reader (Q19). It would help to say that in the docs section.

3. Bloom sizing. With 8 KiB and k=4, the false-positive rate is about:

  • 0.4% at Q19's 4,754 keys
  • 4% at 10k
  • 50% at ~30k
  • 99% at 100k

RuntimeFilterApplyExec only skips at full saturation (~191k distinct keys), so in between every probe row is hashed for little pruning. prepare already holds the merged Bloom, so a cheap guard is to bypass when its fill ratio implies a high false-positive rate. The MinMax bounds aren't used for row filtering either; a range check is cheap and stays selective after the Bloom saturates.

4. Tests. runtime_filters_preserve_adaptive_planning (stage_build_side.rs:171) can't fail. Its StatsExec tables aren't DataSourceExec, so no filter is ever placed; I checked, and there is no tap or apply in either pass. assert_plan! snapshots of the instrumented stage plans over MemTable scans would pin the plan shape the way the other AQE rules' tests do. Cover the broadcast, fallback and sibling flows. That would also cover the EXPLAIN diff I asked for on #2701. Plain EXPLAIN renders the static planner's stages, so these operators only appear under EXPLAIN ANALYZE; an excerpt for Q19 in the PR body would help.

5. Benchmark details. Could you add the cluster setup, the settings used for A and B (AQE, prefer_hash_join, stage_build_side), and the run count? With three runs per side the ±3% rows are hard to read; Q19 looks real. The numbers predate #2671, whose distinct counts feed the join estimates, so a re-run on main would be good. TPC-DS star joins would exercise this more than TPC-H.

Smaller things:

  • Each successful build task keeps an 8–9 KB payload per filter in TaskInfo for the life of the job, so 10k tasks is about 80 MB per filter. Worth a line in the docs, as #2671 did for HLL.
  • instrument unwraps the boundary's stage id (runtime_filter_planner.rs:182-188). An internal error would be safer in the scheduler.
  • A consumer stage that re-runs after a rollback keeps the filter it first bound, because prepared is never cleared. That's fine for deterministic builds; worth a comment, or rebinding on a new attempt.
  • configs.md asks users to set prefer_hash_join = true, but DynamicJoinSelectionExec::to_actual_join says AQE deliberately ignores it. Is it needed?

@wirybeaver

wirybeaver commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. To clarify, each query has 3 measured runs per arm, thus the total measure number is 3*3 = 9. I’ve updated the PR body to make this explicit.

Will evaluate in this PR

  • Reuse existing filters for partitioned-fallback probes. [Done] (adopted)
  • Freeze filters on first actual task binding. [Done] (no gain)
  • Near-saturation Bloom bypass. [Done] (adopted)
  • Skipping sibling-only producers in zero-wait mode. [Done] (no gain)

I previously removed a broader speculative multi-join producer guard after SF10 experiments did not establish a clear benefit. The narrower sibling-only case still needs evaluation.

Already deferred to personal PRs

  • Task 3: MinMax-based Parquet pruning — PR #3. General row-level MinMax filtering can be evaluated there.
  • Task 4: Optional nonzero waiting, deadline wakeups, and early release for fully saturated partial Blooms — PR #13.

@wirybeaver wirybeaver changed the title feat: apply completed distributed runtime filters in AQE feat: add zero-wait distributed runtime filtering to AQE Nov 4, 2026
Attach taps to executable stage copies without changing AQE planning or stage selection. At first dispatch, freeze only summaries with complete current-attempt coverage and bypass unavailable producers. Keep saturated Bloom hashing bounded while continuing NULL and MinMax collection.
@wirybeaver

wirybeaver commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

TPC-DS SF10 candidate evaluation

Candidate / query A: Task 2 before candidates (ms) B: A + candidate (ms) P50 latency change A/B output checks
C1 — Reuse filters for partitioned fallback (D1) 370.44 163.23 -55.94% Matched DataFusion
C2 — Freeze on actual task binding (D2) 365.23 364.45 -0.21% Matched DataFusion
C3 — Bypass Bloom at ≥99% occupancy (D3) 212.36 176.43 -16.92% Matched DataFusion
C4 — Skip sibling-only producers (D4) 406.40 412.09 +1.40% Matched DataFusion
D1–D4 queries and query options

D1 — selective build, partitioned fallback

-- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 4096;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;

D2 — selective build, sibling stages

-- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;

D3 — near-saturated Bloom

-- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 1500000;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;
SET datafusion.optimizer.default_filter_selectivity = 100;
SET datafusion.optimizer.hash_join_single_partition_threshold = 0;
SET datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE CAST(c.c_birth_month AS VARCHAR) IN ('1', '2')
AND s.ss_quantity BETWEEN 10 AND 90;

D4 — broad build, sibling stages

-- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;

SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month <= 11
AND s.ss_quantity BETWEEN 10 AND 90;
Per-visit results — all timed standard and directed queries
Candidate Workload Round Arm Client query latency (ms)
C1 Q3 1 A 139.573
C1 Q3 1 B 141.601
C1 Q7 1 A 381.603
C1 Q7 1 B 376.289
C1 Q13 1 A 450.944
C1 Q13 1 B 463.980
C1 Q17 1 A 276.631
C1 Q17 1 B 272.633
C1 Q19 1 A 206.235
C1 Q19 1 B 210.868
C1 Q25 1 A 343.506
C1 Q25 1 B 351.103
C1 Q29 1 A 281.152
C1 Q29 1 B 279.793
C1 Q42 1 A 142.039
C1 Q42 1 B 140.895
C1 Q52 1 A 146.114
C1 Q52 1 B 144.760
C1 Q55 1 A 142.534
C1 Q55 1 B 141.652
C1 Q72 1 A 1341.684
C1 Q72 1 B 1361.048
C1 Q82 1 A 350.855
C1 Q82 1 B 343.780
C1 Q96 1 A 90.381
C1 Q96 1 B 90.704
C1 D1 1 A 370.439
C1 D1 1 B 165.261
C1 Q3 2 B 143.802
C1 Q3 2 A 144.062
C1 Q7 2 B 388.767
C1 Q7 2 A 382.769
C1 Q13 2 B 461.861
C1 Q13 2 A 469.104
C1 Q17 2 B 272.624
C1 Q17 2 A 272.777
C1 Q19 2 B 214.112
C1 Q19 2 A 215.072
C1 Q25 2 B 352.044
C1 Q25 2 A 355.051
C1 Q29 2 B 286.462
C1 Q29 2 A 286.535
C1 Q42 2 B 143.247
C1 Q42 2 A 142.377
C1 Q52 2 B 147.225
C1 Q52 2 A 148.907
C1 Q55 2 B 141.157
C1 Q55 2 A 144.260
C1 Q72 2 B 1372.871
C1 Q72 2 A 1371.554
C1 Q82 2 B 346.336
C1 Q82 2 A 342.188
C1 Q96 2 B 90.486
C1 Q96 2 A 92.562
C1 D1 2 B 163.231
C1 D1 2 A 359.540
C1 Q3 3 A 142.779
C1 Q3 3 B 142.353
C1 Q7 3 A 399.583
C1 Q7 3 B 390.156
C1 Q13 3 A 467.432
C1 Q13 3 B 462.071
C1 Q17 3 A 275.397
C1 Q17 3 B 277.305
C1 Q19 3 A 212.801
C1 Q19 3 B 215.422
C1 Q25 3 A 350.130
C1 Q25 3 B 357.501
C1 Q29 3 A 285.706
C1 Q29 3 B 289.286
C1 Q42 3 A 145.310
C1 Q42 3 B 143.834
C1 Q52 3 A 147.807
C1 Q52 3 B 150.047
C1 Q55 3 A 144.370
C1 Q55 3 B 142.970
C1 Q72 3 A 1361.111
C1 Q72 3 B 1407.371
C1 Q82 3 A 350.761
C1 Q82 3 B 346.947
C1 Q96 3 A 93.205
C1 Q96 3 B 91.096
C1 D1 3 A 358.885
C1 D1 3 B 163.331
C1 Q3 4 B 143.655
C1 Q3 4 A 143.944
C1 Q7 4 B 390.426
C1 Q7 4 A 402.178
C1 Q13 4 B 478.196
C1 Q13 4 A 464.199
C1 Q17 4 B 275.247
C1 Q17 4 A 273.162
C1 Q19 4 B 217.395
C1 Q19 4 A 215.928
C1 Q25 4 B 360.807
C1 Q25 4 A 355.689
C1 Q29 4 B 288.119
C1 Q29 4 A 316.799
C1 Q42 4 B 149.521
C1 Q42 4 A 145.937
C1 Q52 4 B 149.501
C1 Q52 4 A 158.581
C1 Q55 4 B 146.730
C1 Q55 4 A 148.443
C1 Q72 4 B 1349.907
C1 Q72 4 A 1361.220
C1 Q82 4 B 357.459
C1 Q82 4 A 356.170
C1 Q96 4 B 93.309
C1 Q96 4 A 89.533
C1 D1 4 B 161.566
C1 D1 4 A 373.229
C1 Q3 5 A 145.926
C1 Q3 5 B 145.528
C1 Q7 5 A 388.449
C1 Q7 5 B 390.472
C1 Q13 5 A 459.964
C1 Q13 5 B 462.119
C1 Q17 5 A 292.691
C1 Q17 5 B 277.124
C1 Q19 5 A 210.255
C1 Q19 5 B 211.269
C1 Q25 5 A 354.752
C1 Q25 5 B 354.024
C1 Q29 5 A 287.953
C1 Q29 5 B 283.471
C1 Q42 5 A 144.695
C1 Q42 5 B 141.589
C1 Q52 5 A 150.491
C1 Q52 5 B 146.455
C1 Q55 5 A 143.938
C1 Q55 5 B 144.481
C1 Q72 5 A 1363.989
C1 Q72 5 B 1386.571
C1 Q82 5 A 349.198
C1 Q82 5 B 363.567
C1 Q96 5 A 89.659
C1 Q96 5 B 91.040
C1 D1 5 A 376.422
C1 D1 5 B 157.786
C2 Q3 1 A 140.449
C2 Q3 1 B 144.952
C2 Q7 1 A 380.068
C2 Q7 1 B 384.166
C2 Q13 1 A 453.019
C2 Q13 1 B 453.472
C2 Q17 1 A 271.375
C2 Q17 1 B 269.430
C2 Q19 1 A 208.208
C2 Q19 1 B 206.711
C2 Q25 1 A 354.884
C2 Q25 1 B 343.782
C2 Q29 1 A 272.152
C2 Q29 1 B 280.976
C2 Q42 1 A 143.541
C2 Q42 1 B 140.206
C2 Q52 1 A 144.011
C2 Q52 1 B 148.081
C2 Q55 1 A 141.222
C2 Q55 1 B 145.483
C2 Q72 1 A 1349.611
C2 Q72 1 B 1367.826
C2 Q82 1 A 348.322
C2 Q82 1 B 348.998
C2 Q96 1 A 92.629
C2 Q96 1 B 91.566
C2 D2 1 A 369.558
C2 D2 1 B 167.770
C2 Q3 2 B 142.831
C2 Q3 2 A 147.322
C2 Q7 2 B 381.347
C2 Q7 2 A 391.021
C2 Q13 2 B 473.401
C2 Q13 2 A 458.773
C2 Q17 2 B 278.646
C2 Q17 2 A 280.106
C2 Q19 2 B 210.382
C2 Q19 2 A 219.228
C2 Q25 2 B 353.646
C2 Q25 2 A 347.326
C2 Q29 2 B 283.881
C2 Q29 2 A 285.423
C2 Q42 2 B 144.261
C2 Q42 2 A 144.887
C2 Q52 2 B 154.472
C2 Q52 2 A 149.364
C2 Q55 2 B 145.078
C2 Q55 2 A 149.314
C2 Q72 2 B 1396.799
C2 Q72 2 A 1380.563
C2 Q82 2 B 347.037
C2 Q82 2 A 348.790
C2 Q96 2 B 91.038
C2 Q96 2 A 90.441
C2 D2 2 B 364.451
C2 D2 2 A 378.450
C2 Q3 3 A 146.382
C2 Q3 3 B 142.836
C2 Q7 3 A 393.537
C2 Q7 3 B 391.289
C2 Q13 3 A 477.184
C2 Q13 3 B 470.162
C2 Q17 3 A 280.792
C2 Q17 3 B 275.195
C2 Q19 3 A 216.842
C2 Q19 3 B 218.255
C2 Q25 3 A 355.617
C2 Q25 3 B 355.687
C2 Q29 3 A 288.733
C2 Q29 3 B 288.661
C2 Q42 3 A 149.192
C2 Q42 3 B 143.577
C2 Q52 3 A 156.906
C2 Q52 3 B 154.383
C2 Q55 3 A 146.512
C2 Q55 3 B 146.376
C2 Q72 3 A 1350.820
C2 Q72 3 B 1380.919
C2 Q82 3 A 346.257
C2 Q82 3 B 349.253
C2 Q96 3 A 92.227
C2 Q96 3 B 90.405
C2 D2 3 A 164.580
C2 D2 3 B 369.568
C2 Q3 4 B 144.769
C2 Q3 4 A 143.479
C2 Q7 4 B 386.732
C2 Q7 4 A 394.957
C2 Q13 4 B 474.784
C2 Q13 4 A 462.351
C2 Q17 4 B 278.212
C2 Q17 4 A 279.024
C2 Q19 4 B 212.298
C2 Q19 4 A 211.063
C2 Q25 4 B 354.208
C2 Q25 4 A 353.842
C2 Q29 4 B 285.890
C2 Q29 4 A 289.008
C2 Q42 4 B 145.812
C2 Q42 4 A 142.695
C2 Q52 4 B 153.845
C2 Q52 4 A 150.268
C2 Q55 4 B 145.638
C2 Q55 4 A 147.457
C2 Q72 4 B 1364.210
C2 Q72 4 A 1398.951
C2 Q82 4 B 357.387
C2 Q82 4 A 355.815
C2 Q96 4 B 90.712
C2 Q96 4 A 92.204
C2 D2 4 B 371.228
C2 D2 4 A 365.232
C2 Q3 5 A 159.938
C2 Q3 5 B 146.792
C2 Q7 5 A 402.791
C2 Q7 5 B 401.886
C2 Q13 5 A 472.417
C2 Q13 5 B 464.697
C2 Q17 5 A 276.186
C2 Q17 5 B 281.513
C2 Q19 5 A 214.593
C2 Q19 5 B 210.577
C2 Q25 5 A 356.861
C2 Q25 5 B 360.634
C2 Q29 5 A 287.504
C2 Q29 5 B 283.985
C2 Q42 5 A 146.353
C2 Q42 5 B 144.598
C2 Q52 5 A 150.100
C2 Q52 5 B 148.459
C2 Q55 5 A 145.773
C2 Q55 5 B 147.225
C2 Q72 5 A 1347.075
C2 Q72 5 B 1379.192
C2 Q82 5 A 355.741
C2 Q82 5 B 347.907
C2 Q96 5 A 92.095
C2 Q96 5 B 92.762
C2 D2 5 A 361.319
C2 D2 5 B 162.052
C3 Q3 1 A 143.838
C3 Q3 1 B 147.828
C3 Q7 1 A 385.528
C3 Q7 1 B 385.969
C3 Q13 1 A 462.876
C3 Q13 1 B 469.505
C3 Q17 1 A 276.654
C3 Q17 1 B 275.708
C3 Q19 1 A 213.995
C3 Q19 1 B 212.383
C3 Q25 1 A 350.395
C3 Q25 1 B 352.774
C3 Q29 1 A 287.388
C3 Q29 1 B 280.856
C3 Q42 1 A 147.354
C3 Q42 1 B 146.256
C3 Q52 1 A 148.012
C3 Q52 1 B 147.974
C3 Q55 1 A 142.654
C3 Q55 1 B 144.428
C3 Q72 1 A 1352.516
C3 Q72 1 B 1386.678
C3 Q82 1 A 355.173
C3 Q82 1 B 346.486
C3 Q96 1 A 90.528
C3 Q96 1 B 92.406
C3 D3 1 A 213.775
C3 D3 1 B 175.904
C3 Q3 2 B 143.649
C3 Q3 2 A 148.690
C3 Q7 2 B 383.065
C3 Q7 2 A 380.904
C3 Q13 2 B 470.390
C3 Q13 2 A 467.996
C3 Q17 2 B 279.166
C3 Q17 2 A 277.221
C3 Q19 2 B 216.734
C3 Q19 2 A 219.234
C3 Q25 2 B 357.086
C3 Q25 2 A 351.216
C3 Q29 2 B 283.767
C3 Q29 2 A 285.298
C3 Q42 2 B 144.700
C3 Q42 2 A 144.893
C3 Q52 2 B 152.225
C3 Q52 2 A 153.677
C3 Q55 2 B 146.715
C3 Q55 2 A 146.738
C3 Q72 2 B 1359.548
C3 Q72 2 A 1438.394
C3 Q82 2 B 345.175
C3 Q82 2 A 349.696
C3 Q96 2 B 91.648
C3 Q96 2 A 89.571
C3 D3 2 B 176.433
C3 D3 2 A 211.525
C3 Q3 3 A 145.060
C3 Q3 3 B 143.178
C3 Q7 3 A 386.213
C3 Q7 3 B 402.068
C3 Q13 3 A 467.670
C3 Q13 3 B 470.949
C3 Q17 3 A 278.097
C3 Q17 3 B 278.001
C3 Q19 3 A 215.962
C3 Q19 3 B 215.260
C3 Q25 3 A 356.260
C3 Q25 3 B 361.252
C3 Q29 3 A 286.757
C3 Q29 3 B 288.625
C3 Q42 3 A 147.212
C3 Q42 3 B 144.469
C3 Q52 3 A 152.423
C3 Q52 3 B 152.584
C3 Q55 3 A 148.437
C3 Q55 3 B 145.682
C3 Q72 3 A 1384.417
C3 Q72 3 B 1394.090
C3 Q82 3 A 348.340
C3 Q82 3 B 350.663
C3 Q96 3 A 91.193
C3 Q96 3 B 93.827
C3 D3 3 A 211.847
C3 D3 3 B 186.382
C3 Q3 4 B 144.931
C3 Q3 4 A 145.613
C3 Q7 4 B 398.820
C3 Q7 4 A 392.707
C3 Q13 4 B 471.512
C3 Q13 4 A 491.067
C3 Q17 4 B 275.074
C3 Q17 4 A 279.370
C3 Q19 4 B 213.704
C3 Q19 4 A 214.631
C3 Q25 4 B 358.677
C3 Q25 4 A 359.556
C3 Q29 4 B 284.779
C3 Q29 4 A 281.839
C3 Q42 4 B 144.958
C3 Q42 4 A 142.834
C3 Q52 4 B 152.798
C3 Q52 4 A 154.628
C3 Q55 4 B 146.574
C3 Q55 4 A 143.393
C3 Q72 4 B 1365.393
C3 Q72 4 A 1390.942
C3 Q82 4 B 347.582
C3 Q82 4 A 354.104
C3 Q96 4 B 90.916
C3 Q96 4 A 91.038
C3 D3 4 B 179.405
C3 D3 4 A 212.362
C3 Q3 5 A 143.722
C3 Q3 5 B 145.566
C3 Q7 5 A 392.497
C3 Q7 5 B 397.473
C3 Q13 5 A 487.558
C3 Q13 5 B 477.143
C3 Q17 5 A 280.617
C3 Q17 5 B 281.618
C3 Q19 5 A 221.811
C3 Q19 5 B 213.506
C3 Q25 5 A 350.116
C3 Q25 5 B 359.556
C3 Q29 5 A 282.055
C3 Q29 5 B 287.609
C3 Q42 5 A 148.076
C3 Q42 5 B 148.504
C3 Q52 5 A 152.960
C3 Q52 5 B 151.998
C3 Q55 5 A 144.455
C3 Q55 5 B 146.729
C3 Q72 5 A 1375.539
C3 Q72 5 B 1367.740
C3 Q82 5 A 360.113
C3 Q82 5 B 354.839
C3 Q96 5 A 90.430
C3 Q96 5 B 90.914
C3 D3 5 A 215.562
C3 D3 5 B 176.021
C4 Q3 1 A 142.440
C4 Q3 1 B 149.397
C4 Q7 1 A 390.001
C4 Q7 1 B 392.755
C4 Q13 1 A 463.714
C4 Q13 1 B 477.979
C4 Q17 1 A 276.616
C4 Q17 1 B 277.898
C4 Q19 1 A 219.950
C4 Q19 1 B 218.480
C4 Q25 1 A 352.529
C4 Q25 1 B 353.686
C4 Q29 1 A 289.276
C4 Q29 1 B 285.099
C4 Q42 1 A 142.568
C4 Q42 1 B 145.566
C4 Q52 1 A 149.266
C4 Q52 1 B 147.445
C4 Q55 1 A 146.317
C4 Q55 1 B 145.327
C4 Q72 1 A 1345.623
C4 Q72 1 B 1433.301
C4 Q82 1 A 347.653
C4 Q82 1 B 352.537
C4 Q96 1 A 90.393
C4 Q96 1 B 96.324
C4 D4 1 A 412.539
C4 D4 1 B 417.481
C4 Q3 2 B 145.595
C4 Q3 2 A 148.267
C4 Q7 2 B 387.438
C4 Q7 2 A 401.299
C4 Q13 2 B 475.409
C4 Q13 2 A 475.017
C4 Q17 2 B 276.746
C4 Q17 2 A 277.535
C4 Q19 2 B 213.899
C4 Q19 2 A 218.838
C4 Q25 2 B 354.369
C4 Q25 2 A 358.076
C4 Q29 2 B 301.437
C4 Q29 2 A 287.606
C4 Q42 2 B 149.378
C4 Q42 2 A 147.480
C4 Q52 2 B 153.272
C4 Q52 2 A 151.356
C4 Q55 2 B 145.157
C4 Q55 2 A 147.559
C4 Q72 2 B 1403.837
C4 Q72 2 A 1424.927
C4 Q82 2 B 358.864
C4 Q82 2 A 349.899
C4 Q96 2 B 93.489
C4 Q96 2 A 94.113
C4 D4 2 B 410.699
C4 D4 2 A 404.287
C4 Q3 3 A 146.191
C4 Q3 3 B 145.601
C4 Q7 3 A 396.820
C4 Q7 3 B 388.376
C4 Q13 3 A 464.328
C4 Q13 3 B 474.095
C4 Q17 3 A 279.279
C4 Q17 3 B 275.461
C4 Q19 3 A 217.989
C4 Q19 3 B 212.997
C4 Q25 3 A 355.699
C4 Q25 3 B 360.235
C4 Q29 3 A 293.584
C4 Q29 3 B 287.144
C4 Q42 3 A 147.233
C4 Q42 3 B 142.296
C4 Q52 3 A 153.980
C4 Q52 3 B 152.960
C4 Q55 3 A 147.240
C4 Q55 3 B 146.861
C4 Q72 3 A 1369.950
C4 Q72 3 B 1381.866
C4 Q82 3 A 361.870
C4 Q82 3 B 350.817
C4 Q96 3 A 92.760
C4 Q96 3 B 90.126
C4 D4 3 A 406.402
C4 D4 3 B 412.274
C4 Q3 4 B 147.833
C4 Q3 4 A 145.064
C4 Q7 4 B 392.941
C4 Q7 4 A 398.162
C4 Q13 4 B 458.999
C4 Q13 4 A 473.364
C4 Q17 4 B 279.919
C4 Q17 4 A 273.116
C4 Q19 4 B 219.165
C4 Q19 4 A 216.472
C4 Q25 4 B 355.572
C4 Q25 4 A 356.746
C4 Q29 4 B 282.681
C4 Q29 4 A 289.236
C4 Q42 4 B 146.930
C4 Q42 4 A 147.123
C4 Q52 4 B 146.180
C4 Q52 4 A 155.828
C4 Q55 4 B 148.257
C4 Q55 4 A 145.163
C4 Q72 4 B 1366.750
C4 Q72 4 A 1395.273
C4 Q82 4 B 350.561
C4 Q82 4 A 350.865
C4 Q96 4 B 91.931
C4 Q96 4 A 94.543
C4 D4 4 B 395.323
C4 D4 4 A 416.333
C4 Q3 5 A 145.156
C4 Q3 5 B 142.605
C4 Q7 5 A 397.587
C4 Q7 5 B 407.597
C4 Q13 5 A 472.586
C4 Q13 5 B 459.130
C4 Q17 5 A 277.855
C4 Q17 5 B 273.623
C4 Q19 5 A 215.256
C4 Q19 5 B 211.242
C4 Q25 5 A 350.923
C4 Q25 5 B 354.980
C4 Q29 5 A 283.470
C4 Q29 5 B 290.061
C4 Q42 5 A 147.915
C4 Q42 5 B 145.058
C4 Q52 5 A 151.396
C4 Q52 5 B 155.833
C4 Q55 5 A 145.627
C4 Q55 5 B 144.945
C4 Q72 5 A 1347.422
C4 Q72 5 B 1367.505
C4 Q82 5 A 358.905
C4 Q82 5 B 363.043
C4 Q96 5 A 91.419
C4 Q96 5 B 89.483
C4 D4 5 A 401.409
C4 D4 5 B 412.087

@wirybeaver

Copy link
Copy Markdown
Contributor Author

@andygrove, could you take another look? The A/B experiments showed benefits from reusing filters for partitioned fallback and bypassing Bloom checks at ≥99% occupancy in their targeted cases, and I've added them in two separate commits. I've also rerun the benchmarks against the upstream base and updated the PR description with the results.

@wirybeaver
wirybeaver requested a review from andygrove November 11, 2026 16:47

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wirybeaver, this is a big step forward. Reusing the completed build for the partitioned fallback is exactly the case I hoped would light up, and the D1 result shows it. Thanks also for the much more thorough benchmark write-up. The paired AB/BA rounds and the correctness checks against DataFusion make the numbers easy to trust. Dropping the sibling-only skip after measuring it makes sense to me.

A few things from the last round still look open:

  • runtime_filters_preserve_adaptive_planning still can't fail, see inline.
  • instrument still unwraps the boundary's stage id, see inline.
  • Could the docs section mention that each successful build task keeps an ~8–9 KB payload per filter in TaskInfo for the life of the job?
  • Could it also say which joins benefit today? That's broadcast joins like Q19 plus the partitioned fallback.
  • The docs ask users to set prefer_hash_join = true and enable_dynamic_filter_pushdown = false, but I couldn't find anything that checks either one. Are they required? If so, what happens with the defaults? If not, could we drop them so the setup is just the one flag?
  • A short comment on prepared noting that a re-run consumer keeps the filter it first bound would help the next reader.

On the Bloom bypass, 99% occupancy is roughly 96% false positives, so there's still a band where we hash every probe row for little pruning. I'm happy for that to be part of the MinMax and sizing work in Task 3.

@wirybeaver

Copy link
Copy Markdown
Contributor Author

Thanks, @andygrove. The docs now show a single enable flag and describe the supported join paths and per-task memory cost. I also added a comment above prepared clarifying that retried consumers keep their original filter or bypass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants