Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 9, 2026 21:11
6bfdfaf to
52659af
Compare
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-2
branch
from
November 10, 2026 10:21
d927948 to
cab7f00
Compare
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 10, 2026 10:22
52659af to
ef70cff
Compare
…crate (datafusion-contrib#866) Breaks the dependency cycle between `datafusion-distributed` and `datafusion-distributed-benchmarks`. Before this change, the core crate dev-depended on the benchmarks crate (for TPC-H/TPC-DS/ClickBench data in tests), and the benchmarks crate depended on the core crate. This implied that for every change in `datafusion-distributed`, the `datafusion-distributed-benchmarks` needed to be recompiled (as it's dependant) and then `datafusion-distributed` itself needs to be re-recompiled again, causing compilation times to be very slow. Here are some numbers of adding a small trivial change to one test before vs after: ## Before (32.58s) ``` time cargo test --lib test_select_all Compiling datafusion-distributed v4.0.0 (//datafusion-distributed/datafusion/distributed) Compiling datafusion-distributed-benchmarks v0.1.0 (//datafusion-distributed/datafusion/distributed-benchmarks) Finished `test` profile [unoptimized + debuginfo] target(s) in 32.58s ``` ## After (17.01s) ``` time cargo test --lib test_select_all Compiling datafusion-distributed v4.0.0 (//datafusion-distributed/datafusion/distributed) Finished `test` profile [unoptimized + debuginfo] target(s) in 17.01s ``` So for testing targets, this cuts compilation time in half
…tafusion-contrib#868) Compiles all integration tests of `datafusion-distributed` into one test binary instead of one binary per file. Before this change, each of the 32 files in `tests/` (plus `dynamic_filtering/`) produced its own ~300 MB binary that linked the whole dependency tree. After a library edit, cargo relinked every one of them. Running a single file's tests locally: `cargo test --features integration --test integration udfs::`. Rebuilding the library plus all test targets after a one-line library edit went from 15-16s to 4-8s locally
…rib#872) AQE stage preparation currently lives in a large async closure, which makes subsequent startup changes harder to review. Move that callback into `DynamicPlanBuilder`, implementing the existing `NetworkBoundaryBuilder` trait. Task setup still joins all destinations before sampling kickoff; coalesce behavior, task counts, dynamic filters, statistics and metrics are unchanged. This is a preliminary refactor for datafusion-contrib#852. Validation: 343 library tests passed, one ignored; formatting passed. The moved callback body was checked for equivalence after accounting for borrowed fields and the removed closure wrappers. Co-authored-by: Codex GPT-6 <noreply@localhost>
Task setup skew can postpone useful work on a ready worker. Add a Criterion benchmark for that scheduling case before the execution changes in datafusion-contrib#852. This PR changes no library behavior and adds no dependencies. The benchmark runs one CPU-heavy task and one light task over localhost gRPC, in both static and AQE modes. Each mode has three cases: no routing delay, a 250 ms routing delay on the light task, and the same delay on the heavy task. It measures execution through complete output collection, excluding worker startup and planning. Every iteration checks output `[0, 1]` and exactly one heavy remote execution. An untimed preflight reports whether CPU work starts and finishes before delayed sibling routing returns. Applied unchanged to datafusion-contrib#852, the benchmark demonstrates the conditional benefit: | Mode | Case | Baseline | With datafusion-contrib#852 | Paired change | |---|---|---:|---:|---:| | Static | No delay | 228.15 ms | 227.84 ms | -0.14% | | Static | Delayed light task | 479.29 ms | 294.10 ms | **-38.64%** | | Static | Delayed heavy task | 479.09 ms | 478.60 ms | -0.10% | | AQE | No delay | 228.23 ms | 227.86 ms | -0.16% | | AQE | Delayed light task | 479.37 ms | 294.36 ms | **-38.60%** | | AQE | Delayed heavy task | 479.52 ms | 478.79 ms | -0.15% | In both delayed-light preflights, heavy work started and finished before sibling routing returned with datafusion-contrib#852; neither witness occurred on the baseline. The controls stayed essentially flat. This demonstrates value when ready useful work can overlap a sibling's setup delay; it does not predict representative S3 or multi-host speedups. Validation: optimized builds and all six cases passed on baseline and candidate. Formatting and Criterion-target Clippy with warnings denied passed. <details> <summary>Measurement scope and reproduction</summary> Measurements used baseline `0291e08` (main `e5d6ea3` plus this benchmark) and candidate source matching datafusion-contrib#852 at `354d386`. The benchmark and lockfile were identical. The candidate binary was built before the final commit; source hashes were verified against that commit. This was a controlled synthetic experiment on a shared host, with the same four-CPU affinity for both binaries, four Tokio threads, ten Criterion samples per case, one-second warmup and a five-second target measurement window. Three outer pairs were attempted in baseline/candidate, candidate/baseline, baseline/candidate order. Two opposite-order pairs passed the predefined resource gate. The entire third pair was excluded because baseline selected-CPU average utilization p95 was 85.31%, exceeding the 85% threshold; the maximum-utilization threshold was 98%. Raw results for both sides of the excluded pair were retained. Results were not filtered by measured latency. Times above are medians of the accepted per-run Criterion medians; changes are medians of the paired percentage changes. Two outer pairs illustrate this scheduling case, rather than establishing production performance or precise population estimates. ```shell cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --save-baseline main # Run the same benchmark on the candidate revision: cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --baseline main ``` Use separate build directories or rebuild both local packages when switching revisions. Confirm the candidate's delayed-light preflight shows early work and the baseline's does not before interpreting timings. </details> --------- Co-authored-by: Codex GPT-6 <noreply@localhost>
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 11, 2026 14:35
77858c6 to
c59f43e
Compare
datafusion-contrib#868 added unquoted `run:` commands ending with `::` in `.github/workflows/ci.yml`. Because YAML treats a trailing `:` before a newline as a mapping value indicator, `.github/workflows/ci.yml` has been failing workflow parsing before any jobs start. Quote those `run:` commands so `.github/workflows/ci.yml` parses cleanly.
…-contrib#876) Distributed window queries can fail with `Expects PARTITION BY expression to be ordered` when a partition key is a projected literal. Planning correctly omits the constant key from the sort, but `DistributedCodec` drops its constant metadata when reconstructing the worker plan. Preserve `AcrossPartitions` metadata in the existing equivalence-class protobuf message for shuffle, coalesce, and broadcast boundaries. The additive field distinguishes per-partition constants, uniform constants with unknown values, and known values (including null). Reproducer using the repository's `weather` fixture and two localhost workers: ```sql SELECT typ, "RainToday", "WindGustDir" FROM ( SELECT typ, "RainToday", "WindGustDir", row_number() OVER (PARTITION BY typ, "RainToday", "WindGustDir") AS rn FROM (SELECT 'weather' AS typ, * FROM weather) WHERE "RainToday" = 'No' AND "WindGustDir" IN ('N', 'S') ) WHERE rn = 1 ORDER BY "WindGustDir"; ``` The integration regression fails with the original error on `main` and returns the two expected rows with this patch. The codec round-trip test covers all constant variants across all three boundary types. Validation: - `cargo test -p datafusion-distributed --features integration` - `cargo clippy -p datafusion-distributed --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check`
<!-- If this is your first time contributing to the project, please consider reviewing https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md In particular: All contributions are subject to the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). The DCO text is also included verbatim in the [dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt) file in the root directory of the repository. --> Just a small fix to the insta settings
…contrib#890) Every time a new config option lands in DataFusion or in `DistributedConfig`, benchmarking it means first adding a dedicated flag to `RunOpt` and wiring it into the session builder. That's easy to forget, it keeps `RunOpt` growing, and any option without a flag simply can't be benchmarked. This makes `dfbench run` take config options generically instead. They're plain `key=value` pairs, and any `datafusion.*` or `distributed.*` key works, in either form: ```yaml # .df-bench.yaml (local, gitignored) distributed.broadcast_joins: true datafusion.execution.target_partitions: 10 ``` ```shell dfbench run --dataset $DATASET -c distributed.broadcast_joins=true -c datafusion.execution.target_partitions=10 ``` The file defaults to `.df-bench.yaml` in the current directory, and `--config-path` points to a different one. Each entry goes through DataFusion's own config setter, in this order, so later sources win: ``` DATAFUSION_* env vars ──▶ .df-bench.yaml (or --config-path) ──▶ -c key=value ... (lowest) (file order) (highest) │ ▼ ConfigOptions::set(key, value) unknown keys fail the run ``` Options that aren't DataFusion configs, such as `--workers`, `-i`, `--debug` and `--work-unit-file-scan`, stay as flags. All the flags that only set a config option are gone, for example `--partitions`, `--batch-size`, `--compression` and `--dynamic`. Their replacement is the matching config key. Metrics collection is on by default, since `--debug` and dynamic task count read the collected metrics. Use `distributed.collect_metrics: false` to turn it off.
…datafusion-contrib#841) > [!WARNING] > This PR is big, and it's exclusively dev tooling with no implications in the production code. A very light review is recommended, as the idea is that LLMs are the main ones maintaining this tool. <!-- If this is your first time contributing to the project, please consider reviewing https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md In particular: All contributions are subject to the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). The DCO text is also included verbatim in the [dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt) file in the root directory of the repository. --> Closes datafusion-contrib#523 Ships a tool + a SKILL.md for LLMs (and humans) to debug execution plans and identify bottlenecks and weak spots. There's two main pieces this PR ships: - A plan dump generator function, `generate_plan_dump(plan)`, that generates a compressed logeable b64 string for people to add to their own observability layer. - A CLI for inspecting that plan (+ a SKILL.md to use it) ## Dump size Dumps are meant to live in logs, so they need to be small in size and CPU footprint. Here's a table that compares logging the ASCII plan (even if it was zstd-compressed and base64-encoded) with the plan dumps proposed in this PR: | Query | ASCII plan: bytes (CPU time) | ASCII + zstd + base64: bytes (CPU time) | Plan dump (proto + zstd + base64): bytes (CPU time) | |---|---:|---:|---:| | q1 | 40,614 (846 µs) | 6,716 (949 µs) | 3,640 (485 µs) | | q2 | 227,396 (3,544 µs) | 26,040 (4,184 µs) | 16,112 (1,701 µs) | | q3 | 101,454 (1,581 µs) | 13,436 (1,853 µs) | 6,948 (890 µs) | | q4 | 74,524 (1,300 µs) | 12,236 (1,590 µs) | 6,884 (712 µs) | | q5 | 158,965 (2,515 µs) | 18,984 (2,976 µs) | 10,036 (1,202 µs) | | q6 | 33,105 (573 µs) | 4,500 (656 µs) | 1,592 (419 µs) | | q7 | 175,123 (2,878 µs) | 22,648 (3,539 µs) | 13,024 (1,463 µs) | | q8 | 211,001 (3,358 µs) | 23,412 (3,843 µs) | 12,744 (1,533 µs) | | q9 | 175,677 (2,906 µs) | 23,828 (4,039 µs) | 14,008 (1,688 µs) | | q10 | 121,009 (1,948 µs) | 16,032 (2,254 µs) | 8,528 (974 µs) | | q11 | 154,003 (2,491 µs) | 18,296 (2,795 µs) | 10,524 (1,106 µs) | | q12 | 72,108 (1,150 µs) | 9,644 (1,344 µs) | 4,968 (665 µs) | | q13 | 79,315 (1,473 µs) | 13,884 (1,737 µs) | 8,096 (716 µs) | | q14 | 59,553 (844 µs) | 7,856 (992 µs) | 3,452 (507 µs) | | q16 | 105,161 (1,892 µs) | 15,836 (2,283 µs) | 9,008 (1,040 µs) | | q17 | 97,139 (1,454 µs) | 12,872 (1,717 µs) | 7,176 (762 µs) | | q18 | 129,084 (2,060 µs) | 17,200 (2,315 µs) | 9,468 (954 µs) | | q19 | 93,076 (1,455 µs) | 8,608 (1,577 µs) | 4,404 (1,111 µs) | | q20 | 154,741 (2,621 µs) | 22,196 (3,075 µs) | 13,080 (1,320 µs) | | q21 | 183,115 (3,058 µs) | 23,992 (3,564 µs) | 14,296 (1,548 µs) | | q22 | 94,338 (2,954 µs) | 12,200 (3,344 µs) | 6,576 (2,187 µs) | | **Total** | **2,540,501 (42,901 µs)** | **330,416 (50,626 µs)** | **184,564 (22,983 µs)** | ## Usage ### 1. Log a dump from your service Once the query has been fully executed, fold the worker metrics back into the plan and log its dump next to the rest of the query's telemetry: ```rust let plan = ctx.sql(sql).await?.create_physical_plan().await?; let batches = execute_stream(plan.clone(), ctx.task_ctx())?.try_collect::<Vec<_>>().await?; let plan = rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::PerTask).await?; tracing::info!(plan_dump = %generate_plan_dump(&plan)?, "query finished"); ``` A dump is a single base64 line. Here's a complete one for TPC-H q3 (SF1, 4 workers), 2.9 KB, which you can paste into `dfd plan` to try out the commands below: ``` KLUv/WIAgIcqbUQAY33rHsVztSify0X5XBYV/K4/UC9xHCPTIs6RVSPAHH9P2C7ZNpzK6iri0FonOuvDyIyx6jG0s2FyioWt9KxGx2pI27J9sYJP7el9YhoAN6yp9XwD4OMp25IMew7Lr29yEVsJvCo0JfAyABkANywAhB/3D39Z6Aj6Sjc2z5XUEB2m2g7aDtprmJlNp8fV5NHJWKJD331e9hFenqK10j8O4zcAblg0aWEaIUiQ/pS3SlTlu+sDqUeFT1+xrxCEFhYX1+3uweUmvrUCG6l6WEv2XhoEFEZKkxZG5diW+GC/3xvbFjo7IpWst73saNEV6U4Fjy79ZMlztysl19jNHa8PogR7zDp19SN8wkEN5OUAGqZuQavsnDLFZU4AFV4MbKUzPEQqPhutVGQPxbR7nzL0WlgECZ2jqCmwL4i7aoDwbOJhhf7Kn9i8VLwm8m9SmwuxbWkZ+XtR+bt+Qf326l5Zd7iiqkNTiXm4aYmEUWBMFddSNeV/CDRuJhwHuDyYQ3NANY1/lgVEMDxSyppFTuM/PX26B3NolbCe6riDrUYlGJYSLi0hrhmBOggpMSpr10/W/Zla5l155XVr4FldZbiRzbePOwPeH7ZZH59PELbipneIbTU/Q7M9WI3UjT1+NXN3IqEgbMV9pKmxbjVS+0m2fMPWQ9gKggYyMOAEGDFgEMAKIrxgcc3w02NYHLhqTOl+kk17tzR2oLGpq4T1cP23Q0pySiWspwOzctFHktTI9mVvPMl2V0fLSj5w9fQRJCsOWOfqaJkFO1cEmdEux4vswIWrAsID15M+/SQJTG0jvYEl4Cx7WDtZIE7MQJGY6mVGIM3R1j3u1mPr/q5b27pfIA5s6x7fzbbuve9Ad9q6B6HpbOvew5XY1r2IMp2ER0Ia8l7wBMYgeQWxrXbQLbbZUXRhVRIeCV/wBN43libZ3RMkV00qex9w09q7E/8uz1691ahuNyOtloY8GvJikIxBsmri8nVNAFL3QJ/e6qvENqNQnWE220+IGWpV3C0LvKkYxgRLIOHNBGZaHy85lOhIbEFgYVlpgaHCCSykcMH/uTrQo39w9x+dfdEVgiG9RGIJHQuwZmisFFirpaZVjVyqbNS4FrGqEQsWrAPuDgHI74F1AEJLCvk6QZV+MrXTPUGaLN8/lmc76JN86IX+VOVwlYKoE7/+q841QmWnwqJjLtdX8nwgPbacyKnFBYMhvYNGkMEVjV4QYmABhgwwXsigSP9eilT8onT2XHwitrdX2QaB0aDhJXJKSpQMB9EEBRmCEGNYnQ9OoHCxS+BEFmYdJTW4TUa5jukYygXXc0GS52j5j+4EY1tzq2VZQGplv3LbjIh0YCNUW/1qBlUEUAiAD9MHELbdGrgRHWEFH3smd7w7BVnLr1ZPQO8917vzaNT95vDtlazZTg1QTTYNHk1PkzvA1q8+tFu8Bk0wdDHRXC2hZ2W64qxqqBgkUJAdlwVuJJ2NQlRLJCtbXKhnksfSMfuj/SKv9mwGUHZCTxCEE91qWJ1tU+9N1VLCSgTIHhE5mFZ/lCO/yl8nCo3t0Cd3DPK4++gyHVmcQyFVCicDmtc51MZ2p1GwmgMQmEYEDtKm3qvkakGscgPfromaPBtYPwdk9lwW7QESrg8WIOcNsPz/0adeF+yaaaNdewUNhWDMHO59zvkG+zFK3x2BA+W5AGDV1ozn9T7sxXpmN1buzIL0+K2keR4OPu0QGvU4g/GIkNL10Bx5KNE+nZPRLVqCSss7LsqD561xjPF0vXJyLvtk7Dl1DRpHs5nzNA4CDgfc4482Mvw2o9rYjnPmmorQyrPFjvOcgzhYK6D4+uw4ZQfkp3U/ymd6SCNbbAqS52IXXUiKGhjTaWKHEuWfoewJhU88EdbhEVySuC0vADs0BTOyGU+hRsQB2VJeAiSk7Asr7kaBsPyFhP8hUDX6FpdsKC3C8aDAGD43oAjpOvvtYHjH/xwdnX2BHJ/0dB0b0iFiVUe0nkL3yHnIJAZuSjNfcpokRJ0y+B45XExgyiJf2c0wc9KoPPlADo2Ff2aAj9XPVXRsI8bVj2nglO04bECGGOnhuCzY0h0mUnvDN700tnUmvq1i272wSXNBnucZt9gQAopL7rhm4pINYCLZpdJOgzrXzPhqnPD0gFfcBtnbYRdY0SCJ+3wAGxue3RsTSxd5cXBpAAzOsC7P3NaUIF2DVN5yUyfhXvMO84AqB/7yqGTBa5zKkrmLdQZWY9OJWiablxq9BQorh5Dsbs2EwDlrn2wtZ+73U8M8Y5OcQl0hYU+M8tkbk7S5WZZ1DVQVdITAePji6t0/5SLkuGX8AzzYZv8ppErctZ1JJqKaeOJbroe46DFQVm6F7iVjYBMlURSFoiSKJJHc0ANT+hwuKCvlnoxoBMBYiNEJJ7T0FW7rHPOocX1ctbYtIDMbWpR/snHweZGAEQVKngerTvnXTuA+6rvk+iQZKT31rZsiQAeFywKQEroJENyuyjkJTqIcpHB1QWVBDoSrswSvojkADpK8T1MGv5xUB/JXkp5tJRdxQfNdziwEscQdeZ5nH+7lnlF8Qz275YWpi8ScsJcL8fd4K+ayWmuDsaZ4aBfHHRKd6fB6ORxKSwYu414o49GKQScI/zoqKmwo1BuFhXpsd7IkR3fobCY5qGgrdgsd50/w2Igpj4jV3ZDFPHFiKfmofC4VvAR887cc8RENRbB+JqI1G8KhKFFaBxMjxhrxD2+Rjw9ruGapfTtnqJZWKwjP3ERZvYA7q018i6wTIQTWUYgixmKeDIlZzdbA15F4xq2W7ACSx3EAzxG3PgmSIOdSm2RaLH3jK1vsDGMt3EwIRNDMkuGujdOCke+a5HiGk7FpCA85cpqSWDCEByFdefFTolh97Er1DsoVtULrT1qnhXTkWGQlBLOF2SZrTPPnEQM= ``` You can try this out yourself by checking out the branch of this PR locally and installing the `dfd` CLI with: ``` cargo install --path datafusion/distributed-cli/ pbpaste | dfd plan ``` ### 2. Get an overview of the stages Copy the dump from your logs and pass it to `dfd plan` as an argument, as a file path or through stdin. With no other arguments, it shows the tree of stages with their stage-level metrics, added up across tasks. Some metrics are elided below: ```rust $ pbpaste | dfd plan DistributedExec │ output_rows=11.62 K │ elapsed_compute=5.43ms │ output_bytes=363.1 KB │ … └─ Stage 3 (Coalesce) tasks=2 partitions=16 │ output_rows=11.62 K │ elapsed_compute=77.81ms │ output_bytes=363.1 KB │ … ├─ Stage 1 (Shuffle) tasks=1 partitions=32 │ output_rows=147.1 K │ elapsed_compute=6.61ms │ output_bytes=2.2 MB │ … └─ Stage 2 (Shuffle) tasks=2 partitions=32 output_rows=3.24 M elapsed_compute=21.42ms output_bytes=123.7 MB … ``` ### 3. Inspect a single stage `--stage <id>` shows the nodes of that stage with their metrics. The network boundaries point at the stages they read from, which you can open next. Stage 0 is the head of the plan, run by the coordinator: ```rust $ pbpaste | dfd plan --stage 3 Stage 3 (Coalesce) tasks=2 partitions=16 │ output_rows=11.62 K │ elapsed_compute=77.81ms │ output_bytes=363.1 KB │ … └─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority] │ output_rows=11.62 K │ elapsed_compute=38.59µs │ output_bytes=363.1 KB │ … └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true] │ output_rows=11.62 K │ elapsed_compute=72.29µs │ output_bytes=363.1 KB │ … └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)] │ output_rows=11.62 K │ elapsed_compute=2.97ms │ output_bytes=437.6 KB │ … └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5] │ output_rows=30.52 K │ elapsed_compute=17.58ms │ output_bytes=12.0 MB │ … ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1 │ output_rows=147.1 K │ elapsed_compute=1.53ms │ output_bytes=2.2 MB │ … └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2 output_rows=3.24 M elapsed_compute=55.63ms output_bytes=123.7 MB … ``` ### 4. Filter metrics `--metric <text>` leaves only the metrics whose name contains that text. It can be repeated, like `-m bytes -m rows`, and `--exact` matches whole names instead. Here's where the CPU time of a stage goes: ```rust $ pbpaste | dfd plan --stage 3 --metric elapsed_compute Stage 3 (Coalesce) tasks=2 partitions=16 │ elapsed_compute=77.81ms └─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority] │ elapsed_compute=38.59µs └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true] │ elapsed_compute=72.29µs └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)] │ elapsed_compute=2.97ms └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5] │ elapsed_compute=17.58ms ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1 │ elapsed_compute=1.53ms └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2 elapsed_compute=55.63ms ``` ### 5. Break metrics down per task Metrics are added up across the tasks of a stage. `--verbose` shows the value of each task instead, which tells apart a stage that is slow everywhere from one with a straggler: ```rust $ pbpaste | dfd plan --stage 3 --metric elapsed_compute --verbose Stage 3 (Coalesce) tasks=2 partitions=16 │ elapsed_compute={0:23.59ms, 1:54.22ms} └─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority] │ elapsed_compute={0:26.55µs, 1:12.04µs} └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true] │ elapsed_compute={0:35.42µs, 1:36.87µs} └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)] │ elapsed_compute={0:1.92ms, 1:1.04ms} └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5] │ elapsed_compute={0:8.54ms, 1:9.04ms} ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1 │ elapsed_compute={1:1.53ms} └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2 elapsed_compute={0:13.07ms, 1:42.56ms} ``` The output is colored in a terminal. `FORCE_COLOR=1 dfd plan --stage 2 < dump.txt | less -R` keeps the colors when paging through big stages.
…-contrib#891) Remote benchmarks can't be debugged with `dfd plan` yet. The remote benchmark worker only returns the ASCII plan, and plan dumps can only be generated in Rust. So a remote run has no dump to inspect, while a local `dfbench --debug` run does. The worker now also returns `plan_dump` in its query response, next to the existing ASCII `plan`. It's generated after the elapsed time is measured, so it doesn't affect reported timings. ``` GET /?sql=... ──▶ { "plan": "<ascii>", "plan_dump": "KLUv/...", "elapsed_ms": ..., ... } │ remote client (-d) prints it as one stdout line ──▶ dfd plan ``` The remote client that reads the dump is gabotechs/datafusion-distributed-dev-tools#75. That client treats the field as optional, so it keeps working against workers built from older revisions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ng (datafusion-contrib#894) Some custom `ExecutionPlan` implementations just aren't ready to run in a distributed context: they may depend on local state, assume they see the whole dataset, or not be serializable to workers. Today there's no way to tell the distributed planner "leave plans containing this node alone", so users either have to drop distribution for the whole session or risk wrong results. This adds `with_distributed_blacklisted_node::<T>()` (and its `set_` variant) to `DistributedExt`. Like the event handlers, the blacklist lives as a `SessionConfig` extension, so it can be set on a `SessionConfig`, `SessionStateBuilder`, `SessionState` or `SessionContext`. When the plan coming out of DataFusion's physical planning contains any blacklisted node, `create_distributed_plan` returns it untouched and the query runs on a single node. ``` SessionStateBuilder::new() .with_distributed_planner() .with_distributed_blacklisted_node::<MyLocalOnlyExec>() physical plan │ ▼ contains a blacklisted node type? │ │ no yes │ │ ▼ ▼ ┌─────────────────┐ ┌──────────────────────┐ │ DistributedExec │ │ original single-node │ │ ├ Stage 2 │ │ plan, unchanged │ │ └ Stage 1 │ └──────────────────────┘ └─────────────────┘ ``` The check sits in `create_distributed_plan` rather than only in the query planner, so users who call it directly get the same behavior.
…hes (datafusion-contrib#893) `DistributedExec` forwards the head stage's batches through a channel with room for one batch, but it returned a failure from its task (`msg?`). DataFusion's receiver stream reports task results separately from that channel and polls the two in turn, so the failure can reach the consumer before the batch still waiting in the channel. The query fails either way, but the consumer never sees that last batch. Send the failure through the channel too, behind the waiting batches, and stop forwarding after it. The regression test runs a head stage that yields one row and then fails, and reads the output only once the head stage has yielded the failure, so the row is already waiting in the channel. Its first read is pending, which makes the next read check the task's result first; without the fix, it receives the failure before the row. `MockExec` gains `with_error_notify`, which signals when its stream yields an error. Signed-off-by: Ahmed Farghal <me@asoli.dev>
…ame (datafusion-contrib#884) Labels are how a metric tells you *what* it measured, for example which task or which file a number belongs to. When workers aggregate metrics by name before sending them to the coordinator (the default, `aggregate_metrics_on_workers_before_collection`), we were using DataFusion's `aggregate_by_name`, which throws all labels away. Whatever consumes those metrics later, like the coordinator's plan rendering or anyone inspecting the metrics through the public API, never sees them. Some labels can't survive aggregation: if every partition carries a different `file`, there is no single value to keep. But a label that is identical across all instances of a metric is just as true after summing as before, so there's no reason to lose it. Now we keep exactly those labels: ``` worker, per partition sent to the coordinator output_rows{task=0, file=a.parquet} = 1 before: output_rows = 6 output_rows{task=0, file=b.parquet} = 2 ──► output_rows{task=0, file=c.parquet} = 3 after: output_rows{task=0} = 6 ▲ ▲ same everywhere differs → kept → dropped ``` A label that is missing from some instances is also dropped, since it isn't shared by all of them. Everything else about the aggregation behaves as it did before. --------- Co-authored-by: Jayant Shrivastava <jshrivastava03@gmail.com>
…berg scan nodes (datafusion-contrib#855) > [!NOTE] > This is part of a stack of PRs that implement dynamic filtering for Iceberg. The previous PR can be found here - datafusion-contrib#850 > Relevant changes will also be ported to datafusion-iceberg. ## Summary This PR adds support for accepting and storing expressions, including dynamic filters, in both Iceberg datasources (Eager and Lazy). Based on a config, `iceberg.dynamic_filtering_enabled`, operators like `HashJoinExec` are also informed that the datasource is a consumer of these dynamic filters. The dynamic filters also show up in `EXPLAIN ANALYZE` plans in the DataSource in the following format: ``` DataSourceExec: format=iceberg, table=default.taxi, projection=[trip_distance, pickup_date], predicate=trip_distance >= 2, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ] ``` These filters are **not** yet used for any optimizations - they're only stored and reported. Future PRs will use them to improve data pruning in the following order - * Convert the dynamic filter`PhysicalExpr` to `iceberg::Predicate` * Use predicates containing partition keys to skip processing `FileScanTask`s * Use predicates containing other columns to skip row groups as soon as the filters become available (dependency on #3351 in iceberg-rust) ## Changes * Both the `Lazy` and `Eager` datasources implement `try_pushdown_filters` - they extract and store the filters within the scan config, and report `PushedDown::No` to DF so that it continues apply its own filters * Adds a new iceberg-specific `dynamic_filtering_enabled` flag * Depending on the value of the flag, `apply_expressions` registers its use of dynamic filters, which in turn makes operators like `HashJoinExec` actually build them at runtime --------- Co-authored-by: Anirudh <6716458+Perseus@users.noreply.github.com>
…rib#897) Lazy Iceberg scans reported the snapshot's `total-records` as exact even when merge-on-read delete files remained. DataFusion could then answer `COUNT(*)` from the pre-delete count and skip reading the table. The regression test has nine data rows and two position deletes: it returned **9 before the fix and 7 after it**. Only report an exact snapshot row count when `total-delete-files` explicitly parses as zero. Other populated snapshots keep an inexact estimate, including summaries with a missing or invalid delete-file count. Tables confirmed to have no delete files retain the existing metadata-only count optimization. Extend the shared fixture builder to write and commit real position delete files, and exercise the normal SQL planning and execution path in the regression test. Validation (Rust 1.98.1): - Confirmed the regression fails against `d83f011` with only the test and fixture additions. - `cargo +1.98.1 test --locked -p datafusion-distributed-iceberg -p datafusion-distributed-iceberg-test-tools` - `cargo +1.98.1 clippy --locked -p datafusion-distributed -p datafusion-distributed-iceberg -p datafusion-distributed-iceberg-test-tools --all-targets --all-features -- -D warnings` - `cargo +1.98.1 fmt --all -- --check`
<!-- If this is your first time contributing to the project, please consider reviewing https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md In particular: All contributions are subject to the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). The DCO text is also included verbatim in the [dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt) file in the root directory of the repository. -->
Adds a translator from a DataFusion physical filter to an iceberg predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the filters hash joins build while the query runs: IN lists, struct(a, b) IN (...) for joins on several columns, and the CASE a partitioned join builds with one branch per partition. The predicate matches every row the filter keeps, and can match more, so the filter still runs on every row. An AND can drop a side it can't convert. A NOT is converted only when its inside was converted fully. Floats and times aren't converted, and neither are LIKE prefixes with non-ASCII text. iceberg-rust handles these in ways that could skip matching rows or fail the read. Nothing calls it yet. Partition skipping will use it next.
…ters The converter only depends on DataFusion and iceberg, so it belongs with the shared helpers. The partition skipper in common/ no longer imports from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and day partitions, with dynamic filtering on and off, and checks they return the same rows. The cases use keys on partition edges, keys in no partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only ASCII prefixes are converted.
…pper The skipper becomes DynamicFilterPruner, with one prune method that returns the task to read, or None to skip it. Later uses of the dynamic filters on each file can go in the same place. The code copied from iceberg-rust moves to partition_filter.rs, behind the same PartitionFilter API that iceberg-rust is adding. Once the crate's iceberg-rust version has it, that file can be deleted and the import changed.
DataFusion's dynamic filters use comparisons, IN lists, IS NULL, IS NOT NULL, AND, OR and CASE. This removes the conversions for NOT, NOT IN, LIKE, isnan, starts_with and bare boolean columns. Without NOT, nothing reads the `exact` flag, so the `Converted` wrapper goes too. `datum` no longer converts partition values from int to long or from float to double. Files written before a column was widened now hit the error arm, so we don't skip them.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 18, 2026 09:44
1b1af3d to
249fd05
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.