Skip to content

feat: add support for skipping partitions using dynamic filters - #21

Draft
Perseus wants to merge 31 commits into
anirudh.singh/iceberg-dyn-filtering-2from
anirudh.singh/iceberg-dyn-filtering-3
Draft

Perseus wants to merge 31 commits into
anirudh.singh/iceberg-dyn-filtering-2from
anirudh.singh/iceberg-dyn-filtering-3

Conversation

@Perseus

@Perseus Perseus commented Nov 9, 2026

Copy link
Copy Markdown
Owner

No description provided.

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 6bfdfaf to 52659af Compare November 9, 2026 21:11
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-2 branch from d927948 to cab7f00 Compare November 10, 2026 10:21
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 52659af to ef70cff Compare November 10, 2026 10:22
gabotechs and others added 4 commits November 10, 2026 17:24
…crate (datafusion-contrib#866)

Breaks the dependency cycle between `datafusion-distributed` and
`datafusion-distributed-benchmarks`.

Before this change, the core crate dev-depended on the benchmarks crate
(for TPC-H/TPC-DS/ClickBench data in tests), and the benchmarks crate
depended on the core crate. This implied that for every change in
`datafusion-distributed`, the `datafusion-distributed-benchmarks` needed
to be recompiled (as it's dependant) and then `datafusion-distributed`
itself needs to be re-recompiled again, causing compilation times to be
very slow.

Here are some numbers of adding a small trivial change to one test
before vs after:

## Before (32.58s)

```
time cargo test --lib test_select_all
   Compiling datafusion-distributed v4.0.0 (//datafusion-distributed/datafusion/distributed)
   Compiling datafusion-distributed-benchmarks v0.1.0 (//datafusion-distributed/datafusion/distributed-benchmarks)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 32.58s
```

## After (17.01s)

```
time cargo test --lib test_select_all
   Compiling datafusion-distributed v4.0.0 (//datafusion-distributed/datafusion/distributed)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 17.01s
```

So for testing targets, this cuts compilation time in half
…tafusion-contrib#868)

Compiles all integration tests of `datafusion-distributed` into one test
binary instead of one binary per file.

Before this change, each of the 32 files in `tests/` (plus
`dynamic_filtering/`) produced its own ~300 MB binary that linked the
whole dependency tree. After a library edit, cargo relinked every one of
them.

Running a single file's tests locally: `cargo test --features
integration --test integration udfs::`.

Rebuilding the library plus all test targets after a one-line library
edit went from 15-16s to 4-8s locally
…rib#872)

AQE stage preparation currently lives in a large async closure, which
makes subsequent startup changes harder to review.

Move that callback into `DynamicPlanBuilder`, implementing the existing
`NetworkBoundaryBuilder` trait. Task setup still joins all destinations
before sampling kickoff; coalesce behavior, task counts, dynamic
filters, statistics and metrics are unchanged. This is a preliminary
refactor for datafusion-contrib#852.

Validation: 343 library tests passed, one ignored; formatting passed.
The moved callback body was checked for equivalence after accounting for
borrowed fields and the removed closure wrappers.

Co-authored-by: Codex GPT-6 <noreply@localhost>
Task setup skew can postpone useful work on a ready worker. Add a
Criterion benchmark for that scheduling case before the execution
changes in datafusion-contrib#852. This PR changes no library behavior and adds no
dependencies.

The benchmark runs one CPU-heavy task and one light task over localhost
gRPC, in both static and AQE modes. Each mode has three cases: no
routing delay, a 250 ms routing delay on the light task, and the same
delay on the heavy task. It measures execution through complete output
collection, excluding worker startup and planning. Every iteration
checks output `[0, 1]` and exactly one heavy remote execution. An
untimed preflight reports whether CPU work starts and finishes before
delayed sibling routing returns.

Applied unchanged to datafusion-contrib#852, the benchmark demonstrates the conditional
benefit:

| Mode | Case | Baseline | With datafusion-contrib#852 | Paired change |
|---|---|---:|---:|---:|
| Static | No delay | 228.15 ms | 227.84 ms | -0.14% |
| Static | Delayed light task | 479.29 ms | 294.10 ms | **-38.64%** |
| Static | Delayed heavy task | 479.09 ms | 478.60 ms | -0.10% |
| AQE | No delay | 228.23 ms | 227.86 ms | -0.16% |
| AQE | Delayed light task | 479.37 ms | 294.36 ms | **-38.60%** |
| AQE | Delayed heavy task | 479.52 ms | 478.79 ms | -0.15% |

In both delayed-light preflights, heavy work started and finished before
sibling routing returned with datafusion-contrib#852; neither witness occurred on the
baseline. The controls stayed essentially flat. This demonstrates value
when ready useful work can overlap a sibling's setup delay; it does not
predict representative S3 or multi-host speedups.

Validation: optimized builds and all six cases passed on baseline and
candidate. Formatting and Criterion-target Clippy with warnings denied
passed.

<details>
<summary>Measurement scope and reproduction</summary>

Measurements used baseline `0291e08` (main `e5d6ea3` plus this
benchmark) and candidate source matching datafusion-contrib#852 at `354d386`. The
benchmark and lockfile were identical. The candidate binary was built
before the final commit; source hashes were verified against that
commit.

This was a controlled synthetic experiment on a shared host, with the
same four-CPU affinity for both binaries, four Tokio threads, ten
Criterion samples per case, one-second warmup and a five-second target
measurement window. Three outer pairs were attempted in
baseline/candidate, candidate/baseline, baseline/candidate order. Two
opposite-order pairs passed the predefined resource gate. The entire
third pair was excluded because baseline selected-CPU average
utilization p95 was 85.31%, exceeding the 85% threshold; the
maximum-utilization threshold was 98%. Raw results for both sides of the
excluded pair were retained. Results were not filtered by measured
latency.

Times above are medians of the accepted per-run Criterion medians;
changes are medians of the paired percentage changes. Two outer pairs
illustrate this scheduling case, rather than establishing production
performance or precise population estimates.

```shell
cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --save-baseline main
# Run the same benchmark on the candidate revision:
cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --baseline main
```

Use separate build directories or rebuild both local packages when
switching revisions. Confirm the candidate's delayed-light preflight
shows early work and the baseline's does not before interpreting
timings.

</details>

---------

Co-authored-by: Codex GPT-6 <noreply@localhost>
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 77858c6 to c59f43e Compare November 11, 2026 14:35
stuhood and others added 22 commits November 12, 2026 08:01
datafusion-contrib#868 added unquoted `run:` commands ending with `::` in
`.github/workflows/ci.yml`. Because YAML treats a trailing `:` before a
newline as a mapping value indicator, `.github/workflows/ci.yml` has
been failing workflow parsing before any jobs start.

Quote those `run:` commands so `.github/workflows/ci.yml` parses
cleanly.
…-contrib#876)

Distributed window queries can fail with `Expects PARTITION BY
expression to be ordered` when a partition key is a projected literal.
Planning correctly omits the constant key from the sort, but
`DistributedCodec` drops its constant metadata when reconstructing the
worker plan.

Preserve `AcrossPartitions` metadata in the existing equivalence-class
protobuf message for shuffle, coalesce, and broadcast boundaries. The
additive field distinguishes per-partition constants, uniform constants
with unknown values, and known values (including null).

Reproducer using the repository's `weather` fixture and two localhost
workers:

```sql
SELECT typ, "RainToday", "WindGustDir"
FROM (
    SELECT typ, "RainToday", "WindGustDir",
           row_number() OVER (PARTITION BY typ, "RainToday", "WindGustDir") AS rn
    FROM (SELECT 'weather' AS typ, * FROM weather)
    WHERE "RainToday" = 'No' AND "WindGustDir" IN ('N', 'S')
)
WHERE rn = 1
ORDER BY "WindGustDir";
```

The integration regression fails with the original error on `main` and
returns the two expected rows with this patch. The codec round-trip test
covers all constant variants across all three boundary types.

Validation:
- `cargo test -p datafusion-distributed --features integration`
- `cargo clippy -p datafusion-distributed --all-targets --all-features
-- -D warnings`
- `cargo fmt --all -- --check`
<!--
If this is your first time contributing to the project, 
please consider reviewing
https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md
In particular:

All contributions are subject to the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/). The DCO
text is also included verbatim in the
[dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt)
file in the root directory of the repository.

-->

Just a small fix to the insta settings
…contrib#890)

Every time a new config option lands in DataFusion or in
`DistributedConfig`, benchmarking it means first adding a dedicated flag
to `RunOpt` and wiring it into the session builder. That's easy to
forget, it keeps `RunOpt` growing, and any option without a flag simply
can't be benchmarked.

This makes `dfbench run` take config options generically instead.
They're plain `key=value` pairs, and any `datafusion.*` or
`distributed.*` key works, in either form:

```yaml
# .df-bench.yaml (local, gitignored)
distributed.broadcast_joins: true
datafusion.execution.target_partitions: 10
```

```shell
dfbench run --dataset $DATASET -c distributed.broadcast_joins=true -c datafusion.execution.target_partitions=10
```

The file defaults to `.df-bench.yaml` in the current directory, and
`--config-path` points to a different one. Each entry goes through
DataFusion's own config setter, in this order, so later sources win:

```
 DATAFUSION_* env vars ──▶ .df-bench.yaml (or --config-path) ──▶ -c key=value ...
     (lowest)                    (file order)                       (highest)
                                     │
                                     ▼
                  ConfigOptions::set(key, value)   unknown keys fail the run
```

Options that aren't DataFusion configs, such as `--workers`, `-i`,
`--debug` and `--work-unit-file-scan`, stay as flags. All the flags that
only set a config option are gone, for example `--partitions`,
`--batch-size`, `--compression` and `--dynamic`. Their replacement is
the matching config key.

Metrics collection is on by default, since `--debug` and dynamic task
count read the collected metrics. Use `distributed.collect_metrics:
false` to turn it off.
…datafusion-contrib#841)

> [!WARNING]
> This PR is big, and it's exclusively dev tooling with no implications
in the production code. A very light review is recommended, as the idea
is that LLMs are the main ones maintaining this tool.

<!--
If this is your first time contributing to the project, 
please consider reviewing
https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md
In particular:

All contributions are subject to the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/). The DCO
text is also included verbatim in the
[dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt)
file in the root directory of the repository.

-->

Closes
datafusion-contrib#523

Ships a tool + a SKILL.md for LLMs (and humans) to debug execution plans
and identify bottlenecks and weak spots.

There's two main pieces this PR ships:
- A plan dump generator function, `generate_plan_dump(plan)`, that
generates a compressed logeable b64 string for people to add to their
own observability layer.
- A CLI for inspecting that plan (+ a SKILL.md to use it)

## Dump size

Dumps are meant to live in logs, so they need to be small in size and
CPU footprint. Here's a table that compares logging the ASCII plan (even
if it was zstd-compressed and base64-encoded) with the plan dumps
proposed in this PR:

| Query | ASCII plan: bytes (CPU time) | ASCII + zstd + base64: bytes
(CPU time) | Plan dump (proto + zstd + base64): bytes (CPU time) |
|---|---:|---:|---:|
| q1 | 40,614 (846 µs) | 6,716 (949 µs) | 3,640 (485 µs) |
| q2 | 227,396 (3,544 µs) | 26,040 (4,184 µs) | 16,112 (1,701 µs) |
| q3 | 101,454 (1,581 µs) | 13,436 (1,853 µs) | 6,948 (890 µs) |
| q4 | 74,524 (1,300 µs) | 12,236 (1,590 µs) | 6,884 (712 µs) |
| q5 | 158,965 (2,515 µs) | 18,984 (2,976 µs) | 10,036 (1,202 µs) |
| q6 | 33,105 (573 µs) | 4,500 (656 µs) | 1,592 (419 µs) |
| q7 | 175,123 (2,878 µs) | 22,648 (3,539 µs) | 13,024 (1,463 µs) |
| q8 | 211,001 (3,358 µs) | 23,412 (3,843 µs) | 12,744 (1,533 µs) |
| q9 | 175,677 (2,906 µs) | 23,828 (4,039 µs) | 14,008 (1,688 µs) |
| q10 | 121,009 (1,948 µs) | 16,032 (2,254 µs) | 8,528 (974 µs) |
| q11 | 154,003 (2,491 µs) | 18,296 (2,795 µs) | 10,524 (1,106 µs) |
| q12 | 72,108 (1,150 µs) | 9,644 (1,344 µs) | 4,968 (665 µs) |
| q13 | 79,315 (1,473 µs) | 13,884 (1,737 µs) | 8,096 (716 µs) |
| q14 | 59,553 (844 µs) | 7,856 (992 µs) | 3,452 (507 µs) |
| q16 | 105,161 (1,892 µs) | 15,836 (2,283 µs) | 9,008 (1,040 µs) |
| q17 | 97,139 (1,454 µs) | 12,872 (1,717 µs) | 7,176 (762 µs) |
| q18 | 129,084 (2,060 µs) | 17,200 (2,315 µs) | 9,468 (954 µs) |
| q19 | 93,076 (1,455 µs) | 8,608 (1,577 µs) | 4,404 (1,111 µs) |
| q20 | 154,741 (2,621 µs) | 22,196 (3,075 µs) | 13,080 (1,320 µs) |
| q21 | 183,115 (3,058 µs) | 23,992 (3,564 µs) | 14,296 (1,548 µs) |
| q22 | 94,338 (2,954 µs) | 12,200 (3,344 µs) | 6,576 (2,187 µs) |
| **Total** | **2,540,501 (42,901 µs)** | **330,416 (50,626 µs)** |
**184,564 (22,983 µs)** |

## Usage

### 1. Log a dump from your service

Once the query has been fully executed, fold the worker metrics back
into the plan and log its dump next to the rest of the query's
telemetry:

```rust
let plan = ctx.sql(sql).await?.create_physical_plan().await?;
let batches = execute_stream(plan.clone(), ctx.task_ctx())?.try_collect::<Vec<_>>().await?;

let plan = rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::PerTask).await?;
tracing::info!(plan_dump = %generate_plan_dump(&plan)?, "query finished");
```

A dump is a single base64 line. Here's a complete one for TPC-H q3 (SF1,
4 workers), 2.9 KB, which you can paste into `dfd plan` to try out the
commands below:

```
KLUv/WIAgIcqbUQAY33rHsVztSify0X5XBYV/K4/UC9xHCPTIs6RVSPAHH9P2C7ZNpzK6iri0FonOuvDyIyx6jG0s2FyioWt9KxGx2pI27J9sYJP7el9YhoAN6yp9XwD4OMp25IMew7Lr29yEVsJvCo0JfAyABkANywAhB/3D39Z6Aj6Sjc2z5XUEB2m2g7aDtprmJlNp8fV5NHJWKJD331e9hFenqK10j8O4zcAblg0aWEaIUiQ/pS3SlTlu+sDqUeFT1+xrxCEFhYX1+3uweUmvrUCG6l6WEv2XhoEFEZKkxZG5diW+GC/3xvbFjo7IpWst73saNEV6U4Fjy79ZMlztysl19jNHa8PogR7zDp19SN8wkEN5OUAGqZuQavsnDLFZU4AFV4MbKUzPEQqPhutVGQPxbR7nzL0WlgECZ2jqCmwL4i7aoDwbOJhhf7Kn9i8VLwm8m9SmwuxbWkZ+XtR+bt+Qf326l5Zd7iiqkNTiXm4aYmEUWBMFddSNeV/CDRuJhwHuDyYQ3NANY1/lgVEMDxSyppFTuM/PX26B3NolbCe6riDrUYlGJYSLi0hrhmBOggpMSpr10/W/Zla5l155XVr4FldZbiRzbePOwPeH7ZZH59PELbipneIbTU/Q7M9WI3UjT1+NXN3IqEgbMV9pKmxbjVS+0m2fMPWQ9gKggYyMOAEGDFgEMAKIrxgcc3w02NYHLhqTOl+kk17tzR2oLGpq4T1cP23Q0pySiWspwOzctFHktTI9mVvPMl2V0fLSj5w9fQRJCsOWOfqaJkFO1cEmdEux4vswIWrAsID15M+/SQJTG0jvYEl4Cx7WDtZIE7MQJGY6mVGIM3R1j3u1mPr/q5b27pfIA5s6x7fzbbuve9Ad9q6B6HpbOvew5XY1r2IMp2ER0Ia8l7wBMYgeQWxrXbQLbbZUXRhVRIeCV/wBN43libZ3RMkV00qex9w09q7E/8uz1691ahuNyOtloY8GvJikIxBsmri8nVNAFL3QJ/e6qvENqNQnWE220+IGWpV3C0LvKkYxgRLIOHNBGZaHy85lOhIbEFgYVlpgaHCCSykcMH/uTrQo39w9x+dfdEVgiG9RGIJHQuwZmisFFirpaZVjVyqbNS4FrGqEQsWrAPuDgHI74F1AEJLCvk6QZV+MrXTPUGaLN8/lmc76JN86IX+VOVwlYKoE7/+q841QmWnwqJjLtdX8nwgPbacyKnFBYMhvYNGkMEVjV4QYmABhgwwXsigSP9eilT8onT2XHwitrdX2QaB0aDhJXJKSpQMB9EEBRmCEGNYnQ9OoHCxS+BEFmYdJTW4TUa5jukYygXXc0GS52j5j+4EY1tzq2VZQGplv3LbjIh0YCNUW/1qBlUEUAiAD9MHELbdGrgRHWEFH3smd7w7BVnLr1ZPQO8917vzaNT95vDtlazZTg1QTTYNHk1PkzvA1q8+tFu8Bk0wdDHRXC2hZ2W64qxqqBgkUJAdlwVuJJ2NQlRLJCtbXKhnksfSMfuj/SKv9mwGUHZCTxCEE91qWJ1tU+9N1VLCSgTIHhE5mFZ/lCO/yl8nCo3t0Cd3DPK4++gyHVmcQyFVCicDmtc51MZ2p1GwmgMQmEYEDtKm3qvkakGscgPfromaPBtYPwdk9lwW7QESrg8WIOcNsPz/0adeF+yaaaNdewUNhWDMHO59zvkG+zFK3x2BA+W5AGDV1ozn9T7sxXpmN1buzIL0+K2keR4OPu0QGvU4g/GIkNL10Bx5KNE+nZPRLVqCSss7LsqD561xjPF0vXJyLvtk7Dl1DRpHs5nzNA4CDgfc4482Mvw2o9rYjnPmmorQyrPFjvOcgzhYK6D4+uw4ZQfkp3U/ymd6SCNbbAqS52IXXUiKGhjTaWKHEuWfoewJhU88EdbhEVySuC0vADs0BTOyGU+hRsQB2VJeAiSk7Asr7kaBsPyFhP8hUDX6FpdsKC3C8aDAGD43oAjpOvvtYHjH/xwdnX2BHJ/0dB0b0iFiVUe0nkL3yHnIJAZuSjNfcpokRJ0y+B45XExgyiJf2c0wc9KoPPlADo2Ff2aAj9XPVXRsI8bVj2nglO04bECGGOnhuCzY0h0mUnvDN700tnUmvq1i272wSXNBnucZt9gQAopL7rhm4pINYCLZpdJOgzrXzPhqnPD0gFfcBtnbYRdY0SCJ+3wAGxue3RsTSxd5cXBpAAzOsC7P3NaUIF2DVN5yUyfhXvMO84AqB/7yqGTBa5zKkrmLdQZWY9OJWiablxq9BQorh5Dsbs2EwDlrn2wtZ+73U8M8Y5OcQl0hYU+M8tkbk7S5WZZ1DVQVdITAePji6t0/5SLkuGX8AzzYZv8ppErctZ1JJqKaeOJbroe46DFQVm6F7iVjYBMlURSFoiSKJJHc0ANT+hwuKCvlnoxoBMBYiNEJJ7T0FW7rHPOocX1ctbYtIDMbWpR/snHweZGAEQVKngerTvnXTuA+6rvk+iQZKT31rZsiQAeFywKQEroJENyuyjkJTqIcpHB1QWVBDoSrswSvojkADpK8T1MGv5xUB/JXkp5tJRdxQfNdziwEscQdeZ5nH+7lnlF8Qz275YWpi8ScsJcL8fd4K+ayWmuDsaZ4aBfHHRKd6fB6ORxKSwYu414o49GKQScI/zoqKmwo1BuFhXpsd7IkR3fobCY5qGgrdgsd50/w2Igpj4jV3ZDFPHFiKfmofC4VvAR887cc8RENRbB+JqI1G8KhKFFaBxMjxhrxD2+Rjw9ruGapfTtnqJZWKwjP3ERZvYA7q018i6wTIQTWUYgixmKeDIlZzdbA15F4xq2W7ACSx3EAzxG3PgmSIOdSm2RaLH3jK1vsDGMt3EwIRNDMkuGujdOCke+a5HiGk7FpCA85cpqSWDCEByFdefFTolh97Er1DsoVtULrT1qnhXTkWGQlBLOF2SZrTPPnEQM=
```

You can try this out yourself by checking out the branch of this PR
locally and installing the `dfd` CLI with:

```
cargo install --path datafusion/distributed-cli/
pbpaste | dfd plan
```

### 2. Get an overview of the stages

Copy the dump from your logs and pass it to `dfd plan` as an argument,
as a file path or through stdin. With no other arguments, it shows the
tree of stages with their stage-level metrics, added up across tasks.
Some metrics are elided below:

```rust
$ pbpaste | dfd plan
DistributedExec
│ output_rows=11.62 K
│ elapsed_compute=5.43ms
│ output_bytes=363.1 KB
│ …
└─ Stage 3 (Coalesce) tasks=2 partitions=16
   │ output_rows=11.62 K
   │ elapsed_compute=77.81ms
   │ output_bytes=363.1 KB
   │ …
   ├─ Stage 1 (Shuffle) tasks=1 partitions=32
   │    output_rows=147.1 K
   │    elapsed_compute=6.61ms
   │    output_bytes=2.2 MB
   │    …
   └─ Stage 2 (Shuffle) tasks=2 partitions=32
        output_rows=3.24 M
        elapsed_compute=21.42ms
        output_bytes=123.7 MB
        …
```

### 3. Inspect a single stage

`--stage <id>` shows the nodes of that stage with their metrics. The
network boundaries point at the stages they read from, which you can
open next. Stage 0 is the head of the plan, run by the coordinator:

```rust
$ pbpaste | dfd plan --stage 3
Stage 3 (Coalesce) tasks=2 partitions=16
│ output_rows=11.62 K
│ elapsed_compute=77.81ms
│ output_bytes=363.1 KB
│ …
└─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority]
   │ output_rows=11.62 K
   │ elapsed_compute=38.59µs
   │ output_bytes=363.1 KB
   │ …
   └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true]
      │ output_rows=11.62 K
      │ elapsed_compute=72.29µs
      │ output_bytes=363.1 KB
      │ …
      └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)]
         │ output_rows=11.62 K
         │ elapsed_compute=2.97ms
         │ output_bytes=437.6 KB
         │ …
         └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5]
            │ output_rows=30.52 K
            │ elapsed_compute=17.58ms
            │ output_bytes=12.0 MB
            │ …
            ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1
            │    output_rows=147.1 K
            │    elapsed_compute=1.53ms
            │    output_bytes=2.2 MB
            │    …
            └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2
                 output_rows=3.24 M
                 elapsed_compute=55.63ms
                 output_bytes=123.7 MB
                 …
```

### 4. Filter metrics

`--metric <text>` leaves only the metrics whose name contains that text.
It can be repeated, like `-m bytes -m rows`, and `--exact` matches whole
names instead. Here's where the CPU time of a stage goes:

```rust
$ pbpaste | dfd plan --stage 3 --metric elapsed_compute
Stage 3 (Coalesce) tasks=2 partitions=16
│ elapsed_compute=77.81ms
└─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority]
   │ elapsed_compute=38.59µs
   └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true]
      │ elapsed_compute=72.29µs
      └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)]
         │ elapsed_compute=2.97ms
         └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5]
            │ elapsed_compute=17.58ms
            ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1
            │    elapsed_compute=1.53ms
            └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2
                 elapsed_compute=55.63ms
```

### 5. Break metrics down per task

Metrics are added up across the tasks of a stage. `--verbose` shows the
value of each task instead, which tells apart a stage that is slow
everywhere from one with a straggler:

```rust
$ pbpaste | dfd plan --stage 3 --metric elapsed_compute --verbose
Stage 3 (Coalesce) tasks=2 partitions=16
│ elapsed_compute={0:23.59ms, 1:54.22ms}
└─ ProjectionExec: expr=[l_orderkey@0 as l_orderkey, sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 as revenue, o_orderdate@1 as o_orderdate, o_shippriority@2 as o_shippriority]
   │ elapsed_compute={0:26.55µs, 1:12.04µs}
   └─ SortExec: expr=[sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)@3 DESC, o_orderdate@1 ASC NULLS LAST], preserve_partitioning=[true]
      │ elapsed_compute={0:35.42µs, 1:36.87µs}
      └─ AggregateExec: mode=SinglePartitioned, gby=[l_orderkey@2 as l_orderkey, o_orderdate@0 as o_orderdate, o_shippriority@1 as o_shippriority], aggr=[sum(lineitem.l_extendedprice * 1 - lineitem.l_discount) as sum(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)]
         │ elapsed_compute={0:1.92ms, 1:1.04ms}
         └─ HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, o_shippriority@2, l_orderkey@3, l_extendedprice@4, l_discount@5]
            │ elapsed_compute={0:8.54ms, 1:9.04ms}
            ├─ [Stage 1] => NetworkShuffleExec: output_partitions=16, input_tasks=1
            │    elapsed_compute={1:1.53ms}
            └─ [Stage 2] => NetworkShuffleExec: output_partitions=16, input_tasks=2
                 elapsed_compute={0:13.07ms, 1:42.56ms}
```

The output is colored in a terminal. `FORCE_COLOR=1 dfd plan --stage 2 <
dump.txt | less -R` keeps the colors when paging through big stages.
…-contrib#891)

Remote benchmarks can't be debugged with `dfd plan` yet. The remote
benchmark worker only returns the ASCII plan, and plan dumps can only be
generated in Rust. So a remote run has no dump to inspect, while a local
`dfbench --debug` run does.

The worker now also returns `plan_dump` in its query response, next to
the existing ASCII `plan`. It's generated after the elapsed time is
measured, so it doesn't affect reported timings.

```
GET /?sql=...  ──▶  { "plan": "<ascii>", "plan_dump": "KLUv/...", "elapsed_ms": ..., ... }
                                              │
remote client (-d) prints it as one stdout line ──▶ dfd plan
```

The remote client that reads the dump is
gabotechs/datafusion-distributed-dev-tools#75. That client treats the
field as optional, so it keeps working against workers built from older
revisions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ng (datafusion-contrib#894)

Some custom `ExecutionPlan` implementations just aren't ready to run in
a distributed context: they may depend on local state, assume they see
the whole dataset, or not be serializable to workers. Today there's no
way to tell the distributed planner "leave plans containing this node
alone", so users either have to drop distribution for the whole session
or risk wrong results.

This adds `with_distributed_blacklisted_node::<T>()` (and its `set_`
variant) to `DistributedExt`. Like the event handlers, the blacklist
lives as a `SessionConfig` extension, so it can be set on a
`SessionConfig`, `SessionStateBuilder`, `SessionState` or
`SessionContext`. When the plan coming out of DataFusion's physical
planning contains any blacklisted node, `create_distributed_plan`
returns it untouched and the query runs on a single node.

```
SessionStateBuilder::new()
    .with_distributed_planner()
    .with_distributed_blacklisted_node::<MyLocalOnlyExec>()

            physical plan
                  │
                  ▼
   contains a blacklisted node type?
          │                 │
         no                yes
          │                 │
          ▼                 ▼
  ┌─────────────────┐  ┌──────────────────────┐
  │ DistributedExec │  │ original single-node │
  │   ├ Stage 2     │  │ plan, unchanged      │
  │   └ Stage 1     │  └──────────────────────┘
  └─────────────────┘
```

The check sits in `create_distributed_plan` rather than only in the
query planner, so users who call it directly get the same behavior.
…hes (datafusion-contrib#893)

`DistributedExec` forwards the head stage's batches through a channel
with room for one batch, but it returned a failure from its task
(`msg?`). DataFusion's receiver stream reports task results separately
from that channel and polls the two in turn, so the failure can reach
the consumer before the batch still waiting in the channel. The query
fails either way, but the consumer never sees that last batch.

Send the failure through the channel too, behind the waiting batches,
and stop forwarding after it.

The regression test runs a head stage that yields one row and then
fails, and reads the output only once the head stage has yielded the
failure, so the row is already waiting in the channel. Its first read is
pending, which makes the next read check the task's result first;
without the fix, it receives the failure before the row. `MockExec`
gains `with_error_notify`, which signals when its stream yields an
error.

Signed-off-by: Ahmed Farghal <me@asoli.dev>
…ame (datafusion-contrib#884)

Labels are how a metric tells you *what* it measured, for example which
task or which file a number belongs to. When workers aggregate metrics
by name before sending them to the coordinator (the default,
`aggregate_metrics_on_workers_before_collection`), we were using
DataFusion's `aggregate_by_name`, which throws all labels away. Whatever
consumes those metrics later, like the coordinator's plan rendering or
anyone inspecting the metrics through the public API, never sees them.

Some labels can't survive aggregation: if every partition carries a
different `file`, there is no single value to keep. But a label that is
identical across all instances of a metric is just as true after summing
as before, so there's no reason to lose it. Now we keep exactly those
labels:

```
 worker, per partition                        sent to the coordinator

 output_rows{task=0, file=a.parquet} = 1      before:  output_rows = 6
 output_rows{task=0, file=b.parquet} = 2  ──►
 output_rows{task=0, file=c.parquet} = 3      after:   output_rows{task=0} = 6
               ▲            ▲
       same everywhere   differs
           → kept        → dropped
```

A label that is missing from some instances is also dropped, since it
isn't shared by all of them. Everything else about the aggregation
behaves as it did before.

---------

Co-authored-by: Jayant Shrivastava <jshrivastava03@gmail.com>
…berg scan nodes (datafusion-contrib#855)

> [!NOTE]
> This is part of a stack of PRs that implement dynamic filtering for
Iceberg. The previous PR can be found here -
datafusion-contrib#850
> Relevant changes will also be ported to datafusion-iceberg. 

## Summary

This PR adds support for accepting and storing expressions, including
dynamic filters, in both Iceberg datasources (Eager and Lazy).

Based on a config, `iceberg.dynamic_filtering_enabled`, operators like
`HashJoinExec` are also informed that the datasource is a consumer of
these dynamic filters.

The dynamic filters also show up in `EXPLAIN ANALYZE` plans in the
DataSource in the following format:
```
DataSourceExec: format=iceberg, table=default.taxi, projection=[trip_distance, pickup_date], predicate=trip_distance >= 2, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ]
```

These filters are **not** yet used for any optimizations - they're only
stored and reported.

Future PRs will use them to improve data pruning in the following order
-

* Convert the dynamic filter`PhysicalExpr` to `iceberg::Predicate`
* Use predicates containing partition keys to skip processing
`FileScanTask`s
* Use predicates containing other columns to skip row groups as soon as
the filters become available (dependency on #3351 in iceberg-rust)


## Changes

* Both the `Lazy` and `Eager` datasources implement
`try_pushdown_filters` - they extract and store the filters within the
scan config, and report `PushedDown::No` to DF so that it continues
apply its own filters
* Adds a new iceberg-specific `dynamic_filtering_enabled` flag 
* Depending on the value of the flag, `apply_expressions` registers its
use of dynamic filters, which in turn makes operators like
`HashJoinExec` actually build them at runtime

---------

Co-authored-by: Anirudh <6716458+Perseus@users.noreply.github.com>
…rib#897)

Lazy Iceberg scans reported the snapshot's `total-records` as exact even
when merge-on-read delete files remained. DataFusion could then answer
`COUNT(*)` from the pre-delete count and skip reading the table. The
regression test has nine data rows and two position deletes: it returned
**9 before the fix and 7 after it**.

Only report an exact snapshot row count when `total-delete-files`
explicitly parses as zero. Other populated snapshots keep an inexact
estimate, including summaries with a missing or invalid delete-file
count. Tables confirmed to have no delete files retain the existing
metadata-only count optimization.

Extend the shared fixture builder to write and commit real position
delete files, and exercise the normal SQL planning and execution path in
the regression test.

Validation (Rust 1.98.1):
- Confirmed the regression fails against `d83f011` with only the test
and fixture additions.
- `cargo +1.98.1 test --locked -p datafusion-distributed-iceberg -p
datafusion-distributed-iceberg-test-tools`
- `cargo +1.98.1 clippy --locked -p datafusion-distributed -p
datafusion-distributed-iceberg -p
datafusion-distributed-iceberg-test-tools --all-targets --all-features
-- -D warnings`
- `cargo +1.98.1 fmt --all -- --check`
<!--
If this is your first time contributing to the project, 
please consider reviewing
https://github.com/datafusion-contrib/datafusion-distributed/blob/main/CONTRIBUTING.md
In particular:

All contributions are subject to the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/). The DCO
text is also included verbatim in the
[dco.txt](https://github.com/datafusion-contrib/datafusion-distributed/blob/main/dco.txt)
file in the root directory of the repository.

-->
Adds a translator from a DataFusion physical filter to an iceberg
predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE
prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the
filters hash joins build while the query runs: IN lists, struct(a, b) IN
(...) for joins on several columns, and the CASE a partitioned join builds
with one branch per partition.

The predicate matches every row the filter keeps, and can match more, so
the filter still runs on every row. An AND can drop a side it can't
convert. A NOT is converted only when its inside was converted fully.

Floats and times aren't converted, and neither are LIKE prefixes with
non-ASCII text. iceberg-rust handles these in ways that could skip
matching rows or fail the read.

Nothing calls it yet. Partition skipping will use it next.
…ters

The converter only depends on DataFusion and iceberg, so it belongs with
the shared helpers. The partition skipper in common/ no longer imports
from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows
how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and
day partitions, with dynamic filtering on and off, and checks they
return the same rows. The cases use keys on partition edges, keys in no
partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only
ASCII prefixes are converted.
…pper

The skipper becomes DynamicFilterPruner, with one prune method that
returns the task to read, or None to skip it. Later uses of the dynamic
filters on each file can go in the same place.

The code copied from iceberg-rust moves to partition_filter.rs, behind
the same PartitionFilter API that iceberg-rust is adding. Once the
crate's iceberg-rust version has it, that file can be deleted and the
import changed.
DataFusion's dynamic filters use comparisons, IN lists, IS NULL,
IS NOT NULL, AND, OR and CASE. This removes the conversions for NOT,
NOT IN, LIKE, isnan, starts_with and bare boolean columns. Without
NOT, nothing reads the `exact` flag, so the `Converted` wrapper goes
too.

`datum` no longer converts partition values from int to long or from
float to double. Files written before a column was widened now hit
the error arm, so we don't skip them.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 1b1af3d to 249fd05 Compare November 18, 2026 09:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants