Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
47e969a
refactor: move benchmark dataset generation into datafusion-datasets …
gabotechs Nov 10, 2026
fa4a1bb
refactor: compile all integration tests into a single test binary (#868)
gabotechs Nov 10, 2026
33a15eb
refactor: name the adaptive network boundary builder (#872)
joybestourous Nov 10, 2026
cb6c9c4
perf: benchmark task startup overlap (#873)
joybestourous Nov 10, 2026
de765e8
quote CI run commands ending with trailing :: (#878)
stuhood Nov 12, 2026
a976e71
fix: preserve constant metadata across network boundaries (#876)
alexanderbianchi Nov 12, 2026
7fed65b
Fix insta workspace (#885)
gabotechs Nov 14, 2026
c43b42d
feat: configure dfbench through a YAML file and -c flags (#890)
gabotechs Nov 15, 2026
9e37cab
feat: add plan dumps that can be logged and displayed with `dfd plan`…
gabotechs Nov 16, 2026
c2d1986
feat: return a plan dump from the remote benchmark worker (#891)
gabotechs Nov 16, 2026
8d03b50
feat: allow blacklisting execution plan nodes from distributed planni…
gabotechs Nov 16, 2026
9106556
fix: deliver DistributedExec's terminal error after its buffered batc…
AhmedSoliman Nov 17, 2026
d83f011
feat: preserve consistent labels when aggregating worker metrics by n…
gabotechs Nov 17, 2026
82b547b
feat: accept and store expressions (including dynamic filters) in Ice…
Perseus Nov 18, 2026
e3f4618
fix: apply Iceberg deletes before answering COUNT(*) (#897)
alexanderbianchi Nov 18, 2026
676f79b
Follow downcast delegate in blacklist (#899)
gabotechs Nov 18, 2026
13220b9
feat: turn DataFusion filters into Iceberg predicates
Perseus Nov 6, 2026
cb5531b
feat: add support for skipping partitions using dynamic filters
Perseus Nov 9, 2026
9d08b25
nit: reword a comment the spell check flags
Perseus Nov 10, 2026
9159ab9
nit: remove the unused skip stub, do less work per file, fix comments
Perseus Nov 10, 2026
07e0d1b
test: show join filters skipping partitions on workers
Perseus Nov 10, 2026
10037c6
test: cover top-N, month partitions and null partitions in partition …
Perseus Nov 10, 2026
2311943
refactor: move the physical filter converter under common/dynamic_fil…
Perseus Nov 11, 2026
4bfadd4
feat: count the files that partition skipping keeps and skips
Perseus Nov 11, 2026
42a0ed7
test: compare query results with partition skipping on and off
Perseus Nov 11, 2026
ae2d92e
docs: rewrite the comments in the physical filter converter
Perseus Nov 11, 2026
e598082
refactor: split the copied iceberg-rust code out of the partition ski…
Perseus Nov 14, 2026
53f2914
nit: doc cleanup + small refactor/simplification
Perseus Nov 15, 2026
fc6451a
nit: fix a typo the spell check flags
Perseus Nov 15, 2026
dbca505
fix: simplify partition_filter's docs and comments
Perseus Nov 16, 2026
249fd05
refactor: convert only what dynamic filters use
Perseus Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions .agents/skills/benchmark/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,15 @@ DATASET=$(target/release/dfbench prepare-tpch --scale-factor 10) # only the pa
target/release/dfbench run --dataset $DATASET --workers 8 -i 5
```

`dfbench run --help` lists all options.
`dfbench run --help` lists all options. Config options (`datafusion.*`, `distributed.*`) come from
a local, gitignored `.df-bench.yaml` in the current directory (or `--config-path <file>`),
overridden by repeated `-c key=value` flags, e.g. `-c distributed.dynamic_task_count=true`.

## Plan dumps

`--debug` prints a plan dump per iteration to stdout (everything else goes to stderr), which
`dfd plan` (`-p datafusion-distributed-cli`) inspects; see the `plan-dump` skill. `dfbench` reads
`DATAFUSION_*` env vars, e.g. `DATAFUSION_OPTIMIZER_JOIN_REORDERING=false`.

## Iceberg

Expand Down Expand Up @@ -48,8 +56,16 @@ statistics. See `datafusion/distributed-benchmarks/README.md` for S3 output and
so a profiler attached to `dfbench` sees everything.
- Each run stores results under `testdata/<dataset>/.results/<current git branch>/` and compares
against the previous run. Running on `main` overwrites `main`'s baseline.
- `dfbench run` already repeats every query `-i` times and reports the p50, so one run per
configuration is enough; never repeat whole runs in rounds. For more reliable numbers, raise
`-i` (e.g. `-i 10`).
- Background load (other builds or tests, indexing) skews results heavily. Check it before
trusting numbers, and alternate the sides of a comparison over several rounds.
trusting numbers.
- Criterion benches (`cargo bench -p datafusion-distributed-benchmarks --bench <name>`) already
take many samples per case, so run each side once instead of repeating runs: filter to the
case you care about (e.g. `-- 'plan_dump/generate/0$'`), save one side with
`-- --save-baseline <name>` and compare the other with `-- --baseline <name>`. Benches build
their fixtures for every case even when filtered, so a full run is much slower than one case.
- CPU profiles: record with [samply](https://github.com/mstange/samply) and query them from the
terminal with Mozilla's [profiler-cli](https://github.com/firefox-devtools/profiler/tree/main/profiler-cli)
(`npm install -g @firefox-devtools/profiler-cli@latest`; run `profiler-cli guide` first):
Expand Down
50 changes: 50 additions & 0 deletions .agents/skills/plan-dump/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
name: plan-dump
description: Inspect executed distributed plans with `dfd plan`. Use when asked why a query is slow, where its time or rows go, how two plans of the same query differ (flags on/off, join reordering, branches), or whether a plan change could be returning wrong results.
---

# Inspecting plan dumps with `dfd plan`

A plan dump is one base64 line holding an executed plan with its stages and per-task metrics,
produced by `generate_plan_dump` after `rewrite_distributed_plan_with_metrics`. `dfd plan`
takes a dump as an argument, a file path, or stdin.

## Reading a dump

- **Default view.** With no options, it prints the tree of stages with stage-level metrics.
`--stage N` shows the nodes of one stage. Stage 0 is the head, run by the coordinator.
- **Per-task metrics.** `-v` shows metrics per task rather than aggregated, which is how skew
shows up.
- **Filtering metrics.** `-m TEXT` keeps metrics whose name contains `TEXT`, and is repeatable.
`--exact` matches whole names.
- `-m output_rows --exact` on the stage tree gives a compact skeleton of the whole plan.
- `-m none --exact` hides every metric, leaving plain node lines to grep.
- **Long lines.** Node lines are long, so cut them when scanning.
- **Stage-level metrics:**
- `elapsed_compute` is the CPU time of all the stage's nodes.
- `output_rows` and `output_bytes` are measured where the stage is read: what crossed the
network.
- `plan_added_at`, `plan_executed_at` and `plan_finished_at` are offsets from query start.
They give the timeline and the critical path.
- Dynamic task count adds `estimated_output_bytes` and `output_bytes_q_error`. These show how
wrong the sampled statistics were, and adaptive replanning decides from them.
- **Node metrics worth knowing:**
- On joins: `build_input_rows`, `input_rows` and `probe_hit_rate`.
- On filters: `selectivity`.

## Comparing two plans of the same query

- **Stage ids don't necessarily match across runs.** They are renumbered whenever the plan's
shape changes, so match nodes by content: node type plus its keys or expressions (join keys,
grouping, predicates).
- **Broadcast counts multiply.** Broadcast rows are counted once per consumer task, both in a
broadcast stage's `output_rows` and in a `CollectLeft` join's `build_input_rows`. A broadcast
turning into a shuffle drops them by that factor without losing any rows.
- **Logical row counts should match.** A node computing the same thing in both plans (the same
join, final aggregation or filter) should report the same `output_rows`, whatever its physical
shape. When it differs, the results differ, unless a `LIMIT`/TopK above it stops execution
early or dynamic filters prune its input differently.
- **Equal counts don't prove equal results.** Mixed-up columns keep every count intact. Check
the column indices in projections and sort expressions where the plans diverge.
- **Runs can differ.** Adaptive decisions come from sampled runtime statistics, so runs of the
same query can produce different plans. Group runs by plan shape before comparing timings.
18 changes: 9 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpch --test tpch_correctness_test
- run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_correctness_test::"
env:
ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }}

Expand All @@ -62,7 +62,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpch --test tpch_sorted_correctness_test
- run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_sorted_correctness_test::"
env:
ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }}

Expand All @@ -71,14 +71,14 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpch --test tpch_plans_test
- run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_plans_test::"

tpch-sorted-plans-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpch --test tpch_sorted_plans_test
- run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_sorted_plans_test::"

tpcds-correctness-test:
runs-on: ubuntu-latest
Expand All @@ -90,7 +90,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpcds --test tpcds_correctness_test shard${{ matrix.shard }}
- run: cargo test -p datafusion-distributed --features tpcds --test integration tpcds_correctness_test::tests::test_tpcds_shard${{ matrix.shard }}
env:
ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }}

Expand All @@ -99,7 +99,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpcds --test tpcds_plans_test
- run: "cargo test -p datafusion-distributed --features tpcds --test integration tpcds_plans_test::"

clickbench-correctness-test:
runs-on: ubuntu-latest
Expand All @@ -114,7 +114,7 @@ jobs:
with:
path: testdata/clickbench_parquet/
key: "clickbench_parquet"
- run: cargo test -p datafusion-distributed --features clickbench --test clickbench_correctness_test
- run: "cargo test -p datafusion-distributed --features clickbench --test integration clickbench_correctness_test::"
env:
ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }}

Expand All @@ -127,7 +127,7 @@ jobs:
with:
path: testdata/clickbench_parquet/
key: "clickbench_parquet"
- run: cargo test -p datafusion-distributed --features clickbench --test clickbench_plans_test
- run: "cargo test -p datafusion-distributed --features clickbench --test integration clickbench_plans_test::"

iceberg-test:
runs-on: ubuntu-latest
Expand All @@ -145,7 +145,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup
- run: cargo test -p datafusion-distributed --features tpch,chaos-tests --test chaos
- run: "cargo test -p datafusion-distributed --features tpch,chaos-tests --test integration chaos::"
env:
ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }}

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ testdata/clickbench/*
!testdata/clickbench/queries
codegen/target/*
codegen/Cargo.lock
.df-bench*
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ about a user's networking or deployment stack.
- `datafusion/distributed-iceberg-test-tools/`: shared Iceberg test fixtures.
- `datafusion/distributed-console/`: terminal console for observing a cluster.
- `datafusion/distributed-cli/`: distributed variant of the DataFusion CLI.
- `datafusion/datasets/`: TPC-H, TPC-DS and ClickBench
dataset generation shared by tests and benchmarks; it must not depend on
`datafusion-distributed`.
- `datafusion/distributed-benchmarks/`, `datafusion/distributed-iceberg-benchmarks/`,
`datafusion/distributed-remote-worker/`: local benchmark crates, remote
benchmark worker, and microbenchmarks; do not change benchmark behavior as an
Expand Down Expand Up @@ -95,6 +98,7 @@ Read the relevant contributor guide before starting work in these areas:
- Performance and benchmark changes:
`docs/source/contributor-guide/04-benchmarks.md`
- Running, comparing or profiling benchmarks: `.agents/skills/benchmark/SKILL.md`
- Inspecting executed plans and comparing them (`dfd plan`): `.agents/skills/plan-dump/SKILL.md`
- Linking upstream DataFusion/arrow-rs source to debug or fix them:
`.agents/skills/upstream-source/SKILL.md`

Expand Down
62 changes: 50 additions & 12 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
[workspace]
members = [
"datafusion/datasets",
"datafusion/distributed",
"datafusion/distributed-benchmarks",
"datafusion/distributed-cli",
Expand Down
4 changes: 4 additions & 0 deletions codegen/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ fn generate_iceberg(repo_root: &Path) -> Result<(), Box<dyn std::error::Error>>
".iceberg.DataFusionPartitioning",
"::datafusion_proto::protobuf::Partitioning",
)
.extern_path(
".iceberg.DataFusionPhysicalExprNode",
"::datafusion_proto::protobuf::PhysicalExprNode",
)
.compile_protos(&[proto_file], &[proto_dir])?;

Ok(())
Expand Down
25 changes: 25 additions & 0 deletions datafusion/datasets/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
[package]
name = "datafusion-datasets"
version = "0.1.0"
edition = "2024"
description = "TPC-H, TPC-DS and ClickBench dataset generation for DataFusion Distributed benchmarks and tests"
publish = false

[dependencies]
datafusion = { workspace = true, features = ["parquet"] }
tokio = { version = "1.48", features = ["full"] }
parquet = { version = "59" }
structopt = { version = "0.3.26" }
futures = "0.3.31"
indicatif = "0.18"
arrow = { version = "59" }
tpchgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" }
tpchgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" }
tpcdsgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" }
tpcdsgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" }
reqwest = "0.12"
object_store = { version = "0.13", features = ["aws"] }
aws-config = "1"
aws-credential-types = "1"
url = "2"
tempfile = "3"
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use super::progress::dataset_progress;
use super::{common, output::DatasetOutput};
use crate::progress::dataset_progress;
use crate::{common, output::DatasetOutput};
use arrow::array::Array;
use arrow::compute::cast;
use arrow::datatypes::{DataType, Field, Schema};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ use datafusion::prelude::{ParquetReadOptions, SessionContext};
use std::fs;
use std::path::Path;

/// Returns the workspace root directory (two levels above the benchmarks crate).
/// Returns the workspace root directory (two levels above this crate).
pub(crate) fn workspace_root() -> &'static Path {
Path::new(env!("CARGO_MANIFEST_DIR"))
.ancestors()
.nth(2)
.expect("benchmarks crate should be at datafusion/distributed-benchmarks in the workspace")
.expect("datasets crate should be at datafusion/datasets in the workspace")
}

pub fn get_queries(path: &str) -> Vec<String> {
Expand Down
Loading