diff --git a/.agents/skills/benchmark/SKILL.md b/.agents/skills/benchmark/SKILL.md index c2300d62c..fe8a7e267 100644 --- a/.agents/skills/benchmark/SKILL.md +++ b/.agents/skills/benchmark/SKILL.md @@ -16,7 +16,15 @@ DATASET=$(target/release/dfbench prepare-tpch --scale-factor 10) # only the pa target/release/dfbench run --dataset $DATASET --workers 8 -i 5 ``` -`dfbench run --help` lists all options. +`dfbench run --help` lists all options. Config options (`datafusion.*`, `distributed.*`) come from +a local, gitignored `.df-bench.yaml` in the current directory (or `--config-path `), +overridden by repeated `-c key=value` flags, e.g. `-c distributed.dynamic_task_count=true`. + +## Plan dumps + +`--debug` prints a plan dump per iteration to stdout (everything else goes to stderr), which +`dfd plan` (`-p datafusion-distributed-cli`) inspects; see the `plan-dump` skill. `dfbench` reads +`DATAFUSION_*` env vars, e.g. `DATAFUSION_OPTIMIZER_JOIN_REORDERING=false`. ## Iceberg @@ -48,8 +56,16 @@ statistics. See `datafusion/distributed-benchmarks/README.md` for S3 output and so a profiler attached to `dfbench` sees everything. - Each run stores results under `testdata//.results//` and compares against the previous run. Running on `main` overwrites `main`'s baseline. +- `dfbench run` already repeats every query `-i` times and reports the p50, so one run per + configuration is enough; never repeat whole runs in rounds. For more reliable numbers, raise + `-i` (e.g. `-i 10`). - Background load (other builds or tests, indexing) skews results heavily. Check it before - trusting numbers, and alternate the sides of a comparison over several rounds. + trusting numbers. +- Criterion benches (`cargo bench -p datafusion-distributed-benchmarks --bench `) already + take many samples per case, so run each side once instead of repeating runs: filter to the + case you care about (e.g. `-- 'plan_dump/generate/0$'`), save one side with + `-- --save-baseline ` and compare the other with `-- --baseline `. Benches build + their fixtures for every case even when filtered, so a full run is much slower than one case. - CPU profiles: record with [samply](https://github.com/mstange/samply) and query them from the terminal with Mozilla's [profiler-cli](https://github.com/firefox-devtools/profiler/tree/main/profiler-cli) (`npm install -g @firefox-devtools/profiler-cli@latest`; run `profiler-cli guide` first): diff --git a/.agents/skills/plan-dump/SKILL.md b/.agents/skills/plan-dump/SKILL.md new file mode 100644 index 000000000..673eb2333 --- /dev/null +++ b/.agents/skills/plan-dump/SKILL.md @@ -0,0 +1,50 @@ +--- +name: plan-dump +description: Inspect executed distributed plans with `dfd plan`. Use when asked why a query is slow, where its time or rows go, how two plans of the same query differ (flags on/off, join reordering, branches), or whether a plan change could be returning wrong results. +--- + +# Inspecting plan dumps with `dfd plan` + +A plan dump is one base64 line holding an executed plan with its stages and per-task metrics, +produced by `generate_plan_dump` after `rewrite_distributed_plan_with_metrics`. `dfd plan` +takes a dump as an argument, a file path, or stdin. + +## Reading a dump + +- **Default view.** With no options, it prints the tree of stages with stage-level metrics. + `--stage N` shows the nodes of one stage. Stage 0 is the head, run by the coordinator. +- **Per-task metrics.** `-v` shows metrics per task rather than aggregated, which is how skew + shows up. +- **Filtering metrics.** `-m TEXT` keeps metrics whose name contains `TEXT`, and is repeatable. + `--exact` matches whole names. + - `-m output_rows --exact` on the stage tree gives a compact skeleton of the whole plan. + - `-m none --exact` hides every metric, leaving plain node lines to grep. +- **Long lines.** Node lines are long, so cut them when scanning. +- **Stage-level metrics:** + - `elapsed_compute` is the CPU time of all the stage's nodes. + - `output_rows` and `output_bytes` are measured where the stage is read: what crossed the + network. + - `plan_added_at`, `plan_executed_at` and `plan_finished_at` are offsets from query start. + They give the timeline and the critical path. + - Dynamic task count adds `estimated_output_bytes` and `output_bytes_q_error`. These show how + wrong the sampled statistics were, and adaptive replanning decides from them. +- **Node metrics worth knowing:** + - On joins: `build_input_rows`, `input_rows` and `probe_hit_rate`. + - On filters: `selectivity`. + +## Comparing two plans of the same query + +- **Stage ids don't necessarily match across runs.** They are renumbered whenever the plan's + shape changes, so match nodes by content: node type plus its keys or expressions (join keys, + grouping, predicates). +- **Broadcast counts multiply.** Broadcast rows are counted once per consumer task, both in a + broadcast stage's `output_rows` and in a `CollectLeft` join's `build_input_rows`. A broadcast + turning into a shuffle drops them by that factor without losing any rows. +- **Logical row counts should match.** A node computing the same thing in both plans (the same + join, final aggregation or filter) should report the same `output_rows`, whatever its physical + shape. When it differs, the results differ, unless a `LIMIT`/TopK above it stops execution + early or dynamic filters prune its input differently. +- **Equal counts don't prove equal results.** Mixed-up columns keep every count intact. Check + the column indices in projections and sort expressions where the plans diverge. +- **Runs can differ.** Adaptive decisions come from sampled runtime statistics, so runs of the + same query can produce different plans. Group runs by plan shape before comparing timings. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5463ef4a5..2d5a9cb2a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -49,7 +49,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpch --test tpch_correctness_test + - run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_correctness_test::" env: ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }} @@ -62,7 +62,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpch --test tpch_sorted_correctness_test + - run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_sorted_correctness_test::" env: ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }} @@ -71,14 +71,14 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpch --test tpch_plans_test + - run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_plans_test::" tpch-sorted-plans-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpch --test tpch_sorted_plans_test + - run: "cargo test -p datafusion-distributed --features tpch --test integration tpch_sorted_plans_test::" tpcds-correctness-test: runs-on: ubuntu-latest @@ -90,7 +90,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpcds --test tpcds_correctness_test shard${{ matrix.shard }} + - run: cargo test -p datafusion-distributed --features tpcds --test integration tpcds_correctness_test::tests::test_tpcds_shard${{ matrix.shard }} env: ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }} @@ -99,7 +99,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpcds --test tpcds_plans_test + - run: "cargo test -p datafusion-distributed --features tpcds --test integration tpcds_plans_test::" clickbench-correctness-test: runs-on: ubuntu-latest @@ -114,7 +114,7 @@ jobs: with: path: testdata/clickbench_parquet/ key: "clickbench_parquet" - - run: cargo test -p datafusion-distributed --features clickbench --test clickbench_correctness_test + - run: "cargo test -p datafusion-distributed --features clickbench --test integration clickbench_correctness_test::" env: ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }} @@ -127,7 +127,7 @@ jobs: with: path: testdata/clickbench_parquet/ key: "clickbench_parquet" - - run: cargo test -p datafusion-distributed --features clickbench --test clickbench_plans_test + - run: "cargo test -p datafusion-distributed --features clickbench --test integration clickbench_plans_test::" iceberg-test: runs-on: ubuntu-latest @@ -145,7 +145,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - - run: cargo test -p datafusion-distributed --features tpch,chaos-tests --test chaos + - run: "cargo test -p datafusion-distributed --features tpch,chaos-tests --test integration chaos::" env: ADAPTIVE: ${{ matrix.planning_mode == 'adaptive' }} diff --git a/.gitignore b/.gitignore index 9c9baee8c..21b6e9039 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ testdata/clickbench/* !testdata/clickbench/queries codegen/target/* codegen/Cargo.lock +.df-bench* diff --git a/AGENTS.md b/AGENTS.md index 6a47d8238..4ff0f8003 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,6 +40,9 @@ about a user's networking or deployment stack. - `datafusion/distributed-iceberg-test-tools/`: shared Iceberg test fixtures. - `datafusion/distributed-console/`: terminal console for observing a cluster. - `datafusion/distributed-cli/`: distributed variant of the DataFusion CLI. +- `datafusion/datasets/`: TPC-H, TPC-DS and ClickBench + dataset generation shared by tests and benchmarks; it must not depend on + `datafusion-distributed`. - `datafusion/distributed-benchmarks/`, `datafusion/distributed-iceberg-benchmarks/`, `datafusion/distributed-remote-worker/`: local benchmark crates, remote benchmark worker, and microbenchmarks; do not change benchmark behavior as an @@ -95,6 +98,7 @@ Read the relevant contributor guide before starting work in these areas: - Performance and benchmark changes: `docs/source/contributor-guide/04-benchmarks.md` - Running, comparing or profiling benchmarks: `.agents/skills/benchmark/SKILL.md` +- Inspecting executed plans and comparing them (`dfd plan`): `.agents/skills/plan-dump/SKILL.md` - Linking upstream DataFusion/arrow-rs source to debug or fix them: `.agents/skills/upstream-source/SKILL.md` diff --git a/Cargo.lock b/Cargo.lock index 762d2cf2f..5d72ad4fb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2211,6 +2211,29 @@ dependencies = [ "tokio", ] +[[package]] +name = "datafusion-datasets" +version = "0.1.0" +dependencies = [ + "arrow", + "aws-config", + "aws-credential-types", + "datafusion", + "futures", + "indicatif", + "object_store", + "parquet", + "reqwest 0.12.28", + "structopt", + "tempfile", + "tokio", + "tpcdsgen", + "tpcdsgen-arrow", + "tpchgen", + "tpchgen-arrow", + "url", +] + [[package]] name = "datafusion-datasource" version = "55.0.0" @@ -2382,12 +2405,13 @@ dependencies = [ "arrow-ipc", "arrow-select", "async-trait", + "base64 0.22.1", "bytes", "chrono", "crossbeam-queue", "dashmap", "datafusion", - "datafusion-distributed-benchmarks", + "datafusion-datasets", "datafusion-proto", "delegate", "futures", @@ -2415,6 +2439,7 @@ dependencies = [ "tower", "url", "uuid", + "zstd 0.13.3", ] [[package]] @@ -2423,29 +2448,21 @@ version = "0.1.0" dependencies = [ "arrow", "async-trait", - "aws-config", - "aws-credential-types", "built", "criterion", "datafusion", + "datafusion-datasets", "datafusion-distributed", "env_logger", "futures", - "indicatif", - "object_store", "parquet", - "reqwest 0.12.28", "serde", "serde_json", + "serde_yaml_ng", "sketches-ddsketch", "structopt", - "tempfile", "tokio", "tonic", - "tpcdsgen", - "tpcdsgen-arrow", - "tpchgen", - "tpchgen-arrow", "url", ] @@ -2459,6 +2476,7 @@ dependencies = [ "datafusion-distributed", "dirs", "env_logger", + "prost", "tokio", ] @@ -2471,8 +2489,8 @@ dependencies = [ "color-eyre", "crossterm", "datafusion", + "datafusion-datasets", "datafusion-distributed", - "datafusion-distributed-benchmarks", "futures", "ratatui", "structopt", @@ -2512,6 +2530,7 @@ version = "0.1.0" dependencies = [ "async-trait", "datafusion", + "datafusion-datasets", "datafusion-distributed-benchmarks", "datafusion-distributed-iceberg", "env_logger", @@ -7031,6 +7050,19 @@ dependencies = [ "syn 3.0.5", ] +[[package]] +name = "serde_yaml_ng" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4db627b98b36d4203a7b458cf3573730f2bb591b28871d916dfa9efabfd41f" +dependencies = [ + "indexmap 2.14.2", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "sha1" version = "0.10.7" @@ -8089,6 +8121,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" diff --git a/Cargo.toml b/Cargo.toml index de8d9fceb..2ef245005 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,6 @@ [workspace] members = [ + "datafusion/datasets", "datafusion/distributed", "datafusion/distributed-benchmarks", "datafusion/distributed-cli", diff --git a/codegen/src/main.rs b/codegen/src/main.rs index dd9440dd5..f8922b032 100644 --- a/codegen/src/main.rs +++ b/codegen/src/main.rs @@ -73,6 +73,10 @@ fn generate_iceberg(repo_root: &Path) -> Result<(), Box> ".iceberg.DataFusionPartitioning", "::datafusion_proto::protobuf::Partitioning", ) + .extern_path( + ".iceberg.DataFusionPhysicalExprNode", + "::datafusion_proto::protobuf::PhysicalExprNode", + ) .compile_protos(&[proto_file], &[proto_dir])?; Ok(()) diff --git a/datafusion/datasets/Cargo.toml b/datafusion/datasets/Cargo.toml new file mode 100644 index 000000000..68a074255 --- /dev/null +++ b/datafusion/datasets/Cargo.toml @@ -0,0 +1,25 @@ +[package] +name = "datafusion-datasets" +version = "0.1.0" +edition = "2024" +description = "TPC-H, TPC-DS and ClickBench dataset generation for DataFusion Distributed benchmarks and tests" +publish = false + +[dependencies] +datafusion = { workspace = true, features = ["parquet"] } +tokio = { version = "1.48", features = ["full"] } +parquet = { version = "59" } +structopt = { version = "0.3.26" } +futures = "0.3.31" +indicatif = "0.18" +arrow = { version = "59" } +tpchgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } +tpchgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } +tpcdsgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } +tpcdsgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } +reqwest = "0.12" +object_store = { version = "0.13", features = ["aws"] } +aws-config = "1" +aws-credential-types = "1" +url = "2" +tempfile = "3" diff --git a/datafusion/distributed-benchmarks/src/datasets/clickbench.rs b/datafusion/datasets/src/clickbench.rs similarity index 98% rename from datafusion/distributed-benchmarks/src/datasets/clickbench.rs rename to datafusion/datasets/src/clickbench.rs index b52f9f2d0..2f120ea10 100644 --- a/datafusion/distributed-benchmarks/src/datasets/clickbench.rs +++ b/datafusion/datasets/src/clickbench.rs @@ -1,5 +1,5 @@ -use super::progress::dataset_progress; -use super::{common, output::DatasetOutput}; +use crate::progress::dataset_progress; +use crate::{common, output::DatasetOutput}; use arrow::array::Array; use arrow::compute::cast; use arrow::datatypes::{DataType, Field, Schema}; diff --git a/datafusion/distributed-benchmarks/src/datasets/common.rs b/datafusion/datasets/src/common.rs similarity index 93% rename from datafusion/distributed-benchmarks/src/datasets/common.rs rename to datafusion/datasets/src/common.rs index aa53ed1f7..62416bf39 100644 --- a/datafusion/distributed-benchmarks/src/datasets/common.rs +++ b/datafusion/datasets/src/common.rs @@ -3,12 +3,12 @@ use datafusion::prelude::{ParquetReadOptions, SessionContext}; use std::fs; use std::path::Path; -/// Returns the workspace root directory (two levels above the benchmarks crate). +/// Returns the workspace root directory (two levels above this crate). pub(crate) fn workspace_root() -> &'static Path { Path::new(env!("CARGO_MANIFEST_DIR")) .ancestors() .nth(2) - .expect("benchmarks crate should be at datafusion/distributed-benchmarks in the workspace") + .expect("datasets crate should be at datafusion/datasets in the workspace") } pub fn get_queries(path: &str) -> Vec { diff --git a/datafusion/distributed-benchmarks/src/datasets/mod.rs b/datafusion/datasets/src/lib.rs similarity index 68% rename from datafusion/distributed-benchmarks/src/datasets/mod.rs rename to datafusion/datasets/src/lib.rs index 3cc5a8533..c3b37f91b 100644 --- a/datafusion/distributed-benchmarks/src/datasets/mod.rs +++ b/datafusion/datasets/src/lib.rs @@ -2,6 +2,9 @@ pub mod clickbench; pub(crate) mod common; pub mod output; pub mod parquet_layout; +pub mod prepare_clickbench; +pub mod prepare_tpcds; +pub mod prepare_tpch; mod progress; pub mod tpcds; pub mod tpch; diff --git a/datafusion/distributed-benchmarks/src/datasets/output.rs b/datafusion/datasets/src/output.rs similarity index 99% rename from datafusion/distributed-benchmarks/src/datasets/output.rs rename to datafusion/datasets/src/output.rs index 978cd6a1b..9f8f3ecaa 100644 --- a/datafusion/distributed-benchmarks/src/datasets/output.rs +++ b/datafusion/datasets/src/output.rs @@ -14,7 +14,7 @@ use futures::TryStreamExt; use tokio::io::AsyncWriteExt; use url::Url; -use crate::datasets::common::workspace_root; +use crate::common::workspace_root; const SUCCESS: &str = "_SUCCESS"; diff --git a/datafusion/distributed-benchmarks/src/datasets/parquet_layout.rs b/datafusion/datasets/src/parquet_layout.rs similarity index 99% rename from datafusion/distributed-benchmarks/src/datasets/parquet_layout.rs rename to datafusion/datasets/src/parquet_layout.rs index 608be8313..01343aa46 100644 --- a/datafusion/distributed-benchmarks/src/datasets/parquet_layout.rs +++ b/datafusion/datasets/src/parquet_layout.rs @@ -1,4 +1,4 @@ -use super::output::DatasetOutput; +use crate::output::DatasetOutput; use arrow::array::RecordBatch; use arrow::compute::cast; use arrow::datatypes::{DataType, Field, Schema}; diff --git a/datafusion/distributed-benchmarks/src/prepare_clickbench.rs b/datafusion/datasets/src/prepare_clickbench.rs similarity index 89% rename from datafusion/distributed-benchmarks/src/prepare_clickbench.rs rename to datafusion/datasets/src/prepare_clickbench.rs index 1626cdcf3..6e966cdda 100644 --- a/datafusion/distributed-benchmarks/src/prepare_clickbench.rs +++ b/datafusion/datasets/src/prepare_clickbench.rs @@ -1,7 +1,7 @@ use datafusion::common::{Result, exec_datafusion_err}; -use datafusion_distributed_benchmarks::datasets::clickbench::{ClickBenchSpec, generate_data}; -use datafusion_distributed_benchmarks::datasets::output::TESTDATA_ROOT; +use crate::clickbench::{ClickBenchSpec, generate_data}; +use crate::output::TESTDATA_ROOT; use structopt::StructOpt; /// Prepare ClickBench parquet files for benchmarks diff --git a/datafusion/distributed-benchmarks/src/prepare_tpcds.rs b/datafusion/datasets/src/prepare_tpcds.rs similarity index 81% rename from datafusion/distributed-benchmarks/src/prepare_tpcds.rs rename to datafusion/datasets/src/prepare_tpcds.rs index 094a8b738..4426f2442 100644 --- a/datafusion/distributed-benchmarks/src/prepare_tpcds.rs +++ b/datafusion/datasets/src/prepare_tpcds.rs @@ -1,7 +1,7 @@ +use crate::output::TESTDATA_ROOT; +use crate::parquet_layout::ParquetLayout; +use crate::tpcds::{TpcdsSpec, generate_data}; use datafusion::common::{Result, exec_datafusion_err}; -use datafusion_distributed_benchmarks::datasets::output::TESTDATA_ROOT; -use datafusion_distributed_benchmarks::datasets::parquet_layout::ParquetLayout; -use datafusion_distributed_benchmarks::datasets::tpcds::{TpcdsSpec, generate_data}; use structopt::StructOpt; /// Prepare TPC-DS parquet files for benchmarks diff --git a/datafusion/distributed-benchmarks/src/prepare_tpch.rs b/datafusion/datasets/src/prepare_tpch.rs similarity index 88% rename from datafusion/distributed-benchmarks/src/prepare_tpch.rs rename to datafusion/datasets/src/prepare_tpch.rs index c9f2f1cd2..23a1afd05 100644 --- a/datafusion/distributed-benchmarks/src/prepare_tpch.rs +++ b/datafusion/datasets/src/prepare_tpch.rs @@ -15,10 +15,10 @@ // specific language governing permissions and limitations // under the License. +use crate::output::TESTDATA_ROOT; +use crate::parquet_layout::ParquetLayout; +use crate::tpch::{TpchSpec, generate_data}; use datafusion::common::{Result, exec_datafusion_err}; -use datafusion_distributed_benchmarks::datasets::output::TESTDATA_ROOT; -use datafusion_distributed_benchmarks::datasets::parquet_layout::ParquetLayout; -use datafusion_distributed_benchmarks::datasets::tpch::{TpchSpec, generate_data}; use structopt::StructOpt; /// Generate TPC-H parquet files for benchmarks diff --git a/datafusion/distributed-benchmarks/src/datasets/progress.rs b/datafusion/datasets/src/progress.rs similarity index 100% rename from datafusion/distributed-benchmarks/src/datasets/progress.rs rename to datafusion/datasets/src/progress.rs diff --git a/datafusion/distributed-benchmarks/src/datasets/tpcds.rs b/datafusion/datasets/src/tpcds.rs similarity index 98% rename from datafusion/distributed-benchmarks/src/datasets/tpcds.rs rename to datafusion/datasets/src/tpcds.rs index 60ffffc32..2977d80ab 100644 --- a/datafusion/distributed-benchmarks/src/datasets/tpcds.rs +++ b/datafusion/datasets/src/tpcds.rs @@ -1,6 +1,6 @@ -use super::parquet_layout::{ParquetLayout, TableColumns, TableWriter}; -use super::progress::dataset_progress; -use super::{common, output::DatasetOutput}; +use crate::parquet_layout::{ParquetLayout, TableColumns, TableWriter}; +use crate::progress::dataset_progress; +use crate::{common, output::DatasetOutput}; use arrow::datatypes::{Field, Schema}; use arrow::record_batch::RecordBatch; use datafusion::common::exec_datafusion_err; diff --git a/datafusion/distributed-benchmarks/src/datasets/tpch.rs b/datafusion/datasets/src/tpch.rs similarity index 96% rename from datafusion/distributed-benchmarks/src/datasets/tpch.rs rename to datafusion/datasets/src/tpch.rs index bb9f52076..41af6dfd1 100644 --- a/datafusion/distributed-benchmarks/src/datasets/tpch.rs +++ b/datafusion/datasets/src/tpch.rs @@ -1,6 +1,6 @@ -use super::parquet_layout::{ParquetLayout, TableColumns, TableWriter}; -use super::progress::dataset_progress; -use super::{common, output::DatasetOutput}; +use crate::parquet_layout::{ParquetLayout, TableColumns, TableWriter}; +use crate::progress::dataset_progress; +use crate::{common, output::DatasetOutput}; use datafusion::common::{DataFusionError, exec_datafusion_err}; use indicatif::HumanCount; use tpchgen::generators::{ diff --git a/datafusion/distributed-benchmarks/Cargo.toml b/datafusion/distributed-benchmarks/Cargo.toml index 08e548e02..bdad6aaaf 100644 --- a/datafusion/distributed-benchmarks/Cargo.toml +++ b/datafusion/distributed-benchmarks/Cargo.toml @@ -7,6 +7,7 @@ default-run = "dfbench" [dependencies] async-trait = "0.1.89" datafusion = { workspace = true } +datafusion-datasets = { path = "../datasets" } datafusion-distributed = { path = "../distributed", features = [ "integration", "system-metrics", @@ -16,22 +17,13 @@ parquet = { version = "59" } structopt = { version = "0.3.26" } serde = { version = "1.0.219", features = ["derive"] } serde_json = "1.0.141" +serde_yaml_ng = "0.10" env_logger = "0.11.8" futures = "0.3.31" -indicatif = "0.18" arrow = { version = "59", features = ["test_utils"] } tonic = { version = "0.14.1", features = ["transport"] } -tpchgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } -tpchgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } -tpcdsgen = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } -tpcdsgen-arrow = { git = "https://github.com/clflushopt/tpchgen-rs", rev = "f42708c32e7fd816b8ad8d2a880c1802a71b669f" } -reqwest = "0.12" sketches-ddsketch = "0.3" -object_store = { version = "0.13", features = ["aws"] } -aws-config = "1" -aws-credential-types = "1" url = "2" -tempfile = "3" [dev-dependencies] criterion = "0.5" @@ -66,3 +58,11 @@ harness = false [[bench]] name = "metrics_collection" harness = false + +[[bench]] +name = "plan_dump" +harness = false + +[[bench]] +name = "startup_overlap" +harness = false diff --git a/datafusion/distributed-benchmarks/README.md b/datafusion/distributed-benchmarks/README.md index d5c01fb6f..f44e82c8f 100644 --- a/datafusion/distributed-benchmarks/README.md +++ b/datafusion/distributed-benchmarks/README.md @@ -42,7 +42,7 @@ options: With `--sorted`, the sort keys declared for each table are listed in `table_columns` in -`src/datasets/tpch.rs` and `tpcds.rs`. +`../datasets/src/tpch.rs` and `tpcds.rs`. `_SUCCESS` is written once every table is complete, and rerunning a `prepare-*` command for a complete dataset does nothing. Any other non-empty destination, @@ -111,7 +111,8 @@ cargo run -p datafusion-distributed-benchmarks --release -- run --dataset $DATAS The same command is used for running distributed benchmarks: ```shell -cargo run -p datafusion-distributed-benchmarks --release -- run --workers 8 --dataset $DATASET --file-scan-config-bytes-per-partition 16777216 +cargo run -p datafusion-distributed-benchmarks --release -- run --workers 8 --dataset $DATASET \ + -c distributed.file_scan_config_bytes_per_partition=16777216 ``` - `--workers`: Amount of localhost workers used in the query. They all run in @@ -120,9 +121,25 @@ cargo run -p datafusion-distributed-benchmarks --release -- run --workers 8 --da attached to the benchmark process observes both the coordinator and the workers on the same threads. - `--dataset`: Dataset path, as above. -- `--file-scan-config-bytes-per-partition`: How many bytes each partition is - expected to scan. Lower values produce more partitions/tasks. Defaults to the - engine default when unset. +- `-c key=value`: Sets any `datafusion.*` or `distributed.*` config option, and + can be repeated. These take precedence over the config file. + +### Config options + +Config options are read from the `.df-bench.yaml` file in the current directory +when present, or from the file given with `--config-path`. It holds flat +`key: value` pairs, applied in file order on top of any `DATAFUSION_*` +environment variables, and `-c key=value` flags are applied last: + +```yaml +datafusion.execution.target_partitions: 8 +distributed.broadcast_joins: true +``` + +`distributed.dynamic_plan_remainder_replanning` only takes effect together with +`distributed.dynamic_task_count: true`. Metrics are collected by default, as +`--debug` and dynamic task count read them; set `distributed.collect_metrics: false` +to turn them off. ### Iceberg benchmarks @@ -164,9 +181,9 @@ Run each representation with its format-specific binary, then compare them with ```shell cargo run -p datafusion-distributed-benchmarks --release -- run \ - --workers 2 --dataset $DATASET --partitions 2 + --workers 2 --dataset $DATASET -c datafusion.execution.target_partitions=2 cargo run -p datafusion-distributed-iceberg-benchmarks --release -- run \ - --workers 2 --dataset $ICEBERG --partitions 2 + --workers 2 --dataset $ICEBERG -c datafusion.execution.target_partitions=2 dfbench compare $DATASET $ICEBERG dfbench compare $DATASET@base $ICEBERG@candidate diff --git a/datafusion/distributed-benchmarks/benches/metrics_collection.rs b/datafusion/distributed-benchmarks/benches/metrics_collection.rs index cbbe88466..b24177a4f 100644 --- a/datafusion/distributed-benchmarks/benches/metrics_collection.rs +++ b/datafusion/distributed-benchmarks/benches/metrics_collection.rs @@ -1,15 +1,15 @@ use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use datafusion::error::DataFusionError; use datafusion::physical_plan::collect; +use datafusion_datasets::output::TESTDATA_ROOT; +use datafusion_datasets::parquet_layout::ParquetLayout; +use datafusion_datasets::tpch::TpchSpec; +use datafusion_datasets::{register_tables, tpch}; use datafusion_distributed::test_utils::localhost::start_localhost_context; use datafusion_distributed::{ DefaultSessionBuilder, DistributedExec, DistributedExt, DistributedMetricsFormat, display_plan_ascii, rewrite_distributed_plan_with_metrics, }; -use datafusion_distributed_benchmarks::datasets::output::TESTDATA_ROOT; -use datafusion_distributed_benchmarks::datasets::parquet_layout::ParquetLayout; -use datafusion_distributed_benchmarks::datasets::tpch::TpchSpec; -use datafusion_distributed_benchmarks::datasets::{register_tables, tpch}; use std::path::PathBuf; use std::sync::Arc; use tokio::runtime::Builder as RuntimeBuilder; diff --git a/datafusion/distributed-benchmarks/benches/plan_dump.rs b/datafusion/distributed-benchmarks/benches/plan_dump.rs new file mode 100644 index 000000000..d80a15560 --- /dev/null +++ b/datafusion/distributed-benchmarks/benches/plan_dump.rs @@ -0,0 +1,83 @@ +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; +use datafusion::error::DataFusionError; +use datafusion::physical_plan::collect; +use datafusion_datasets::output::TESTDATA_ROOT; +use datafusion_datasets::parquet_layout::ParquetLayout; +use datafusion_datasets::tpch::TpchSpec; +use datafusion_datasets::{register_tables, tpch}; +use datafusion_distributed::test_utils::localhost::start_localhost_context; +use datafusion_distributed::{ + DefaultSessionBuilder, DistributedExec, DistributedExt, DistributedMetricsFormat, + generate_plan_dump, rewrite_distributed_plan_with_metrics, +}; +use std::path::PathBuf; +use std::sync::Arc; +use tokio::runtime::Builder as RuntimeBuilder; + +const NUM_WORKERS: usize = 10; +const TARGET_PARTITIONS: usize = 128; + +/// Measures [generate_plan_dump] on executed TPC-H plans with their metrics, which is the part of +/// dumping a plan that runs in an application. Executing the queries and collecting their metrics +/// happens once per query, outside of the measurement. +fn plan_dump(c: &mut Criterion) { + let rt = RuntimeBuilder::new_multi_thread() + .enable_all() + .build() + .expect("tokio runtime"); + + let mut group = c.benchmark_group("plan_dump"); + + let tpch_spec = TpchSpec { + location: TESTDATA_ROOT.to_string(), + scale_factor: 0.1, + layout: ParquetLayout::default(), + }; + + let (ctx, _join_set, _workers) = rt + .block_on(async move { + tpch::generate_data(&tpch_spec).await.unwrap(); + + let (ctx, join_set, workers) = + start_localhost_context(NUM_WORKERS, DefaultSessionBuilder).await; + let ctx = ctx.with_distributed_metrics_collection(true)?; + ctx.sql(&format!( + "SET datafusion.execution.target_partitions = {TARGET_PARTITIONS}" + )) + .await?; + register_tables(&ctx, &PathBuf::from(tpch_spec.dataset())).await?; + + Ok::<_, DataFusionError>((ctx, join_set, workers)) + }) + .unwrap(); + + for (i, query) in tpch::get_queries().iter().enumerate() { + if query == "q15" { + // q15 is weird because it has views, rather than complicating the code here, let's just + // skip it. + continue; + } + let ctx = ctx.clone(); + + let plan_with_metrics = rt + .block_on(async move { + let sql = tpch::get_query(query)?; + let plan = ctx.sql(&sql).await?.create_physical_plan().await?; + if !plan.is::() { + panic!("The plan was not distributed") + } + let _ = collect(Arc::clone(&plan), ctx.task_ctx()).await?; + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::PerTask).await + }) + .unwrap(); + + group.bench_function(BenchmarkId::new("generate", i), |b| { + b.iter(|| generate_plan_dump(&plan_with_metrics).unwrap()) + }); + } + + group.finish() +} + +criterion_group!(benches, plan_dump); +criterion_main!(benches); diff --git a/datafusion/distributed-benchmarks/benches/planning.rs b/datafusion/distributed-benchmarks/benches/planning.rs index f8a30e67e..d2b530e2e 100644 --- a/datafusion/distributed-benchmarks/benches/planning.rs +++ b/datafusion/distributed-benchmarks/benches/planning.rs @@ -1,11 +1,11 @@ use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use datafusion::prelude::SessionContext; +use datafusion_datasets::output::TESTDATA_ROOT; +use datafusion_datasets::parquet_layout::ParquetLayout; +use datafusion_datasets::tpch::TpchSpec; +use datafusion_datasets::{register_tables, tpch}; use datafusion_distributed::test_utils::in_memory_channel_resolver::InMemoryWorkerResolver; use datafusion_distributed::{DistributedExt, create_distributed_plan}; -use datafusion_distributed_benchmarks::datasets::output::TESTDATA_ROOT; -use datafusion_distributed_benchmarks::datasets::parquet_layout::ParquetLayout; -use datafusion_distributed_benchmarks::datasets::tpch::TpchSpec; -use datafusion_distributed_benchmarks::datasets::{register_tables, tpch}; use std::path::PathBuf; use std::sync::Arc; use tokio::runtime::Builder as RuntimeBuilder; diff --git a/datafusion/distributed-benchmarks/benches/startup_overlap.rs b/datafusion/distributed-benchmarks/benches/startup_overlap.rs new file mode 100644 index 000000000..f171d7189 --- /dev/null +++ b/datafusion/distributed-benchmarks/benches/startup_overlap.rs @@ -0,0 +1,229 @@ +//! # Task startup overlap +//! +//! ```shell +//! cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --save-baseline main +//! # Run the same benchmark on the candidate revision: +//! cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --baseline main +//! ``` +//! +//! This controlled benchmark runs two tasks over localhost gRPC in static and AQE +//! mode. Task 0 performs 100 million dependent integer operations; task 1 returns +//! one row. The existing localhost fixture starts three endpoints so both active +//! tasks can route away from the coordinator's local endpoint. Each query must +//! return task indices `[0, 1]`, and the CPU function must execute remotely once. +//! +//! `delay_light_task` adds 250 ms to task 1's routing setup, allowing task 0's CPU +//! work to overlap that wait. `delay_heavy_task` delays task 0 instead, keeping the +//! wait on the critical path; `no_delay` provides the other control. The delay is +//! only in routing setup, not in the CPU workload. Criterion measures execution +//! through complete output collection; worker startup and query planning are +//! excluded. An untimed preflight also reports first-batch time and whether remote +//! CPU work started and finished before the delayed task returned a ready +//! destination. The witnesses are reported without requiring overlap, so the +//! benchmark also runs before the optimization. +//! +//! Run both revisions on a quiet host and alternate comparisons. Verify a positive +//! overlap witness in the candidate's `delay_light_task` preflight and its absence +//! in the baseline before interpreting timings. This isolates scheduling value +//! under controlled setup skew; it does not replace representative suite results. + +use arrow::array::Int64Array; +use arrow::datatypes::DataType; +use async_trait::async_trait; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; +use datafusion::common::{Result, exec_err}; +use datafusion::execution::SessionState; +use datafusion::logical_expr::{ScalarUDF, Volatility, create_udf}; +use datafusion::physical_plan::execute_stream; +use datafusion::prelude::SessionContext; +use datafusion_distributed::test_utils::localhost::start_localhost_context; +use datafusion_distributed::test_utils::routing::{ + URLEmitterExtensionCodec, URLEmitterFunction, UrlEmitterRouteTaskHandler, + url_emitter_desired_task_count, url_emitter_scale_up_leaf_node, +}; +use datafusion_distributed::{ + DistributedExt, MappedWorkerSessionBuilderExt, RouteTaskEvent, RouteTaskEventResponse, + RouteTaskHandler, WorkerQueryContext, +}; +use futures::TryStreamExt; +use std::hint::black_box; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::time::{Duration, Instant}; +use tokio::runtime::Builder as RuntimeBuilder; +use tokio::time::timeout; + +const QUERY: &str = "SELECT cpu_work(task_index) FROM url_emitter(2, 2, 'startup')"; +const SETUP_DELAY: Duration = Duration::from_millis(250); +const WAIT: Duration = Duration::from_secs(10); + +fn startup_overlap(c: &mut Criterion) { + let rt = RuntimeBuilder::new_multi_thread() + .worker_threads(4) + .enable_all() + .build() + .unwrap(); + let mut group = c.benchmark_group("startup_overlap"); + group.sample_size(10); + group.warm_up_time(Duration::from_secs(1)); + group.measurement_time(Duration::from_secs(5)); + + for (mode, adaptive) in [("static", false), ("aqe", true)] { + for (scenario, delayed_task) in [ + ("no_delay", None), + ("delay_light_task", Some(1)), + ("delay_heavy_task", Some(0)), + ] { + let probe = Arc::new(Probe::default()); + let udf = Arc::new(cpu_work(Some(Arc::clone(&probe)))); + let worker = build_state.map(move |builder| { + Ok(builder + .with_scalar_functions(vec![Arc::clone(&udf)]) + .build()) + }); + // Routing uses URLs 2 and 1, avoiding URL 0's local-worker shortcut. + let (mut ctx, _workers, _) = rt.block_on(start_localhost_context(3, worker)); + ctx.set_distributed_desired_task_count_handler(url_emitter_desired_task_count); + ctx.set_distributed_scale_up_leaf_node_handler(url_emitter_scale_up_leaf_node); + ctx.set_distributed_user_codec(URLEmitterExtensionCodec); + ctx.register_udtf("url_emitter", Arc::new(URLEmitterFunction)); + ctx.register_udf(cpu_work(None)); + let ctx = ctx + .with_distributed_dynamic_task_count(adaptive) + .unwrap() + .with_distributed_route_task_handler(SetupDelay { + delayed_task, + probe: Arc::clone(&probe), + }); + rt.block_on(ctx.sql("SET datafusion.execution.target_partitions = 2")) + .unwrap(); + + let (first, total) = rt.block_on(run_query(&ctx, &probe)).unwrap(); + println!( + "preflight {mode}/{scenario}: first={first:?} total={total:?} \ + heavy_started_before_ready={} heavy_finished_before_ready={}", + probe.started_before_ready.load(Ordering::SeqCst), + probe.finished_before_ready.load(Ordering::SeqCst), + ); + group.bench_function(BenchmarkId::new(mode, scenario), |b| { + b.iter_custom(|iterations| { + rt.block_on(async { + let mut elapsed = Duration::ZERO; + for _ in 0..iterations { + elapsed += run_query(&ctx, &probe).await.unwrap().1; + } + elapsed + }) + }) + }); + } + } + group.finish(); +} + +async fn run_query(ctx: &SessionContext, probe: &Probe) -> Result<(Duration, Duration)> { + let plan = ctx.sql(QUERY).await?.create_physical_plan().await?; + probe.delayed_ready.store(false, Ordering::SeqCst); + probe.started_before_ready.store(false, Ordering::SeqCst); + probe.finished_before_ready.store(false, Ordering::SeqCst); + probe.heavy_executions.store(0, Ordering::SeqCst); + let start = Instant::now(); + let mut output = execute_stream(plan, ctx.task_ctx())?; + let first = timeout(WAIT, output.try_next()).await.unwrap()?.unwrap(); + let first_elapsed = start.elapsed(); + let mut batches = vec![first]; + batches.extend( + timeout(WAIT, output.try_collect::>()) + .await + .unwrap()?, + ); + let total_elapsed = start.elapsed(); + let mut values: Vec<_> = batches + .iter() + .flat_map(|batch| { + batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values() + .to_vec() + }) + .collect(); + values.sort_unstable(); + assert_eq!(values, vec![0, 1]); + assert_eq!(probe.heavy_executions.load(Ordering::SeqCst), 1); + Ok((first_elapsed, total_elapsed)) +} + +async fn build_state(ctx: WorkerQueryContext) -> Result { + Ok(ctx + .builder + .with_distributed_user_codec(URLEmitterExtensionCodec) + .build()) +} + +#[derive(Default)] +struct Probe { + delayed_ready: AtomicBool, + started_before_ready: AtomicBool, + finished_before_ready: AtomicBool, + heavy_executions: AtomicUsize, +} + +struct SetupDelay { + delayed_task: Option, + probe: Arc, +} + +#[async_trait] +impl RouteTaskHandler for SetupDelay { + async fn handle(&self, event: RouteTaskEvent<'_>) -> Option> { + let delayed = self.delayed_task == Some(event.task_key.task_number); + if delayed { + tokio::time::sleep(SETUP_DELAY).await; + } + let response = UrlEmitterRouteTaskHandler.handle(event).await?; + if delayed || self.delayed_task.is_none() { + self.probe.delayed_ready.store(true, Ordering::SeqCst); + } + Some(response) + } +} + +fn cpu_work(probe: Option>) -> ScalarUDF { + create_udf( + "cpu_work", + vec![DataType::Int64], + DataType::Int64, + Volatility::Volatile, + Arc::new(move |args| { + let Some(probe) = &probe else { + return exec_err!("cpu_work must execute on a remote worker"); + }; + let input = args[0].clone().into_array(1)?; + let input = input.as_any().downcast_ref::().unwrap(); + for task in input.values() { + if *task == 0 { + probe.heavy_executions.fetch_add(1, Ordering::SeqCst); + probe.started_before_ready.store( + !probe.delayed_ready.load(Ordering::SeqCst), + Ordering::SeqCst, + ); + let mut value = 1u64; + for _ in 0..100_000_000 { + value = black_box(value.wrapping_mul(17).wrapping_add(1)); + } + probe.finished_before_ready.store( + !probe.delayed_ready.load(Ordering::SeqCst), + Ordering::SeqCst, + ); + } + } + Ok(args[0].clone()) + }), + ) +} + +criterion_group!(benches, startup_overlap); +criterion_main!(benches); diff --git a/datafusion/distributed-benchmarks/src/backend.rs b/datafusion/distributed-benchmarks/src/backend.rs index 9bdbadbc7..08115fb45 100644 --- a/datafusion/distributed-benchmarks/src/backend.rs +++ b/datafusion/distributed-benchmarks/src/backend.rs @@ -5,7 +5,7 @@ use datafusion::error::Result; use datafusion::execution::SessionStateBuilder; use datafusion::prelude::SessionContext; -use crate::datasets::register_tables; +use datafusion_datasets::register_tables; #[async_trait] pub trait BenchmarkBackend: Send + Sync + 'static { diff --git a/datafusion/distributed-benchmarks/src/lib.rs b/datafusion/distributed-benchmarks/src/lib.rs index 345ba9520..41142c668 100644 --- a/datafusion/distributed-benchmarks/src/lib.rs +++ b/datafusion/distributed-benchmarks/src/lib.rs @@ -1,6 +1,5 @@ pub mod backend; pub mod compare; -pub mod datasets; mod local_workers; pub mod results; pub mod run; diff --git a/datafusion/distributed-benchmarks/src/main.rs b/datafusion/distributed-benchmarks/src/main.rs index c633cb1f3..c9af510e5 100644 --- a/datafusion/distributed-benchmarks/src/main.rs +++ b/datafusion/distributed-benchmarks/src/main.rs @@ -1,9 +1,7 @@ //! DataFusion Distributed benchmark runner -mod prepare_clickbench; -mod prepare_tpcds; -mod prepare_tpch; use datafusion::error::Result; +use datafusion_datasets::{prepare_clickbench, prepare_tpcds, prepare_tpch}; use datafusion_distributed_benchmarks::{backend::ParquetBenchmarkBackend, compare, run}; use structopt::StructOpt; diff --git a/datafusion/distributed-benchmarks/src/results.rs b/datafusion/distributed-benchmarks/src/results.rs index b9e926dbb..3d2a8a42a 100644 --- a/datafusion/distributed-benchmarks/src/results.rs +++ b/datafusion/distributed-benchmarks/src/results.rs @@ -101,13 +101,13 @@ impl BenchmarkRun { "=== Comparing {} results from branch '{}' [prev] with '{}' [new] ===", self.dataset, previous.branch, self.branch ); - println!("{header}"); + eprintln!("{header}"); // Print machine information - println!("os: {}", std::env::consts::OS); - println!("arch: {}", std::env::consts::ARCH); - println!("cpu cores: {}", get_available_parallelism()); - println!("workers: {} -> {}", previous.workers, self.workers); - println!("{}", "=".repeat(header.len())); + eprintln!("os: {}", std::env::consts::OS); + eprintln!("arch: {}", std::env::consts::ARCH); + eprintln!("cpu cores: {}", get_available_parallelism()); + eprintln!("workers: {} -> {}", previous.workers, self.workers); + eprintln!("{}", "=".repeat(header.len())); for query in self.results.iter() { let Some(prev_query) = previous.results.iter().find(|v| v.id == query.id) else { continue; @@ -244,15 +244,15 @@ impl BenchResult { let new_err = self.iterations.iter().find_map(|v| v.error.clone()); match (prev_err, new_err) { (Some(_prev_err), None) => { - println!("{}: Previously failed, but now succeeded ๐ŸŸ ", self.id); + eprintln!("{}: Previously failed, but now succeeded ๐ŸŸ ", self.id); return; } (None, Some(_new_err)) => { - println!("{}: Previously succeeded, but now failed โŒ", self.id); + eprintln!("{}: Previously succeeded, but now failed โŒ", self.id); return; } (Some(_prev_err), Some(_new_err)) => { - println!("{}: Previously failed, and now also failed โŒ", self.id); + eprintln!("{}: Previously failed, and now also failed โŒ", self.id); return; } (None, None) => {} @@ -267,7 +267,7 @@ impl BenchResult { let f = p50 as f64 / p50_prev as f64; (f, "slower", if f > 1.2 { "โŒ" } else { "โœ–" }) }; - println!( + eprintln!( "{:>8}: prev={p50_prev:>4} ms, new={p50:>4} ms, diff={f:.2} {tag} {emoji}", self.id ); @@ -324,7 +324,7 @@ pub fn print_comparison_total(base: &[BenchResult], new: &[BenchResult]) { let f = total_new as f64 / total_prev.max(1) as f64; (f, "slower", if f > 1.2 { "โŒ" } else { "โœ–" }) }; - println!( + eprintln!( "{:>8}: prev={total_prev} ms, new={total_new} ms, diff={f:.2} {tag} {emoji}", "TOTAL" ); @@ -336,9 +336,9 @@ pub fn print_comparison_total(base: &[BenchResult], new: &[BenchResult]) { fn print_q_error_comparison(label: &str, prev: Vec, new: Vec) { match (median(prev), median(new)) { - (Some(prev), Some(new)) => println!("{label:>8}: prev={prev:.2}x, new={new:.2}x"), - (Some(prev), None) => println!("{label:>8}: prev={prev:.2}x, new=n/a"), - (None, Some(new)) => println!("{label:>8}: prev=n/a, new={new:.2}x"), + (Some(prev), Some(new)) => eprintln!("{label:>8}: prev={prev:.2}x, new={new:.2}x"), + (Some(prev), None) => eprintln!("{label:>8}: prev={prev:.2}x, new=n/a"), + (None, Some(new)) => eprintln!("{label:>8}: prev=n/a, new={new:.2}x"), (None, None) => {} } } diff --git a/datafusion/distributed-benchmarks/src/run.rs b/datafusion/distributed-benchmarks/src/run.rs index ba1957e8b..1a78d4f21 100644 --- a/datafusion/distributed-benchmarks/src/run.rs +++ b/datafusion/distributed-benchmarks/src/run.rs @@ -16,30 +16,30 @@ // under the License. use crate::backend::BenchmarkBackend; -use crate::datasets::{clickbench, tpcds, tpch}; use crate::local_workers::LocalWorkers; use crate::results::{BenchResult, BenchmarkRun, QueryIter, dataset_path}; use crate::stats::stats_estimation_q_error; -use datafusion::arrow::ipc::CompressionType; use datafusion::arrow::record_batch::RecordBatch; use datafusion::common::instant::Instant; use datafusion::common::tree_node::{Transformed, TreeNode}; -use datafusion::common::utils::get_available_parallelism; -use datafusion::common::{config_err, exec_err, not_impl_err}; +use datafusion::common::{config_datafusion_err, config_err, exec_err, not_impl_err}; use datafusion::datasource::source::DataSourceExec; use datafusion::error::{DataFusionError, Result}; use datafusion::execution::SessionStateBuilder; use datafusion::physical_plan::{ExecutionPlan, collect}; use datafusion::prelude::*; +use datafusion_datasets::{clickbench, tpcds, tpch}; use datafusion_distributed::test_utils::localhost::LocalHostWorkerResolver; use datafusion_distributed::test_utils::work_unit_file_scan::{ WorkUnitFileScanCodec, WorkUnitFileScanConfig, WorkUnitFileScanRule, work_unit_file_scan_desired_task_count, work_unit_file_scan_scale_up_leaf_node, }; use datafusion_distributed::{ - DistributedExt, DistributedMetricsFormat, NetworkBoundaryExt, SessionStateBuilderExt, - display_plan_ascii, rewrite_distributed_plan_with_metrics, + DistributedConfig, DistributedExt, DistributedMetricsFormat, NetworkBoundaryExt, + SessionStateBuilderExt, display_plan_ascii, generate_plan_dump, + rewrite_distributed_plan_with_metrics, }; +use serde_yaml_ng::{Mapping, Value}; use std::fs; use std::path::{Path, PathBuf}; use std::sync::Arc; @@ -72,42 +72,9 @@ pub struct RunOpt { #[structopt(long, default_value = "0")] workers: usize, - /// Number of bytes each partition is expected to scan from parquet files. Lower values - /// produce more partitions/tasks. Defaults to the engine default when unset. - #[structopt(long)] - file_scan_config_bytes_per_partition: Option, - - /// Task count scale factor for when nodes in stages change the cardinality of the data - #[structopt(long)] - cardinality_task_sf: Option, - - /// Collects metrics across network boundaries - #[structopt(long)] - collect_metrics: bool, - - /// Collects metrics across network boundaries - #[structopt(long, default_value = "lz4")] - compression: String, - - /// Sets the limits of tasks for each stage - #[structopt(long, default_value = "0")] - max_tasks_per_stage: usize, - - /// Activate dynamic task count - #[structopt(long)] - dynamic: bool, - /// Number of iterations of each test run - #[structopt(short = "i", long = "iterations", default_value = "5")] - iterations: usize, - - /// Number of partitions to process in parallel. Defaults to the number of available cores. - #[structopt(short = "n", long = "partitions")] - partitions: Option, - - /// Batch size when reading CSV or Parquet files - #[structopt(short = "s", long = "batch-size")] - batch_size: Option, + #[structopt(short = "i", long = "iterations")] + iterations: Option, /// Activate debug mode to see more details #[structopt(short, long)] @@ -119,13 +86,17 @@ pub struct RunOpt { #[structopt(long = "work-unit-file-scan")] work_unit_file_scan: bool, - /// Fan-out threshold for choosing between Direct and TwoPhase shuffle modes. In Direct mode - /// the producer hashes into `consumer_task_count ร— consumer_partition_count` global buckets; - /// TwoPhase mode is used when that product meets or exceeds this value, reducing the - /// producer-side hash to `consumer_task_count` buckets and adding a local RepartitionExec - /// on the consumer side. - #[structopt(long = "two-step-shuffle-fanout-threshold", default_value = "128")] - two_step_shuffle_fanout_threshold: usize, + /// YAML file with DataFusion and distributed config options as flat `key: value` pairs, such + /// as `datafusion.execution.target_partitions: 8` or `distributed.broadcast_joins: true`. + /// Defaults to `.df-bench.yaml` in the current directory, which is skipped if it doesn't exist. + #[structopt(long)] + config_path: Option, + + /// A DataFusion or distributed config option as `key=value`, such as + /// `-c distributed.dynamic_task_count=true`. Can be repeated, and takes precedence over the + /// values in the config file. + #[structopt(short = "c", long = "config", parse(try_from_str = parse_config_entry))] + config: Vec<(String, String)>, } /// The query suite of a dataset path such as `tpch_parquet/scale_factor=1/...`: the benchmark @@ -161,14 +132,54 @@ fn queries_for_dataset(dataset: &str) -> Result, DataFusio } } +/// The default `--config-path`, relative to the current directory. +const DEFAULT_CONFIG_PATH: &str = ".df-bench.yaml"; + +fn parse_config_entry(entry: &str) -> Result<(String, String), String> { + let (key, value) = entry + .split_once('=') + .ok_or_else(|| format!("expected `key=value`, got `{entry}`"))?; + Ok((key.trim().to_string(), value.trim().to_string())) +} + +/// Reads the flat `key: value` config entries of a YAML file, in file order. +fn read_config_file(path: &Path) -> Result> { + let yaml = fs::read_to_string(path) + .map_err(|e| config_datafusion_err!("Cannot read {}: {e}", path.display()))?; + // A file with only comments is a null document. + let entries: Option = serde_yaml_ng::from_str(&yaml) + .map_err(|e| config_datafusion_err!("Invalid YAML in {}: {e}", path.display()))?; + entries + .unwrap_or_default() + .into_iter() + .map(|(key, value)| { + let Value::String(key) = key else { + return config_err!("Config keys in {} must be strings", path.display()); + }; + let value = match value { + Value::Bool(v) => v.to_string(), + Value::Number(v) => v.to_string(), + Value::String(v) => v, + v => return config_err!("Config value of {key} must be a scalar, got {v:?}"), + }; + Ok((key, value)) + }) + .collect() +} + impl RunOpt { - fn config(&self) -> Result { - SessionConfig::from_env().map(|mut config| { - if let Some(batch_size) = self.batch_size { - config = config.with_batch_size(batch_size); + /// The config entries of the config file followed by the `--config` ones, so that applying + /// them in order makes the `--config` ones take precedence. + fn config_entries(&self) -> Result> { + let mut entries = match &self.config_path { + Some(path) => read_config_file(path)?, + None if Path::new(DEFAULT_CONFIG_PATH).exists() => { + read_config_file(Path::new(DEFAULT_CONFIG_PATH))? } - config.with_target_partitions(self.partitions.unwrap_or_else(get_available_parallelism)) - }) + None => vec![], + }; + entries.extend(self.config.iter().cloned()); + Ok(entries) } pub async fn run(self, mut backend: B) -> Result<()> { @@ -185,30 +196,11 @@ impl RunOpt { ) -> Result<()> { let mut builder = SessionStateBuilder::new() .with_default_features() - .with_config(self.config()?) + .with_config(SessionConfig::from_env()?) .with_distributed_worker_resolver(LocalHostWorkerResolver::new( local_workers.ports.to_vec(), )) .with_distributed_planner() - .with_distributed_cardinality_effect_task_scale_factor( - self.cardinality_task_sf.unwrap_or(1.0), - )? - .with_distributed_dynamic_task_count(self.dynamic)? - .with_distributed_compression(match self.compression.as_str() { - "zstd" => Some(CompressionType::ZSTD), - "lz4" => Some(CompressionType::LZ4_FRAME), - "none" => None, - v => return config_err!("Unknown compression type {v}"), - })? - .with_distributed_children_isolator_unions(true)? - .with_distributed_broadcast_joins(true)? - .with_distributed_two_step_shuffle_fanout_threshold( - self.two_step_shuffle_fanout_threshold, - )? - .with_distributed_metrics_collection( - self.collect_metrics || self.debug || self.dynamic, - )? - .with_distributed_max_tasks_per_stage(self.max_tasks_per_stage)? .with_distributed_user_codec(WorkUnitFileScanCodec) .with_distributed_desired_task_count_handler(work_unit_file_scan_desired_task_count) .with_distributed_scale_up_leaf_node_handler(work_unit_file_scan_scale_up_leaf_node) @@ -216,17 +208,20 @@ impl RunOpt { dse.data_source() .downcast_ref::() .map(|v| &v.feed) - }); + }) + .with_distributed_metrics_collection(true)?; + + let config_entries = self.config_entries()?; + let options = builder.config().get_or_insert_default().options_mut(); + for (key, value) in &config_entries { + options.set(key, value)?; + } if let Some(local_worker) = local_workers.workers.first() { let url = Url::parse(&format!("http://127.0.0.1:{}", local_workers.ports[0])).unwrap(); builder.set_distributed_local_worker_context(local_worker.to_local_worker_context(url)) } - if let Some(bytes_per_partition) = self.file_scan_config_bytes_per_partition { - builder.set_distributed_file_scan_config_bytes_per_partition(bytes_per_partition)?; - } - if self.work_unit_file_scan { builder = builder.with_physical_optimizer_rule(Arc::new(WorkUnitFileScanRule)) } @@ -236,7 +231,8 @@ impl RunOpt { backend.register_tables(&ctx, &self.get_path()?).await?; let dataset_suite = dataset_suite(&self.dataset); - println!("Running benchmarks with the following options: {self:?}"); + eprintln!("Running benchmarks with the following options: {self:?}"); + eprintln!("Config: {config_entries:?}"); let mut benchmark_run = BenchmarkRun::new(self.dataset.clone(), local_workers.ports.len()); for (id, sql) in queries_for_dataset(dataset_suite)? { @@ -262,13 +258,22 @@ impl RunOpt { sql: &str, ctx: &SessionContext, ) -> Result { + let dynamic = + DistributedConfig::from_session_config(&ctx.copied_config())?.dynamic_task_count; let mut bench_query = BenchResult { id: id.to_string(), dataset: self.dataset.clone(), iterations: vec![], }; + let mut iterations = 5; // Default to 5 iterations. + if self.debug { + iterations = 1; + } + if let Some(value) = self.iterations { + iterations = value + } - 'outer: for i in 0..self.iterations { + 'outer: for i in 0..iterations { let start = Instant::now(); for query in sql.split(";").map(|v| v.trim()) { @@ -284,7 +289,7 @@ impl RunOpt { let elapsed = start.elapsed(); let ms = elapsed.as_secs_f64() * 1000.0; let row_count = result.iter().map(|b| b.num_rows()).sum(); - let physical_plan = if self.dynamic || self.debug { + let physical_plan = if self.debug || dynamic { rewrite_distributed_plan_with_metrics( physical_plan, DistributedMetricsFormat::PerTask, @@ -293,25 +298,26 @@ impl RunOpt { } else { physical_plan }; - let stats_q_error = match self.dynamic { + let stats_q_error = match dynamic { true => stats_estimation_q_error(&physical_plan), false => None, }; if let Some(q_error) = stats_q_error { - println!( + eprintln!( "Query {id} iteration {i} took {ms:.1} ms, stats q-error P50 {:.2}x, P95 {:.2}x and returned {row_count} rows", q_error.p50, q_error.p95 ); } else { - println!( + eprintln!( "Query {id} iteration {i} took {ms:.1} ms and returned {row_count} rows" ); } if self.debug { - println!( + eprintln!( "=== Physical plan with metrics ===\n{}\n", display_plan_ascii(physical_plan.as_ref(), true) ); + println!("{}", generate_plan_dump(&physical_plan)?); } bench_query.iterations.push(QueryIter { @@ -324,7 +330,7 @@ impl RunOpt { }); } Err(err) => { - println!("Query {id} iteration {i} failed: {err}"); + eprintln!("Query {id} iteration {i} failed: {err}"); bench_query.iterations.push(QueryIter { elapsed: Duration::from_millis(0), row_count: 0, @@ -338,7 +344,7 @@ impl RunOpt { } } } - println!("Query {id} p50 time: {} ms", bench_query.p50()); + eprintln!("Query {id} p50 time: {} ms", bench_query.p50()); Ok(bench_query) } diff --git a/datafusion/distributed-cli/Cargo.toml b/datafusion/distributed-cli/Cargo.toml index 0f2206794..4084915df 100644 --- a/datafusion/distributed-cli/Cargo.toml +++ b/datafusion/distributed-cli/Cargo.toml @@ -2,7 +2,7 @@ name = "datafusion-distributed-cli" version = "0.1.0" edition = "2024" -description = "DataFusion's CLI with distributed execution capabilities" +description = "DataFusion's CLI with distributed execution capabilities, and a viewer for DataFusion Distributed plan dumps" license = "Apache-2.0" documentation = "https://datafusion-contrib.github.io/datafusion-distributed/" repository = "https://github.com/datafusion-contrib/datafusion-distributed" @@ -14,6 +14,7 @@ datafusion-distributed = { path = "../distributed", version = "4.0.0", features datafusion-cli = { version = "55", default-features = false } tokio = { version = "1.48", features = ["full"] } clap = { version = "4", features = ["derive"] } +prost = { version = "0.14", features = ["no-recursion-limit"]} env_logger = "0.11" dirs = "6" diff --git a/datafusion/distributed-cli/README.md b/datafusion/distributed-cli/README.md index 99bca2889..81ac16479 100644 --- a/datafusion/distributed-cli/README.md +++ b/datafusion/distributed-cli/README.md @@ -39,3 +39,59 @@ SET distributed.files_per_task = 1; The usage is exactly the same as the original CLI: https://datafusion.apache.org/user-guide/cli/usage.html + +## Displaying plan dumps + +`datafusion_distributed::generate_plan_dump` encodes an executed plan, with its metrics, in a +string that can be logged. `dfd plan` displays it as the tree of stages: + +```bash +dfd plan +``` + +and `--stage` displays the nodes of one stage with their metrics, where stage 0 is the head of the +plan: + +```bash +dfd plan --stage 2 +``` + +Metrics are added up across all tasks, like `output_rows=17`. `--verbose` displays them per task +instead, like `output_rows={0:10, 1:7}`, which is longer but shows how the work was spread: + +```bash +dfd plan --stage 2 --verbose +``` + +`--metric` displays only the metrics whose name contains a text, and can be repeated to display the +metrics matching any of them, like the bytes and rows of every node: + +```bash +dfd plan --stage 2 --metric bytes --metric rows +``` + +Like `cargo test -- --exact` does with test names, `--exact` matches whole metric names instead, so +`output_bytes` does not display `estimated_output_bytes` too: + +```bash +dfd plan --metric output_bytes --exact +``` + +Besides their own metrics, stages show the `elapsed_compute` of all their nodes, and, when their +output was estimated for choosing task counts dynamically, `output_bytes_q_error`: the ratio between +the estimated and the actual bytes they returned, the larger over the smaller. + +The argument can also be the path of a file containing the plan dump: + +```bash +dfd plan plan_dump.txt --stage 2 +``` + +Without an argument, the plan dump is read from stdin: + +```bash +pbpaste | dfd plan --stage 2 +``` + +The output is colored when printed to a terminal, unless `NO_COLOR` is set. `FORCE_COLOR=1` keeps +the colors through a pipe, e.g. into `less -R`. diff --git a/datafusion/distributed-cli/src/main.rs b/datafusion/distributed-cli/src/main.rs index 7b51224e9..0d4cdc57d 100644 --- a/datafusion/distributed-cli/src/main.rs +++ b/datafusion/distributed-cli/src/main.rs @@ -1,6 +1,7 @@ use clap::{Parser, Subcommand}; use std::process::ExitCode; +mod plan; mod query; #[derive(Debug, Parser, PartialEq)] @@ -15,6 +16,8 @@ enum Command { /// Run queries in an interactive shell, or from commands and files, like datafusion-cli does, /// but distributed across in-memory workers. Query(query::Args), + /// Display a plan dump produced by `datafusion_distributed::generate_plan_dump`. + Plan(plan::Args), } #[tokio::main] @@ -23,6 +26,7 @@ pub async fn main() -> ExitCode { env_logger::init(); let result = match Args::parse().command { Command::Query(args) => query::run(args).await, + Command::Plan(args) => plan::run(args), }; if let Err(e) = result { println!("Error: {e}"); diff --git a/datafusion/distributed-cli/src/plan.rs b/datafusion/distributed-cli/src/plan.rs new file mode 100644 index 000000000..9382c5006 --- /dev/null +++ b/datafusion/distributed-cli/src/plan.rs @@ -0,0 +1,74 @@ +use datafusion::common::{exec_datafusion_err, exec_err}; +use datafusion::error::Result; +use datafusion_distributed::{DisplayPlanDumpOptions, DistributedMetricsFormat, display_plan_dump}; +use std::io::{self, IsTerminal}; +use std::path::Path; +use std::{env, fs}; + +#[derive(Debug, clap::Args, PartialEq)] +pub(super) struct Args { + /// The plan dump, as produced by `datafusion_distributed::generate_plan_dump`, or the path of a + /// file containing it. If not given, it is read from stdin. + dump: Option, + + /// Display the nodes of this stage with their metrics instead of the tree of stages. Stage 0 is + /// the head of the plan, run by the coordinator. + #[clap(short, long)] + stage: Option, + + /// Enable verbose display mode. In this mode, metrics are displayed per-task, rather than + /// globally. + #[clap(short, long)] + verbose: bool, + + /// Display only the metrics whose name contains this text. Repeat it to display the metrics + /// matching any of them, like `-m bytes -m rows`. + #[clap(short, long = "metric", value_name = "TEXT")] + metrics: Vec, + + /// Match the `--metric` filters with whole metric names, like `cargo test -- --exact` does with + /// test names, so `-m output_bytes --exact` leaves out `estimated_output_bytes`. + #[clap(long)] + exact: bool, +} + +/// Displays a plan dump. +pub(super) fn run(args: Args) -> Result<()> { + let dump = match args.dump { + // Plan dumps are base64, so arguments with other characters, like `.`, are paths too. + Some(path) if Path::new(&path).is_file() || !is_base64(&path) => fs::read_to_string(&path) + .map_err(|err| exec_datafusion_err!("Failed to read the plan dump in {path}: {err}"))?, + Some(dump) => dump, + None if io::stdin().is_terminal() => { + return exec_err!("Pass a plan dump as an argument, or pipe it through stdin"); + } + None => io::read_to_string(io::stdin())?, + }; + let mut opts = DisplayPlanDumpOptions::default().with_color(color_enabled()); + if let Some(stage) = args.stage { + opts = opts.with_stage_id(stage); + } + + opts = opts.with_metrics_format(match args.verbose { + true => DistributedMetricsFormat::PerTask, + false => DistributedMetricsFormat::Aggregated, + }); + for filter in args.metrics { + opts = opts.with_metric_filter(filter); + } + opts = opts.with_exact_metric_filters(args.exact); + print!("{}", display_plan_dump(dump, opts)?); + Ok(()) +} + +fn is_base64(text: &str) -> bool { + (text.trim().bytes()).all(|byte| byte.is_ascii_alphanumeric() || b"+/=".contains(&byte)) +} + +/// Output is colored when printed to a terminal, unless `NO_COLOR` is set. `FORCE_COLOR` keeps +/// colors when piping the output, e.g. into `less -R`. +fn color_enabled() -> bool { + let force = env::var("FORCE_COLOR").is_ok_and(|value| !value.is_empty() && value != "0"); + let disable = env::var_os("NO_COLOR").is_some_and(|value| !value.is_empty()); + force || (!disable && io::stdout().is_terminal()) +} diff --git a/datafusion/distributed-console/Cargo.toml b/datafusion/distributed-console/Cargo.toml index 900ee22b1..5f086d1f1 100644 --- a/datafusion/distributed-console/Cargo.toml +++ b/datafusion/distributed-console/Cargo.toml @@ -19,6 +19,6 @@ tokio-stream = "0.1.18" [dev-dependencies] arrow = "59" async-trait = "0.1.89" -datafusion-distributed-benchmarks = { path = "../distributed-benchmarks" } +datafusion-datasets = { path = "../datasets" } futures = "0.3.31" url = "2.5.7" diff --git a/datafusion/distributed-console/examples/tpcds_runner.rs b/datafusion/distributed-console/examples/tpcds_runner.rs index 22fb676ba..298d26f20 100644 --- a/datafusion/distributed-console/examples/tpcds_runner.rs +++ b/datafusion/distributed-console/examples/tpcds_runner.rs @@ -4,13 +4,13 @@ use datafusion::common::runtime::JoinSet; use datafusion::execution::SessionStateBuilder; use datafusion::physical_plan::execute_stream; use datafusion::prelude::SessionContext; +use datafusion_datasets::{ + output::TESTDATA_ROOT, parquet_layout::ParquetLayout, register_tables, tpcds, +}; use datafusion_distributed::{ DistributedExt, DistributedMetricsFormat, SessionStateBuilderExt, WorkerResolver, display_plan_ascii, }; -use datafusion_distributed_benchmarks::datasets::{ - output::TESTDATA_ROOT, parquet_layout::ParquetLayout, register_tables, tpcds, -}; use futures::TryStreamExt; use std::error::Error; use std::num::NonZeroUsize; diff --git a/datafusion/distributed-iceberg-benchmarks/Cargo.toml b/datafusion/distributed-iceberg-benchmarks/Cargo.toml index 0048a859c..a230a23a2 100644 --- a/datafusion/distributed-iceberg-benchmarks/Cargo.toml +++ b/datafusion/distributed-iceberg-benchmarks/Cargo.toml @@ -8,6 +8,7 @@ publish = false [dependencies] async-trait = "0.1.89" datafusion = { workspace = true, features = ["parquet", "sql"] } +datafusion-datasets = { path = "../datasets" } datafusion-distributed-benchmarks = { path = "../distributed-benchmarks" } datafusion-distributed-iceberg = { path = "../distributed-iceberg" } env_logger = "0.11.8" diff --git a/datafusion/distributed-iceberg-benchmarks/README.md b/datafusion/distributed-iceberg-benchmarks/README.md index 963056d27..66ab27d62 100644 --- a/datafusion/distributed-iceberg-benchmarks/README.md +++ b/datafusion/distributed-iceberg-benchmarks/README.md @@ -11,7 +11,7 @@ DATASET=$(cargo run -p datafusion-distributed-benchmarks --release -- prepare-tp ICEBERG=$(cargo run -p datafusion-distributed-iceberg-benchmarks --release -- prepare \ --input $DATASET) cargo run -p datafusion-distributed-iceberg-benchmarks --release -- run \ - --workers 2 --dataset $ICEBERG --partitions 2 + --workers 2 --dataset $ICEBERG -c datafusion.execution.target_partitions=2 ``` Preparation writes an immutable `tpch_iceberg` dataset by default, whose path repeats the input's diff --git a/datafusion/distributed-iceberg-benchmarks/src/dataset.rs b/datafusion/distributed-iceberg-benchmarks/src/dataset.rs index 5ce03d59b..290811030 100644 --- a/datafusion/distributed-iceberg-benchmarks/src/dataset.rs +++ b/datafusion/distributed-iceberg-benchmarks/src/dataset.rs @@ -9,7 +9,7 @@ use datafusion::common::{DataFusionError, exec_datafusion_err, exec_err}; use datafusion::execution::context::SessionContext; use datafusion::physical_plan::{ExecutionPlan, ExecutionPlanProperties}; use datafusion::prelude::ParquetReadOptions; -use datafusion_distributed_benchmarks::datasets::output::DatasetOutput; +use datafusion_datasets::output::DatasetOutput; use datafusion_distributed_iceberg::IcebergIntegrationOptions; use datafusion_distributed_iceberg::iceberg; use futures::StreamExt; diff --git a/datafusion/distributed-iceberg-benchmarks/src/prepare.rs b/datafusion/distributed-iceberg-benchmarks/src/prepare.rs index c7ee26b62..676c173c8 100644 --- a/datafusion/distributed-iceberg-benchmarks/src/prepare.rs +++ b/datafusion/distributed-iceberg-benchmarks/src/prepare.rs @@ -2,7 +2,7 @@ use std::path::PathBuf; use super::dataset::{convert_parquet_to_iceberg, output_path}; use datafusion::common::{Result, exec_datafusion_err}; -use datafusion_distributed_benchmarks::datasets::output::DatasetOutput; +use datafusion_datasets::output::DatasetOutput; use structopt::StructOpt; /// Convert local Parquet tables into an Iceberg dataset. diff --git a/datafusion/distributed-iceberg-test-tools/src/append.rs b/datafusion/distributed-iceberg-test-tools/src/append.rs index e209b658e..1fc7dec75 100644 --- a/datafusion/distributed-iceberg-test-tools/src/append.rs +++ b/datafusion/distributed-iceberg-test-tools/src/append.rs @@ -2,7 +2,7 @@ use std::collections::HashMap; use std::sync::Arc; use std::time::{SystemTime, UNIX_EPOCH}; -use datafusion::arrow::array::{RecordBatch, new_null_array}; +use datafusion::arrow::array::{Int64Array, RecordBatch, StringArray, new_null_array}; use datafusion::arrow::compute::cast; use datafusion::arrow::datatypes::SchemaRef as ArrowSchemaRef; use datafusion::common::{exec_err, not_impl_err}; @@ -11,9 +11,12 @@ use datafusion::parquet::file::properties::WriterProperties; use iceberg::arrow::{RecordBatchPartitionSplitter, schema_to_arrow_schema}; use iceberg::spec::{ DataFile, DataFileFormat, FormatVersion, MAIN_BRANCH, ManifestListWriter, - ManifestWriterBuilder, Operation, Snapshot, SnapshotSummaryCollector, Summary, + ManifestWriterBuilder, Operation, PartitionKey, Snapshot, SnapshotSummaryCollector, Summary, }; use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; +use iceberg::writer::base_writer::position_delete_writer::{ + PositionDeleteFileWriterBuilder, position_delete_schema, +}; use iceberg::writer::file_writer::ParquetWriterBuilder; use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, @@ -24,18 +27,18 @@ use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use crate::FixtureTable; use crate::error::df_err; -/// One commit that adds data files, started by [`FixtureTable::append`]. +/// One commit that adds data or position delete files, started by [`FixtureTable::append`]. /// /// Each [`Append::file`] becomes exactly one Parquet data file, so tests /// control how rows are laid out across files. [`Append::commit`] adds them /// all in one snapshot on top of the current one. Snapshots are numbered 1, /// 2, 3, ..., unlike iceberg-rust's `fast_append`, which picks random ids, so -/// that tests can show snapshot ids. Only adds data files: no deletes, and -/// format version 2 only. +/// that tests can show snapshot ids. Supports format version 2 only. pub struct Append<'a> { table: &'a mut FixtureTable, writer_properties: WriterProperties, files: Vec, + position_deletes: Vec<(DataFile, Vec)>, } impl<'a> Append<'a> { @@ -44,6 +47,7 @@ impl<'a> Append<'a> { table, writer_properties: WriterProperties::default(), files: Vec::new(), + position_deletes: Vec::new(), } } @@ -63,8 +67,19 @@ impl<'a> Append<'a> { self } - /// Writes the data files and commits them as a new snapshot of the main - /// branch. Returns the data files in the order they were added. + /// Adds a position delete file for the given data file's zero-based row positions. + pub fn position_deletes( + mut self, + file: &DataFile, + positions: impl IntoIterator, + ) -> Self { + self.position_deletes + .push((file.clone(), positions.into_iter().collect())); + self + } + + /// Writes the data and delete files and commits them as a new snapshot of the + /// main branch. Returns only the data files, in the order they were added. pub async fn commit(self) -> Result> { let table = self.table.table(); let metadata = table.metadata(); @@ -125,31 +140,80 @@ impl<'a> Append<'a> { data_files.extend(writer.close().await.map_err(df_err)?); } - let mut manifest_writer = ManifestWriterBuilder::new( - file_io - .new_output(format!( - "{}/metadata/snap-{snapshot_id}-m0.avro", - metadata.location() - )) - .map_err(df_err)?, - Some(snapshot_id), - schema.clone(), - partition_spec.as_ref().clone(), - ) - .build_v2_data(); - let mut summary = SnapshotSummaryCollector::default(); - for data_file in &data_files { - manifest_writer - .add_file(data_file.clone(), sequence_number) + let mut delete_files = Vec::new(); + let delete_schema = position_delete_schema(); + let delete_writer_builder = + PositionDeleteFileWriterBuilder::new(RollingFileWriterBuilder::new( + ParquetWriterBuilder::new(WriterProperties::default(), delete_schema.clone()), + usize::MAX, + file_io.clone(), + DefaultLocationGenerator::new(metadata).map_err(df_err)?, + DefaultFileNameGenerator::new( + format!("{snapshot_id}-deletes"), + None, + DataFileFormat::Parquet, + ), + )); + for (file, mut positions) in self.position_deletes { + positions.sort_unstable(); + positions.dedup(); + let batch = RecordBatch::try_new( + Arc::new(schema_to_arrow_schema(&delete_schema).map_err(df_err)?), + vec![ + Arc::new(StringArray::from(vec![file.file_path(); positions.len()])), + Arc::new(Int64Array::from(positions)), + ], + )?; + let partition_key = PartitionKey::new( + partition_spec.as_ref().clone(), + schema.clone(), + file.partition().clone(), + ); + let mut writer = delete_writer_builder + .build(Some(partition_key)) + .await .map_err(df_err)?; - summary.add_file(data_file, schema.clone(), partition_spec.clone()); + writer.write(batch).await.map_err(df_err)?; + delete_files.extend(writer.close().await.map_err(df_err)?); + } + + let mut manifests = Vec::new(); + let mut summary = SnapshotSummaryCollector::default(); + for (name, files, is_delete) in + [("m0", &data_files, false), ("deletes", &delete_files, true)] + { + if files.is_empty() { + continue; + } + let builder = ManifestWriterBuilder::new( + file_io + .new_output(format!( + "{}/metadata/snap-{snapshot_id}-{name}.avro", + metadata.location() + )) + .map_err(df_err)?, + Some(snapshot_id), + schema.clone(), + partition_spec.as_ref().clone(), + ); + let mut manifest_writer = if is_delete { + builder.build_v2_deletes() + } else { + builder.build_v2_data() + }; + for data_file in files { + manifest_writer + .add_file(data_file.clone(), sequence_number) + .map_err(df_err)?; + summary.add_file(data_file, schema.clone(), partition_spec.clone()); + } + manifests.push( + manifest_writer + .write_manifest_file() + .await + .map_err(df_err)?, + ); } - let mut manifests = vec![ - manifest_writer - .write_manifest_file() - .await - .map_err(df_err)?, - ]; if let Some(parent) = metadata.current_snapshot() { let parent_manifests = table .manifest_list_reader(parent) @@ -183,7 +247,11 @@ impl<'a> Append<'a> { .with_timestamp_ms(now_ms()) .with_manifest_list(manifest_list) .with_summary(Summary { - operation: Operation::Append, + operation: if delete_files.is_empty() { + Operation::Append + } else { + Operation::Overwrite + }, additional_properties: with_totals( summary.build(), metadata.current_snapshot().map(|parent| parent.summary()), diff --git a/datafusion/distributed-iceberg-test-tools/src/table.rs b/datafusion/distributed-iceberg-test-tools/src/table.rs index 5e5f32c58..f1ea9bc94 100644 --- a/datafusion/distributed-iceberg-test-tools/src/table.rs +++ b/datafusion/distributed-iceberg-test-tools/src/table.rs @@ -165,7 +165,7 @@ impl FixtureTable { self.table.metadata_location_result().map_err(df_err) } - /// Starts a commit that appends data files. + /// Starts a commit that adds data or position delete files. pub fn append(&mut self) -> Append<'_> { Append::new(self) } diff --git a/datafusion/distributed-iceberg/README.md b/datafusion/distributed-iceberg/README.md index 5d5c5a4c9..9c5b6507b 100644 --- a/datafusion/distributed-iceberg/README.md +++ b/datafusion/distributed-iceberg/README.md @@ -90,6 +90,14 @@ statistics and task-size estimates. Selected-file column statistics remain a separate change. Task-local eager variants and decoded plans report unknown statistics rather than applying whole-snapshot aggregates to a subset. +Both scans keep the filters DataFusion pushes into them, including the dynamic +filters that joins, top-N sorts and aggregates compute while the query runs. Scans +return `PushedDown::No` for these filters, so DataFusion still checks every row. A join +or aggregate only computes a dynamic filter when a scan below it is a consumer of that +filter. Enable `iceberg.dynamic_filtering_enabled` to make Iceberg scans consumers of the +dynamic filters pushed into them. With it on, `EXPLAIN` shows them on the scan as +`dynamic_filters=`. Scans don't skip data with these filters yet. + ```bash cargo test -p datafusion-distributed-iceberg -p datafusion-distributed-iceberg-test-tools ``` diff --git a/datafusion/distributed-iceberg/src/common/dynamic_filters/mod.rs b/datafusion/distributed-iceberg/src/common/dynamic_filters/mod.rs new file mode 100644 index 000000000..7cc259ed0 --- /dev/null +++ b/datafusion/distributed-iceberg/src/common/dynamic_filters/mod.rs @@ -0,0 +1,3 @@ +pub(crate) mod partition_filter; +pub(crate) mod physical_predicate; +pub(crate) mod pruner; diff --git a/datafusion/distributed-iceberg/src/common/dynamic_filters/partition_filter.rs b/datafusion/distributed-iceberg/src/common/dynamic_filters/partition_filter.rs new file mode 100644 index 000000000..baaf5f1d6 --- /dev/null +++ b/datafusion/distributed-iceberg/src/common/dynamic_filters/partition_filter.rs @@ -0,0 +1,299 @@ +//! A copy of the code iceberg-rust uses to skip data files by their partition values. That +//! code isn't part of iceberg-rust's public API yet. When it becomes public, this file should +//! be deleted and its callers should use the public API. +//! Ref: https://github.com/apache/iceberg-rust/issues/3675 +//! +//! Each part of this file copies one part of iceberg-rust: +//! +//! | This file | iceberg-rust | +//! |------------------------------|--------------------------------------------------------------| +//! | `PartitionFilter::try_new` | `TableScanBuilder::with_filter`, `PartitionFilterCache::get` | +//! | `PartitionFilter::can_match` | `ExpressionEvaluator::eval` | +//! | `project`, `project_column` | `InclusiveProjection` | +//! | `evaluate` | `ExpressionEvaluatorVisitor` | +//! | `partition_value`, `datum` | `StructAccessor::get` | +//! +//! The copy works the same way as iceberg-rust, except for one case in `datum` that's +//! explained there. + +use std::sync::Arc; + +use iceberg::expr::{Bind, BoundPredicate, BoundReference, Predicate, PredicateOperator}; +use iceberg::spec::{ + Datum, Literal, PartitionSpec, PartitionSpecRef, PrimitiveLiteral, PrimitiveType, Schema, + SchemaRef, Struct, Type, +}; +use iceberg::{Error, ErrorKind, Result}; + +/// Checks whether a data file's partition can hold rows that match a filter, as a table scan +/// does when it plans files. +pub(crate) struct PartitionFilter { + /// The filter projected onto the spec's partition columns, bound to its partition schema. + predicate: BoundPredicate, + spec: PartitionSpecRef, +} + +impl PartitionFilter { + /// Creates a partition filter for `filter` on data files written with `partition_spec`. + /// + /// If `partition_spec` uses a column that `schema` no longer has, every partition matches. + /// Returns an error if `filter` doesn't bind to `schema`, or if it can't be projected onto + /// `partition_spec`. + pub(crate) fn try_new( + filter: Predicate, + schema: SchemaRef, + partition_spec: &PartitionSpecRef, + case_sensitive: bool, + ) -> Result { + let filter = filter + .rewrite_not() + .bind(Arc::clone(&schema), case_sensitive)?; + let has_dropped_source_column = partition_spec + .fields() + .iter() + .any(|field| schema.field_by_id(field.source_id).is_none()); + let predicate = if has_dropped_source_column { + BoundPredicate::AlwaysTrue + } else { + let partition_type = partition_spec.partition_type(&schema)?; + let partition_schema = Arc::new( + Schema::builder() + .with_schema_id(partition_spec.spec_id()) + .with_fields(partition_type.fields().to_owned()) + .build()?, + ); + project(&filter, partition_spec)? + .rewrite_not() + .bind(partition_schema, case_sensitive)? + }; + Ok(Self { + predicate, + spec: Arc::clone(partition_spec), + }) + } + + /// Returns whether a data file with `partition` can hold rows that match the filter. + /// Returns an error if `partition` doesn't match the partition spec's fields. + pub(crate) fn can_match(&self, partition: &Struct) -> Result { + evaluate(&self.predicate, &self.spec, partition) + } +} + +/// Rewrites a predicate on table columns into one on the spec's partition columns. The result +/// matches every partition that holds a matching row, and can match more. +fn project(predicate: &BoundPredicate, spec: &PartitionSpec) -> Result { + Ok(match predicate { + BoundPredicate::AlwaysTrue => Predicate::AlwaysTrue, + BoundPredicate::AlwaysFalse => Predicate::AlwaysFalse, + BoundPredicate::And(expr) => { + let [left, right] = expr.inputs(); + project(left, spec)?.and(project(right, spec)?) + } + BoundPredicate::Or(expr) => { + let [left, right] = expr.inputs(); + project(left, spec)?.or(project(right, spec)?) + } + // `PartitionFilter::try_new` removes every NOT first, so none should reach here. If one + // does, this returns an error, as iceberg-rust does. + BoundPredicate::Not(_) => { + return Err(Error::new( + ErrorKind::Unexpected, + "found a NOT that `PartitionFilter::try_new` should have removed", + )); + } + BoundPredicate::Unary(expr) => project_column(expr.term(), predicate, spec)?, + BoundPredicate::Binary(expr) => project_column(expr.term(), predicate, spec)?, + BoundPredicate::Set(expr) => project_column(expr.term(), predicate, spec)?, + }) +} + +/// Projects a predicate on one column through every partition field built from that column. +/// A column with no partition field gives `AlwaysTrue`. +fn project_column( + column: &BoundReference, + predicate: &BoundPredicate, + spec: &PartitionSpec, +) -> Result { + spec.fields() + .iter() + .filter(|field| field.source_id == column.field().id) + .try_fold(Predicate::AlwaysTrue, |projected, field| { + // `None` means the transform can't express the predicate, so it keeps every file. + Ok(match field.transform.project(&field.name, predicate)? { + Some(part) => projected.and(part), + None => projected, + }) + }) +} + +/// Checks whether one data file's partition values pass a filter on the partition columns. +fn evaluate(predicate: &BoundPredicate, spec: &PartitionSpec, partition: &Struct) -> Result { + match predicate { + BoundPredicate::AlwaysTrue => Ok(true), + BoundPredicate::AlwaysFalse => Ok(false), + BoundPredicate::And(expr) => { + let [left, right] = expr.inputs(); + let (left, right) = ( + evaluate(left, spec, partition)?, + evaluate(right, spec, partition)?, + ); + Ok(left && right) + } + BoundPredicate::Or(expr) => { + let [left, right] = expr.inputs(); + let (left, right) = ( + evaluate(left, spec, partition)?, + evaluate(right, spec, partition)?, + ); + Ok(left || right) + } + BoundPredicate::Not(_) => Err(Error::new( + ErrorKind::Unexpected, + "found a NOT that `PartitionFilter::try_new` should have removed", + )), + BoundPredicate::Unary(expr) => { + let value = partition_value(expr.term(), spec, partition)?; + let is_nan = value.as_ref().is_some_and(Datum::is_nan); + Ok(match expr.op() { + PredicateOperator::IsNull => value.is_none(), + PredicateOperator::NotNull => value.is_some(), + PredicateOperator::IsNan => is_nan, + PredicateOperator::NotNan => !is_nan, + _ => true, + }) + } + BoundPredicate::Binary(expr) => { + let Some(value) = partition_value(expr.term(), spec, partition)? else { + return Ok(matches!( + expr.op(), + PredicateOperator::NotEq | PredicateOperator::NotStartsWith + )); + }; + let literal = expr.literal(); + Ok(match expr.op() { + PredicateOperator::StartsWith => starts_with(&value, literal), + PredicateOperator::NotStartsWith => !starts_with(&value, literal), + PredicateOperator::LessThan => &value < literal, + PredicateOperator::LessThanOrEq => &value <= literal, + PredicateOperator::GreaterThan => &value > literal, + PredicateOperator::GreaterThanOrEq => &value >= literal, + PredicateOperator::Eq => &value == literal, + PredicateOperator::NotEq => &value != literal, + _ => true, + }) + } + BoundPredicate::Set(expr) => { + let Some(value) = partition_value(expr.term(), spec, partition)? else { + return Ok(matches!(expr.op(), PredicateOperator::NotIn)); + }; + let contains = expr.literals().contains(&value); + Ok(match expr.op() { + PredicateOperator::In => contains, + PredicateOperator::NotIn => !contains, + _ => true, + }) + } + } +} + +/// Returns the partition value that `column` points to, with the partition column's type. +/// Returns `None` when the value is null. +/// +/// iceberg-rust reads this value with `StructAccessor::get`, which isn't public. So this finds +/// the value's position by looking up its partition field ID in the spec. +fn partition_value( + column: &BoundReference, + spec: &PartitionSpec, + partition: &Struct, +) -> Result> { + let field = column.field(); + let position = spec + .fields() + .iter() + .position(|partition_field| partition_field.field_id == field.id) + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!("partition field {} is not in the spec", field.id), + ) + })?; + let Some(value) = partition.fields().get(position) else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("the partition has no value at {position}"), + )); + }; + let Some(value) = value else { + return Ok(None); + }; + let (Literal::Primitive(literal), Type::Primitive(r#type)) = (value, field.field_type.as_ref()) + else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("partition field {} is not a primitive", field.id), + )); + }; + datum(r#type, literal).map(Some) +} + +/// Builds a `Datum` of type `r#type` from a stored partition value. iceberg-rust uses +/// `Datum::new` for this, which isn't public, so this uses the public constructors. +fn datum(r#type: &PrimitiveType, literal: &PrimitiveLiteral) -> Result { + Ok(match (r#type, literal) { + (PrimitiveType::Boolean, PrimitiveLiteral::Boolean(value)) => Datum::bool(*value), + (PrimitiveType::Int, PrimitiveLiteral::Int(value)) => Datum::int(*value), + (PrimitiveType::Long, PrimitiveLiteral::Long(value)) => Datum::long(*value), + (PrimitiveType::Float, PrimitiveLiteral::Float(value)) => Datum::float(value.0), + (PrimitiveType::Double, PrimitiveLiteral::Double(value)) => Datum::double(value.0), + (PrimitiveType::Date, PrimitiveLiteral::Int(value)) => Datum::date(*value), + (PrimitiveType::Time, PrimitiveLiteral::Long(value)) => Datum::time_micros(*value)?, + (PrimitiveType::Timestamp, PrimitiveLiteral::Long(value)) => { + Datum::timestamp_micros(*value) + } + (PrimitiveType::Timestamptz, PrimitiveLiteral::Long(value)) => { + Datum::timestamptz_micros(*value) + } + (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(value)) => { + Datum::timestamp_nanos(*value) + } + (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(value)) => { + Datum::timestamptz_nanos(*value) + } + (PrimitiveType::String, PrimitiveLiteral::String(value)) => Datum::string(value), + (PrimitiveType::Binary, PrimitiveLiteral::Binary(value)) => { + Datum::binary(value.iter().copied()) + } + (PrimitiveType::Fixed(_), PrimitiveLiteral::Binary(value)) => { + Datum::fixed(value.iter().copied()) + } + // Decimal and uuid have no public constructor that takes the raw value, so these go + // through their byte encoding. + (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(value)) => { + Datum::try_from_bytes(&value.to_be_bytes(), r#type.clone())? + } + (PrimitiveType::Uuid, PrimitiveLiteral::UInt128(value)) => { + Datum::try_from_bytes(&value.to_be_bytes(), PrimitiveType::Uuid)? + } + // The stored value doesn't fit its column's type. This happens in files written before + // a column was widened, for example from int to long. iceberg-rust builds the value + // anyway. None of its comparisons then match, so it skips the file. The public + // constructors can't build a value like that, so we return an error and don't skip + // the file. + _ => { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("can't read a {type} partition value"), + )); + } + }) +} + +/// Returns whether `value` starts with `prefix` +fn starts_with(value: &Datum, prefix: &Datum) -> bool { + match (value.literal(), prefix.literal()) { + (PrimitiveLiteral::String(value), PrimitiveLiteral::String(prefix)) => { + value.starts_with(prefix.as_str()) + } + _ => false, + } +} diff --git a/datafusion/distributed-iceberg/src/common/dynamic_filters/physical_predicate.rs b/datafusion/distributed-iceberg/src/common/dynamic_filters/physical_predicate.rs new file mode 100644 index 000000000..a38adc9f0 --- /dev/null +++ b/datafusion/distributed-iceberg/src/common/dynamic_filters/physical_predicate.rs @@ -0,0 +1,581 @@ +use std::sync::Arc; + +use datafusion::arrow::array::StructArray; +use datafusion::functions::core::r#struct::StructFunc; +use datafusion::logical_expr::Operator; +use datafusion::physical_expr::expressions::{ + BinaryExpr, CaseExpr, Column, InListExpr, IsNotNullExpr, IsNullExpr, Literal, +}; +use datafusion::physical_expr::{PhysicalExpr, ScalarFunctionExpr}; +use datafusion::scalar::ScalarValue; +use iceberg::expr::{Predicate, Reference}; +use iceberg::spec::{Datum, PrimitiveType}; + +/// Turns a DataFusion filter into an iceberg predicate. The predicate matches every row that the +/// filter would match, and *can* match more if the DF expression cannot be translated into an +/// Iceberg predicate completely. +/// +/// Returns `None` when no part of the filter can be converted. +pub(crate) fn convert_physical_filter_to_predicate( + filter: &Arc, +) -> Option { + convert(filter) +} + +/// Converts a given `PhysicalExpr` into an `iceberg::Predicate` +/// +/// We only convert the expressions DataFusion's dynamic filters use: comparisons, IN lists, +/// IS NULL, IS NOT NULL, AND, OR and CASE. +/// +/// Returns None if the expression could not be converted +fn convert(expr: &Arc) -> Option { + if let Some(binary) = expr.downcast_ref::() { + return convert_binary(binary); + } + if let Some(case) = expr.downcast_ref::() { + return convert_case(case); + } + if let Some(in_list) = expr.downcast_ref::() { + return convert_in_list(in_list); + } + if let Some(is_null) = expr.downcast_ref::() { + let column = column_name(is_null.arg())?; + return Some(Reference::new(column).is_null()); + } + if let Some(is_not_null) = expr.downcast_ref::() { + let column = column_name(is_not_null.arg())?; + return Some(Reference::new(column).is_not_null()); + } + // A bare `true`, `false` or NULL. NULL matches nothing. + match literal_value(expr)? { + ScalarValue::Boolean(Some(true)) => Some(Predicate::AlwaysTrue), + ScalarValue::Boolean(_) | ScalarValue::Null => Some(Predicate::AlwaysFalse), + _ => None, + } +} + +/// Converts a binary expression into an `iceberg::Predicate` +fn convert_binary(binary: &BinaryExpr) -> Option { + let (left, right) = (binary.left(), binary.right()); + match binary.op() { + // An AND expression can be converted if *either* side can be converted + // If one of the sides are dropped, iceberg would return extra rows + // which would then be filtered out by DataFusion + Operator::And => both(convert(left), convert(right)), + // An OR expression can only be converted if *both* sides can be converted + // We cannot drop one of the sides, as that would lead to data being skipped incorrectly + Operator::Or => Some(convert(left)?.or(convert(right)?)), + op => compare(left, *op, right), + } +} + +/// ANDs two converted sides. An AND can drop a side it can't convert: +/// `a = 1 AND f(b)` becomes `a = 1`. The result matches extra rows, which is fine. +fn both(left: Option, right: Option) -> Option { + match (left, right) { + (Some(left), Some(right)) => Some(left.and(right)), + (Some(side), None) | (None, Some(side)) => Some(side), + (None, None) => None, + } +} + +/// Converts a CaseExpr into an `iceberg::Predicate` +/// +/// In a CASE statement, each WHEN/THEN pair applies a filter over the rows. +/// +/// `CASE WHEN c1 THEN x WHEN c2 THEN y ELSE z END` becomes `(c1 AND x) OR (c2 AND y) OR z`. +/// It ignores that the second branch only applies when `c1` is false, so it can match +/// extra rows. +/// +/// `CASE a WHEN 1 THEN x ...` works the same way, with `a = 1` as the WHEN. +/// +/// A missing ELSE means NULL, which matches nothing. +fn convert_case(case: &CaseExpr) -> Option { + // This handles the two variants for CASE + // 1. CASE WHEN a = 1 THEN ... + // In this variant, `case.expr()` is `None` because the expression is imbued in the WHEN clause. + // We pass the WHEN clause to our general predicate converter for this. + // + // 2. CASE a WHEN 0 THEN ... + // In this variant, `case.expr()` is `Some(a)`, and each WHEN clause compares its own + // expression with this expression. + // To handle this, we convert this into a comparison predicate. + // Iceberg predicates can't always represent a given expression; if that is the case, we drop + // the predicate for the WHEN clause. + let convert_when = |when| match case.expr() { + Some(value) => compare(value, Operator::Eq, when), + None => convert(when), + }; + + case.when_then_expr() + .iter() + // Converts each individual WHEN/THEN pair into a conjunctive pair + // WHEN x > 5 THEN y < 3 becomes (x > 5 AND y < 3) + // If either side is dropped because it can't be converted, the filter is still valid + // and will just match more rows + .map(|(when, then)| both(convert_when(when), convert(then))) + // If there is an ELSE at the end, convert it into a predicate directly + .chain(case.else_expr().map(convert)) + .collect::>>()? + .into_iter() + // Combines all predicates in a disjunction, i.e, OR + // Multiple WHEN/THEN statements are joined together with ORs + // E.g: + // WHEN x > 5 THEN y ELSE + // WHEN x > 10 THEN z ELSE ... + // becomes + // (x > 5 AND y) OR (x > 10 AND z) + .reduce(Predicate::or) +} + +/// Converts a column <-> value comparison into an `iceberg::Predicate` +fn compare( + left: &Arc, + op: Operator, + right: &Arc, +) -> Option { + // Try to match ` ` first + // If unsuccessful, swap the order of the two inputs and check again + // If unsuccessful again, we can't convert this expression and we bail early + let (column, op, datum) = match (column_name(left), literal_datum(right)) { + (Some(column), Some(datum)) => (column, op, datum), + _ => (column_name(right)?, op.swap()?, literal_datum(left)?), + }; + + let column = Reference::new(column); + match op { + Operator::Eq => Some(column.equal_to(datum)), + Operator::NotEq => Some(column.not_equal_to(datum)), + Operator::Lt => Some(column.less_than(datum)), + Operator::LtEq => Some(column.less_than_or_equal_to(datum)), + Operator::Gt => Some(column.greater_than(datum)), + Operator::GtEq => Some(column.greater_than_or_equal_to(datum)), + _ => None, + } +} + +/// Converts an InList into an `iceberg::Predicate` +fn convert_in_list(in_list: &InListExpr) -> Option { + // Dynamic filters only build IN lists, so we don't convert NOT IN. + if in_list.negated() { + return None; + } + + // Something like `struct(a, b) IN (...)`, which checks several columns at once. + let Some(column) = column_name(in_list.expr()) else { + return convert_struct_in_list(in_list); + }; + + let literals = in_list + .list() + .iter() + // We expect every value in the list to be a literal, i.e, x IN (1, 2, 3) + // If it isn't, we can't convert this list, and bail early + .map(literal_value) + .collect::>>()? + .into_iter() + // NULL never equals anything, so `x IN (1, NULL)` matches the same rows as `x IN (1)` + .filter(|value| !value.is_null()) + .map(scalar_value_to_datum) + .collect::>>()?; + + Some(Reference::new(column).is_in(literals)) +} + +/// Converts a multi-column InList into an `iceberg::Predicate` +/// +/// Hash joins on two or more columns build filters like `struct(a, b) IN ({1, 3}, {2, 4})`. +/// +/// Iceberg can't check pairs, so each column is checked on its own: +/// `a IN (1, 2) AND b IN (3, 4)`. This can result in extra rows from pairs like `{1, 4}`, +/// which is okay. +/// +/// A column with a NULL value is left out, because the join decides if NULL matches NULL. +fn convert_struct_in_list(in_list: &InListExpr) -> Option { + let columns = + ScalarFunctionExpr::try_downcast_func::(in_list.expr().as_ref())?.args(); + + let items = in_list + .list() + .iter() + // If any of the items in the list aren't struct literals, we can't convert this + // and we bail early + .map(|item| match literal_value(item)? { + ScalarValue::Struct(item) => Some(item), + _ => None, + }) + .collect::>>()?; + + // One IN list per plain column. A column left out only makes the filter match more rows. + columns + .iter() + .enumerate() + .filter_map(|(index, column)| { + let column = column_name(column)?; + let values = field_values(&items, index)?; + Some(Reference::new(column).is_in(values)) + }) + .reduce(Predicate::and) +} + +/// Field `index` of each struct as an Iceberg value, or `None` if any of them can't be converted. +/// Each struct is a one-row array, so its value is in row 0. +fn field_values(items: &[&Arc], index: usize) -> Option> { + items + .iter() + .map(|item| { + let field = item.columns().get(index)?; + scalar_value_to_datum(&ScalarValue::try_from_array(field, 0).ok()?) + }) + .collect() +} + +fn column_name(expr: &Arc) -> Option<&str> { + expr.downcast_ref::().map(Column::name) +} + +fn literal_value(expr: &Arc) -> Option<&ScalarValue> { + expr.downcast_ref::().map(Literal::value) +} + +fn literal_datum(expr: &Arc) -> Option { + scalar_value_to_datum(literal_value(expr)?) +} + +/// Converts a DataFusion value to an iceberg value, for the types iceberg columns use. +/// +/// Floats aren't converted. DataFusion sorts NaN above every number and -0.0 below 0.0. +/// Iceberg doesn't, so a float filter like `a > 1.0` could skip rows DataFusion keeps. +/// +/// Times aren't converted either, because iceberg-rust's file reader can't compare them yet. +fn scalar_value_to_datum(value: &ScalarValue) -> Option { + match value { + ScalarValue::Boolean(Some(v)) => Some(Datum::bool(*v)), + ScalarValue::Int32(Some(v)) => Some(Datum::int(*v)), + ScalarValue::Int64(Some(v)) => Some(Datum::long(*v)), + ScalarValue::Decimal128(Some(v), precision, scale) => { + let data_type = PrimitiveType::Decimal { + precision: u32::from(*precision), + scale: u32::try_from(*scale).ok()?, + }; + Datum::try_from_bytes(&v.to_be_bytes(), data_type).ok() + } + ScalarValue::Date32(Some(v)) => Some(Datum::date(*v)), + ScalarValue::TimestampMicrosecond(Some(v), None) => Some(Datum::timestamp_micros(*v)), + ScalarValue::TimestampMicrosecond(Some(v), Some(_)) => Some(Datum::timestamptz_micros(*v)), + ScalarValue::TimestampNanosecond(Some(v), None) => Some(Datum::timestamp_nanos(*v)), + ScalarValue::TimestampNanosecond(Some(v), Some(_)) => Some(Datum::timestamptz_nanos(*v)), + ScalarValue::Utf8(Some(v)) + | ScalarValue::LargeUtf8(Some(v)) + | ScalarValue::Utf8View(Some(v)) => Some(Datum::string(v)), + ScalarValue::Binary(Some(v)) + | ScalarValue::LargeBinary(Some(v)) + | ScalarValue::BinaryView(Some(v)) => Some(Datum::binary(v.clone())), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use datafusion::arrow::array::{ArrayRef, Int64Array, StructArray}; + use datafusion::arrow::datatypes::{DataType, Field, Fields, Schema}; + use datafusion::common::{DFSchema, Result}; + use datafusion::config::ConfigOptions; + use datafusion::functions::core::r#struct; + use datafusion::logical_expr::Operator; + use datafusion::physical_expr::expressions::{ + BinaryExpr, Column, InListExpr, col, in_list, lit, + }; + use datafusion::physical_expr::{PhysicalExpr, ScalarFunctionExpr}; + use datafusion::prelude::SessionContext; + use datafusion::scalar::ScalarValue; + use insta::assert_snapshot; + + use super::convert_physical_filter_to_predicate; + + // The filters below use the columns from `schema()`: `a` and `b` are integers, `s` is a + // string, `f` is a float and `flag` is a boolean. + + #[test] + fn converts_null_checks() { + assert_snapshot!(convert_sql("a IS NULL"), @"a IS NULL"); + assert_snapshot!(convert_sql("a IS NOT NULL"), @"a IS NOT NULL"); + } + + #[test] + fn converts_comparisons_with_a_literal() { + assert_snapshot!(convert_sql("a = 5"), @"a = 5"); + assert_snapshot!(convert_sql("a != 5"), @"a != 5"); + assert_snapshot!(convert_sql("a < 5"), @"a < 5"); + assert_snapshot!(convert_sql("a <= 5"), @"a <= 5"); + assert_snapshot!(convert_sql("a > 5"), @"a > 5"); + assert_snapshot!(convert_sql("a >= 5"), @"a >= 5"); + } + + #[test] + fn flips_comparisons_with_the_literal_on_the_left() { + assert_snapshot!(convert_sql("5 < a"), @"a > 5"); + assert_snapshot!(convert_sql("5 >= a"), @"a <= 5"); + } + + #[test] + fn leaves_comparisons_between_columns_unconverted() { + assert_snapshot!(convert_sql("a < b"), @"None"); + } + + #[test] + fn leaves_float_comparisons_unconverted() { + assert_snapshot!(convert_sql("f > 1.5"), @"None"); + assert_snapshot!(convert_sql("a = 1 AND f > 1.5"), @"a = 1"); + } + + #[test] + fn converts_in_lists() { + assert_snapshot!(convert_sql("a IN (1, 2)"), @"a IN (1, 2)"); + } + + #[test] + fn leaves_not_in_lists_unconverted() { + assert_snapshot!(convert_sql("a NOT IN (1, 2)"), @"None"); + } + + #[test] + fn drops_nulls_from_an_in_list() -> Result<()> { + // Built by hand, the way a hash join builds it. SQL would wrap the NULL in a cast. + let schema = schema(); + let null = lit(ScalarValue::Int64(None)); + let values = vec![lit(1i64), Arc::clone(&null)]; + let one_or_null = in_list(col("a", &schema)?, values, &false, &schema)?; + let only_null = in_list(col("a", &schema)?, vec![null], &false, &schema)?; + assert_snapshot!(convert_filter(&one_or_null), @"a IN (1)"); + // `a IN (NULL)` matches nothing. + assert_snapshot!(convert_filter(&only_null), @"a IN ()"); + Ok(()) + } + + #[test] + fn leaves_in_lists_with_a_column_item_unconverted() { + assert_snapshot!(convert_sql("a IN (b, 1)"), @"None"); + } + + #[test] + fn leaves_an_in_list_over_an_expression_unconverted() { + assert_snapshot!(convert_sql("a + 1 IN (1, 2)"), @"None"); + } + + #[test] + fn leaves_filters_that_dynamic_filters_dont_use_unconverted() { + assert_snapshot!(convert_sql("NOT (a < 5)"), @"None"); + assert_snapshot!(convert_sql("s LIKE 'ab%'"), @"None"); + assert_snapshot!(convert_sql("flag"), @"None"); + } + + #[test] + fn converts_true_false_and_null() { + assert_snapshot!(convert_sql("true"), @"TRUE"); + assert_snapshot!(convert_sql("false"), @"FALSE"); + // A NULL filter keeps no rows. + assert_snapshot!(convert_sql("NULL"), @"FALSE"); + } + + #[test] + fn converts_an_and() { + assert_snapshot!(convert_sql("a = 1 AND b = 2"), @"(a = 1) AND (b = 2)"); + } + + #[test] + fn drops_the_unconverted_side_of_an_and() { + assert_snapshot!(convert_sql("a = 1 AND a < b"), @"a = 1"); + } + + #[test] + fn converts_an_or_only_when_both_sides_convert() { + // Hash joins build this when rows with a NULL key can still match. + assert_snapshot!( + convert_sql("a IS NULL OR a IN (1, 2)"), + @"(a IS NULL) OR (a IN (1, 2))" + ); + assert_snapshot!(convert_sql("a = 1 OR a < b"), @"None"); + } + + #[test] + fn converts_a_case_to_each_when_and_its_then_ored_together() { + assert_snapshot!( + convert_sql("CASE WHEN a = 1 THEN b = 2 WHEN a = 2 THEN b = 3 ELSE b = 4 END"), + @"(((a = 1) AND (b = 2)) OR ((a = 2) AND (b = 3))) OR (b = 4)" + ); + } + + #[test] + fn converts_case_value_whens_to_equalities() { + assert_snapshot!( + convert_sql("CASE a WHEN 1 THEN b = 2 WHEN 2 THEN b = 3 END"), + @"((a = 1) AND (b = 2)) OR ((a = 2) AND (b = 3))" + ); + } + + #[test] + fn keeps_only_the_thens_when_the_whens_cant_be_converted() { + // `a % 2` stands in for the `hash(a) % N` a partitioned hash join uses. + assert_snapshot!( + convert_sql("CASE a % 2 WHEN 0 THEN b IN (1, 2) WHEN 1 THEN b = 3 ELSE false END"), + @"(b IN (1, 2)) OR (b = 3)" + ); + } + + #[test] + fn keeps_the_when_if_its_then_cant_be_converted() { + assert_snapshot!(convert_sql("CASE WHEN a = 1 THEN a < b ELSE false END"), @"a = 1"); + } + + #[test] + fn converts_a_case_with_an_always_true_branch_to_true() { + // Hash joins use `true` for a partition whose build side was canceled. + assert_snapshot!(convert_sql("CASE a % 2 WHEN 0 THEN b = 1 ELSE true END"), @"TRUE"); + } + + #[test] + fn leaves_a_case_with_an_unconverted_branch_unconverted() { + assert_snapshot!( + convert_sql("CASE a % 2 WHEN 0 THEN b IN (1, 2) WHEN 1 THEN a < b ELSE false END"), + @"None" + ); + assert_snapshot!(convert_sql("CASE WHEN a = 1 THEN b = 2 ELSE a < b END"), @"None"); + } + + #[test] + fn converts_a_multi_column_join_filter_to_an_in_list_per_column() -> Result<()> { + let filter = struct_in(vec![Some(1), Some(2)], vec![Some(5), Some(6)])?; + assert_snapshot!(convert_filter(&filter), @"(a IN (1, 2)) AND (b IN (5, 6))"); + Ok(()) + } + + #[test] + fn drops_join_filter_columns_with_a_null() -> Result<()> { + let filter = struct_in(vec![Some(1), Some(2)], vec![Some(3), None])?; + assert_snapshot!(convert_filter(&filter), @"a IN (1, 2)"); + Ok(()) + } + + #[test] + fn converts_literals_to_iceberg_values() { + // Each value is compared with `a`. The translator doesn't check that the types match. + let values = [ + ScalarValue::Boolean(Some(true)), + ScalarValue::Int32(Some(1)), + ScalarValue::Int64(Some(1)), + ScalarValue::Decimal128(Some(12345), 10, 2), + ScalarValue::Decimal128(Some(-12345), 10, 2), + ScalarValue::Date32(Some(19723)), + ScalarValue::TimestampMicrosecond(Some(1_704_067_200_000_000), None), + ScalarValue::TimestampMicrosecond(Some(1_704_067_200_000_000), Some("+00:00".into())), + ScalarValue::TimestampNanosecond(Some(1_704_067_200_000_000_000), None), + ScalarValue::TimestampNanosecond( + Some(1_704_067_200_000_000_000), + Some("+00:00".into()), + ), + ScalarValue::Utf8(Some("ab".into())), + ScalarValue::LargeUtf8(Some("ab".into())), + ScalarValue::Utf8View(Some("ab".into())), + ScalarValue::Binary(Some(vec![1, 2])), + ScalarValue::LargeBinary(Some(vec![1, 2])), + ScalarValue::BinaryView(Some(vec![1, 2])), + ScalarValue::Float32(Some(1.5)), + ScalarValue::Float64(Some(1.5)), + ScalarValue::Time64Microsecond(Some(3_600_000_000)), + ScalarValue::Int64(None), + ]; + let converted = values + .into_iter() + .map(|value| { + let column = Arc::new(Column::new("a", 0)); + let filter: Arc = + Arc::new(BinaryExpr::new(column, Operator::Eq, lit(value.clone()))); + format!("{value:?} => {}", convert_filter(&filter)) + }) + .collect::>() + .join("\n"); + assert_snapshot!(converted, @r#" + Boolean(true) => a = true + Int32(1) => a = 1 + Int64(1) => a = 1 + Decimal128(123.45,10,2) => a = 123.45 + Decimal128(-123.45,10,2) => a = -123.45 + Date32("2024-01-01") => a = 2024-01-01 + TimestampMicrosecond(1704067200000000, None) => a = 2024-01-01 00:00:00 + TimestampMicrosecond(1704067200000000, Some("+00:00")) => a = 2024-01-01 00:00:00 UTC + TimestampNanosecond(1704067200000000000, None) => a = 2024-01-01 00:00:00 + TimestampNanosecond(1704067200000000000, Some("+00:00")) => a = 2024-01-01 00:00:00 UTC + Utf8("ab") => a = "ab" + LargeUtf8("ab") => a = "ab" + Utf8View("ab") => a = "ab" + Binary("1,2") => a = 0102 + LargeBinary("1,2") => a = 0102 + BinaryView("1,2") => a = 0102 + Float32(1.5) => None + Float64(1.5) => None + Time64Microsecond("3600000000") => None + Int64(NULL) => None + "#); + } + + fn schema() -> Schema { + Schema::new(vec![ + Field::new("a", DataType::Int64, true), + Field::new("b", DataType::Int64, true), + Field::new("s", DataType::Utf8, true), + Field::new("f", DataType::Float64, true), + Field::new("flag", DataType::Boolean, true), + ]) + } + + /// Turns SQL like `a < 5` into a DataFusion filter over [`schema`], then converts it. + fn convert_sql(filter: &str) -> String { + let schema = DFSchema::try_from(schema()).unwrap(); + let ctx = SessionContext::new(); + let expr = ctx.parse_sql_expr(filter, &schema).unwrap(); + let expr = ctx.state().create_physical_expr(expr, &schema).unwrap(); + convert_filter(&expr) + } + + /// Converts the filter and prints the predicate, or `None`. + fn convert_filter(filter: &Arc) -> String { + match convert_physical_filter_to_predicate(filter) { + Some(predicate) => predicate.to_string(), + None => "None".to_string(), + } + } + + /// Builds the filter a hash join on `a` and `b` makes: `struct(a, b) IN (...)`. + /// Each pair takes the values at the same position in `a` and `b`. + fn struct_in(a: Vec>, b: Vec>) -> Result> { + let schema = schema(); + let fields = Fields::from(vec![ + Field::new("a", DataType::Int64, true), + Field::new("b", DataType::Int64, true), + ]); + let columns = ScalarFunctionExpr::new( + "struct", + r#struct(), + vec![col("a", &schema)?, col("b", &schema)?], + Arc::new(Field::new("struct", DataType::Struct(fields.clone()), true)), + Arc::new(ConfigOptions::default()), + ); + let values = StructArray::new( + fields, + vec![ + Arc::new(Int64Array::from(a)) as ArrayRef, + Arc::new(Int64Array::from(b)) as ArrayRef, + ], + None, + ); + Ok(Arc::new(InListExpr::try_new_from_array( + Arc::new(columns), + Arc::new(values), + false, + &schema, + )?)) + } +} diff --git a/datafusion/distributed-iceberg/src/common/dynamic_filters/pruner.rs b/datafusion/distributed-iceberg/src/common/dynamic_filters/pruner.rs new file mode 100644 index 000000000..2f75995b8 --- /dev/null +++ b/datafusion/distributed-iceberg/src/common/dynamic_filters/pruner.rs @@ -0,0 +1,133 @@ +use std::sync::Arc; + +use datafusion::common::HashMap; +use datafusion::physical_expr::{PhysicalExpr, conjunction}; +use datafusion::physical_expr_common::physical_expr::{ + snapshot_generation, snapshot_physical_expr, +}; +use datafusion::physical_plan::metrics::{ + ExecutionPlanMetricsSet, MetricBuilder, MetricType, PruningMetrics, +}; +use iceberg::expr::Predicate; +use iceberg::scan::FileScanTask; + +use crate::common::dynamic_filters::partition_filter::PartitionFilter; +use crate::common::dynamic_filters::physical_predicate::convert_physical_filter_to_predicate; + +/// Uses the scan's dynamic filters on each file just before the reader starts it. Skips files +/// whose partition value can't match the filters. +pub(crate) struct DynamicFilterPruner { + /// The scan's dynamic filters, combined with AND. + dynamic_filters: Arc, + + /// The generation of `dynamic_filters` that `predicate` was built from. + cached_snapshot_generation: u64, + + /// The dynamic filters are stored as iceberg Predicates after being converted + /// `None` if the filters can't be converted + predicate: Option, + + /// A map of partition spec IDs to their corresponding partition filter executors + partition_filters: HashMap>, + + /// Metrics to track pruning + files: PruningMetrics, +} + +impl DynamicFilterPruner { + pub(crate) fn new( + dynamic_filters: Vec>, + metrics: &ExecutionPlanMetricsSet, + partition: usize, + ) -> Self { + Self { + dynamic_filters: conjunction(dynamic_filters), + cached_snapshot_generation: 0, + predicate: None, + partition_filters: HashMap::new(), + files: MetricBuilder::new(metrics) + .with_type(MetricType::Summary) + .pruning_metrics("files_pruned_by_partition", partition), + } + } + + /// Whether the filters rule out every row in the task's partition. Counts the task as pruned + /// or matched + pub(crate) fn should_prune(&mut self, task: &FileScanTask) -> bool { + let prune = self.can_skip(task); + if prune { + self.files.add_pruned(1); + } else { + self.files.add_matched(1); + } + prune + } + + /// Whether no row in the task's partition can match the filters. + fn can_skip(&mut self, task: &FileScanTask) -> bool { + let (Some(spec), Some(partition)) = (task.partition_spec(), task.partition()) else { + return false; + }; + if spec.is_unpartitioned() { + return false; + } + self.refresh(); + let Some(predicate) = &self.predicate else { + return false; + }; + + // Every `FileScanTask` within a scan has the same Iceberg schema, taken from the selected + // Iceberg snapshot. + // The only difference in partition filters can come from the partition spec being different + // between data files. + // The PartitionFilter struct - + // 1. Binds the `Predicate`, which just contains column names and values, to the actual + // schema, resolving column names to Iceberg field IDs etc. + // 2. Projects the column filter into a partition filter. A partition spec is not limited + // to just direct column filters; it can have expressions like `bucket(2, x)` or + // `day(ts)`. A column filter like `ts >= '2024-03-15 10:00'` becomes + // `ts_day >= 2024-03-15` which can be checked against every data file's partition value. + // These steps can be expensive, so we maintain a cache of a `PartitionFilter` per spec ID. + let partition_filter = self + .partition_filters + .entry(spec.spec_id()) + .or_insert_with(|| { + PartitionFilter::try_new( + predicate.clone(), + task.schema_ref(), + spec, + task.case_sensitive(), + ) + .ok() + }); + let Some(filter) = partition_filter.as_ref() else { + return false; + }; + + // Check whether the file can be dropped, based on the partition values. + // If there's an error while evaluating the filter, + // we return `false` so that the file is not skipped + filter.can_match(partition).is_ok_and(|matches| !matches) + } + + /// Rebuilds the predicate when any of the filters has changed. + fn refresh(&mut self) { + // Read the current generation before snapshotting the filters. + // The two reads aren't atomic, so a filter can change between them. + // + // In this order, the snapshot can be newer than the generation we have stored + // which is okay - because we're still doing the most 'efficient' pruning + // + // In the opposite order, we could store a newer generation but an older filter, + // which would prune less until the filter updated. + let generation = snapshot_generation(&self.dynamic_filters); + if self.cached_snapshot_generation == generation { + return; + } + self.cached_snapshot_generation = generation; + self.partition_filters.clear(); + self.predicate = snapshot_physical_expr(Arc::clone(&self.dynamic_filters)) + .ok() + .and_then(|snapshot| convert_physical_filter_to_predicate(&snapshot)); + } +} diff --git a/datafusion/distributed-iceberg/src/common/mod.rs b/datafusion/distributed-iceberg/src/common/mod.rs index 77c185916..984b811f8 100644 --- a/datafusion/distributed-iceberg/src/common/mod.rs +++ b/datafusion/distributed-iceberg/src/common/mod.rs @@ -2,6 +2,7 @@ mod error; pub(crate) use error::{df_err, iceberg_err}; +mod dynamic_filters; mod file_partition; pub(crate) mod scan_config; pub(crate) use file_partition::file_partition; diff --git a/datafusion/distributed-iceberg/src/common/scan_config.rs b/datafusion/distributed-iceberg/src/common/scan_config.rs index 600d6e48c..51dd51a04 100644 --- a/datafusion/distributed-iceberg/src/common/scan_config.rs +++ b/datafusion/distributed-iceberg/src/common/scan_config.rs @@ -1,10 +1,15 @@ +use crate::common::dynamic_filters::pruner::DynamicFilterPruner; use crate::{IcebergConfig, common::df_err}; use datafusion::arrow::datatypes::SchemaRef; use datafusion::common::stats::Precision; +use datafusion::common::tree_node::{Transformed, TreeNode, TreeNodeRecursion}; use datafusion::common::{ColumnStatistics, Statistics}; use datafusion::error::Result; use datafusion::execution::{SendableRecordBatchStream, TaskContext}; -use datafusion::physical_expr::Partitioning; +use datafusion::physical_expr::expressions::{DynamicFilterPhysicalExpr, InListExpr, lit}; +use datafusion::physical_expr::{Partitioning, PhysicalExpr, conjunction, split_conjunction}; +use datafusion::physical_expr_common::physical_expr::snapshot_physical_expr; +use datafusion::physical_plan::apply_expression_roots; use datafusion::physical_plan::limit::LimitStream; use datafusion::physical_plan::metrics::{ BaselineMetrics, ExecutionPlanMetricsSet, MetricBuilder, MetricCategory, MetricType, @@ -17,6 +22,7 @@ use iceberg::io::FileIO; use iceberg::scan::FileScanTaskStream; use iceberg::spec::SnapshotRef; use std::fmt; +use std::future::ready; use std::sync::Arc; /// Execution settings shared by assigned-file and feed-backed scans. @@ -30,6 +36,8 @@ pub(crate) struct ScanConfig { pub(crate) metrics: ExecutionPlanMetricsSet, pub(crate) iceberg_file_io: FileIO, pub(crate) iceberg_runtime: iceberg::Runtime, + pub(crate) filters: Option>, + pub(crate) dynamic_filtering_enabled: bool, // Clones share the delete loader across this source's local partitions. reader: ArrowReader, } @@ -57,10 +65,73 @@ impl ScanConfig { metrics: Default::default(), iceberg_file_io, iceberg_runtime, + filters: None, + dynamic_filtering_enabled: config.dynamic_filtering_enabled, reader, } } + /// ANDs `filters` onto the scan's filters. + pub(crate) fn push_filters(&mut self, filters: Vec>) { + self.filters = Some(conjunction(self.filters.take().into_iter().chain(filters))); + } + + /// Visits the scan's filters when dynamic filtering is enabled. + pub(crate) fn apply_expressions( + &self, + f: &mut dyn FnMut(&Arc) -> Result, + ) -> Result { + if !self.dynamic_filtering_enabled { + return Ok(TreeNodeRecursion::Continue); + } + apply_expression_roots(self.filters.iter(), f) + } + + /// Writes the scan's dynamic filters into `f` when dynamic filtering is enabled. A join filter + /// can list thousands of keys, so each IN list prints as its count. + pub(crate) fn fmt_dynamic_filters(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if !self.dynamic_filtering_enabled { + return Ok(()); + } + let Some(filters) = &self.filters else { + return Ok(()); + }; + let dynamic_filters = dynamic_filters(filters); + if dynamic_filters.is_empty() { + return Ok(()); + } + write!(f, ", dynamic_filters=")?; + for (i, filter) in dynamic_filters.into_iter().enumerate() { + if i > 0 { + write!(f, " AND ")?; + } + // A dynamic filter is at generation 1 until it gets its first values. + if filter.snapshot_generation() == 1 { + write!(f, "DynamicFilter [ empty ]")?; + continue; + } + let current = snapshot_physical_expr(filter) + .and_then(|current| { + current.transform_up(|expr| match expr.downcast_ref::() { + Some(in_list) => { + let not = if in_list.negated() { "NOT " } else { "" }; + let text = format!( + "{} {not}IN ([{} elements])", + in_list.expr(), + in_list.len() + ); + Ok(Transformed::yes(lit(text))) + } + None => Ok(Transformed::no(expr)), + }) + }) + .map_err(|_| fmt::Error)? + .data; + write!(f, "DynamicFilter [ {current} ]")?; + } + Ok(()) + } + pub(crate) fn open( &self, tasks: FileScanTaskStream, @@ -74,9 +145,19 @@ impl ScanConfig { .with_category(MetricCategory::Bytes) .gauge("bytes_scanned", partition); + let mut pruner = self.dynamic_filter_pruner(partition); // The reader pulls a task when it starts reading that task's file. It reads ahead, - // so with a LIMIT this can count files that never produced output. - let tasks = tasks.inspect_ok(move |_| files_opened.add(1)).boxed(); + // so with a LIMIT this can count files that never produced output. Skipped files + // are dropped before this, and the pruner counts them. + let tasks = tasks + .try_filter(move |task| { + let prune = pruner + .as_mut() + .is_some_and(|pruner| pruner.should_prune(task)); + ready(!prune) + }) + .inspect_ok(move |_| files_opened.add(1)) + .boxed(); let result = self.reader.clone().read(tasks).map_err(df_err)?; let reader_metrics = result.metrics().clone(); @@ -98,6 +179,32 @@ impl ScanConfig { Ok(Box::pin(LimitStream::new(stream, 0, self.fetch, metrics))) } + + /// A pruner over the scan's dynamic filters. `None` when dynamic filtering is off or the + /// scan has no dynamic filter. + fn dynamic_filter_pruner(&self, partition: usize) -> Option { + if !self.dynamic_filtering_enabled { + return None; + } + let dynamic_filters = dynamic_filters(self.filters.as_ref()?); + (!dynamic_filters.is_empty()) + .then(|| DynamicFilterPruner::new(dynamic_filters, &self.metrics, partition)) + } +} + +/// The parts of `filters` that contain a dynamic filter. It only looks at the expressions, so +/// it doesn't subscribe to the filters' updates. +fn dynamic_filters(filters: &Arc) -> Vec> { + split_conjunction(filters) + .into_iter() + .filter(|filter| { + matches!( + filter.exists(|expr| Ok(expr.is::())), + Ok(true) + ) + }) + .cloned() + .collect() } impl fmt::Debug for ScanConfig { @@ -109,6 +216,8 @@ impl fmt::Debug for ScanConfig { .field("metrics", &self.metrics) .field("iceberg_file_io", &self.iceberg_file_io) .field("iceberg_runtime", &self.iceberg_runtime) + .field("filters", &self.filters) + .field("dynamic_filtering_enabled", &self.dynamic_filtering_enabled) .finish_non_exhaustive() } } @@ -120,6 +229,7 @@ impl fmt::Debug for ScanConfig { /// https://github.com/apache/iceberg-rust/blob/4168a0b2950dc5f85588e5cb3ab6796e5228b309/crates/iceberg/src/spec/snapshot_summary.rs#L46-L47 const TOTAL_RECORDS: &str = "total-records"; const TOTAL_FILE_SIZE: &str = "total-files-size"; +const TOTAL_DELETE_FILES: &str = "total-delete-files"; /// Getting stats out of snapshot's additional properties (no I/O overhead) pub(crate) fn stats_from_snapshot( @@ -136,10 +246,23 @@ pub(crate) fn stats_from_snapshot( }; let props = &snap.summary().additional_properties; + // total-records counts data rows before merge-on-read deletes. Only an + // explicit zero delete-file count makes it safe to answer COUNT(*) from + // this summary. Missing or invalid optional counts must also force a scan. + let has_no_deletes = props + .get(TOTAL_DELETE_FILES) + .and_then(|v| v.parse::().ok()) + == Some(0); let num_rows = props .get(TOTAL_RECORDS) .and_then(|v| v.parse().ok()) - .map(Precision::Exact) + .map(|rows| { + if has_no_deletes { + Precision::Exact(rows) + } else { + Precision::Inexact(rows) + } + }) .unwrap_or(Precision::Absent); let total_byte_size = props .get(TOTAL_FILE_SIZE) diff --git a/datafusion/distributed-iceberg/src/config.rs b/datafusion/distributed-iceberg/src/config.rs index d9c24e850..2f9e38c53 100644 --- a/datafusion/distributed-iceberg/src/config.rs +++ b/datafusion/distributed-iceberg/src/config.rs @@ -18,6 +18,10 @@ extensions_options! { pub column_stats_enabled: bool, default = false /// Discover and retain file tasks during physical planning instead of execution. pub eager_file_planning: bool, default = false + /// (Experimental) Whether Iceberg scans are consumers of the dynamic filters pushed + /// into them, so joins and aggregates compute those filters. Scans don't skip data + /// with them yet. + pub dynamic_filtering_enabled: bool, default = false } } diff --git a/datafusion/distributed-iceberg/src/eager/codec.rs b/datafusion/distributed-iceberg/src/eager/codec.rs index 9aa97478d..b447b4d69 100644 --- a/datafusion/distributed-iceberg/src/eager/codec.rs +++ b/datafusion/distributed-iceberg/src/eager/codec.rs @@ -95,16 +95,17 @@ impl PhysicalExtensionCodec for EagerIcebergCodec { .collect::>>() }) .collect::>>()?; - Ok(DataSourceExec::from_data_source( - EagerIcebergDataSource::from_file_groups( - scan.schema, - groups, - scan.iceberg_file_io, - scan.iceberg_runtime, - scan.fetch, - ctx, - )?, - )) + let mut source = EagerIcebergDataSource::from_file_groups( + scan.schema, + groups, + scan.iceberg_file_io, + scan.iceberg_runtime, + scan.fetch, + ctx, + )?; + source.scan.filters = scan.filters; + source.scan.dynamic_filtering_enabled = scan.dynamic_filtering_enabled; + Ok(DataSourceExec::from_data_source(source)) } fn try_encode( @@ -175,14 +176,22 @@ impl EagerIcebergCodec { proto.custom_file_io.as_deref(), ctx, )?; - Ok(ScanConfig::new( + let filters = proto + .filters + .as_ref() + .map(|filters| proto_converter.proto_to_physical_expr(filters, &schema, &decode_ctx)) + .transpose()?; + let mut scan = ScanConfig::new( schema, partitioning, fetch, iceberg_file_io, self.iceberg_runtime.clone(), ctx, - )) + ); + scan.filters = filters; + scan.dynamic_filtering_enabled = proto.dynamic_filtering_enabled; + Ok(scan) } fn encode_config( @@ -203,6 +212,13 @@ impl EagerIcebergCodec { fetch: source.fetch().map(|value| value as u64), storage_properties: file_io.properties, custom_file_io: file_io.custom, + filters: source + .scan + .filters + .as_ref() + .map(|filters| proto_converter.physical_expr_to_proto(filters, self)) + .transpose()?, + dynamic_filtering_enabled: source.scan.dynamic_filtering_enabled, }) } } diff --git a/datafusion/distributed-iceberg/src/eager/data_source.rs b/datafusion/distributed-iceberg/src/eager/data_source.rs index b4017fb3a..06df942f4 100644 --- a/datafusion/distributed-iceberg/src/eager/data_source.rs +++ b/datafusion/distributed-iceberg/src/eager/data_source.rs @@ -3,12 +3,14 @@ use crate::eager::planned_files::PlannedFiles; use datafusion::arrow::datatypes::SchemaRef; use datafusion::common::tree_node::TreeNodeRecursion; use datafusion::common::{Statistics, exec_err}; +use datafusion::config::ConfigOptions; use datafusion::datasource::source::DataSource; use datafusion::error::Result; use datafusion::execution::{SendableRecordBatchStream, TaskContext}; use datafusion::physical_expr::projection::ProjectionExprs; use datafusion::physical_expr::{EquivalenceProperties, Partitioning, PhysicalExpr}; use datafusion::physical_plan::DisplayFormatType; +use datafusion::physical_plan::filter_pushdown::{FilterPushdownPropagation, PushedDown}; use datafusion::physical_plan::metrics::ExecutionPlanMetricsSet; use futures::StreamExt; use iceberg::io::FileIO; @@ -144,9 +146,9 @@ impl EagerIcebergDataSource { impl DataSource for EagerIcebergDataSource { fn apply_expressions( &self, - _f: &mut dyn FnMut(&Arc) -> Result, + f: &mut dyn FnMut(&Arc) -> Result, ) -> Result { - Ok(TreeNodeRecursion::Continue) + self.scan.apply_expressions(f) } fn open( &self, @@ -171,7 +173,7 @@ impl DataSource for EagerIcebergDataSource { if let Some(fetch) = self.scan.fetch { write!(f, ", fetch={fetch}")?; } - Ok(()) + self.scan.fmt_dynamic_filters(f) } fn output_partitioning(&self) -> Partitioning { self.scan.partitioning.clone() @@ -200,4 +202,21 @@ impl DataSource for EagerIcebergDataSource { fn metrics(&self) -> ExecutionPlanMetricsSet { self.scan.metrics.clone() } + + fn try_pushdown_filters( + &self, + filters: Vec>, + _config: &ConfigOptions, + ) -> Result>> { + let result = FilterPushdownPropagation::with_parent_pushdown_result(vec![ + PushedDown::No; + filters.len() + ]); + if filters.is_empty() { + return Ok(result); + } + let mut source = self.clone(); + source.scan.push_filters(filters); + Ok(result.with_updated_node(Arc::new(source))) + } } diff --git a/datafusion/distributed-iceberg/src/lazy/codec.rs b/datafusion/distributed-iceberg/src/lazy/codec.rs index 779f6e716..44c3212b2 100644 --- a/datafusion/distributed-iceberg/src/lazy/codec.rs +++ b/datafusion/distributed-iceberg/src/lazy/codec.rs @@ -103,16 +103,24 @@ impl PhysicalExtensionCodec for LazyIcebergCodec { proto.custom_file_io.as_deref(), ctx, )?; + let filters = proto + .filters + .as_ref() + .map(|filters| proto_converter.proto_to_physical_expr(filters, &schema, &decode_ctx)) + .transpose()?; + let mut scan = ScanConfig::new( + schema, + partitioning, + fetch, + iceberg_file_io, + self.iceberg_runtime.clone(), + ctx, + ); + scan.filters = filters; + scan.dynamic_filtering_enabled = proto.dynamic_filtering_enabled; Ok(DataSourceExec::from_data_source(LazyIcebergDataSource { - scan: ScanConfig::new( - schema, - partitioning, - fetch, - iceberg_file_io, - self.iceberg_runtime.clone(), - ctx, - ), + scan, column_stats: None, table_snapshot: None, feed, @@ -150,6 +158,13 @@ impl PhysicalExtensionCodec for LazyIcebergCodec { fetch: source.scan.fetch.map(|value| value as u64), storage_properties: file_io.properties, custom_file_io: file_io.custom, + filters: source + .scan + .filters + .as_ref() + .map(|filters| proto_converter.physical_expr_to_proto(filters, self)) + .transpose()?, + dynamic_filtering_enabled: source.scan.dynamic_filtering_enabled, }; proto.encode(buf).map_err(|error| { proto_error(format!("failed to encode LazyIcebergDataSource: {error}")) @@ -161,12 +176,16 @@ impl PhysicalExtensionCodec for LazyIcebergCodec { mod tests { use datafusion::common::Statistics; use datafusion::datasource::source::DataSource; + use datafusion::logical_expr::Operator; + use datafusion::physical_expr::expressions::{binary, col, lit}; + use datafusion::prelude::SessionConfig; use datafusion_distributed_iceberg_test_tools::create_taxi_table; use datafusion_proto::physical_plan::DefaultPhysicalProtoConverter; use iceberg::io::{FileIOBuilder, LocalFsStorageFactory}; use iceberg::table::Table; use super::*; + use crate::IcebergConfig; use crate::common::df_err; use crate::table_provider::scan::{IcebergDataSourceOptions, plan_scan}; @@ -186,6 +205,10 @@ mod tests { .runtime(iceberg::Runtime::current()) .build() .map_err(df_err)?; + let config = SessionConfig::new().with_option_extension(IcebergConfig { + dynamic_filtering_enabled: true, + ..Default::default() + }); let options = IcebergDataSourceOptions { fetch: Some(10), ..Default::default() @@ -194,9 +217,21 @@ mod tests { table, taxi.schema()?, options, - Arc::new(TaskContext::default()), + Arc::new(TaskContext::default().with_session_config(config.clone())), ) .await?; + let schema = taxi.schema()?; + let filter = binary( + col("fare_amount", &schema)?, + Operator::Gt, + lit(10.0), + &schema, + )?; + let pushed = iceberg_source(&plan)?.try_pushdown_filters(vec![filter], config.options())?; + let Some(pushed_source) = pushed.updated_node else { + return internal_err!("the scan should keep the filter it was offered"); + }; + let plan: Arc = Arc::new(DataSourceExec::new(pushed_source)); let codec = LazyIcebergCodec::new(Arc::new(LocalFsStorageFactory), iceberg::Runtime::current()); let proto_converter = DefaultPhysicalProtoConverter {}; @@ -231,6 +266,11 @@ mod tests { decoded.partition_statistics(None)?.as_ref(), &Statistics::new_unknown(&decoded.scan.schema) ); + let Some(filters) = &decoded.scan.filters else { + return internal_err!("the decoded scan should keep the filter"); + }; + insta::assert_snapshot!(filters, @"fare_amount@8 > 10"); + assert!(decoded.scan.dynamic_filtering_enabled); Ok(()) } diff --git a/datafusion/distributed-iceberg/src/lazy/data_source.rs b/datafusion/distributed-iceberg/src/lazy/data_source.rs index 38686490e..97339bab3 100644 --- a/datafusion/distributed-iceberg/src/lazy/data_source.rs +++ b/datafusion/distributed-iceberg/src/lazy/data_source.rs @@ -126,9 +126,9 @@ impl LazyIcebergDataSource { impl DataSource for LazyIcebergDataSource { fn apply_expressions( &self, - _f: &mut dyn FnMut(&Arc) -> Result, + f: &mut dyn FnMut(&Arc) -> Result, ) -> Result { - Ok(TreeNodeRecursion::Continue) + self.scan.apply_expressions(f) } fn open( @@ -149,20 +149,20 @@ impl DataSource for LazyIcebergDataSource { fn fmt_as(&self, _t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result { write!(f, "format=iceberg")?; - let Some(feed) = self.feed.inner() else { - return Ok(()); - }; - write!(f, ", table={}", feed.iceberg_table.identifier())?; - if let Some(projection) = &feed.projection { - write!(f, ", projection=[{}]", projection.join(", "))?; + // Workers' copies have no feed, but still show their dynamic filters. + if let Some(feed) = self.feed.inner() { + write!(f, ", table={}", feed.iceberg_table.identifier())?; + if let Some(projection) = &feed.projection { + write!(f, ", projection=[{}]", projection.join(", "))?; + } + if let Some(predicate) = &feed.predicates { + write!(f, ", predicate={predicate}")?; + } + if let Some(fetch) = self.scan.fetch { + write!(f, ", fetch={fetch}")?; + } } - if let Some(predicate) = &feed.predicates { - write!(f, ", predicate={predicate}")?; - } - if let Some(fetch) = self.scan.fetch { - write!(f, ", fetch={fetch}")?; - } - Ok(()) + self.scan.fmt_dynamic_filters(f) } fn output_partitioning(&self) -> Partitioning { @@ -212,12 +212,17 @@ impl DataSource for LazyIcebergDataSource { filters: Vec>, _config: &ConfigOptions, ) -> Result>> { - // TODO: Allow this DataSource to be pushed down filters. Some filters might be more - // straight forward to accept, like simple predicates, but some others might require - // a bit more work, like dynamic filters. - Ok(FilterPushdownPropagation::with_parent_pushdown_result( - vec![PushedDown::No; filters.len()], - )) + // The scan keeps the filters but doesn't apply them, so DataFusion still checks every row. + let result = FilterPushdownPropagation::with_parent_pushdown_result(vec![ + PushedDown::No; + filters.len() + ]); + if filters.is_empty() { + return Ok(result); + } + let mut source = self.clone(); + source.scan.push_filters(filters); + Ok(result.with_updated_node(Arc::new(source))) } fn try_pushdown_sort( diff --git a/datafusion/distributed-iceberg/src/proto/generated/iceberg.rs b/datafusion/distributed-iceberg/src/proto/generated/iceberg.rs index 1a19d3ea9..8e512a298 100644 --- a/datafusion/distributed-iceberg/src/proto/generated/iceberg.rs +++ b/datafusion/distributed-iceberg/src/proto/generated/iceberg.rs @@ -15,6 +15,13 @@ pub struct IcebergDataSource { ::std::collections::HashMap<::prost::alloc::string::String, ::prost::alloc::string::String>, #[prost(bytes = "vec", optional, tag = "6")] pub custom_file_io: ::core::option::Option<::prost::alloc::vec::Vec>, + /// Filters pushed into the scan. A row must pass all of them. + #[prost(message, optional, tag = "7")] + pub filters: ::core::option::Option<::datafusion_proto::protobuf::PhysicalExprNode>, + /// Whether the scan takes part in dynamic filtering. It reports its filters to + /// the operators that fill them in. + #[prost(bool, tag = "8")] + pub dynamic_filtering_enabled: bool, } #[allow(clippy::all)] #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] @@ -36,6 +43,13 @@ pub struct ScanConfig { ::std::collections::HashMap<::prost::alloc::string::String, ::prost::alloc::string::String>, #[prost(bytes = "vec", optional, tag = "5")] pub custom_file_io: ::core::option::Option<::prost::alloc::vec::Vec>, + /// Filters pushed into the scan. A row must pass all of them. + #[prost(message, optional, tag = "6")] + pub filters: ::core::option::Option<::datafusion_proto::protobuf::PhysicalExprNode>, + /// Whether the scan takes part in dynamic filtering. It reports its filters to + /// the operators that fill them in. + #[prost(bool, tag = "7")] + pub dynamic_filtering_enabled: bool, } #[allow(clippy::all)] #[derive(Clone, PartialEq, ::prost::Message)] diff --git a/datafusion/distributed-iceberg/src/proto/iceberg.proto b/datafusion/distributed-iceberg/src/proto/iceberg.proto index b02d978ac..613c4f33c 100644 --- a/datafusion/distributed-iceberg/src/proto/iceberg.proto +++ b/datafusion/distributed-iceberg/src/proto/iceberg.proto @@ -8,6 +8,11 @@ message IcebergDataSource { optional uint64 fetch = 4; map storage_properties = 5; optional bytes custom_file_io = 6; + // Filters pushed into the scan. A row must pass all of them. + DataFusionPhysicalExprNode filters = 7; + // Whether the scan takes part in dynamic filtering. It reports its filters to + // the operators that fill them in. + bool dynamic_filtering_enabled = 8; } message WorkUnitFeed { @@ -17,6 +22,7 @@ message WorkUnitFeed { // The generated Rust fields use DataFusion's canonical protobuf types. message DataFusionSchema {} message DataFusionPartitioning {} +message DataFusionPhysicalExprNode {} message ScanConfig { DataFusionSchema schema = 1; @@ -24,6 +30,11 @@ message ScanConfig { optional uint64 fetch = 3; map storage_properties = 4; optional bytes custom_file_io = 5; + // Filters pushed into the scan. A row must pass all of them. + DataFusionPhysicalExprNode filters = 6; + // Whether the scan takes part in dynamic filtering. It reports its filters to + // the operators that fill them in. + bool dynamic_filtering_enabled = 7; } message AssignedScan { ScanConfig config = 1; diff --git a/datafusion/distributed-iceberg/tests/dynamic_filtering.rs b/datafusion/distributed-iceberg/tests/dynamic_filtering.rs new file mode 100644 index 000000000..09a8d5438 --- /dev/null +++ b/datafusion/distributed-iceberg/tests/dynamic_filtering.rs @@ -0,0 +1,97 @@ +mod tests { + use datafusion::common::Result; + use datafusion::physical_plan::collect; + use datafusion_distributed::{ + DistributedExt, display_plan_ascii, rewrite_distributed_plan_with_dynamic_filters, + }; + use datafusion_distributed_iceberg::IcebergExt; + use datafusion_distributed_iceberg_test_tools::IcebergTestHarness; + use datafusion_distributed_iceberg_test_tools::create_taxi_table; + use std::sync::Arc; + + #[tokio::test] + async fn lazy_planning_scan_gets_the_join_filter() -> Result<()> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(create_taxi_table().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + .build() + .await?; + + let plan = harness + .physical_plan( + "SELECT COUNT(*) AS trips FROM taxi \ + JOIN (VALUES (DATE '2024-01-10')) AS days(day) ON taxi.pickup_date = days.day \ + WHERE taxi.trip_distance >= 2.0", + ) + .await?; + collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + + insta::assert_snapshot!(display_plan_ascii(plan.as_ref(), false), @" + ProjectionExec: expr=[count(Int64(1))@0 as trips] + AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))] + CoalescePartitionsExec + AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))] + HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(day@0, pickup_date@0)], projection=[] + ProjectionExec: expr=[column1@0 as day] + DataSourceExec: partitions=1, partition_sizes=[1] + FilterExec: trip_distance@0 >= 2, projection=[pickup_date@1] + DataSourceExec: format=iceberg, table=default.taxi, projection=[trip_distance, pickup_date], predicate=trip_distance >= 2, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ] + "); + Ok(()) + } + + #[tokio::test] + async fn eager_planning_workers_get_the_join_filter() -> Result<()> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(create_taxi_table().await?) + .with_config_option("iceberg.eager_file_planning", true) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + .with_workers(2) + .configure_session(|state| { + state + .with_distributed_file_scan_config_bytes_per_partition(1)? + .with_distributed_dynamic_filter_collection(true) + })? + .build() + .await?; + + let plan = harness + .physical_plan( + "SELECT COUNT(*) AS trips FROM taxi \ + JOIN (VALUES (DATE '2024-01-10')) AS days(day) ON taxi.pickup_date = days.day \ + WHERE taxi.trip_distance >= 2.0", + ) + .await?; + let task_ctx = harness.state().task_ctx(); + collect(Arc::clone(&plan), Arc::clone(&task_ctx)).await?; + let plan = rewrite_distributed_plan_with_dynamic_filters(plan, &task_ctx).await?; + + insta::assert_snapshot!(display_plan_ascii(plan.as_ref(), false), @" + โ”Œโ”€โ”€โ”€โ”€โ”€ DistributedExec + โ”‚ ProjectionExec: expr=[count(Int64(1))@0 as trips] + โ”‚ AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))] + โ”‚ CoalescePartitionsExec + โ”‚ [Stage 2] => NetworkCoalesceExec: output_partitions=8, input_tasks=2 + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + โ”Œโ”€โ”€โ”€โ”€โ”€ Stage 2 โ”€โ”€ tasks=2, partitions=8 + โ”‚ AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))] + โ”‚ HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(day@0, pickup_date@0)], projection=[] + โ”‚ CoalescePartitionsExec + โ”‚ [Stage 1] => NetworkBroadcastExec: partitions_per_consumer=1, stage_partitions=2, input_tasks=1 + โ”‚ FilterExec: trip_distance@0 >= 2, projection=[pickup_date@1] + โ”‚ DistributedLeafExec: + โ”‚ t0: DataSourceExec: format=iceberg, planned_files=3, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ] + โ”‚ t1: DataSourceExec: format=iceberg, planned_files=4, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ] + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + โ”Œโ”€โ”€โ”€โ”€โ”€ Stage 1 โ”€โ”€ tasks=1, partitions=2 + โ”‚ BroadcastExec: input_partitions=1, consumer_tasks=2, output_partitions=2 + โ”‚ ProjectionExec: expr=[column1@0 as day] + โ”‚ DataSourceExec: partitions=1, partition_sizes=[1] + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + "); + Ok(()) + } +} diff --git a/datafusion/distributed-iceberg/tests/partition_skipping.rs b/datafusion/distributed-iceberg/tests/partition_skipping.rs new file mode 100644 index 000000000..bd6fc1f68 --- /dev/null +++ b/datafusion/distributed-iceberg/tests/partition_skipping.rs @@ -0,0 +1,514 @@ +mod tests { + use datafusion::arrow::array::{ + ArrayRef, Date32Array, Int32Array, Int64Array, RecordBatch, StringArray, + TimestampMicrosecondArray, + }; + use datafusion::arrow::datatypes::SchemaRef; + use datafusion::arrow::util::pretty::pretty_format_batches; + use datafusion::common::Result; + use datafusion::datasource::source::DataSourceExec; + use datafusion::physical_plan::metrics::MetricValue; + use datafusion::physical_plan::{ExecutionPlan, collect}; + use datafusion_distributed::test_utils::metrics::node_metrics; + use datafusion_distributed::{ + DistributedExt, DistributedLeafExec, DistributedMetricsFormat, + rewrite_distributed_plan_with_metrics, + }; + use datafusion_distributed_iceberg::IcebergExt; + use datafusion_distributed_iceberg_test_tools::create_taxi_table; + use datafusion_distributed_iceberg_test_tools::{ + FixtureCatalog, FixtureTable, IcebergTestHarness, + }; + use iceberg::TableIdent; + use iceberg::spec::{NestedField, PrimitiveType, Schema, Transform, Type}; + use std::error::Error; + use std::sync::Arc; + + // The taxi table has one file per day, for the 7 days from 2024-01-08. The join keeps two of + // them, so its filter lets the scan skip the other 5 files. + const TWO_DAYS_OF_TRIPS: &str = "SELECT COUNT(*) AS trips FROM taxi \ + JOIN (VALUES (DATE '2024-01-10'), (DATE '2024-01-12')) AS days(day) \ + ON taxi.pickup_date = days.day"; + + #[tokio::test] + async fn lazy_planning_workers_skip_files_the_join_filter_rules_out() -> Result<()> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(create_taxi_table().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + .with_workers(2) + .configure_session(|state| { + state.with_distributed_file_scan_config_bytes_per_partition(1) + })? + .build() + .await?; + + let plan = harness.physical_plan(TWO_DAYS_OF_TRIPS).await?; + let batches = collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + + insta::assert_snapshot!(pretty_format_batches(&batches)?, @" + +-------+ + | trips | + +-------+ + | 50000 | + +-------+ + "); + // The first `DataSourceExec` is the join's `VALUES` table. The second is the taxi scan. + assert_eq!(node_metrics::(&plan, "files_opened", 1), 2); + Ok(()) + } + + #[tokio::test] + async fn eager_planning_workers_skip_files_the_join_filter_rules_out() -> Result<()> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(create_taxi_table().await?) + .with_config_option("iceberg.eager_file_planning", true) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + .with_workers(2) + .configure_session(|state| { + state.with_distributed_file_scan_config_bytes_per_partition(1) + })? + .build() + .await?; + + let plan = harness.physical_plan(TWO_DAYS_OF_TRIPS).await?; + let batches = collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + + insta::assert_snapshot!(pretty_format_batches(&batches)?, @" + +-------+ + | trips | + +-------+ + | 50000 | + +-------+ + "); + assert_eq!( + node_metrics::(&plan, "files_opened", 0), + 2 + ); + Ok(()) + } + + #[tokio::test] + async fn top_n_skips_files_once_it_fills_up() -> Result<()> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(create_taxi_table().await?) + .with_config_option("iceberg.eager_file_planning", true) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + // Read one file at a time, so top-N's filter tightens before the next file starts. + .with_config_option("iceberg.data_file_concurrency_limit", 1) + .configure_session(|mut state| { + let session = state.config().get_or_insert_default(); + session.options_mut().execution.target_partitions = 1; + Ok(state) + })? + .build() + .await?; + + let plan = harness + .physical_plan("SELECT pickup_date FROM taxi ORDER BY pickup_date LIMIT 3") + .await?; + let batches = collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + + insta::assert_snapshot!(pretty_format_batches(&batches)?, @" + +-------------+ + | pickup_date | + +-------------+ + | 2024-01-08 | + | 2024-01-08 | + | 2024-01-08 | + +-------------+ + "); + // The first file, 2024-01-08, fills the top 3. The filter then becomes + // `pickup_date < 2024-01-08`, which rules out the other 6 days. + assert_eq!(node_metrics::(&plan, "files_opened", 0), 1); + Ok(()) + } + + #[tokio::test] + async fn month_partitions_skip_files_from_other_months() -> Result<(), Box> { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(events_by_month().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + .build() + .await?; + + let plan = harness + .physical_plan( + "SELECT events.day FROM events \ + JOIN (VALUES (DATE '2024-02-14')) AS days(day) ON events.day = days.day", + ) + .await?; + let batches = collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + + insta::assert_snapshot!(pretty_format_batches(&batches)?, @" + +------------+ + | day | + +------------+ + | 2024-02-14 | + +------------+ + "); + // The first `DataSourceExec` is the join's `VALUES` table. The second is the events scan, + // which opens only February's file. + assert_eq!(node_metrics::(&plan, "files_opened", 1), 1); + Ok(()) + } + + #[tokio::test] + async fn null_partitions_are_read_when_the_filter_can_match_null() -> Result<(), Box> + { + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(groups_by_key().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + // Read one file at a time, so top-N's filter tightens before the next file starts. + .with_config_option("iceberg.data_file_concurrency_limit", 1) + .configure_session(|mut state| { + let session = state.config().get_or_insert_default(); + session.options_mut().execution.target_partitions = 1; + Ok(state) + })? + .build() + .await?; + + let plan = harness + .physical_plan("SELECT k FROM groups ORDER BY k NULLS FIRST LIMIT 2") + .await?; + let batches = collect(Arc::clone(&plan), harness.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + + insta::assert_snapshot!(pretty_format_batches(&batches)?, @" + +---+ + | k | + +---+ + | | + | | + +---+ + "); + // The scan reads `k = 1` first, and top-N's filter becomes `k IS NULL OR k < 1`. That + // skips the `k = 2` file and keeps the null file. + assert_eq!(node_metrics::(&plan, "files_opened", 0), 2); + Ok(()) + } + + #[tokio::test] + async fn validate_predicate_translation() -> Result<(), Box> { + let skipping = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(readings().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", true) + // Read one file at a time, so top-N's filter tightens before the next file starts. + .with_config_option("iceberg.data_file_concurrency_limit", 1) + // A join with more than 2 keys filters only on their bounds, so the cases cover + // both kinds of join filter. + .with_config_option( + "datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values", + 2, + ) + .configure_session(|mut state| { + let session = state.config().get_or_insert_default(); + session.options_mut().execution.target_partitions = 1; + Ok(state) + })? + .build() + .await?; + let reading_every_file = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(readings().await?) + .with_config_option("iceberg.eager_file_planning", false) + .with_config_option("iceberg.dynamic_filtering_enabled", false) + .with_config_option("iceberg.data_file_concurrency_limit", 1) + .with_config_option( + "datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values", + 2, + ) + .configure_session(|mut state| { + let session = state.config().get_or_insert_default(); + session.options_mut().execution.target_partitions = 1; + Ok(state) + })? + .build() + .await?; + + // Keys on partition edges, keys in no partition, and nulls, for each transform. + let schema = skipping.table().schema()?; + let cases = [ + join(&schema, "id", &["3"]), + join(&schema, "id", &["0"]), + join(&schema, "id", &["2", "6"]), + join(&schema, "id", &["2", "3", "4"]), + join(&schema, "id", &["NULL"]), + join(&schema, "id", &["1", "NULL"]), + join(&schema, "name", &["'avocado'"]), + join(&schema, "name", &["'b'"]), + join(&schema, "name", &["'รฉclair'"]), + join(&schema, "name", &["'รฉ'"]), + join(&schema, "name", &["'zebra'"]), + join(&schema, "day", &["'2024-02-01'"]), + join(&schema, "day", &["'2024-01-15'"]), + join(&schema, "day", &["'2024-04-01'"]), + join(&schema, "day", &["'2023-12-31'", "'2024-03-01'"]), + join(&schema, "n", &["12"]), + join(&schema, "n", &["99"]), + join(&schema, "n", &["-1", "13"]), + join(&schema, "at", &["'2024-01-02T00:00:00'"]), + join(&schema, "at", &["'2024-01-01T23:59:59.999999'"]), + join(&schema, "at", &["'2024-01-04T00:00:00'"]), + top_n("id", 1), + top_n("id DESC", 1), + top_n("id NULLS FIRST", 2), + top_n("id DESC NULLS LAST", 2), + top_n("name", 1), + top_n("name DESC", 1), + top_n("day", 2), + top_n("day DESC", 1), + top_n("n", 1), + top_n("at", 1), + top_n("at DESC", 2), + ]; + + // Each case must return the same rows whether the scan skips files or reads them all. + // The snapshot shows how many files each case kept. + let mut skipped = Vec::new(); + for (case, sql) in cases { + let plan = skipping.physical_plan(&sql).await?; + let rows = collect(Arc::clone(&plan), skipping.state().task_ctx()).await?; + let plan = + rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated) + .await?; + let expected = reading_every_file.query(&sql).await?.1; + assert_eq!( + pretty_format_batches(&rows)?.to_string(), + expected, + "{case}" + ); + let files = scan_metric(&plan, "files_pruned_by_partition") + .unwrap_or_else(|| panic!("{case}: the scan has no dynamic filter")); + let rows: usize = rows.iter().map(RecordBatch::num_rows).sum(); + skipped.push(format!("{case}: rows: {rows}, files: {files}")); + } + insta::assert_snapshot!(skipped.join("\n"), @" + join on id with 3: rows: 1, files: 6 total โ†’ 1 matched + join on id with 0: rows: 0, files: 6 total โ†’ 0 matched + join on id with 2, 6: rows: 2, files: 6 total โ†’ 2 matched + join on id with 2, 3, 4: rows: 3, files: 6 total โ†’ 3 matched + join on id with NULL: rows: 0, files: 0 total โ†’ 0 matched + join on id with 1, NULL: rows: 1, files: 6 total โ†’ 1 matched + join on name with 'avocado': rows: 1, files: 6 total โ†’ 2 matched + join on name with 'b': rows: 0, files: 6 total โ†’ 1 matched + join on name with 'รฉclair': rows: 1, files: 6 total โ†’ 1 matched + join on name with 'รฉ': rows: 0, files: 6 total โ†’ 1 matched + join on name with 'zebra': rows: 0, files: 6 total โ†’ 0 matched + join on day with '2024-02-01': rows: 1, files: 6 total โ†’ 2 matched + join on day with '2024-01-15': rows: 0, files: 6 total โ†’ 1 matched + join on day with '2024-04-01': rows: 0, files: 6 total โ†’ 0 matched + join on day with '2023-12-31', '2024-03-01': rows: 2, files: 6 total โ†’ 2 matched + join on n with 12: rows: 1, files: 6 total โ†’ 3 matched + join on n with 99: rows: 0, files: 6 total โ†’ 1 matched + join on n with -1, 13: rows: 2, files: 6 total โ†’ 4 matched + join on at with '2024-01-02T00:00:00': rows: 1, files: 6 total โ†’ 2 matched + join on at with '2024-01-01T23:59:59.999999': rows: 1, files: 6 total โ†’ 1 matched + join on at with '2024-01-04T00:00:00': rows: 0, files: 6 total โ†’ 0 matched + ORDER BY id LIMIT 1: rows: 1, files: 6 total โ†’ 1 matched + ORDER BY id DESC LIMIT 1: rows: 1, files: 6 total โ†’ 6 matched + ORDER BY id NULLS FIRST LIMIT 2: rows: 2, files: 6 total โ†’ 3 matched + ORDER BY id DESC NULLS LAST LIMIT 2: rows: 2, files: 6 total โ†’ 5 matched + ORDER BY name LIMIT 1: rows: 1, files: 6 total โ†’ 2 matched + ORDER BY name DESC LIMIT 1: rows: 1, files: 6 total โ†’ 5 matched + ORDER BY day LIMIT 2: rows: 2, files: 6 total โ†’ 3 matched + ORDER BY day DESC LIMIT 1: rows: 1, files: 6 total โ†’ 5 matched + ORDER BY n LIMIT 1: rows: 1, files: 6 total โ†’ 6 matched + ORDER BY at LIMIT 1: rows: 1, files: 6 total โ†’ 2 matched + ORDER BY at DESC LIMIT 2: rows: 2, files: 6 total โ†’ 5 matched + "); + Ok(()) + } + + // Table `events`, partitioned by the month of `day`, with one file for each of January, + // February and March 2024. + async fn events_by_month() -> Result> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::optional(1, "day", Type::Primitive(PrimitiveType::Date)).into(), + ]) + .build()?; + let fixtures = FixtureCatalog::try_new().await?; + let mut table = fixtures + .table(TableIdent::from_strs(["default", "events"])?, schema) + .partition_by("day", Transform::Month) + .create() + .await?; + // Days since 1970-01-01. + let january = [19727, 19742]; // 2024-01-05, 2024-01-20 + let february = [19756, 19767]; // 2024-02-03, 2024-02-14 + let march = [19783]; // 2024-03-01 + let mut append = table.append(); + for days in [&january[..], &february[..], &march[..]] { + let days: ArrayRef = Arc::new(Date32Array::from(days.to_vec())); + append = append.file(RecordBatch::try_from_iter([("day", days)])?); + } + append.commit().await?; + Ok(table) + } + + // Table `groups`, partitioned by `k`, with one file each for `k = 1`, `k = 2` and a null `k`, + // in that order. Each file has two rows. + async fn groups_by_key() -> Result> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::optional(1, "k", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build()?; + let fixtures = FixtureCatalog::try_new().await?; + let mut table = fixtures + .table(TableIdent::from_strs(["default", "groups"])?, schema) + .partition_by("k", Transform::Identity) + .create() + .await?; + let mut append = table.append(); + for k in [Some(1), Some(2), None] { + let keys: ArrayRef = Arc::new(Int32Array::from(vec![k, k])); + append = append.file(RecordBatch::try_from_iter([("k", keys)])?); + } + append.commit().await?; + Ok(table) + } + + // Table `readings`, partitioned by `id`, the first letter of `name`, the month of `day`, 4 + // buckets of `n` and the day of `at`. Each row is its own file, and the last row is all + // nulls. + async fn readings() -> Result> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::optional(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "day", Type::Primitive(PrimitiveType::Date)).into(), + NestedField::optional(4, "n", Type::Primitive(PrimitiveType::Long)).into(), + NestedField::optional(5, "at", Type::Primitive(PrimitiveType::Timestamp)).into(), + ]) + .build()?; + let fixtures = FixtureCatalog::try_new().await?; + let mut table = fixtures + .table(TableIdent::from_strs(["default", "readings"])?, schema) + .partition_by("id", Transform::Identity) + .partition_by("name", Transform::Truncate(1)) + .partition_by("day", Transform::Month) + .partition_by("n", Transform::Bucket(4)) + .partition_by("at", Transform::Day) + .create() + .await?; + let id: ArrayRef = Arc::new(Int32Array::from(vec![ + Some(1), + Some(2), + Some(3), + Some(4), + Some(6), + None, + ])); + let name: ArrayRef = Arc::new(StringArray::from(vec![ + Some("apple"), + Some("avocado"), + Some("banana"), + Some("รฉclair"), + Some("cherry"), + None, + ])); + // Days since 1970-01-01: 2024-01-31, 2024-02-01, 2024-02-29, 2024-03-01, 2023-12-31. + let day: ArrayRef = Arc::new(Date32Array::from(vec![ + Some(19753), + Some(19754), + Some(19782), + Some(19783), + Some(19722), + None, + ])); + let n: ArrayRef = Arc::new(Int64Array::from(vec![ + Some(10), + Some(11), + Some(12), + Some(13), + Some(-1), + None, + ])); + // Microseconds since 1970-01-01: 2024-01-01 23:59:59.999999, 2024-01-02 00:00:00, + // 2024-01-02 12:00:00, 2024-01-03 00:00:00, 2023-12-31 23:59:59. + let at: ArrayRef = Arc::new(TimestampMicrosecondArray::from(vec![ + Some(1704153599999999), + Some(1704153600000000), + Some(1704196800000000), + Some(1704240000000000), + Some(1704067199000000), + None, + ])); + let rows = RecordBatch::try_from_iter([ + ("id", id), + ("name", name), + ("day", day), + ("n", n), + ("at", at), + ])?; + let mut append = table.append(); + for row in 0..rows.num_rows() { + append = append.file(rows.slice(row, 1)); + } + append.commit().await?; + Ok(table) + } + + /// A case that joins `readings` with `keys` on `column`. Each key is cast to the column's + /// type, so the join filters the column itself, without a cast around it. + fn join(schema: &SchemaRef, column: &str, keys: &[&str]) -> (String, String) { + let data_type = schema.field_with_name(column).unwrap().data_type(); + let values = keys + .iter() + .map(|key| format!("(arrow_cast({key}, '{data_type}'))")) + .collect::>() + .join(", "); + ( + format!("join on {column} with {}", keys.join(", ")), + format!( + "SELECT t.* FROM readings t JOIN (VALUES {values}) AS k(x) \ + ON t.{column} = k.x ORDER BY t.id" + ), + ) + } + + /// A case that reads the first `limit` rows of `readings` in `order`. + fn top_n(order: &str, limit: usize) -> (String, String) { + ( + format!("ORDER BY {order} LIMIT {limit}"), + format!("SELECT * FROM readings ORDER BY {order} LIMIT {limit}"), + ) + } + + /// The metric `name` of the Iceberg scan in `plan`, summed over its partitions. + fn scan_metric(plan: &Arc, name: &str) -> Option { + plan.metrics() + .and_then(|metrics| metrics.sum_by_name(name)) + .or_else(|| { + plan.children() + .into_iter() + .find_map(|child| scan_metric(child, name)) + }) + } +} diff --git a/datafusion/distributed-iceberg/tests/statistics.rs b/datafusion/distributed-iceberg/tests/statistics.rs index ae8319f5f..e7b711412 100644 --- a/datafusion/distributed-iceberg/tests/statistics.rs +++ b/datafusion/distributed-iceberg/tests/statistics.rs @@ -217,7 +217,8 @@ mod tests { .await?; let stats = query_statistics(&harness, "SELECT * FROM taxi").await?; - assert_eq!(stats.num_rows, Precision::Exact(42)); + // This historical summary omits the optional delete-file count. + assert_eq!(stats.num_rows, Precision::Inexact(42)); assert_eq!(stats.total_byte_size, Precision::Exact(4_242)); Ok(()) } @@ -322,6 +323,33 @@ mod tests { Ok(()) } + #[tokio::test] + async fn count_star_applies_position_deletes() -> Result<(), Box> { + let (mut table, files) = create_metrics_table().await?; + // Nine data rows, with two positions removed from the first file. + table + .append() + .position_deletes(&files[0], [0, 2]) + .commit() + .await?; + let harness = IcebergTestHarness::builder() + .with_session_extension(|state| state.with_iceberg_integration(Default::default())) + .with_table(table) + .with_config_option("iceberg.eager_file_planning", false) + .build() + .await?; + let (_, batches) = harness.query("SELECT count(*) FROM metrics").await?; + + insta::assert_snapshot!(batches, @" + +----------+ + | count(*) | + +----------+ + | 7 | + +----------+ + "); + Ok(()) + } + // Observe the query's output statistics, including projection and propagation. async fn query_statistics(harness: &IcebergTestHarness, sql: &str) -> Result> { let plan = harness.physical_plan(sql).await?; diff --git a/datafusion/distributed-remote-worker/src/main.rs b/datafusion/distributed-remote-worker/src/main.rs index a50896bf4..7d7dc5f1a 100644 --- a/datafusion/distributed-remote-worker/src/main.rs +++ b/datafusion/distributed-remote-worker/src/main.rs @@ -18,7 +18,7 @@ use datafusion_distributed::test_utils::work_unit_file_scan::{ use datafusion_distributed::{ ChannelResolver, DistributedExt, DistributedMetricsFormat, NetworkBoundaryExt, SessionStateBuilderExt, Stage, Worker, WorkerQueryContext, WorkerResolver, display_plan_ascii, - get_distributed_channel_resolver, get_distributed_worker_resolver, + generate_plan_dump, get_distributed_channel_resolver, get_distributed_worker_resolver, rewrite_distributed_plan_with_metrics, }; use futures::{StreamExt, TryFutureExt}; @@ -223,6 +223,8 @@ async fn main() -> Result<(), Box> { let elapsed = start.elapsed(); let ms = elapsed.as_secs_f64() * 1000.0; + // Generated after measuring, so it doesn't count towards the query time. + let plan_dump = generate_plan_dump(&physical).map_err(err)?; info!("Finished executing query:\n{sql}\n\n{plan}"); info!("Returned {count} rows in {ms} ms"); abort_notifier.finished(); @@ -230,6 +232,7 @@ async fn main() -> Result<(), Box> { Ok::<_, (StatusCode, String)>(Json(QueryResult { count, plan, + plan_dump, elapsed_ms: ms, tasks: task_count, stats_q_error_p50: stats_q_error.map(|q_error| q_error.p50), @@ -270,6 +273,8 @@ mod built_info { #[derive(Serialize)] struct QueryResult { plan: String, + /// Single-line plan dump, readable with `dfd plan`. + plan_dump: String, count: usize, elapsed_ms: f64, tasks: usize, diff --git a/datafusion/distributed/Cargo.toml b/datafusion/distributed/Cargo.toml index 6081907ea..1e5135056 100644 --- a/datafusion/distributed/Cargo.toml +++ b/datafusion/distributed/Cargo.toml @@ -7,10 +7,13 @@ license = "Apache-2.0" documentation = "https://datafusion-contrib.github.io/datafusion-distributed/" repository = "https://github.com/datafusion-contrib/datafusion-distributed" readme = "../../README.md" +autotests = false +# Every integration test is a module of `tests/main.rs`, so they all link into one binary +# instead of one binary per file. [[test]] -name = "dynamic_filtering" -path = "tests/dynamic_filtering/main.rs" +name = "integration" +path = "tests/main.rs" required-features = ["integration"] # The examples live at the repository root (`examples/`), outside this package directory. @@ -68,6 +71,8 @@ uuid = "1.19" delegate = "0.13.4" dashmap = "6.0.1" prost = "0.14.1" +base64 = "0.22.1" +zstd = "0.13" rand = "0.9" bytes = "1.11" pin-project = "1.1.10" @@ -123,7 +128,7 @@ chaos-tests = [] sysinfo = ["dep:sysinfo"] [dev-dependencies] -datafusion-distributed-benchmarks = { path = "../distributed-benchmarks" } +datafusion-datasets = { path = "../datasets" } # The crate's own tests drive `SessionContext::sql` and register parquet tables, so they need # both features regardless of which ones a consumer selects. Requesting them here keeps # `--no-default-features` test runs compiling without forcing either on library consumers. diff --git a/datafusion/distributed/src/codec/distributed_codec.rs b/datafusion/distributed/src/codec/distributed_codec.rs index 85cd9b623..2e53861d9 100644 --- a/datafusion/distributed/src/codec/distributed_codec.rs +++ b/datafusion/distributed/src/codec/distributed_codec.rs @@ -15,7 +15,9 @@ use datafusion::arrow::datatypes::SchemaRef; use datafusion::common::{Result, internal_datafusion_err}; use datafusion::error::DataFusionError; use datafusion::execution::TaskContext; -use datafusion::physical_expr::equivalence::{EquivalenceClass, EquivalenceGroup}; +use datafusion::physical_expr::equivalence::{ + AcrossPartitions, ConstExpr, EquivalenceClass, EquivalenceGroup, +}; use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr}; use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType}; use datafusion::physical_plan::union::UnionExec; @@ -513,11 +515,29 @@ fn serialize_equivalence_group( .eq_group() .iter() .map(|class| { - class + let expressions = class .iter() .map(|expr| proto_converter.physical_expr_to_proto(expr, codec)) - .collect::>>() - .map(|expressions| EquivalenceClassProto { expressions }) + .collect::>>()?; + let constant = match class + .canonical_expr() + .and_then(|expr| properties.eq_group().is_expr_constant(expr)) + { + None => None, + Some(AcrossPartitions::Heterogeneous) => { + Some(AcrossPartitionsProto::Heterogeneous(())) + } + Some(AcrossPartitions::Uniform(None)) => { + Some(AcrossPartitionsProto::UniformUnknown(())) + } + Some(AcrossPartitions::Uniform(Some(value))) => { + Some(AcrossPartitionsProto::UniformValue((&value).try_into()?)) + } + }; + Ok(EquivalenceClassProto { + expressions, + constant, + }) }) .collect() } @@ -528,22 +548,34 @@ fn parse_equivalence_properties( decode_ctx: &PhysicalPlanDecodeContext<'_>, proto_converter: &dyn PhysicalProtoConverterExtension, ) -> Result { - let classes = equivalence_classes - .into_iter() - .map(|class| { - class - .expressions - .iter() - .map(|expr| { - proto_converter.proto_to_physical_expr(expr, schema.as_ref(), decode_ctx) - }) - .collect::>>() - .map(EquivalenceClass::new) - }) - .collect::>>()?; + let mut classes = Vec::with_capacity(equivalence_classes.len()); + let mut constants = Vec::new(); + for class in equivalence_classes { + let expressions = class + .expressions + .iter() + .map(|expr| proto_converter.proto_to_physical_expr(expr, schema.as_ref(), decode_ctx)) + .collect::>>()?; + if let Some(constant) = class.constant { + let expr = expressions.first().ok_or_else(|| { + internal_datafusion_err!("Constant equivalence class has no expressions") + })?; + let across = match constant { + AcrossPartitionsProto::Heterogeneous(()) => AcrossPartitions::Heterogeneous, + AcrossPartitionsProto::UniformUnknown(()) => AcrossPartitions::Uniform(None), + AcrossPartitionsProto::UniformValue(value) => { + AcrossPartitions::Uniform(Some((&value).try_into()?)) + } + }; + constants.push(ConstExpr::new(Arc::clone(expr), across)); + } + classes.push(EquivalenceClass::new(expressions)); + } let mut properties = EquivalenceProperties::new(schema); properties.add_equivalence_group(EquivalenceGroup::new(classes))?; + // Singleton column classes are otherwise discarded as trivial during reconstruction. + properties.add_constants(constants)?; Ok(properties) } @@ -663,6 +695,19 @@ pub struct EquivalenceClassProto { /// Expressions known to produce equal values. #[prost(message, repeated, tag = "1")] expressions: Vec, + /// Constant metadata can exist without a literal in the expression list. + #[prost(oneof = "AcrossPartitionsProto", tags = "2, 3, 4")] + constant: Option, +} + +#[derive(Clone, PartialEq, ::prost::Oneof)] +pub enum AcrossPartitionsProto { + #[prost(message, tag = "2")] + Heterogeneous(()), + #[prost(message, tag = "3")] + UniformUnknown(()), + #[prost(message, tag = "4")] + UniformValue(protobuf::ScalarValue), } #[derive(Clone, PartialEq, ::prost::Message)] @@ -810,6 +855,7 @@ mod tests { const DEFAULT_MODE: ShuffleMode = ShuffleMode::Direct; use datafusion::arrow::datatypes::{DataType, Field}; + use datafusion::common::ScalarValue; use datafusion::physical_expr::{LexOrdering, PhysicalExpr}; use datafusion::physical_plan::empty::EmptyExec; use datafusion::physical_plan::filter::FilterExec; @@ -1198,6 +1244,10 @@ mod tests { let schema = Arc::new(Schema::new(vec![ Field::new("a", DataType::Int32, false), Field::new("b", DataType::Int32, false), + Field::new("c0", DataType::Int32, true), + Field::new("c1", DataType::Int32, true), + Field::new("c2", DataType::Int32, true), + Field::new("c3", DataType::Int32, true), ])); let a: Arc = Arc::new(Column::new("a", 0)); let b: Arc = Arc::new(Column::new("b", 1)); @@ -1206,6 +1256,21 @@ mod tests { equivalence_properties.add_equal_conditions(a.clone(), b.clone())?; equivalence_properties.add_ordering([PhysicalSortExpr::new_default(a.clone())]); + let constants = [ + AcrossPartitions::Uniform(Some(ScalarValue::Int32(Some(7)))), + AcrossPartitions::Uniform(Some(ScalarValue::Int32(None))), + AcrossPartitions::Uniform(None), + AcrossPartitions::Heterogeneous, + ]; + let constants = constants + .into_iter() + .enumerate() + .map(|(i, across)| { + ConstExpr::new(Arc::new(Column::new(&format!("c{i}"), i + 2)), across) + }) + .collect::>(); + equivalence_properties.add_constants(constants.clone())?; + let plans: Vec<(&str, Arc)> = vec![ ( "shuffle", @@ -1251,6 +1316,11 @@ mod tests { .exprs_equal(&a, &b), "{name} lost the equivalence relationship" ); + assert_eq!( + decoded.properties().equivalence_properties().constants(), + constants, + "{name} lost constant metadata" + ); if name == "shuffle" { assert_eq!( decoded.properties().output_ordering(), diff --git a/datafusion/distributed/src/coordinator/distributed.rs b/datafusion/distributed/src/coordinator/distributed.rs index 179498c73..6bc4d59c3 100644 --- a/datafusion/distributed/src/coordinator/distributed.rs +++ b/datafusion/distributed/src/coordinator/distributed.rs @@ -266,9 +266,13 @@ impl ExecutionPlan for DistributedExec { internal_datafusion_err!("DistributedExec was already prepared for execution") })?; let mut stream = head_stage.execute(partition, context)?; + // Send a failure through the channel too, behind the batches already + // waiting there. Returned from this task instead, it could reach the + // consumer before them. while let Some(msg) = stream.next().await { - if tx.send(Ok(msg?)).await.is_err() { - break; // channel closed + let failed = msg.is_err(); + if tx.send(msg).await.is_err() || failed { + break; // channel closed, or the stream failed } } Ok(()) diff --git a/datafusion/distributed/src/coordinator/mod.rs b/datafusion/distributed/src/coordinator/mod.rs index a6990579f..53f931674 100644 --- a/datafusion/distributed/src/coordinator/mod.rs +++ b/datafusion/distributed/src/coordinator/mod.rs @@ -9,4 +9,5 @@ mod store; pub use distributed::DistributedExec; pub(crate) use dynamic_filter_registry::DynamicFilterRegistry; +pub(crate) use prepare_dynamic_plan::ESTIMATED_OUTPUT_BYTES_METRIC; pub(crate) use store::Store; diff --git a/datafusion/distributed/src/coordinator/prepare_dynamic_plan.rs b/datafusion/distributed/src/coordinator/prepare_dynamic_plan.rs index e8453d4d9..033855e70 100644 --- a/datafusion/distributed/src/coordinator/prepare_dynamic_plan.rs +++ b/datafusion/distributed/src/coordinator/prepare_dynamic_plan.rs @@ -2,8 +2,8 @@ use crate::common::{TreeNodeExt, element_wise_sum, vec_avg_reduce, vec_div, vec_ use crate::coordinator::distributed::PreparedPlan; use crate::coordinator::query_coordinator::QueryCoordinator; use crate::distributed_planner::{ - InjectNetworkBoundaryContext, NetworkBoundaryBuilderResult, ProducerHead, calculate_cost, - inject_network_boundaries, + InjectNetworkBoundaryContext, NetworkBoundaryBuilder, NetworkBoundaryBuilderResult, + ProducerHead, calculate_cost, inject_network_boundaries, }; use crate::dynamic_filtering::{ is_remote_dynamic_filtering_enabled, orphan_dynamic_filter_consumers, @@ -14,6 +14,7 @@ use crate::{ BytesCounterMetric, CoordinatorToWorkerMsg, LoadInfo, MaxGaugeMetric, NetworkBoundaryExt, NetworkCoalesceExec, Stage, TaskCountAnnotation, }; +use async_trait::async_trait; use dashmap::DashMap; use datafusion::common::stats::Precision; use datafusion::common::tree_node::{Transformed, TreeNode, TreeNodeRecursion}; @@ -29,128 +30,20 @@ use std::num::NonZeroUsize; use std::sync::Arc; use tokio_stream::wrappers::UnboundedReceiverStream; +/// Stage metric with the bytes a stage is estimated to return, from sampling its output. +pub(crate) const ESTIMATED_OUTPUT_BYTES_METRIC: &str = "estimated_output_bytes"; + pub(super) async fn prepare_dynamic_plan( query_coordinator: &Arc, base_plan: &Arc, ) -> Result { - let plans_for_viz = Arc::new(PlanReconstructor::default()); + let plans_for_viz = PlanReconstructor::default(); let head_stage = inject_network_boundaries( Arc::clone(base_plan), - |mut input_stage: LocalStage, nb_type: TypeId, nb_ctx: &InjectNetworkBoundaryContext| { - let mut metrics = MetricsSet::new(); - - // At this point, input_stage.plan has two kind of leaf nodes: - // - The ones that naturally do not read from any children, like DataSourceExec - // - Network boundaries whose Stage was set to Stage::Remote by a previous iteration - // of this same function. - // Both types of leaf nodes contain very valuable and accurate statistics that are used - // here for computing an estimation of the compute cost (measured in bytes): - // - DataSourceExec (or natural leaf nodes) contain stats pulled directly from their - // data source, like parquet files. - // - Network boundaries contain statistics collected from runtime information, gathered - // by the SamplerExec injected by this same function. - let cost = calculate_cost(&input_stage.plan)?; - metrics.push(BytesCounterMetric::new_metric( - "cpu_cost", - *cost.cpu.get_value().unwrap_or(&0), - )); - metrics.push(BytesCounterMetric::new_metric( - "memory_cost", - *cost.memory.get_value().unwrap_or(&0), - )); - metrics.push(BytesCounterMetric::new_metric( - "network_cost", - *cost.network.get_value().unwrap_or(&0), - )); - let compute_based_task_count = *cost.cpu.get_value().unwrap_or(&0) as f64 - / nb_ctx.d_cfg.dynamic_bytes_per_partition.max(1) as f64 - / input_stage - .plan - .output_partitioning() - .partition_count() - .max(1) as f64; - let compute_based_task_count = compute_based_task_count.min(nb_ctx.max_tasks()? as f64); - let task_count = nb_ctx - .task_count(&input_stage.plan)? - .merge(TaskCountAnnotation::soft(compute_based_task_count))?; - - // Propagate the final task_count inferred based on runtime statistics and compute cost. - // Here is where leaf nodes are scaled up by ScaleUpLeafNodeHandler, and the - // plan is finally left ready for distribution. - input_stage.plan = nb_ctx - .propagate_task_count_until_network_boundaries(&input_stage.plan, task_count)?; - input_stage.tasks = task_count.as_usize(); - // In order to infer the compute the cost of the stage above this one, here a sampler - // is injected to gather runtime statistics. - input_stage.plan = ProducerHead::insert_sampler(input_stage.plan)?; - let dynamic_filter_anchors = - if is_remote_dynamic_filtering_enabled(query_coordinator.session_config())? { - orphan_dynamic_filter_consumers(&input_stage.plan)? - } else { - vec![] - }; - - let mut load_info_rxs = Vec::with_capacity(input_stage.tasks); - - let query_coordinator = Arc::clone(query_coordinator); - let plans_for_viz = Arc::clone(&plans_for_viz); - - Ok(async move { - let mut stage_coordinator = query_coordinator.stage_coordinator(&input_stage); - - let mut futures = Vec::with_capacity(input_stage.tasks); - for task_i in 0..input_stage.tasks { - futures.push(stage_coordinator.init_bidirectional_stream(task_i)); - } - let results = futures::future::try_join_all(futures).await?; - - let mut workers = Vec::with_capacity(input_stage.tasks); - for (task_i, (url, worker_tx, worker_rx_stream)) in results.into_iter().enumerate() - { - workers.push(url); - load_info_rxs.push({ - let rx = - stage_coordinator.worker_to_coordinator_task(task_i, worker_rx_stream); - UnboundedReceiverStream::new(rx) - }); - let _ = worker_tx.send(CoordinatorToWorkerMsg::KickOffSampling); - stage_coordinator.coordinator_to_worker_task(task_i, worker_tx)?; - } - stage_coordinator.seal_dynamic_filter_stage()?; - - let (stats, consumer_tc) = if nb_type == TypeId::of::() { - (None, TaskCountAnnotation::exact(NonZeroUsize::MIN, 0.0)) - } else { - let (stats, new_metrics) = - gather_runtime_statistics(load_info_rxs, &input_stage.plan).await?; - metrics.extend(new_metrics); - // returning Desired(0) here is our way to tell the planner that we don't care - // about the task count assigned to the network boundary in the consumer stage, - // and we don't want it to affect other task count decisions. A 0.0 hint does - // not mask smaller fractional values via max-merge, and does not inflate - // UNION sums when this boundary sits under a child of a - // ChildrenIsolatorUnionExec. - (Some(Arc::new(stats)), TaskCountAnnotation::soft(0.0)) - }; - - // Capture the output partitioning of the (rescaled, sampler-wrapped) input plan - // before it's moved: the returned stage is remote and carries no plan to read it - // back from. - let input_properties = Arc::clone(input_stage.plan.properties()); - plans_for_viz.insert(input_stage.num, input_stage.plan, metrics); - Ok(NetworkBoundaryBuilderResult { - consumer_task_count: consumer_tc, - input_stage: Stage::Remote(RemoteStage { - query_id: input_stage.query_id, - num: input_stage.num, - workers, - runtime_stats: stats, - dynamic_filter_anchors, - }), - input_properties, - }) - }) + DynamicPlanBuilder { + query_coordinator, + plans_for_viz: &plans_for_viz, }, query_coordinator.session_config(), ) @@ -162,6 +55,129 @@ pub(super) async fn prepare_dynamic_plan( }) } +struct DynamicPlanBuilder<'a> { + query_coordinator: &'a Arc, + plans_for_viz: &'a PlanReconstructor, +} + +#[async_trait] +impl NetworkBoundaryBuilder for DynamicPlanBuilder<'_> { + async fn build<'a>( + &'a self, + mut input_stage: LocalStage, + nb_type: TypeId, + nb_ctx: &'a InjectNetworkBoundaryContext<'a>, + ) -> Result { + let mut metrics = MetricsSet::new(); + + // At this point, input_stage.plan has two kind of leaf nodes: + // - The ones that naturally do not read from any children, like DataSourceExec + // - Network boundaries whose Stage was set to Stage::Remote by a previous iteration + // of this same function. + // Both types of leaf nodes contain very valuable and accurate statistics that are used + // here for computing an estimation of the compute cost (measured in bytes): + // - DataSourceExec (or natural leaf nodes) contain stats pulled directly from their + // data source, like parquet files. + // - Network boundaries contain statistics collected from runtime information, gathered + // by the SamplerExec injected by this same function. + let cost = calculate_cost(&input_stage.plan)?; + metrics.push(BytesCounterMetric::new_metric( + "cpu_cost", + *cost.cpu.get_value().unwrap_or(&0), + )); + metrics.push(BytesCounterMetric::new_metric( + "memory_cost", + *cost.memory.get_value().unwrap_or(&0), + )); + metrics.push(BytesCounterMetric::new_metric( + "network_cost", + *cost.network.get_value().unwrap_or(&0), + )); + let compute_based_task_count = *cost.cpu.get_value().unwrap_or(&0) as f64 + / nb_ctx.d_cfg.dynamic_bytes_per_partition.max(1) as f64 + / input_stage + .plan + .output_partitioning() + .partition_count() + .max(1) as f64; + let compute_based_task_count = compute_based_task_count.min(nb_ctx.max_tasks()? as f64); + let task_count = nb_ctx + .task_count(&input_stage.plan)? + .merge(TaskCountAnnotation::soft(compute_based_task_count))?; + + // Propagate the final task_count inferred based on runtime statistics and compute cost. + // Here is where leaf nodes are scaled up by ScaleUpLeafNodeHandler, and the + // plan is finally left ready for distribution. + input_stage.plan = + nb_ctx.propagate_task_count_until_network_boundaries(&input_stage.plan, task_count)?; + input_stage.tasks = task_count.as_usize(); + // In order to infer the compute the cost of the stage above this one, here a sampler + // is injected to gather runtime statistics. + input_stage.plan = ProducerHead::insert_sampler(input_stage.plan)?; + let dynamic_filter_anchors = + if is_remote_dynamic_filtering_enabled(self.query_coordinator.session_config())? { + orphan_dynamic_filter_consumers(&input_stage.plan)? + } else { + vec![] + }; + + let mut load_info_rxs = Vec::with_capacity(input_stage.tasks); + + let mut stage_coordinator = self.query_coordinator.stage_coordinator(&input_stage); + + let mut futures = Vec::with_capacity(input_stage.tasks); + for task_i in 0..input_stage.tasks { + futures.push(stage_coordinator.init_bidirectional_stream(task_i)); + } + let results = futures::future::try_join_all(futures).await?; + + let mut workers = Vec::with_capacity(input_stage.tasks); + for (task_i, (url, worker_tx, worker_rx_stream)) in results.into_iter().enumerate() { + workers.push(url); + load_info_rxs.push({ + let rx = stage_coordinator.worker_to_coordinator_task(task_i, worker_rx_stream); + UnboundedReceiverStream::new(rx) + }); + let _ = worker_tx.send(CoordinatorToWorkerMsg::KickOffSampling); + stage_coordinator.coordinator_to_worker_task(task_i, worker_tx)?; + } + stage_coordinator.seal_dynamic_filter_stage()?; + + let (stats, consumer_tc) = if nb_type == TypeId::of::() { + (None, TaskCountAnnotation::exact(NonZeroUsize::MIN, 0.0)) + } else { + let (stats, new_metrics) = + gather_runtime_statistics(load_info_rxs, &input_stage.plan).await?; + metrics.extend(new_metrics); + // returning Desired(0) here is our way to tell the planner that we don't care + // about the task count assigned to the network boundary in the consumer stage, + // and we don't want it to affect other task count decisions. A 0.0 hint does + // not mask smaller fractional values via max-merge, and does not inflate + // UNION sums when this boundary sits under a child of a + // ChildrenIsolatorUnionExec. + (Some(Arc::new(stats)), TaskCountAnnotation::soft(0.0)) + }; + + // Capture the output partitioning of the (rescaled, sampler-wrapped) input plan + // before it's moved: the returned stage is remote and carries no plan to read it + // back from. + let input_properties = Arc::clone(input_stage.plan.properties()); + self.plans_for_viz + .insert(input_stage.num, input_stage.plan, metrics); + Ok(NetworkBoundaryBuilderResult { + consumer_task_count: consumer_tc, + input_stage: Stage::Remote(RemoteStage { + query_id: input_stage.query_id, + num: input_stage.num, + workers, + runtime_stats: stats, + dynamic_filter_anchors, + }), + input_properties, + }) + } +} + /// Reconstructs the plan dynamically as stages get transitioned to Remote and get sent to the /// respective workers. /// @@ -323,7 +339,7 @@ async fn gather_runtime_statistics( let total_byte_size: usize = per_col_byte_size.iter().sum(); new_metrics.push(BytesCounterMetric::new_metric( - "estimated_output_bytes", + ESTIMATED_OUTPUT_BYTES_METRIC, total_byte_size, )); diff --git a/datafusion/distributed/src/distributed_ext.rs b/datafusion/distributed/src/distributed_ext.rs index 57c25b9a3..176bf62da 100644 --- a/datafusion/distributed/src/distributed_ext.rs +++ b/datafusion/distributed/src/distributed_ext.rs @@ -2,6 +2,7 @@ use crate::codec::{set_distributed_user_codec, set_distributed_user_codec_arc}; use crate::config_extension_ext::{ set_distributed_option_extension, set_distributed_option_extension_from_headers, }; +use crate::distributed_planner::add_distributed_blacklisted_node; use crate::events::{ DesiredTaskCountHandler, DesiredTaskCountHandlers, RouteTaskHandler, RouteTaskHandlers, ScaleUpLeafNodeHandler, ScaleUpLeafNodeHandlers, TaskFinishedHandler, TaskFinishedHandlers, @@ -842,6 +843,26 @@ pub trait DistributedExt: Sized { /// Same as [DistributedExt::with_distributed_task_finished_handler] but with an in-place /// mutation. fn set_distributed_task_finished_handler(&mut self, handler: T); + + /// Prevents distributed planning for any plan containing a node of type `T`. Such plans are + /// left untouched and run on a single node. + /// + /// This is different from providing a [DesiredTaskCountHandler] that returns an exact count of + /// 1 in the sense that the distributed planner is fully disabled, instead of just throttling + /// that particular node to 1 task. + /// + /// ```rust + /// # use datafusion::execution::SessionStateBuilder; + /// # use datafusion::physical_plan::empty::EmptyExec; + /// # use datafusion_distributed::DistributedExt; + /// + /// SessionStateBuilder::new() + /// .with_distributed_blacklisted_node::(); + /// ``` + fn with_distributed_blacklisted_node(self) -> Self; + + /// Same as [DistributedExt::with_distributed_blacklisted_node] but with an in-place mutation. + fn set_distributed_blacklisted_node(&mut self); } /// Trait to have a unified interface for getting structs & properties from SessionConfig that are used in distributed context. @@ -1062,6 +1083,10 @@ impl DistributedExt for SessionConfig { TaskFinishedHandlers::push_custom(self, Arc::new(h)); } + fn set_distributed_blacklisted_node(&mut self) { + add_distributed_blacklisted_node::(self); + } + delegate! { to self { #[call(set_distributed_option_extension)] @@ -1189,6 +1214,10 @@ impl DistributedExt for SessionConfig { #[call(set_distributed_task_finished_handler)] #[expr($;self)] fn with_distributed_task_finished_handler(mut self, h: H) -> Self; + + #[call(set_distributed_blacklisted_node)] + #[expr($;self)] + fn with_distributed_blacklisted_node(mut self) -> Self; } } } @@ -1362,6 +1391,11 @@ impl DistributedExt for SessionStateBuilder { #[call(set_distributed_task_finished_handler)] #[expr($;self)] fn with_distributed_task_finished_handler(mut self, h: H) -> Self; + + fn set_distributed_blacklisted_node(&mut self); + #[call(set_distributed_blacklisted_node)] + #[expr($;self)] + fn with_distributed_blacklisted_node(mut self) -> Self; } } } @@ -1537,6 +1571,11 @@ impl DistributedExt for SessionState { #[call(set_distributed_task_finished_handler)] #[expr($;self)] fn with_distributed_task_finished_handler(mut self, h: H) -> Self; + + fn set_distributed_blacklisted_node(&mut self); + #[call(set_distributed_blacklisted_node)] + #[expr($;self)] + fn with_distributed_blacklisted_node(mut self) -> Self; } } } @@ -1705,6 +1744,11 @@ impl DistributedExt for SessionContext { #[call(set_distributed_task_finished_handler)] #[expr($;self)] fn with_distributed_task_finished_handler(self, h: H) -> Self; + + fn set_distributed_blacklisted_node(&mut self); + #[call(set_distributed_blacklisted_node)] + #[expr($;self)] + fn with_distributed_blacklisted_node(self) -> Self; } } } diff --git a/datafusion/distributed/src/distributed_planner/blacklisted_nodes.rs b/datafusion/distributed/src/distributed_planner/blacklisted_nodes.rs new file mode 100644 index 000000000..2a9bfa2ac --- /dev/null +++ b/datafusion/distributed/src/distributed_planner/blacklisted_nodes.rs @@ -0,0 +1,39 @@ +use datafusion::common::HashSet; +use datafusion::common::Result; +use datafusion::common::tree_node::TreeNode; +use datafusion::execution::config::SessionConfig; +use datafusion::physical_plan::ExecutionPlan; +use std::any::TypeId; +use std::sync::Arc; + +/// [SessionConfig] extension holding the [ExecutionPlan] types that prevent a plan from being +/// distributed. +#[derive(Default, Clone)] +struct BlacklistedNodes(HashSet); + +pub(crate) fn add_distributed_blacklisted_node(cfg: &mut SessionConfig) { + let mut blacklisted = cfg + .get_extension::() + .map(|v| v.as_ref().clone()) + .unwrap_or_default(); + blacklisted.0.insert(TypeId::of::()); + cfg.set_extension(Arc::new(blacklisted)); +} + +/// Whether `plan` contains any node blacklisted in `cfg`. +pub(crate) fn contains_blacklisted_node( + plan: &Arc, + cfg: &SessionConfig, +) -> Result { + let Some(blacklisted) = cfg.get_extension::() else { + return Ok(false); + }; + // `as_ref()` is needed: calling `type_id()` on the `Arc` returns the Arc's own type id. + plan.exists(|node| { + let type_id = match node.downcast_delegate() { + Some(delegate) => delegate.type_id(), + None => node.as_ref().type_id(), + }; + Ok(blacklisted.0.contains(&type_id)) + }) +} diff --git a/datafusion/distributed/src/distributed_planner/distributed_query_planner.rs b/datafusion/distributed/src/distributed_planner/distributed_query_planner.rs index 0f5dd9621..8ecbab25c 100644 --- a/datafusion/distributed/src/distributed_planner/distributed_query_planner.rs +++ b/datafusion/distributed/src/distributed_planner/distributed_query_planner.rs @@ -1,4 +1,5 @@ use crate::common::TreeNodeExt; +use crate::distributed_planner::blacklisted_nodes::contains_blacklisted_node; use crate::distributed_planner::inject_network_boundaries::{ CardinalityBasedNetworkBoundaryBuilder, inject_network_boundaries, }; @@ -90,6 +91,10 @@ pub fn create_distributed_plan( return Ok(original_plan); } + if contains_blacklisted_node(&original_plan, session.config())? { + return Ok(original_plan); + } + if let Some(analyze) = original_plan.downcast_ref::() { // Recursively distribute the query being analyzed. Replace `AnalyzeExec` only when its // input became distributed; otherwise retain DataFusion's native implementation. @@ -182,6 +187,7 @@ fn create_distributed_exec( mod tests { use crate::assert_snapshot; use crate::test_utils::plans::{TestPlanBuilder, build_side_one_desired_task_count_handler}; + use datafusion::physical_plan::aggregates::AggregateExec; /* schema for the "weather" table MinTemp [type=DOUBLE] [repetitiontype=OPTIONAL] @@ -1037,4 +1043,23 @@ mod tests { โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ "); } + + #[tokio::test] + async fn test_blacklisted_node_keeps_plan_single_node() { + let query = r#" + SELECT count(*), "RainToday" FROM weather GROUP BY "RainToday" + "#; + let physical_plan_ascii = TestPlanBuilder::default() + .with_distributed_blacklisted_node::() + .physical_plan_as_ascii(query, false) + .await; + + assert_snapshot!(physical_plan_ascii, @" + ProjectionExec: expr=[count(Int64(1))@1 as count(*), RainToday@0 as RainToday] + AggregateExec: mode=FinalPartitioned, gby=[RainToday@0 as RainToday], aggr=[count(Int64(1))] + RepartitionExec: partitioning=Hash([RainToday@0], 4), input_partitions=3 + AggregateExec: mode=Partial, gby=[RainToday@0 as RainToday], aggr=[count(Int64(1))] + DataSourceExec: file_groups={3 groups: [[/testdata/weather/result-000000.parquet], [/testdata/weather/result-000001.parquet], [/testdata/weather/result-000002.parquet]]}, projection=[RainToday], file_type=parquet + "); + } } diff --git a/datafusion/distributed/src/distributed_planner/mod.rs b/datafusion/distributed/src/distributed_planner/mod.rs index b49645214..ff32d02f7 100644 --- a/datafusion/distributed/src/distributed_planner/mod.rs +++ b/datafusion/distributed/src/distributed_planner/mod.rs @@ -1,3 +1,4 @@ +mod blacklisted_nodes; mod distributed_config; mod distributed_query_planner; mod inject_network_boundaries; @@ -11,10 +12,12 @@ mod push_fetch_into_network_coalesce; mod session_state_builder_ext; mod statistics; +pub(crate) use blacklisted_nodes::add_distributed_blacklisted_node; pub use distributed_config::DistributedConfig; pub use distributed_query_planner::create_distributed_plan; pub(crate) use inject_network_boundaries::{ - InjectNetworkBoundaryContext, NetworkBoundaryBuilderResult, inject_network_boundaries, + InjectNetworkBoundaryContext, NetworkBoundaryBuilder, NetworkBoundaryBuilderResult, + inject_network_boundaries, }; pub use network_boundary::{NetworkBoundary, NetworkBoundaryExt, ProducerHead}; pub use session_state_builder_ext::SessionStateBuilderExt; diff --git a/datafusion/distributed/src/lib.rs b/datafusion/distributed/src/lib.rs index dbdc91ea6..d4d7a401a 100644 --- a/datafusion/distributed/src/lib.rs +++ b/datafusion/distributed/src/lib.rs @@ -39,9 +39,10 @@ pub use execution_plans::{ }; pub use metrics::{ AvgLatencyMetric, BytesCounterMetric, BytesMetricExt, DISTRIBUTED_DATAFUSION_TASK_ID_LABEL, - DistributedMetricsFormat, FirstLatencyMetric, GaugeMetricExt, LatencyMetricExt, MaxGaugeMetric, - MaxLatencyMetric, MinLatencyMetric, P50LatencyMetric, P75LatencyMetric, P95LatencyMetric, - P99LatencyMetric, rewrite_distributed_plan_with_metrics, + DisplayPlanDumpOptions, DistributedMetricsFormat, FirstLatencyMetric, GaugeMetricExt, + LatencyMetricExt, MaxGaugeMetric, MaxLatencyMetric, MinLatencyMetric, P50LatencyMetric, + P75LatencyMetric, P95LatencyMetric, P99LatencyMetric, display_plan_dump, generate_plan_dump, + rewrite_distributed_plan_with_metrics, }; pub use protocol::LocalWorkerContext; diff --git a/datafusion/distributed/src/metrics/aggregate_by_name.rs b/datafusion/distributed/src/metrics/aggregate_by_name.rs new file mode 100644 index 000000000..d14d1e2ed --- /dev/null +++ b/datafusion/distributed/src/metrics/aggregate_by_name.rs @@ -0,0 +1,91 @@ +use datafusion::common::HashMap; +use datafusion::physical_plan::Metric; +use datafusion::physical_plan::metrics::{Label, MetricsSet}; +use std::sync::Arc; + +/// Same as [MetricsSet::aggregate_by_name], but also keeps every label that is present with the +/// same value in all the aggregated instances of a metric name. Labels that are missing from some +/// instances, or that have different values across them, are dropped, as there is no single value +/// that could represent them in the aggregated metric. +pub(crate) fn aggregate_by_name(metrics: &MetricsSet) -> MetricsSet { + // Metric names in first-seen order, so the output order is deterministic. + let mut names = vec![]; + let mut map: HashMap<&str, (Metric, Vec