Skip to content

Spark 4.2: Drop dynamic pruning filters that nest pruning filters - #49

Open
robert3005 wants to merge 1 commit into
mainfrom
rk/spark-nested-dynamic-pruning
Open

robert3005 wants to merge 1 commit into
mainfrom
rk/spark-nested-dynamic-pruning

Conversation

@robert3005

Copy link
Copy Markdown
Member

Summary

  • Add the optimizer rule NestedDynamicPruning, which drops each dynamic pruning filter whose filtering plan holds pruning filters of its own. The Iceberg session extensions install it.
  • Without the rule, a common table expression that joins with itself makes the plan grow by a factor of three per level of joins. The test query has five levels on a partition column: 121 pruning filters without the rule, and 2 with it.
  • This is the rule from Spark 4.2, Vortex: faster scans of Vortex tables #48, taken out so that it can merge by itself. It does not depend on runtime filters for data columns.

Checks

  • TestNestedDynamicPruning passes on Spark 4.2 (8 tests). Without the rule, the plan of the test query has 121 pruning filters, which is above the limit of 20 in the test.
  • spotlessApply on the two Spark 4.2 modules.
  • Not run: the full Spark test suites, and the other Spark versions. No backports are in this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_018SWbEtEQLKKebxSCarAZrU

Spark adds one pruning subquery per join that can prune a scan, holding
the plan of the join's other side. When a common table expression joins
with itself, each subquery copies the subqueries of that side, and the
plan grows by a factor of three per level of joins: five levels on a
partition column give 121 pruning filters.

The Iceberg session extensions now install NestedDynamicPruning, which
drops each pruning filter whose filtering plan holds pruning filters of
its own. A pruning filter only skips rows that the join drops, so the
result does not change.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@github-actions github-actions Bot added the SPARK label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant