Skip to content

test: cover remote dynamic filters with a work unit feed - #896

Open
raghavyadav01 wants to merge 1 commit into
datafusion-contrib:mainfrom
raghavyadav01:test/work-unit-feed-remote-dynamic-filters
Open

raghavyadav01 wants to merge 1 commit into
datafusion-contrib:mainfrom
raghavyadav01:test/work-unit-feed-remote-dynamic-filters

Conversation

@raghavyadav01

@raghavyadav01 raghavyadav01 commented Nov 16, 2026 •

Copy link
Copy Markdown

Closes #770.

Adds remote_dynamic_filters_with_feed to tests/work_unit_feed.rs. It runs a partitioned join with a work unit feed on the build side and the weather parquet scan on the probe side, so the join's dynamic filter has to go through the coordinator to reach the scan in another stage while the coordinator is also streaming the feed's work units.

The test checks that:

  • the result is the same with dynamic filter pushdown off,
  • the coordinator received dynamic filter updates,
  • the probe scan ends up with the join's filter (Cloud9am between 0 and 1, matching the feed's partition values), via rewrite_distributed_plan_with_dynamic_filters in the snapshot.

The feed is on the build side because RowGeneratorExec doesn't accept pushed-down filters, so it can't be a filter consumer itself. Happy to extend it if you had a different case in mind.

A partitioned join with a work unit feed on the build side and a parquet
scan on the probe side, in different stages. The join dynamic filter has
to reach the scan through the coordinator while the feed streams its work
units over the same channel.

Signed-off-by: Raghav Yadav <raghavmnnit@gmail.com>
@raghavyadav01
raghavyadav01 force-pushed the test/work-unit-feed-remote-dynamic-filters branch from 7fae510 to cd5f547 Compare November 18, 2026 23:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[dynamic filtering] add integration test for work unit feeds

2 participants