Repository navigation
tp: search tables for pipeline WHERE instead of testing every row - #8298
Draft
LalitMaganti wants to merge 1 commit into
Draft
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
LalitMaganti
force-pushed
the
dev/lalitm/pipe-scan-pushdown
branch
from
October 29, 2026 19:26
7e5a674 to
66b52d2
Compare
🎨 Perfetto UI Builds & Tests
|
LalitMaganti
force-pushed
the
dev/lalitm/pipe-scan-pushdown
branch
from
October 29, 2026 20:35
66b52d2 to
465554a
Compare
A pipeline WHERE always scanned the whole table and tested each row, even where the dataframe can find the rows directly: a binary search of a sorted column such as `id` or `ts`, an index made with CREATE PERFETTO INDEX, or nothing at all when a value can match no row. The executor already has SortedScan, IndexScan and EmptySource for this, but the pipeline planner never chose them. A bound pass now moves into each scan a WHERE reads directly the conditions the scan can answer by searching, choosing as the dataframe's own planner does, and drops the WHERE if nothing is left for it to test. Results are unchanged; only how the rows are found is. The table's indexes are bound with its columns when a plan is loaded, so nothing new is serialized and an index made after the pipeline was written is still used.
LalitMaganti
force-pushed
the
dev/lalitm/pipe-scan-pushdown
branch
from
October 29, 2026 20:56
465554a to
ad2a687
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A pipeline WHERE always scanned the whole table and tested each row,
even where the dataframe can find the rows directly: a binary search of
a sorted column such as
idorts, an index made withCREATE PERFETTO INDEX, or nothing at all when a value can match no row.
The executor already has SortedScan, IndexScan and EmptySource for
this, but the pipeline planner never chose them.
Changes
SearchScans, a bound pass: for each WHERE directly on a scan, theconditions the scan can answer by searching move into the scan with
their literals. A WHERE left with nothing to test is dropped
(
LogicalPlan::Bypass).Scan::PlanSearchchooses as the dataframe's own planner does(
SortedFilterStrategy,GetBestIndexForFilterSpecs), andScan::Loweremits the SortedScan, IndexScan or EmptySource chosen.so nothing new is serialized and an index made after the pipeline was
written is still used.
Results are unchanged; only how the rows are found is.
Testing
conditions no row can match, and a new diff test.