feat: Support vector post-filtering and scoped FTS prefiltering - #91
zhangstar333 wants to merge 1 commit into
Conversation
db548af to
4a76e6e
Compare
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Gate recommendation: approve.
Vector queries now support SQL postfiltering within an explicit fragment domain. Filtered prepared FTS scans retain global scores while restricting prefilter work to the selected segments, and unfiltered and empty INDEX_ONLY scans preserve their existing behavior.
lance-gatekeeper
Bot
added
the
K-approved
| } | ||
| if apply_fragment_filter_after_nearest { | ||
| self.apply_fragment_filter(&mut scanner)?; | ||
| } |
There was a problem hiding this comment.
[P1] with_fragments does not become a post-filter based on call order
Scanner::with_fragments only stores self.fragments; the builder does not preserve whether it was called before or after nearest(). During plan creation, that fragment scope still restricts relevant index segments and flat-search inputs before Top-K. Moving this call below nearest() therefore bypasses ensure_not_fragment_scan(), but does not implement post-ranking filtering.
For example, if the global Top-K rows are all in fragment 0 while fragment 1 is selected, a real post-filter should return no rows, whereas this plan can return the Top-K rows from fragment 1.
Please implement the fragment predicate as an explicit post-ranking execution node, or clarify that fragments define the search domain and keep the prefilter semantics. Add a two-fragment indexed regression test that distinguishes these outcomes.
| @@ -558,6 +568,7 @@ impl LanceScanner { | |||
| struct PreparedFtsExecution { | |||
| context: Arc<FtsQueryContextInner>, | |||
| segments: Vec<IndexMetadata>, | |||
| scope_prefilter_to_fts_segments: bool, | |||
| batch_size: Option<usize>, | |||
| scan_statistics_callback: Option<ExecutionStatsCallback>, | |||
| } | |||
| @@ -596,11 +607,21 @@ impl PreparedScanner { | |||
| let Some(distributed_fts) = self.distributed_fts else { | |||
| return self.scanner.try_into_stream().await; | |||
| }; | |||
| let plan = self.scanner.create_plan().await?; | |||
| let selected_segments_have_current_fragments = segments_have_current_fragments( | |||
| let selected_fragments = selected_current_fts_fragments( | |||
| &distributed_fts.context.dataset, | |||
| &distributed_fts.segments, | |||
| )?; | |||
| let selected_segments_have_current_fragments = !selected_fragments.is_empty(); | |||
| let mut scanner = self.scanner; | |||
| if distributed_fts.scope_prefilter_to_fts_segments | |||
| && selected_segments_have_current_fragments | |||
| { | |||
| // The scanner is already split by the selected FTS segment(s). Applying the same | |||
| // fragment scope before plan creation lets Lance restrict scalar-index segment loads | |||
| // for the TVF prefilter without changing unfiltered FTS scans. | |||
There was a problem hiding this comment.
[P1] Add regression coverage for both new query-planning branches
This PR introduces two behavior changes, but the only test diff updates a helper call in the existing unfiltered prepared-FTS plan test.
Please add:
- An indexed, two-fragment vector test for
fragment_ids + prefilter=falsewhose expected result distinguishes global Top-K post-filtering from fragment-scoped Top-K. - A prepared-FTS test with at least two segments, one selected segment, a SQL filter, and
prefilter=true. Assert returned IDs and global scores, and use scan statistics or plan assertions to verify that only the selected segment’s fragments are loaded for the prefilter.
This is required because both branches affect result-selection boundaries, not just execution cost.
2 participants
prefilter=false, defer applying the fragment filter until afternearest()is configured, allowing Lance to apply it as a post-filter.