feat(table): execute a primary-key vector search over engine-planned splits - #771
feat(table): execute a primary-key vector search over engine-planned splits#771JunRuiLee wants to merge 1 commit into
Conversation
0a22b36 to
3b027b4
Compare
…splits A planner running in Paimon Java enumerates one BucketVectorSearchSplit per bucket -- a bucket is never divided, because the ANN current-segment decision needs the bucket's whole active file set -- and ships its bytes to a worker outside the JVM. Decoding those bytes, resolving their index files and planning from them already landed; this is the part that runs the search and returns rows. `VectorSearchBuilder::execute_read_for_bucket_splits` takes the splits as the plan: their payload files, per-file row ranges and pinned snapshot are used as given, and the table's index manifest is not read. Everything after planning is the ordinary primary-key vector read, so search, optional refine, Top-K and materialization are shared with `execute_read` rather than reimplemented, and the output is the same -- projected user columns plus `__paimon_search_score`, best-first. The Top-K is local to the splits passed in; a caller distributing one call per bucket merges the per-bucket results itself. Rejected rather than worked around: a data-evolution column, which plans through the global index instead, so answering it would silently use a different plan than the caller supplied; and an empty split list, which pins no snapshot and would otherwise be indistinguishable from a query that matched nothing. `paimon_vector_search_builder_execute_read_for_bucket_splits` exposes it over the C ABI as a new symbol, with the array-of-buffers marshalling the existing terminal does not need. A null array, a zero count or a null entry is the caller's input error and is reported as one, rather than reaching the decoder as corrupt data. The end-to-end test is driven by a fixture where BOTH halves come from one Java run: the table directory written by the ivf-flat indexer, and the split bytes its own planner produced over that same table. A split names its files by generated UUID, so bytes from a different run would reference files that do not exist. Java records a row range only for a file its pre-filter narrowed, and the fixture's query has no pre-filter, so its rangeFileCount is zero and the committed split exercises the omitted-file normalization. The listed case is built on top of it, and is what distinguishes this route from the manifest one: with the manifest plan substituted, that test is the one that fails.
3b027b4 to
32f6b5b
Compare
| params.index_type.clone(), | ||
| self.filter.clone(), | ||
| ) | ||
| .plan_for_bucket_vector_splits(splits)?; |
There was a problem hiding this comment.
[P1] Bound or preserve decoded row ranges before bitmap expansion
This new entry point forwards engine-supplied DataFileMeta.row_count values into plan_for_bucket_vector_splits without a resource bound. When a file omits its range entry (which is valid in the Java encoding), planning synthesizes [0, row_count - 1], and the search path materializes that range into a RoaringTreemap. Both roaring 0.11.4 and the current 0.11.5 implementation iterate every high-u32 shard in insert_range and construct a full u32 bitmap for each intermediate shard; row_count = 2^40 already creates 256 full bitmaps (about 16.8 million containers), while a value near i64::MAX can hang or OOM the worker before any storage I/O. The later i32 row-position validation runs only after this expansion. Please reject unreasonable row counts/endpoints and, preferably, preserve unrestricted or interval-form selections through the ANN layer instead of eagerly materializing every allowed position.
| limit, | ||
| &plan, | ||
| ¶ms, | ||
| ) |
There was a problem hiding this comment.
[P2] Keep unrestricted files off the filtered ANN path
A Java split with rangeFileCount == 0 is the normal no-prefilter form (including the committed fixture), but the planner normalizes every omitted file into an explicit full-file range. This call then converts those ranges into RoaringTreemaps and passes Some(filter) to ANN even though every row is allowed. Lumina subsequently collects the bitmap into a Vec and invokes search_with_filter, adding O(live_rows) setup and at least 8 * live_rows bytes for the ID vector per segment search. Please preserve an unrestricted sentinel (with a per-file unrestricted/restricted/excluded state for mixed splits), and add a backend-facing test asserting that the no-prefilter fixture uses the unfiltered ANN path.
2 participants
Why
Final step of #755. paimon-rust can run a primary-key vector search it plans itself, but not act as
the execution kernel for an engine that plans elsewhere — the shape Doris needs for an external
Paimon table (apache/doris#65883): planning runs in Paimon Java on the FE, and each bucket is
shipped to a BE that calls in here.
Steps 1-3 (#746, #752, #757) built up to it, but
plan_for_bucket_vector_splitshad no caller, sonone of it was reachable.
How
VectorSearchBuilder::execute_read_for_bucket_splitstakes the splits as the plan — their payloadfiles, per-file row ranges and pinned snapshot are used as given, no index manifest is read.
Everything after planning is shared with
execute_read, so search, refine, Top-K and materializationare not reimplemented and the output is the same. The Top-K is local to the splits passed in; a
caller distributing one call per bucket merges the results itself.
Rejected rather than worked around: a data-evolution column (plans through the global index, so
answering it would use a different plan than the caller supplied), and an empty split list (pins no
snapshot, otherwise indistinguishable from a query that matched nothing).
paimon_vector_search_builder_execute_read_for_bucket_splitsexposes it as a NEW C symbol. Thearray-of-buffers marshalling is the only logic the C layer adds, so a null array, zero count or null
entry is reported as the caller's input error rather than reaching the decoder as corrupt data.
One thing worth a reviewer's attention
"The split route agrees with the manifest route" is not a sufficient test — it is also what a
read that ignored the split and re-planned from the manifest would produce. I checked by substituting
the manifest plan: every obvious assertion still passed.
What separates them is the split's per-file row ranges, which nothing else carries. The fixture's own
rangeFileCountis zero — the asymmetry #755 called out, since Java records a range only for a fileits pre-filter narrowed — so the committed bytes cover the omitted-file normalization, and
restricts_the_read_to_the_splits_row_rangesbuilds the listed case on top. With the manifest plansubstituted, that is the test that fails.
Fixture
Both halves come from one Java run: the table written by the
ivf-flatindexer, and the split bytesits own planner produced over that same table (a split names files by generated UUID, so bytes from
another run would reference files that do not exist). Generated by a new
PkVectorSplitFixtureGeneratorinpaimon-vector, which I'll send to apache/paimon separately; thecommand and config are in the test's header so it can be regenerated. The split embeds an absolute
bucket path, as Java serializes and a real engine ships, so the test stages into a temp dir and
rewrites that one string.
Testing
6 integration tests, 2 C-binding tests.
cargo test -p paimon --lib2586 passed,--features fulltext2661; fmt and clippy clean forpaimonandpaimon-c.Independent of #770 — different files, and this commit's tests pass on
mainwithout it. #770 isstill worth landing first: this C symbol is the first caller to take split bytes from outside the
process, and #770 hardens the decoder those bytes reach.
cargo test -p paimon-cis 72 passed / 1 failed;vector_search_append_filter_returns_invalid_inputfails identically on unmodified
main, so it is pre-existing. Happy to look at it separately.