fix(index): filter stale vector segment rows before top-k

[lance-gatefixer]

Root cause

A physical vector-index segment can retain pre-update rows after its fragment ownership bitmap is pruned. Because ownership was applied only after the sub-index local top-k, those stale rows could displace closer rows that the segment still owns, and no later filter could recover the missing candidates.

The initial repair classified stale segments by loading the historical dataset manifest during every query. That added object-store reads after index prewarming and did not account for later inputs incorporated into a merged segment.

Fix

• Intersect the shared dataset prefilter with each stale segment ownership mask before both partition and streaming sub-index searches.
• Persist immutable physical fragment coverage in vector segment details when segments are built, and union all source provenance in both merge paths.
• Compare physical coverage with current ownership without loading historical manifests.
• Use conservative ownership filtering for legacy or remapped segments whose exact physical provenance is unavailable.
• Preserve provenance when legacy vector details are inferred without exposing it in user-facing index configuration.

Performance safeguard

Append-only deltas with known physical coverage retain an empty prefilter and the existing unfiltered sub-index fast paths. Classification performs no object-store I/O, so a query after index prewarming remains at zero reads.

Validation

• cargo test -p lance test_prewarm_ivf_pq_multiple_deltas -- --nocapture
• cargo test -p lance test_append_only_deltas_keep_empty_prefilter_fast_path -- --nocapture
• cargo test -p lance test_merge_index_metadata_reports_progress -- --nocapture
• cargo test -p lance test_query_delta_indices -- --nocapture (5 passed)
• cargo test -p lance test_physical_coverage_does_not_suppress_details_inference -- --nocapture
• uv run pytest python/tests/test_vector_index.py::test_segment_ownership_filter_precedes_partition_topk (2 passed)
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings
• make build
• uv run make lint

Fixes #8348

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The masking decision needs each segment's complete physical coverage, including fragments incorporated by segment merges. Preserving immutable physical-coverage provenance when segments are built or merged, with a conservative cached fallback for legacy segments, would let queries apply the segment ownership mask exactly where coverage diverges while retaining append-only fast paths.

[lance-gatekeeper]

Detection misses stale rows incorporated from a later input of a merged segment. Vector merging physically unions every source bitmap, but both merge paths record the minimum source dataset_version; this loop therefore inspects only fragments in the oldest snapshot. If a later fragment is updated and pruned from ownership, this returns false, so its stale physical rows can again displace owned candidates in the segment-local top-k.

Please retain and consult immutable physical coverage for each segment (including all merge inputs), applying global mask ∩ current ownership whenever it diverges. Unknown legacy provenance can use a cached conservative validation/filter path; that also avoids rescanning historical manifests per append-only segment on every query.

Executed reproducer

I temporarily added this test beside the new fast-path test:

#[tokio::test]
async fn gate_repro_merged_segment_later_fragment_is_not_inspected() {
    let first = lance_datagen::gen_batch()
        .col("vector", array::rand_vec::<Float32Type>(lance_datagen::Dimension::from(4)))
        .into_reader_rows(RowCount::from(20), BatchCount::from(1));
    let mut dataset = Dataset::write(first, "memory://", None).await.unwrap();
    let oldest_version = dataset.manifest.version;
    let first_fragments = dataset.fragment_bitmap.as_ref().clone();
    let field_id = dataset.schema().field("vector").unwrap().id;

    let later = lance_datagen::gen_batch()
        .col("vector", array::rand_vec::<Float32Type>(lance_datagen::Dimension::from(4)))
        .into_reader_rows(RowCount::from(20), BatchCount::from(1));
    dataset = Dataset::write(
        later,
        "memory://",
        Some(WriteParams { mode: WriteMode::Append, ..Default::default() }),
    ).await.unwrap();

    let merged_after_later_fragment_update = IndexMetadata {
        uuid: Uuid::new_v4(),
        fields: vec![field_id],
        name: "vector_idx".to_string(),
        dataset_version: oldest_version,
        fragment_bitmap: Some(first_fragments),
        index_details: None,
        index_version: 0,
        created_at: None,
        base_id: None,
        files: None,
    };
    assert!(
        segment_needs_ownership_filter(&dataset, &merged_after_later_fragment_update).await,
        "a merged segment's later physical fragments must be considered",
    );
}

CARGO_TARGET_DIR=/home/agent/tmp/pr8351-target cargo test --locked -p lance gate_repro_merged_segment_later_fragment_is_not_inspected -- --nocapture

Expected: the helper returns true. Observed: the assertion failed because it returned false. The temporary test was then removed and the checkout restored clean.

[lance-gatefixer]

Addressed in 556894e. Vector segment details now retain immutable physical fragment coverage at build time, both merge paths union every source coverage, and query classification compares that provenance with current ownership without historical manifest reads. Legacy or remapped segments with unknown provenance use the conservative ownership filter. Focused coverage now exercises merged later fragments, append-only fast paths, merge preservation, and details inference.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The persisted physical-coverage model fixes the earlier historical-snapshot gap, but it still loses correctness across segment merging and fragment-ID remapping, and coverage-only metadata is misread by older Lance readers.

A complete revision should filter each merge input by its own current ownership before coalescing, remap or invalidate provenance whenever fragment IDs change, and persist provenance without changing the established empty vector-details sentinel.

[lance-gatekeeper]

This set union cannot represent that one source holds a stale copy of a fragment also present in a fresh source. Both merge implementations copy raw source rows without applying each source segment's ownership, so stale physical {F} / owned {} plus fresh {F} / owned {F} becomes one merged segment with physical = owned = {F}. The query classifier then disables the mask, and stale vectors can fill the local top-k again.

Filter each source by its ownership while writing the merged partitions, or rebuild from current rows. A post-merge bitmap cannot recover which duplicate came from the stale source.

Reproducer

I wrote two 20-row fragments (A vectors [1;4], B vectors [0;4]), built one-partition IVF_FLAT, rewrote B to [10;4], created its fresh delta with OptimizeOptions::append(), queried, merged both segments with OptimizeOptions::merge(2), and queried again.

CARGO_TARGET_DIR=/home/agent/tmp/pr8351-verify-target CARGO_INCREMENTAL=0 cargo run --locked

Expected both queries to return A at distance 4. Observed:

BEFORE_MERGE_IDS=[0, 1, 2, 3, 4]
BEFORE_MERGE_DISTS=[4.0, 4.0, 4.0, 4.0, 4.0]
AFTER_MERGE_IDS=[100, 101, 102, 103, 104]
AFTER_MERGE_DISTS=[0.0, 0.0, 0.0, 0.0, 0.0]

The reproducer ran on the immediate parent 556894e7; this head changes only #[cfg(test)] code in create.rs, so the exercised production path is unchanged.

[lance-gatefixer]

Addressed in 2e0f73b. Stale or unknown optimize inputs are rebuilt from current owned rows, while direct segment merges now filter every source before writing each partition (including exact stable-row-ID filters) and recompute the written partition lengths.

[lance-gatefixer]

No additional code change was needed at current head 6b8f4a09c: direct vector segment merges filter every source by exact ownership before writing, while stale or unknown optimize inputs rebuild from current owned rows. The per-source ownership regression passes.

[lance-gatefixer]

No additional code change was needed at current head 1c7ff252f: direct vector segment merges still filter each source by exact ownership, while unsafe optimize inputs rebuild from current rows. The per-source ownership regression passes on this head.

[lance-gatefixer]

No code change was needed at current head ca23eed: ordinary vector optimize applies exact per-source stable-row-ID filters before merging, while inputs with unsafe physical provenance still rebuild from current rows. test_vector_merge_filters_stable_row_id_replacements passes on this head.

[lance-gatekeeper]

This comparison assumes physical coverage, ownership, and the current dataset use the same fragment-ID domain. They do not after index reuse: the fragment-reuse projection remaps only ownership, and stable-row-ID compaction likewise moves ownership to new fragment IDs while reusing the index and its old physical bitmap. After a later update prunes the new ID, physical={0}, owned={}, current={1} makes this intersection empty, so the stale segment is searched without its ownership mask.

Remap physical provenance alongside ownership, or clear it to unknown at every fragment-ID remap/reuse boundary so the conservative path is used.

Reproducer

I executed the exact decision expression with the post-remap state:

let current = RoaringBitmap::from_iter([1]);
let owned = RoaringBitmap::new();
let physical = RoaringBitmap::from_iter([0]);
let unowned_physical = &physical - &owned;
let actual = unowned_physical.intersection_len(&current) > 0;
assert!(!actual); // observed current behavior

Output:

(physical-owned).intersection_len(current)=0
segment_needs_ownership_filter=false
filtering_required_for_remapped_then_updated_rows=true

The current head changes only tests outside this path.

[lance-gatefixer]

Addressed in 2e0f73b. Fragment-reuse projection now invalidates physical coverage whenever ownership is remapped, preserving the real vector configuration while forcing the conservative path in the new fragment-ID domain; the persistent compaction remap paths do the same.

[lance-gatefixer]

No additional code change was needed at current head 6b8f4a09c: fragment reuse invalidates physical coverage when an applicable mapping can touch immutable coverage, preserving the conservative path across fragment-ID domains. The affected and unaffected provenance regression passes.

[lance-gatefixer]

No additional code change was needed at current head 1c7ff252f: applicable fragment-reuse mappings still invalidate physical provenance so remapped segments take the conservative path, while disjoint mappings preserve exact coverage. The provenance regression passes on this head.

[lance-gatefixer]

No code change was needed at current head ca23eed: applicable fragment-reuse mappings invalidate physical provenance and take the conservative path, while older or disjoint mappings preserve exact coverage. test_fragment_reuse_projection_invalidates_only_affected_vector_coverage passes on this head.

[lance-gatekeeper]

This turns the established empty vector-details sentinel into a nonempty configuration payload. This merge path starts from vector_index_details_default(), so field 10 is the only serialized value. Older readers see a nonempty Any.value and skip disk inference; after dropping the unknown field they decode the known defaults as L2 / IVF_FLAT. A newly merged Cosine/PQ index can therefore drive incorrect metric planning in those readers.

Keep physical provenance outside VectorIndexDetails (for example, as optional index metadata), or at minimum never emit a coverage-only details payload and preserve the real source configuration here.

Reproducer

A temporary binary constructed this exact coverage-only payload and applied the base reader logic (Any.value.is_empty(), followed by decoding the known fields). It exited successfully with:

source=COSINE/IVF_PQ any_value_len=22 prior_needs_inference=false prior_metric=L2 prior_type=IVF_FLAT

The current head does not change this writer or protobuf path.

[lance-gatefixer]

Addressed in 2e0f73b. Physical coverage is no longer serialized into an otherwise empty VectorIndexDetails payload, and direct merges preserve a configured source payload before adding coverage. Legacy/default details therefore remain the established empty sentinel for older readers.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The fragment-level correctness fixes are effective, but vector segment merging still needs exact per-source ownership without converting fast paths into full retraining or unbounded buffering. Preserve that contract at the merge-input boundary, and rebuild only when provenance genuinely cannot prove safety.

[lance-gatekeeper]

This rebuild decision misses row-level ownership changes inside a still-owned fragment. With stable row IDs, MERGE-INSERT can delete old rows in that fragment and append replacements with the same IDs. Here physical coverage still equals owned coverage, so this returns false; the raw optimize merge coalesces both copies, after which the unified live-ID mask admits stale vectors.

Apply each source segment's exact live-row-ID filter before the ordinary optimize merge, as the direct auxiliary merge now does, or source-rebuild selected stable-ID segments whose owned fragments carry deletions.

Reproducer

I added an isolated Rust regression with 40 stable-ID rows: IDs 20–39 started at vector [0;4], MERGE-INSERT updated them to [10;4], and IDs 0–19 remained at [1;4]. It then ran append followed by OptimizeOptions::merge(2).

CARGO_TARGET_DIR=/home/agent/tmp/pr8351-focused.pviaMS/target cargo test -p lance test_review_repro_stable_row_id_merge_insert_filters_stale_vectors -- --nocapture

On this head, one run found 60 rows in the merged index for the 40-row dataset (left: 60, right: 40). With the query assertion first, the zero-vector query returned stale IDs [20, 21, 22, 23, 24]; expected IDs were <20 at distance 4.

[lance-gatefixer]

Addressed in 6b8f4a0. Stable-row-ID optimize merges now source-rebuild selected vector segments whose owned fragments carry deletion vectors, so merge-insert replacements are indexed once from current rows. The regression verifies 40 physical index rows and excludes stale zero-vector neighbors.

[lance-gatefixer]

Addressed in 6b8f4a09c. Stable-row-ID optimize merges rebuild selected vector segments whose owned fragments carry deletions, so replacement rows are indexed once from current data. The regression confirms 40 physical rows and excludes stale neighbors.

[lance-gatekeeper]

This clears exact provenance for every vector segment whenever any fragment-reuse index exists, even when no reuse mapping can touch the segment. Missing provenance then makes segment_merge_requires_rebuild return true, turning a suffix merge or rebalance into a full source scan and independent vector retraining.

Invalidate only for reuse versions that could apply to the immutable physical fragment chain (or that are not strictly older than the segment); preserve provenance for demonstrably disjoint or strictly newer segments. Comparing only the mutable ownership bitmap is insufficient because it may already have been remapped.

Reproducer

An isolated unit used a vector segment at dataset version 3 with owned/physical coverage {10} and an FRI version 2 mapping {0} to {1}.

CARGO_TARGET_DIR=/home/agent/tmp/remap-compat-target CARGO_INCREMENTAL=0 cargo test --locked -p lance gate_repro_fragment_reuse_preserves_unaffected_new_vector_coverage -- --nocapture

The ownership remained {10}, but the provenance assertion failed with left: None, right: Some(RoaringBitmap<[10]>).

[lance-gatefixer]

Addressed in 6b8f4a0. Fragment reuse now invalidates provenance only when a reuse version at or after the segment build can touch immutable physical coverage; strictly older or disjoint mappings preserve exact coverage. The regression covers a version-3 segment on {10} against the version-2 mapping {0}->{1}.

[lance-gatefixer]

Addressed in 6b8f4a09c. Fragment reuse now invalidates provenance only when a mapping at or after segment creation can touch immutable physical coverage; older or disjoint mappings preserve exact coverage. The selective-invalidation regression passes.

[lance-gatekeeper]

This replaces the default FLAT/SQ/HNSW per-partition streaming path with windows that materialize 512 partitions across every shard. ShardMergeReader can retain the current window plus a prefetched window, while shard reads within each window use unbounded fanout. At one target-sized 4,096 rows per partition and dimension 1,536, a single IVF_FLAT window holds 2,097,152 vectors, about 12 GiB of float data; two resident windows are about 24 GiB before Arrow overhead. This makes ordinary distributed merges vulnerable to OOM.

Keep the prior streaming writer when no filter is required. For filtered sources, stream/filter a bounded partition or batch at a time and accumulate only the emitted partition lengths.

Capacity check

python3 -c 'rows=512*4096; gib=rows*1536*4/1024**3; print(f"rows_per_window={rows} float32_vector_gib={gib:.1f} two_windows_gib={2*gib:.1f}")'

Output: rows_per_window=2097152 float32_vector_gib=12.0 two_windows_gib=24.0.

[lance-gatefixer]

Addressed in 6b8f4a0. FLAT, SQ, and HNSW auxiliary merges again stream one shard partition at a time: unfiltered rows use the existing streaming writer, while filtered rows are masked and written batch-by-batch with emitted lengths accumulated. Filtered and unfiltered IVF_FLAT merge regressions pass.

[lance-gatefixer]

Addressed in 6b8f4a09c. FLAT, SQ, and HNSW auxiliary merges stream one shard partition at a time: unfiltered rows keep the existing streaming writer, while filtered rows are masked and written batch by batch. Both filtered and unfiltered IVF_FLAT regressions pass.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The replacement-row and provenance fixes are now correct, but the stable-row-ID rebuild guard still turns already-clean vector segments into fresh scans and retraining on later merges.

Use deletion provenance to rebuild only when a covered deletion is at least as new as the segment; a segment built after that deletion already filtered those rows and should retain the auxiliary merge fast path.

[lance-gatekeeper]

Any covered deletion file triggers this branch even when the index was built afterward. In that case deletion_file.read_version < index.dataset_version, so the segment was created from a snapshot that already excludes those deleted rows; nevertheless both merge paths call build_fresh_vector_segment, which independently scans/retrains the segment despite OptimizeOptions::retrain == false. This makes every later merge involving that clean segment pay full rebuild cost.

A focused regression on this head creates a stable-row-ID dataset, deletes id < 20, builds IVF_FLAT after the delete, verifies deletion_file.read_version < index.dataset_version, and expects this helper to return false. Running cargo test --locked -p lance gate_repro_fresh_stable_index_after_deletion_keeps_merge_fast_path -- --nocapture fails at that final assertion. Please compare deletion provenance to index.dataset_version (and remain conservative only when provenance cannot establish ordering) so only deletions the segment could not have incorporated force a rebuild.

[lance-gatefixer]

Addressed in 1c7ff252f. Stable-row-ID segments now rebuild only when a covered deletion has read_version >= index.dataset_version; deletions already incorporated when the segment was built retain the auxiliary merge fast path. The new pre-build deletion regression and the existing post-build replacement regression both pass.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The version-aware rebuild guard now preserves auxiliary merging for stable-row-ID segments that already incorporate a deletion while still rebuilding segments that predate replacement rows. The focused fast-path and stale-replacement regressions both pass, and the earlier physical-coverage, FRI, and bounded per-source merge fixes remain intact.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Per-segment ownership filters now remove stale stable-row-ID postings without forcing fresh rebuilds, so this revision preserves vector merge topology while retaining replacement-row correctness. Physical-provenance mismatches still take the conservative rebuild path.