Skip to content

feat: add synchronous index prewarm C and C++ APIs - #94

Merged
yanghua merged 1 commit into
lance-format:mainfrom
Gabriel39:dev/synchronous-index-prewarm
Oct 1, 2026
Merged

yanghua merged 1 commit into
lance-format:mainfrom
Gabriel39:dev/synchronous-index-prewarm

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

C and C++ callers currently cannot invoke the Rust SDK's index prewarm operation before serving queries. Add lance_dataset_prewarm_index(dataset, index_name) and Dataset::prewarm_index(index_name) as synchronous bindings to the existing SDK method.

The operation uses one dataset snapshot and its existing session, prewarms all segments of the named logical index, and preserves the dataset version. It uses the established FFI error/panic handling and rejects NULL, empty, and invalid UTF-8 arguments. The README and public header explain shared-session reuse, eviction, partial failures, snapshot scope, and the distinction between index prewarming and result-row reads. No Lance Rust or dependency changes are required.

Tests cover invalid inputs, missing indexes, multi-segment B-tree and IVF-Flat indexes, historical snapshots after index removal, repeated calls, and dataset/session handle lifetimes. An independent cold query records partition loads; queries from a newly opened handle sharing the prewarmed session return the expected nearest neighbors with zero partition loads. C and C++ runtime tests exercise the exported symbol and wrapper.

Validation:

  • cargo fmt -- --check
  • cargo check --all-targets
  • cargo clippy --all-targets -- -D warnings
  • cargo test — 479 passed, 0 failed (3 separately executed tests ignored by default)
  • cargo test --test compile_and_run_test -- --ignored --test-threads=1 — 3 passed, 0 failed

Implements the upstream binding portion of M1 in apache/doris#68692. The Doris dependency update and synchronous SQL integration remain separate follow-up work after this merges.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The thin binding preserves the SDK’s snapshot, error, and shared-session cache semantics. Local regression and C/C++ runtime tests confirm multi-segment prewarm and cache reuse after reopening handles, with correct query results and zero partition loads. No blocking issues found.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 1, 2026

@yanghua yanghua left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a non-blocked nit.

Comment thread tests/c_api_test.rs
let version = lance_dataset_version(ds);
let historical = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), version);
assert!(!historical.is_null());
assert_eq!(lance_dataset_prewarm_index(ds, c"key_idx".as_ptr()), 0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Assert the scalar index was actually warmed

This test currently proves that the B-tree call succeeds for a multi-segment snapshot, but not that either segment was loaded into the session cache. Could we also query both segments through the same session and assert the relevant cache-hit/load statistics? That would prevent a no-op or existence-only implementation from satisfying the test.

@yanghua
yanghua merged commit 9846834 into lance-format:main Oct 1, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants