feat: add synchronous index prewarm C and C++ APIs - #94
Conversation
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The thin binding preserves the SDK’s snapshot, error, and shared-session cache semantics. Local regression and C/C++ runtime tests confirm multi-segment prewarm and cache reuse after reopening handles, with correct query results and zero partition loads. No blocking issues found.
| let version = lance_dataset_version(ds); | ||
| let historical = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), version); | ||
| assert!(!historical.is_null()); | ||
| assert_eq!(lance_dataset_prewarm_index(ds, c"key_idx".as_ptr()), 0); |
There was a problem hiding this comment.
[P2] Assert the scalar index was actually warmed
This test currently proves that the B-tree call succeeds for a multi-segment snapshot, but not that either segment was loaded into the session cache. Could we also query both segments through the same session and assert the relevant cache-hit/load statistics? That would prevent a no-op or existence-only implementation from satisfying the test.
C and C++ callers currently cannot invoke the Rust SDK's index prewarm operation before serving queries. Add
lance_dataset_prewarm_index(dataset, index_name)andDataset::prewarm_index(index_name)as synchronous bindings to the existing SDK method.The operation uses one dataset snapshot and its existing session, prewarms all segments of the named logical index, and preserves the dataset version. It uses the established FFI error/panic handling and rejects NULL, empty, and invalid UTF-8 arguments. The README and public header explain shared-session reuse, eviction, partial failures, snapshot scope, and the distinction between index prewarming and result-row reads. No Lance Rust or dependency changes are required.
Tests cover invalid inputs, missing indexes, multi-segment B-tree and IVF-Flat indexes, historical snapshots after index removal, repeated calls, and dataset/session handle lifetimes. An independent cold query records partition loads; queries from a newly opened handle sharing the prewarmed session return the expected nearest neighbors with zero partition loads. C and C++ runtime tests exercise the exported symbol and wrapper.
Validation:
cargo fmt -- --checkcargo check --all-targetscargo clippy --all-targets -- -D warningscargo test— 479 passed, 0 failed (3 separately executed tests ignored by default)cargo test --test compile_and_run_test -- --ignored --test-threads=1— 3 passed, 0 failedImplements the upstream binding portion of M1 in apache/doris#68692. The Doris dependency update and synchronous SQL integration remain separate follow-up work after this merges.