Add an optional Foyer-backed data-file cache - #73
Draft
zhangstar333 wants to merge 3 commits into
Draft
zhangstar333 wants to merge 3 commits into
zhangstar333 wants to merge 3 commits into
Conversation
zhangstar333
marked this pull request as ready for review
September 3, 2026 05:02
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 3, 2026 05:22
d7c1d5c to
195de5f
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
zhangstar333
marked this pull request as draft
September 3, 2026 05:55
lance-gatekeeper
Bot
removed
K-approved
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 3, 2026 07:35
195de5f to
ceb8f35
Compare
zhangstar333
marked this pull request as ready for review
September 3, 2026 07:35
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 3, 2026 07:46
ceb8f35 to
570a3d0
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 3, 2026 08:13
570a3d0 to
b1d20b5
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
zhangstar333
force-pushed
the
lance_foyer
branch
2 times, most recently
from
September 3, 2026 08:31
b1d20b5 to
a39bc8b
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
lance-gatekeeper
Bot
removed
the
K-risk
This was referenced Sep 30, 2026
Merged
Merged
Contributor
|
The follow-up fixes for cross-origin cache collisions and repeated size-entry disk writes are submitted to this PR's source branch in zhangstar333#3 (source commit Metadata, size, and data blocks are isolated per live underlying store. Because the wrapper API does not expose a complete stable backend identity, new instances and process restarts start cold; this includes Dataset opens that create a new store. The same live store can still reuse cached entries. Warm reads check both hybrid tiers and no longer reinsert unchanged size records. All 461 regular Rust tests passed, including HTTP endpoint isolation and actual disk-write checks (40 KB before versus zero after five warm reads). Apache Doris #68613 and #68615 now consume a patch regenerated from the exact source commit, with complete source-tree equality verified. The separate macOS shell-harness fix remains downstream. All three opt-in native consumer tests also passed: C calls, C++ calls, and static OSS HTTP transport. |
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 30, 2026 07:51
bd8751e to
7538a8b
Compare
zhangstar333
force-pushed
the
lance_foyer
branch
from
September 30, 2026 08:12
7538a8b to
b1ff549
Compare
zhangstar333
marked this pull request as ready for review
September 30, 2026 08:17
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
fix: isolate Foyer cache origins and avoid warm size writes
lance-gatekeeper
Bot
removed
the
K-changes
There was a problem hiding this comment.
The cross-origin blocker is fixed by the merged source-branch follow-up. The isolation, disk-recovery, warm-disk-write, repeated-scan, and restore-statistics regressions pass.
New store instances and process restarts start cold, so the cache helps most with long-lived dataset handles. Cold streamed reads still issue one sequential origin request per cache block, and concurrent misses are not coalesced; measure the intended remote workload with the chosen block size before broad rollout. Treat cached bytes as disposable. No further change is required for this revision.
lance-gatekeeper
Bot
added
K-approved
zhangstar333
marked this pull request as draft
September 30, 2026 09:17
lance-gatekeeper
Bot
removed
K-approved
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Sep 30, 2026
…68613) Unfiltered ANN searches scoped to complete index segments can spend time materializing redundant row-ID allowlists. Update lance-c to an immutable upstream revision containing the merged Lance development-branch fix lance-format/lance#9599 and ANN diagnostics from lance-format/lance#9602, through lance-format/lance-c#89. Keep all master changes in `thirdparty/`. Remove the superseded local lance-c patch chain and retain the Foyer cache integration while its interface remains outside upstream main. The upstream revision also carries the merged lance-format/lance#9537 PQ scoring change and ANN stage timing diagnostics; no Doris FE or BE code changes are included here. The prefilter fast path requires complete visible coverage of every selected segment and no predicate. Partial coverage and predicates retain row-ID prefiltering, while deletions and unindexed fallback remain effective. Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This includes parallel-path partition-preparation timing. The upstream callback regression checks timer presence and nanosecond units while accepting valid zero durations. The Foyer wrapper caches immutable ObjectMeta and Attributes independently of HTTP response extensions, avoiding repeated HEAD requests on warm range reads. It does not replay transport extensions from cached metadata. The HTTP regression checks HEAD/GET counts, returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD bypass. Generate the retained Foyer patch from source commit `24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned lance-c baseline. The header records both revisions and the regeneration command. The source is submitted as zhangstar333/lance-c#3 against the branch behind lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris consumes the exact reviewed source revision while that supplementary PR awaits merge. Scope metadata, size, and block keys to a random namespace for each live underlying object-store instance. The wrapping API omits a complete stable backend identity; identical bucket/path names alone cannot distinguish S3-compatible endpoints. Weak identity records preserve sharing for the same live store without retaining it, while new stores and process restarts start cold. A fresh Dataset open that creates a new store consequently needs to warm its own cache. This deliberately reduces reuse across instances to prevent returning another origin's data; it does not claim unchanged cache-hit rates or production latency. Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid cache: look up both tiers first and populate only absent or invalid size records. Regression tests cover real HTTP endpoints sharing bucket/path/ETag, batched data and NotFound isolation, replaced origins after disk recovery, weak-reference lifetime, and actual disk-write bytes. Five warm range reads wrote 40 KB before the fix and zero bytes after it in the regression. Fingerprint the patch in the downloader so existing source trees refresh after updates, including legacy empty markers. Identical patches reuse cached sources. Check platform definitions under nounset with simulated Darwin x86_64/arm64, and make the optional ADBC source guard safe when unset on master. Downloader lifecycle and platform handling remain downstream in Doris. Validation: 461 regular Rust tests passed on the final source; Rust formatting and diff checks passed. GNU patch and git apply checks passed, and all 87 tracked source files match the recorded source commit. Downloader tests passed on master, branch-4.1, and the downstream hotfix branch for fresh extraction, idempotence, re-extraction, generic markers, legacy/mismatched Foyer markers, and patch failure. Shell syntax and simulated macOS initialization passed. All three opt-in native consumer tests passed: C calls, C++ calls, and static OSS HTTP transport. Full Doris builds and BE integration execution remain in PR CI.
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Sep 30, 2026
…nch-4.1) (#68615) Unfiltered ANN searches scoped to complete index segments can materialize redundant row-ID allowlists, and the existing scan profile leaves the remaining search work unexplained. Update the upstream dependency and expose ANN stage timings on `branch-4.1`. This includes the dependency integration from #68613: adopt the merged lance-format/lance#9599 and lance-format/lance#9602 through lance-format/lance-c#89, retain the merged lance-format/lance#9537 PQ scoring fix, remove superseded local lance-c patches, and retain only the Foyer cache integration. Add BE profile counters for index opening, partition loading/preparation, prefilter readiness, CPU queue wait, partition search, query lookup-table preparation, fused distance/TopK work, and result materialization. Export operator baselines for ANN, sort/merge, take, and vector-distance work. Extend the existing indexed multivector regression to check that stage timers reach the Doris profile, alongside existing result and prefilter assertions. Timings accumulate across concurrent work and overlap parent stages. Distance/TopK includes candidate filtering in fused paths; these counters must not be summed to reconstruct wall time. `docs/lance-ann-profile.md` documents the boundaries, supported paths, and relationship to the existing scanner and prefilter timers. The BE changes add observability; the prefilter optimization itself remains in the upstream dependency. Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This includes parallel-path partition-preparation timing. The upstream callback regression checks timer presence and nanosecond units while accepting valid zero durations. The Foyer wrapper caches immutable ObjectMeta and Attributes independently of HTTP response extensions, avoiding repeated HEAD requests on warm range reads. It does not replay transport extensions from cached metadata. The HTTP regression checks HEAD/GET counts, returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD bypass. Generate the retained Foyer patch from source commit `24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned lance-c baseline. The header records both revisions and the regeneration command. The source is submitted as zhangstar333/lance-c#3 against the branch behind lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris consumes the exact reviewed source revision while that supplementary PR awaits merge. Scope metadata, size, and block keys to a random namespace for each live underlying object-store instance. The wrapping API omits a complete stable backend identity; identical bucket/path names alone cannot distinguish S3-compatible endpoints. Weak identity records preserve sharing for the same live store without retaining it, while new stores and process restarts start cold. A fresh Dataset open that creates a new store consequently needs to warm its own cache. This deliberately reduces reuse across instances to prevent returning another origin's data; it does not claim unchanged cache-hit rates or production latency. Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid cache: look up both tiers first and populate only absent or invalid size records. Regression tests cover real HTTP endpoints sharing bucket/path/ETag, batched data and NotFound isolation, replaced origins after disk recovery, weak-reference lifetime, and actual disk-write bytes. Five warm range reads wrote 40 KB before the fix and zero bytes after it in the regression. Fingerprint the patch in the downloader so existing source trees refresh after updates, including legacy empty markers. Identical patches reuse cached sources. Check platform definitions under nounset with simulated Darwin x86_64/arm64, and make the optional ADBC source guard safe when unset on master. Downloader lifecycle and platform handling remain downstream in Doris. Validation: 461 regular Rust tests passed on the final source; Rust formatting and diff checks passed. GNU patch and git apply checks passed, and all 87 tracked source files match the recorded source commit. Downloader tests passed on master, branch-4.1, and the downstream hotfix branch for fresh extraction, idempotence, re-extraction, generic markers, legacy/mismatched Foyer markers, and patch failure. Shell syntax and simulated macOS initialization passed. All three opt-in native consumer tests passed: C calls, C++ calls, and static OSS HTTP transport. Full Doris builds and BE integration execution remain in PR CI.
This was referenced Sep 30, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#71 not merge to main branch, only as patch.