Repository navigation
Conversation
An incremental read walks back from `to` to its checkpoint `from`. It looked `from` up too, so a checkpoint that expiry removed failed the read even when its child was kept, and a gap in the history failed with the same NotFound as a snapshot that was never an ancestor. The walk now stops at the child of `from`, and a missing snapshot other than `to` is the new Error::TruncatedHistory that ADR 0009 calls for. The walk is its own iterator so the plan-time range check can share it. files_added_by read only the manifests written inside the range and kept only Added entries. A replace that rewrites a manifest carries a file forward as Existing under the id of the snapshot that added it, so those rows were dropped. It now reads every manifest `to` lists and keeps the live entries added in the range. Part of #89 Signed-off-by: Mayur Kale <hello@mayurkale.com>
8 tasks
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 33 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #115 +/- ##
==========================================
+ Coverage 99.04% 99.08% +0.03%
==========================================
Files 16 16
Lines 2521 2626 +105
==========================================
+ Hits 2497 2602 +105
Misses 24 24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The storage half of #89: an incremental read now tells expired history
apart from a snapshot that was never an ancestor, and finds rows that a
manifest rewrite carried forward.
Part of #89
How
infostash_model::ErrorgainsTruncatedHistory, the variant ADR 0009says engine: fall back to a full refresh at plan time and fix files_added_by #89 adds. It is never retried; the reader must read the table in
full.
toback tofromis a private iterator,ancestors_after, whichappends_betweenuses and the plan-time checkwill share. It looks
toup first, as before, stops at the child offromwithout lookingfromup, and turns a missing ancestor intoTruncatedHistory. A parent ofNoneis still the non-ancestorNotFound, and the non-append refusal is unchanged.files_added_byreads every manifest thattolists and keeps thelive entries (
AddedorExisting) whose snapshot is in the range. Itused to skip manifests written outside the range and keep only
Addedentries, which drops a file a later replace carries forward as
Existing.Things to know
plan-time range function, the full refresh, removing the compile-time
check of compiler: add an incremental eligibility classifier with a function allow-list #84 and the non-retryable outcome in
crates/dist.incrementally instead of failing.
files_ofkeeps its meaning: anExistingentrynever carries the id of the snapshot that lists it, so asking for one
commit's files still returns only what it added.
Tests
files_added_in_the_range_are_found_in_a_rewritten_manifest: two filesadded in one manifest, then a replace removes one, rewriting the
manifest. The range holding the first replace returns exactly the kept
file: not the removed one, not the files it deleted, and nothing from
before the range.
a_range_whose_history_expired_is_truncated_not_missing: four appendswith expiry simulated by dropping snapshots from the metadata. Dropping
older snapshots or the checkpoint alone still reads the range; dropping
the checkpoint and its child gives
TruncatedHistory; a made-upfromand a missing
tostayNotFound.