From c9a620642f48ee6fae3831a8304141d260a09817 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 10:44:31 +0300 Subject: [PATCH 1/4] feat(repo): sparse widen on lazy partial clones (P37b-ii, ADR-0048) - fill: `fill_view` fills the closure of a sparse view under a root level by level, in id batches of at most 1024; `ensure_view` runs it on a lazy, online partial clone before any ref, spec or file mutation. It is called by switch, adopt and fast-forward merge, undo and `sparse set`, outside the store lock; anything else is left to the no-network materialize preflight. - `sparse set` may widen past the promisor filter on a lazy clone (other partial clones still refuse). materialize takes the spec that was in effect as the bound for its removal walk, replacing the filter-prefix bound, so a widened view can be narrowed again. - by-id replies are capped at min(--max-pack-size, 512 MiB), 0 meaning 512 MiB, on the wire server and LocalTransport. A refused batch is split in half and retried; a refused single is final. A pre-stream PackTooLarge refusal no longer marks a StdioTransport desynced, so the split retry reuses the connection. - demo/run_lazy_fetch_demo.sh covers widen, commit, push, a second clone, gc, verify and an offline undo. Co-Authored-By: Claude Sonnet 5.5 --- CLAUDE.md | 6 +- ROADMAP.md | 7 +- crates/repo/src/fill.rs | 435 +++++++++++++++++- crates/repo/src/oplog.rs | 2 + crates/repo/src/partial_walks.rs | 66 +++ crates/repo/src/repo.rs | 17 +- crates/repo/src/sparse.rs | 20 +- crates/repo/src/stdio_transport.rs | 132 +++++- crates/repo/src/transport.rs | 29 +- crates/repo/src/wire.rs | 7 +- crates/repo/src/worktree.rs | 42 +- demo/run_lazy_fetch_demo.sh | 37 +- .../adr/0048-presence-based-partial-clones.md | 19 +- docs/adr/README.md | 2 +- 14 files changed, 771 insertions(+), 50 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 10af20d0..ab12c46e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,7 @@ the project tree if desired. ## Capability map (what's built, by phase) -All 36 phases, P37a and P37b-i are built and tested. One line of current fact per phase; the +All 36 phases, P37a, P37b-i and P37b-ii are built and tested. One line of current fact per phase; the authoritative rationale and full semantics live in the linked ADR, the design in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not @@ -152,7 +152,7 @@ the code, those win. | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | -| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. Sparse widen (37b-ii) and merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) | +| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. Merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) | ## Standing boundaries & gotchas @@ -169,7 +169,7 @@ transport-adjacent. The rest, imperatively: capability, and no DTO field capable of carrying protected ciphertext or identity material. - **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`, - `sc work`, `sc export`, `sparse disable`, an empty `sparse set`, `sc rewrap`, + `sc work`, `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`, `sc grant`/`revoke`, leaving a private branch, `sc push` to a git remote, and `sc fetch` from any native remote but the promisor origin.** `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what diff --git a/ROADMAP.md b/ROADMAP.md index 300ae641..acdaf7e2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -548,12 +548,13 @@ across every phase. ## Active -**P37b — presence-based walks and the lifted P27 refusals (37b-i built).** +**P37b — presence-based walks and the lifted P27 refusals (37b-i, 37b-ii built).** P37a (built) added the batched `GetObjects` verb and `sc cat`. 37b-i (built) made push, verify and gc ask "is this object present?" instead of "is this path in the filter?", which also fixed gc and fast-forward failing after an -ordinary fetch on a partial clone; the remaining work is sparse widen over a -filled prefix (37b-ii). Merge, rebase and sessions on partial clones are directions +ordinary fetch on a partial clone. 37b-ii (built) added sparse widen on a lazy +clone: the view closure is filled by id before switch, fast-forward, undo and +`sparse set`, and a by-id reply is capped and split on refusal. Merge, rebase and sessions on partial clones are directions with constraints, each needing its own design record; `export` and `sparse disable` stay refused. Design and open questions in [ADR-0048](docs/adr/0048-presence-based-partial-clones.md). diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index 4865bfa4..d92ae362 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -11,13 +11,14 @@ //! unchanged, so push, gc and verify (all filter-scoped walks) behave exactly //! as they did before the fill; they simply find a few extra present objects. -use std::collections::BTreeSet; +use std::collections::{BTreeSet, HashSet}; use std::sync::Arc; use scl_core::{EntryKind, Object, ObjectId}; use crate::error::{Error, Result}; use crate::repo::Repo; +use crate::sparse::Sparse; use crate::stdio_transport::open_transport; use crate::transport::{ingest_pack_file_checked, TempPackGuard, Transport}; @@ -95,6 +96,76 @@ impl Repo { Ok(ingested.len()) } + /// Fetch `ids` from `transport` in as many round trips as it takes: batches + /// of at most [`crate::wire::MAX_GET_OBJECTS_IDS`], and a batch the server + /// refuses as over its reply cap is split in half and retried. A batch of + /// one that is still refused is final (an object above the server's cap + /// cannot be fetched by id), so splitting never loops. + fn fetch_ids_batched( + &self, + transport: &dyn Transport, + ids: &[ObjectId], + stats: &mut FillStats, + ) -> Result<()> { + for chunk in ids.chunks(crate::wire::MAX_GET_OBJECTS_IDS) { + self.fetch_ids_split(transport, chunk, stats)?; + } + Ok(()) + } + + fn fetch_ids_split( + &self, + transport: &dyn Transport, + ids: &[ObjectId], + stats: &mut FillStats, + ) -> Result<()> { + match self.fetch_by_id_from(transport, ids) { + Ok(n) => { + stats.requests += 1; + stats.objects += n; + Ok(()) + } + Err(Error::PackTooLarge(_)) if ids.len() > 1 => { + let (left, right) = ids.split_at(ids.len() / 2); + self.fetch_ids_split(transport, left, stats)?; + self.fetch_ids_split(transport, right, stats) + } + Err(e) => Err(e), + } + } + + /// Make the closure of `sparse`'s view under `root` present before an + /// operation that lays the tree out (ADR-0048, the view closure + /// obligation). A lazy, online partial clone fetches what is missing by + /// id, level by level; anything else returns `Ok` and leaves the no-network + /// preflight inside `materialize` to report a gap before it writes. Call + /// it BEFORE any ref, spec or working-tree mutation, and not while holding + /// the store lock: the fill takes it. + pub(crate) fn ensure_view(&self, root: ObjectId, sparse: &Sparse) -> Result { + self.ensure_view_impl(root, sparse, crate::promisor::offline()) + } + + /// `offline` is a parameter so tests can exercise `SC_OFFLINE` without + /// mutating the process environment. + fn ensure_view_impl( + &self, + root: ObjectId, + sparse: &Sparse, + offline: bool, + ) -> Result { + // An empty spec is a full checkout: never fill the whole tree on a + // partial clone's behalf (the preflight reports the gap instead). + if sparse.is_full() { + return Ok(FillStats::default()); + } + let mut filler = Filler::new(self, offline)?; + if filler.origin.is_none() { + return Ok(FillStats::default()); + } + filler.fill_view(root, sparse)?; + Ok(filler.stats) + } + /// The bytes of the file at `path` as of `refname`'s tip (`sc cat /// :`). On a lazy partial clone, trees and the blob missing /// locally are fetched by id on the way down (one round trip per tree @@ -227,11 +298,92 @@ impl<'a> Filler<'a> { self.stats.objects += n; Ok(()) } + + /// Ensure every id in `ids` is in the local store, fetching the absent + /// ones in batches. `path` names what the caller is laying out, for the + /// gap error when filling is not allowed. + fn ensure_many(&mut self, ids: &BTreeSet, path: &str) -> Result<()> { + let absent: Vec = { + let store_arc = self.repo.vfs.store(); + let store = store_arc.lock().unwrap(); + ids.iter() + .copied() + .filter(|id| !store.contains(id)) + .collect() + }; + let Some(first) = absent.first() else { + return Ok(()); + }; + let Some(origin) = self.origin.clone() else { + return Err(if self.repo.promisor()?.is_some() { + crate::promisor::partial_gap_hint(path) + } else { + Error::Core(scl_core::Error::NotFound(*first)) + }); + }; + if self.transport.is_none() { + self.transport = Some(open_transport(&origin)?); + } + let transport = self.transport.as_deref().expect("opened above"); + self.repo + .fetch_ids_batched(transport, &absent, &mut self.stats) + } + + /// Fill the closure of `sparse`'s view under `root`, one tree depth per + /// round (ADR-0048). Each round reads every tree of the current level + /// (present, or just arrived: a tree can land before its children, so + /// present trees are read too), collects the absent blobs and subtrees + /// under the view, and fetches them together in id batches. A duplicate + /// subtree is read once per path. Progress is by presence, so an + /// interrupted fill resumes where it stopped. + fn fill_view(&mut self, root: ObjectId, sparse: &Sparse) -> Result<()> { + self.ensure_many(&BTreeSet::from([root]), "")?; + let mut frontier: Vec<(ObjectId, String)> = vec![(root, String::new())]; + let mut seen: HashSet<(ObjectId, String)> = HashSet::new(); + while !frontier.is_empty() { + let mut want = BTreeSet::new(); + let mut next = Vec::new(); + for (tree_id, path) in &frontier { + let tree = { + let store_arc = self.repo.vfs.store(); + let mut store = store_arc.lock().unwrap(); + store.get_tree(tree_id)? + }; + for e in tree.entries { + let child = if path.is_empty() { + e.name.clone() + } else { + format!("{path}/{}", e.name) + }; + match e.kind { + EntryKind::Blob => { + if sparse.matches(&child) { + want.insert(e.id); + } + } + EntryKind::Tree => { + if sparse.should_descend(&child) { + want.insert(e.id); + if seen.insert((e.id, child.clone())) { + next.push((e.id, child)); + } + } + } + } + } + } + let hint = frontier.first().map(|(_, p)| p.clone()).unwrap_or_default(); + self.ensure_many(&want, &hint)?; + frontier = next; + } + Ok(()) + } } #[cfg(test)] mod tests { use super::*; + use crate::sparse::Sparse; use std::io::{Read, Write}; /// Source repo with `src/a.txt`, `docs/readme.md` and `docs/api/guide.md`. @@ -616,4 +768,285 @@ mod tests { assert!(store_has(&want.id()), "verified objects are retained"); fx.cleanup(); } + + /// Serves from a real transport but refuses any by-id request over + /// `max_ids` as too large, like a server whose reply cap is lower than the + /// batch. + struct Capped { + inner: Box, + max_ids: usize, + } + + impl Transport for Capped { + fn list_refs(&self) -> Result> { + self.inner.list_refs() + } + fn head_branch(&self) -> Result { + self.inner.head_branch() + } + fn has_object(&self, id: &ObjectId) -> Result { + self.inner.has_object(id) + } + fn get_object(&self, id: &ObjectId) -> Result> { + self.inner.get_object(id) + } + fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { + self.inner.put_object(id, bytes) + } + fn update_ref(&self, b: &str, id: &ObjectId, old: Option<&ObjectId>) -> Result<()> { + self.inner.update_ref(b, id, old) + } + fn get_pack( + &self, + w: &[ObjectId], + h: &[ObjectId], + f: Option<&[String]>, + out: &mut dyn Write, + ) -> Result<()> { + self.inner.get_pack(w, h, f, out) + } + fn put_pack(&self, src: &mut dyn Read) -> Result> { + self.inner.put_pack(src) + } + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + if ids.len() > self.max_ids { + return Err(Error::PackTooLarge( + "by-id response exceeds the limit".into(), + )); + } + self.inner.get_objects(ids, out) + } + } + + #[test] + fn a_refused_batch_is_split_in_half_and_a_refused_single_is_final() { + let fx = Fixture::new("split"); + let dst = fx.clone(true); + let ids = [fx.docs_tree, fx.api_tree, fx.guide_blob, fx.readme_blob]; + + let capped = Capped { + inner: open_transport(fx.src_root.to_str().unwrap()).unwrap(), + max_ids: 2, + }; + let mut stats = FillStats::default(); + dst.fetch_ids_batched(&capped, &ids, &mut stats).unwrap(); + // 4 ids refused as one batch, then served as 2 + 2. + assert_eq!( + stats, + FillStats { + requests: 2, + objects: 4 + } + ); + assert!(ids.iter().all(|id| fx.has(&dst, id))); + fx.cleanup(); + + // A single object that is itself over the cap cannot be split further. + let fx = Fixture::new("split-final"); + let dst = fx.clone(true); + let none = Capped { + inner: open_transport(fx.src_root.to_str().unwrap()).unwrap(), + max_ids: 0, + }; + let err = dst + .fetch_ids_batched(&none, &[fx.docs_tree], &mut FillStats::default()) + .unwrap_err(); + assert!(matches!(err, Error::PackTooLarge(_)), "got {err:?}"); + assert!(!fx.has(&dst, &fx.docs_tree)); + fx.cleanup(); + } + + /// Serves synthetic blobs by id and records the size of every request. + struct Synth { + objs: std::collections::HashMap, + sizes: std::cell::RefCell>, + } + + impl Transport for Synth { + fn list_refs(&self) -> Result> { + unimplemented!() + } + fn head_branch(&self) -> Result { + unimplemented!() + } + fn has_object(&self, _: &ObjectId) -> Result { + unimplemented!() + } + fn get_object(&self, _: &ObjectId) -> Result> { + unimplemented!() + } + fn put_object(&self, _: &ObjectId, _: &[u8]) -> Result<()> { + unimplemented!() + } + fn update_ref(&self, _: &str, _: &ObjectId, _: Option<&ObjectId>) -> Result<()> { + unimplemented!() + } + fn get_pack( + &self, + _: &[ObjectId], + _: &[ObjectId], + _: Option<&[String]>, + _: &mut dyn Write, + ) -> Result<()> { + unimplemented!() + } + fn put_pack(&self, _: &mut dyn Read) -> Result> { + unimplemented!() + } + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + self.sizes.borrow_mut().push(ids.len()); + let mut w = scl_core::pack::PackWriter::new(&mut *out, ids.len() as u32)?; + for id in ids { + let o = &self.objs[id]; + w.write_object(id, &o.encode())?; + } + w.finish()?; + Ok(Vec::new()) + } + } + + #[test] + fn a_wide_level_is_chunked_by_count_before_anything_is_split_by_size() { + let fx = Fixture::new("chunks"); + let dst = fx.clone(true); + let objs: std::collections::HashMap = (0..2500u32) + .map(|i| { + let o = Object::blob(i.to_le_bytes().to_vec()); + (o.id(), o) + }) + .collect(); + let ids: Vec = objs.keys().copied().collect(); + let t = Synth { + objs, + sizes: Default::default(), + }; + let mut stats = FillStats::default(); + dst.fetch_ids_batched(&t, &ids, &mut stats).unwrap(); + // 2500 ids: two full 1024-id requests and one of 452, never split. + assert_eq!(*t.sizes.borrow(), vec![1024, 1024, 452]); + assert_eq!( + stats, + FillStats { + requests: 3, + objects: 2500 + } + ); + fx.cleanup(); + } + + #[test] + fn ensure_view_fills_the_closure_level_by_level() { + let fx = Fixture::new("view"); + let dst = fx.clone(true); + let root = dst.snapshot(&fx.tip).unwrap().root; + let view = Sparse::new(vec!["docs/".to_string()]); + + // Offline: nothing is fetched and the call is a no-op, so the + // preflight in `materialize` is what reports the gap. + let stats = dst.ensure_view_impl(root, &view, true).unwrap(); + assert_eq!(stats, FillStats::default()); + assert!(!fx.has(&dst, &fx.docs_tree)); + + let stats = dst.ensure_view_impl(root, &view, false).unwrap(); + // docs tree; then readme.md + docs/api; then guide.md. + assert_eq!( + stats, + FillStats { + requests: 3, + objects: 4 + } + ); + assert!(fx.has(&dst, &fx.docs_tree)); + assert!(fx.has(&dst, &fx.api_tree)); + assert!(fx.has(&dst, &fx.guide_blob)); + assert!(fx.has(&dst, &fx.readme_blob)); + + // Present already: a second call costs nothing. + let stats = dst.ensure_view_impl(root, &view, false).unwrap(); + assert_eq!(stats, FillStats::default()); + // An empty spec (a full checkout) is never filled on a clone's behalf. + let stats = dst + .ensure_view_impl(root, &Sparse::default(), false) + .unwrap(); + assert_eq!(stats, FillStats::default()); + fx.cleanup(); + } + + #[test] + fn sparse_set_outside_the_filter_fills_then_narrowing_back_removes_the_files() { + let fx = Fixture::new("widen"); + let dst = fx.clone(true); + assert!(fx.dst_root.join("src/a.txt").exists()); + + dst.set_sparse(&["docs/".to_string()], None).unwrap(); + assert_eq!( + std::fs::read(fx.dst_root.join("docs/api/guide.md")).unwrap(), + b"guide body" + ); + assert!(fx.dst_root.join("docs/readme.md").exists()); + assert!( + !fx.dst_root.join("src/a.txt").exists(), + "old view was removed" + ); + + // Back to src/: the docs files lie outside the promisor filter, so the + // old-view bound is what lets the removal walk find them. + dst.set_sparse(&["src/".to_string()], None).unwrap(); + assert!(fx.dst_root.join("src/a.txt").exists()); + assert!(!fx.dst_root.join("docs/readme.md").exists()); + assert!(!fx.dst_root.join("docs/api/guide.md").exists()); + fx.cleanup(); + } + + #[test] + fn sparse_set_outside_the_filter_is_refused_unless_the_clone_is_lazy() { + let fx = Fixture::new("widen-nonlazy"); + let dst = fx.clone(false); + let err = dst.set_sparse(&["docs/".to_string()], None).unwrap_err(); + assert!(matches!(err, Error::GapOutsideFilter(_)), "got {err:?}"); + assert!(!fx.has(&dst, &fx.docs_tree)); + fx.cleanup(); + } + + /// The demo flow: widen the view past the filter, edit and commit there, + /// push, and a second clone sees it; gc and verify still work, and an undo + /// of the commit needs no network once the closure is present. + #[test] + fn widen_commit_push_then_a_second_clone_sees_it() { + let fx = Fixture::new("widen-push"); + let dst = fx.clone(true); + dst.set_sparse(&["src/".to_string(), "docs/".to_string()], None) + .unwrap(); + std::fs::write(fx.dst_root.join("docs/api/guide.md"), b"guide, edited").unwrap(); + let new_tip = dst.commit("t", "edit docs from a widened view").unwrap(); + dst.push("origin").unwrap(); + + // A fresh full clone of the origin sees the edit. + let other_root = + std::env::temp_dir().join(format!("scl-fill-widen-push-other-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&other_root); + let other = Repo::clone_to(&fx.src_root, &other_root).unwrap(); + assert_eq!( + std::fs::read(other_root.join("docs/api/guide.md")).unwrap(), + b"guide, edited" + ); + assert_eq!(other.head_tip().unwrap(), Some(new_tip)); + drop(other); + std::fs::remove_dir_all(&other_root).unwrap(); + + // gc keeps the filled closure; verify counts no gap inside the view. + dst.gc(std::time::Duration::from_secs(0)).unwrap(); + assert_eq!( + std::fs::read(fx.dst_root.join("docs/readme.md")).unwrap(), + b"readme body" + ); + // Undo needs no network: the closure it restores is already present. + std::fs::remove_dir_all(&fx.src_root).unwrap(); + dst.undo().unwrap(); + assert_eq!( + std::fs::read(fx.dst_root.join("docs/api/guide.md")).unwrap(), + b"guide body" + ); + fx.cleanup(); + } } diff --git a/crates/repo/src/oplog.rs b/crates/repo/src/oplog.rs index 76b8c77a..79e2bfc7 100644 --- a/crates/repo/src/oplog.rs +++ b/crates/repo/src/oplog.rs @@ -418,6 +418,8 @@ impl Repo { let restore = if rematerialize { let target_tip = will_be_tip.expect("checked non-unborn above"); let snap = self.snapshot(&target_tip)?; + // A lazy, online partial clone fills the restored view first. + self.ensure_view(snap.root, &self.sparse_spec()?)?; { let store_arc = self.vfs().store(); let mut store = store_arc.lock().unwrap(); diff --git a/crates/repo/src/partial_walks.rs b/crates/repo/src/partial_walks.rs index f477c3c4..c22485bf 100644 --- a/crates/repo/src/partial_walks.rs +++ b/crates/repo/src/partial_walks.rs @@ -445,3 +445,69 @@ fn old_root_gap_outside_the_view_does_not_break_switch_or_undo() { std::fs::remove_dir_all(r).unwrap(); } } + +/// On a lazy clone the fast-forward that 37b-i refuses with `GapInView` now +/// heals itself: the missing blob is fetched by id before anything is written. +#[test] +fn lazy_clone_heals_the_fetch_gap_on_fast_forward() { + let (fx, clone) = Fx::new("lazy-ff"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + assert!(!Fx::has(&clone, &fx.shared_blob)); + + clone.merge("origin/main", "t").unwrap(); + assert!(Fx::has(&clone, &fx.shared_blob)); + assert_eq!( + std::fs::read(fx.clone_root.join("src/readme.txt")).unwrap(), + SHARED + ); + assert_eq!( + std::fs::read(fx.clone_root.join("src/aaa.txt")).unwrap(), + b"fetched normally" + ); + drop(clone); + fx.cleanup(); +} + +#[test] +fn lazy_switch_and_undo_fill_the_target_view_first() { + let (fx, clone) = Fx::new("lazy-switch"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + refs::write_branch_tip(clone.layout(), "feat", &fx.c2).unwrap(); + clone.switch("feat").unwrap(); + assert!(Fx::has(&clone, &fx.shared_blob)); + assert!(fx.clone_root.join("src/readme.txt").exists()); + + // Undo back to main (a switch) then forward again: both present already. + clone.undo().unwrap(); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "main"); + drop(clone); + fx.cleanup(); +} + +/// A fill that cannot reach the origin fails the operation before it changes +/// anything: no ref move, no spec change, no file written. +#[test] +fn a_failed_fill_leaves_refs_spec_and_files_untouched() { + let (fx, clone) = Fx::new("lazy-offline"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + refs::write_branch_tip(clone.layout(), "feat", &fx.c2).unwrap(); + // The origin goes away. + std::fs::remove_dir_all(&fx.origin_root).unwrap(); + + let spec_before = clone.sparse_spec().unwrap(); + assert!(clone.switch("feat").is_err()); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "main"); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + + assert!(clone.merge("origin/main", "t").is_err()); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + + assert!(clone.set_sparse(&["docs/".to_string()], None).is_err()); + assert_eq!(clone.sparse_spec().unwrap(), spec_before); + assert!(fx.clone_root.join("src/a.txt").exists()); + drop(clone); + fx.cleanup(); +} diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 2144b7e2..7b6183c2 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -1268,11 +1268,14 @@ impl Repo { Some(ours) => ours, None => { // Unborn local branch: adopt theirs wholesale (fast-forward from empty). - let store_arc = self.vfs.store(); - let mut store = store_arc.lock().unwrap(); - let theirs_snap = store.get_snapshot(&theirs)?; + let theirs_snap = self.snapshot(&theirs)?; let theirs_root = theirs_snap.root; let theirs_protection = theirs_snap.protection; + // Fill the view's closure first (lazy partial clones only); + // it takes the store lock, so it runs before ours. + self.ensure_view(theirs_root, &self.sparse_spec()?)?; + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; let skipped = worktree::materialize( &self.layout, @@ -1311,6 +1314,11 @@ impl Repo { let theirs_root = theirs_snap.root; let theirs_protection = theirs_snap.protection; let ours_root = store.get_snapshot(&ours)?.root; + // Fill the view's closure first (lazy partial clones only); + // the fill takes the store lock, so release ours around it. + drop(store); + self.ensure_view(theirs_root, &self.sparse_spec()?)?; + let mut store = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; let skipped = worktree::materialize( &self.layout, @@ -1822,6 +1830,9 @@ impl Repo { let snap = store_arc.lock().unwrap().get_snapshot(&target_tip)?; (snap.root, snap.protection) }; + // A lazy, online partial clone fills the target view's closure now, + // before the first stale file is removed or written (ADR-0048). + self.ensure_view(target_root, &self.sparse_spec()?)?; if head_priv.is_some() { // Remove private-tip paths the public target doesn't track, // BEFORE materialize lays the target down. diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index 66657a16..73bf9e4c 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -170,10 +170,15 @@ impl Repo { promisor.prefixes().join(", ") ))); } - for p in prefixes { - let bare = p.trim_end_matches('/'); - if !promisor.matches(bare) { - return Err(crate::promisor::partial_gap_hint(p)); + // A lazy clone may widen the view past its fetch filter: the + // closure is filled by id below. Any other partial clone never + // fetched outside the filter, so it refuses. + if !crate::promisor::is_lazy(&self.layout) { + for p in prefixes { + let bare = p.trim_end_matches('/'); + if !promisor.matches(bare) { + return Err(crate::promisor::partial_gap_hint(p)); + } } } } @@ -185,6 +190,9 @@ impl Repo { )); } let spec = Sparse::new(prefixes.to_vec()); + // The view in effect now, read before it is replaced: it bounds the + // removal walk in the re-lay below. + let old_spec = self.sparse_spec()?; let tip = self.head_tip()?; let snap = match tip { Some(tip) => Some(self.snapshot(&tip)?), @@ -195,6 +203,7 @@ impl Repo { // a failure below would leave the new spec over a tree that cannot be // laid out (ADR-0048). if let Some(snap) = &snap { + self.ensure_view(snap.root, &spec)?; let store_arc = self.vfs().store(); let mut s = store_arc.lock().unwrap(); crate::worktree::preflight_view(self.layout(), &mut s, snap.root, &spec)?; @@ -206,7 +215,7 @@ impl Repo { let store_arc = self.vfs().store(); let mut s = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; - let skipped = crate::worktree::materialize( + let skipped = crate::worktree::materialize_with_old_view( self.layout(), &mut s, snap.root, @@ -214,6 +223,7 @@ impl Repo { &snap.protection, identity, &spec, + Some(&old_spec), Some(&mut cache), )?; // The sparse spec (the operation's durable commit point) is already diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index ee3f4fa7..9d37b871 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -54,6 +54,45 @@ impl WireClient { let mut rw = self.rw.borrow_mut(); wire::write_frame(&mut rw.1, &Request::Bye.encode()) } + + /// [`WireClient::get_objects`], also reporting through `refused_in_sync` + /// whether the failure was the server refusing the request with a + /// `PackTooLarge` error frame. That reply is sent before any pack byte, so + /// the connection is still in sync and the caller may retry on it with a + /// smaller batch (ADR-0048). Every other failure leaves reply frames + /// unread or the stream in an unknown state. + pub(crate) fn get_objects_flagged( + &self, + ids: &[ObjectId], + out: &mut dyn Write, + refused_in_sync: &mut bool, + ) -> Result> { + if ids.len() > wire::MAX_GET_OBJECTS_IDS { + return Err(Error::InvalidArgument(format!( + "by-id fetch takes at most {} ids per call", + wire::MAX_GET_OBJECTS_IDS + ))); + } + let mut rw = self.rw.borrow_mut(); + wire::write_frame( + &mut rw.1, + &Request::GetObjects { ids: ids.to_vec() }.encode(), + )?; + let frame = wire::read_frame(&mut rw.0)?; + let body = wire::parse_response(frame); + if matches!(body, Err(Error::PackTooLarge(_))) { + *refused_in_sync = true; + } + let not_sent = wire::decode_ids_body_capped(&body?, ids.len())?; + let mut seen = std::collections::BTreeSet::new(); + if let Some(dup) = not_sent.iter().find(|id| !seen.insert(**id)) { + return Err(Error::Protocol(format!( + "reply lists object {dup} as not sent more than once" + ))); + } + wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP)?; + Ok(not_sent) + } } impl Transport for WireClient { @@ -138,27 +177,7 @@ impl Transport for WireClient { /// stream is capped (`GET_OBJECTS_RESPONSE_CAP`) so a hostile server /// cannot make a by-id fetch spool without bound. fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { - if ids.len() > wire::MAX_GET_OBJECTS_IDS { - return Err(Error::InvalidArgument(format!( - "by-id fetch takes at most {} ids per call", - wire::MAX_GET_OBJECTS_IDS - ))); - } - let mut rw = self.rw.borrow_mut(); - wire::write_frame( - &mut rw.1, - &Request::GetObjects { ids: ids.to_vec() }.encode(), - )?; - let frame = wire::read_frame(&mut rw.0)?; - let not_sent = wire::decode_ids_body_capped(&wire::parse_response(frame)?, ids.len())?; - let mut seen = std::collections::BTreeSet::new(); - if let Some(dup) = not_sent.iter().find(|id| !seen.insert(**id)) { - return Err(Error::Protocol(format!( - "reply lists object {dup} as not sent more than once" - ))); - } - wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP)?; - Ok(not_sent) + self.get_objects_flagged(ids, out, &mut false) } } @@ -274,9 +293,14 @@ impl Transport for StdioTransport { self.client.put_pack(src) } fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { - let r = self.client.get_objects(ids, out); - if r.is_err() { - // Any failure may have left unread reply frames on the pipe. + let mut refused_in_sync = false; + let r = self + .client + .get_objects_flagged(ids, out, &mut refused_in_sync); + if r.is_err() && !refused_in_sync { + // Any other failure may have left unread reply frames on the pipe. + // A server-side size refusal is sent before any pack byte, so the + // connection stays usable for a retry with a smaller batch. self.desynced.set(true); } r @@ -684,6 +708,66 @@ mod tests { ); } + /// A server-side size refusal arrives before any pack byte, so the client + /// reports it as in sync and the same connection serves the retry. + #[test] + fn a_pack_too_large_refusal_leaves_the_connection_usable_for_a_retry() { + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + // First request: refused as too large. + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_err( + &mut server_write, + wire::EC_TOO_LARGE, + "by-id response exceeds the limit", + ) + .unwrap(); + // Second request, on the same connection: served (nothing to send). + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::ids_body(&[])).unwrap(); + wire::write_pack_stream(&mut server_write, &mut std::io::empty(), 1024).unwrap(); + }); + let client = WireClient::handshake(client_read, client_write).unwrap(); + let id = Object::blob(b"wanted".to_vec()).id(); + let mut in_sync = false; + let err = client + .get_objects_flagged(&[id], &mut Vec::new(), &mut in_sync) + .unwrap_err(); + assert!(matches!(err, Error::PackTooLarge(_)), "got {err:?}"); + assert!(in_sync, "a pre-stream refusal keeps the stream in sync"); + let mut in_sync = false; + client + .get_objects_flagged(&[id], &mut Vec::new(), &mut in_sync) + .expect("the retry is served on the same connection"); + assert!(!in_sync); + server.join().unwrap(); + } + + /// Any other failure is not known to be in sync. + #[test] + fn a_malformed_reply_is_not_reported_as_in_sync() { + let id = Object::blob(b"wanted".to_vec()).id(); + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::ids_body(&[id, id])).unwrap(); + }); + let client = WireClient::handshake(client_read, client_write).unwrap(); + let mut in_sync = false; + let other = Object::blob(b"other".to_vec()).id(); + assert!(client + .get_objects_flagged(&[id, other], &mut Vec::new(), &mut in_sync) + .is_err()); + assert!(!in_sync, "only a size refusal is reported as in sync"); + server.join().unwrap(); + } + #[test] fn a_desynced_stdio_transport_kills_a_child_that_never_exits_on_drop() { // `sleep` ignores the wire protocol and never exits on its own, which diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 2aa13276..74dd1b79 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -82,6 +82,23 @@ pub trait Transport { } } +/// The most canonical object bytes one by-id reply carries (ADR-0048). It sits +/// well under the client's 1 GiB read cap so pack record headers and zstd +/// framing cannot push a legitimate reply past it. +const BY_ID_REPLY_CAP: u64 = 512 << 20; + +/// The by-id reply limit for a server whose `--max-pack-size` is +/// `max_pack_size` (0 means unlimited): the smaller of that and +/// [`BY_ID_REPLY_CAP`]. Shared by the wire server and [`LocalTransport`], so a +/// lazy clone of a local path cannot spool an unbounded reply either. +pub(crate) fn by_id_reply_cap(max_pack_size: u64) -> u64 { + if max_pack_size == 0 { + BY_ID_REPLY_CAP + } else { + max_pack_size.min(BY_ID_REPLY_CAP) + } +} + /// Canonical encoded size of `obj`, computed without copying a large payload: /// a blob or sealed object encodes as one tag byte plus its payload; the small /// structural kinds are encoded to measure them. @@ -520,7 +537,7 @@ impl Transport for LocalTransport { crate::wire::MAX_GET_OBJECTS_IDS ))); } - let (not_sent, guard) = self.build_objects_tempfile(ids, 0)?; + let (not_sent, guard) = self.build_objects_tempfile(ids, by_id_reply_cap(0))?; let mut f = std::fs::File::open(guard.path())?; std::io::copy(&mut f, out)?; Ok(not_sent) @@ -1069,4 +1086,14 @@ mod tests { std::fs::remove_dir_all(&layout.root).unwrap(); } + + #[test] + fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_512_mib() { + const MIB: u64 = 1 << 20; + // 0 means "unlimited" for --max-pack-size, but a by-id reply is still + // bounded so the client's read cap always covers it. + assert_eq!(by_id_reply_cap(0), 512 * MIB); + assert_eq!(by_id_reply_cap(64 * MIB), 64 * MIB); + assert_eq!(by_id_reply_cap(16 << 30), 512 * MIB); + } } diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index da5d8983..beb24571 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -1075,9 +1075,10 @@ fn serve_session( // remotes refuse: they refuse partial clone (ADR-0046), and // by-id reads exist only for partial clones. let result = match &transport { - ServeTransport::Local(t) => { - t.build_objects_tempfile(&ids, policy.max_pack_size) - } + ServeTransport::Local(t) => t.build_objects_tempfile( + &ids, + crate::transport::by_id_reply_cap(policy.max_pack_size), + ), ServeTransport::Bucket { .. } => Err(Error::InvalidArgument( "bucket remotes do not support by-id object fetch".into(), )), diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 0f192b04..5ef76eae 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -616,9 +616,13 @@ pub(crate) fn preflight_view( let bare = p.trim_end_matches('/'); path == bare || path.starts_with(&format!("{bare}/")) }) - .or_else(|| promisor.prefixes().first()) .cloned() - .unwrap_or_else(|| path.to_string()); + // Outside the filter (a view widened past it): name the path + // itself, which `sc backfill` accepts as a prefix. For the root + // tree there is no path, so fall back to the first filter prefix. + .or_else(|| (!path.is_empty()).then(|| path.to_string())) + .or_else(|| promisor.prefixes().first().cloned()) + .unwrap_or_default(); Error::GapInView { path: if path.is_empty() { "(root tree)".to_string() @@ -692,6 +696,38 @@ pub(crate) fn materialize( protection: &Protection, identity: Option<&scl_crypto::SecretKey>, sparse: &Sparse, + cache: Option<&mut ProtectedCache>, +) -> Result> { + materialize_with_old_view( + layout, + store, + target_root, + old_root, + protection, + identity, + sparse, + None, + cache, + ) +} + +/// [`materialize`] for a caller that is CHANGING the sparse spec (`sparse set`). +/// `old_view` is the spec that was in effect when `old_root` was laid out: the +/// removal walk is bounded by it, because everything on disk lies inside it +/// and nothing outside it was ever written. `None` means the spec is +/// unchanged, so `sparse` itself is the bound (ADR-0048; this replaces the +/// promisor-filter bound, which stops being a superset of the view once a lazy +/// clone can widen it past the filter). +#[allow(clippy::too_many_arguments)] +pub(crate) fn materialize_with_old_view( + layout: &Layout, + store: &mut Store, + target_root: ObjectId, + old_root: Option, + protection: &Protection, + identity: Option<&scl_crypto::SecretKey>, + sparse: &Sparse, + old_view: Option<&Sparse>, mut cache: Option<&mut ProtectedCache>, ) -> Result> { // Fail before the first write if the view's closure is not all present @@ -716,7 +752,7 @@ pub(crate) fn materialize( // old unfiltered walk exactly (`Sparse::default()` falls back to // `tree_file_ids` inside `tree_file_ids_sparse`). let (old_walk_bound, partial) = match crate::promisor::load(layout)? { - Some(p) => (Sparse::new(p.prefixes().to_vec()), true), + Some(_) => (old_view.unwrap_or(sparse).clone(), true), None => (Sparse::default(), false), }; let old_files = if partial { diff --git a/demo/run_lazy_fetch_demo.sh b/demo/run_lazy_fetch_demo.sh index accdec68..934caeba 100755 --- a/demo/run_lazy_fetch_demo.sh +++ b/demo/run_lazy_fetch_demo.sh @@ -123,4 +123,39 @@ cmp "$W/http.out" "$ORIGIN/docs/api/guide.bin" || fail "http cat must be byte-id echo "read-only sc+http server: lazy cat byte-identical, +3 objects, sibling not fetched ✔" echo -echo "RESULT: lazy by-id fetch verified (path-only fetch, offline/non-lazy gap errors, http v5)" +echo "=== 8: widen the view past the filter, commit, push, undo offline (ADR-0048) ===" +WIDE="$W/wide" +"$SC" clone --filter src/ --lazy "$ORIGIN" "$WIDE" >/dev/null +[ ! -e "$WIDE/docs" ] || fail "docs/ must not be on disk before the widen" +# Offline, the widen is refused before anything changes: the spec stays put. +SPEC_BEFORE=$(cat "$WIDE/.sc/sparse") +if (cd "$WIDE" && SC_OFFLINE=1 "$SC" sparse set src/ docs/) >/dev/null 2>"$W/wide-off.err"; then + fail "an offline widen of a lazy clone must fail" +fi +[ "$(cat "$WIDE/.sc/sparse")" = "$SPEC_BEFORE" ] || fail "a refused widen must leave .sc/sparse unchanged" +[ ! -e "$WIDE/docs" ] || fail "a refused widen must write nothing" +echo "offline widen: refused, .sc/sparse unchanged, nothing written ✔" +# Online, it fills the closure by id and lays the files out. +(cd "$WIDE" && "$SC" sparse set src/ docs/ >/dev/null) +cmp "$WIDE/docs/api/guide.bin" "$ORIGIN/docs/api/guide.bin" || fail "widened file must be byte-identical" +cmp "$WIDE/docs/api/other.bin" "$ORIGIN/docs/api/other.bin" || fail "the whole docs/ closure must be filled" +echo "sparse set src/ docs/: closure filled, files byte-identical ✔" +printf 'edited from a widened view\n' > "$WIDE/docs/api/guide.bin" +(cd "$WIDE" && "$SC" commit -m "edit docs" --author demo >/dev/null && "$SC" push origin >/dev/null) +SECOND="$W/second" +"$SC" clone "$ORIGIN" "$SECOND" >/dev/null +[ "$(cat "$SECOND/docs/api/guide.bin")" = "edited from a widened view" ] || fail "a second clone must see the edit" +echo "commit + push from the widened view: a second clone sees it ✔" +(cd "$WIDE" && "$SC" gc >/dev/null) || fail "gc must work after a widen" +(cd "$WIDE" && "$SC" verify >/dev/null) || fail "verify must work after a widen" +echo "gc and verify after the widen ✔" +# The closure is present, so undo needs no network even with the origin gone. +mv "$ORIGIN" "$ORIGIN.gone" +(cd "$WIDE" && SC_OFFLINE=1 "$SC" undo >/dev/null) || fail "undo of the commit must work offline" +# The origin working tree still holds the pre-edit file (a push moves refs only). +cmp "$WIDE/docs/api/guide.bin" "$ORIGIN.gone/docs/api/guide.bin" || fail "undo must restore the pre-edit file" +mv "$ORIGIN.gone" "$ORIGIN" +echo "undo after a widen, offline ✔" + +echo +echo "RESULT: lazy by-id fetch verified (path-only fetch, offline/non-lazy gap errors, http v5, widen/commit/push/undo)" diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 5c72ed69..7195b659 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -1,6 +1,6 @@ # ADR-0048: Presence-based walks for partial clones (P37b) -- **Status:** Accepted (37b-i built) +- **Status:** Accepted (37b-i, 37b-ii built) - **Date:** 2026-10-16 - **Phase:** 37b - **Builds on:** ADR-0047 (want-by-id fetch, 37a), ADR-0037 (partial clone), ADR-0034 (sparse), ADR-0035 (streaming transfer), ADR-0039/0041 (size limits) @@ -250,6 +250,17 @@ checkout (it equals disable), so it is refused on a partial clone too, in 37b-i; otherwise `ensure_view` would fill the whole tip and persist a full spec, bypassing the refusal. +*As built in 37b-ii:* `Repo::ensure_view` (`fill.rs`) runs before any ref, spec +or file mutation at switch, adopt and fast-forward merge, undo (against +`will_be_tip`) and `sparse set` (against the proposed spec), and not while the +store lock is held. It does nothing on a non-lazy, offline or full-spec call, +so the `materialize` preflight remains the single no-network backstop. +`materialize_with_old_view` takes the spec that was in effect, which bounds the +removal walk; every other caller passes the unchanged spec as the bound. A lazy +clone may `sparse set` outside the filter; any other partial clone still +refuses. The conflict-materialization and abort sites in the decision list stay +unreachable on a partial clone until merge and replay lift their refusals. + ### 5. Reply size limits 37a left a mismatch: the client reads at most 1 GiB per by-id reply while the @@ -321,7 +332,7 @@ review established, as constraints on that work: fetch-then-gc repro from the Context, the filtered-`get_pack`-still-cuts test, a lost-local-object push that must refuse, a push to a non-mirror remote. Existing partial-clone demos stay green unchanged. -- **37b-ii, widen.** `fill_closure`, `ensure_view` at every materializer call +- **37b-ii, widen (built).** `fill_closure`, `ensure_view` at every materializer call site with the ordering tests, reply-cap alignment with batch splitting, `sparse set` outside the filter. Demo: widen, commit, push, a second clone sees it; gc and verify after; undo after a widen while offline. @@ -368,6 +379,10 @@ review established, as constraints on that work: and `:186`) leaves in-view files reading as deleted at the next commit. Pre-existing; the old-spec parameter in decision 4 is a natural place to close it. +10. **Private branches on a partial clone** have no `ensure_view` or + preflight (`private.rs` materializes through its own path). Untested: + whether switching to one can hit a raw `NotFound` when public base + content is absent. ## Alternatives considered diff --git a/docs/adr/README.md b/docs/adr/README.md index 5381043e..532aa790 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -58,7 +58,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 | | [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (37a built) | 37 | -| [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (37b-i built) | 37b | +| [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (37b-i, 37b-ii built) | 37b | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, From a0f205858f841a937fee193b72b87d3ced68332c Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 10:56:38 +0300 Subject: [PATCH 2/4] fix(repo): address P37b-ii review (rebase site, crash marker, split rule) - fast-forward rebase now fills the view first, like merge; a test scans the sources so every materializer call must call ensure_view or carry an ensure-view-exempt note. - sparse set records the replaced spec in .sc/sparse-prev until the re-lay finishes; the removal walk is bounded by the union, so a crash strands no files (closes ADR-0048 open question 9 for narrowing). - the client splits a by-id batch only on a pre-stream server refusal; its own read cap tripping mid-stream is a protocol error. - fill_view reads a repeated subtree under a matched path once by id. - tests for lazy rebase/adopt/undo heal, non-lazy rebase refusal, the crash marker, and gc after a widen; stale doc comments corrected. Co-Authored-By: Claude Sonnet 5.5 --- crates/repo/src/fill.rs | 25 ++- crates/repo/src/layout.rs | 8 + crates/repo/src/partial_walks.rs | 160 ++++++++++++++++++ crates/repo/src/replay.rs | 9 + crates/repo/src/repo.rs | 3 + crates/repo/src/sparse.rs | 65 +++++++ crates/repo/src/stdio_transport.rs | 8 +- crates/repo/src/sync.rs | 1 + crates/repo/src/workspace.rs | 1 + crates/repo/src/worktree.rs | 57 ++++--- .../adr/0048-presence-based-partial-clones.md | 19 ++- 11 files changed, 316 insertions(+), 40 deletions(-) diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index d92ae362..a4dcf4a5 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -7,9 +7,10 @@ //! `Store::get` stays network-free, so gc, verify, materialize and the desktop //! read model can never block on the network. //! -//! Filling only ADDS objects the filter did not cover. The promisor filter is -//! unchanged, so push, gc and verify (all filter-scoped walks) behave exactly -//! as they did before the fill; they simply find a few extra present objects. +//! Filling only ADDS objects. The promisor filter is unchanged, and push, gc +//! and verify walk what the store holds (ADR-0048), so they simply find the +//! extra present objects. On a lazy clone `ensure_view` also fills a sparse +//! view's closure before an operation lays the tree out. use std::collections::{BTreeSet, HashSet}; use std::sync::Arc; @@ -339,7 +340,10 @@ impl<'a> Filler<'a> { fn fill_view(&mut self, root: ObjectId, sparse: &Sparse) -> Result<()> { self.ensure_many(&BTreeSet::from([root]), "")?; let mut frontier: Vec<(ObjectId, String)> = vec![(root, String::new())]; - let mut seen: HashSet<(ObjectId, String)> = HashSet::new(); + // Under a path the view matches, every descendant is in view whatever + // its path, so a repeated subtree is read once by id. Above that (the + // ancestors of a view prefix) the verdict depends on the path. + let mut seen: HashSet<(ObjectId, Option)> = HashSet::new(); while !frontier.is_empty() { let mut want = BTreeSet::new(); let mut next = Vec::new(); @@ -364,7 +368,8 @@ impl<'a> Filler<'a> { EntryKind::Tree => { if sparse.should_descend(&child) { want.insert(e.id); - if seen.insert((e.id, child.clone())) { + let key = (!sparse.matches(&child)).then(|| child.clone()); + if seen.insert((e.id, key)) { next.push((e.id, child)); } } @@ -1036,10 +1041,12 @@ mod tests { // gc keeps the filled closure; verify counts no gap inside the view. dst.gc(std::time::Duration::from_secs(0)).unwrap(); - assert_eq!( - std::fs::read(fx.dst_root.join("docs/readme.md")).unwrap(), - b"readme body" - ); + for id in [fx.docs_tree, fx.api_tree, fx.readme_blob] { + assert!(fx.has(&dst, &id), "gc dropped a filled object"); + } + // The gap frontier is unchanged by the widen: nothing in the view is + // missing, so only objects outside it could still be gaps. + assert_eq!(dst.partial_gap_count(&[new_tip]).unwrap(), Some(0)); // Undo needs no network: the closure it restores is already present. std::fs::remove_dir_all(&fx.src_root).unwrap(); dst.undo().unwrap(); diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index 2ba77532..87ab597f 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -110,6 +110,14 @@ impl Layout { pub fn promisor_lazy_path(&self) -> PathBuf { self.dot_sc.join("promisor-lazy") } + /// `.sc/sparse-prev` — the sparse spec in effect before an in-flight `sparse + /// set` (ADR-0048). Written before the new spec is stored and removed once + /// the tree is re-laid; while it exists, the removal walk is bounded by the + /// union of both specs, so a crash in between cannot strand files that the + /// new spec no longer names. + pub fn sparse_prev_path(&self) -> PathBuf { + self.dot_sc.join("sparse-prev") + } /// `.sc/serve-tokens.toml` — server access-control tokens (P29): each entry is /// `{label, hash = BLAKE3(raw token), scope}`. Presence of ≥1 entry turns on /// bearer auth for `sc serve --http`. Distinct from `recipients.toml` diff --git a/crates/repo/src/partial_walks.rs b/crates/repo/src/partial_walks.rs index c22485bf..425b78bd 100644 --- a/crates/repo/src/partial_walks.rs +++ b/crates/repo/src/partial_walks.rs @@ -511,3 +511,163 @@ fn a_failed_fill_leaves_refs_spec_and_files_untouched() { drop(clone); fx.cleanup(); } + +/// Fast-forward rebase is a materializer site too: a lazy clone heals the gap. +#[test] +fn lazy_clone_heals_the_fetch_gap_on_fast_forward_rebase() { + let (fx, clone) = Fx::new("lazy-rebase"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + assert!(!Fx::has(&clone, &fx.shared_blob)); + let r = clone.rebase("origin/main", "t", None).unwrap(); + assert!( + matches!(r, crate::replay::RebaseResult::FastForwarded(_)), + "got {r:?}" + ); + assert!(Fx::has(&clone, &fx.shared_blob)); + assert_eq!( + std::fs::read(fx.clone_root.join("src/readme.txt")).unwrap(), + SHARED + ); + drop(clone); + fx.cleanup(); +} + +/// A non-lazy clone still refuses the fast-forward rebase before writing. +#[test] +fn non_lazy_fast_forward_rebase_fails_before_writing() { + let (fx, clone) = Fx::new("rebase-gap"); + clone.fetch("origin").unwrap(); + let err = clone.rebase("origin/main", "t", None).unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + drop(clone); + fx.cleanup(); +} + +/// The unborn-branch adopt path fills the view first on a lazy clone. +#[test] +fn lazy_adopt_into_an_unborn_branch_fills_first() { + let (fx, clone) = Fx::new("lazy-adopt"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + refs::write_head(clone.layout(), "fresh").unwrap(); + assert!(clone.head_tip().unwrap().is_none()); + clone.merge("origin/main", "t").unwrap(); + assert!(Fx::has(&clone, &fx.shared_blob)); + assert_eq!( + std::fs::read(fx.clone_root.join("src/readme.txt")).unwrap(), + SHARED + ); + drop(clone); + fx.cleanup(); +} + +/// Undo on a lazy clone fills the snapshot it restores before moving any ref. +#[test] +fn lazy_undo_fills_the_restored_view_before_moving_refs() { + let (fx, clone) = Fx::new("lazy-undo"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.fetch("origin").unwrap(); + let head = refs::current_branch(clone.layout()).unwrap(); + let cur = refs::read_branch_tip(clone.layout(), "main").unwrap(); + crate::oplog::record( + clone.layout(), + "synthetic", + &head, + &head, + &[("main".to_string(), Some(fx.c2), cur)], + ) + .unwrap(); + clone.undo().unwrap(); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(fx.c2) + ); + assert!(Fx::has(&clone, &fx.shared_blob), "the gap was filled"); + assert_eq!( + std::fs::read(fx.clone_root.join("src/readme.txt")).unwrap(), + SHARED + ); + drop(clone); + fx.cleanup(); +} + +/// A crash between storing the new sparse spec and re-laying the tree must not +/// strand files the new spec no longer names: the pending marker keeps the old +/// view in the removal walk's bound until a later materialize finishes. +#[test] +fn an_interrupted_sparse_set_is_finished_by_the_next_materialize() { + let (fx, clone) = Fx::new_in_sync("sparse-crash"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone + .set_sparse(&["src/".to_string(), "docs/".to_string()], None) + .unwrap(); + assert!(fx.clone_root.join("docs/readme.txt").exists()); + // Simulate the crash window of `sparse set src/`: the previous spec is + // recorded and the new one stored, but the tree was never re-laid. + let prev = clone.sparse_spec().unwrap(); + crate::sparse::store_prev(clone.layout(), &prev).unwrap(); + crate::sparse::store(clone.layout(), &Sparse::new(vec!["src/".to_string()])).unwrap(); + assert!( + fx.clone_root.join("docs/readme.txt").exists(), + "still on disk" + ); + + // The next operation that lays the tree out removes the stranded files. + clone.set_sparse(&["src/".to_string()], None).unwrap(); + assert!(!fx.clone_root.join("docs/readme.txt").exists()); + assert!( + !fx.clone_root.join("docs").exists() || !fx.clone_root.join("docs/readme.txt").exists() + ); + assert!( + !clone.layout().sparse_prev_path().exists(), + "the marker is cleared once the re-lay completes" + ); + drop(clone); + fx.cleanup(); +} + +/// Every non-test call of the materializer either fills the view's closure +/// first (`ensure_view`) or carries an `ensure-view-exempt` note saying why it +/// cannot run on a partial clone. A new call site cannot be added silently. +#[test] +fn every_materializer_call_site_fills_or_is_exempt() { + let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src"); + let mut checked = 0; + for entry in std::fs::read_dir(&dir).unwrap() { + let path = entry.unwrap().path(); + if path.extension().and_then(|e| e.to_str()) != Some("rs") { + continue; + } + let name = path.file_name().unwrap().to_string_lossy().into_owned(); + if name == "worktree.rs" || name == "partial_walks.rs" { + continue; // the definition, and this test + } + let text = std::fs::read_to_string(&path).unwrap(); + let body = text + .split("#[cfg(test)]\nmod tests") + .next() + .unwrap_or(&text); + let lines: Vec<&str> = body.lines().collect(); + for (i, line) in lines.iter().enumerate() { + let t = line.trim(); + if t.starts_with("//") || t.starts_with("///") { + continue; + } + let call = t.contains("worktree::materialize(") + || t.contains("worktree::materialize_with_old_view("); + if !call { + continue; + } + checked += 1; + let window = lines[i.saturating_sub(60)..i].join("\n"); + assert!( + window.contains("ensure_view(") || window.contains("ensure-view-exempt"), + "{name}:{}: materializer call without ensure_view or an ensure-view-exempt note", + i + 1 + ); + } + } + assert!(checked >= 15, "the scan found only {checked} call sites"); +} diff --git a/crates/repo/src/replay.rs b/crates/repo/src/replay.rs index 56eb93eb..5ab34aae 100644 --- a/crates/repo/src/replay.rs +++ b/crates/repo/src/replay.rs @@ -480,6 +480,7 @@ impl Repo { // Protection-aware materialize: a PROTECTED entry decrypts // for `identity` when possible, else is skipped (never // writes ciphertext to disk) — same as merge's clean path. + // ensure-view-exempt: replay is refused on partial clones worktree::materialize( &self.layout, &mut store, @@ -604,6 +605,7 @@ impl Repo { // No identity at abort time: nothing is decrypted, so `materialize` // records nothing — pass `None`, no cache to open or save // (P33 Task 7 item 7). + // ensure-view-exempt: replay is refused on partial clones skipped = worktree::materialize( &self.layout, &mut store, @@ -718,6 +720,11 @@ impl Repo { let target_snap = store.get_snapshot(&target_tip)?; let target_root = target_snap.root; let target_protection = target_snap.protection; + // Fill the view's closure first (lazy partial clones only); + // the fill takes the store lock, so release ours around it. + drop(store); + self.ensure_view(target_root, &self.sparse_spec()?)?; + let mut store = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; worktree::materialize( &self.layout, @@ -1013,6 +1020,7 @@ impl Repo { // (each step pruned its wrap map to its own merged tree): a // PROTECTED entry decrypts for `identity` when possible, else is // skipped — never writes ciphertext to disk. + // ensure-view-exempt: replay is refused on partial clones worktree::materialize( &self.layout, &mut store, @@ -1230,6 +1238,7 @@ impl Repo { // No identity at abort time: nothing is decrypted, so `materialize` // records nothing — pass `None`, no cache to open or save // (P33 Task 7 item 7). + // ensure-view-exempt: replay is refused on partial clones worktree::materialize( &self.layout, &mut store, diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 7b6183c2..03c3c828 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -1514,6 +1514,7 @@ impl Repo { let skipped = { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); + // ensure-view-exempt: conflict materialization; merge is refused on partial clones worktree::materialize( &self.layout, &mut store, @@ -1617,6 +1618,7 @@ impl Repo { // skipped content, so nothing is lost by not surfacing the list // here (the `Err`/`Stopped` return can't carry it). let mut cache = self.open_protected_cache()?; + // ensure-view-exempt: conflict materialization; merge is refused on partial clones let _skipped = worktree::materialize( &self.layout, &mut store, @@ -1681,6 +1683,7 @@ impl Repo { // (not decrypted). With nothing decrypted, `materialize` records // nothing into a cache, so pass `None` — no cache to open or save // (P33 Task 7 item 7). + // ensure-view-exempt: merge abort; no merge can be in progress on a partial clone skipped = worktree::materialize( &self.layout, &mut store, diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index 73bf9e4c..d5013597 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -109,6 +109,50 @@ pub fn store(layout: &Layout, s: &Sparse) -> Result<()> { Ok(()) } +/// Record `s` as the spec in effect before an in-flight `sparse set`. +pub(crate) fn store_prev(layout: &Layout, s: &Sparse) -> Result<()> { + let text = format!("{}\n", s.prefixes.join("\n")); + scl_core::fsutil::atomic_write_durable(&layout.sparse_prev_path(), text.as_bytes())?; + Ok(()) +} + +/// The spec recorded by an interrupted `sparse set`, if one is pending. +pub(crate) fn load_prev(layout: &Layout) -> Result> { + match std::fs::read_to_string(layout.sparse_prev_path()) { + Ok(text) => Ok(Some(Sparse { + prefixes: text + .lines() + .filter(|l| !l.is_empty()) + .map(String::from) + .collect(), + })), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e.into()), + } +} + +/// Drop the pending-`sparse set` marker. +pub(crate) fn clear_prev(layout: &Layout) -> Result<()> { + remove_if_exists(&layout.sparse_prev_path()) +} + +impl Sparse { + /// A spec that matches whatever either input matches. A full (empty) spec + /// is full in the union. + pub(crate) fn union(&self, other: &Sparse) -> Sparse { + if self.is_full() || other.is_full() { + return Sparse::default(); + } + let mut prefixes = self.prefixes.clone(); + for p in &other.prefixes { + if !prefixes.contains(p) { + prefixes.push(p.clone()); + } + } + Sparse { prefixes } + } +} + /// Clear the sparse spec (equivalent to `store`-ing an empty one). pub fn clear(layout: &Layout) -> Result<()> { remove_if_exists(&layout.sparse_path()) @@ -208,8 +252,25 @@ impl Repo { let mut s = store_arc.lock().unwrap(); crate::worktree::preflight_view(self.layout(), &mut s, snap.root, &spec)?; } + // Crash safety: record the spec being replaced before the new one is + // stored. If the process dies before the re-lay finishes, the marker + // keeps the old view in the removal walk's bound until a later + // materialize completes the job (ADR-0048, open question 9). + let partial = self.promisor()?.is_some(); + if partial { + // An earlier interrupted `sparse set` may have left a marker: keep + // its view in the bound too. + let pending = match load_prev(self.layout())? { + Some(prev) => prev.union(&old_spec), + None => old_spec.clone(), + }; + store_prev(self.layout(), &pending)?; + } store(self.layout(), &spec)?; let Some(snap) = snap else { + if partial { + clear_prev(self.layout())?; + } return Ok(Vec::new()); }; let store_arc = self.vfs().store(); @@ -231,6 +292,9 @@ impl Repo { // is best-effort — never abort a sparse change on cache trouble // (P33 Task 7). cache.save_best_effort(); + if partial { + clear_prev(self.layout())?; + } Ok(skipped) } @@ -282,6 +346,7 @@ impl Repo { let store_arc = self.vfs().store(); let mut s = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; + // ensure-view-exempt: sparse disable is refused on partial clones let skipped = crate::worktree::materialize( self.layout(), &mut s, diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 9d37b871..872bd529 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -90,7 +90,13 @@ impl WireClient { "reply lists object {dup} as not sent more than once" ))); } - wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP)?; + // The client's own read cap tripping is NOT a server refusal: the reply + // is already part-way through the stream, so report it as a protocol + // violation that callers must not answer by splitting the batch. + wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP).map_err(|e| match e { + Error::PackTooLarge(m) => Error::Protocol(format!("by-id reply too large: {m}")), + other => other, + })?; Ok(not_sent) } } diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 89df424c..fc98eaa7 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -179,6 +179,7 @@ impl Repo { let head_snap = store.get_snapshot(&head_tip)?; let head_root = head_snap.root; let head_protection = head_snap.protection; + // ensure-view-exempt: clone: the lazy marker is set after the clone, and the materialize preflight covers it worktree::materialize( &dst_repo.layout, &mut store, diff --git a/crates/repo/src/workspace.rs b/crates/repo/src/workspace.rs index aab0a11e..e06f55ca 100644 --- a/crates/repo/src/workspace.rs +++ b/crates/repo/src/workspace.rs @@ -56,6 +56,7 @@ pub(crate) fn materialize_workspace( // passes an ephemeral one threaded fork->harvest. A `None` cache still // works — a workspace's randomized protected path just re-seals on the // next harvest instead of carrying (spurious-but-safe, per `cache.rs`). + // ensure-view-exempt: an ephemeral work session; `sc work` is refused on partial clones worktree::materialize( &ws, &mut store, diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 5ef76eae..0851f2f7 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -385,9 +385,9 @@ pub(crate) fn tree_file_ids_sparse( /// Like [`tree_file_ids_sparse`], but skips a subtree the store lacks instead /// of failing. For `materialize`'s old-root removal walk on a partial clone: /// the old root was laid out under a preflighted view, so an absent subtree -/// lay outside that view and has nothing on disk to remove. The promisor -/// filter that bounds the walk is only a superset of the view, and a fetch can -/// leave a tree inside the filter but outside the view absent (ADR-0048). +/// lay outside that view and has nothing on disk to remove. A fetch can leave +/// a tree absent that is inside the promisor filter but outside the view +/// (ADR-0048). fn tree_file_ids_sparse_present( store: &mut Store, root: ObjectId, @@ -497,11 +497,10 @@ pub struct Diff { /// `sparse` is full (the sparse-aware flattener falls back to the unfiltered /// one in that case) — the only behavior change is that an out-of-`sparse` /// path is now never fetched from the store at all, instead of being fetched -/// and then filtered out by the loop below. On a partial clone, `sparse` is -/// always a subset of the promisor filter (enforced by `set_sparse`/ -/// `disable_sparse`'s preflight), so this walk never touches a gapped -/// out-of-filter object — an out-of-filter path was never materialized to -/// disk either, so it's expected-absent, not a deletion. +/// and then filtered out by the loop below. On a partial clone this walk never +/// touches an object outside the view, so a gapped out-of-view path is +/// expected-absent, not a deletion. (A lazy clone's view may extend past its +/// promisor filter; the filled closure under the view is present.) pub(crate) fn diff_worktree( layout: &Layout, store: &mut Store, @@ -682,12 +681,11 @@ pub(crate) fn preflight_view( /// /// Gap-tolerant on a partial clone (P27 Task 5): both the target walk and /// the old-root removal walk are bounded so neither ever `store.get`s an -/// out-of-filter (never-fetched) object — the old-root walk is bounded by -/// the promisor filter specifically (not by the possibly-narrower `sparse`), -/// since `sparse`-narrowing is exactly the case that needs to enumerate -/// paths `sparse` no longer matches. This closes the crash a bare -/// `set_sparse`/`disable_sparse` narrowing (or a `switch`) used to hit on -/// any partial clone with a genuine out-of-filter gap elsewhere in the tree. +/// out-of-view (possibly never-fetched) object. The target walk is bounded by +/// `sparse`; the old-root walk by the spec the old root was laid out under +/// (see [`materialize_with_old_view`]), since narrowing is exactly the case +/// that needs to enumerate paths `sparse` no longer matches. Before the first +/// write, a partial clone's view closure is checked present (`preflight_view`). pub(crate) fn materialize( layout: &Layout, store: &mut Store, @@ -739,20 +737,27 @@ pub(crate) fn materialize_with_old_view( // walk here would `NotFound` on them. let target = tree_file_entries_with_perms_sparse(store, target_root, sparse)?; if let Some(old) = old_root { - // The old-root removal walk (P27 Task 5 fix): bounded by the - // promisor filter, not by `sparse` itself. `old` is enumerated to - // find paths that WERE materialized and might now need removing — - // `sparse` (the NEW, possibly narrower spec) can't be the descend - // bound here, since narrowing is exactly the case that needs to - // enumerate paths `sparse` no longer matches. On a partial clone, - // the promisor filter is the widest boundary that's guaranteed - // present in the CAS; anything outside it was never fetched and so - // was never materialized either — nothing to remove there, and - // walking it would `NotFound`. A full (non-partial) repo keeps the - // old unfiltered walk exactly (`Sparse::default()` falls back to + // The old-root removal walk: bounded by the spec the old root was + // laid out under (`old_view`, or `sparse` itself when the spec is + // unchanged), not by the promisor filter (ADR-0048). `old` is + // enumerated to find paths that WERE materialized and might now need + // removing, so the NEW, possibly narrower spec can't be the bound when + // it changed: narrowing is exactly the case that needs to enumerate + // paths it no longer matches. Everything on disk lies inside the old + // view, and the walk skips subtrees the store lacks (they were outside + // it, so nothing of them is on disk). A full (non-partial) repo keeps + // the old unfiltered walk exactly (`Sparse::default()` falls back to // `tree_file_ids` inside `tree_file_ids_sparse`). let (old_walk_bound, partial) = match crate::promisor::load(layout)? { - Some(_) => (old_view.unwrap_or(sparse).clone(), true), + Some(_) => { + let mut bound = old_view.unwrap_or(sparse).clone(); + // An interrupted `sparse set` left the replaced spec behind: + // files laid out under it are still on disk. + if let Some(prev) = crate::sparse::load_prev(layout)? { + bound = bound.union(&prev); + } + (bound, true) + } None => (Sparse::default(), false), }; let old_files = if partial { diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 7195b659..b6efda76 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -251,7 +251,7 @@ checkout (it equals disable), so it is refused on a partial clone too, in spec, bypassing the refusal. *As built in 37b-ii:* `Repo::ensure_view` (`fill.rs`) runs before any ref, spec -or file mutation at switch, adopt and fast-forward merge, undo (against +or file mutation at switch, adopt, fast-forward merge and fast-forward rebase, undo (against `will_be_tip`) and `sparse set` (against the proposed spec), and not while the store lock is held. It does nothing on a non-lazy, offline or full-spec call, so the `materialize` preflight remains the single no-network backstop. @@ -259,7 +259,17 @@ so the `materialize` preflight remains the single no-network backstop. removal walk; every other caller passes the unchanged spec as the bound. A lazy clone may `sparse set` outside the filter; any other partial clone still refuses. The conflict-materialization and abort sites in the decision list stay -unreachable on a partial clone until merge and replay lift their refusals. +unreachable on a partial clone until merge and replay lift their refusals; each +carries an `ensure-view-exempt` note, and a test scans the sources so that a new +materializer call site must either call `ensure_view` or say why it is exempt. +Duplicate subtrees under a matched path are read once by id. + +`sparse set` writes `.sc/sparse-prev` (the replaced spec) before storing the new +one and clears it after the re-lay. While it exists the removal walk is bounded +by the union of both specs, so a crash in between strands no files (this closes +open question 9). The client splits a batch only on a size refusal the server +sent before any pack byte; the client's own read cap tripping mid-stream is a +protocol error, because the stream is no longer in sync. ### 5. Reply size limits @@ -377,8 +387,9 @@ review established, as constraints on that work: differ by id from the same commit on a full clone. A spike should check. 9. **Crash between storing the sparse spec and materializing** (`sparse.rs:178` and `:186`) leaves in-view files reading as deleted at the next commit. - Pre-existing; the old-spec parameter in decision 4 is a natural place to - close it. + Closed in 37b-ii for the files a narrowing would strand, by the + `.sc/sparse-prev` marker (decision 4, as built). The in-view-reads-as-deleted + half after a widen is unchanged. 10. **Private branches on a partial clone** have no `ensure_view` or preflight (`private.rs` materializes through its own path). Untested: whether switching to one can hit a raw `NotFound` when public base From 91f2142bf960522f0d842c923fed09e14c79332b Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 10:58:56 +0300 Subject: [PATCH 3/4] fix(repo): count refused by-id batches as round trips; clarify lazy gap healing in CLAUDE.md Co-Authored-By: Claude Sonnet 5.5 --- CLAUDE.md | 7 ++++--- crates/repo/src/fill.rs | 8 +++++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ab12c46e..170c4701 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -174,9 +174,10 @@ transport-adjacent. The rest, imperatively: `sc fetch` from any native remote but the promisor origin.** `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what a clone *asked for*, not what is present: after a fetch an in-filter object - can be absent, and a switch, fast-forward, undo or `sparse set` that needs it - fails with `GapInView` (nothing changed) and names the `sc backfill ` - that repairs it. + can be absent. On a lazy, online clone a switch, fast-forward, rebase, undo + or `sparse set` fetches what its view needs by id first; on a non-lazy or + offline clone it fails with `GapInView` (nothing changed) and names the + `sc backfill ` that repairs it. - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index a4dcf4a5..278af353 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -120,9 +120,11 @@ impl Repo { ids: &[ObjectId], stats: &mut FillStats, ) -> Result<()> { + // Every attempt is a network round trip, including one the server + // refuses as over its reply cap; only ingested objects are counted. + stats.requests += 1; match self.fetch_by_id_from(transport, ids) { Ok(n) => { - stats.requests += 1; stats.objects += n; Ok(()) } @@ -835,11 +837,11 @@ mod tests { }; let mut stats = FillStats::default(); dst.fetch_ids_batched(&capped, &ids, &mut stats).unwrap(); - // 4 ids refused as one batch, then served as 2 + 2. + // 4 ids refused as one batch (still a round trip), then served as 2 + 2. assert_eq!( stats, FillStats { - requests: 2, + requests: 3, objects: 4 } ); From 35d60a18b4e6974dfcab1ea52f87249ce75e43ae Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:06:36 +0300 Subject: [PATCH 4/4] fix(repo): clear the sparse-prev marker once any materialize has used it Co-Authored-By: Claude Sonnet 5.5 --- crates/repo/src/partial_walks.rs | 17 +++++++++++++++++ crates/repo/src/worktree.rs | 11 +++++++++++ 2 files changed, 28 insertions(+) diff --git a/crates/repo/src/partial_walks.rs b/crates/repo/src/partial_walks.rs index 425b78bd..d5e97810 100644 --- a/crates/repo/src/partial_walks.rs +++ b/crates/repo/src/partial_walks.rs @@ -624,6 +624,23 @@ fn an_interrupted_sparse_set_is_finished_by_the_next_materialize() { !clone.layout().sparse_prev_path().exists(), "the marker is cleared once the re-lay completes" ); + + // A marker left by a crash is also cleared by any other materialize (here + // a switch), so it cannot widen later removal walks forever. + crate::sparse::store_prev( + clone.layout(), + &Sparse::new(vec!["src/".to_string(), "docs/".to_string()]), + ) + .unwrap(); + refs::write_branch_tip(clone.layout(), "other", &fx.c2).unwrap(); + clone.switch("other").unwrap(); + assert!(!clone.layout().sparse_prev_path().exists()); + // ...and a file the user creates outside the view afterwards survives the + // next switch (it is not in the old root, and the bound is back to normal). + std::fs::create_dir_all(fx.clone_root.join("docs")).unwrap(); + std::fs::write(fx.clone_root.join("docs/mine.txt"), b"mine").unwrap(); + clone.switch("main").unwrap(); + assert!(fx.clone_root.join("docs/mine.txt").exists()); drop(clone); fx.cleanup(); } diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 0851f2f7..fd9b2fcf 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -736,6 +736,9 @@ pub(crate) fn materialize_with_old_view( // clone never fetched out-of-filter subtrees at all, so an unfiltered // walk here would `NotFound` on them. let target = tree_file_entries_with_perms_sparse(store, target_root, sparse)?; + // Set when an interrupted `sparse set`'s marker widened the removal bound; + // cleared once this call has finished laying the tree out under `sparse`. + let mut prev_marker_used = false; if let Some(old) = old_root { // The old-root removal walk: bounded by the spec the old root was // laid out under (`old_view`, or `sparse` itself when the spec is @@ -755,6 +758,7 @@ pub(crate) fn materialize_with_old_view( // files laid out under it are still on disk. if let Some(prev) = crate::sparse::load_prev(layout)? { bound = bound.union(&prev); + prev_marker_used = true; } (bound, true) } @@ -838,6 +842,13 @@ pub(crate) fn materialize_with_old_view( std::fs::write(&full, &bytes[..])?; } } + // The removal walk above covered the replaced view, and the tree is now + // laid out under `sparse`: the marker has done its job. Leaving it would + // widen every later removal walk and delete files the user created + // outside the view. + if prev_marker_used { + crate::sparse::clear_prev(layout)?; + } Ok(skipped) }