diff --git a/CLAUDE.md b/CLAUDE.md index 2919495d..10af20d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,7 @@ the project tree if desired. ## Capability map (what's built, by phase) -All 36 phases and P37a are built and tested. One line of current fact per phase; the +All 36 phases, P37a and P37b-i are built and tested. One line of current fact per phase; the authoritative rationale and full semantics live in the linked ADR, the design in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not @@ -152,6 +152,7 @@ the code, those win. | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | +| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. Sparse widen (37b-ii) and merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) | ## Standing boundaries & gotchas @@ -168,8 +169,14 @@ transport-adjacent. The rest, imperatively: capability, and no DTO field capable of carrying protected ciphertext or identity material. - **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`, - `sc work`, `sc export`, and `sparse disable`.** `sc backfill --all` converts - to a genuine full clone and re-enables them. + `sc work`, `sc export`, `sparse disable`, an empty `sparse set`, `sc rewrap`, + `sc grant`/`revoke`, leaving a private branch, `sc push` to a git remote, and + `sc fetch` from any native remote but the promisor origin.** `sc backfill --all` + converts to a genuine full clone and re-enables them. The filter records what + a clone *asked for*, not what is present: after a fetch an in-filter object + can be absent, and a switch, fast-forward, undo or `sparse set` that needs it + fails with `GapInView` (nothing changed) and names the `sc backfill ` + that repairs it. - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. diff --git a/ROADMAP.md b/ROADMAP.md index 50e85bd0..300ae641 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -548,12 +548,12 @@ across every phase. ## Active -**P37b — presence-based walks and the lifted P27 refusals (proposed).** -P37a (built) added the batched `GetObjects` verb and `sc cat`; the remaining -work needs push, verify and gc to ask "is this object present?" instead of "is -this path in the filter?" (37b-i, which also fixes gc and fast-forward failing -after an ordinary fetch on a partial clone), then sparse widen over a filled -prefix (37b-ii). Merge, rebase and sessions on partial clones are directions +**P37b — presence-based walks and the lifted P27 refusals (37b-i built).** +P37a (built) added the batched `GetObjects` verb and `sc cat`. 37b-i (built) +made push, verify and gc ask "is this object present?" instead of "is this +path in the filter?", which also fixed gc and fast-forward failing after an +ordinary fetch on a partial clone; the remaining work is sparse widen over a +filled prefix (37b-ii). Merge, rebase and sessions on partial clones are directions with constraints, each needing its own design record; `export` and `sparse disable` stay refused. Design and open questions in [ADR-0048](docs/adr/0048-presence-based-partial-clones.md). diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index ffc7c21c..f579368d 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -3868,6 +3868,16 @@ fn run_push(remote: &str, include_encrypted: bool) -> Result<()> { `sc branch publish {branch}` first" ); } + // A partial clone cannot export: the git bridge walks the whole + // history and every blob, and the first absent object would fail + // it partway through. Refuse up front, like `sc export` does. + if repo.promisor()?.is_some() { + anyhow::bail!( + "refusing to push to a git remote: this is a partial clone \ + (.sc/promisor present) and the git bridge needs the whole history \ + and every blob; run `sc backfill --all` to convert to a full clone first" + ); + } run_push_git(&repo, remote, include_encrypted) } _ => { diff --git a/crates/cli/tests/partial_clone_safety.rs b/crates/cli/tests/partial_clone_safety.rs index c15cbe5a..0395292a 100644 --- a/crates/cli/tests/partial_clone_safety.rs +++ b/crates/cli/tests/partial_clone_safety.rs @@ -249,3 +249,121 @@ fn sparse_widen_beyond_partial_filter_is_refused() { std::fs::remove_dir_all(&src_root).unwrap(); std::fs::remove_dir_all(&dst_root).unwrap(); } + +/// ADR-0048: an ordinary `sc fetch` can leave an in-filter object absent +/// (the origin subtracts the closure of every have, and the new file equals +/// one that is only reachable at an out-of-filter path). `sc gc` must keep +/// working, `sc verify` must count the gap, and a fast-forward must fail +/// before it writes anything, with a repair hint that works. +#[test] +fn fetch_then_gc_and_fast_forward_on_a_partial_clone() { + let src_root = tmp("fetch-gc-src"); + let dst_root = tmp("fetch-gc-dst"); + std::fs::create_dir_all(src_root.join("src")).unwrap(); + std::fs::create_dir_all(src_root.join("docs")).unwrap(); + assert!(sc(&src_root, &["init"]).status.success()); + std::fs::write(src_root.join("src/a.txt"), "src-one\n").unwrap(); + std::fs::write( + src_root.join("docs/readme.txt"), + "same bytes at two paths\n", + ) + .unwrap(); + assert!(sc(&src_root, &["commit", "-m", "c1"]).status.success()); + let out = sc( + &src_root, + &[ + "clone", + src_root.to_str().unwrap(), + dst_root.to_str().unwrap(), + "--filter", + "src/", + ], + ); + assert!(out.status.success(), "clone: {}", stderr(&out)); + + // The origin adds an in-filter file whose bytes equal docs/readme.txt, and + // another in-filter file that sorts before it. + std::fs::write(src_root.join("src/aaa.txt"), "fetched normally\n").unwrap(); + std::fs::write(src_root.join("src/readme.txt"), "same bytes at two paths\n").unwrap(); + assert!(sc(&src_root, &["commit", "-m", "c2"]).status.success()); + + let fetch = sc(&dst_root, &["fetch", "origin"]); + assert!(fetch.status.success(), "fetch: {}", stderr(&fetch)); + + let gc = sc(&dst_root, &["gc"]); + assert!( + gc.status.success(), + "gc must work after an ordinary fetch: {}", + stderr(&gc) + ); + let verify = sc(&dst_root, &["verify"]); + assert!(verify.status.success(), "verify: {}", stderr(&verify)); + + let merge = sc(&dst_root, &["merge", "origin/main"]); + assert!(!merge.status.success(), "the fast-forward must be refused"); + let err = stderr(&merge); + assert!( + err.contains("sc backfill src/"), + "error names the repair: {err}" + ); + assert!( + !dst_root.join("src/aaa.txt").exists() && !dst_root.join("src/readme.txt").exists(), + "nothing was written before the refusal" + ); + + let backfill = sc(&dst_root, &["backfill", "src/"]); + assert!(backfill.status.success(), "backfill: {}", stderr(&backfill)); + let merge = sc(&dst_root, &["merge", "origin/main"]); + assert!(merge.status.success(), "merge: {}", stderr(&merge)); + assert_eq!( + std::fs::read_to_string(dst_root.join("src/readme.txt")).unwrap(), + "same bytes at two paths\n" + ); + + std::fs::remove_dir_all(&src_root).unwrap(); + std::fs::remove_dir_all(&dst_root).unwrap(); +} + +/// The git bridge walks the whole history and every blob, so a partial clone +/// refuses `sc push` to a git remote up front, like `sc export`. +#[test] +fn push_to_a_git_remote_is_refused_on_a_partial_clone() { + let (src_root, dst_root) = make_partial_clone("push-git"); + let bare = tmp("push-git-bare"); + std::fs::create_dir_all(&bare).unwrap(); + let init = Command::new("git") + .args(["init", "-q", "--bare", "-b", "main"]) + .arg(&bare) + .output() + .expect("git runs"); + assert!(init.status.success()); + assert!(sc( + &dst_root, + &["remote", "add", "hub", bare.to_str().unwrap(), "--git"] + ) + .status + .success()); + + let out = sc(&dst_root, &["push", "hub"]); + assert!(!out.status.success()); + let err = stderr(&out); + assert!(err.contains("partial clone"), "names the cause: {err}"); + assert!(err.contains("backfill --all"), "names the repair: {err}"); + + std::fs::remove_dir_all(&src_root).unwrap(); + std::fs::remove_dir_all(&dst_root).unwrap(); + std::fs::remove_dir_all(&bare).unwrap(); +} + +/// `sc sparse set` with no prefixes is a full checkout, which is what `sparse +/// disable` is, so it is refused too (otherwise it would bypass that refusal). +#[test] +fn sparse_set_with_no_prefixes_is_refused_on_a_partial_clone() { + let (src_root, dst_root) = make_partial_clone("sparse-empty"); + let out = sc(&dst_root, &["sparse", "set"]); + assert!(!out.status.success(), "an empty spec must be refused"); + assert!(stderr(&out).contains("backfill"), "{}", stderr(&out)); + assert!(!dst_root.join("docs/b.txt").exists()); + std::fs::remove_dir_all(&src_root).unwrap(); + std::fs::remove_dir_all(&dst_root).unwrap(); +} diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index d9dbe272..763f184b 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -46,6 +46,24 @@ pub enum Error { GappedPathContent(String), #[error("{0} lies outside this partial clone's fetch filter; run `sc backfill {0}` to fetch it first")] GapOutsideFilter(String), + /// An object the checked-out view needs is absent from a partial clone's + /// store even though the path is inside its fetch filter (ADR-0048): an + /// ordinary `fetch` subtracts the closure of every have, so a new object + /// equal to one reachable from a have at an out-of-filter path is left + /// out of the reply. Raised before anything is written, so the working + /// tree and refs are untouched. `prefix` is the filter prefix that + /// `sc backfill` can act on. + #[error("{path} is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill {prefix}` to repair it")] + GapInView { path: String, prefix: String }, + /// A push found an object the local store lacks (referenced by `referrer`) + /// that the destination does not hold either (ADR-0048). Pushing the new + /// tip would leave the destination with a tree that names an object it + /// cannot serve, so the push is refused before any byte is sent. + #[error("refusing to push: object {id} (referenced by {referrer}) is missing locally and the remote does not have it; if it came from the origin run `sc backfill --all`, otherwise restore or recreate the content and commit again, then retry")] + PushGap { + id: scl_core::ObjectId, + referrer: scl_core::ObjectId, + }, /// A dedicated, non-path-shaped refusal for an operation that isn't /// supported on a partial clone at all (merge, cherry-pick/rebase /// replay, `sc ws harvest`, `sc work`) — distinct from diff --git a/crates/repo/src/gc.rs b/crates/repo/src/gc.rs index acd60844..6f0bf71f 100644 --- a/crates/repo/src/gc.rs +++ b/crates/repo/src/gc.rs @@ -14,7 +14,7 @@ use std::collections::BTreeSet; use std::time::{Duration, SystemTime, UNIX_EPOCH}; -use scl_core::{Object, ObjectId, Store}; +use scl_core::{ObjectId, Store}; use crate::error::Result; use crate::layout::Layout; @@ -80,6 +80,23 @@ fn roots(layout: &Layout) -> Result> { Ok(set.into_iter().collect()) } +/// Protect an in-progress operation's decided carried tree. On a partial +/// clone the walk tolerates an absent child, like the main reachability walk +/// does, so a stopped operation cannot brick gc; a full clone stays strict. +fn walk_decided_root( + store: &mut Store, + tree: ObjectId, + partial: bool, + reachable: &mut BTreeSet, +) -> Result<()> { + if partial { + let mut gaps = std::collections::BTreeMap::new(); + reachable::walk_tree_partial(store, tree, tree, reachable, &mut gaps) + } else { + reachable::walk_tree(store, tree, reachable) + } +} + /// Run a gc pass. Caller must already hold the repo lock. pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result { let mut stats = GcStats::default(); @@ -94,60 +111,22 @@ pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result = match &filter { - Some(p) => { - let r = reachable::reachable_objects_filtered(store, &roots, Some(p))?; - let mut included = r.included; - // P27 Task 5 review Critical fix (gc defense-in-depth): a gap id - // means "referenced by an in-filter parent tree but never - // fetched" — the filtered walk deliberately never `get()`s it, - // so it's normally absent locally. But gc must not treat "gap" - // as license to prune an object that IS present locally for any - // reason (belt-and-suspenders alongside the Task 5 commit-side - // refusal above, which closes the one known way such an object - // could be created; this is the backstop for any path that - // doesn't yet route through that guard). Walk every PRESENT gap - // id in, so it — and anything it in turn references — becomes - // part of `reachable` and survives the sweep below: gc never - // prunes reachable content it holds. This walk-in is itself - // ABSENCE-TOLERANT (P27 final review I1, `walk_tree_present_only`), - // not the strict `walk_tree`: `ingest_pack_file`'s write pass is - // not all-or-nothing, so a crash-interrupted `sc backfill` can - // leave a present gap-frontier tree with an absent child — the - // strict walk would `get()` that child and hard-error, which - // `Store::write_pack` below would then repeat on the id anyway - // if it were smuggled into `reachable`. Present content BELOW an - // absent child is not reached and is pruned by the ordinary - // loose-object sweep if it's otherwise disconnected — gc never - // prunes anything connected to the local graph, but it is not - // structurally incapable of pruning locally-disconnected residue. - for gap in &r.gaps { - if included.contains(gap) || !store.contains(gap) { - continue; - } - match store.get(gap)? { - Object::Tree(_) => { - reachable::walk_tree_present_only(store, *gap, &mut included)?; - } - _ => { - included.insert(*gap); - } - } - } - included - } - None => reachable::reachable_objects(store, &roots)?, + // Presence-based reachability on a partial clone (ADR-0048): when + // `.sc/promisor` is present, walk what the store actually holds. An absent + // child of a present tree is a gap, recorded and never read, so a + // genuinely missing object (the premise of a partial clone) cannot + // surface as a `gc` error, and a gap is not garbage either: it is not in + // `reachable`, so the loose-object sweep below never considers it. The + // promisor filter is deliberately not consulted. It records what a clone + // asked for, not what is present, and an ordinary `fetch` can leave an + // in-filter object absent. Every present tree is descended, so content a + // lazy fill or a backfill brought in survives. A full clone + // (`promisor::load` returns `None`) takes the strict walk, unchanged. + let partial = promisor::load(layout)?.is_some(); + let mut reachable: BTreeSet = if partial { + reachable::reachable_objects_partial(store, &roots)?.present + } else { + reachable::reachable_objects(store, &roots)? }; // An in-progress merge's or cherry-pick's decided carried tree // (MERGE_DECIDED_ROOT / PICK_DECIDED_ROOT) is a TREE root, not a @@ -161,16 +140,16 @@ pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result (Fx, Repo) { + Fx::build(tag, true) + } + + /// Like [`Fx::new`], but the origin stays at its first commit, so the + /// clone can push without a fast-forward conflict. + fn new_in_sync(tag: &str) -> (Fx, Repo) { + Fx::build(tag, false) + } + + fn build(tag: &str, advance: bool) -> (Fx, Repo) { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p37b-{tag}-origin-{pid}")); + let clone_root = base.join(format!("scl-p37b-{tag}-clone-{pid}")); + let other_root = base.join(format!("scl-p37b-{tag}-other-{pid}")); + for r in [&origin_root, &clone_root, &other_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(origin_root.join("src")).unwrap(); + std::fs::create_dir_all(origin_root.join("docs")).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("src/a.txt"), b"src-one").unwrap(); + std::fs::write(origin_root.join("docs/readme.txt"), SHARED).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + let docs_tree = { + let arc = origin.vfs().store(); + let mut s = arc.lock().unwrap(); + let root = s.get_snapshot(&c1).unwrap().root; + s.get_tree(&root).unwrap().get("docs").unwrap().id + }; + let shared_blob = Object::blob(SHARED.to_vec()).id(); + + let clone = Repo::clone_url_filtered( + origin_root.to_str().unwrap(), + &clone_root, + Some(&["src/".to_string()]), + ) + .unwrap(); + + let c2 = if advance { + std::fs::write(origin_root.join("src/aaa.txt"), b"fetched normally").unwrap(); + std::fs::write(origin_root.join("src/readme.txt"), SHARED).unwrap(); + origin.commit("t", "c2").unwrap() + } else { + c1 + }; + drop(origin); + + std::fs::create_dir_all(&other_root).unwrap(); + let other = Repo::init(&other_root).unwrap(); + drop(other); + + ( + Fx { + origin_root, + clone_root, + other_root, + c2, + shared_blob, + docs_tree, + }, + clone, + ) + } + + fn has(repo: &Repo, id: &ObjectId) -> bool { + repo.vfs().store().lock().unwrap().contains(id) + } + + fn cleanup(self) { + for r in [&self.origin_root, &self.clone_root, &self.other_root] { + if r.exists() { + std::fs::remove_dir_all(r).unwrap(); + } + assert!(!r.exists(), "fixture {} must be removed", r.display()); + } + } +} + +/// The ADR-0048 repro. After an ordinary fetch the new in-filter blob is +/// absent; `sc gc` must still work, the gap must be counted, and a +/// fast-forward must fail cleanly (before writing anything) instead of +/// failing in `materialize` after it had rewritten part of the tree. +#[test] +fn fetch_leaves_in_filter_gap_gc_works_and_ff_fails_before_writing() { + let (fx, clone) = Fx::new("repro"); + clone.fetch("origin").unwrap(); + assert!( + !Fx::has(&clone, &fx.shared_blob), + "premise: the fetch omitted the blob (it is reachable from a have at docs/)" + ); + + // gc used to fail with `object ... not found` here. + clone.gc(Duration::from_secs(0)).unwrap(); + + // The gap is counted: the never-fetched docs tree, plus the new blob. + assert_eq!(clone.partial_gap_count(&[fx.c2]).unwrap(), Some(2)); + + let head_before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + let err = clone.merge("origin/main", "t").unwrap_err(); + match &err { + Error::GapInView { path, prefix } => { + assert_eq!(path, "src/readme.txt"); + assert_eq!(prefix, "src/"); + } + other => panic!("expected GapInView, got {other:?}"), + } + assert!( + err.to_string().contains("sc backfill src/"), + "the error names the repair: {err}" + ); + // Nothing moved: the ref is unchanged and no file was written, not even + // `src/aaa.txt`, which sorts before the missing `src/readme.txt`. + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + head_before + ); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + assert!(!fx.clone_root.join("src/readme.txt").exists()); + + // `sc backfill src/` repairs it, and the fast-forward then works. + clone.backfill(&["src/".to_string()]).unwrap(); + assert!(Fx::has(&clone, &fx.shared_blob)); + clone.merge("origin/main", "t").unwrap(); + assert_eq!( + std::fs::read(fx.clone_root.join("src/readme.txt")).unwrap(), + SHARED + ); + assert_eq!( + std::fs::read(fx.clone_root.join("src/aaa.txt")).unwrap(), + b"fetched normally" + ); + drop(clone); + fx.cleanup(); +} + +/// `switch` preflights the target before it removes or writes anything. +#[test] +fn switch_to_a_gapped_branch_fails_before_writing() { + let (fx, clone) = Fx::new("switch"); + clone.fetch("origin").unwrap(); + refs::write_branch_tip(clone.layout(), "feat", &fx.c2).unwrap(); + + let err = clone.switch("feat").unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "main"); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + assert!(fx.clone_root.join("src/a.txt").exists()); + drop(clone); + fx.cleanup(); +} + +/// `sc undo` preflights the snapshot it will restore, before any ref moves. +#[test] +fn undo_to_a_gapped_snapshot_fails_before_moving_refs() { + let (fx, clone) = Fx::new("undo"); + clone.fetch("origin").unwrap(); + // Record an operation whose undo would restore `main` to the gapped c2. + let head = refs::current_branch(clone.layout()).unwrap(); + let cur = refs::read_branch_tip(clone.layout(), "main").unwrap(); + crate::oplog::record( + clone.layout(), + "synthetic", + &head, + &head, + &[("main".to_string(), Some(fx.c2), cur)], + ) + .unwrap(); + + let err = clone.undo().unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!(refs::read_branch_tip(clone.layout(), "main").unwrap(), cur); + assert!(!fx.clone_root.join("src/aaa.txt").exists()); + drop(clone); + fx.cleanup(); +} + +/// `sparse set` checks the proposed view before it stores the new spec. +#[test] +fn sparse_set_with_a_gapped_head_keeps_the_old_spec() { + let (fx, clone) = Fx::new_in_sync("sparse-gap"); + // Lose an in-view blob from the store (the working tree still has the + // file, so the tree is clean and `sparse set` reaches the preflight). + let lost = Object::blob(b"src-one".to_vec()).id(); + clone.vfs().store().lock().unwrap().delete(&lost).unwrap(); + let before = clone.sparse_spec().unwrap(); + + let err = clone.set_sparse(&["src/".to_string()], None).unwrap_err(); + assert!( + matches!(err, Error::GapInView { ref path, .. } if path == "src/a.txt"), + "got {err:?}" + ); + assert_eq!(clone.sparse_spec().unwrap(), before, "spec untouched"); + drop(clone); + fx.cleanup(); +} + +/// An empty spec is a full checkout, which a partial clone cannot have. +#[test] +fn sparse_set_with_no_prefixes_is_refused_on_a_partial_clone() { + let (fx, clone) = Fx::new("sparse-empty"); + let before = clone.sparse_spec().unwrap(); + let err = clone.set_sparse(&[], None).unwrap_err(); + assert!( + matches!(err, Error::InvalidArgument(ref m) if m.contains("empty sparse spec")), + "got {err:?}" + ); + assert_eq!(clone.sparse_spec().unwrap(), before); + drop(clone); + fx.cleanup(); +} + +/// A fill brings in objects outside the filter; gc keeps them because they +/// are reachable, not because the filter says so. +#[test] +fn gc_keeps_filled_objects_outside_the_filter() { + let (fx, clone) = Fx::new("gc-fill"); + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + let bytes = clone.cat_at("main", "docs/readme.txt").unwrap(); + assert_eq!(&bytes[..], SHARED); + assert!(Fx::has(&clone, &fx.docs_tree)); + assert!(Fx::has(&clone, &fx.shared_blob)); + + clone.gc(Duration::from_secs(0)).unwrap(); + assert!(Fx::has(&clone, &fx.docs_tree), "filled tree survives gc"); + assert!(Fx::has(&clone, &fx.shared_blob), "filled blob survives gc"); + // And a fresh `cat` still works without refetching anything it holds. + assert_eq!( + &clone.cat_at("main", "docs/readme.txt").unwrap()[..], + SHARED + ); + drop(clone); + fx.cleanup(); +} + +/// A locally created object that is later lost must stop the push, not ship +/// a tree that names it. +#[test] +fn push_refuses_when_a_local_object_is_lost() { + let (fx, clone) = Fx::new_in_sync("push-lost"); + std::fs::write(fx.clone_root.join("src/a.txt"), b"edited locally").unwrap(); + let tip = clone.commit("t", "local edit").unwrap(); + let lost = Object::blob(b"edited locally".to_vec()).id(); + clone.vfs().store().lock().unwrap().delete(&lost).unwrap(); + assert!(!Fx::has(&clone, &lost)); + + let err = clone.push("origin").unwrap_err(); + match err { + Error::PushGap { id, .. } => assert_eq!(id, lost), + other => panic!("expected PushGap, got {other:?}"), + } + // The origin was not touched. + drop(clone); + let origin = Repo::open(&fx.origin_root).unwrap(); + assert!(!Fx::has(&origin, &tip)); + assert_eq!( + origin.head_tip().unwrap(), + Some(fx.c2), + "origin tip unchanged" + ); + drop(origin); + fx.cleanup(); +} + +/// A destination that is not a mirror of the origin lacks the gap objects, so +/// the push is refused instead of writing a snapshot with dangling children. +#[test] +fn push_to_a_non_mirror_remote_is_refused() { + let (fx, clone) = Fx::new_in_sync("push-other"); + clone + .remote_add("other", fx.other_root.to_str().unwrap()) + .unwrap(); + std::fs::write(fx.clone_root.join("src/a.txt"), b"edited locally").unwrap(); + let tip = clone.commit("t", "local edit").unwrap(); + + let err = clone.push("other").unwrap_err(); + match err { + Error::PushGap { id, .. } => assert_eq!(id, fx.docs_tree), + other => panic!("expected PushGap, got {other:?}"), + } + drop(clone); + let other = Repo::open(&fx.other_root).unwrap(); + assert!(!Fx::has(&other, &tip), "nothing was sent"); + drop(other); + fx.cleanup(); +} + +/// A partial clone fetches from its promisor origin only. +#[test] +fn fetch_from_a_non_origin_remote_is_refused() { + let (fx, clone) = Fx::new("fetch-other"); + clone + .remote_add("other", fx.other_root.to_str().unwrap()) + .unwrap(); + let err = clone.fetch("other").unwrap_err(); + assert!( + matches!(err, Error::InvalidArgument(ref m) if m.contains("promisor origin")), + "got {err:?}" + ); + // The origin itself is fine. + clone.fetch("origin").unwrap(); + drop(clone); + fx.cleanup(); +} + +/// Commands that flatten whole trees refuse explicitly on a partial clone. +#[test] +fn whole_tree_commands_refuse_on_a_partial_clone() { + let (fx, clone) = Fx::new("refuse"); + let (sk, pk) = scl_crypto::generate_keypair(); + let rid = pk.recipient_id(); + assert!(matches!( + clone.rewrap(&sk, &[], &[], false), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.grant("src/", &sk, &pk), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.revoke("src/", &rid), + Err(Error::PartialCloneUnsupported(_)) + )); + drop(clone); + fx.cleanup(); +} + +/// The commit graft refuses a brand-new top-level directory only when it +/// lies outside BOTH the promisor filter and the sparse view. A view that +/// was widened past the filter (37b-ii) therefore stops being refused. +#[test] +fn graft_new_name_check_uses_filter_union_sparse() { + let (fx, clone) = Fx::new("graft"); + std::fs::create_dir_all(fx.clone_root.join("tools")).unwrap(); + std::fs::write(fx.clone_root.join("tools/x.txt"), b"tool").unwrap(); + let err = clone.commit("t", "tools").unwrap_err(); + assert!( + matches!(err, Error::GappedPathContent(ref p) if p.contains("tools")), + "outside both: refused, got {err:?}" + ); + + // Widen the view to cover `tools/` (the filter still says `src/`). + crate::sparse::store( + clone.layout(), + &Sparse::new(vec!["src/".to_string(), "tools/".to_string()]), + ) + .unwrap(); + clone.commit("t", "tools").unwrap(); + drop(clone); + fx.cleanup(); +} + +/// A fetch can leave a tree inside the filter but outside the (narrower) +/// sparse view absent. A later switch or undo then lays out from a root that +/// has that gap as its OLD root; the removal walk must skip it (nothing of it +/// is on disk) instead of failing with a raw `NotFound` after refs moved. +#[test] +fn old_root_gap_outside_the_view_does_not_break_switch_or_undo() { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p37b-oldroot-origin-{pid}")); + let clone_root = base.join(format!("scl-p37b-oldroot-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(origin_root.join("src/a")).unwrap(); + std::fs::create_dir_all(origin_root.join("docs")).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("src/a/x.txt"), b"x").unwrap(); + std::fs::write(origin_root.join("docs/d.txt"), SHARED).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + let clone = Repo::clone_url_filtered( + origin_root.to_str().unwrap(), + &clone_root, + Some(&["src/".to_string()]), + ) + .unwrap(); + clone.set_sparse(&["src/a/".to_string()], None).unwrap(); + // `src/b` is byte-identical to `docs`, so its tree id is reachable from a + // have and the fetch leaves it out: inside the filter, outside the view. + std::fs::create_dir_all(origin_root.join("src/b")).unwrap(); + std::fs::write(origin_root.join("src/b/d.txt"), SHARED).unwrap(); + origin.commit("t", "c2").unwrap(); + drop(origin); + + refs::write_branch_tip(clone.layout(), "old", &c1).unwrap(); + clone.fetch("origin").unwrap(); + clone.merge("origin/main", "t").unwrap(); + + clone.switch("old").unwrap(); + let tip_old = refs::read_branch_tip(clone.layout(), "main").unwrap(); + clone.switch("main").unwrap(); + // Undo the second switch, then the first: each lays out across the gap. + clone.undo().unwrap(); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "old"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + tip_old + ); + drop(clone); + for r in [&origin_root, &clone_root] { + std::fs::remove_dir_all(r).unwrap(); + } +} diff --git a/crates/repo/src/promisor.rs b/crates/repo/src/promisor.rs index 0b60836e..754d67cf 100644 --- a/crates/repo/src/promisor.rs +++ b/crates/repo/src/promisor.rs @@ -189,27 +189,37 @@ pub fn offline() -> bool { } impl Repo { + /// Refuse `op` on a partial clone with the explicit + /// [`Error::PartialCloneUnsupported`] message. For commands that flatten + /// whole trees and would otherwise fail with a raw `NotFound` on the + /// first absent object (ADR-0048). + pub(crate) fn refuse_on_partial(&self, op: &str) -> Result<()> { + if self.promisor()?.is_some() { + return Err(partial_clone_unsupported(op)); + } + Ok(()) + } + /// The repo's current promisor marker, if any (thin wrapper over /// `load`). `None` means this is a full clone. pub fn promisor(&self) -> Result> { load(self.layout()) } - /// The out-of-filter gap count for `sc verify`'s partial-clone report - /// (P27 Task 5): walks the filtered reachability from `tips` and returns - /// the number of gaps recorded (ids referenced by an in-filter parent - /// tree but never fetched) — `None` if this is a full clone (no - /// `.sc/promisor`), so a partial clone's expected gaps are never mistaken - /// for missing/corrupt objects. An in-filter object that's genuinely - /// absent still surfaces as an `Err` from the underlying walk (real - /// corruption), not folded into this count. + /// The gap count for `sc verify`'s partial-clone report (ADR-0048): the + /// number of objects a present tree references that the store lacks, + /// from the presence-based walk over `tips`. `None` if this is a full + /// clone (no `.sc/promisor`). A gap is expected on a partial clone, never + /// treated as corruption, and the count is the size of the gap frontier: + /// a fill can replace one missing tree with several missing children, so + /// it is not a measure of progress. pub fn partial_gap_count(&self, tips: &[scl_core::ObjectId]) -> Result> { - let Some(p) = self.promisor()? else { + if self.promisor()?.is_none() { return Ok(None); - }; + } let store_arc = self.vfs().store(); let mut store = store_arc.lock().unwrap(); - let r = crate::reachable::reachable_objects_filtered(&mut *store, tips, Some(&p))?; + let r = crate::reachable::reachable_objects_partial(&mut store, tips)?; Ok(Some(r.gaps.len())) } } diff --git a/crates/repo/src/protect_ops.rs b/crates/repo/src/protect_ops.rs index 34f158c2..35918cc7 100644 --- a/crates/repo/src/protect_ops.rs +++ b/crates/repo/src/protect_ops.rs @@ -142,6 +142,7 @@ impl Repo { new: &scl_crypto::PublicKey, ) -> Result { self.refuse_on_private("sc grant")?; + self.refuse_on_partial("sc grant")?; // P21: same in-progress guard as `protect` — see its comment. if crate::merge_state::in_progress(self.layout()) { return Err(Error::MergeInProgress); @@ -222,6 +223,7 @@ impl Repo { /// already checked out — see the secrets-revoke rationale in ADR-0008). pub fn revoke(&self, prefix: &str, recipient_id: &scl_crypto::RecipientId) -> Result { self.refuse_on_private("sc revoke")?; + self.refuse_on_partial("sc revoke")?; // P21: same in-progress guard as `protect` — see its comment. if crate::merge_state::in_progress(self.layout()) { return Err(Error::MergeInProgress); diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index c0f300e0..aac8aeec 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -2,7 +2,7 @@ //! snapshot tips. Parameterized by an `ObjectSource` so it walks either the //! local `Store` (push) or a remote `Transport` (clone/fetch). Reused by P8 gc. -use std::collections::{BTreeSet, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashSet, VecDeque}; use scl_core::{EntryKind, Object, ObjectId, Store}; @@ -64,57 +64,6 @@ pub(crate) fn walk_tree( walk_tree_filtered(src, root, seen, &mut gaps, None, &mut expanded) } -/// Like [`walk_tree`], but tolerant of an absent child: a missing tree or -/// blob id is simply not inserted into `seen` and not descended into, -/// instead of erroring (P27 final review I1). Used only by gc's -/// defense-in-depth walk-in of a PRESENT gap-frontier tree -/// (`gc.rs::run`): `ingest_pack_file`'s write pass is not all-or-nothing, -/// so a crash-interrupted `sc backfill` (Ctrl-C, power loss) can leave a -/// gap-frontier tree present locally while one of its children never -/// landed. The strict [`walk_tree`] would `get()` that missing child and -/// hard-error, bricking `sc gc` on a partial clone — which this walk-in -/// exists specifically to prevent. Presence is also checked BEFORE -/// inserting an id into `seen` (not just before recursing into it): -/// `gc::run`'s next step, `Store::write_pack`, `get()`s every id in the -/// reachable set it's handed, so an absent id smuggled into `seen` here -/// would just move the same hard error one step later. A present child -/// below an absent one is intentionally NOT reached (matches the -/// documented nuance that gc only protects content connected to the local -/// graph — see ADR-0037's gc section). Takes `&mut Store` directly (not -/// the generic `ObjectSource`) since gc always walks the local store. -pub(crate) fn walk_tree_present_only( - store: &mut Store, - root: ObjectId, - seen: &mut BTreeSet, -) -> Result<()> { - if !store.contains(&root) || !seen.insert(root) { - return Ok(()); - } - let mut stack = vec![root]; - while let Some(tree_id) = stack.pop() { - let tree = match Store::get(store, &tree_id)? { - Object::Tree(t) => t, - _ => return Err(Error::BadRef(format!("expected tree {tree_id}"))), - }; - for e in tree.entries { - if !store.contains(&e.id) { - continue; - } - match e.kind { - EntryKind::Blob => { - seen.insert(e.id); - } - EntryKind::Tree => { - if seen.insert(e.id) { - stack.push(e.id); - } - } - } - } - } - Ok(()) -} - /// The prefix predicate a filtered reachability/tree walk needs. Implemented /// by [`crate::promisor::Promisor`] (P27 Task 1); `crate::sparse::Sparse` /// could implement it too, but that wiring is not required by this task. A @@ -169,14 +118,7 @@ pub fn reachable_objects_filtered( tips: &[ObjectId], filter: Option<&dyn PrefixFilter>, ) -> Result { - let mut included = BTreeSet::new(); let mut gaps = BTreeSet::new(); - let mut snapshots: VecDeque = VecDeque::new(); - for t in tips { - if included.insert(*t) { - snapshots.push_back(*t); - } - } // (id, path) pairs already pushed for expansion, under a filter only. // Hoisted to span EVERY snapshot's walk in this call (not just one), so // a later snapshot whose ROOT tree dedups to an already-expanded @@ -185,6 +127,37 @@ pub fn reachable_objects_filtered( // comment for why a bare-id gate on the root is wrong under a filter // (C1, P27 final review). let mut expanded: HashSet<(ObjectId, String)> = HashSet::new(); + let mut included = walk_snapshots(src, tips, |src, _snapshot, root, included| { + walk_tree_filtered(src, root, included, &mut gaps, filter, &mut expanded) + })?; + // A byte-identical subtree can be gapped at one path and included at + // another (same id, different filter verdict per path) — included + // always wins. See the disjointness note on `Reachable`. + gaps.retain(|id| !included.contains(id)); + Ok(Reachable { + included: std::mem::take(&mut included), + gaps, + }) +} + +/// The snapshot-level walk shared by the filtered and the presence-based +/// reachability: every tip, its parents, its `secrets` registry objects, and +/// a private branch's manifest closure. `walk_root` is called once per +/// snapshot with that snapshot's id and root tree id, and owns the tree +/// descent (and so the gap policy). Snapshots, parents and manifests are +/// always read strictly — they are never gaps. +fn walk_snapshots( + src: &mut S, + tips: &[ObjectId], + mut walk_root: impl FnMut(&mut S, ObjectId, ObjectId, &mut BTreeSet) -> Result<()>, +) -> Result> { + let mut included = BTreeSet::new(); + let mut snapshots: VecDeque = VecDeque::new(); + for t in tips { + if included.insert(*t) { + snapshots.push_back(*t); + } + } while let Some(sid) = snapshots.pop_front() { let snap = match src.get(&sid)? { Object::Snapshot(s) => s, @@ -229,20 +202,89 @@ pub fn reachable_objects_filtered( for id in snap.secrets.values() { included.insert(*id); } - walk_tree_filtered( - src, - snap.root, - &mut included, - &mut gaps, - filter, - &mut expanded, - )?; + walk_root(src, sid, snap.root, &mut included)?; } - // A byte-identical subtree can be gapped at one path and included at - // another (same id, different filter verdict per path) — included - // always wins. See the disjointness note on `Reachable`. - gaps.retain(|id| !included.contains(id)); - Ok(Reachable { included, gaps }) + Ok(included) +} + +/// The result of a presence-based walk (ADR-0048): `present` are the tree and +/// blob objects the walk reached and the store holds, plus the snapshot-level +/// ids it takes without a presence check, as the strict walk does (manifest +/// closure ids and secrets); `gaps` maps every absent object that +/// a present tree (or snapshot, for a root tree) references to its referrer. +/// The two are disjoint by construction, since an id is either in the store +/// or not. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct PartialReach { + pub present: BTreeSet, + pub gaps: BTreeMap, +} + +/// Presence-based reachability for a partial clone's own store (ADR-0048). +/// Unlike [`reachable_objects_filtered`], this takes no filter: it descends +/// every tree the store holds and records an absent child as a gap together +/// with the tree that references it, never reading it. The promisor filter +/// says what a clone *asked for*, not what is present (an ordinary `fetch` +/// can leave an in-filter object absent), so every client-side decision — +/// what gc keeps, what push sends, what verify reports — asks the store. +/// +/// A snapshot's root tree can be absent too (a commit that moves an old +/// out-of-filter subtree to the root is dropped by the fetch's `haves` +/// subtraction), so an absent root is a gap, not an error. Snapshots, +/// parents, secrets and manifests stay strict. Verdicts depend only on +/// presence, never on path, so the ordinary bare-id expansion dedup applies. +pub fn reachable_objects_partial(store: &mut Store, tips: &[ObjectId]) -> Result { + let mut gaps = BTreeMap::new(); + let present = walk_snapshots(store, tips, |store, snapshot, root, present| { + walk_tree_partial(store, snapshot, root, present, &mut gaps) + })?; + Ok(PartialReach { present, gaps }) +} + +/// Presence-based descent from one tree `root`, recording what it reaches in +/// `present` and what it finds absent in `gaps` (id to the referrer, the +/// first one seen). `referrer` is whatever names `root`: the snapshot for a +/// snapshot root, or `root` itself for a decided-root tree. `pub(crate)` so +/// gc can protect an in-progress operation's decided tree with the same +/// gap policy. +pub(crate) fn walk_tree_partial( + store: &mut Store, + referrer: ObjectId, + root: ObjectId, + present: &mut BTreeSet, + gaps: &mut BTreeMap, +) -> Result<()> { + if !store.contains(&root) { + gaps.entry(root).or_insert(referrer); + return Ok(()); + } + if !present.insert(root) { + return Ok(()); + } + let mut stack = vec![root]; + while let Some(tree_id) = stack.pop() { + let tree = match Store::get(store, &tree_id)? { + Object::Tree(t) => t, + _ => return Err(Error::BadRef(format!("expected tree {tree_id}"))), + }; + for e in tree.entries { + if !store.contains(&e.id) { + gaps.entry(e.id).or_insert(tree_id); + continue; + } + match e.kind { + EntryKind::Blob => { + present.insert(e.id); + } + EntryKind::Tree => { + if present.insert(e.id) { + stack.push(e.id); + } + } + } + } + } + Ok(()) } /// Filtered clones exclude private branches entirely (ADR-0044): sealed @@ -887,4 +929,61 @@ mod tests { other => panic!("expected NotFound for missing in-filter object, got {other:?}"), } } + + #[test] + fn partial_walk_on_a_full_store_equals_the_strict_walk_with_no_gaps() { + let (arc, snap_id, ..) = two_subtree_repo(); + let mut s = arc.lock().unwrap(); + let strict = reachable_objects(&mut *s, &[snap_id]).unwrap(); + let r = reachable_objects_partial(&mut s, &[snap_id]).unwrap(); + assert_eq!(r.present, strict); + assert!(r.gaps.is_empty()); + // The filtered walk keeps its meaning: against the same full store it + // still cuts out-of-filter objects (the origin relies on that to build + // a filtered transfer), which the presence walk does not. + let filter = crate::promisor::Promisor::new("origin", vec!["src/".into()]); + let f = reachable_objects_filtered(&mut *s, &[snap_id], Some(&filter)).unwrap(); + assert!(!f.gaps.is_empty()); + assert!(f.included.len() < strict.len()); + } + + #[test] + fn partial_walk_records_absent_children_with_their_referrer() { + let (arc, snap_id, root_id, src_tree_id, src_blob_id, docs_tree_id, docs_blob_id) = + two_subtree_repo(); + let mut s = arc.lock().unwrap(); + // The out-of-filter `docs` subtree is absent (a partial clone), and so + // is the in-filter blob under `src` (a fetch that omitted it). + s.delete(&docs_tree_id).unwrap(); + s.delete(&docs_blob_id).unwrap(); + s.delete(&src_blob_id).unwrap(); + let r = reachable_objects_partial(&mut s, &[snap_id]).unwrap(); + assert_eq!(r.gaps.get(&docs_tree_id), Some(&root_id)); + assert_eq!(r.gaps.get(&src_blob_id), Some(&src_tree_id)); + // Nothing below an absent tree is reachable, so it is not listed. + assert!(!r.gaps.contains_key(&docs_blob_id)); + assert_eq!(r.gaps.len(), 2); + assert!(r.present.contains(&snap_id)); + assert!(r.present.contains(&root_id)); + assert!(r.present.contains(&src_tree_id)); + assert!(r.present.iter().all(|id| s.contains(id))); + } + + #[test] + fn partial_walk_treats_an_absent_root_tree_as_a_gap() { + let (arc, snap_id, root_id, ..) = two_subtree_repo(); + let mut s = arc.lock().unwrap(); + s.delete(&root_id).unwrap(); + let r = reachable_objects_partial(&mut s, &[snap_id]).unwrap(); + assert_eq!(r.gaps.get(&root_id), Some(&snap_id)); + assert_eq!(r.present, BTreeSet::from([snap_id])); + } + + #[test] + fn partial_walk_still_requires_the_snapshot_itself() { + let (arc, snap_id, ..) = two_subtree_repo(); + let mut s = arc.lock().unwrap(); + s.delete(&snap_id).unwrap(); + assert!(reachable_objects_partial(&mut s, &[snap_id]).is_err()); + } } diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 764bcf1b..2144b7e2 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -1805,6 +1805,11 @@ impl Repo { if let Some(m) = self.manifest_at(&target_tip)? { return self.switch_to_private(name, target_tip, m, identity, &old_paths); } + // Leaving a private branch flattens the public target's whole tree + // below, which a partial clone cannot do. + if head_priv.is_some() { + self.refuse_on_partial("switching away from a private branch")?; + } // Public target. When leaving a private branch there is no public // old root to hand `materialize` — the stale-path removal below uses // `old_paths` directly instead. diff --git a/crates/repo/src/rewrap.rs b/crates/repo/src/rewrap.rs index d635d061..22575f3a 100644 --- a/crates/repo/src/rewrap.rs +++ b/crates/repo/src/rewrap.rs @@ -45,6 +45,7 @@ impl Repo { dry_run: bool, ) -> Result { self.refuse_on_private("sc rewrap")?; + self.refuse_on_partial("sc rewrap")?; // Refuse mid-merge/mid-pick: completion unions the OLD wraps back in // over rewrap's cutover, silently resurrecting a stripped recipient // at the tip (final-review I1). Checked even on --dry-run — a diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index 4543a9c6..66657a16 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -160,6 +160,16 @@ impl Repo { // Preflight — before any disk write (the spec file or the working // tree) — so a refused widen leaves nothing partially written. if let Some(promisor) = self.promisor()? { + // No prefixes means a full checkout, which is what `sparse + // disable` is, and that is refused on a partial clone. + if prefixes.is_empty() { + return Err(Error::InvalidArgument(format!( + "cannot set an empty sparse spec (a full checkout) on a partial clone: \ + only [{}] was fetched; run `sc backfill --all` to convert to a full \ + clone first, or name a prefix within it", + promisor.prefixes().join(", ") + ))); + } for p in prefixes { let bare = p.trim_end_matches('/'); if !promisor.matches(bare) { @@ -175,11 +185,24 @@ impl Repo { )); } let spec = Sparse::new(prefixes.to_vec()); + let tip = self.head_tip()?; + let snap = match tip { + Some(tip) => Some(self.snapshot(&tip)?), + None => None, + }; + // Check the PROPOSED view's closure against HEAD before the spec is + // stored: once it is stored the previous view cannot be recovered and + // a failure below would leave the new spec over a tree that cannot be + // laid out (ADR-0048). + if let Some(snap) = &snap { + let store_arc = self.vfs().store(); + let mut s = store_arc.lock().unwrap(); + crate::worktree::preflight_view(self.layout(), &mut s, snap.root, &spec)?; + } store(self.layout(), &spec)?; - let Some(tip) = self.head_tip()? else { + let Some(snap) = snap else { return Ok(Vec::new()); }; - let snap = self.snapshot(&tip)?; let store_arc = self.vfs().store(); let mut s = store_arc.lock().unwrap(); let mut cache = self.open_protected_cache()?; diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index d6c0f66f..89df424c 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -349,6 +349,21 @@ impl Repo { let url = cfg .url(remote) .ok_or_else(|| Error::NoSuchRemote(remote.to_string()))?; + // A partial clone fetches from its promisor origin only (ADR-0048). + // Its promise is that every object a present tree names but the store + // lacks is held by the origin. Another remote subtracts the closure + // of a have it holds, and the origin may not hold that object, so a + // fetch from it could leave a gap nobody can fill. + if let Some(p) = self.promisor()? { + if !same_remote_url(url, &p.origin) { + return Err(Error::InvalidArgument(format!( + "refusing to fetch from {remote}: this is a partial clone and only its \ + promisor origin ({}) can fill the gaps a fetch leaves. Fetch from the \ + origin, or run `sc backfill --all` to convert to a full clone first", + p.origin + ))); + } + } let transport = open_transport(url)?; let remote_refs = transport.list_refs()?; @@ -424,26 +439,30 @@ impl Repo { { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); - // Partial-clone push (P27 Task 4): an unfiltered reachability - // walk would descend into any gapped (never-fetched) - // out-of-filter subtree and NotFound on the missing tree/blob - // object. When this repo is a partial clone, walk with its - // promisor filter instead — this sends exactly the (possibly - // newly-committed) in-filter objects; the origin already has - // every out-of-filter object untouched (a P24/P15 carry-by-id - // commit never wrote to them), so a full clone of the origin - // still sees the intact gapped subtree after the push lands. - let promisor = self.promisor()?; - let reachable = match &promisor { - Some(p) => { - reachable::reachable_objects_filtered( - &mut *store, - &[local_tip], - Some(p as &dyn reachable::PrefixFilter), - )? - .included + // Partial-clone push (ADR-0048): walk what the store holds. An + // unfiltered walk would descend into a never-fetched subtree and + // `NotFound`, and a filtered one would trust the filter to say + // what is present, which an ordinary fetch can falsify. Absent + // children of present trees come back as gaps instead. The + // destination already holds every untouched out-of-filter object + // (a carry-by-id commit never wrote to it), so gaps are not sent, + // but each must exist there: a gap that is a lost local object or + // that this destination never had would leave a pushed tree + // naming something it cannot serve. Check before any byte moves. + let partial = self.promisor()?.is_some(); + let reachable = if partial { + let r = reachable::reachable_objects_partial(&mut store, &[local_tip])?; + for (gap, referrer) in &r.gaps { + if !transport.has_object(gap)? { + return Err(Error::PushGap { + id: *gap, + referrer: *referrer, + }); + } } - None => reachable::reachable_objects(&mut *store, &[local_tip])?, + r.present + } else { + reachable::reachable_objects(&mut *store, &[local_tip])? }; // Sender seam (P22 Task 3): unlike `fetch`/`clone`, `push` builds // its outgoing set directly from a local reachability walk @@ -483,6 +502,13 @@ impl Repo { } } +/// Whether two configured remote URLs name the same place: equal after +/// trimming trailing slashes, which the stored promisor origin and a +/// `[remote]` entry can differ by. +fn same_remote_url(a: &str, b: &str) -> bool { + a.trim_end_matches('/') == b.trim_end_matches('/') +} + /// The tips we already hold locally: every local branch and remote-tracking ref /// target that is present in the store. Passed as `haves` so a fetch pulls only /// the delta the remote has beyond what we already have. Safe to advertise diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 07cabe6c..0f192b04 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -246,10 +246,11 @@ pub(crate) fn graft_out_of_sparse( // The I1 check inside the loop below only fires when the parent // already has a same-name entry there; this closes that gap by // scanning the built side for any survivor once up front, against the - // PROMISOR filter (not `sparse` — `sparse` can be narrower than the - // fetch filter, but never wider, since `set_sparse`/`disable_sparse` - // refuse widening past it; the filter is the actual "was this ever - // fetched?" authority). Refuse loudly rather than let it land in the + // UNION of the promisor filter and the sparse view (ADR-0048): the + // filter records what a clone asked for, and a widened view can reach + // paths the filter does not, so a path is refused only if it lies + // outside both. (Before widening existed `sparse` was never wider than + // the filter, so nothing is newly refused.) Refuse loudly rather than let it land in the // new root, where `sc gc` would classify it as an unreachable gap and // prune it — even though it's genuinely reachable and the origin never // had it (P27 Task 5 review Critical). @@ -262,7 +263,7 @@ pub(crate) fn graft_out_of_sparse( } else { format!("{prefix}/{}", e.name) }; - if !promisor.should_descend(&path) { + if !promisor.should_descend(&path) && !sparse.should_descend(&path) { return Err(Error::GappedPathContent(path)); } } @@ -377,7 +378,25 @@ pub(crate) fn tree_file_ids_sparse( return tree_file_ids(store, root); } let mut out = BTreeMap::new(); - walk_tree_sparse(store, root, String::new(), sparse, &mut out)?; + walk_tree_sparse(store, root, String::new(), sparse, &mut out, false)?; + Ok(out) +} + +/// Like [`tree_file_ids_sparse`], but skips a subtree the store lacks instead +/// of failing. For `materialize`'s old-root removal walk on a partial clone: +/// the old root was laid out under a preflighted view, so an absent subtree +/// lay outside that view and has nothing on disk to remove. The promisor +/// filter that bounds the walk is only a superset of the view, and a fetch can +/// leave a tree inside the filter but outside the view absent (ADR-0048). +fn tree_file_ids_sparse_present( + store: &mut Store, + root: ObjectId, + sparse: &Sparse, +) -> Result> { + let mut out = BTreeMap::new(); + if store.contains(&root) { + walk_tree_sparse(store, root, String::new(), sparse, &mut out, true)?; + } Ok(out) } @@ -387,6 +406,7 @@ fn walk_tree_sparse( prefix: String, sparse: &Sparse, out: &mut BTreeMap, + skip_absent: bool, ) -> Result<()> { let tree: Tree = store.get_tree(&tree_id)?; for e in tree.entries { @@ -402,8 +422,8 @@ fn walk_tree_sparse( } } EntryKind::Tree => { - if sparse.should_descend(&path) { - walk_tree_sparse(store, e.id, path, sparse, out)?; + if sparse.should_descend(&path) && (!skip_absent || store.contains(&e.id)) { + walk_tree_sparse(store, e.id, path, sparse, out, skip_absent)?; } } } @@ -571,6 +591,75 @@ pub(crate) fn safe_join(root: &Path, rel: &str) -> Result { Ok(root.join(rel)) } +/// Check, before anything is written, that every object the sparse view needs +/// under `root` is in the store (ADR-0048, the view closure obligation). Only a +/// partial clone can have such a gap, so a full clone returns immediately and +/// keeps its existing behavior. On a gap this fails with +/// [`Error::GapInView`], naming the path and the filter prefix `sc backfill` +/// can repair, instead of the raw `NotFound` the write loop would raise after +/// it had already rewritten part of the working tree. No network: filling the +/// gap on a lazy clone is a separate, later step. +pub(crate) fn preflight_view( + layout: &Layout, + store: &mut Store, + root: ObjectId, + sparse: &Sparse, +) -> Result<()> { + let Some(promisor) = crate::promisor::load(layout)? else { + return Ok(()); + }; + let gap = |path: &str| { + let prefix = promisor + .prefixes() + .iter() + .find(|p| { + let bare = p.trim_end_matches('/'); + path == bare || path.starts_with(&format!("{bare}/")) + }) + .or_else(|| promisor.prefixes().first()) + .cloned() + .unwrap_or_else(|| path.to_string()); + Error::GapInView { + path: if path.is_empty() { + "(root tree)".to_string() + } else { + path.to_string() + }, + prefix, + } + }; + if !store.contains(&root) { + return Err(gap("")); + } + let mut stack = vec![(root, String::new())]; + while let Some((tree_id, path)) = stack.pop() { + let tree: Tree = store.get_tree(&tree_id)?; + for e in tree.entries { + let child = if path.is_empty() { + e.name.clone() + } else { + format!("{path}/{}", e.name) + }; + match e.kind { + EntryKind::Blob => { + if sparse.matches(&child) && !store.contains(&e.id) { + return Err(gap(&child)); + } + } + EntryKind::Tree => { + if sparse.should_descend(&child) { + if !store.contains(&e.id) { + return Err(gap(&child)); + } + stack.push((e.id, child)); + } + } + } + } + } + Ok(()) +} + /// Materialize a snapshot's file tree into the working dir. /// /// `sparse` filters the disk view: a target entry outside `sparse` is never @@ -605,6 +694,10 @@ pub(crate) fn materialize( sparse: &Sparse, mut cache: Option<&mut ProtectedCache>, ) -> Result> { + // Fail before the first write if the view's closure is not all present + // (partial clones only; ADR-0048). The check lives here so every caller + // is covered and none can forget it. + preflight_view(layout, store, target_root, sparse)?; // Gap-tolerant when sparse is active (P27 Task 4): a promisor-filtered // clone never fetched out-of-filter subtrees at all, so an unfiltered // walk here would `NotFound` on them. @@ -622,11 +715,16 @@ pub(crate) fn materialize( // walking it would `NotFound`. A full (non-partial) repo keeps the // old unfiltered walk exactly (`Sparse::default()` falls back to // `tree_file_ids` inside `tree_file_ids_sparse`). - let old_walk_bound = match crate::promisor::load(layout)? { - Some(p) => Sparse::new(p.prefixes().to_vec()), - None => Sparse::default(), + let (old_walk_bound, partial) = match crate::promisor::load(layout)? { + Some(p) => (Sparse::new(p.prefixes().to_vec()), true), + None => (Sparse::default(), false), + }; + let old_files = if partial { + tree_file_ids_sparse_present(store, old, &old_walk_bound)? + } else { + tree_file_ids_sparse(store, old, &old_walk_bound)? }; - for p in tree_file_ids_sparse(store, old, &old_walk_bound)?.keys() { + for p in old_files.keys() { if !target.contains_key(p) || !sparse.matches(p) { let full = safe_join(&layout.root, p)?; let _ = std::fs::remove_file(full); diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 16e857e7..5c72ed69 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -1,6 +1,6 @@ # ADR-0048: Presence-based walks for partial clones (P37b) -- **Status:** Proposed +- **Status:** Accepted (37b-i built) - **Date:** 2026-10-16 - **Phase:** 37b - **Builds on:** ADR-0047 (want-by-id fetch, 37a), ADR-0037 (partial clone), ADR-0034 (sparse), ADR-0035 (streaming transfer), ADR-0039/0041 (size limits) @@ -148,7 +148,8 @@ is not all-or-nothing, so a stored tree does not prove a stored closure. ### 3. Verify and gc -- `sc verify` on a partial clone reports `partial: N gaps` from the presence +- `sc verify` on a partial clone keeps its `partial: N object(s) outside + filter [...]` line (the demos match on it) but counts N from the presence walk. N is the size of the gap frontier: it can rise when a fill replaces one missing tree with several missing children, and it is not a measure of progress. An optional origin check over the gap set is a follow-on. @@ -181,7 +182,17 @@ clone and fails with the existing gap hint on a non-lazy or offline one in-filter gap case above for lazy clones; non-lazy clones keep the explicit `sc backfill` answer. -`ensure_view` has two halves, and they ship separately. The **preflight** +`ensure_view` has two halves, and they ship separately. **As built in 37b-i, +the preflight lives at the top of `worktree::materialize`**, so it runs before +the first file write at every materializer call site (clone, switch, adopt, +fast-forward merge, conflict materialization, aborts, undo, sparse) and no +site can forget it. Sites that mutate a ref or the spec *before* they reach +`materialize` also call it explicitly, ahead of the mutation: `sc undo` +(against `will_be_tip`) and `sparse set` (against the proposed spec). The +check raises `Error::GapInView`, which names the missing path and the filter +prefix `sc backfill` repairs. The fill half is still 37b-ii. + +The original two halves: The **preflight** (check that the view's closure at the target root is present, otherwise fail with the gap hint, before any mutation) needs no network and lands in 37b-i, so a fast-forward or switch after an ordinary fetch fails cleanly with an @@ -226,7 +237,12 @@ truth: (`worktree.rs:288`), stays on the sparse view: relaxing it would silently discard an edit under a carried subtree. - **Materialize.** The old-view bound comes from the spec that was in effect, - passed in, not from the filter prefixes. + passed in, not from the filter prefixes. *Moved to 37b-ii:* until a widen + exists the sparse view is always inside the filter, so the filter-prefix + bound covers everything that was ever laid out. A superset of the view is + not necessarily present, though (a fetch can leave a tree inside the filter + but outside the view absent), so 37b-i makes that removal walk skip absent + subtrees; nothing of them is on disk. `sc sparse disable` stays refused on a partial clone; whole history and a full tree stay `sc backfill --all`. A `sc sparse set` with no prefixes means a full @@ -296,10 +312,10 @@ review established, as constraints on that work: ## Phasing -- **37b-i, engine.** `reachable_objects_partial`; gc, push and verify on it; +- **37b-i, engine (built).** `reachable_objects_partial`; gc, push and verify on it; the push gap-edge check; graft authority for new names as the union; - materialize takes the old spec; the `ensure_view` preflight at every - materializer site; refusals for the commands above, for `sc push` to a git + the `ensure_view` preflight at every materializer site (materialize + taking the old spec moved to 37b-ii, see decision 4); refusals for the commands above, for `sc push` to a git remote, for `sc fetch` from a non-origin remote, and for a zero-prefix `sc sparse set`. Regression tests: the fetch-then-gc repro from the Context, the filtered-`get_pack`-still-cuts diff --git a/docs/adr/README.md b/docs/adr/README.md index a92dab2c..5381043e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -58,7 +58,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 | | [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (37a built) | 37 | -| [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Proposed | 37b | +| [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (37b-i built) | 37b | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5,