diff --git a/CLAUDE.md b/CLAUDE.md index d55f347b..b79da4a6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,7 @@ the project tree if desired. ## Capability map (what's built, by phase) -All 36 phases, P37a, P37b-i and P37b-ii are built and tested. One line of current fact per phase; the +All 36 phases, P37a, P37b-i, P37b-ii and P37c-ii are built and tested. One line of current fact per phase; the authoritative rationale and full semantics live in the linked ADR, the design in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not @@ -152,7 +152,8 @@ the code, those win. | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | -| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. Merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) | +| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | +| P37c-ii | Clean merge on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. Cherry-pick/rebase and conflicts are next | [0049](docs/adr/0049-merge-on-partial-clones.md) | ## Standing boundaries & gotchas @@ -168,10 +169,10 @@ transport-adjacent. The rest, imperatively: renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or identity material. -- **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`, +- **Partial clones refuse cherry-pick/rebase, `sc ws fork`/`harvest`, `sc work`, `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`, - `sc grant`/`revoke`, creating, switching to or leaving a private branch, `sc push` to a git remote, and - `sc fetch` from any native remote but the promisor origin.** `sc backfill --all` + `sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge that has conflicts (no ref, state or working-tree change), `sc push` to a git remote, and + `sc fetch` from any native remote but the promisor origin.** A clean `sc merge` works (P37c-ii). `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what a clone *asked for*, not what is present: after a fetch an in-filter object can be absent. On a lazy, online clone a switch, fast-forward, rebase, undo diff --git a/ROADMAP.md b/ROADMAP.md index 89ddfab4..3b526251 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -558,11 +558,11 @@ clone: the view closure is filled by id before switch, fast-forward, undo and `sparse disable` stay refused. Design and open questions in [ADR-0048](docs/adr/0048-presence-based-partial-clones.md). -**P37c — merge on partial clones (proposed; 37c-i engine built, not yet wired).** A tree-recursive three-way merge +**P37c — merge on partial clones (proposed; 37c-i engine and 37c-ii clean merge built).** A tree-recursive three-way merge that carries a subtree by id where the three sides agree, checks protection rules before carrying, never drops wraps for subtrees it does not enumerate, and fills (lazy) or refuses (otherwise) the region where the sides differ, -the region that intersects a protection rule, and the sparse view's closure. Clean merges first, then conflicts, then cherry-pick/rebase; sessions +the region that intersects a protection rule, and the sparse view's closure. Clean merges are built (`sc merge` on a partial clone; a conflict is refused before any write). Conflicts are next, then cherry-pick/rebase; sessions are undecided. Design in [ADR-0049](docs/adr/0049-merge-on-partial-clones.md). diff --git a/crates/cli/tests/partial_clone_safety.rs b/crates/cli/tests/partial_clone_safety.rs index 0395292a..b473b820 100644 --- a/crates/cli/tests/partial_clone_safety.rs +++ b/crates/cli/tests/partial_clone_safety.rs @@ -165,9 +165,14 @@ fn status_diff_and_gc_succeed_on_fresh_partial_clone() { fn backfill_all_lifts_partial_clone_refusals() { let (src_root, dst_root) = make_partial_clone("backfill-all"); - // Merge is refused while partial: create a divergent (non-conflicting — - // different files, so a clean merge is the only thing this test is - // exercising) branch first. + // A merge with conflicts is refused while partial (a clean one works, P37c-ii). + // Create a divergent, non-conflicting branch (different files) for the clean + // merge after backfill, and a clashing one (same file) for the refusal. + assert!(sc(&dst_root, &["branch", "clash"]).status.success()); + assert!(sc(&dst_root, &["switch", "clash"]).status.success()); + std::fs::write(dst_root.join("src/a.txt"), "clash-edit\n").unwrap(); + assert!(sc(&dst_root, &["commit", "-m", "clash"]).status.success()); + assert!(sc(&dst_root, &["switch", "main"]).status.success()); assert!(sc(&dst_root, &["branch", "feature"]).status.success()); assert!(sc(&dst_root, &["switch", "feature"]).status.success()); std::fs::write(dst_root.join("src/feature.txt"), "feature-only\n").unwrap(); @@ -176,16 +181,24 @@ fn backfill_all_lifts_partial_clone_refusals() { std::fs::write(dst_root.join("src/a.txt"), "main-edit\n").unwrap(); assert!(sc(&dst_root, &["commit", "-m", "main"]).status.success()); - let merge_before = sc(&dst_root, &["merge", "feature"]); + let merge_before = sc(&dst_root, &["merge", "clash"]); assert!( !merge_before.status.success(), - "merge must still be refused before backfill --all" + "a conflicting merge must still be refused before backfill --all" ); assert!( stderr(&merge_before).contains("backfill --all"), "refusal must point at `sc backfill --all`: {}", stderr(&merge_before) ); + assert!( + !dst_root.join("src/a.txt.theirs").exists(), + "a refused merge writes nothing" + ); + assert_eq!( + std::fs::read_to_string(dst_root.join("src/a.txt")).unwrap(), + "main-edit\n" + ); let backfill = sc(&dst_root, &["backfill", "--all"]); assert!( @@ -207,6 +220,22 @@ fn backfill_all_lifts_partial_clone_refusals() { stderr(&sparse_disable) ); + // The conflicting merge is no longer refused: it reaches the ordinary + // conflict state (markers in the file), which `--abort` undoes. + let conflict_after = sc(&dst_root, &["merge", "clash"]); + assert!( + !stderr(&conflict_after).contains("partial clone"), + "a conflicting merge must no longer be refused after backfill --all: {}", + stderr(&conflict_after) + ); + assert!( + std::fs::read_to_string(dst_root.join("src/a.txt")) + .unwrap() + .contains("<<<<<<<"), + "the ordinary conflict markers are written" + ); + assert!(sc(&dst_root, &["merge", "--abort"]).status.success()); + let merge_after = sc(&dst_root, &["merge", "feature"]); assert!( merge_after.status.success(), diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index 278af353..af05851d 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -169,6 +169,21 @@ impl Repo { Ok(filler.stats) } + /// A filler for one tree merge's rounds (ADR-0049 decision 5): the + /// connection to the promisor origin opens on the first fill and is reused + /// by every later round of the same merge. + pub(crate) fn needs_filler(&self) -> Result> { + self.needs_filler_impl(crate::promisor::offline()) + } + + /// `offline` is a parameter so tests can exercise `SC_OFFLINE` without + /// mutating the process environment. + fn needs_filler_impl(&self, offline: bool) -> Result> { + Ok(NeedsFiller { + filler: Filler::new(self, offline)?, + }) + } + /// The bytes of the file at `path` as of `refname`'s tip (`sc cat /// :`). On a lazy partial clone, trees and the blob missing /// locally are fetched by id on the way down (one round trip per tree @@ -254,6 +269,50 @@ impl Repo { } } +/// Fills what a tree merge asks for, round after round, over one connection. +pub(crate) struct NeedsFiller<'a> { + filler: Filler<'a>, +} + +impl NeedsFiller<'_> { + /// Make the objects a tree merge asked for present. Each `(id, path)` is + /// what `merge_trees` could not read and where it found it. A lazy, online + /// partial clone fetches them by id in batches; any other partial clone + /// fails with the gap error for the first path (`sc backfill `), and + /// with the whole-clone refusal when the gap is the root, which no prefix + /// can repair. Call it with the store lock released: the fill takes it. + /// Fetched objects stay (they are immutable CAS additions), so a retry + /// resumes from current presence. Returns what this call cost. + pub(crate) fn fill(&mut self, needs: &[(ObjectId, String)]) -> Result { + let Some((_, first)) = needs.first() else { + return Ok(FillStats::default()); + }; + let before = self.filler.stats; + let ids: BTreeSet = needs.iter().map(|(id, _)| *id).collect(); + if let Err(e) = self.filler.ensure_many(&ids, first) { + return Err(match e { + Error::GapOutsideFilter(p) if p.is_empty() => { + crate::promisor::partial_clone_unsupported("merge") + } + other => other, + }); + } + // The batch is all-or-nothing on the wire, but trust presence, not the + // reply: the caller's no-progress check rests on it. + let store_arc = self.filler.repo.vfs.store(); + let store = store_arc.lock().unwrap(); + if let Some(id) = ids.iter().find(|id| !store.contains(id)) { + return Err(Error::Core(scl_core::Error::NotFound(*id))); + } + drop(store); + let after = self.filler.stats; + Ok(FillStats { + requests: after.requests - before.requests, + objects: after.objects - before.objects, + }) + } +} + /// Makes objects present, dialing the promisor remote only when the clone is /// lazy and not offline. The connection opens on first need. struct Filler<'a> { diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 47bfe770..7f25671f 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -14,13 +14,12 @@ pub mod ignore; pub mod layout; pub mod lock; pub mod merge; -// Wired into `merge` on partial clones in P37c-ii (ADR-0049); until then only -// its differential tests call it. pub mod merge_state; -#[cfg_attr(not(test), allow(dead_code))] pub(crate) mod merge_tree; pub mod oplog; #[cfg(test)] +mod partial_merge; +#[cfg(test)] mod partial_walks; pub mod pick_state; pub mod private; diff --git a/crates/repo/src/merge.rs b/crates/repo/src/merge.rs index 3ef14574..ba3bad4e 100644 --- a/crates/repo/src/merge.rs +++ b/crates/repo/src/merge.rs @@ -562,11 +562,18 @@ fn plain_input( identity: Option<&scl_crypto::SecretKey>, path: &str, ) -> Result>> { + // The identity requirement is checked before the blob is read, so a lazy + // partial clone refuses a keyless merge of a protected path without first + // fetching its ciphertext (ADR-0049 open question 1). + let sk = if entry.2 & PROTECTED == 0 { + None + } else { + Some(identity.ok_or_else(|| Error::ProtectedMergeNeedsIdentity(path.to_string()))?) + }; let bytes = blob_bytes(store, entry.0)?; - if entry.2 & PROTECTED == 0 { + let Some(sk) = sk else { return Ok(scl_crypto::Zeroizing::new(bytes)); - } - let sk = identity.ok_or_else(|| Error::ProtectedMergeNeedsIdentity(path.to_string()))?; + }; protect::decrypt_with(&bytes, &entry.0, prots, sk, path) } diff --git a/crates/repo/src/merge_tree.rs b/crates/repo/src/merge_tree.rs index 2dc478d5..f2186659 100644 --- a/crates/repo/src/merge_tree.rs +++ b/crates/repo/src/merge_tree.rs @@ -30,6 +30,7 @@ use std::collections::{BTreeMap, BTreeSet}; use scl_core::{EntryKind, FileMode, ObjectId, Protection, Store, TreeEntry}; use crate::error::{Error, Result}; +use crate::fill::FillStats; use crate::merge::{check_path_collisions, merge_path, FileEntry, FileMerge, MergeAcc}; /// What the merge carried by id instead of reading. @@ -225,6 +226,62 @@ pub(crate) fn merge_trees( })) } +impl crate::repo::Repo { + /// Run [`merge_trees`] to completion on this repo's store, filling what it + /// asks for between passes (ADR-0049 decision 5). Every pass takes the + /// store lock and releases it before the fill, which takes it too. A pass + /// that needs objects discards its work, so a round is idempotent and an + /// interrupted fill resumes from current presence. + /// + /// An id asked for again after it was filled means the store reports it + /// present while the traversal cannot read it (a stale index entry): that + /// would loop forever, so it fails instead. + pub(crate) fn tree_merge_filled( + &self, + base: Option<(ObjectId, &Protection)>, + ours: (ObjectId, &Protection), + theirs: (ObjectId, &Protection), + identity: Option<&scl_crypto::SecretKey>, + ) -> Result<(TreeMergeDone, FillStats)> { + let mut filler = self.needs_filler()?; + fill_loop( + || { + let store_arc = self.vfs().store(); + let mut store = store_arc.lock().unwrap(); + merge_trees(&mut store, base, ours, theirs, identity) + }, + |needs| filler.fill(needs), + ) + } +} + +/// The pass-then-fill loop of [`Repo::tree_merge_filled`], with the pass and +/// the fill as parameters so the no-progress guard can be tested without a +/// store whose index disagrees with itself. +fn fill_loop( + mut pass: impl FnMut() -> Result, + mut fill: impl FnMut(&[(ObjectId, String)]) -> Result, +) -> Result<(TreeMergeDone, FillStats)> { + let mut stats = FillStats::default(); + let mut asked: BTreeSet = BTreeSet::new(); + loop { + let needs = match pass()? { + TreeMerge::Done(done) => return Ok((done, stats)), + TreeMerge::Needs(needs) => needs, + }; + if let Some((id, path)) = needs.iter().find(|(id, _)| asked.contains(id)) { + return Err(Error::Protocol(format!( + "merge asked again for object {id} (at {path:?}) after it was fetched; \ + the object store and its index disagree" + ))); + } + asked.extend(needs.iter().map(|(id, _)| *id)); + let round = fill(&needs)?; + stats.requests += round.requests; + stats.objects += round.objects; + } +} + /// Visit one name at `path` with its (up to three) entries. fn visit(store: &mut Store, w: &mut Walk, path: &str, ents: [Option; 3]) -> Result<()> { let present: Vec<&TreeEntry> = ents.iter().flatten().collect(); @@ -1082,4 +1139,69 @@ mod tests { let err = merge_trees(&mut s, None, (ours, &p), (theirs, &p), None).unwrap_err(); assert!(matches!(err, Error::PathCollision(_)), "got {err:?}"); } + + fn empty_done() -> TreeMerge { + TreeMerge::Done(TreeMergeDone { + merge: FileMerge { + files: vec![], + sidecars: vec![], + conflicts: vec![], + wrapped_carry: Default::default(), + }, + carried: Carried::Entries(vec![]), + }) + } + + #[test] + fn fill_loop_fills_then_finishes_and_sums_the_cost() { + let a = scl_core::Object::blob(b"a".to_vec()).id(); + let b = scl_core::Object::blob(b"b".to_vec()).id(); + let mut passes = vec![ + TreeMerge::Needs(vec![(a, "x".into())]), + TreeMerge::Needs(vec![(b, "x/y".into())]), + empty_done(), + ] + .into_iter(); + let mut filled: Vec = Vec::new(); + let (_, stats) = fill_loop( + || Ok(passes.next().unwrap()), + |needs| { + filled.extend(needs.iter().map(|(id, _)| *id)); + Ok(FillStats { + requests: 1, + objects: needs.len(), + }) + }, + ) + .unwrap(); + assert_eq!(filled, vec![a, b]); + assert_eq!( + stats, + FillStats { + requests: 2, + objects: 2 + } + ); + } + + #[test] + fn fill_loop_fails_when_a_filled_id_is_asked_for_again() { + // The store claims to hold the object (the fill succeeds) while the + // traversal still cannot read it: without the guard this spins forever. + let a = scl_core::Object::blob(b"a".to_vec()).id(); + let mut rounds = 0; + let err = fill_loop( + || Ok(TreeMerge::Needs(vec![(a, "x".into())])), + |_| { + rounds += 1; + Ok(FillStats::default()) + }, + ) + .unwrap_err(); + assert_eq!(rounds, 1, "one fill, then the repeat is caught"); + assert!( + matches!(&err, Error::Protocol(m) if m.contains("asked again")), + "got {err:?}" + ); + } } diff --git a/crates/repo/src/partial_merge.rs b/crates/repo/src/partial_merge.rs new file mode 100644 index 00000000..f5e5d8c8 --- /dev/null +++ b/crates/repo/src/partial_merge.rs @@ -0,0 +1,447 @@ +//! Tests for clean merges on a partial clone (ADR-0049, P37c-ii). +//! +//! The fixture is an origin with `src/`, `docs/`, `docs/api/` and `lib/`, a +//! `--filter src/` clone of it, and two origin branches `b1` and `b2` cut from +//! the first commit. The clone's view is `src/`, so everything the tests touch +//! under `docs/` is outside it. + +use std::path::{Path, PathBuf}; + +use scl_core::{Object, ObjectId}; + +use crate::error::Error; +use crate::refs; +use crate::repo::Repo; + +const SRC_A: &[u8] = b"one\ntwo\nthree\n"; + +struct Fx { + origin_root: PathBuf, + clone_root: PathBuf, + origin: Repo, +} + +impl Fx { + /// An origin on `main` at one commit, with `extra` written into that first + /// commit by the caller-supplied closure. + fn origin(tag: &str, extra: impl FnOnce(&Path)) -> Fx { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p37c-{tag}-origin-{pid}")); + let clone_root = base.join(format!("scl-p37c-{tag}-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + for d in ["src", "docs/api", "lib"] { + std::fs::create_dir_all(origin_root.join(d)).unwrap(); + } + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("src/a.txt"), SRC_A).unwrap(); + std::fs::write(origin_root.join("docs/a.md"), b"a\n").unwrap(); + std::fs::write(origin_root.join("docs/b.md"), b"b\n").unwrap(); + std::fs::write(origin_root.join("docs/api/guide.md"), b"g\n").unwrap(); + std::fs::write(origin_root.join("lib/x.txt"), b"x\n").unwrap(); + extra(&origin_root); + origin.commit("t", "c1").unwrap(); + Fx { + origin_root, + clone_root, + origin, + } + } + + /// Cut `name` from the origin's `main`, write `files` on it, commit, and + /// return to `main`. Returns the new tip. + fn origin_branch(&self, name: &str, files: &[(&str, &[u8])]) -> ObjectId { + self.origin.branch(name).unwrap(); + self.origin.switch(name).unwrap(); + for (p, bytes) in files { + std::fs::write(self.origin_root.join(p), bytes).unwrap(); + } + let tip = self.origin.commit("t", &format!("edit on {name}")).unwrap(); + self.origin.switch("main").unwrap(); + tip + } + + /// A `--filter src/` clone, lazy or not, that has fetched every origin branch. + fn clone(&self, lazy: bool) -> Repo { + let clone = Repo::clone_url_filtered( + self.origin_root.to_str().unwrap(), + &self.clone_root, + Some(&["src/".to_string()]), + ) + .unwrap(); + crate::promisor::set_lazy(clone.layout(), lazy).unwrap(); + clone.fetch("origin").unwrap(); + clone + } + + fn cleanup(self) { + drop(self.origin); + for r in [&self.origin_root, &self.clone_root] { + if r.exists() { + std::fs::remove_dir_all(r).unwrap(); + } + assert!(!r.exists(), "fixture {} must be removed", r.display()); + } + } +} + +fn has(repo: &Repo, id: &ObjectId) -> bool { + repo.vfs().store().lock().unwrap().contains(id) +} + +fn blob_id(bytes: &[u8]) -> ObjectId { + Object::blob(bytes.to_vec()).id() +} + +/// The tree id of directory `name` directly under `snap`'s root. +fn dir_id(repo: &Repo, snap: &ObjectId, name: &str) -> ObjectId { + let arc = repo.vfs().store(); + let mut s = arc.lock().unwrap(); + let root = s.get_snapshot(snap).unwrap().root; + s.get_tree(&root).unwrap().get(name).unwrap().id +} + +/// Commit an edit to `src/a.txt` on the clone's `main`. +fn clone_edit_src(fx: &Fx, clone: &Repo, bytes: &[u8]) { + std::fs::write(fx.clone_root.join("src/a.txt"), bytes).unwrap(); + clone.commit("t", "ours").unwrap(); +} + +/// Disjoint edits merge without reading, or fetching, the other side's region. +/// The clone is not lazy, so a merge that needed the network could not succeed. +#[test] +fn disjoint_merge_carries_the_other_sides_subtree_without_a_fetch() { + let fx = Fx::origin("carry", |_| {}); + let b1 = fx.origin_branch("b1", &[("docs/api/guide.md", b"g changed\n")]); + let theirs_docs = dir_id(&fx.origin, &b1, "docs"); + let clone = fx.clone(false); + assert!( + !has(&clone, &theirs_docs), + "premise: the docs tree is a gap" + ); + + clone_edit_src(&fx, &clone, b"one\nTWO\nthree\n"); + let ours = refs::read_branch_tip(clone.layout(), "main").unwrap(); + let (merged, _) = clone.merge_with_identity("origin/b1", "t", None).unwrap(); + + let snap = clone.snapshot(&merged).unwrap(); + assert_eq!(snap.parents.len(), 2); + assert_eq!(Some(snap.parents[0]), ours); + assert_eq!( + dir_id(&clone, &merged, "docs"), + theirs_docs, + "their docs tree is carried by id" + ); + assert!(!has(&clone, &theirs_docs), "and never read or fetched"); + assert_eq!( + std::fs::read(fx.clone_root.join("src/a.txt")).unwrap(), + b"one\nTWO\nthree\n" + ); + assert!(!fx.clone_root.join("docs").exists()); + assert!(!clone.merge_in_progress()); + // The merged snapshot references a subtree the store does not hold: gc and + // push must walk what is present and cope (ADR-0048). + clone.gc(std::time::Duration::from_secs(0)).unwrap(); + assert!(has(&clone, &merged), "gc keeps the merge"); + assert!(!has(&clone, &theirs_docs), "and does not invent the gap"); + drop(clone); + fx.cleanup(); +} + +/// A region both sides changed is outside the view: a lazy clone fetches it by +/// id (trees only, the blobs are decided by id) and a non-lazy one is refused +/// before it writes anything. +#[test] +fn differing_region_outside_the_view_fills_when_lazy_and_is_refused_otherwise() { + let fx = Fx::origin("fill", |_| {}); + let b1 = fx.origin_branch("b1", &[("docs/a.md", b"a1\n")]); + let b2 = fx.origin_branch("b2", &[("docs/b.md", b"b2\n")]); + let clone = fx.clone(false); + clone.merge("origin/b1", "t").unwrap(); // fast-forward + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(b1) + ); + + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + let err = clone.merge("origin/b2", "t").unwrap_err(); + assert!( + matches!(&err, Error::GapOutsideFilter(p) if p == "docs"), + "got {err:?}" + ); + assert!(err.to_string().contains("sc backfill docs"), "{err}"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before, + "refused before any write" + ); + assert!(!clone.merge_in_progress()); + assert!(!fx.clone_root.join("docs").exists()); + + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + let (merged, _) = clone.merge_with_identity("origin/b2", "t", None).unwrap(); + let snap = clone.snapshot(&merged).unwrap(); + assert_eq!(snap.parents, vec![b1, b2]); + // The differing region was read: its trees and the two changed blobs + // arrived. The untouched `docs/api` subtree is identical on all three sides, + // so it was carried and never fetched. Nothing is on disk: docs/ is outside + // the view. + assert!(has(&clone, &blob_id(b"a1\n"))); + assert!(has(&clone, &blob_id(b"b2\n"))); + let api = dir_id(&clone, &b1, "docs"); + let api = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + s.get_tree(&api).unwrap().get("api").unwrap().id + }; + assert!(!has(&clone, &api), "the identical subtree stays a gap"); + assert!(!fx.clone_root.join("docs").exists()); + let docs = dir_id(&clone, &merged, "docs"); + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + let docs = s.get_tree(&docs).unwrap(); + assert_eq!(docs.get("a.md").unwrap().id, blob_id(b"a1\n")); + assert_eq!(docs.get("b.md").unwrap().id, blob_id(b"b2\n")); + assert_eq!(docs.get("api").unwrap().id, api); + drop(s); + drop(clone); + fx.cleanup(); +} + +/// A conflict is refused before any merge output, state or working-tree write. +#[test] +fn conflict_on_a_partial_clone_is_refused_before_any_write() { + let fx = Fx::origin("conflict", |_| {}); + fx.origin_branch("b1", &[("src/a.txt", b"ONE\ntwo\nthree\n")]); + fx.origin_branch("b2", &[("src/a.txt", b"uno\ntwo\nthree\n")]); + let clone = fx.clone(true); + clone.merge("origin/b1", "t").unwrap(); + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + + let err = clone.merge("origin/b2", "t").unwrap_err(); + assert!( + matches!(&err, Error::PartialCloneUnsupported(op) if op.contains("conflicts")), + "got {err:?}" + ); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before + ); + assert!(!clone.merge_in_progress()); + assert_eq!( + std::fs::read(fx.clone_root.join("src/a.txt")).unwrap(), + b"ONE\ntwo\nthree\n", + "no markers" + ); + assert!(!fx.clone_root.join("src/a.txt.theirs").exists()); + drop(clone); + fx.cleanup(); +} + +/// An origin whose `vault/` is protected for `pk` (rule plus ciphertext). +fn protected_origin(tag: &str, pk: &scl_crypto::PublicKey) -> Fx { + let fx = Fx::origin(tag, |root| { + std::fs::create_dir_all(root.join("vault")).unwrap(); + std::fs::write(root.join("vault/s.txt"), b"l1\nl2\nl3\n").unwrap(); + }); + fx.origin + .protect("vault/", std::slice::from_ref(pk), None) + .unwrap(); + fx +} + +/// A protected subtree no side touched is carried, so its blobs were never +/// enumerated: the merged snapshot must keep every wrap, or no recipient could +/// decrypt the carried files again (ADR-0049 decision 4). +#[test] +fn a_carried_protected_subtree_keeps_its_wraps() { + let (sk, pk) = scl_crypto::generate_keypair(); + let fx = protected_origin("wraps", &pk); + let base = fx.origin.head_tip().unwrap().unwrap(); + let base_snap = fx.origin.snapshot(&base).unwrap(); + assert!(!base_snap.protection.wrapped.is_empty(), "premise: sealed"); + let vault = dir_id(&fx.origin, &base, "vault"); + fx.origin_branch("b1", &[("docs/a.md", b"a1\n")]); + let clone = fx.clone(false); + assert!(!has(&clone, &vault), "premise: vault is outside the filter"); + + clone_edit_src(&fx, &clone, b"one\nTWO\nthree\n"); + let (merged, _) = clone.merge_with_identity("origin/b1", "t", None).unwrap(); + let snap = clone.snapshot(&merged).unwrap(); + assert_eq!(dir_id(&clone, &merged, "vault"), vault, "carried by id"); + assert!(!has(&clone, &vault), "and never fetched"); + for (id, wraps) in &base_snap.protection.wrapped { + assert_eq!( + snap.protection.wrapped.get(id), + Some(wraps), + "the wraps of {id} survive the merge byte for byte" + ); + } + // And they still open the carried ciphertext: the recipient decrypts the + // blob (read from the origin, which holds it) under the MERGED policy. + let (cipher_id, _) = base_snap.protection.wrapped.iter().next().unwrap(); + let cipher = { + let arc = fx.origin.vfs().store(); + let mut s = arc.lock().unwrap(); + match s.get(cipher_id).unwrap() { + Object::Blob(b) => b, + other => panic!("expected a blob, got {}", other.kind_name()), + } + }; + let plain = + crate::protect::decrypt_with(&cipher, cipher_id, &[&snap.protection], &sk, "vault/s.txt") + .unwrap(); + assert_eq!(&plain[..], b"l1\nl2\nl3\n"); + assert_eq!(snap.protection.prefixes, base_snap.protection.prefixes); + drop(clone); + fx.cleanup(); +} + +/// A keyless merge of a protected path both sides changed refuses without +/// fetching its ciphertext; with the identity it merges the content. +#[test] +fn keyless_protected_merge_refuses_before_fetching_the_ciphertext() { + let (sk, pk) = scl_crypto::generate_keypair(); + let fx = protected_origin("identity", &pk); + // The origin materialized `vault/s.txt` as plaintext at its protect commit; + // each branch rewrites it, so the edit is committed under the rule. + let b1 = fx.origin_branch("b1", &[("vault/s.txt", b"L1\nl2\nl3\n")]); + let b2 = fx.origin_branch("b2", &[("vault/s.txt", b"l1\nl2\nL3\n")]); + let cipher = |tip: &ObjectId| { + let arc = fx.origin.vfs().store(); + let mut s = arc.lock().unwrap(); + let root = s.get_snapshot(tip).unwrap().root; + let vault = s.get_tree(&root).unwrap().get("vault").unwrap().id; + s.get_tree(&vault).unwrap().get("s.txt").unwrap().id + }; + let (c1, c2) = (cipher(&b1), cipher(&b2)); + assert_ne!(c1, c2); + + let clone = fx.clone(true); + clone.merge("origin/b1", "t").unwrap(); // fast-forward + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + let err = clone.merge("origin/b2", "t").unwrap_err(); + assert!( + matches!(&err, Error::ProtectedMergeNeedsIdentity(p) if p == "vault/s.txt"), + "got {err:?}" + ); + assert!(!has(&clone, &c1), "ciphertext not fetched for a refusal"); + assert!(!has(&clone, &c2), "ciphertext not fetched for a refusal"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before + ); + + let (merged, _) = clone + .merge_with_identity("origin/b2", "t", Some(&sk)) + .unwrap(); + let snap = clone.snapshot(&merged).unwrap(); + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + let vault = s.get_tree(&snap.root).unwrap().get("vault").unwrap().id; + let entry = s.get_tree(&vault).unwrap().get("s.txt").unwrap().clone(); + assert_ne!(entry.perms & scl_core::PROTECTED, 0, "sealed again"); + assert!( + snap.protection.wrapped.contains_key(&entry.id), + "with fresh wraps for the merged content" + ); + // The merged plaintext reaches neither the store nor the disk; the sealed + // blob decrypts for the recipient to the clean three-way result. + assert!( + !s.contains(&blob_id(b"L1\nl2\nL3\n")), + "no plaintext in the CAS" + ); + let sealed = match s.get(&entry.id).unwrap() { + Object::Blob(b) => b, + other => panic!("expected a blob, got {}", other.kind_name()), + }; + let plain = + crate::protect::decrypt_with(&sealed, &entry.id, &[&snap.protection], &sk, "vault/s.txt") + .unwrap(); + assert_eq!(&plain[..], b"L1\nl2\nL3\n"); + assert!(!fx.clone_root.join("vault").exists(), "outside the view"); + drop(s); + drop(clone); + fx.cleanup(); +} + +/// The merged tree carries a subtree that is absent inside the sparse view: +/// the view's closure is filled before anything is written, on a lazy clone, +/// and a non-lazy one fails with the view-gap error and changes nothing. +#[test] +fn a_carried_subtree_inside_the_view_is_filled_before_materializing() { + let fx = Fx::origin("view", |_| {}); + fx.origin_branch("b1", &[("docs/api/guide.md", b"g changed\n")]); + let clone = fx.clone(true); + clone + .set_sparse(&["src/".to_string(), "docs/".to_string()], None) + .unwrap(); + assert!(fx.clone_root.join("docs/api/guide.md").exists()); + clone_edit_src(&fx, &clone, b"one\nTWO\nthree\n"); + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + + // Their docs subtree is carried, so it is not read, yet it lies in the view. + crate::promisor::set_lazy(clone.layout(), false).unwrap(); + let err = clone.merge("origin/b1", "t").unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before + ); + assert_eq!( + std::fs::read(fx.clone_root.join("docs/api/guide.md")).unwrap(), + b"g\n", + "nothing was laid out" + ); + assert!(!clone.merge_in_progress()); + + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.merge("origin/b1", "t").unwrap(); + assert_eq!( + std::fs::read(fx.clone_root.join("docs/api/guide.md")).unwrap(), + b"g changed\n" + ); + drop(clone); + fx.cleanup(); +} + +/// A fill interrupted partway leaves its objects behind and nothing else; the +/// retry resumes from what is present and succeeds. +#[test] +fn an_interrupted_fill_changes_nothing_and_the_retry_succeeds() { + let fx = Fx::origin("retry", |_| {}); + let b1 = fx.origin_branch("b1", &[("docs/a.md", b"a1\n")]); + let b2 = fx.origin_branch("b2", &[("docs/b.md", b"b2\n")]); + let clone = fx.clone(true); + clone.merge("origin/b1", "t").unwrap(); // fast-forward + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + + // Round one by hand: fetch the docs trees of both sides, then lose the origin + // before the blobs arrive. + let docs = [&b1, &b2].map(|tip| dir_id(&fx.origin, tip, "docs")); + let needs: Vec<(ObjectId, String)> = docs.iter().map(|id| (*id, "docs".to_string())).collect(); + clone.needs_filler().unwrap().fill(&needs).unwrap(); + assert!(docs.iter().all(|id| has(&clone, id)), "trees arrived"); + let gone = fx.origin_root.with_extension("gone"); + std::fs::rename(&fx.origin_root, &gone).unwrap(); + let err = clone.merge("origin/b2", "t").unwrap_err(); + assert!( + !matches!(err, Error::MergeConflicts(_) | Error::UpToDate), + "got {err:?}" + ); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before + ); + assert!(!clone.merge_in_progress()); + assert!(!fx.clone_root.join("docs").exists()); + assert!(docs.iter().all(|id| has(&clone, id)), "progress is kept"); + + std::fs::rename(&gone, &fx.origin_root).unwrap(); + let (merged, _) = clone.merge_with_identity("origin/b2", "t", None).unwrap(); + assert_eq!(clone.snapshot(&merged).unwrap().parents, vec![b1, b2]); + drop(clone); + fx.cleanup(); +} diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 7287a7d4..71e26654 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -1344,22 +1344,68 @@ impl Repo { } } - // Merge/pick completion guard (P27 Task 5, T5-I4): a real three-way - // merge flattens BOTH sides' full trees (`crate::merge::three_way`); - // on a partial clone that would touch out-of-filter content this - // clone never fetched. `three_way`'s unfiltered walk already fails - // with a raw `NotFound` there today — refuse explicitly and loudly - // here instead, before ever calling it, so the failure names the - // real cause (a partial clone) and points at `sc backfill` rather - // than surfacing as a confusing corruption-shaped error. Only the - // fast-forward/adopt paths above (which never rebuild a tree, only - // gap-tolerant `materialize`) are exempt. - if self.promisor()?.is_some() { - return Err(crate::promisor::partial_clone_unsupported("merge")); - } + // A partial clone cannot flatten both sides: most of their objects are + // absent. It merges tree by tree instead (ADR-0049), reading only the + // regions where the sides differ or a protection rule applies, and + // fetching what those need by id on a lazy, online clone (outside the + // store lock). Everything else is carried by id, unread. + let partial = self.promisor()?.is_some(); // Real three-way merge. - let (merge_result, ours_root, ours_protection, theirs_protection) = { + let (merge_result, tree_carried, ours_root, ours_protection, theirs_protection) = if partial + { + let (base_snap, ours_snap, theirs_snap) = { + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); + let base = crate::merge::merge_base(&mut store, ours, theirs)? + .ok_or(Error::NoCommonAncestor)?; + ( + store.get_snapshot(&base)?, + store.get_snapshot(&ours)?, + store.get_snapshot(&theirs)?, + ) + }; + let secrets = crate::merge::merge_secrets( + &base_snap.secrets, + &ours_snap.secrets, + &theirs_snap.secrets, + )?; + let (done, _fill) = self.tree_merge_filled( + Some((base_snap.root, &base_snap.protection)), + (ours_snap.root, &ours_snap.protection), + (theirs_snap.root, &theirs_snap.protection), + identity, + )?; + // Conflicts need a state contract this phase does not build + // (ADR-0049 decision 8): refuse before any merge output, merge + // state or working-tree write. + if !done.merge.conflicts.is_empty() { + return Err(crate::promisor::partial_clone_unsupported( + "a merge with conflicts", + )); + } + let wrapped_carry = done.merged_wraps( + &ours_snap.protection, + &theirs_snap.protection, + &base_snap.protection, + ); + let carried_any = done.carried_any(); + let fm = done.merge; + let m = crate::merge::Merge { + files: fm.files, + sidecars: fm.sidecars, + conflicts: fm.conflicts, + secrets, + wrapped_carry, + }; + ( + m, + Some((done.carried, carried_any)), + ours_snap.root, + ours_snap.protection, + theirs_snap.protection, + ) + } else { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); let base = crate::merge::merge_base(&mut store, ours, theirs)? @@ -1369,7 +1415,7 @@ impl Repo { let ours_root = store.get_snapshot(&ours)?.root; let ours_protection = store.get_snapshot(&ours)?.protection; let theirs_protection = store.get_snapshot(&theirs)?.protection; - (m, ours_root, ours_protection, theirs_protection) + (m, None, ours_root, ours_protection, theirs_protection) }; // Union protection rules across both sides: governs which recipients a @@ -1471,7 +1517,18 @@ impl Repo { carried.extend(encrypted); let all = carried; - let merged_root = self.vfs.write_tree_with_perms(&all)?; + // Carried subtrees (partial clone only) are overlaid by id; nothing + // below them was read, and the plaintext rules above never saw them. + let merged_root = match &tree_carried { + Some((crate::merge_tree::Carried::Root(id), _)) => { + debug_assert!(all.is_empty(), "a carried root leaves nothing enumerated"); + *id + } + Some((crate::merge_tree::Carried::Entries(entries), _)) => { + self.vfs.write_tree_with_carried(&all, entries)? + } + None => self.vfs.write_tree_with_perms(&all)?, + }; // Merged wrap map: union the carried wraps (from `wrapped_carry`, for // ciphertext that survived unchanged/one-sided) with the freshly @@ -1488,7 +1545,11 @@ impl Repo { // Prune to blobs actually reachable in the merged tree (commit's // rebuild discipline — a path that lost its rule mid-merge and no // longer resolves to that blob must not leave a stale wrap entry). - { + // Not when a subtree was carried: its blobs were never enumerated, so + // pruning would drop their wraps for good. The map is then the + // complete input union (ADR-0049 decision 4). + let carried_any = tree_carried.as_ref().is_some_and(|(_, any)| *any); + if !carried_any { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); let reachable: std::collections::BTreeSet = @@ -1503,6 +1564,11 @@ impl Repo { wrapped, }; + // The merged tree may carry a subtree that is absent inside the sparse + // view: fill its closure now (lazy partial clones only), before any ref, + // spec or working-tree write and outside the store lock. + self.ensure_view(merged_root, &self.sparse_spec()?)?; + // Materialize the merged tree into the working dir. Protection-aware: // a merged PROTECTED entry decrypts for `identity` when possible, else // is skipped (never writes ciphertext to disk) and reported to the @@ -1514,7 +1580,6 @@ impl Repo { let skipped = { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); - // ensure-view-exempt: conflict materialization; merge is refused on partial clones worktree::materialize( &self.layout, &mut store, @@ -1618,7 +1683,7 @@ impl Repo { // skipped content, so nothing is lost by not surfacing the list // here (the `Err`/`Stopped` return can't carry it). let mut cache = self.open_protected_cache()?; - // ensure-view-exempt: conflict materialization; merge is refused on partial clones + // ensure-view-exempt: conflict materialization; a partial clone refuses conflicts before any write let _skipped = worktree::materialize( &self.layout, &mut store, diff --git a/crates/vfs/src/lib.rs b/crates/vfs/src/lib.rs index 85a0d003..b74865d7 100644 --- a/crates/vfs/src/lib.rs +++ b/crates/vfs/src/lib.rs @@ -106,7 +106,33 @@ impl Repo { map.insert(normalize(path), (id, *mode, *perms)); } } - self.build_subtree_inner(&map, "") + self.build_subtree_inner(&map, &BTreeMap::new(), "") + } + + /// Like [`write_tree_with_perms`][Self::write_tree_with_perms], and also + /// places `carried` subtrees verbatim: each `(path, entry)` is inserted at + /// `path` with its id, mode and perms untouched and nothing below it read. + /// Ancestor directories are built from `files` and the carried paths + /// together. A carried path that is also a file, or the ancestor of one, is + /// the caller's collision to reject first (`merge::check_path_collisions`). + pub fn write_tree_with_carried( + &self, + files: &[(String, Vec, FileMode, u8)], + carried: &[(String, TreeEntry)], + ) -> Result { + let mut map: BTreeMap = BTreeMap::new(); + { + let mut store = self.store.lock().unwrap(); + for (path, bytes, mode, perms) in files { + let id = store.put(Object::blob(bytes.clone()))?; + map.insert(normalize(path), (id, *mode, *perms)); + } + } + let carried: BTreeMap = carried + .iter() + .map(|(p, e)| (normalize(p), e.clone())) + .collect(); + self.build_subtree_inner(&map, &carried, "") } /// Fork an in-memory worktree from a base snapshot. Cheap: allocates only an @@ -133,7 +159,7 @@ impl Repo { .iter() .map(|(p, (id, mode))| (p.clone(), (*id, *mode, 0u8))) .collect(); - self.build_subtree_inner(&with_perms, "") + self.build_subtree_inner(&with_perms, &BTreeMap::new(), "") } /// The single tree builder: reads each blob entry's `perms` byte from the map @@ -142,6 +168,7 @@ impl Repo { fn build_subtree_inner( &self, files: &BTreeMap, + carried: &BTreeMap, prefix: &str, ) -> Result { // Partition entries directly under `prefix` into files vs subdirectories. @@ -165,13 +192,27 @@ impl Repo { } } } + for (path, entry) in carried { + let Some(rest) = strip_prefix_dir(path, prefix) else { + continue; + }; + match rest.split_once('/') { + None => entries.push(TreeEntry { + name: rest.to_string(), + ..entry.clone() + }), + Some((dir, _)) => { + subdirs.insert(dir.to_string(), ()); + } + } + } for dir in subdirs.keys() { let child_prefix = if prefix.is_empty() { format!("{dir}/") } else { format!("{prefix}{dir}/") }; - let sub_id = self.build_subtree_inner(files, &child_prefix)?; + let sub_id = self.build_subtree_inner(files, carried, &child_prefix)?; entries.push(TreeEntry { name: dir.clone(), kind: EntryKind::Tree, @@ -459,6 +500,51 @@ mod tests { Repo::new(Store::new(StoreConfig::default())) } + #[test] + fn write_tree_with_carried_places_subtrees_verbatim_beside_built_ones() { + let r = repo(); + // A subtree the store has never seen: only its entry is placed. + let absent = Object::blob(b"never stored".to_vec()).id(); + let carried = TreeEntry { + name: "ignored".into(), + kind: EntryKind::Tree, + id: absent, + mode: FileMode(0o700), + perms: 3, + }; + let root = r + .write_tree_with_carried( + &[ + ("a.txt".into(), b"a".to_vec(), FileMode::FILE, 0), + ("dir/b.txt".into(), b"b".to_vec(), FileMode::FILE, 0), + ], + &[ + ("dir/sub".into(), carried.clone()), + ("top".into(), carried.clone()), + ("only/deep/er".into(), carried), + ], + ) + .unwrap(); + let store = r.store(); + let mut s = store.lock().unwrap(); + let t = s.get_tree(&root).unwrap(); + let names: Vec<&str> = t.entries.iter().map(|e| e.name.as_str()).collect(); + assert_eq!(names, vec!["a.txt", "dir", "only", "top"]); + let top = t.get("top").unwrap(); + assert_eq!((top.id, top.mode, top.perms), (absent, FileMode(0o700), 3)); + let dir = s.get_tree(&t.get("dir").unwrap().id).unwrap(); + let dn: Vec<&str> = dir.entries.iter().map(|e| e.name.as_str()).collect(); + assert_eq!( + dn, + vec!["b.txt", "sub"], + "built and carried share a directory" + ); + assert_eq!(dir.get("sub").unwrap().id, absent); + let only = s.get_tree(&t.get("only").unwrap().id).unwrap(); + let deep = s.get_tree(&only.get("deep").unwrap().id).unwrap(); + assert_eq!(deep.get("er").unwrap().id, absent); + } + fn seed(repo: &Repo) -> ObjectId { repo.commit_files( &[ diff --git a/demo/run_lazy_fetch_demo.sh b/demo/run_lazy_fetch_demo.sh index 934caeba..a0c54fb1 100755 --- a/demo/run_lazy_fetch_demo.sh +++ b/demo/run_lazy_fetch_demo.sh @@ -158,4 +158,42 @@ mv "$ORIGIN.gone" "$ORIGIN" echo "undo after a widen, offline ✔" echo -echo "RESULT: lazy by-id fetch verified (path-only fetch, offline/non-lazy gap errors, http v5, widen/commit/push/undo)" +echo "=== 9: merge on a partial clone carries the other side's region unread (ADR-0049) ===" +MORIGIN="$W/morigin" +mkdir -p "$MORIGIN" +( + cd "$MORIGIN" + "$SC" init >/dev/null + mkdir -p src docs/api + printf 'one\ntwo\nthree\n' > src/a.txt + printf 'guide\n' > docs/api/guide.md + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > docs/api/big.bin + "$SC" commit -m "base" --author demo >/dev/null + "$SC" branch feature + "$SC" switch feature >/dev/null + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > docs/api/big.bin + cp docs/api/big.bin "$W/feature-big.bin" + "$SC" commit -m "feature edits docs" --author demo >/dev/null + "$SC" switch main >/dev/null +) +MERGE="$W/merge" +"$SC" clone --filter src/ --lazy "$MORIGIN" "$MERGE" >/dev/null +printf 'one\nTWO\nthree\n' > "$MERGE/src/a.txt" +(cd "$MERGE" && "$SC" commit -m "local src edit" --author demo >/dev/null && "$SC" fetch origin >/dev/null) +MBEFORE=$(objects "$MERGE") +(cd "$MERGE" && "$SC" merge origin/feature --author demo >/dev/null) || fail "disjoint merge on a partial clone must succeed" +[ ! -e "$MERGE/docs" ] || fail "docs/ is outside the view and must not be written" +[ "$(cat "$MERGE/src/a.txt")" = "$(printf 'one\nTWO\nthree')" ] || fail "the local src edit must survive the merge" +MAFTER=$(objects "$MERGE") +# The merge commit and the merged root/src trees are new; the 64 KiB docs blob is not. +[ "$(($MAFTER - $MBEFORE))" -le 4 ] || fail "a disjoint merge must not fetch the other side's region ($MBEFORE -> $MAFTER)" +echo "disjoint merge on a lazy partial clone: docs/ untouched and unfetched ($MBEFORE -> $MAFTER objects) ✔" +(cd "$MERGE" && "$SC" push origin >/dev/null) || fail "pushing the merge must work" +MFULL="$W/merge-full" +"$SC" clone "$MORIGIN" "$MFULL" >/dev/null +[ "$(cat "$MFULL/src/a.txt")" = "$(printf 'one\nTWO\nthree')" ] || fail "a full re-clone must see the local edit" +cmp "$MFULL/docs/api/big.bin" "$W/feature-big.bin" || fail "the merge must carry feature's docs blob byte-identical" +echo "push + full re-clone: merged result has both sides, docs byte-identical ✔" + +echo +echo "RESULT: lazy by-id fetch verified (path-only fetch, offline/non-lazy gap errors, http v5, widen/commit/push/undo, merge)" diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index eab8a4e2..e85a293a 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -1,6 +1,6 @@ # ADR-0049: Merge on partial clones (P37c) -- **Status:** Proposed (37c-i built: engine and differential tests; not yet wired into `sc merge`) +- **Status:** Proposed (37c-i engine and 37c-ii clean merge built; conflicts, replay and sessions open) - **Date:** 2026-10-17 - **Phase:** 37c - **Builds on:** ADR-0048 (presence-based partial clones), ADR-0047 (want-by-id fetch), ADR-0012 (three-way merge), ADR-0025 (protected merge and replay), ADR-0043 (randomized sealing) @@ -70,7 +70,7 @@ Over-retention costs size: wraps are inline in the snapshot, and an object above ### 5. The engine is pull-based; filling happens outside the store lock -`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(missing)`, where each entry is `(ObjectId, path)`: a subtree **or blob** it must read and does not hold (the leaf merge and the encryption step read blob bytes, so a differing or rule-intersecting region outside the filter needs its blobs too), with the path at which the traversal found it (an id alone does not identify a path, since subtrees alias, and an absent root has no path to recover from tree data; the root is reported as the empty path). The caller drops the lock, fills those ids (lazy and online; blobs are ordinary by-id requests, batched and split like trees) or fails with `GapInView` using the first reported path and its filter prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. +`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(missing)`, where each entry is `(ObjectId, path)`: a subtree **or blob** it must read and does not hold (the leaf merge and the encryption step read blob bytes, so a differing or rule-intersecting region outside the filter needs its blobs too), with the path at which the traversal found it (an id alone does not identify a path, since subtrees alias, and an absent root has no path to recover from tree data; the root is reported as the empty path). The caller drops the lock, fills those ids (lazy and online; blobs are ordinary by-id requests, batched and split like trees) or fails with the gap error for the first reported path (`GapOutsideFilter`, whose message names `sc backfill `; the whole-clone refusal when the path is the root) otherwise, and calls again. Each round is idempotent, and progress is by presence. The merged tree can also carry a subtree that is absent **inside the sparse view** (the fetch case from ADR-0048). So after the merge and before materializing, the caller runs `ensure_view(merged_root, sparse)`, as every other materializer site does. The claim that network traffic is bounded to the differing region is therefore "the differing region, the rule-intersecting region, and the view's closure". @@ -117,7 +117,7 @@ A second differential runs each merge on a partial clone and a full clone. The n ## Phasing - **37c-i, engine (built).** The content/mode split and the collision check on the flatten merge (with the ADR-0012 amendment and tests), `merge_trees` returning `TreeMerge` or `Needs`, the policy-intersection test, `TreeMergeDone::merged_wraps` (the wrap-union rule; assembling and pruning the merged snapshot's `protection.wrapped` from it is 37c-ii), and the differential tests (file sets with modes and perms, conflicts, sidecars, wraps, carried-subtree policy, and fills on a store with objects removed). Nothing calls the new engine yet except tests. Decision 9's cases built from real operations (protect, grant, revoke, with decryption by each identity), equal plaintext sealed independently, an empty-directory entry on a carried side, merge then gc, and an interrupted fill then retry need the repo layer and land with 37c-ii. -- **37c-ii, clean merge on a partial clone.** `merge_with_identity` uses the engine when the clone is partial: fill loop outside the lock, `ensure_view(merged_root)`, then the existing pipeline. A conflict is refused up front. Demo: two branches touching disjoint directories merge on a `--filter` clone without fetching either side's out-of-view content; a merge whose differing region is outside the view fills it on a lazy clone and is refused on a non-lazy one. +- **37c-ii, clean merge on a partial clone (built).** `merge_with_identity` uses the engine when the clone is partial: `Repo::tree_merge_filled` runs a pass under the store lock, fills what it asked for outside it (`Repo::fill_needs`, batched and split like every by-id fetch) and repeats; a conflict is refused with `PartialCloneUnsupported` before any write; the merged root is built by `Vfs::write_tree_with_carried` (enumerated files plus carried subtrees verbatim, or the carried root as is); `merged_wraps` is the wrap map, unpruned when anything was carried; `ensure_view(merged_root)` runs before the first ref, spec or working-tree write. The differing region's blobs are fetched too: the leaf merge returns file bytes even when one side wins by id, so a region both sides touched is read in full while an untouched subtree (identical on all three sides) is not. Cases from decision 9 that need real repo operations are covered by `partial_merge.rs`: disjoint carry without a fetch, the differing region (fill when lazy, gap error otherwise), a conflict refused, a carried protected subtree keeping its wraps, a keyless protected merge refusing before fetching (the merged result decrypts for the recipient and its plaintext never reaches the CAS), a carried subtree inside the view, and an interrupted fill followed by a retry. The carried wraps are checked byte for byte and by decrypting the carried ciphertext under the merged policy. Merge then gc is covered by the disjoint-carry test. Still open from decision 9: equal plaintext sealed independently, crossed revokes and regrants, an empty-directory entry on a carried side, and a randomized partial-versus-full differential. Demo: `demo/run_lazy_fetch_demo.sh` section 9. - **37c-iii, conflicts.** First an amendment to this record that fixes decision 8's contract, then markers for present in-view conflicts, completion from the decided root, abort, `sc conflicts` and `sc resolve`. - **37c-iv, replay.** `cherry-pick` and `rebase` call the same engine; `replay.rs:176` is the only guard that blocks them, and `three_way_files` is already their merge. Replay adds protected re-encryption under the replayed rules and the resumable-rebase state. - **Sessions** (`sc ws`, `sc work`): undecided. They materialize whole checkouts per agent and auto-merge on harvest. Whether they should work on a partial clone at all is a product question. @@ -132,8 +132,8 @@ A second differential runs each merge on a partial clone and a full clone. The n ## Open questions -1. **Ciphertext fetched before a refusal it cannot avoid.** `plain_input` reads a protected blob before it checks for an identity (`merge.rs:565`, `:569`), so a lazy partial merge of a content-divergent protected path would fetch the ciphertext and then refuse with `ProtectedMergeNeedsIdentity`. 37c-ii should check the identity requirement before reporting `Needs`. -2. **Fill progress.** `merge_trees` detects an absent tree with `store.contains` but an absent blob through a `NotFound` from `get`. If those ever disagree (a stale pack index entry), a round asks for an id the filler thinks is present. The 37c-ii fill loop must detect a round that makes no progress and fail. +1. **Ciphertext fetched before a refusal it cannot avoid.** *Resolved in 37c-ii:* `plain_input` checks the identity requirement before it reads the blob, so a keyless lazy merge refuses without fetching the ciphertext (test `keyless_protected_merge_refuses_before_fetching_the_ciphertext`). The identity-free paths are unchanged. +2. **Fill progress.** *Resolved in 37c-ii:* `Repo::tree_merge_filled` fails with `Error::Protocol` when a pass asks again for an id an earlier round filled (a stale index entry would otherwise loop forever), and `fill_needs` verifies presence after each fill. Covered by `fill_loop_fails_when_a_filled_id_is_asked_for_again`. 3. **Mode change against a delete.** Ours changes only the mode and theirs deletes the file: the delete wins silently (content decides, as before). Git reports modify/delete. Unchanged by this work; decide separately. 4. **Pre-existing collisions.** Before the collision check, a clean cherry-pick or rebase that produced a file and a directory with the same name may already have committed such a snapshot. The fix prevents new ones and does not repair old ones. 5. **A `sc verify` check for plaintext under a rule** (decision 3's accepted divergence). @@ -141,6 +141,9 @@ A second differential runs each merge on a partial clone and a full clone. The n 7. **Switching full clones to the new engine.** Out of scope; decide after the differential has soaked. 8. **Merging a public branch INTO a private one** routes through `merge_into_private` and stays refused on a partial clone (ADR-0048 q10). 9. **Empty-directory normalization.** Decision 7 accepts the divergence; revisit if it shows up in real histories. +10. **Fill round trips.** A leaf reads its inputs one at a time and a pass records only the first missing blob per path, so a diff3-merged file both sides edited can take up to three rounds (ours, theirs, base), and every round re-walks the region. One connection serves all rounds of a merge (`NeedsFiller`). Collecting every missing input of a leaf in one pass, or having a pass skip finished regions, is a follow-on; measure first. +11. **Gap error names one path.** On a non-lazy clone the error names the first missing path; a second gap shows only after the first `sc backfill`. Whether `sc backfill ` fetches the merge-base version of a file (the gap can be a blob path) is unverified. +12. **Plaintext copies across rounds.** A pass that returns `Needs` discards decrypted protected inputs held in plain `Vec`s that are not wiped. The full merge has the same exposure once; the fill loop repeats it per round. Memory hygiene only. ## Alternatives considered diff --git a/docs/adr/README.md b/docs/adr/README.md index 8961222a..fd814c60 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -59,7 +59,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (37a built) | 37 | | [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (37b-i, 37b-ii built) | 37b | -| [0049](0049-merge-on-partial-clones.md) | Merge on partial clones | Proposed (37c-i built) | 37c | +| [0049](0049-merge-on-partial-clones.md) | Merge on partial clones | Proposed (37c-i, 37c-ii built) | 37c | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5,