Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ the project tree if desired.

## Capability map (what's built, by phase)

All 36 phases, P37a, P37b-i and P37b-ii are built and tested. One line of current fact per phase; the
All 36 phases, P37a, P37b-i, P37b-ii and P37c-ii are built and tested. One line of current fact per phase; the
authoritative rationale and full semantics live in the linked ADR, the design
in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is
archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not
Expand Down Expand Up @@ -152,7 +152,8 @@ the code, those win.
| P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) |
| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) |
| P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat <ref>:<path>` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |
| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. Merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) |
| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) |
| P37c-ii | Clean merge on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill <path>`); a carried protected subtree keeps the complete input wrap union; a conflict is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. Cherry-pick/rebase and conflicts are next | [0049](docs/adr/0049-merge-on-partial-clones.md) |

## Standing boundaries & gotchas

Expand All @@ -168,10 +169,10 @@ transport-adjacent. The rest, imperatively:
renderer has five typed repository commands, no general filesystem/shell
capability, and no DTO field capable of carrying protected ciphertext or
identity material.
- **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`,
- **Partial clones refuse cherry-pick/rebase, `sc ws fork`/`harvest`,
`sc work`, `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`,
`sc grant`/`revoke`, creating, switching to or leaving a private branch, `sc push` to a git remote, and
`sc fetch` from any native remote but the promisor origin.** `sc backfill --all`
`sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge that has conflicts (no ref, state or working-tree change), `sc push` to a git remote, and
`sc fetch` from any native remote but the promisor origin.** A clean `sc merge` works (P37c-ii). `sc backfill --all`
converts to a genuine full clone and re-enables them. The filter records what
a clone *asked for*, not what is present: after a fetch an in-filter object
can be absent. On a lazy, online clone a switch, fast-forward, rebase, undo
Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -558,11 +558,11 @@ clone: the view closure is filled by id before switch, fast-forward, undo and
`sparse disable` stay refused. Design and open questions in
[ADR-0048](docs/adr/0048-presence-based-partial-clones.md).

**P37c — merge on partial clones (proposed; 37c-i engine built, not yet wired).** A tree-recursive three-way merge
**P37c — merge on partial clones (proposed; 37c-i engine and 37c-ii clean merge built).** A tree-recursive three-way merge
that carries a subtree by id where the three sides agree, checks protection
rules before carrying, never drops wraps for subtrees it does not enumerate,
and fills (lazy) or refuses (otherwise) the region where the sides differ,
the region that intersects a protection rule, and the sparse view's closure. Clean merges first, then conflicts, then cherry-pick/rebase; sessions
the region that intersects a protection rule, and the sparse view's closure. Clean merges are built (`sc merge` on a partial clone; a conflict is refused before any write). Conflicts are next, then cherry-pick/rebase; sessions
are undecided. Design in
[ADR-0049](docs/adr/0049-merge-on-partial-clones.md).

Expand Down
39 changes: 34 additions & 5 deletions crates/cli/tests/partial_clone_safety.rs
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,14 @@ fn status_diff_and_gc_succeed_on_fresh_partial_clone() {
fn backfill_all_lifts_partial_clone_refusals() {
let (src_root, dst_root) = make_partial_clone("backfill-all");

// Merge is refused while partial: create a divergent (non-conflicting —
// different files, so a clean merge is the only thing this test is
// exercising) branch first.
// A merge with conflicts is refused while partial (a clean one works, P37c-ii).
// Create a divergent, non-conflicting branch (different files) for the clean
// merge after backfill, and a clashing one (same file) for the refusal.
assert!(sc(&dst_root, &["branch", "clash"]).status.success());
assert!(sc(&dst_root, &["switch", "clash"]).status.success());
std::fs::write(dst_root.join("src/a.txt"), "clash-edit\n").unwrap();
assert!(sc(&dst_root, &["commit", "-m", "clash"]).status.success());
assert!(sc(&dst_root, &["switch", "main"]).status.success());
assert!(sc(&dst_root, &["branch", "feature"]).status.success());
assert!(sc(&dst_root, &["switch", "feature"]).status.success());
std::fs::write(dst_root.join("src/feature.txt"), "feature-only\n").unwrap();
Expand All @@ -176,16 +181,24 @@ fn backfill_all_lifts_partial_clone_refusals() {
std::fs::write(dst_root.join("src/a.txt"), "main-edit\n").unwrap();
assert!(sc(&dst_root, &["commit", "-m", "main"]).status.success());

let merge_before = sc(&dst_root, &["merge", "feature"]);
let merge_before = sc(&dst_root, &["merge", "clash"]);
assert!(
!merge_before.status.success(),
"merge must still be refused before backfill --all"
"a conflicting merge must still be refused before backfill --all"
);
assert!(
stderr(&merge_before).contains("backfill --all"),
"refusal must point at `sc backfill --all`: {}",
stderr(&merge_before)
);
assert!(
!dst_root.join("src/a.txt.theirs").exists(),
"a refused merge writes nothing"
);
assert_eq!(
std::fs::read_to_string(dst_root.join("src/a.txt")).unwrap(),
"main-edit\n"
);

let backfill = sc(&dst_root, &["backfill", "--all"]);
assert!(
Expand All @@ -207,6 +220,22 @@ fn backfill_all_lifts_partial_clone_refusals() {
stderr(&sparse_disable)
);

// The conflicting merge is no longer refused: it reaches the ordinary
// conflict state (markers in the file), which `--abort` undoes.
let conflict_after = sc(&dst_root, &["merge", "clash"]);
assert!(
!stderr(&conflict_after).contains("partial clone"),
"a conflicting merge must no longer be refused after backfill --all: {}",
stderr(&conflict_after)
);
assert!(
std::fs::read_to_string(dst_root.join("src/a.txt"))
.unwrap()
.contains("<<<<<<<"),
"the ordinary conflict markers are written"
);
assert!(sc(&dst_root, &["merge", "--abort"]).status.success());

let merge_after = sc(&dst_root, &["merge", "feature"]);
assert!(
merge_after.status.success(),
Expand Down
59 changes: 59 additions & 0 deletions crates/repo/src/fill.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,21 @@ impl Repo {
Ok(filler.stats)
}

/// A filler for one tree merge's rounds (ADR-0049 decision 5): the
/// connection to the promisor origin opens on the first fill and is reused
/// by every later round of the same merge.
pub(crate) fn needs_filler(&self) -> Result<NeedsFiller<'_>> {
self.needs_filler_impl(crate::promisor::offline())
}

/// `offline` is a parameter so tests can exercise `SC_OFFLINE` without
/// mutating the process environment.
fn needs_filler_impl(&self, offline: bool) -> Result<NeedsFiller<'_>> {
Ok(NeedsFiller {
filler: Filler::new(self, offline)?,
})
}

/// The bytes of the file at `path` as of `refname`'s tip (`sc cat
/// <ref>:<path>`). On a lazy partial clone, trees and the blob missing
/// locally are fetched by id on the way down (one round trip per tree
Expand Down Expand Up @@ -254,6 +269,50 @@ impl Repo {
}
}

/// Fills what a tree merge asks for, round after round, over one connection.
pub(crate) struct NeedsFiller<'a> {
filler: Filler<'a>,
}

impl NeedsFiller<'_> {
/// Make the objects a tree merge asked for present. Each `(id, path)` is
/// what `merge_trees` could not read and where it found it. A lazy, online
/// partial clone fetches them by id in batches; any other partial clone
/// fails with the gap error for the first path (`sc backfill <path>`), and
/// with the whole-clone refusal when the gap is the root, which no prefix
/// can repair. Call it with the store lock released: the fill takes it.
/// Fetched objects stay (they are immutable CAS additions), so a retry
/// resumes from current presence. Returns what this call cost.
pub(crate) fn fill(&mut self, needs: &[(ObjectId, String)]) -> Result<FillStats> {
let Some((_, first)) = needs.first() else {
return Ok(FillStats::default());
};
let before = self.filler.stats;
let ids: BTreeSet<ObjectId> = needs.iter().map(|(id, _)| *id).collect();
if let Err(e) = self.filler.ensure_many(&ids, first) {
return Err(match e {
Error::GapOutsideFilter(p) if p.is_empty() => {
crate::promisor::partial_clone_unsupported("merge")
}
other => other,
});
}
// The batch is all-or-nothing on the wire, but trust presence, not the
// reply: the caller's no-progress check rests on it.
let store_arc = self.filler.repo.vfs.store();
let store = store_arc.lock().unwrap();
if let Some(id) = ids.iter().find(|id| !store.contains(id)) {
return Err(Error::Core(scl_core::Error::NotFound(*id)));
}
drop(store);
let after = self.filler.stats;
Ok(FillStats {
requests: after.requests - before.requests,
objects: after.objects - before.objects,
})
}
}

/// Makes objects present, dialing the promisor remote only when the clone is
/// lazy and not offline. The connection opens on first need.
struct Filler<'a> {
Expand Down
5 changes: 2 additions & 3 deletions crates/repo/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,12 @@ pub mod ignore;
pub mod layout;
pub mod lock;
pub mod merge;
// Wired into `merge` on partial clones in P37c-ii (ADR-0049); until then only
// its differential tests call it.
pub mod merge_state;
#[cfg_attr(not(test), allow(dead_code))]
pub(crate) mod merge_tree;
pub mod oplog;
#[cfg(test)]
mod partial_merge;
#[cfg(test)]
mod partial_walks;
pub mod pick_state;
pub mod private;
Expand Down
13 changes: 10 additions & 3 deletions crates/repo/src/merge.rs
Original file line number Diff line number Diff line change
Expand Up @@ -562,11 +562,18 @@ fn plain_input(
identity: Option<&scl_crypto::SecretKey>,
path: &str,
) -> Result<scl_crypto::Zeroizing<Vec<u8>>> {
// The identity requirement is checked before the blob is read, so a lazy
// partial clone refuses a keyless merge of a protected path without first
// fetching its ciphertext (ADR-0049 open question 1).
let sk = if entry.2 & PROTECTED == 0 {
None
} else {
Some(identity.ok_or_else(|| Error::ProtectedMergeNeedsIdentity(path.to_string()))?)
};
let bytes = blob_bytes(store, entry.0)?;
if entry.2 & PROTECTED == 0 {
let Some(sk) = sk else {
return Ok(scl_crypto::Zeroizing::new(bytes));
}
let sk = identity.ok_or_else(|| Error::ProtectedMergeNeedsIdentity(path.to_string()))?;
};
protect::decrypt_with(&bytes, &entry.0, prots, sk, path)
}

Expand Down
122 changes: 122 additions & 0 deletions crates/repo/src/merge_tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ use std::collections::{BTreeMap, BTreeSet};
use scl_core::{EntryKind, FileMode, ObjectId, Protection, Store, TreeEntry};

use crate::error::{Error, Result};
use crate::fill::FillStats;
use crate::merge::{check_path_collisions, merge_path, FileEntry, FileMerge, MergeAcc};

/// What the merge carried by id instead of reading.
Expand Down Expand Up @@ -225,6 +226,62 @@ pub(crate) fn merge_trees(
}))
}

impl crate::repo::Repo {
/// Run [`merge_trees`] to completion on this repo's store, filling what it
/// asks for between passes (ADR-0049 decision 5). Every pass takes the
/// store lock and releases it before the fill, which takes it too. A pass
/// that needs objects discards its work, so a round is idempotent and an
/// interrupted fill resumes from current presence.
///
/// An id asked for again after it was filled means the store reports it
/// present while the traversal cannot read it (a stale index entry): that
/// would loop forever, so it fails instead.
pub(crate) fn tree_merge_filled(
&self,
base: Option<(ObjectId, &Protection)>,
ours: (ObjectId, &Protection),
theirs: (ObjectId, &Protection),
identity: Option<&scl_crypto::SecretKey>,
) -> Result<(TreeMergeDone, FillStats)> {
let mut filler = self.needs_filler()?;
fill_loop(
|| {
let store_arc = self.vfs().store();
let mut store = store_arc.lock().unwrap();
merge_trees(&mut store, base, ours, theirs, identity)
},
|needs| filler.fill(needs),
)
}
}

/// The pass-then-fill loop of [`Repo::tree_merge_filled`], with the pass and
/// the fill as parameters so the no-progress guard can be tested without a
/// store whose index disagrees with itself.
fn fill_loop(
mut pass: impl FnMut() -> Result<TreeMerge>,
mut fill: impl FnMut(&[(ObjectId, String)]) -> Result<FillStats>,
) -> Result<(TreeMergeDone, FillStats)> {
let mut stats = FillStats::default();
let mut asked: BTreeSet<ObjectId> = BTreeSet::new();
loop {
let needs = match pass()? {
TreeMerge::Done(done) => return Ok((done, stats)),
TreeMerge::Needs(needs) => needs,
};
if let Some((id, path)) = needs.iter().find(|(id, _)| asked.contains(id)) {
return Err(Error::Protocol(format!(
"merge asked again for object {id} (at {path:?}) after it was fetched; \
the object store and its index disagree"
)));
}
asked.extend(needs.iter().map(|(id, _)| *id));
let round = fill(&needs)?;
stats.requests += round.requests;
stats.objects += round.objects;
}
}

/// Visit one name at `path` with its (up to three) entries.
fn visit(store: &mut Store, w: &mut Walk, path: &str, ents: [Option<TreeEntry>; 3]) -> Result<()> {
let present: Vec<&TreeEntry> = ents.iter().flatten().collect();
Expand Down Expand Up @@ -1082,4 +1139,69 @@ mod tests {
let err = merge_trees(&mut s, None, (ours, &p), (theirs, &p), None).unwrap_err();
assert!(matches!(err, Error::PathCollision(_)), "got {err:?}");
}

fn empty_done() -> TreeMerge {
TreeMerge::Done(TreeMergeDone {
merge: FileMerge {
files: vec![],
sidecars: vec![],
conflicts: vec![],
wrapped_carry: Default::default(),
},
carried: Carried::Entries(vec![]),
})
}

#[test]
fn fill_loop_fills_then_finishes_and_sums_the_cost() {
let a = scl_core::Object::blob(b"a".to_vec()).id();
let b = scl_core::Object::blob(b"b".to_vec()).id();
let mut passes = vec![
TreeMerge::Needs(vec![(a, "x".into())]),
TreeMerge::Needs(vec![(b, "x/y".into())]),
empty_done(),
]
.into_iter();
let mut filled: Vec<ObjectId> = Vec::new();
let (_, stats) = fill_loop(
|| Ok(passes.next().unwrap()),
|needs| {
filled.extend(needs.iter().map(|(id, _)| *id));
Ok(FillStats {
requests: 1,
objects: needs.len(),
})
},
)
.unwrap();
assert_eq!(filled, vec![a, b]);
assert_eq!(
stats,
FillStats {
requests: 2,
objects: 2
}
);
}

#[test]
fn fill_loop_fails_when_a_filled_id_is_asked_for_again() {
// The store claims to hold the object (the fill succeeds) while the
// traversal still cannot read it: without the guard this spins forever.
let a = scl_core::Object::blob(b"a".to_vec()).id();
let mut rounds = 0;
let err = fill_loop(
|| Ok(TreeMerge::Needs(vec![(a, "x".into())])),
|_| {
rounds += 1;
Ok(FillStats::default())
},
)
.unwrap_err();
assert_eq!(rounds, 1, "one fill, then the repeat is caught");
assert!(
matches!(&err, Error::Protocol(m) if m.contains("asked again")),
"got {err:?}"
);
}
}
Loading
Loading