Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ the project tree if desired.

## Capability map (what's built, by phase)

All 36 phases and P37a are built and tested. One line of current fact per phase; the
All 36 phases, P37a and P37b-i are built and tested. One line of current fact per phase; the
authoritative rationale and full semantics live in the linked ADR, the design
in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is
archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not
Expand Down Expand Up @@ -152,6 +152,7 @@ the code, those win.
| P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) |
| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) |
| P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat <ref>:<path>` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |
| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. Sparse widen (37b-ii) and merge/replay are next | [0048](docs/adr/0048-presence-based-partial-clones.md) |

## Standing boundaries & gotchas

Expand All @@ -168,8 +169,14 @@ transport-adjacent. The rest, imperatively:
capability, and no DTO field capable of carrying protected ciphertext or
identity material.
- **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`,
`sc work`, `sc export`, and `sparse disable`.** `sc backfill --all` converts
to a genuine full clone and re-enables them.
`sc work`, `sc export`, `sparse disable`, an empty `sparse set`, `sc rewrap`,
`sc grant`/`revoke`, leaving a private branch, `sc push` to a git remote, and
`sc fetch` from any native remote but the promisor origin.** `sc backfill --all`
converts to a genuine full clone and re-enables them. The filter records what
a clone *asked for*, not what is present: after a fetch an in-filter object
can be absent, and a switch, fast-forward, undo or `sparse set` that needs it
fails with `GapInView` (nothing changed) and names the `sc backfill <prefix>`
that repairs it.
- **Bucket remotes hold public content plaintext at rest** — bucket ACL is
the perimeter (sealed content stays ciphertext, unchanged); partial-clone
`filter` against bucket remotes is refused.
Expand Down
12 changes: 6 additions & 6 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,12 +548,12 @@ across every phase.

## Active

**P37b — presence-based walks and the lifted P27 refusals (proposed).**
P37a (built) added the batched `GetObjects` verb and `sc cat`; the remaining
work needs push, verify and gc to ask "is this object present?" instead of "is
this path in the filter?" (37b-i, which also fixes gc and fast-forward failing
after an ordinary fetch on a partial clone), then sparse widen over a filled
prefix (37b-ii). Merge, rebase and sessions on partial clones are directions
**P37b — presence-based walks and the lifted P27 refusals (37b-i built).**
P37a (built) added the batched `GetObjects` verb and `sc cat`. 37b-i (built)
made push, verify and gc ask "is this object present?" instead of "is this
path in the filter?", which also fixed gc and fast-forward failing after an
ordinary fetch on a partial clone; the remaining work is sparse widen over a
filled prefix (37b-ii). Merge, rebase and sessions on partial clones are directions
with constraints, each needing its own design record; `export` and `sparse
disable` stay refused. Design and open questions in
[ADR-0048](docs/adr/0048-presence-based-partial-clones.md).
Expand Down
10 changes: 10 additions & 0 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3868,6 +3868,16 @@ fn run_push(remote: &str, include_encrypted: bool) -> Result<()> {
`sc branch publish {branch}` first"
);
}
// A partial clone cannot export: the git bridge walks the whole
// history and every blob, and the first absent object would fail
// it partway through. Refuse up front, like `sc export` does.
if repo.promisor()?.is_some() {
anyhow::bail!(
"refusing to push to a git remote: this is a partial clone \
(.sc/promisor present) and the git bridge needs the whole history \
and every blob; run `sc backfill --all` to convert to a full clone first"
);
}
run_push_git(&repo, remote, include_encrypted)
}
_ => {
Expand Down
118 changes: 118 additions & 0 deletions crates/cli/tests/partial_clone_safety.rs
Original file line number Diff line number Diff line change
Expand Up @@ -249,3 +249,121 @@ fn sparse_widen_beyond_partial_filter_is_refused() {
std::fs::remove_dir_all(&src_root).unwrap();
std::fs::remove_dir_all(&dst_root).unwrap();
}

/// ADR-0048: an ordinary `sc fetch` can leave an in-filter object absent
/// (the origin subtracts the closure of every have, and the new file equals
/// one that is only reachable at an out-of-filter path). `sc gc` must keep
/// working, `sc verify` must count the gap, and a fast-forward must fail
/// before it writes anything, with a repair hint that works.
#[test]
fn fetch_then_gc_and_fast_forward_on_a_partial_clone() {
let src_root = tmp("fetch-gc-src");
let dst_root = tmp("fetch-gc-dst");
std::fs::create_dir_all(src_root.join("src")).unwrap();
std::fs::create_dir_all(src_root.join("docs")).unwrap();
assert!(sc(&src_root, &["init"]).status.success());
std::fs::write(src_root.join("src/a.txt"), "src-one\n").unwrap();
std::fs::write(
src_root.join("docs/readme.txt"),
"same bytes at two paths\n",
)
.unwrap();
assert!(sc(&src_root, &["commit", "-m", "c1"]).status.success());
let out = sc(
&src_root,
&[
"clone",
src_root.to_str().unwrap(),
dst_root.to_str().unwrap(),
"--filter",
"src/",
],
);
assert!(out.status.success(), "clone: {}", stderr(&out));

// The origin adds an in-filter file whose bytes equal docs/readme.txt, and
// another in-filter file that sorts before it.
std::fs::write(src_root.join("src/aaa.txt"), "fetched normally\n").unwrap();
std::fs::write(src_root.join("src/readme.txt"), "same bytes at two paths\n").unwrap();
assert!(sc(&src_root, &["commit", "-m", "c2"]).status.success());

let fetch = sc(&dst_root, &["fetch", "origin"]);
assert!(fetch.status.success(), "fetch: {}", stderr(&fetch));

let gc = sc(&dst_root, &["gc"]);
assert!(
gc.status.success(),
"gc must work after an ordinary fetch: {}",
stderr(&gc)
);
let verify = sc(&dst_root, &["verify"]);
assert!(verify.status.success(), "verify: {}", stderr(&verify));

let merge = sc(&dst_root, &["merge", "origin/main"]);
assert!(!merge.status.success(), "the fast-forward must be refused");
let err = stderr(&merge);
assert!(
err.contains("sc backfill src/"),
"error names the repair: {err}"
);
assert!(
!dst_root.join("src/aaa.txt").exists() && !dst_root.join("src/readme.txt").exists(),
"nothing was written before the refusal"
);

let backfill = sc(&dst_root, &["backfill", "src/"]);
assert!(backfill.status.success(), "backfill: {}", stderr(&backfill));
let merge = sc(&dst_root, &["merge", "origin/main"]);
assert!(merge.status.success(), "merge: {}", stderr(&merge));
assert_eq!(
std::fs::read_to_string(dst_root.join("src/readme.txt")).unwrap(),
"same bytes at two paths\n"
);

std::fs::remove_dir_all(&src_root).unwrap();
std::fs::remove_dir_all(&dst_root).unwrap();
}

/// The git bridge walks the whole history and every blob, so a partial clone
/// refuses `sc push` to a git remote up front, like `sc export`.
#[test]
fn push_to_a_git_remote_is_refused_on_a_partial_clone() {
let (src_root, dst_root) = make_partial_clone("push-git");
let bare = tmp("push-git-bare");
std::fs::create_dir_all(&bare).unwrap();
let init = Command::new("git")
.args(["init", "-q", "--bare", "-b", "main"])
.arg(&bare)
.output()
.expect("git runs");
assert!(init.status.success());
assert!(sc(
&dst_root,
&["remote", "add", "hub", bare.to_str().unwrap(), "--git"]
)
.status
.success());

let out = sc(&dst_root, &["push", "hub"]);
assert!(!out.status.success());
let err = stderr(&out);
assert!(err.contains("partial clone"), "names the cause: {err}");
assert!(err.contains("backfill --all"), "names the repair: {err}");

std::fs::remove_dir_all(&src_root).unwrap();
std::fs::remove_dir_all(&dst_root).unwrap();
std::fs::remove_dir_all(&bare).unwrap();
}

/// `sc sparse set` with no prefixes is a full checkout, which is what `sparse
/// disable` is, so it is refused too (otherwise it would bypass that refusal).
#[test]
fn sparse_set_with_no_prefixes_is_refused_on_a_partial_clone() {
let (src_root, dst_root) = make_partial_clone("sparse-empty");
let out = sc(&dst_root, &["sparse", "set"]);
assert!(!out.status.success(), "an empty spec must be refused");
assert!(stderr(&out).contains("backfill"), "{}", stderr(&out));
assert!(!dst_root.join("docs/b.txt").exists());
std::fs::remove_dir_all(&src_root).unwrap();
std::fs::remove_dir_all(&dst_root).unwrap();
}
18 changes: 18 additions & 0 deletions crates/repo/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,24 @@ pub enum Error {
GappedPathContent(String),
#[error("{0} lies outside this partial clone's fetch filter; run `sc backfill {0}` to fetch it first")]
GapOutsideFilter(String),
/// An object the checked-out view needs is absent from a partial clone's
/// store even though the path is inside its fetch filter (ADR-0048): an
/// ordinary `fetch` subtracts the closure of every have, so a new object
/// equal to one reachable from a have at an out-of-filter path is left
/// out of the reply. Raised before anything is written, so the working
/// tree and refs are untouched. `prefix` is the filter prefix that
/// `sc backfill` can act on.
#[error("{path} is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill {prefix}` to repair it")]
GapInView { path: String, prefix: String },
/// A push found an object the local store lacks (referenced by `referrer`)
/// that the destination does not hold either (ADR-0048). Pushing the new
/// tip would leave the destination with a tree that names an object it
/// cannot serve, so the push is refused before any byte is sent.
#[error("refusing to push: object {id} (referenced by {referrer}) is missing locally and the remote does not have it; if it came from the origin run `sc backfill --all`, otherwise restore or recreate the content and commit again, then retry")]
PushGap {
id: scl_core::ObjectId,
referrer: scl_core::ObjectId,
},
/// A dedicated, non-path-shaped refusal for an operation that isn't
/// supported on a partial clone at all (merge, cherry-pick/rebase
/// replay, `sc ws harvest`, `sc work`) — distinct from
Expand Down
95 changes: 37 additions & 58 deletions crates/repo/src/gc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
use std::collections::BTreeSet;
use std::time::{Duration, SystemTime, UNIX_EPOCH};

use scl_core::{Object, ObjectId, Store};
use scl_core::{ObjectId, Store};

use crate::error::Result;
use crate::layout::Layout;
Expand Down Expand Up @@ -80,6 +80,23 @@ fn roots(layout: &Layout) -> Result<Vec<ObjectId>> {
Ok(set.into_iter().collect())
}

/// Protect an in-progress operation's decided carried tree. On a partial
/// clone the walk tolerates an absent child, like the main reachability walk
/// does, so a stopped operation cannot brick gc; a full clone stays strict.
fn walk_decided_root(
store: &mut Store,
tree: ObjectId,
partial: bool,
reachable: &mut BTreeSet<ObjectId>,
) -> Result<()> {
if partial {
let mut gaps = std::collections::BTreeMap::new();
reachable::walk_tree_partial(store, tree, tree, reachable, &mut gaps)
} else {
reachable::walk_tree(store, tree, reachable)
}
}

/// Run a gc pass. Caller must already hold the repo lock.
pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result<GcStats> {
let mut stats = GcStats::default();
Expand All @@ -94,60 +111,22 @@ pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result<GcStat
oplog::trim_older_than(layout, cutoff)?;

let roots = roots(layout)?;
// Gap-tolerant reachability on a partial clone (P27 Task 5): when
// `.sc/promisor` is present, walk with the filtered reachability walk and
// keep only `.included` — an out-of-filter child is recorded in `.gaps`
// by id and never `get()`'d, so a genuinely-missing out-of-filter object
// (the whole premise of a partial clone) can never surface as a `gc`
// error, and a gap is never treated as unreachable garbage either (it's
// simply not in `reachable` at all, so the loose-object sweep below never
// considers it — it was never fetched, so it's never `store.list_loose`d
// in the first place). A full clone (`promisor::load` returns `None`)
// takes the unfiltered path, unchanged.
let filter = promisor::load(layout)?;
let mut reachable: BTreeSet<ObjectId> = match &filter {
Some(p) => {
let r = reachable::reachable_objects_filtered(store, &roots, Some(p))?;
let mut included = r.included;
// P27 Task 5 review Critical fix (gc defense-in-depth): a gap id
// means "referenced by an in-filter parent tree but never
// fetched" — the filtered walk deliberately never `get()`s it,
// so it's normally absent locally. But gc must not treat "gap"
// as license to prune an object that IS present locally for any
// reason (belt-and-suspenders alongside the Task 5 commit-side
// refusal above, which closes the one known way such an object
// could be created; this is the backstop for any path that
// doesn't yet route through that guard). Walk every PRESENT gap
// id in, so it — and anything it in turn references — becomes
// part of `reachable` and survives the sweep below: gc never
// prunes reachable content it holds. This walk-in is itself
// ABSENCE-TOLERANT (P27 final review I1, `walk_tree_present_only`),
// not the strict `walk_tree`: `ingest_pack_file`'s write pass is
// not all-or-nothing, so a crash-interrupted `sc backfill` can
// leave a present gap-frontier tree with an absent child — the
// strict walk would `get()` that child and hard-error, which
// `Store::write_pack` below would then repeat on the id anyway
// if it were smuggled into `reachable`. Present content BELOW an
// absent child is not reached and is pruned by the ordinary
// loose-object sweep if it's otherwise disconnected — gc never
// prunes anything connected to the local graph, but it is not
// structurally incapable of pruning locally-disconnected residue.
for gap in &r.gaps {
if included.contains(gap) || !store.contains(gap) {
continue;
}
match store.get(gap)? {
Object::Tree(_) => {
reachable::walk_tree_present_only(store, *gap, &mut included)?;
}
_ => {
included.insert(*gap);
}
}
}
included
}
None => reachable::reachable_objects(store, &roots)?,
// Presence-based reachability on a partial clone (ADR-0048): when
// `.sc/promisor` is present, walk what the store actually holds. An absent
// child of a present tree is a gap, recorded and never read, so a
// genuinely missing object (the premise of a partial clone) cannot
// surface as a `gc` error, and a gap is not garbage either: it is not in
// `reachable`, so the loose-object sweep below never considers it. The
// promisor filter is deliberately not consulted. It records what a clone
// asked for, not what is present, and an ordinary `fetch` can leave an
// in-filter object absent. Every present tree is descended, so content a
// lazy fill or a backfill brought in survives. A full clone
// (`promisor::load` returns `None`) takes the strict walk, unchanged.
let partial = promisor::load(layout)?.is_some();
let mut reachable: BTreeSet<ObjectId> = if partial {
reachable::reachable_objects_partial(store, &roots)?.present
} else {
reachable::reachable_objects(store, &roots)?
};
// An in-progress merge's or cherry-pick's decided carried tree
// (MERGE_DECIDED_ROOT / PICK_DECIDED_ROOT) is a TREE root, not a
Expand All @@ -161,16 +140,16 @@ pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result<GcStat
// tree forever.
if merge_state::read_merge_head(layout)?.is_some() {
if let Some(tree) = merge_state::read_decided_root(layout)? {
reachable::walk_tree(store, tree, &mut reachable)?;
walk_decided_root(store, tree, partial, &mut reachable)?;
}
}
if pick_state::read_pick_head(layout)?.is_some() {
if let Some(tree) = pick_state::read_decided_root(layout)? {
reachable::walk_tree(store, tree, &mut reachable)?;
walk_decided_root(store, tree, partial, &mut reachable)?;
}
}
if let Some(tree) = rebase_state::read_decided_root(layout)? {
reachable::walk_tree(store, tree, &mut reachable)?;
walk_decided_root(store, tree, partial, &mut reachable)?;
}

// Transcript index (P30): a Transcript is likewise reachable from no
Expand Down
2 changes: 2 additions & 0 deletions crates/repo/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ pub mod lock;
pub mod merge;
pub mod merge_state;
pub mod oplog;
#[cfg(test)]
mod partial_walks;
pub mod pick_state;
pub mod private;
pub mod promisor;
Expand Down
Loading
Loading