Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .claude/skills/sc-cli-reference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,23 @@ cargo run --bin sc -- backfill --all # fetch EVERY remaining object (no
# genuine full clone and merge/pick/rebase/ws
# fork/harvest/sc work/export/sparse disable all
# work again (P27 final review I2)
cargo run --bin sc -- clone --filter <prefix…> --lazy <src> <dst>
# lazy partial clone (P37): also writes
# .sc/promisor-lazy so `sc cat` may fetch missing
# trees/blobs by object id; requires --filter
cargo run --bin sc -- backfill --lazy [prefix…]
# mark an existing partial clone lazy (P37); with
# prefixes, also backfills them; conflicts --all
cargo run --bin sc -- cat <ref>:<path> # print one file as of a ref (P37). On a lazy
# partial clone fetches only the absent trees and
# the blob on that path (one GetObjects round trip
# per level). SC_OFFLINE=1 forces the gap error.
# Refuses protected paths and private branches
bash demo/run_lazy_fetch_demo.sh # by-id fetch proof (P37a): cat of one path adds
# exactly 3 objects (2 trees + blob), the sibling
# stays unfetched, offline/non-lazy keep the gap
# error, and a read-only sc+http server answers
# GetObjects (protocol v5)
bash demo/run_partial_clone_demo.sh # partial clone proof (P27): sc clone --filter
# src/ fetches fewer objects than a full clone
# (object-store count + sc verify's gap report);
Expand Down
15 changes: 15 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1060,3 +1060,18 @@ at accepted `BranchManifest` metadata and becomes `private_opaque`; authors,
messages, paths, timestamps, and inner topology never cross IPC. The WebView has
no generic dialog, filesystem, URL, process, or object-store command. See
ADR-0045.

## Phase 37a — want-by-id fetch (built)

A partial clone marked lazy (`.sc/promisor-lazy`) can read a path outside its
fetch filter without `sc backfill`. `sc cat <ref>:<path>` walks from the ref's
root tree; each absent tree, then the blob, is fetched with one `GetObjects`
round trip (`PROTOCOL_VERSION` 5): the client names ids, the server answers
with the objects it can serve plus the ids it will not send, with no haves and
no reachability walk. `Store::get` stays network-free; the only code that dials
the promisor after clone is `Repo::fetch_by_id_from` in `crates/repo/src/
fill.rs`, which verifies the reply against the request before writing. A fill
only adds objects, so push, gc and verify (filter-scoped walks) are unchanged.
Sparse widening over a filled path and the lifted P27 refusals need
presence-based walks and are phase 37b. See ADR-0047 and
`docs/research/scion-evaluation.md`.
7 changes: 4 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ the project tree if desired.

## Capability map (what's built, by phase)

All 36 phases are built and tested. One line of current fact per phase; the
All 36 phases and P37a are built and tested. One line of current fact per phase; the
authoritative rationale and full semantics live in the linked ADR, the design
in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is
archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not
Expand Down Expand Up @@ -151,15 +151,16 @@ the code, those win.
| P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) |
| P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) |
| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) |
| P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat <ref>:<path>` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |

## Standing boundaries & gotchas

Cross-cutting current facts that bite. Security boundaries are consolidated in
`docs/THREAT-MODEL.md` — read it before touching anything crypto- or
transport-adjacent. The rest, imperatively:

- **Wire protocol is version 4** (bumped from 3 in P34 for the two additive
sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the
- **Wire protocol is version 5** (bumped from 4 in P37 for the additive
`GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the
transfer path only — a larger blob commits locally but fails every
subsequent push/fetch/clone at the receiver.
- **The P35 desktop app is a keyless, read-only top-level adapter.** Its
Expand Down
21 changes: 11 additions & 10 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,12 +548,12 @@ across every phase.

## Active

**P37 — Want-by-id fetch and gap fill (proposed, not started).** Batched
`GetObjects` verb + an explicit `fill_gaps` seam so partial clones can run
merge/rebase/ws/work/export without `Store::get` ever touching the network;
design in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md), prior art
in `docs/research/scion-evaluation.md`. Phases: 37a (verb + read-only entry
points), 37b (lift the P27 refusals).
**P37b — presence-based walks and the lifted P27 refusals (proposed).**
P37a (built) added the batched `GetObjects` verb and `sc cat`; the remaining
work needs push, verify and gc to ask "is this object present?" instead of "is
this path in the filter?", then sparse widen over a filled prefix, then merge,
rebase, `ws`, `work` and `export` on partial clones. Design and open questions
in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md).

## Completed phases (usability-first ordering)

Expand Down Expand Up @@ -591,6 +591,7 @@ points), 37b (lift the P27 refusals).
| **P33 — Randomized protected encryption** | Close equality confirmation for new protected content | fresh DEK + nonce per seal, dual-read/randomized-write, keyed local stat cache, rewrap upgrade | [0043](docs/adr/0043-randomized-protected-encryption.md) |
| **P34 — Private branches** | Keep an entire branch opaque before publication | sealed object closure behind a manifest; O(1) grant, KEK-rotating revoke, scanner-gated publish | [0044](docs/adr/0044-per-branch-access-control.md) |
| **P35 — Native desktop browser** | Prove native `.sc` objects can power a polished visual tool | Tauri app opens a repository and browses refs, snapshot DAG/provenance, public trees, and first-parent diffs while locked/private states remain safe | [0045](docs/adr/0045-native-desktop-read-model.md) |
| **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |

> **Prior art.** Phases P5–P9 adapt decisions from the sibling project
> [git.agentic](https://github.com/git-agentic/git.agentic) (same BLAKE3
Expand Down Expand Up @@ -743,9 +744,9 @@ scale-&-reach horizon):
origin automatically on any gap, matching git's promisor default. Real
scope/risk (offline breakage, surprise latency, fetch-during-gc), so it
stays deferred, not accidentally missing.
**Now scoped as P37 (ADR-0047, Proposed):** an opt-in explicit `fill_gaps`
seam over a batched by-id `GetObjects` verb, keeping `Store::get`
network-free.
**Now scoped as P37 (ADR-0047):** an opt-in `.sc/promisor-lazy` marker over a
batched by-id `GetObjects` verb, keeping `Store::get` network-free. P37a
(`sc cat`) is built; the rest is P37b.
- **Per-case gap-tolerant merge/rebase/ws-harvest/sc-work on a partial
clone (P27).** These are refused entirely today
(`Error::PartialCloneUnsupported`) rather than gap-tolerantly
Expand All @@ -754,7 +755,7 @@ scale-&-reach horizon):
individually is the follow-on if the "backfill to a full clone first"
workaround proves too coarse in practice.
**Now scoped as P37b (ADR-0047, Proposed):** lifted one operation at a
time on top of the P37a `fill_gaps` seam.
time once push, verify and gc are presence-based.
- **Blob-size/object-count clone filters (P27).** `sc clone --filter` is
prefix-only, matching the sparse path-boundary machinery already built;
git's `--filter=blob:limit` size-based filtering is a separate axis,
Expand Down
96 changes: 91 additions & 5 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,12 @@ enum Cmd {
/// over a git-bridge remote.
#[arg(long)]
filter: Vec<String>,
/// Mark the partial clone lazy (P37): `sc cat` may fetch the trees
/// and blobs it needs from the promisor remote by object id, instead
/// of failing with a gap error. `SC_OFFLINE=1` turns it off per
/// process. Requires `--filter`.
#[arg(long, requires = "filter")]
lazy: bool,
},
/// Serve a repo to a remote `sc` client, either over stdin/stdout
/// (invoked by `ssh` for ssh:// remotes) or over TCP (`sc+http://`
Expand Down Expand Up @@ -499,8 +505,22 @@ enum Cmd {
/// and merge/pick/rebase/`ws harvest`/`sc work`/export/sparse
/// disable all work again. Mutually exclusive with `prefixes`
/// (P27 final review I2).
#[arg(long)]
#[arg(long, conflicts_with = "lazy")]
all: bool,
/// Mark this partial clone lazy (P37): `sc cat` may fetch missing
/// trees and blobs by object id (`SC_OFFLINE=1` disables it per
/// process). May be given with no prefixes.
#[arg(long)]
lazy: bool,
},
/// Print one file as of a ref: `sc cat <ref>:<path>` (P37). On a lazy
/// partial clone, missing trees and the blob are fetched by object id.
/// Public content only: protected paths and private branches are refused.
/// `<ref>` is a branch or remote-tracking ref (`main`, `origin/main`);
/// `HEAD` is not accepted.
Cat {
/// `<ref>:<path>`, e.g. `main:docs/api/guide.md`.
target: String,
},
/// Session transcript operations (P30): attach a sealed agent-session
/// transcript to a commit, list/inspect them, and sign them.
Expand Down Expand Up @@ -942,7 +962,9 @@ fn main() -> Result<()> {
dst,
git,
filter,
} => run_clone(src, dst, git, filter),
lazy,
} => run_clone(src, dst, git, filter, lazy),
Cmd::Cat { target } => run_cat(&target),
Cmd::Serve {
sub,
stdio,
Expand Down Expand Up @@ -1022,7 +1044,11 @@ fn main() -> Result<()> {
Cmd::Sign { r#ref, identity } => run_sign(&r#ref, identity),
Cmd::Verify { r#ref, require } => run_verify(r#ref, require),
Cmd::Sparse { op } => run_sparse(op),
Cmd::Backfill { prefixes, all } => run_backfill(prefixes, all),
Cmd::Backfill {
prefixes,
all,
lazy,
} => run_backfill(prefixes, all, lazy),
Cmd::Transcript { op } => run_transcript(op),
}
}
Expand Down Expand Up @@ -3395,7 +3421,20 @@ fn resolve_ids_to_pubkeys(
Ok(out)
}

fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec<String>) -> Result<()> {
/// A relative local clone source is recorded as the remote and promisor
/// origin; resolved from inside the clone it would name the wrong directory,
/// so persist the absolute path. URLs (`ssh://`, `sc+http://`, ...) and paths
/// that do not resolve are returned unchanged.
fn absolute_local_source(src: String) -> String {
if src.contains("://") {
return src;
}
std::fs::canonicalize(&src)
.map(|p| p.display().to_string())
.unwrap_or(src)
}

fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec<String>, lazy: bool) -> Result<()> {
// Auto-detect unambiguous git URL forms (https/http, scp-style, file://):
// those can never be sc-native, so no flag is needed. Bare `ssh://` is
// ambiguous — it means an sc-native remote (ADR-0022) unless `--git`
Expand All @@ -3407,8 +3446,12 @@ fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec<String>) -> Resul
}
return run_clone_git(&src, &dst);
}
let src = absolute_local_source(src);
let filter_opt = (!filter.is_empty()).then_some(filter.as_slice());
let repo = scl_repo::Repo::clone_url_filtered(&src, &dst, filter_opt)?;
if lazy {
scl_repo::promisor::set_lazy(repo.layout(), true)?;
Comment on lines +3452 to +3453

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve an absolute origin for relative lazy clones

When --lazy is used with a relative local source such as sc clone --filter src/ --lazy source clone, the promisor records source verbatim. The user must then enter clone to run sc cat, where Filler::ensure_present resolves that path relative to the clone and dials clone/source rather than the original repository. Consequently, on-demand reads outside the filter fail for ordinary relative-path clones; canonicalize the local promisor origin before persisting/marking the clone lazy.

Useful? React with 👍 / 👎.

}
let n = repo.branches()?.len();
if filter_opt.is_some() {
println!(
Expand All @@ -3428,8 +3471,20 @@ fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec<String>) -> Resul
/// origin and widen `.sc/promisor`. Errors if this repo is not a partial
/// clone. `--all` ignores `prefixes` and instead fetches every remaining
/// object, then removes `.sc/promisor` entirely (P27 final review I2).
fn run_backfill(prefixes: Vec<String>, all: bool) -> Result<()> {
fn run_backfill(prefixes: Vec<String>, all: bool, lazy: bool) -> Result<()> {
let repo = open_repo()?;
if lazy {
if repo.promisor()?.is_none() {
anyhow::bail!("not a partial clone (.sc/promisor absent); --lazy has nothing to mark");
}
if !prefixes.is_empty() {
repo.backfill(&prefixes)?;
println!("backfilled {} prefix(es)", prefixes.len());
}
scl_repo::promisor::set_lazy(repo.layout(), true)?;
println!("marked lazy: `sc cat` may fetch missing objects by id");
return Ok(());
}
if all {
if !prefixes.is_empty() {
anyhow::bail!("--all fetches every remaining object; pass either --all or explicit prefixes, not both");
Expand All @@ -3445,6 +3500,20 @@ fn run_backfill(prefixes: Vec<String>, all: bool) -> Result<()> {
Ok(())
}

/// `sc cat <ref>:<path>`: write one file's bytes to stdout. Branch names are
/// flat (no `:`), so the first `:` splits ref from path.
fn run_cat(target: &str) -> Result<()> {
let (refname, path) = target
.split_once(':')
.filter(|(r, p)| !r.is_empty() && !p.is_empty())
.ok_or_else(|| anyhow::anyhow!("expected <ref>:<path>, got '{target}'"))?;
let repo = open_repo()?;
let bytes = repo.cat_at(refname, path)?;
use std::io::Write;
std::io::stdout().write_all(&bytes)?;
Ok(())
}

/// Clone from a hosted Git URL: init + remote add origin --git + fetch +
/// adopt the remote's default branch (P10's unborn fast-forward adoption).
/// Reached by auto-detect for unambiguous git URL forms, or by `--git` for
Expand Down Expand Up @@ -4586,6 +4655,7 @@ mod tests {
dst.clone(),
false,
Vec::new(),
false,
)
.unwrap();
let repo = scl_repo::Repo::open(&dst).unwrap();
Expand All @@ -4602,6 +4672,21 @@ mod tests {
std::fs::remove_dir_all(&root).unwrap();
}

#[test]
fn clone_source_is_recorded_as_an_absolute_path_but_urls_are_untouched() {
// A relative source would resolve against the clone's own directory
// once the user cd's into it, so the recorded origin must be absolute.
let abs = absolute_local_source(".".to_string());
assert!(std::path::Path::new(&abs).is_absolute(), "got {abs}");
for url in ["ssh://host/srv/repo", "sc+http://127.0.0.1:1/x"] {
assert_eq!(absolute_local_source(url.to_string()), url);
}
assert_eq!(
absolute_local_source("no/such/dir/anywhere".to_string()),
"no/such/dir/anywhere"
);
}

#[test]
fn clone_bare_ssh_url_without_git_flag_stays_sc_native() {
// Env mutation (SC_SSH) — serialize with the shared git-env lock.
Expand All @@ -4620,6 +4705,7 @@ mod tests {
root.join("cloned-ssh"),
false,
Vec::new(),
false,
)
.unwrap_err();
std::env::remove_var("SC_SSH");
Expand Down
Loading
Loading