Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .claude/skills/sc-cli-reference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,18 @@ cargo run --bin sc -- clone --filter <prefix…> <src> <dst> # partial clone (
# prefixes; writes .sc/promisor + .sc/sparse to
# the same prefixes; not supported over a
# git-bridge remote
cargo run --bin sc -- clone --filter blob:limit=<size> [--filter <prefix…>] [--lazy] <src> <dst>
# blob-size partial clone (ADR-0055, #217):
# <size> is bytes or a KiB/MiB/GiB suffix; blobs of
# AT LEAST that size that no ref tip reaches are
# left out (old revisions of large files), every tip
# and the checkout stay complete; blob:limit=0 = no
# blobs beyond the tips. `blob:` is reserved in
# --filter (blob:none is an error), once only; a
# limit above 1 GiB is an error.
# Server-evaluated: wire v6, upgrade both sides.
# Repair: `sc backfill <file>`, a lazy fill, or
# `backfill --all`; fetch is not limit-aware
cargo run --bin sc -- backfill <prefix…> # widen a partial clone: fetch objects under
# these prefixes from the promisor origin and
# widen .sc/promisor; errors if this repo isn't
Expand All @@ -408,7 +420,10 @@ bash demo/run_lazy_fetch_demo.sh # by-id fetch proof (P37a): cat of
# exactly 3 objects (2 trees + blob), the sibling
# stays unfetched, offline/non-lazy keep the gap
# error, and a read-only sc+http server answers
# GetObjects (protocol v5)
# GetObjects
bash demo/run_blob_limit_demo.sh # blob-size filter proof (ADR-0055): old large
# blob left out, tip checkout whole, verify/gc/
# push fine, --lazy, prefix+limit, backfill repair
bash demo/run_partial_clone_demo.sh # partial clone proof (P27): sc clone --filter
# src/ fetches fewer objects than a full clone
# (object-store count + sc verify's gap report);
Expand Down
21 changes: 20 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,8 @@ phases but the shape is stable:
`transcript-drops` — the derived set of dropped transcripts (`sc transcript
drop`), rebuilt from the store on clone.
- `sparse` — the local, uncommitted sparse-checkout prefix spec (P24);
`promisor` — a partial clone's fetch-filter + origin (P27).
`promisor` — a partial clone's fetch-filter + origin (P27);
`promisor-blob-limit` — its blob-size limit and whole-tree-or-prefix scope (ADR-0055).
- `serve-tokens.toml` — bearer-token access control for `sc serve --http` (P29);
tokens may carry an expiry and be rotated in place (`sc serve token add
--expires` / `rotate`). `serve-tls/` — the auto-minted `sc+https://` server
Expand Down Expand Up @@ -1234,6 +1235,24 @@ by id (`ensure_view`) before switch, fast-forward, undo and `sparse set`;
object cap; the client reads at most 4 GiB) and split when refused. `sc export` and `sparse
disable` stay refused on a partial clone. See ADR-0048.

## Phase 27 follow-on — blob-size clone filter (built)

`sc clone --filter blob:limit=<size>` (alone or with prefixes) leaves out blobs
of at least `<size>` bytes that no ref tip's root tree reaches. Only the server
knows a blob's size before sending it (tree entries carry none), so
`GetPack` gained a trailing optional `blob_limit` and `PROTOCOL_VERSION` is 6.
The sender's walk, `reachable_objects_limited`, walks every want's own tree
first (every ref tip stays complete, so the initial checkout is whole), then
moves over-limit blobs reached only through history from `included` to `gaps`,
measuring each distinct blob once with `Store::blob_size`. The client side adds
nothing to the gap model: absence is the gap (37b). `.sc/promisor-blob-limit`
records the limit and an explicit `AllPaths`/`Prefixes` scope (a size-only clone
is `AllPaths`, with a full sparse view); `PackFilter` is the single definition of
a filtered transfer (a prefix or a limit). A lazy size-only clone fills a gapped
revision by id even though its view is full; otherwise the gap error names the
file and `sc backfill <path>` repairs it. `sc fetch` is unfiltered, and the
object-count limit is deferred. See ADR-0055.

## Phase 37c — merge, cherry-pick and rebase on a partial clone (built)

`merge_trees` is a tree-recursive three-way merge that carries a subtree **by
Expand Down
20 changes: 15 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ the code, those win.
| P24 | Sparse checkouts: `.sc/sparse` local prefix spec; checkout-only (CAS keeps everything) | [0034](docs/adr/0034-sparse-checkouts.md) |
| P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) |
| P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) |
| P27 | Partial clone: `sc clone --filter <prefix…>`, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) |
| P27 | Partial clone: `sc clone --filter <prefix…>`, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3. Blob-size filter (#217): `--filter blob:limit=<size>` (combinable with prefixes; `blob:` is reserved) leaves out blobs of at least that size unless a ref tip reaches them, evaluated by the server (`GetPack.blob_limit`, wire v6); the gap is absence as in ADR-0048, `sc backfill <path>` or a lazy fill repairs it, `.sc/promisor-blob-limit` records limit and scope | [0037](docs/adr/0037-partial-clone.md), [0055](docs/adr/0055-blob-size-clone-filter.md) |
| P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) |
| P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) |
| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; retained signed tombstones ride a transfer when their snapshot is in it or was collected by the sender, only verifying signatures are indexed, and held-but-untrusted tombstones are re-judged from `.sc/transcript-pending` when the trusted-signer set changes (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) |
Expand All @@ -163,8 +163,11 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in
`docs/THREAT-MODEL.md` — read it before touching anything crypto- or
transport-adjacent. The rest, imperatively:

- **Wire protocol is version 5** (bumped from 4 in P37 for the additive
`GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB;
- **Wire protocol is version 6** (bumped from 5 in #217 for `GetPack.blob_limit`,
ADR-0055; 4 to 5 in P37 for the additive `GetObjects` verb; 3 to 4 in P34 for
the sealed-branch object kinds). Servers and clients upgrade together: every
mixed pair fails closed at the handshake, ssh remotes included (they run the
remote host's own `sc` binary). The max object size (`scl_core::max_object_size()`, default 256 MiB;
`sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to
1 GiB, out of range aborts startup, ADR-0039 addendum) guards the
transfer path only — a larger blob commits locally but fails every
Expand All @@ -184,7 +187,7 @@ transport-adjacent. The rest, imperatively:
identity material. Opening a repository can write the derived transcript-drop
indexes (`.sc/transcript-drops`, `.sc/transcript-pending`) when the trusted
signer set changed; no key material is involved (#230).
- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`,
- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and `sparse disable`; both are allowed on a size-only `AllPaths` clone, ADR-0055), on a non-lazy clone a `sparse set` outside the filter, `sc rewrap`,
`sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge, cherry-pick or rebase whose conflict lies outside the view (a `.theirs` sidecar sits beside its conflicted path, so it is in view whenever that path is; ADR-0049) (no ref, state or working-tree change; on every clone, a conflict stop that needs sealing outside the view is refused the same way), `sc push` to a git remote, and
`sc fetch` from any native remote but the promisor origin.** `sc merge`, `sc cherry-pick` and `sc rebase` work, including conflicts inside the view (P37c). `sc ws` and `sc work` work too (ADR-0050): workspaces and agents see the clone's view only, and harvest lands through the same merge. `sc backfill --all`
converts to a genuine full clone and re-enables them. The filter records what
Expand All @@ -193,6 +196,13 @@ transport-adjacent. The rest, imperatively:
or `sparse set` fetches what its view needs by id first; on a non-lazy or
offline clone it fails with `GapInView` (nothing changed) and names the
`sc backfill <prefix>` that repairs it.
A `--filter blob:limit=<size>` clone (ADR-0055) is partial too: a blob of at
least that size is left out unless a ref tip reaches it, tips are complete at
clone time only, a later fetch can bring one that points at a gapped blob, and
with a limit the gap errors name the file's own path (`GapBySize`, or
`GapInView` with the limit as a cause). A size-only clone has an `AllPaths`
scope and a full sparse view, and a lazy one still fills on switch, undo, merge
and replay. `sc fetch` is not limit-aware, and the object-count limit is deferred.
- **Bucket remotes hold public content plaintext at rest** — bucket ACL is
the perimeter (sealed content stays ciphertext, unchanged); partial-clone
`filter` against bucket remotes is refused.
Expand Down Expand Up @@ -313,7 +323,7 @@ transport-adjacent. The rest, imperatively:
Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of
truth — do not re-list items here); entries that name an issue mirror the open
GitHub issues. Standing entries: `sc ws` workspace refresh (#215),
blob-size clone filters (#217), bucket compaction and gc (#216), the non-unix
a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), bucket compaction and gc (#216), the non-unix
`.sc/local-key` ACL (#214), and the open
sparse, private-branch and abort edge cases (#241, #246, #247, #248).

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,8 @@ via `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system`), bucket (`sc+wal://`, `sc+s3://`),
and Git (including hosted GitHub) network transports, plus private branches
sealed to a recipient set until publish and a read-only Tauri desktop
browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds
27 independent end-to-end proofs, one per major capability. Remaining
28 independent end-to-end proofs, one per major capability. Remaining
follow-ons (transcript drop delivery gaps, bucket compaction and gc, `sc ws`
workspace refresh, size/count clone filters, and similar hardening
workspace refresh, a limit-aware fetch and an object-count clone limit, and similar hardening
items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open
GitHub issues rather than being open MVP scope.
20 changes: 16 additions & 4 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,16 @@ across every phase.
carried even under a rule); conflicts inside the view are marked and
resolved as on a full clone, conflicts outside it are refused before any
write. (ADR-0049.)
- **Phase 27 follow-on — Blob-size clone filter (#217).** `sc clone --filter
blob:limit=<size>` (combinable with prefix filters; `blob:` is reserved) leaves
out blobs of at least that size that no ref tip's root tree reaches, so old
revisions of large files stay on the server while every tip and the checkout
are complete. The server evaluates it (`GetPack.blob_limit`, `PROTOCOL_VERSION`
6; client and server upgrade together). The gap is absence (ADR-0048), recorded
by `.sc/promisor-blob-limit` with an explicit `AllPaths`/`Prefixes` scope; a lazy
clone fills by id (a full view included), and otherwise the gap error names the
file and `sc backfill <path>` repairs it. Proven by `demo/run_blob_limit_demo.sh`.
(ADR-0055.)
- **Phase 37d — Agent sessions on a partial clone.** `sc ws` and `sc work` work
on a partial clone: a workspace sees the host's sparse view, and harvest lands
through the ADR-0049 merge. (ADR-0050.)
Expand Down Expand Up @@ -666,6 +676,7 @@ under Deferred below.
| **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |
| **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) |
| **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) |
| **P27 follow-on — Blob-size clone filter (#217)** | Clone a repo with large binary history without fetching old revisions | `sc clone --filter blob:limit=1MiB <src> <dst>` (with or without prefixes) leaves out old large blobs, the tip checkout is whole; verify/gc/push work; `--lazy` fills by id, `sc backfill <path>` repairs; wire v6; proven by `demo/run_blob_limit_demo.sh` | [0055](docs/adr/0055-blob-size-clone-filter.md) |
| **P37d — Sessions on partial clones** | Agent workspaces on a partial clone | `sc ws fork/run/harvest/abandon` and `sc work` see the clone's sparse view; harvest lands through the ADR-0049 merge, conflicts fall back to `work-<i>` | [0050](docs/adr/0050-sessions-on-partial-clones.md) |

> **Prior art.** Phases P5–P9 adapt decisions from the sibling project
Expand Down Expand Up @@ -790,10 +801,11 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in
everything else reports a gap with the `sc backfill` repair. Dialing the promisor origin
from any read path is still deferred (offline breakage, surprise latency,
fetch-during-gc).
- **Blob-size and object-count clone filters (#217, P27).** `sc clone --filter`
is prefix-only; git's `--filter=blob:limit` size axis needs a decision on the
repair path for a size gap, where the filter is evaluated, and protected
content first.
- **Remainder of #217 (P27): a limit-aware `sc fetch` and an object-count clone
limit.** The blob-size filter shipped (ADR-0055); `sc fetch` stays unfiltered,
so the limit bounds the initial transfer, not later growth, and a limit-aware
fetch needs per-path exemptions for backfilled files. An object-count limit has
no stable definition (the transfer set is id-ordered) and no git precedent.
- **`sc rewrap` on partial clones — refused (P37).** `Repo::rewrap` calls
`refuse_on_partial`; a paths-only/streaming rebuild that would lift the
refusal remains a possible follow-on.
Expand Down
23 changes: 13 additions & 10 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -310,8 +310,13 @@ enum Cmd {
git: bool,
/// Partial clone (P27): only fetch objects under this path prefix
/// (repeatable). Writes `.sc/promisor` + `.sc/sparse`; missing
/// objects can later be pulled in with `sc backfill`. Not supported
/// over a git-bridge remote.
/// objects can later be pulled in with `sc backfill`. Also accepts
/// `blob:limit=<size>` (bytes, or a KiB/MiB/GiB suffix; ADR-0055),
/// once: blobs of at least that size are left out unless a ref tip
/// reaches them, so old versions of large files stay on the server.
/// `blob:` is reserved (a prefix cannot start with it); `blob:limit=0`
/// leaves out every blob no tip reaches. Not supported over a
/// git-bridge remote.
#[arg(long)]
filter: Vec<String>,
/// Mark the partial clone lazy (P37): `sc cat` may fetch the trees
Expand Down Expand Up @@ -3009,10 +3014,7 @@ fn run_verify(refname: Option<String>, require: bool) -> Result<()> {
// EXPECTED to be absent on a partial clone, never missing/corrupt — print
// a count instead of treating them as a verify failure.
if let Some(gaps) = repo.partial_gap_count(&[tip])? {
let prefixes = repo
.promisor()?
.map(|p| p.prefixes().join(", "))
.unwrap_or_default();
let prefixes = repo.promisor()?.map(|p| p.describe()).unwrap_or_default();
print_line(&format!(
"partial: {gaps} object(s) outside filter [{prefixes}]"
));
Expand Down Expand Up @@ -3769,19 +3771,20 @@ fn run_clone(
return run_clone_git(&src, &dst);
}
let src = absolute_local_source(src);
let filter_opt = (!filter.is_empty()).then_some(filter.as_slice());
let repo = scl_repo::Repo::clone_url_opts(&src, &dst, filter_opt, no_transcripts)?;
let (prefixes, blob_limit) = scl_repo::promisor::parse_clone_filters(&filter)?;
let repo =
scl_repo::Repo::clone_url_limited(&src, &dst, &prefixes, blob_limit, no_transcripts)?;
if lazy {
scl_repo::promisor::set_lazy(repo.layout(), true)?;
}
let n = repo.branches()?.len();
if filter_opt.is_some() {
if let Some(promisor) = repo.promisor()? {
println!(
"cloned {} into {} ({} branch(es), partial: {})",
src,
dst.display(),
n,
filter.join(", ")
promisor.describe()
);
} else {
println!("cloned {} into {} ({} branch(es))", src, dst.display(), n);
Expand Down
Loading
Loading