From 94f88114a6b312867775de74a0527965d005e375 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:43:22 +0300 Subject: [PATCH 1/7] docs(adr): propose ADR-0055 blob-size clone filter (ref #217) Records why a size filter needs a server-evaluated GetPack field (PROTOCOL_VERSION 6), the tip-complete semantics, the gap model and repair, and defers the object-count limit. Not implemented; awaiting an owner decision on the wire bump. --- docs/adr/0055-blob-size-clone-filter.md | 208 ++++++++++++++++++++++++ docs/adr/README.md | 1 + 2 files changed, 209 insertions(+) create mode 100644 docs/adr/0055-blob-size-clone-filter.md diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md new file mode 100644 index 00000000..f531ac84 --- /dev/null +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -0,0 +1,208 @@ +# ADR-0055: Blob-size clone filter (`--filter blob:limit=`) + +- **Status:** Proposed. Needs an owner decision: it requires a wire change (`PROTOCOL_VERSION` 5 to 6). Nothing is implemented. +- **Date:** 2026-10-20 +- **Phase:** 27 follow-on (issue #217) +- **Builds on:** ADR-0037 (partial clone), ADR-0047 (want-by-id fetch), ADR-0048 (presence-based walks), ADR-0049/0050 (merge and sessions on partial clones), ADR-0044 (private branches), ADR-0014 (protected paths) + +## Context + +`sc clone --filter` takes path prefixes only. Issue #217 asks for git's +`blob:limit` axis (skip blobs above a size) and an object-count limit. + +ADR-0048 already made a partial clone a store plus one promise: every id a +present object references and the store lacks is held by the origin. The +filter records what a clone *asked for*; presence decides what exists. A blob +left out by size is the same kind of gap as a blob left out by path, so the +client side needs almost nothing new. The open questions are who learns the +size, and what a checkout does with a gapped file. + +## Findings that shape the design + +1. **Only the server can know a blob's size before receiving it.** A + `TreeEntry` is `{name, kind, id, mode, perms}` (`core/src/object.rs:119`); + there is no size. The wire has no size query: `HasObject` returns a bool, + `GetObject`/`GetObjects` return the object itself (`wire.rs:118`). The + by-id reply caps (ADR-0048 decision 5) refuse an over-cap *batch*, they do + not report sizes. +2. **The prefix filter cannot carry it.** `GetPack` ships `filter: + Vec` of path prefixes (`wire.rs:138`, `put_strs`). A reserved + entry such as `blob:limit=1048576` in that list would be read by a v5 + server as a path prefix and silently send almost nothing, a fail-unsafe + misparse. ADR-0047 rejected the same trick for the lazy flag. +3. **The walk that cuts a transfer never reads blobs.** + `walk_tree_filtered` decides blob inclusion from the path alone + (`reachable.rs:347`). A size rule means the server reads (or at least + sizes) each candidate blob once during the walk. +4. **Git has no object-count filter.** Its filters are `blob:none`, + `blob:limit`, `tree:`, `object:type`, `sparse:oid`, `combine`. A + count limit would also be undefined here: `reachable_objects_filtered` + collects into a `BTreeSet` ordered by id, so "the first N objects" would + be an arbitrary, server-dependent set, and a client could not tell a + cut-off object from a missing one by any rule. + +## Decisions + +### 1. User-facing form + +`sc clone --filter blob:limit= `, with `` parsed by +`scl_core::parse_byte_size` (`1048576`, `1MiB`, `512KiB`). It combines with +prefix filters in the same repeatable flag: `--filter src/ --filter +blob:limit=1MiB` means "only `src/`, and within it no blob above 1 MiB except +as decision 4 says". `blob:limit=0` is legal and means no blobs beyond the +tips. + +`--filter` values starting with `blob:` become a reserved grammar. An unknown +`blob:` form (`blob:none`, a misspelling) is an error, not a prefix. Cost: a +path prefix that literally begins with `blob:` can no longer be named in +`--filter`. Alternative: a separate `--blob-limit` flag, which has no +collision. This ADR recommends the issue's form; it is an owner choice. + +### 2. Object-count limit: deferred, not built + +No stable definition (finding 4), no git precedent, no stated use case. +`ROADMAP.md` keeps a one-line deferred entry for it; the blob-size part +leaves the deferred list when this lands. + +### 3. The server evaluates the filter: wire change, `PROTOCOL_VERSION` 6 + +Findings 1 and 2 leave no client-side design that saves transfer. + +- `Request::GetPack` gains a trailing optional `blob_limit` (a tag byte, then + a `u64` when present; absent means no limit, so `blob:limit=0` stays + representable). `PROTOCOL_VERSION` 5 to 6; a v5 peer is rejected at the + handshake, as at P25, P27, P34 and P37. +- `Transport::get_pack` gains the same parameter. Every implementation + (`LocalTransport`, `WireClient`, stdio, http) forwards it; the two wrappers + forward by hand as in ADR-0047. +- `build_pack_tempfile` passes it to `reachable_objects_filtered`, which + drops a blob over the limit into `gaps` (never `included`). The `haves` + side stays unfiltered, as ADR-0037 requires. +- Bucket remotes already refuse partial clone (ADR-0046) and refuse this. + +Alternatives considered and rejected: + +- **Client-side drop on ingest, no bump.** The client downloads every blob and + discards the large ones. It saves disk and gives the same gap model, but + not one byte of transfer, which is the point of the feature. It would also + hide the missing wire support behind a flag that does not do what git users + expect. +- **A size-query verb (`ObjectSizes`) plus client-chosen fetches.** Also a + new verb and a bump. The client would walk all history level by level over + `GetObjects` to learn blob ids first, then ask sizes, then fetch: many + round trips for something the server does in one walk. +- **Sentinel in the prefix list.** Finding 2. + +Server cost: each candidate blob is read once to measure it, then read again +when written to the pack if kept. This is O(blobs under the filter) per +clone and backfill, never per fetch. A size index is a later optimization. + +### 4. Tips are complete; the limit applies to history + +The server exempts blobs reachable from the root tree of each *want* (every +ref tip the client asked for) that match the prefix filter. The limit applies +to blobs reachable only through parent snapshots. + +Reason: checkout, `sc switch` between branches, and every operation that +materializes a tip need those blobs, and the clone's own initial checkout +runs before any lazy marker can exist (`sync.rs`, the +`ensure-view-exempt: clone` note). A strict limit would make a non-lazy +clone fail at its own checkout with `GapInView`, and a lazy one would fetch +the same large files in a second round trip right after clone. The +alternative that skips large tracked files at checkout would need a third +"tracked but absent" state in status, diff, commit graft, materialize and the +merge engine (the sparse machinery covers paths, not sizes) and is not a small +change. + +What this saves: old revisions of large files, and large files only reachable +from history. That is where git's `blob:limit` pays off for binary assets +too. What it does not do: skip a large file at HEAD. The issue's phrase +"reports or lazily fills large ones" is therefore met by the history case: +operations reaching an old gapped blob fill it by id on a lazy clone and fail +with the gap error on a non-lazy one (decision 5). + +### 5. Gap record, presentation and repair + +- **No new gap record.** Absence is the gap (ADR-0048). Push, verify and gc + already walk presence (`reachable_objects_partial`) and need no change; the + tests for them are new, the code is not. +- **Marker:** `.sc/promisor-blob-limit`, local and uncommitted, one decimal + byte count. Not a field in `.sc/promisor`, for the ADR-0047 reason (that + file is `origin` plus one path prefix per line). `Repo::promisor()` returns + a clone with the limit marker as partial even when the prefix list is empty; + an empty prefix list together with the marker means the whole tree is in + scope. `Promisor::matches`/`should_descend` and its consumers + (`graft_out_of_sparse`, the materialize old-view bound, push and verify + messages) need that case checked one by one; it is the main implementation + risk. +- **Presentation:** `sc verify` keeps `partial: N object(s) outside filter + [...]`, with `blob:limit=` printed in the bracket beside any + prefixes. `sc clone` prints one line naming the limit and the gap count. +- **Repair, lazy and online:** nothing to do. `ensure_view`, `sc cat` and the + ADR-0049 merge fill by id. +- **Repair, otherwise:** the gap error already names `sc backfill `. + For a size gap the prefix is the gapped file's own path, which + `Promisor::matches` accepts as an exact boundary. `sc backfill ` sends + every version of that path without applying the limit (backfill passes no + `blob_limit`). `sc backfill --all` removes the markers and converts to a full + clone, as before (`promisor::remove` must also delete the limit marker). + Error text gains one clause when the limit marker exists: "(blob:limit=1MiB + left this out of the clone)". No `--min-size` form: it would be a second + size parser on the repair side with no use that `` does not cover. +- **`sc fetch` is unchanged:** it stays unfiltered (ADR-0048 Context), so new + large blobs from new commits arrive in full. A limit-aware fetch needs the + exemption list for backfilled paths in the request, which is a larger design. + Deferred; the clone's limit therefore bounds the initial transfer, not later + growth. + +### 6. Interactions + +- **Protected content.** The size is the stored object's payload length. A + protected file's blob is ciphertext, so its size is the plaintext size plus + the sealing overhead; the server cannot see plaintext. A file near the + limit may flip sides from its plaintext size by that overhead. Protected + and plain blobs are treated by the same rule. No new information leaks: + ciphertext length is already visible to anyone who can fetch the object. +- **Private branches.** Unchanged: a filtered transfer already excludes them + (ADR-0044). +- **Sealed objects, signatures, transcripts.** Never subject to the limit; + they are not path-reached blobs. +- **Sparse and prefix filters.** The prefix decides scope first, the limit + prunes inside it. `.sc/sparse` is still set to the prefixes (none for a + size-only clone, meaning a full view). +- **Push, verify, gc.** Presence-based already (ADR-0048); a size gap is + checked against the push destination like any other gap. +- **Merge, pick, rebase, ws, work** (ADR-0049/0050). A merge base or an old + version that is a size gap is "asked for" by the engine and filled by id on + a lazy, online clone, or fails with the gap error. No new engine code. +- **Wire caps.** A gapped blob over the by-id reply cap behaves as in + ADR-0048 decision 5. + +## Implementation outline (once accepted) + +1. Wire v6: `GetPack.blob_limit`, `Transport::get_pack`, all transports, a + version test, the wire doc comments. +2. Server walk: size-aware, tip-exempt `reachable_objects_filtered`; tests + that a filtered pack omits a large history blob and keeps a large tip + blob; a differential test against the unlimited walk. +3. `Repo`: clone option, `.sc/promisor-blob-limit`, whole-tree scope case in + `Promisor`, `promisor::remove`, backfill passing no limit. +4. CLI: parse `blob:limit=` in `run_clone`, reject unknown `blob:` + forms, report line, `sc verify` text. +5. Tests: push, verify and gc on a size-limited clone; the gap error text; + prefix plus size; `sc backfill ` repair; lazy fill by id; partial + clone refusals unchanged. +6. Docs: ADR-0037 and ADR-0048 pointers, CLAUDE.md capability row and bullets, + ROADMAP, ARCHITECTURE, THREAT-MODEL (one line on sizes), the + `sc-cli-reference` skill, a demo script. + +## Open questions for the owner + +1. Accept the `PROTOCOL_VERSION` 6 bump? (Recommended: yes; no useful design + avoids it.) +2. Tips complete, limit on history only (decision 4), versus a strict limit? + (Recommended: tips complete.) +3. `--filter blob:limit=...` with `blob:` reserved, or a separate + `--blob-limit` flag? (Recommended: the issue's form.) +4. Object-count limit deferred, and `sc fetch` not limit-aware in the first + version. (Recommended: both deferred; PR says `Refs #217`.) diff --git a/docs/adr/README.md b/docs/adr/README.md index f3ba57be..9c220f8e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -65,6 +65,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0052](0052-secret-injection-by-fd.md) | Secret injection by file descriptor (`sc run --secrets-fd`) | Accepted (built) | — | | [0053](0053-tls-pin-management.md) | TLS pin management (`sc tls`) | Accepted (built) | — | | [0054](0054-tls-ca-validation.md) | CA-path validation for `sc+https://` | Accepted (built) | — | +| [0055](0055-blob-size-clone-filter.md) | Blob-size clone filter (`--filter blob:limit=`) | Proposed (needs wire v6 decision) | 27 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, From 74f129826c6ead92f7b7802396ed0c672f2cfda4 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:23:50 +0300 Subject: [PATCH 2/7] feat(repo,cli): blob-size clone filter, wire v6 (ref #217) sc clone --filter blob:limit= leaves out blobs of at least that size that no ref tip's root tree reaches. The server evaluates it: GetPack gains a trailing optional blob_limit and PROTOCOL_VERSION is 6. PackFilter is the single definition of an active filter (prefix or limit), so private-branch wants, the bucket refusal and the clone's ref skipping all follow it. Promisor gets an explicit AllPaths/Prefixes scope and a limit, persisted in .sc/promisor-blob-limit. ensure_view now fills for a size-only (full-view) lazy clone, and gap errors name the file and the limit. Refs #217 --- crates/cli/src/main.rs | 23 +- crates/cli/tests/blob_limit_filter.rs | 196 ++++++ crates/repo/src/bucket_transport.rs | 19 +- crates/repo/src/error.rs | 26 +- crates/repo/src/fill.rs | 27 +- crates/repo/src/http_transport.rs | 3 +- crates/repo/src/layout.rs | 7 + crates/repo/src/lib.rs | 4 +- crates/repo/src/partial_size.rs | 894 ++++++++++++++++++++++++++ crates/repo/src/partial_walks.rs | 2 +- crates/repo/src/private.rs | 38 ++ crates/repo/src/promisor.rs | 303 +++++++-- crates/repo/src/reachable.rs | 107 ++- crates/repo/src/sparse.rs | 4 +- crates/repo/src/stdio_transport.rs | 34 +- crates/repo/src/sync.rs | 42 +- crates/repo/src/transport.rs | 34 +- crates/repo/src/wire.rs | 172 ++++- crates/repo/src/worktree.rs | 35 +- 19 files changed, 1841 insertions(+), 129 deletions(-) create mode 100644 crates/cli/tests/blob_limit_filter.rs create mode 100644 crates/repo/src/partial_size.rs diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 26269204..d2b3091d 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -310,8 +310,13 @@ enum Cmd { git: bool, /// Partial clone (P27): only fetch objects under this path prefix /// (repeatable). Writes `.sc/promisor` + `.sc/sparse`; missing - /// objects can later be pulled in with `sc backfill`. Not supported - /// over a git-bridge remote. + /// objects can later be pulled in with `sc backfill`. Also accepts + /// `blob:limit=` (bytes, or a KiB/MiB/GiB suffix; ADR-0055), + /// once: blobs of at least that size are left out unless a ref tip + /// reaches them, so old versions of large files stay on the server. + /// `blob:` is reserved (a prefix cannot start with it); `blob:limit=0` + /// leaves out every blob no tip reaches. Not supported over a + /// git-bridge remote. #[arg(long)] filter: Vec, /// Mark the partial clone lazy (P37): `sc cat` may fetch the trees @@ -2984,10 +2989,7 @@ fn run_verify(refname: Option, require: bool) -> Result<()> { // EXPECTED to be absent on a partial clone, never missing/corrupt — print // a count instead of treating them as a verify failure. if let Some(gaps) = repo.partial_gap_count(&[tip])? { - let prefixes = repo - .promisor()? - .map(|p| p.prefixes().join(", ")) - .unwrap_or_default(); + let prefixes = repo.promisor()?.map(|p| p.describe()).unwrap_or_default(); print_line(&format!( "partial: {gaps} object(s) outside filter [{prefixes}]" )); @@ -3722,19 +3724,20 @@ fn run_clone( return run_clone_git(&src, &dst); } let src = absolute_local_source(src); - let filter_opt = (!filter.is_empty()).then_some(filter.as_slice()); - let repo = scl_repo::Repo::clone_url_opts(&src, &dst, filter_opt, no_transcripts)?; + let (prefixes, blob_limit) = scl_repo::promisor::parse_clone_filters(&filter)?; + let repo = + scl_repo::Repo::clone_url_limited(&src, &dst, &prefixes, blob_limit, no_transcripts)?; if lazy { scl_repo::promisor::set_lazy(repo.layout(), true)?; } let n = repo.branches()?.len(); - if filter_opt.is_some() { + if let Some(promisor) = repo.promisor()? { println!( "cloned {} into {} ({} branch(es), partial: {})", src, dst.display(), n, - filter.join(", ") + promisor.describe() ); } else { println!("cloned {} into {} ({} branch(es))", src, dst.display(), n); diff --git a/crates/cli/tests/blob_limit_filter.rs b/crates/cli/tests/blob_limit_filter.rs new file mode 100644 index 00000000..b3bc0c49 --- /dev/null +++ b/crates/cli/tests/blob_limit_filter.rs @@ -0,0 +1,196 @@ +//! ADR-0055: `sc clone --filter blob:limit=` through the real binary. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn sc(dir: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_sc")) + .args(args) + .current_dir(dir) + .output() + .expect("sc runs") +} + +fn stdout(o: &Output) -> String { + String::from_utf8_lossy(&o.stdout).into_owned() +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-blob-limit-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + d +} + +const OLD: &[u8] = &[b'o'; 4096]; +const NEW: &[u8] = &[b'n'; 4096]; + +/// An origin whose first commit holds a 4 KiB `big.bin` that the second +/// replaced, plus a small file. +fn origin(tag: &str) -> PathBuf { + let root = tmp(&format!("{tag}-origin")); + std::fs::create_dir_all(&root).unwrap(); + assert!(sc(&root, &["init"]).status.success()); + std::fs::write(root.join("big.bin"), OLD).unwrap(); + std::fs::write(root.join("small.txt"), "one\n").unwrap(); + assert!(sc(&root, &["commit", "-m", "c1"]).status.success()); + std::fs::write(root.join("big.bin"), NEW).unwrap(); + assert!(sc(&root, &["commit", "-m", "c2"]).status.success()); + root +} + +#[test] +fn size_limited_clone_checks_out_complete_and_reports_the_gap() { + let src = origin("clone"); + let dst = tmp("clone-dst"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(out.status.success(), "{}", stderr(&out)); + assert!( + stdout(&out).contains("partial: blob:limit=1024"), + "{}", + stdout(&out) + ); + assert_eq!(std::fs::read(dst.join("big.bin")).unwrap(), NEW); + assert_eq!( + std::fs::read_to_string(dst.join(".sc/promisor-blob-limit")).unwrap(), + "limit 1024\nscope all-paths\n" + ); + let v = sc(&dst, &["verify"]); + assert!(v.status.success(), "{}", stderr(&v)); + let text = stdout(&v); + assert!( + text.contains("partial: 1 object(s) outside filter [blob:limit=1024]"), + "{text}" + ); + assert!(sc(&dst, &["gc"]).status.success()); + assert!(sc(&dst, &["status"]).status.success()); + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&dst).unwrap(); +} + +#[test] +fn lazy_and_backfill_repair_a_size_gap() { + let src = origin("lazy"); + let dst = tmp("lazy-dst"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + "--lazy", + ], + ); + assert!( + out.status.success(), + "--lazy works with only blob: {}", + stderr(&out) + ); + assert!(dst.join(".sc/promisor-lazy").exists()); + + // Non-lazy sibling: backfill by the file's own path repairs the gap. + let dst2 = tmp("lazy-dst2"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst2.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(out.status.success(), "{}", stderr(&out)); + let b = sc(&dst2, &["backfill", "big.bin"]); + assert!(b.status.success(), "{}", stderr(&b)); + let v = stdout(&sc(&dst2, &["verify"])); + assert!(v.contains("partial: 0 object(s)"), "{v}"); + assert!( + std::fs::read_to_string(dst2.join(".sc/promisor-blob-limit")) + .unwrap() + .contains("scope all-paths"), + "backfill keeps the whole-tree scope" + ); + let all = sc(&dst2, &["backfill", "--all"]); + assert!(all.status.success(), "{}", stderr(&all)); + assert!(!dst2.join(".sc/promisor").exists()); + assert!(!dst2.join(".sc/promisor-blob-limit").exists()); + for d in [&src, &dst, &dst2] { + std::fs::remove_dir_all(d).unwrap(); + } +} + +#[test] +fn filter_grammar_errors_are_actionable() { + let src = origin("grammar"); + let dst = tmp("grammar-dst"); + let run = |filters: &[&str]| { + let mut args = vec!["clone", src.to_str().unwrap(), dst.to_str().unwrap()]; + for f in filters { + args.push("--filter"); + args.push(f); + } + sc(&src, &args) + }; + let none = run(&["blob:none"]); + assert!(!none.status.success()); + assert!(stderr(&none).contains("blob:limit=0"), "{}", stderr(&none)); + let dup = run(&["blob:limit=1", "blob:limit=2"]); + assert!(!dup.status.success()); + assert!(stderr(&dup).contains("more than once"), "{}", stderr(&dup)); + let bad = run(&["blob:limit=lots"]); + assert!(!bad.status.success()); + assert!(!dst.exists(), "a rejected filter leaves no destination"); + // `sc clone --help` documents the grammar. + let help = sc(&src, &["clone", "--help"]); + assert!( + stdout(&help).contains("blob:limit="), + "{}", + stdout(&help) + ); + std::fs::remove_dir_all(&src).unwrap(); +} + +#[test] +fn a_bucket_remote_refuses_a_size_only_clone() { + let src = origin("bucket"); + let bucket = tmp("bucket-wal"); + std::fs::create_dir_all(&bucket).unwrap(); + let url = format!("sc+wal://{}", bucket.display()); + assert!(sc(&src, &["remote", "add", "bk", &url]).status.success()); + assert!(sc(&src, &["push", "bk"]).status.success()); + let dst = tmp("bucket-dst"); + let out = sc( + &src, + &[ + "clone", + &url, + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(!out.status.success(), "a bucket remote must refuse"); + assert!( + stderr(&out).contains("partial clone from bucket remotes"), + "{}", + stderr(&out) + ); + for d in [&src, &bucket, &dst] { + let _ = std::fs::remove_dir_all(d); + } +} diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index b16c9043..a913311b 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -608,7 +608,7 @@ impl Transport for BucketTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { if filter.is_some() { @@ -922,7 +922,22 @@ mod tests { // filter is refused loudly, not ignored let filt = vec!["src/".to_string()]; assert!(t - .get_pack(&[tip], &[], Some(&filt), &mut Vec::new()) + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&filt, None).as_ref(), + &mut Vec::new() + ) + .is_err()); + // ... and so is a size-only filter (ADR-0055): "filter active" is + // `prefixes || blob_limit`, not "prefixes non-empty". + assert!(t + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&[], Some(1024)).as_ref(), + &mut Vec::new() + ) .is_err()); drop(t); std::fs::remove_dir_all(&broot).unwrap(); diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index 42af300c..fa4aea62 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -53,8 +53,20 @@ pub enum Error { /// out of the reply. Raised before anything is written, so the working /// tree and refs are untouched. `prefix` is the filter prefix that /// `sc backfill` can act on. - #[error("{path} is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill {prefix}` to repair it")] - GapInView { path: String, prefix: String }, + #[error("{path} is missing from this partial clone's local objects ({}); nothing was changed. Run `sc backfill {prefix}` to repair it", gap_cause(*.blob_limit))] + GapInView { + path: String, + prefix: String, + /// The clone's `blob:limit` (ADR-0055), when it has one: then `prefix` + /// is the file's own path, and the message names the limit as a cause. + blob_limit: Option, + }, + /// A path inside a partial clone's scope is absent and the clone was made + /// with a blob-size limit (ADR-0055): the blob is a large old version the + /// clone left out, or a fetch brought a tip that points at one. `sc + /// backfill ` fetches every version of that file. + #[error("{path} was left out of this partial clone (blob:limit={limit}); run `sc backfill {path}` to fetch it first")] + GapBySize { path: String, limit: u64 }, /// A push found an object the local store lacks (referenced by `referrer`) /// that the destination does not hold either (ADR-0048). Pushing the new /// tip would leave the destination with a tree that names an object it @@ -190,3 +202,13 @@ pub enum Error { } pub type Result = std::result::Result; + +/// The cause clause of [`Error::GapInView`]. +fn gap_cause(blob_limit: Option) -> String { + match blob_limit { + Some(limit) => format!( + "blob:limit={limit} left this out of the clone, or an earlier fetch left a gap inside the checked-out view" + ), + None => "an earlier fetch left a gap inside the checked-out view".to_string(), + } +} diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index af05851d..76581c69 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -157,8 +157,11 @@ impl Repo { offline: bool, ) -> Result { // An empty spec is a full checkout: never fill the whole tree on a - // partial clone's behalf (the preflight reports the gap instead). - if sparse.is_full() { + // partial clone's behalf (the preflight reports the gap instead). A + // size-only clone (ADR-0055) is the exception: its view is always full + // and what is missing under one root is exactly the large old blobs + // the operation needs, so the fill is bounded by that one revision. + if sparse.is_full() && !self.promisor()?.is_some_and(|p| p.is_all_paths()) { return Ok(FillStats::default()); } let mut filler = Filler::new(self, offline)?; @@ -345,10 +348,9 @@ impl<'a> Filler<'a> { return Ok(()); } let Some(origin) = self.origin.clone() else { - return Err(if self.repo.promisor()?.is_some() { - crate::promisor::partial_gap_hint(path) - } else { - Error::Core(scl_core::Error::NotFound(*id)) + return Err(match self.repo.promisor()? { + Some(p) => crate::promisor::gap_error(Some(&p), path), + None => Error::Core(scl_core::Error::NotFound(*id)), }); }; if self.transport.is_none() { @@ -377,10 +379,9 @@ impl<'a> Filler<'a> { return Ok(()); }; let Some(origin) = self.origin.clone() else { - return Err(if self.repo.promisor()?.is_some() { - crate::promisor::partial_gap_hint(path) - } else { - Error::Core(scl_core::Error::NotFound(*first)) + return Err(match self.repo.promisor()? { + Some(p) => crate::promisor::gap_error(Some(&p), path), + None => Error::Core(scl_core::Error::NotFound(*first)), }); }; if self.transport.is_none() { @@ -759,7 +760,7 @@ mod tests { &self, _: &[ObjectId], _: &[ObjectId], - _: Option<&[String]>, + _: Option<&crate::promisor::PackFilter>, _: &mut dyn Write, ) -> Result<()> { unimplemented!() @@ -866,7 +867,7 @@ mod tests { &self, w: &[ObjectId], h: &[ObjectId], - f: Option<&[String]>, + f: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.inner.get_pack(w, h, f, out) @@ -951,7 +952,7 @@ mod tests { &self, _: &[ObjectId], _: &[ObjectId], - _: Option<&[String]>, + _: Option<&crate::promisor::PackFilter>, _: &mut dyn Write, ) -> Result<()> { unimplemented!() diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index f0f853af..58680f9e 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -646,7 +646,7 @@ impl Transport for HttpTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.client.get_pack(wants, haves, filter, out) @@ -2784,6 +2784,7 @@ mod tests { wants: vec![tip], haves: vec![], filter: vec![], + blob_limit: None, } .encode(), ) diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index 607a63f0..ffff4a0f 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -109,6 +109,13 @@ impl Layout { pub fn promisor_path(&self) -> PathBuf { self.dot_sc.join("promisor") } + /// `.sc/promisor-blob-limit` — the blob-size limit a partial clone was + /// made with and whether its scope is every path (ADR-0055). A separate + /// file for the same reason as `promisor-lazy`: a line in `.sc/promisor` + /// is a path prefix, and any text is a legal path. + pub fn promisor_blob_limit_path(&self) -> PathBuf { + self.dot_sc.join("promisor-blob-limit") + } /// `.sc/promisor-lazy` — empty marker (P37): this partial clone may fill /// gaps from its promisor remote by object id (ADR-0047). A separate file, /// not a field in `.sc/promisor`, because that file's lines after diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 18c4202d..94db1fed 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -27,6 +27,8 @@ mod partial_replay; #[cfg(test)] mod partial_sessions; #[cfg(test)] +mod partial_size; +#[cfg(test)] mod partial_walks; pub mod pick_state; pub mod private; @@ -71,7 +73,7 @@ pub use gc::GcStats; pub use git_marks::MarksStore; pub use http_transport::ScHttpUrl; pub use oplog::{OpRecord, UndoOutcome}; -pub use promisor::Promisor; +pub use promisor::{PackFilter, Promisor, Scope}; pub use protect_ops::PrefixRecipient; pub use remote::{RemoteConfig, RemoteKind}; pub use replay::{PickResult, RebaseResult}; diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs new file mode 100644 index 00000000..8cfa41b2 --- /dev/null +++ b/crates/repo/src/partial_size.rs @@ -0,0 +1,894 @@ +//! Tests for the blob-size clone filter (ADR-0055, `--filter blob:limit=`). +//! +//! The fixture is an origin whose history has large blobs that only an old +//! commit references, so a size-limited clone leaves them out while every ref +//! tip stays complete. + +use std::path::PathBuf; +use std::time::Duration; + +use scl_core::{Object, ObjectId}; + +use crate::error::Error; +use crate::promisor::{PackFilter, Scope}; +use crate::refs; +use crate::replay::{PickResult, RebaseResult}; +use crate::repo::Repo; + +const LIMIT: u64 = 1024; + +fn blob_id(bytes: &[u8]) -> ObjectId { + Object::blob(bytes.to_vec()).id() +} + +fn has(repo: &Repo, id: &ObjectId) -> bool { + repo.vfs().store().lock().unwrap().contains(id) +} + +/// 300 numbered lines (about 2.7 KB, over `LIMIT`) with `edits` replacing +/// lines by index, so two edits to different lines merge cleanly. +fn text(edits: &[(usize, &str)]) -> Vec { + let mut out = String::new(); + for n in 0..300 { + match edits.iter().find(|(i, _)| *i == n) { + Some((_, replacement)) => out.push_str(&format!("{replacement}\n")), + None => out.push_str(&format!("line {n}\n")), + } + } + out.into_bytes() +} + +fn huge(tag: u8) -> Vec { + vec![tag; 8192] +} + +struct Sz { + origin_root: PathBuf, + clone_root: PathBuf, + origin: Repo, + c1: ObjectId, + c2: ObjectId, +} + +impl Sz { + /// An origin with `big.txt` (a text over the limit), `small.txt` and + /// `src/huge.bin` at `c1`, each rewritten at `c2` (`main`). Everything the + /// first commit holds that the second replaced is history only. + fn new(tag: &str) -> Sz { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-{tag}-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-{tag}-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(origin_root.join("src")).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("big.txt"), text(&[])).unwrap(); + std::fs::write(origin_root.join("small.txt"), b"s1\n").unwrap(); + std::fs::write(origin_root.join("src/huge.bin"), huge(1)).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + std::fs::write(origin_root.join("big.txt"), text(&[(0, "ours")])).unwrap(); + std::fs::write(origin_root.join("small.txt"), b"s2\n").unwrap(); + std::fs::write(origin_root.join("src/huge.bin"), huge(2)).unwrap(); + let c2 = origin.commit("t", "c2").unwrap(); + Sz { + origin_root, + clone_root, + origin, + c1, + c2, + } + } + + fn clone(&self, prefixes: &[&str], limit: u64, lazy: bool) -> Repo { + let prefixes: Vec = prefixes.iter().map(|s| s.to_string()).collect(); + let clone = Repo::clone_url_limited( + self.origin_root.to_str().unwrap(), + &self.clone_root, + &prefixes, + Some(limit), + false, + ) + .unwrap(); + crate::promisor::set_lazy(clone.layout(), lazy).unwrap(); + clone + } + + fn cleanup(self) { + drop(self.origin); + for r in [&self.origin_root, &self.clone_root] { + if r.exists() { + std::fs::remove_dir_all(r).unwrap(); + } + assert!(!r.exists(), "fixture {} must be removed", r.display()); + } + } +} + +/// The core promise: the history-only large blobs are left out, every tip blob +/// (large or small) is present, the checkout is complete, and the clone knows +/// its scope and limit across a reload. +#[test] +fn size_only_clone_leaves_out_history_blobs_and_keeps_every_tip_complete() { + let sz = Sz::new("core"); + let clone = sz.clone(&[], LIMIT, false); + + assert!(!has(&clone, &blob_id(&text(&[]))), "old big.txt is a gap"); + assert!(!has(&clone, &blob_id(&huge(1))), "old huge.bin is a gap"); + assert!(has(&clone, &blob_id(&text(&[(0, "ours")]))), "tip big.txt"); + assert!(has(&clone, &blob_id(&huge(2))), "tip huge.bin"); + assert!(has(&clone, &blob_id(b"s1\n")), "small history blob is kept"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "the checkout materializes the large tip file" + ); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(2) + ); + + let promisor = clone.promisor().unwrap().unwrap(); + assert!(promisor.is_all_paths()); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert!(promisor.prefixes().is_empty()); + assert_eq!(promisor.describe(), format!("blob:limit={LIMIT}")); + assert_eq!( + std::fs::read_to_string(clone.layout().promisor_blob_limit_path()).unwrap(), + "limit 1024\nscope all-paths\n" + ); + assert!(clone.sparse_spec().unwrap().is_full(), "the view is full"); + // verify counts the two large old blobs as the gap frontier. + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(2)); + drop(clone); + sz.cleanup(); +} + +/// A blob of exactly `limit` bytes is left out (size >= limit, like git); one +/// byte less is kept. `blob:limit=0` leaves out every blob no tip reaches. +#[test] +fn boundary_is_inclusive_and_zero_keeps_only_tip_blobs() { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-edge-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-edge-clone-{pid}")); + let zero_root = base.join(format!("scl-p27sz-edge-zero-{pid}")); + for r in [&origin_root, &clone_root, &zero_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(&origin_root).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + let eq = vec![b'e'; LIMIT as usize]; + let lt = vec![b'l'; LIMIT as usize - 1]; + std::fs::write(origin_root.join("eq.bin"), &eq).unwrap(); + std::fs::write(origin_root.join("lt.bin"), <).unwrap(); + origin.commit("t", "c1").unwrap(); + std::fs::write(origin_root.join("eq.bin"), b"gone").unwrap(); + std::fs::write(origin_root.join("lt.bin"), b"gone too").unwrap(); + origin.commit("t", "c2").unwrap(); + + let clone = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &clone_root, + &[], + Some(LIMIT), + false, + ) + .unwrap(); + assert!(!has(&clone, &blob_id(&eq)), "size == limit is left out"); + assert!(has(&clone, &blob_id(<)), "size == limit - 1 is kept"); + drop(clone); + + let zero = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &zero_root, + &[], + Some(0), + false, + ) + .unwrap(); + assert!( + !has(&zero, &blob_id(<)), + "limit 0 leaves out every old blob" + ); + assert!(has(&zero, &blob_id(b"gone")), "but every tip blob is kept"); + assert_eq!( + std::fs::read(zero_root.join("eq.bin")).unwrap(), + b"gone", + "the checkout is complete" + ); + drop(zero); + drop(origin); + for r in [&origin_root, &clone_root, &zero_root] { + std::fs::remove_dir_all(r).unwrap(); + } +} + +/// A clone that combines a prefix with a limit: the prefix decides scope, the +/// limit prunes inside it, and the promisor keeps both. +#[test] +fn prefix_plus_limit_scopes_first_then_prunes() { + let sz = Sz::new("combo"); + let clone = sz.clone(&["src/"], LIMIT, false); + assert!( + !has(&clone, &blob_id(&huge(1))), + "in scope, over limit, old" + ); + assert!(has(&clone, &blob_id(&huge(2))), "in scope, over limit, tip"); + assert!( + !has(&clone, &blob_id(&text(&[(0, "ours")]))), + "outside the prefix, even at the tip" + ); + let promisor = clone.promisor().unwrap().unwrap(); + assert!(!promisor.is_all_paths()); + assert_eq!(promisor.prefixes(), &["src/".to_string()]); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert_eq!( + promisor.describe(), + format!("src/, blob:limit={LIMIT}"), + "verify names both" + ); + assert_eq!( + std::fs::read_to_string(clone.layout().promisor_blob_limit_path()).unwrap(), + "limit 1024\nscope prefixes\n" + ); + assert!(!sz.clone_root.join("big.txt").exists()); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(2) + ); + drop(clone); + sz.cleanup(); +} + +/// `sc backfill ` on a size-only clone fetches every version of the path, +/// and the whole-tree scope and the limit survive it and a reload (B1). +#[test] +fn backfill_keeps_the_whole_tree_scope_and_limit() { + let sz = Sz::new("backfill"); + let clone = sz.clone(&[], LIMIT, false); + clone.backfill(&["src/huge.bin".to_string()]).unwrap(); + assert!(has(&clone, &blob_id(&huge(1))), "the old version arrived"); + assert!( + !has(&clone, &blob_id(&text(&[]))), + "an unrelated path is untouched" + ); + let promisor = clone.promisor().unwrap().unwrap(); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert!(promisor.is_all_paths(), "still every path"); + assert!(promisor.prefixes().is_empty(), "widen did not add a prefix"); + assert!(matches!( + crate::promisor::load(clone.layout()).unwrap().unwrap(), + p if p == promisor + )); + // The gap count falls by the repaired blob. + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(1)); + + clone.backfill_all().unwrap(); + assert!(clone.promisor().unwrap().is_none(), "now a full clone"); + assert!( + !clone.layout().promisor_blob_limit_path().exists(), + "the limit marker went with it" + ); + assert!(has(&clone, &blob_id(&text(&[])))); + drop(clone); + sz.cleanup(); +} + +/// Marker persistence: the explicit scope round-trips, a stray limit file with +/// no `.sc/promisor` is ignored, and a prefix scope with a limit reloads as such. +#[test] +fn scope_and_limit_round_trip_through_the_markers() { + let root = std::env::temp_dir().join(format!("scl-p27sz-marker-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let layout = crate::layout::Layout::at(&root); + std::fs::create_dir_all(&layout.dot_sc).unwrap(); + + let all = crate::Promisor::with_blob_limit("o", vec![], 5); + crate::promisor::store(&layout, &all).unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), Some(all)); + + let pref = crate::Promisor::with_blob_limit("o", vec!["a/".into(), "b".into()], 7); + crate::promisor::store(&layout, &pref).unwrap(); + let back = crate::promisor::load(&layout).unwrap().unwrap(); + assert_eq!(back, pref); + assert!(matches!(back.prefixes(), [a, b] if a == "a/" && b == "b")); + + // Storing a limit-less marker removes the limit file. + crate::promisor::store(&layout, &crate::Promisor::new("o", vec!["a/".into()])).unwrap(); + assert!(!layout.promisor_blob_limit_path().exists()); + + // A limit file without `.sc/promisor` is leftover state: a full clone. + std::fs::write( + layout.promisor_blob_limit_path(), + "limit 9\nscope all-paths\n", + ) + .unwrap(); + std::fs::remove_file(layout.promisor_path()).unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), None); + + // Malformed limit file is an error, not a silent unlimited clone. + crate::promisor::store(&layout, &crate::Promisor::new("o", vec![])).unwrap(); + std::fs::write(layout.promisor_blob_limit_path(), "limit nope\n").unwrap(); + assert!(crate::promisor::load(&layout).is_err()); + std::fs::remove_dir_all(&root).unwrap(); +} + +/// Widening an all-paths scope is a no-op, and widening a prefix scope keeps +/// its limit. +#[test] +fn widen_cannot_narrow_an_all_paths_scope() { + let mut p = crate::Promisor::with_blob_limit("o", vec![], 5); + p.widen(&["x/".into()]); + assert!(p.is_all_paths()); + assert!(p.matches("anything/at/all")); + assert!(p.should_descend("a/b")); + assert!(matches!( + Scope::AllPaths, + ref s if *s == Scope::AllPaths + )); + let mut q = crate::Promisor::with_blob_limit("o", vec!["a/".into()], 5); + q.widen(&["b/".into()]); + assert_eq!(q.prefixes(), &["a/".to_string(), "b/".to_string()]); + assert_eq!(q.blob_limit(), Some(5)); +} + +/// `PackFilter` is the one definition of "filter active": a limit alone counts. +#[test] +fn pack_filter_is_active_for_a_prefix_or_a_limit() { + assert!(PackFilter::new(&[], None).is_none()); + assert!(PackFilter::new(&["a/".to_string()], None).is_some()); + let f = PackFilter::new(&[], Some(0)).expect("a limit of zero is still a filter"); + assert_eq!(f.blob_limit(), Some(0)); + assert!(f.prefixes().is_empty()); +} + +/// `--filter` grammar (decision 1): one limit, no `blob:none`, `blob:` reserved. +#[test] +fn filter_grammar_rejects_duplicates_and_unknown_blob_forms() { + use crate::promisor::parse_clone_filters as parse; + let v = |xs: &[&str]| xs.iter().map(|s| s.to_string()).collect::>(); + assert_eq!( + parse(&v(&["src/", "blob:limit=1MiB"])).unwrap(), + (vec!["src/".to_string()], Some(1 << 20)) + ); + assert_eq!(parse(&v(&["blob:limit=0"])).unwrap(), (vec![], Some(0))); + assert_eq!( + parse(&v(&["blob:limit=512KiB"])).unwrap().1, + Some(512 << 10) + ); + assert_eq!(parse(&v(&["blob:limit=2048"])).unwrap().1, Some(2048)); + let dup = parse(&v(&["blob:limit=1", "blob:limit=2"])).unwrap_err(); + assert!(dup.to_string().contains("more than once"), "{dup}"); + let none = parse(&v(&["blob:none"])).unwrap_err(); + assert!(none.to_string().contains("blob:limit=0"), "{none}"); + let other = parse(&v(&["blob:oops"])).unwrap_err(); + assert!(other.to_string().contains("reserved"), "{other}"); + assert!(parse(&v(&["blob:limit=lots"])).is_err()); + assert!(parse(&v(&["blob:limit="])).is_err()); +} + +/// gc, verify and push on a size-limited clone (presence-based, ADR-0048). +#[test] +fn gc_verify_and_push_work_on_a_size_limited_clone() { + let sz = Sz::new("gcpush"); + let clone = sz.clone(&[], LIMIT, false); + let old_big = blob_id(&text(&[])); + + clone.gc(Duration::from_secs(0)).unwrap(); + assert!(has(&clone, &blob_id(&huge(2))), "gc keeps the tip"); + assert!(!has(&clone, &old_big), "and does not invent the gap"); + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(2)); + + // A commit on the clone carries the gapped history by id and pushes. + std::fs::write(sz.clone_root.join("small.txt"), b"s3\n").unwrap(); + let tip = clone.commit("t", "local").unwrap(); + // The origin handle holds the repo lock; release it for the push. + let Sz { + origin_root, + clone_root, + origin, + .. + } = sz; + drop(origin); + assert_eq!(clone.push("origin").unwrap(), tip); + drop(clone); + let origin = Repo::open(&origin_root).unwrap(); + assert!(has(&origin, &old_big), "the origin still holds the history"); + assert_eq!(origin.head_tip().unwrap(), Some(tip)); + assert!(origin.snapshot(&tip).is_ok()); + drop(origin); + for r in [&origin_root, &clone_root] { + std::fs::remove_dir_all(r).unwrap(); + } +} + +/// A destination that is not a mirror lacks the gapped history, so the push is +/// refused before sending a snapshot that points at it. +#[test] +fn push_to_a_non_mirror_refuses_on_a_size_gap() { + let sz = Sz::new("pushother"); + let clone = sz.clone(&[], LIMIT, false); + let other_root = std::env::temp_dir().join(format!("scl-p27sz-other-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&other_root); + std::fs::create_dir_all(&other_root).unwrap(); + drop(Repo::init(&other_root).unwrap()); + clone + .remote_add("other", other_root.to_str().unwrap()) + .unwrap(); + std::fs::write(sz.clone_root.join("small.txt"), b"s3\n").unwrap(); + clone.commit("t", "local").unwrap(); + let err = clone.push("other").unwrap_err(); + match err { + Error::PushGap { id, .. } => { + assert!(id == blob_id(&text(&[])) || id == blob_id(&huge(1)), "{id}") + } + other => panic!("expected PushGap, got {other:?}"), + } + drop(clone); + std::fs::remove_dir_all(&other_root).unwrap(); + sz.cleanup(); +} + +/// Tip-exemption ordering (D7): a tip that is an ancestor of another tip keeps +/// its own blobs, so switching to it works on a non-lazy clone. +#[test] +fn an_ancestor_tip_is_exempt_even_though_another_tip_reaches_it() { + let sz = Sz::new("anc"); + // `old` is a ref at c1, which is also main's parent. + sz.origin.branch("old").unwrap(); + refs::write_branch_tip(sz.origin.layout(), "old", &sz.c1).unwrap(); + let clone = sz.clone(&[], LIMIT, false); + assert!( + has(&clone, &blob_id(&text(&[]))), + "old tip's big.txt is kept" + ); + assert!(has(&clone, &blob_id(&huge(1)))); + clone.fetch("origin").unwrap(); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + clone.switch("old").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(1) + ); + drop(clone); + sz.cleanup(); +} + +/// A tip whose root tree equals an older subtree: `side` is the first commit +/// (root R), and `main` moved everything into `x/` and then deleted it, so R +/// also appears at `x` in a non-tip. The tip's blob must be kept. +#[test] +fn a_tip_root_equal_to_an_older_subtree_is_still_expanded() { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-eq-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-eq-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(&origin_root).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("big.bin"), huge(7)).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + std::fs::create_dir_all(origin_root.join("x")).unwrap(); + std::fs::rename(origin_root.join("big.bin"), origin_root.join("x/big.bin")).unwrap(); + origin.commit("t", "move").unwrap(); + std::fs::remove_dir_all(origin_root.join("x")).unwrap(); + std::fs::write(origin_root.join("tiny.txt"), b"t").unwrap(); + origin.commit("t", "drop").unwrap(); + refs::write_branch_tip(origin.layout(), "side", &c1).unwrap(); + + let clone = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &clone_root, + &[], + Some(LIMIT), + false, + ) + .unwrap(); + assert!( + has(&clone, &blob_id(&huge(7))), + "side's tip reaches the blob; the older x/ copy must not hide it" + ); + drop(clone); + drop(origin); + for r in [&origin_root, &clone_root] { + std::fs::remove_dir_all(r).unwrap(); + } +} + +/// The walk itself: limit semantics at the `reachable` level, both orderings of +/// the wants, and `None` equals the unlimited walk. +#[test] +fn limited_walk_orders_and_none_match_the_unlimited_walk() { + let sz = Sz::new("walk"); + let arc = sz.origin.vfs().store(); + let mut store = arc.lock().unwrap(); + let all = crate::promisor::Promisor::with_blob_limit("", vec![], LIMIT); + let f: &dyn crate::reachable::PrefixFilter = &all; + let full = crate::reachable::reachable_objects(&mut *store, &[sz.c2]).unwrap(); + + let none = + crate::reachable::reachable_objects_limited(&mut *store, &[sz.c2], Some(f), None).unwrap(); + assert_eq!(none.included, full, "no limit keeps everything"); + assert!(none.gaps.is_empty()); + + for tips in [vec![sz.c2], vec![sz.c2, sz.c1], vec![sz.c1, sz.c2]] { + let r = + crate::reachable::reachable_objects_limited(&mut *store, &tips, Some(f), Some(LIMIT)) + .unwrap(); + let old_in_tips = tips.contains(&sz.c1); + assert_eq!( + r.included.contains(&blob_id(&huge(1))), + old_in_tips, + "old huge.bin is kept exactly when c1 is itself a tip ({tips:?})" + ); + assert!(r.included.contains(&blob_id(&huge(2)))); + assert!(r.included.is_disjoint(&r.gaps)); + } + drop(store); + drop(arc); + sz.cleanup(); +} + +/// B3: a lazy size-only clone has a full view, and switch/undo to a revision +/// with a gap must still fill (the `is_full()` guard used to skip it). +#[test] +fn lazy_full_view_switch_and_undo_fill_the_gapped_revision() { + let sz = Sz::new("b3"); + let clone = sz.clone(&[], LIMIT, false); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + + // Non-lazy: refused before writing, naming the file. + let err = clone.switch("old").unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "main"); + + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.switch("old").unwrap(); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "old"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + assert!(has(&clone, &blob_id(&huge(1))), "filled by id"); + + // Undo back to main (already complete) and then forward again is a no-op + // for gaps; undo to the gapped tip on a non-lazy clone fails first. + clone.switch("main").unwrap(); + crate::promisor::set_lazy(clone.layout(), false).unwrap(); + clone.gc(Duration::from_secs(0)).unwrap(); + // gc kept the filled blobs (reachable from `old`), so nothing is missing. + assert!(has(&clone, &blob_id(&huge(1)))); + drop(clone); + sz.cleanup(); +} + +/// D2: tips are complete at clone time only. A fetch can bring a tip that +/// points at a gapped old blob; the non-lazy ff fails before writing with the +/// new clause and the file's own path, `backfill ` repairs it, and a +/// lazy clone heals it by id. +#[test] +fn a_fetch_that_reintroduces_a_gapped_blob_is_a_gap_in_view() { + for lazy in [false, true] { + let sz = Sz::new(if lazy { "d2-lazy" } else { "d2" }); + let clone = sz.clone(&[], LIMIT, lazy); + // Revert big.txt to its c1 content: the blob the clone left out. + std::fs::write(sz.origin_root.join("big.txt"), text(&[])).unwrap(); + sz.origin.commit("t", "revert").unwrap(); + clone.fetch("origin").unwrap(); + assert!( + !has(&clone, &blob_id(&text(&[]))), + "the fetch did not send it" + ); + + if lazy { + clone.merge("origin/main", "t").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + } else { + let err = clone.merge("origin/main", "t").unwrap_err(); + match &err { + Error::GapInView { + path, + prefix, + blob_limit, + } => { + assert_eq!(path, "big.txt"); + assert_eq!(prefix, "big.txt", "the file's own path"); + assert_eq!(*blob_limit, Some(LIMIT)); + } + other => panic!("expected GapInView, got {other:?}"), + } + let msg = err.to_string(); + assert!(msg.contains("blob:limit=1024"), "{msg}"); + assert!(msg.contains("`sc backfill big.txt`"), "{msg}"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "nothing was written" + ); + clone.backfill(&["big.txt".to_string()]).unwrap(); + clone.merge("origin/main", "t").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + } + drop(clone); + sz.cleanup(); + } +} + +/// D3 on a prefix + limit clone: the error names the file, not the covering +/// prefix `src/`. +#[test] +fn gap_error_names_the_file_on_a_prefix_plus_limit_clone() { + let sz = Sz::new("d3"); + let clone = sz.clone(&["src/"], LIMIT, false); + std::fs::write(sz.origin_root.join("src/huge.bin"), huge(1)).unwrap(); + sz.origin.commit("t", "revert").unwrap(); + clone.fetch("origin").unwrap(); + let err = clone.merge("origin/main", "t").unwrap_err(); + match &err { + Error::GapInView { path, prefix, .. } => { + assert_eq!(path, "src/huge.bin"); + assert_eq!(prefix, "src/huge.bin", "not the covering prefix"); + } + other => panic!("expected GapInView, got {other:?}"), + } + assert!( + err.to_string().contains("`sc backfill src/huge.bin`"), + "{err}" + ); + drop(clone); + sz.cleanup(); +} + +/// A prefix-only partial clone keeps its old error text (no limit clause). +#[test] +fn prefix_only_gap_error_is_unchanged() { + let err = Error::GapInView { + path: "src/a.txt".into(), + prefix: "src/".into(), + blob_limit: None, + }; + assert_eq!( + err.to_string(), + "src/a.txt is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill src/` to repair it" + ); +} + +/// Merge / cherry-pick / rebase with a size gap as the merge base (D3 + the +/// ADR-0049 engine): a non-lazy clone is refused before any write with the +/// file's own path, a lazy one fills the base by id and lands the merge. +#[test] +fn replay_and_merge_with_a_size_gap_as_base() { + struct Case(&'static str); + for case in [Case("merge"), Case("pick"), Case("rebase")] { + for lazy in [false, true] { + let sz = Sz::new(&format!("base-{}-{lazy}", case.0)); + // `b1` forks at c1 and edits another line of big.txt, so the base + // (c1's big.txt) is reachable from no tip and is a gap. + refs::write_branch_tip(sz.origin.layout(), "b1", &sz.c1).unwrap(); + sz.origin.switch("b1").unwrap(); + std::fs::write(sz.origin_root.join("big.txt"), text(&[(299, "theirs")])).unwrap(); + std::fs::write(sz.origin_root.join("src/huge.bin"), huge(1)).unwrap(); + sz.origin.commit("t", "b1 edit").unwrap(); + sz.origin.switch("main").unwrap(); + + let clone = sz.clone(&[], LIMIT, lazy); + clone.fetch("origin").unwrap(); + let base_blob = blob_id(&text(&[])); + assert!(!has(&clone, &base_blob), "premise: the base is a gap"); + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + + let run = |c: &Repo| -> Result<(), Error> { + match case.0 { + "merge" => c.merge("origin/b1", "t").map(|_| ()), + "pick" => c + .cherry_pick("origin/b1", "t", None, None) + .map(|r| assert!(matches!(r, PickResult::Picked(_)))), + _ => c.rebase("origin/b1", "t", None).map(|r| { + assert!(matches!( + r, + RebaseResult::Rebased { .. } | RebaseResult::FastForwarded(_) + )) + }), + } + }; + if lazy { + run(&clone).unwrap_or_else(|e| panic!("{} lazy: {e:?}", case.0)); + assert!(has(&clone, &base_blob), "the base was filled by id"); + let merged = std::fs::read(sz.clone_root.join("big.txt")).unwrap(); + let merged = String::from_utf8(merged).unwrap(); + if case.0 != "rebase" { + assert!(merged.starts_with("ours\n"), "{}", &merged[..20]); + assert!(merged.contains("theirs\n")); + } + } else { + let err = run(&clone) + .err() + .unwrap_or_else(|| panic!("{}: non-lazy run succeeded", case.0)); + match &err { + Error::GapBySize { path, limit } => { + assert_eq!(path, "big.txt", "{} names the file", case.0); + assert_eq!(*limit, LIMIT); + } + Error::GapInView { prefix, .. } => { + assert_eq!(prefix, "big.txt", "{} names the file", case.0) + } + other => panic!("{}: unexpected {other:?}", case.0), + } + assert!( + err.to_string().contains("sc backfill big.txt"), + "{}: {err}", + case.0 + ); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before, + "refused before any ref moved" + ); + assert!(!clone.merge_in_progress() && !clone.pick_in_progress()); + // The repair the message names makes the retry succeed. + clone.backfill(&["big.txt".to_string()]).unwrap(); + run(&clone).unwrap(); + } + drop(clone); + sz.cleanup(); + } + } +} + +/// Sessions on a size-only clone: a non-lazy workspace forks the complete tip +/// view and harvests a clean merge, and a lazy clone's fork after a healed +/// fast-forward does too. +#[test] +fn workspace_session_on_a_size_limited_clone_lands() { + use crate::ws::WsHarvestOutcome; + for lazy in [false, true] { + let sz = Sz::new(if lazy { "ws-lazy" } else { "ws" }); + let clone = sz.clone(&[], LIMIT, lazy); + let session = clone.ws_fork(1, "t", None).unwrap(); + let ws = &session.workspaces[0].dir; + assert_eq!( + std::fs::read(ws.join("src/huge.bin")).unwrap(), + huge(2), + "the large tip file is laid out in the workspace" + ); + std::fs::write(ws.join("big.txt"), text(&[(0, "ours"), (150, "ws")])).unwrap(); + std::fs::write( + sz.clone_root.join("big.txt"), + text(&[(0, "ours"), (299, "host")]), + ) + .unwrap(); + clone.commit("t", "host edit").unwrap(); + let outcomes = clone.ws_harvest(None, "t", None).unwrap(); + assert!( + matches!(outcomes.as_slice(), [WsHarvestOutcome::Landed { .. }]), + "got {outcomes:?}" + ); + let landed = + String::from_utf8(std::fs::read(sz.clone_root.join("big.txt")).unwrap()).unwrap(); + assert!(landed.contains("ws\n") && landed.contains("host\n")); + drop(clone); + sz.cleanup(); + } +} + +/// `--lazy` works when the only filter is `blob:`: the marker is set after a +/// size-only clone and `sc cat` fills an old version by id. +#[test] +fn lazy_works_when_the_only_filter_is_a_size_limit() { + let sz = Sz::new("lazycat"); + let clone = sz.clone(&[], LIMIT, true); + assert!(crate::promisor::is_lazy(clone.layout())); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + let bytes = clone.cat_at("old", "src/huge.bin").unwrap(); + assert_eq!(&bytes[..], &huge(1)[..]); + assert!(has(&clone, &blob_id(&huge(1)))); + drop(clone); + sz.cleanup(); +} + +/// A non-lazy clone's `sc cat` of an old version gets the size-aware error. +#[test] +fn cat_of_a_gapped_blob_names_the_limit_when_not_lazy() { + let sz = Sz::new("catgap"); + let clone = sz.clone(&[], LIMIT, false); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + let err = clone.cat_at("old", "src/huge.bin").unwrap_err(); + assert!(matches!(err, Error::GapBySize { .. }), "got {err:?}"); + assert!(err.to_string().contains("blob:limit=1024"), "{err}"); + assert!( + err.to_string().contains("sc backfill src/huge.bin"), + "{err}" + ); + drop(clone); + sz.cleanup(); +} + +/// The partial-clone refusals apply to a size-only clone exactly as before. +#[test] +fn partial_refusals_are_unchanged_on_a_size_limited_clone() { + let sz = Sz::new("refuse"); + let clone = sz.clone(&[], LIMIT, false); + let (sk, pk) = scl_crypto::generate_keypair(); + let rid = pk.recipient_id(); + let err = clone.set_sparse(&[], None).unwrap_err(); + assert!( + matches!(err, Error::InvalidArgument(ref m) if m.contains("empty sparse spec")), + "got {err:?}" + ); + assert!( + err.to_string().contains(&format!("blob:limit={LIMIT}")), + "{err}" + ); + assert!(matches!( + clone.rewrap(&sk, &[], &[], false), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.grant("src/", &sk, &pk), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.revoke("src/", &rid), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!( + clone.disable_sparse(None).is_err(), + "sparse disable stays refused" + ); + drop(clone); + sz.cleanup(); +} + +/// B3 for undo: restoring a ref to a revision with a size gap fills it first on +/// a lazy full-view clone, and refuses before moving anything otherwise. +#[test] +fn undo_to_a_gapped_revision_fills_when_lazy_and_refuses_otherwise() { + for lazy in [false, true] { + let sz = Sz::new(if lazy { "undo-lazy" } else { "undo" }); + let clone = sz.clone(&[], LIMIT, lazy); + let head = refs::current_branch(clone.layout()).unwrap(); + crate::oplog::record( + clone.layout(), + "synthetic", + &head, + &head, + &[("main".to_string(), Some(sz.c1), Some(sz.c2))], + ) + .unwrap(); + if lazy { + clone.undo().unwrap(); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(sz.c1) + ); + assert!( + has(&clone, &blob_id(&huge(1))), + "filled before the ref moved" + ); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(1) + ); + } else { + let err = clone.undo().unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(sz.c2), + "no ref moved" + ); + } + drop(clone); + sz.cleanup(); + } +} diff --git a/crates/repo/src/partial_walks.rs b/crates/repo/src/partial_walks.rs index 01e77601..d5cb6f13 100644 --- a/crates/repo/src/partial_walks.rs +++ b/crates/repo/src/partial_walks.rs @@ -142,7 +142,7 @@ fn fetch_leaves_in_filter_gap_gc_works_and_ff_fails_before_writing() { let head_before = refs::read_branch_tip(clone.layout(), "main").unwrap(); let err = clone.merge("origin/main", "t").unwrap_err(); match &err { - Error::GapInView { path, prefix } => { + Error::GapInView { path, prefix, .. } => { assert_eq!(path, "src/readme.txt"); assert_eq!(prefix, "src/"); } diff --git a/crates/repo/src/private.rs b/crates/repo/src/private.rs index aba30038..19cdb717 100644 --- a/crates/repo/src/private.rs +++ b/crates/repo/src/private.rs @@ -2545,6 +2545,44 @@ mod tests { std::fs::remove_dir_all(&part_dst).unwrap(); } + /// ADR-0055 B2: "filter active" is `prefixes || blob_limit`, so a clone + /// whose only filter is a size limit also leaves the private ref out. + #[test] + fn size_only_clone_excludes_private_branches() { + let (root, repo, (alice_sk, _), _bob) = setup("clonesize"); + repo.branch_private("fix", &alice_sk, &[], &[]).unwrap(); + repo.switch_with_identity("fix", Some(&alice_sk)).unwrap(); + write(&root, "src/private.rs", "sealed\n"); + repo.commit_private("alice", "c1", Some(&alice_sk)).unwrap(); + repo.switch_with_identity("main", Some(&alice_sk)).unwrap(); + drop(repo); + + let part_dst = tmp_root("clonesize-part"); + let part = Repo::clone_url_limited( + &root.display().to_string(), + &part_dst, + &[], + Some(1024), + false, + ) + .unwrap(); + let names: Vec = part + .branches() + .unwrap() + .into_iter() + .map(|(n, _)| n) + .collect(); + assert!( + !names.contains(&"fix".to_string()), + "a size-only clone must exclude private refs: {names:?}" + ); + assert!(names.contains(&"main".to_string())); + drop(part); + + std::fs::remove_dir_all(&root).unwrap(); + std::fs::remove_dir_all(&part_dst).unwrap(); + } + #[test] fn wire_pack_for_a_private_branch_carries_ciphertext_only() { let (root, repo, (alice_sk, _), _bob) = setup("wire"); diff --git a/crates/repo/src/promisor.rs b/crates/repo/src/promisor.rs index 754d67cf..47d2ee7f 100644 --- a/crates/repo/src/promisor.rs +++ b/crates/repo/src/promisor.rs @@ -27,35 +27,89 @@ use crate::error::{Error, Result}; use crate::layout::Layout; use crate::repo::Repo; -/// A partial clone's durable marker: the fetch-filter prefixes + the -/// promisor remote (origin URL) that can fill in objects outside the -/// filter on demand. Absent file = a full clone. -#[derive(Clone, Debug, Default, PartialEq, Eq)] +/// Which paths a partial clone asked for (ADR-0055 B1). A size-only clone +/// (`--filter blob:limit=` with no prefix) is in scope everywhere, which +/// an empty prefix list cannot say: that reads as "matches nothing". +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Scope { + /// Every path (a size-only clone). + AllPaths, + /// Only paths under these prefixes. + Prefixes(Vec), +} + +/// A partial clone's durable marker: the fetch-filter scope + the optional +/// blob-size limit + the promisor remote (origin URL) that can fill in objects +/// outside the filter on demand. Absent file = a full clone. +#[derive(Clone, Debug, PartialEq, Eq)] pub struct Promisor { - prefixes: Vec, + scope: Scope, + blob_limit: Option, pub origin: String, } +impl Default for Promisor { + fn default() -> Self { + Promisor { + scope: Scope::Prefixes(Vec::new()), + blob_limit: None, + origin: String::new(), + } + } +} + impl Promisor { /// Build a marker from an explicit origin + prefix list. pub fn new(origin: impl Into, prefixes: Vec) -> Self { Promisor { - prefixes, + scope: Scope::Prefixes(prefixes), + blob_limit: None, origin: origin.into(), } } + /// Build a marker for a clone made with a blob-size limit (ADR-0055). An + /// empty `prefixes` means every path is in scope ([`Scope::AllPaths`]). + pub fn with_blob_limit( + origin: impl Into, + prefixes: Vec, + blob_limit: u64, + ) -> Self { + Promisor { + scope: if prefixes.is_empty() { + Scope::AllPaths + } else { + Scope::Prefixes(prefixes) + }, + blob_limit: Some(blob_limit), + origin: origin.into(), + } + } + + /// The blob-size limit the clone was made with, if any: blobs of at least + /// this many bytes, reachable only through history, were left out. + pub fn blob_limit(&self) -> Option { + self.blob_limit + } + + /// True when every path is in scope (a size-only clone). + pub fn is_all_paths(&self) -> bool { + matches!(self.scope, Scope::AllPaths) + } + /// Whether `path` falls under any recorded prefix, at a `/` boundary — /// mirrors [`crate::sparse::Sparse::matches`] / `protect::matching_prefix`. - /// Unlike `Sparse`, an empty prefix list matches nothing (a promisor - /// marker with no prefixes would be a degenerate/empty partial clone, - /// not "everything" — callers gate on `Repo::promisor()` returning - /// `None` for "no filter at all"). + /// An empty prefix list matches nothing (a degenerate partial clone); + /// [`Scope::AllPaths`] matches everything. Callers gate on + /// `Repo::promisor()` returning `None` for "no filter at all". pub fn matches(&self, path: &str) -> bool { - self.prefixes.iter().any(|p| { - let bare = p.trim_end_matches('/'); - path == bare || path.starts_with(&format!("{bare}/")) - }) + match &self.scope { + Scope::AllPaths => true, + Scope::Prefixes(prefixes) => prefixes.iter().any(|p| { + let bare = p.trim_end_matches('/'); + path == bare || path.starts_with(&format!("{bare}/")) + }), + } } /// Should a tree walk descend into `path`? True if `path` is itself @@ -64,30 +118,128 @@ impl Promisor { /// content deeper in the tree). The empty path (the root) always /// descends, since every prefix lies under it. pub fn should_descend(&self, path: &str) -> bool { - path.is_empty() - || self.matches(path) - || self - .prefixes - .iter() - .any(|p| p.trim_end_matches('/').starts_with(&format!("{path}/"))) + match &self.scope { + Scope::AllPaths => true, + Scope::Prefixes(prefixes) => { + path.is_empty() + || self.matches(path) + || prefixes + .iter() + .any(|p| p.trim_end_matches('/').starts_with(&format!("{path}/"))) + } + } } - /// The recorded prefixes, in the order they were set. + /// The recorded prefixes, in the order they were set (empty for + /// [`Scope::AllPaths`]). pub fn prefixes(&self) -> &[String] { - &self.prefixes + match &self.scope { + Scope::AllPaths => &[], + Scope::Prefixes(prefixes) => prefixes, + } + } + + /// The filter as one line for messages: the prefixes, then `blob:limit=` + /// when the clone has a limit (`sc verify`, refusals). + pub fn describe(&self) -> String { + let mut parts: Vec = self.prefixes().to_vec(); + if let Some(limit) = self.blob_limit { + parts.push(format!("blob:limit={limit}")); + } + parts.join(", ") } /// Widen the filter to also include `prefixes` (backfill, Task 4). - /// Dedups against the existing set; order is existing-then-new. + /// Dedups against the existing set; order is existing-then-new. An + /// all-paths scope stays all-paths (nothing is wider). pub fn widen(&mut self, prefixes: &[String]) { - for p in prefixes { - if !self.prefixes.iter().any(|existing| existing == p) { - self.prefixes.push(p.clone()); + if let Scope::Prefixes(existing) = &mut self.scope { + for p in prefixes { + if !existing.iter().any(|e| e == p) { + existing.push(p.clone()); + } } } } } +/// The filter a `GetPack` carries (ADR-0037, ADR-0055): path prefixes and an +/// optional blob-size limit. The one place that decides whether a transfer is +/// filtered: a value exists only when it is active (`!prefixes.is_empty() || +/// blob_limit.is_some()`), so every `Option<&PackFilter>` that is `Some` means +/// "partial transfer" on the sender, the bucket refusal and the client's +/// private-ref handling alike. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PackFilter { + prefixes: Vec, + blob_limit: Option, +} + +impl PackFilter { + /// `None` when neither a prefix nor a limit is given (a full transfer). + pub fn new(prefixes: &[String], blob_limit: Option) -> Option { + (!prefixes.is_empty() || blob_limit.is_some()).then(|| PackFilter { + prefixes: prefixes.to_vec(), + blob_limit, + }) + } + + /// The path prefixes; empty means every path. + pub fn prefixes(&self) -> &[String] { + &self.prefixes + } + + /// Blobs of at least this many bytes that are not reachable from a want's + /// own root tree are left out. + pub fn blob_limit(&self) -> Option { + self.blob_limit + } + + /// The path scope as a [`Promisor`] the reachability walk can ask. + pub(crate) fn scope(&self) -> Promisor { + match self.blob_limit { + Some(limit) => Promisor::with_blob_limit(String::new(), self.prefixes.clone(), limit), + None => Promisor::new(String::new(), self.prefixes.clone()), + } + } +} + +/// Split `sc clone --filter` values into path prefixes and the optional blob +/// limit (ADR-0055). A value starting with `blob:` is the reserved size +/// grammar, never a prefix: only `blob:limit=` is accepted, once. A +/// path prefix that literally begins with `blob:` cannot be named here. +pub fn parse_clone_filters(values: &[String]) -> Result<(Vec, Option)> { + let mut prefixes = Vec::new(); + let mut limit = None; + for v in values { + let Some(spec) = v.strip_prefix("blob:") else { + prefixes.push(v.clone()); + continue; + }; + if spec == "none" { + return Err(Error::InvalidArgument( + "unsupported filter `blob:none`; use `blob:limit=0` to leave out every blob \ + that no ref tip reaches" + .into(), + )); + } + let Some(size) = spec.strip_prefix("limit=") else { + return Err(Error::InvalidArgument(format!( + "unsupported filter `{v}`; the only blob filter is `blob:limit=` \ + (`blob:` is reserved and cannot start a path prefix)" + ))); + }; + if limit.is_some() { + return Err(Error::InvalidArgument( + "`blob:limit` given more than once; pass a single `--filter blob:limit=`" + .into(), + )); + } + limit = Some(scl_core::parse_byte_size(size).map_err(Error::InvalidArgument)?); + } + Ok((prefixes, limit)) +} + /// Build the "outside this partial clone's fetch filter" error for `path` /// (P27 Task 5) — used by the sparse-widen preflight /// (`set_sparse`/`disable_sparse`), where `path` is a real prefix `sc @@ -96,6 +248,22 @@ pub fn partial_gap_hint(path: &str) -> Error { Error::GapOutsideFilter(path.to_string()) } +/// The gap error for `path` on this partial clone: [`Error::GapBySize`] when +/// the clone has a blob-size limit and `path` is in its scope (the file's own +/// path is what `sc backfill` repairs), otherwise the prefix hint. +pub fn gap_error(promisor: Option<&Promisor>, path: &str) -> Error { + match promisor { + Some(p) if !path.is_empty() && p.matches(path) => match p.blob_limit() { + Some(limit) => Error::GapBySize { + path: path.to_string(), + limit, + }, + None => partial_gap_hint(path), + }, + _ => partial_gap_hint(path), + } +} + /// Build the "this operation isn't supported on a partial clone at all" /// error (P27 Task 5 review fix): merge, cherry-pick/rebase replay, `sc ws /// harvest`, and `sc work` all need the FULL tree of whatever they touch, @@ -109,7 +277,9 @@ pub fn partial_clone_unsupported(op: &str) -> Error { } /// Load the repo's promisor marker. An absent `.sc/promisor` file reads as -/// `None` — a full clone, not an empty filter. +/// `None` — a full clone, not an empty filter. `.sc/promisor-blob-limit` +/// (ADR-0055) is read only when `.sc/promisor` exists: a stray limit file +/// without it is leftover state and is ignored. pub fn load(layout: &Layout) -> Result> { match std::fs::read_to_string(layout.promisor_path()) { Ok(text) => { @@ -129,18 +299,74 @@ pub fn load(layout: &Layout) -> Result> { )) } }; - let prefixes = lines.map(|l| l.to_string()).collect(); - Ok(Some(Promisor { prefixes, origin })) + let prefixes: Vec = lines.map(|l| l.to_string()).collect(); + let (blob_limit, all_paths) = load_limit(layout)?; + let scope = if all_paths { + Scope::AllPaths + } else { + Scope::Prefixes(prefixes) + }; + Ok(Some(Promisor { + scope, + blob_limit, + origin, + })) } Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), Err(e) => Err(e.into()), } } -/// Persist the repo's promisor marker, atomically. +/// Parse `.sc/promisor-blob-limit`: `limit ` then `scope all-paths` or +/// `scope prefixes`. Absent file = no limit and prefix scope. +fn load_limit(layout: &Layout) -> Result<(Option, bool)> { + let text = match std::fs::read_to_string(layout.promisor_blob_limit_path()) { + Ok(t) => t, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((None, false)), + Err(e) => return Err(e.into()), + }; + let bad = || { + Error::InvalidArgument( + "malformed .sc/promisor-blob-limit: expected 'limit ' then \ + 'scope all-paths|prefixes'" + .into(), + ) + }; + let mut lines = text.lines(); + let limit = lines + .next() + .and_then(|l| l.strip_prefix("limit ")) + .and_then(|n| n.parse::().ok()) + .ok_or_else(bad)?; + let all_paths = match lines.next() { + Some("scope all-paths") => true, + Some("scope prefixes") => false, + _ => return Err(bad()), + }; + Ok((Some(limit), all_paths)) +} + +/// Persist the repo's promisor marker, atomically. The limit file goes first: +/// a crash between the two writes leaves a limit file with no marker (ignored +/// by [`load`]) rather than a marker that has lost its limit. pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { + match p.blob_limit { + Some(limit) => { + let scope = if p.is_all_paths() { + "all-paths" + } else { + "prefixes" + }; + let text = format!("limit {limit}\nscope {scope}\n"); + scl_core::fsutil::atomic_write_durable( + &layout.promisor_blob_limit_path(), + text.as_bytes(), + )?; + } + None => remove_file_if_exists(&layout.promisor_blob_limit_path())?, + } let mut text = format!("origin {}\n", p.origin); - for prefix in &p.prefixes { + for prefix in p.prefixes() { text.push_str(prefix); text.push('\n'); } @@ -148,6 +374,14 @@ pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { Ok(()) } +fn remove_file_if_exists(path: &std::path::Path) -> Result<()> { + match std::fs::remove_file(path) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), + } +} + /// Remove the repo's promisor marker: this repo becomes a genuine full /// clone (P27 final review I2, `Repo::backfill_all`'s escape hatch). A /// no-op if the marker is already absent. Callers MUST only call this @@ -155,11 +389,8 @@ pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { /// complete — this function does not itself verify anything. pub fn remove(layout: &Layout) -> Result<()> { set_lazy(layout, false)?; - match std::fs::remove_file(layout.promisor_path()) { - Ok(()) => Ok(()), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), - Err(e) => Err(e.into()), - } + remove_file_if_exists(&layout.promisor_blob_limit_path())?; + remove_file_if_exists(&layout.promisor_path()) } /// True when this partial clone has opted in to filling gaps by object id diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index 9cf965f9..44f6efe8 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -2,7 +2,7 @@ //! snapshot tips. Parameterized by an `ObjectSource` so it walks either the //! local `Store` (push) or a remote `Transport` (clone/fetch). Reused by P8 gc. -use std::collections::{BTreeMap, BTreeSet, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}; use scl_core::{EntryKind, Object, ObjectId, Store}; @@ -12,12 +12,28 @@ use crate::transport::Transport; /// Minimal read access the walk needs: fetch a decoded object by id. pub trait ObjectSource { fn get(&mut self, id: &ObjectId) -> Result; + + /// Byte length of the blob `id`. The default loads the object; a source + /// that can measure without admitting the blob to a cache overrides it. + fn blob_size(&mut self, id: &ObjectId) -> Result { + Ok(match self.get(id)? { + Object::Blob(b) => b.len() as u64, + _ => return Err(Error::Core(scl_core::Error::WrongKind(*id, "blob"))), + }) + } } impl ObjectSource for Store { fn get(&mut self, id: &ObjectId) -> Result { Ok(Store::get(self, id)?) } + + /// Streams the payload through a fixed buffer and checks its hash, so + /// measuring a large blob neither fills the object cache nor trusts + /// unverified bytes (ADR-0055). + fn blob_size(&mut self, id: &ObjectId) -> Result { + Ok(Store::blob_size(self, id)? as u64) + } } /// An `ObjectSource` backed by a remote `Transport`. @@ -121,8 +137,35 @@ pub fn reachable_objects_filtered( src: &mut impl ObjectSource, tips: &[ObjectId], filter: Option<&dyn PrefixFilter>, +) -> Result { + reachable_objects_limited(src, tips, filter, None) +} + +/// [`reachable_objects_filtered`] plus the blob-size limit (ADR-0055). With +/// `blob_limit = Some(n)`, a blob of at least `n` bytes goes to `gaps` instead +/// of `included`, unless it is reachable from the root tree of one of `tips` +/// (the wants): every tip is complete, so the limit only prunes blobs that are +/// reachable through parent snapshots alone. `blob_limit` needs a `filter` to +/// say which paths are in scope (use an all-paths one for a size-only walk). +/// With `None` this is `reachable_objects_filtered` exactly. +/// +/// Ordering: `walk_snapshots` queues every tip before any parent, so a tip's +/// exempt walk always runs before a non-tip snapshot's limited one, even when +/// one tip is an ancestor of another. The `(id, path)` expansion gate then only +/// ever lets a limited walk skip work an exempt walk already did (a superset), +/// never the reverse; and a tip root tree that equals an older subtree is still +/// expanded at its own `(root, "")` key. +pub fn reachable_objects_limited( + src: &mut impl ObjectSource, + tips: &[ObjectId], + filter: Option<&dyn PrefixFilter>, + blob_limit: Option, ) -> Result { let mut gaps = BTreeSet::new(); + let tip_set: HashSet = tips.iter().copied().collect(); + // Size verdicts by blob id: a blob shared by many paths or versions is + // measured once per request (ADR-0055). + let mut sizes: HashMap = HashMap::new(); // (id, path) pairs already pushed for expansion, under a filter only. // Hoisted to span EVERY snapshot's walk in this call (not just one), so // a later snapshot whose ROOT tree dedups to an already-expanded @@ -131,8 +174,21 @@ pub fn reachable_objects_filtered( // comment for why a bare-id gate on the root is wrong under a filter // (C1, P27 final review). let mut expanded: HashSet<(ObjectId, String)> = HashSet::new(); - let mut included = walk_snapshots(src, tips, |src, _snapshot, root, included| { - walk_tree_filtered(src, root, included, &mut gaps, filter, &mut expanded) + let mut included = walk_snapshots(src, tips, |src, snapshot, root, included| { + let mut gate = blob_limit.map(|limit| SizeGate { + limit, + exempt: tip_set.contains(&snapshot), + verdicts: &mut sizes, + }); + walk_tree_sized( + src, + root, + included, + &mut gaps, + filter, + &mut expanded, + gate.as_mut(), + ) })?; // A byte-identical subtree can be gapped at one path and included at // another (same id, different filter verdict per path) — included @@ -363,6 +419,41 @@ fn walk_tree_filtered( gaps: &mut BTreeSet, filter: Option<&dyn PrefixFilter>, expanded: &mut HashSet<(ObjectId, String)>, +) -> Result<()> { + walk_tree_sized(src, root, included, gaps, filter, expanded, None) +} + +/// The blob-size rule for one snapshot's tree walk (ADR-0055): `exempt` is true +/// for a want's own walk, where every in-scope blob is kept; otherwise a blob +/// of `limit` bytes or more is a gap. `verdicts` caches "keep?" by blob id. +struct SizeGate<'a> { + limit: u64, + exempt: bool, + verdicts: &'a mut HashMap, +} + +impl SizeGate<'_> { + fn keeps(&mut self, src: &mut impl ObjectSource, id: &ObjectId) -> Result { + if self.exempt { + return Ok(true); + } + if let Some(keep) = self.verdicts.get(id) { + return Ok(*keep); + } + let keep = src.blob_size(id)? < self.limit; + self.verdicts.insert(*id, keep); + Ok(keep) + } +} + +fn walk_tree_sized( + src: &mut impl ObjectSource, + root: ObjectId, + included: &mut BTreeSet, + gaps: &mut BTreeSet, + filter: Option<&dyn PrefixFilter>, + expanded: &mut HashSet<(ObjectId, String)>, + mut size: Option<&mut SizeGate<'_>>, ) -> Result<()> { let mut stack: Vec<(ObjectId, String)> = Vec::new(); match filter { @@ -395,7 +486,15 @@ fn walk_tree_filtered( included.insert(e.id); } Some(f) => { - if f.matches(&child_path) { + let keep = f.matches(&child_path) + && match size.as_deref_mut() { + // Already kept via an exempt walk: no need to measure. + Some(gate) if !included.contains(&e.id) => { + gate.keeps(src, &e.id)? + } + _ => true, + }; + if keep { included.insert(e.id); } else { gaps.insert(e.id); diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index f6bee3a2..bc3b0c1c 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -343,7 +343,7 @@ impl Repo { "cannot set an empty sparse spec (a full checkout) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full \ clone first, or name a prefix within it", - promisor.prefixes().join(", ") + promisor.describe() ))); } // A lazy clone may widen the view past its fetch filter: the @@ -747,7 +747,7 @@ impl Repo { "cannot disable sparse checkout (full materialization) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full clone \ first, or `sc sparse set` to a prefix within it", - promisor.prefixes().join(", ") + promisor.describe() ))); } let dirty = self.status()?; diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 965caf0e..3b171187 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -160,7 +160,7 @@ impl Transport for WireClient { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { let mut rw = self.rw.borrow_mut(); @@ -169,7 +169,8 @@ impl Transport for WireClient { &Request::GetPack { wants: wants.to_vec(), haves: haves.to_vec(), - filter: filter.map(|f| f.to_vec()).unwrap_or_default(), + filter: filter.map(|f| f.prefixes().to_vec()).unwrap_or_default(), + blob_limit: filter.and_then(|f| f.blob_limit()), } .encode(), )?; @@ -346,7 +347,7 @@ impl Transport for StdioTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.client.get_pack(wants, haves, filter, out) @@ -655,7 +656,12 @@ mod tests { let mut pack = Vec::new(); client - .get_pack(&[tip], &[], Some(&["src/".to_string()]), &mut pack) + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&["src/".to_string()], None).as_ref(), + &mut pack, + ) .unwrap(); let ids: Vec<_> = scl_core::pack::parse_pack(&pack) .unwrap() @@ -693,21 +699,28 @@ mod tests { } #[test] - fn handshake_rejects_v4_peer() { - // P37 bumped PROTOCOL_VERSION 4 -> 5 for the additive `GetObjects` - // verb: a v4 server cannot decode it, so refuse at the handshake. + fn handshake_rejects_v5_peer() { + // ADR-0055 bumped PROTOCOL_VERSION 5 -> 6: `GetPack` gained the + // trailing `blob_limit`, so a v5 server cannot decode it. Refuse at + // the handshake and name both versions. let (client_read, mut server_write) = std::io::pipe().unwrap(); let (mut server_read, client_write) = std::io::pipe().unwrap(); let server = std::thread::spawn(move || { let f = wire::read_frame(&mut server_read).unwrap(); assert!(matches!( wire::Request::decode(&f).unwrap(), - wire::Request::Hello { version: 5 } + wire::Request::Hello { version: 6 } )); - wire::write_ok(&mut server_write, &wire::u32_body(4)).unwrap(); + wire::write_ok(&mut server_write, &wire::u32_body(5)).unwrap(); }); let err = WireClient::handshake(client_read, client_write).unwrap_err(); - assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + match &err { + Error::Protocol(m) => assert_eq!( + m, "server speaks protocol 5, this client speaks 6", + "got {m}" + ), + other => panic!("expected a protocol error, got {other:?}"), + } server.join().unwrap(); } @@ -1002,6 +1015,7 @@ mod tests { wants: vec![tip], haves: vec![], filter: vec![], + blob_limit: None, } .encode(), ) diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 75cdc350..1989fdba 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -96,12 +96,28 @@ impl Repo { dst: impl AsRef, filter: Option<&[String]>, no_transcripts: bool, + ) -> Result { + Self::clone_url_limited(src_url, dst, filter.unwrap_or(&[]), None, no_transcripts) + } + + /// [`Repo::clone_url_opts`] with the blob-size limit (ADR-0055): the + /// clone is partial when `prefixes` is non-empty or `blob_limit` is set. + /// With a limit and no prefix every path is in scope (`Scope::AllPaths`) + /// and the sparse view stays full. The server leaves out blobs of at least + /// `blob_limit` bytes that no ref tip's root tree reaches, so the initial + /// checkout is always complete. + pub fn clone_url_limited( + src_url: &str, + dst: impl AsRef, + prefixes: &[String], + blob_limit: Option, + no_transcripts: bool, ) -> Result { // Before the network or the destination directory: a blank prefix // would be persisted as a sparse line that reloads as a full view. - if let Some(f) = filter { - crate::sparse::validate_prefixes(f)?; - } + crate::sparse::validate_prefixes(prefixes)?; + let pack_filter = crate::promisor::PackFilter::new(prefixes, blob_limit); + let filter = pack_filter.as_ref(); let transport = open_transport(src_url)?; let remote_refs = transport.list_refs()?; let head_branch = transport.head_branch()?; @@ -177,9 +193,17 @@ impl Repo { // doesn't transfer a pre-existing `.sc/sparse` from the source (out // of P24 scope, sparse config is local like `.scignore`). let sparse_spec = match filter { - Some(prefixes) => { - let promisor = crate::promisor::Promisor::new(src_url, prefixes.to_vec()); + Some(_) => { + let promisor = match blob_limit { + Some(limit) => crate::promisor::Promisor::with_blob_limit( + src_url, + prefixes.to_vec(), + limit, + ), + None => crate::promisor::Promisor::new(src_url, prefixes.to_vec()), + }; crate::promisor::store(&dst_repo.layout, &promisor)?; + // A size-only clone has no prefixes: the view stays full. let sparse = crate::sparse::Sparse::new(prefixes.to_vec()); crate::sparse::store(&dst_repo.layout, &sparse)?; sparse @@ -284,7 +308,9 @@ impl Repo { &mut store, &tips, &[], - Some(prefixes), + // Backfill never applies the clone's blob limit: it fetches + // every version of the named paths (ADR-0055). + crate::promisor::PackFilter::new(prefixes, None).as_ref(), )?; } @@ -611,7 +637,7 @@ fn transfer_objects( store: &mut Store, tips: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result<()> { transfer_objects_opts(layout, transport, store, tips, haves, filter, false) } @@ -625,7 +651,7 @@ fn transfer_objects_opts( store: &mut Store, tips: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, no_transcripts: bool, ) -> Result<()> { let guard = crate::transport::TempPackGuard::new(layout)?; diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 1c4e6eaf..2ab4b4ec 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -47,16 +47,19 @@ pub trait Transport { /// `filter`, when `Some`, narrows the sender's `wants`-side walk to the /// given path prefixes (P27 Task 3) — the wire form of a partial clone's /// `.sc/promisor` prefix list, matched with the same `/`-boundary - /// discipline as `Promisor::matches`/`should_descend`. The `haves`-side - /// walk is always unfiltered (the receiver's full closure is excluded - /// regardless of filter, since a filter only narrows what's sent, never - /// widens what's assumed already held). `filter = None` reproduces the - /// pre-P27 full-transfer behavior byte-for-byte. + /// discipline as `Promisor::matches`/`should_descend` — and/or leaves out + /// blobs of at least `blob_limit` bytes that no want's own root tree + /// reaches (ADR-0055; no prefixes means every path). A `PackFilter` exists + /// only when it is active, so `Some` always means a partial transfer. The + /// `haves`-side walk is always unfiltered (the receiver's full closure is + /// excluded regardless of filter, since a filter only narrows what's sent, + /// never widens what's assumed already held). `filter = None` reproduces + /// the pre-P27 full-transfer behavior byte-for-byte. fn get_pack( &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()>; @@ -238,14 +241,13 @@ impl LocalTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result { use std::collections::BTreeSet; let mut store = self.store.borrow_mut(); // Reachable-from-wants minus reachable-from-haves, computed on this // (the remote) store. `haves` the remote doesn't have are skipped. - let promisor_filter = - filter.map(|prefixes| crate::promisor::Promisor::new(String::new(), prefixes.to_vec())); + let promisor_filter = filter.map(|f| f.scope()); // Filtered transfers exclude private branches entirely (ADR-0044): // a want that resolves to a branch manifest is dropped before the // walk. Serves both transports — the wire server's GetPack delegates @@ -254,10 +256,11 @@ impl LocalTransport { crate::reachable::filter_manifest_wants(&mut *store, wants, promisor_filter.is_some())?; let mut want_set = match &promisor_filter { Some(pf) => { - crate::reachable::reachable_objects_filtered( + crate::reachable::reachable_objects_limited( &mut *store, &wants, Some(pf as &dyn crate::reachable::PrefixFilter), + filter.and_then(|f| f.blob_limit()), )? .included } @@ -726,7 +729,7 @@ impl Transport for LocalTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { let guard = self.build_pack_tempfile(wants, haves, filter)?; @@ -1116,8 +1119,13 @@ mod tests { other => panic!("expected snapshot, got {other:?}"), }; let mut pack = Vec::new(); - t.get_pack(&[tip], &[], Some(&["src/".to_string()]), &mut pack) - .unwrap(); + t.get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&["src/".to_string()], None).as_ref(), + &mut pack, + ) + .unwrap(); let _ = crate::repo::Repo::init(&dst_root).unwrap(); let t2 = LocalTransport::open(&dst_root).unwrap(); diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index a6c454cb..83eb78de 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -28,8 +28,13 @@ use crate::error::{Error, Result}; /// Bumped 4 -> 5 in P37 (ADR-0047): the additive `GetObjects` verb (batched /// by-id object fetch) — a v4 server would answer its opcode with an /// "unknown opcode" protocol error mid-session, so the mismatch is refused at -/// the handshake instead. -pub const PROTOCOL_VERSION: u32 = 5; +/// the handshake instead. Bumped 5 -> 6 in ADR-0055: `Request::GetPack` gains a +/// trailing optional `blob_limit` (the `--filter blob:limit=` clone +/// filter, evaluated by the server because only it knows a blob's size). A v5 +/// peer would reject the longer frame as trailing bytes, or worse, a v6 limit +/// riding in the prefix list would read as a path, so the mismatch is refused +/// at the handshake in both directions instead. +pub const PROTOCOL_VERSION: u32 = 6; // Request opcodes (one per Transport verb, plus session control). const OP_HELLO: u8 = 0x01; @@ -133,12 +138,16 @@ pub enum Request { id: ObjectId, expected_old: Option, }, - /// `filter`: partial-clone prefix list (P27 Task 3), empty = no filter - /// (full transfer, matching pre-P27 behavior byte-for-byte). + /// `filter`: partial-clone prefix list (P27 Task 3), empty = no prefix + /// filter. `blob_limit` (ADR-0055): leave out blobs of at least that many + /// bytes that no want's own root tree reaches. The transfer is filtered + /// when `filter` is non-empty or `blob_limit` is set; with neither it is a + /// full transfer, matching pre-P27 behavior byte-for-byte. GetPack { wants: Vec, haves: Vec, filter: Vec, + blob_limit: Option, }, /// Marker only (P25) — the pack body is no longer embedded in this /// request's frame. Immediately after this request frame, the sender @@ -218,6 +227,9 @@ impl<'a> Cur<'a> { fn u32(&mut self) -> Result { Ok(u32::from_be_bytes(self.take(4)?.try_into().unwrap())) } + fn u64(&mut self) -> Result { + Ok(u64::from_be_bytes(self.take(8)?.try_into().unwrap())) + } fn id(&mut self) -> Result { Ok(ObjectId::from_bytes(self.take(32)?.try_into().unwrap())) } @@ -319,11 +331,19 @@ impl Request { wants, haves, filter, + blob_limit, } => { out.push(OP_GET_PACK); put_ids(&mut out, wants); put_ids(&mut out, haves); put_strs(&mut out, filter); + match blob_limit { + Some(limit) => { + out.push(1); + out.extend_from_slice(&limit.to_be_bytes()); + } + None => out.push(0), + } } Request::PutPack => out.push(OP_PUT_PACK), Request::GetObjects { ids } => { @@ -366,6 +386,11 @@ impl Request { wants: c.ids()?, haves: c.ids()?, filter: c.strs()?, + blob_limit: match c.u8()? { + 0 => None, + 1 => Some(c.u64()?), + t => return Err(Error::Protocol(format!("bad blob_limit tag {t}"))), + }, }, OP_PUT_PACK => Request::PutPack, OP_GET_OBJECTS => Request::GetObjects { @@ -1012,7 +1037,7 @@ fn bucket_get_pack_tempfile( tmp_dir: &std::path::Path, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result { let guard = TempPackGuard::new_in(tmp_dir)?; let mut f = std::fs::File::create(guard.path())?; @@ -1211,16 +1236,15 @@ fn serve_session( wants, haves, filter, + blob_limit, } => { - let filter_opt = if filter.is_empty() { - None - } else { - Some(filter.as_slice()) - }; + let filter_opt = crate::promisor::PackFilter::new(&filter, blob_limit); let result = match &transport { - ServeTransport::Local(t) => t.build_pack_tempfile(&wants, &haves, filter_opt), + ServeTransport::Local(t) => { + t.build_pack_tempfile(&wants, &haves, filter_opt.as_ref()) + } ServeTransport::Bucket { transport: bt, tmp } => { - bucket_get_pack_tempfile(bt, &tmp.0, &wants, &haves, filter_opt) + bucket_get_pack_tempfile(bt, &tmp.0, &wants, &haves, filter_opt.as_ref()) } }; match result { @@ -1529,16 +1553,37 @@ mod tests { wants: vec![some_id(7)], haves: vec![some_id(8), some_id(9)], filter: vec![], + blob_limit: None, }, Request::GetPack { wants: vec![some_id(7)], haves: vec![], filter: vec!["src/".to_string(), "docs/".to_string()], + blob_limit: None, + }, + Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: None, + }, + Request::GetPack { + wants: vec![some_id(7)], + haves: vec![], + filter: vec!["src/".to_string()], + blob_limit: Some(1 << 20), + }, + Request::GetPack { + wants: vec![some_id(7)], + haves: vec![], + filter: vec![], + blob_limit: Some(0), }, Request::GetPack { wants: vec![], haves: vec![], filter: vec![], + blob_limit: Some(u64::MAX), }, Request::PutPack, Request::GetObjects { ids: vec![] }, @@ -1851,6 +1896,93 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + /// ADR-0055: a v5 client (no `blob_limit` field) is refused at the + /// handshake with a message naming both versions, so it never sends a + /// `GetPack` the v6 decoder would misparse. + #[test] + fn serve_rejects_a_v5_client_naming_both_versions() { + let root = tmp_repo("v5client"); + let r = run_session(&root, &[Request::Hello { version: 5 }]); + match &r[0] { + Err(Error::Protocol(m)) => assert_eq!( + m, "unsupported protocol version 5 (server speaks 6)", + "got {m}" + ), + other => panic!("expected a protocol error, got {other:?}"), + } + std::fs::remove_dir_all(&root).unwrap(); + } + + /// ADR-0055, the other direction: a v6 client against a v5 server sees the + /// v5 server's own refusal (its `handshake_hello` text with its version), + /// and no `GetPack` is ever sent. + #[test] + fn a_v6_client_reports_a_v5_servers_refusal() { + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let f = read_frame(&mut server_read).unwrap(); + assert!(matches!( + Request::decode(&f).unwrap(), + Request::Hello { version: 6 } + )); + write_err( + &mut server_write, + EC_PROTOCOL, + "unsupported protocol version 6 (server speaks 5)", + ) + .unwrap(); + }); + let Err(err) = crate::stdio_transport::WireClient::handshake(client_read, client_write) + else { + panic!("the handshake must fail"); + }; + assert_eq!( + err.to_string(), + "wire protocol error: unsupported protocol version 6 (server speaks 5)" + ); + server.join().unwrap(); + } + + /// The trailing `blob_limit` tag is strict: an unknown tag or a truncated + /// limit is a protocol error, not a silently unfiltered pack. + #[test] + fn get_pack_decode_rejects_a_bad_blob_limit_tag() { + let mut ok = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: Some(7), + } + .encode(); + assert!(Request::decode(&ok).is_ok()); + let tag_at = ok.len() - 9; + ok[tag_at] = 9; + assert!(matches!(Request::decode(&ok), Err(Error::Protocol(_)))); + let mut truncated = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: Some(7), + } + .encode(); + truncated.pop(); + assert!(matches!( + Request::decode(&truncated), + Err(Error::Protocol(_)) + )); + // A v5 frame (no trailing field) no longer decodes. + let mut v5 = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: None, + } + .encode(); + v5.pop(); + assert!(matches!(Request::decode(&v5), Err(Error::Protocol(_)))); + } + #[test] fn serve_reports_not_a_repo_as_typed_error() { let root = std::env::temp_dir().join(format!("scl-wire-norepo-{}", std::process::id())); @@ -2954,6 +3086,22 @@ mod tests { let got = scl_core::pack::parse_pack(&out).unwrap(); assert_eq!(got.len(), objects.len()); + // A served `--store` bucket refuses a partial transfer of either kind + // (ADR-0046, ADR-0055 B2): a size-only request is "filtered" too. + for filter in [ + crate::promisor::PackFilter::new(&["src/".to_string()], None), + crate::promisor::PackFilter::new(&[], Some(1024)), + ] { + let err = client + .get_pack(&[tip], &[], filter.as_ref(), &mut Vec::new()) + .unwrap_err(); + assert!( + err.to_string() + .contains("partial clone from bucket remotes"), + "{err}" + ); + } + // PutPack + UpdateRef land a second commit through the same session. let (tip2, objects2) = tiny_history_for_bucket("seed2", b"second commit"); let (pack2, _idx2) = scl_core::pack::build_pack(&objects2).unwrap(); diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index d1c31169..6e171dc6 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -735,20 +735,26 @@ pub(crate) fn preflight_view( return Ok(()); }; let gap = |path: &str| { - let prefix = promisor - .prefixes() - .iter() - .find(|p| { - let bare = p.trim_end_matches('/'); - path == bare || path.starts_with(&format!("{bare}/")) - }) - .cloned() - // Outside the filter (a view widened past it): name the path - // itself, which `sc backfill` accepts as a prefix. For the root - // tree there is no path, so fall back to the first filter prefix. - .or_else(|| (!path.is_empty()).then(|| path.to_string())) - .or_else(|| promisor.prefixes().first().cloned()) - .unwrap_or_default(); + // With a blob limit the missing object is a file, and `sc backfill` + // repairs it by its own path rather than a covering prefix (ADR-0055). + let prefix = if promisor.blob_limit().is_some() && !path.is_empty() { + path.to_string() + } else { + promisor + .prefixes() + .iter() + .find(|p| { + let bare = p.trim_end_matches('/'); + path == bare || path.starts_with(&format!("{bare}/")) + }) + .cloned() + // Outside the filter (a view widened past it): name the path + // itself, which `sc backfill` accepts as a prefix. For the root + // tree there is no path, so fall back to the first filter prefix. + .or_else(|| (!path.is_empty()).then(|| path.to_string())) + .or_else(|| promisor.prefixes().first().cloned()) + .unwrap_or_default() + }; Error::GapInView { path: if path.is_empty() { "(root tree)".to_string() @@ -756,6 +762,7 @@ pub(crate) fn preflight_view( path.to_string() }, prefix, + blob_limit: promisor.blob_limit(), } }; if !store.contains(&root) { From 044312307925472482aeec29f65343bd902d5cda Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:23:51 +0300 Subject: [PATCH 3/7] docs: ADR-0055 accepted, partial-clone docs and blob-limit demo (ref #217) ADR-0055 records the accepted design and the review amendments. Updates ADR-0037/0048, CLAUDE.md, ROADMAP, ARCHITECTURE, THREAT-MODEL, the sc-cli-reference skill and README, and adds demo/run_blob_limit_demo.sh. Refs #217 --- .claude/skills/sc-cli-reference/SKILL.md | 16 +- ARCHITECTURE.md | 21 +- CLAUDE.md | 18 +- README.md | 4 +- ROADMAP.md | 20 +- demo/run_blob_limit_demo.sh | 93 ++++ docs/THREAT-MODEL.md | 16 + docs/adr/0037-partial-clone.md | 7 +- .../adr/0048-presence-based-partial-clones.md | 18 +- docs/adr/0055-blob-size-clone-filter.md | 406 ++++++++++-------- docs/adr/README.md | 2 +- 11 files changed, 430 insertions(+), 191 deletions(-) create mode 100755 demo/run_blob_limit_demo.sh diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 3e93947c..fc7829f4 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -371,6 +371,17 @@ cargo run --bin sc -- clone --filter # partial clone ( # prefixes; writes .sc/promisor + .sc/sparse to # the same prefixes; not supported over a # git-bridge remote +cargo run --bin sc -- clone --filter blob:limit= [--filter ] [--lazy] + # blob-size partial clone (ADR-0055, #217): + # is bytes or a KiB/MiB/GiB suffix; blobs of + # AT LEAST that size that no ref tip reaches are + # left out (old revisions of large files), every tip + # and the checkout stay complete; blob:limit=0 = no + # blobs beyond the tips. `blob:` is reserved in + # --filter (blob:none is an error), once only. + # Server-evaluated: wire v6, upgrade both sides. + # Repair: `sc backfill `, a lazy fill, or + # `backfill --all`; fetch is not limit-aware cargo run --bin sc -- backfill # widen a partial clone: fetch objects under # these prefixes from the promisor origin and # widen .sc/promisor; errors if this repo isn't @@ -397,7 +408,10 @@ bash demo/run_lazy_fetch_demo.sh # by-id fetch proof (P37a): cat of # exactly 3 objects (2 trees + blob), the sibling # stays unfetched, offline/non-lazy keep the gap # error, and a read-only sc+http server answers - # GetObjects (protocol v5) + # GetObjects +bash demo/run_blob_limit_demo.sh # blob-size filter proof (ADR-0055): old large + # blob left out, tip checkout whole, verify/gc/ + # push fine, --lazy, prefix+limit, backfill repair bash demo/run_partial_clone_demo.sh # partial clone proof (P27): sc clone --filter # src/ fetches fewer objects than a full clone # (object-store count + sc verify's gap report); diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 8ea729d1..de319e10 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -338,7 +338,8 @@ phases but the shape is stable: `transcript-drops` — the derived set of dropped transcripts (`sc transcript drop`), rebuilt from the store on clone. - `sparse` — the local, uncommitted sparse-checkout prefix spec (P24); - `promisor` — a partial clone's fetch-filter + origin (P27). + `promisor` — a partial clone's fetch-filter + origin (P27); + `promisor-blob-limit` — its blob-size limit and whole-tree-or-prefix scope (ADR-0055). - `serve-tokens.toml` — bearer-token access control for `sc serve --http` (P29); tokens may carry an expiry and be rotated in place (`sc serve token add --expires` / `rotate`). `serve-tls/` — the auto-minted `sc+https://` server @@ -1216,6 +1217,24 @@ by id (`ensure_view`) before switch, fast-forward, undo and `sparse set`; object cap; the client reads at most 4 GiB) and split when refused. `sc export` and `sparse disable` stay refused on a partial clone. See ADR-0048. +## Phase 27 follow-on — blob-size clone filter (built) + +`sc clone --filter blob:limit=` (alone or with prefixes) leaves out blobs +of at least `` bytes that no ref tip's root tree reaches. Only the server +knows a blob's size before sending it (tree entries carry none), so +`GetPack` gained a trailing optional `blob_limit` and `PROTOCOL_VERSION` is 6. +The sender's walk, `reachable_objects_limited`, walks every want's own tree +first (every ref tip stays complete, so the initial checkout is whole), then +moves over-limit blobs reached only through history from `included` to `gaps`, +measuring each distinct blob once with `Store::blob_size`. The client side adds +nothing to the gap model: absence is the gap (37b). `.sc/promisor-blob-limit` +records the limit and an explicit `AllPaths`/`Prefixes` scope (a size-only clone +is `AllPaths`, with a full sparse view); `PackFilter` is the single definition of +a filtered transfer (a prefix or a limit). A lazy size-only clone fills a gapped +revision by id even though its view is full; otherwise the gap error names the +file and `sc backfill ` repairs it. `sc fetch` is unfiltered, and the +object-count limit is deferred. See ADR-0055. + ## Phase 37c — merge, cherry-pick and rebase on a partial clone (built) `merge_trees` is a tree-recursive three-way merge that carries a subtree **by diff --git a/CLAUDE.md b/CLAUDE.md index bb2bde70..cf4481f8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -142,7 +142,7 @@ the code, those win. | P24 | Sparse checkouts: `.sc/sparse` local prefix spec; checkout-only (CAS keeps everything) | [0034](docs/adr/0034-sparse-checkouts.md) | | P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) | | P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) | -| P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | +| P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3. Blob-size filter (#217): `--filter blob:limit=` (combinable with prefixes; `blob:` is reserved) leaves out blobs of at least that size unless a ref tip reaches them, evaluated by the server (`GetPack.blob_limit`, wire v6); the gap is absence as in ADR-0048, `sc backfill ` or a lazy fill repairs it, `.sc/promisor-blob-limit` records limit and scope | [0037](docs/adr/0037-partial-clone.md), [0055](docs/adr/0055-blob-size-clone-filter.md) | | P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | | P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex | [0038](docs/adr/0038-agent-session-transcripts.md) | @@ -163,8 +163,11 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in `docs/THREAT-MODEL.md` — read it before touching anything crypto- or transport-adjacent. The rest, imperatively: -- **Wire protocol is version 5** (bumped from 4 in P37 for the additive - `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB; +- **Wire protocol is version 6** (bumped from 5 in #217 for `GetPack.blob_limit`, + ADR-0055; 4 to 5 in P37 for the additive `GetObjects` verb; 3 to 4 in P34 for + the sealed-branch object kinds). Servers and clients upgrade together: every + mixed pair fails closed at the handshake, ssh remotes included (they run the + remote host's own `sc` binary). The max object size (`scl_core::max_object_size()`, default 256 MiB; `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every @@ -191,6 +194,13 @@ transport-adjacent. The rest, imperatively: or `sparse set` fetches what its view needs by id first; on a non-lazy or offline clone it fails with `GapInView` (nothing changed) and names the `sc backfill ` that repairs it. + A `--filter blob:limit=` clone (ADR-0055) is partial too: a blob of at + least that size is left out unless a ref tip reaches it, tips are complete at + clone time only, a later fetch can bring one that points at a gapped blob, and + with a limit the gap errors name the file's own path (`GapBySize`, or + `GapInView` with the limit as a cause). A size-only clone has an `AllPaths` + scope and a full sparse view, and a lazy one still fills on switch, undo, merge + and replay. `sc fetch` is not limit-aware, and the object-count limit is deferred. - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. @@ -305,7 +315,7 @@ transport-adjacent. The rest, imperatively: Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of truth — do not re-list items here); entries that name an issue mirror the open GitHub issues. Standing entries: named/concurrent `sc ws` sessions (#215), -blob-size clone filters (#217), bucket compaction and gc (#216), the non-unix +a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), bucket compaction and gc (#216), the non-unix `.sc/local-key` ACL (#214), transcript-drop delivery gaps (#230), and the open sparse, private-branch and abort edge cases (#241, #246, #247, #248). diff --git a/README.md b/README.md index edb031f2..2076006a 100644 --- a/README.md +++ b/README.md @@ -169,8 +169,8 @@ via `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system`), bucket (`sc+wal://`, `sc+s3://`), and Git (including hosted GitHub) network transports, plus private branches sealed to a recipient set until publish and a read-only Tauri desktop browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds -27 independent end-to-end proofs, one per major capability. Remaining +28 independent end-to-end proofs, one per major capability. Remaining follow-ons (transcript drop delivery gaps, bucket compaction and gc, named -concurrent `sc ws` sessions, size/count clone filters, and similar hardening +concurrent `sc ws` sessions, a limit-aware fetch and an object-count clone limit, and similar hardening items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open GitHub issues rather than being open MVP scope. diff --git a/ROADMAP.md b/ROADMAP.md index c21a54c4..c30ab767 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -580,6 +580,16 @@ across every phase. carried even under a rule); conflicts inside the view are marked and resolved as on a full clone, conflicts outside it are refused before any write. (ADR-0049.) +- **Phase 27 follow-on — Blob-size clone filter (#217).** `sc clone --filter + blob:limit=` (combinable with prefix filters; `blob:` is reserved) leaves + out blobs of at least that size that no ref tip's root tree reaches, so old + revisions of large files stay on the server while every tip and the checkout + are complete. The server evaluates it (`GetPack.blob_limit`, `PROTOCOL_VERSION` + 6; client and server upgrade together). The gap is absence (ADR-0048), recorded + by `.sc/promisor-blob-limit` with an explicit `AllPaths`/`Prefixes` scope; a lazy + clone fills by id (a full view included), and otherwise the gap error names the + file and `sc backfill ` repairs it. Proven by `demo/run_blob_limit_demo.sh`. + (ADR-0055.) - **Phase 37d — Agent sessions on a partial clone.** `sc ws` and `sc work` work on a partial clone: a workspace sees the host's sparse view, and harvest lands through the ADR-0049 merge. (ADR-0050.) @@ -653,6 +663,7 @@ under Deferred below. | **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) | | **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) | +| **P27 follow-on — Blob-size clone filter (#217)** | Clone a repo with large binary history without fetching old revisions | `sc clone --filter blob:limit=1MiB ` (with or without prefixes) leaves out old large blobs, the tip checkout is whole; verify/gc/push work; `--lazy` fills by id, `sc backfill ` repairs; wire v6; proven by `demo/run_blob_limit_demo.sh` | [0055](docs/adr/0055-blob-size-clone-filter.md) | | **P37d — Sessions on partial clones** | Agent workspaces on a partial clone | `sc ws fork/run/harvest/abandon` and `sc work` see the clone's sparse view; harvest lands through the ADR-0049 merge, conflicts fall back to `work-` | [0050](docs/adr/0050-sessions-on-partial-clones.md) | > **Prior art.** Phases P5–P9 adapt decisions from the sibling project @@ -778,10 +789,11 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in everything else reports a gap with the `sc backfill` repair. Dialing the promisor origin from any read path is still deferred (offline breakage, surprise latency, fetch-during-gc). -- **Blob-size and object-count clone filters (#217, P27).** `sc clone --filter` - is prefix-only; git's `--filter=blob:limit` size axis needs a decision on the - repair path for a size gap, where the filter is evaluated, and protected - content first. +- **Remainder of #217 (P27): a limit-aware `sc fetch` and an object-count clone + limit.** The blob-size filter shipped (ADR-0055); `sc fetch` stays unfiltered, + so the limit bounds the initial transfer, not later growth, and a limit-aware + fetch needs per-path exemptions for backfilled files. An object-count limit has + no stable definition (the transfer set is id-ordered) and no git precedent. - **`sc rewrap` on partial clones — refused (P37).** `Repo::rewrap` calls `refuse_on_partial`; a paths-only/streaming rebuild that would lift the refusal remains a possible follow-on. diff --git a/demo/run_blob_limit_demo.sh b/demo/run_blob_limit_demo.sh new file mode 100755 index 00000000..b99472ba --- /dev/null +++ b/demo/run_blob_limit_demo.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# P27 follow-on demo: blob-size clone filter (ADR-0055, #217). Proves that +# `sc clone --filter blob:limit=` leaves large OLD blobs out of the +# transfer while every ref tip stays complete (the checkout is whole), that +# `sc verify` / `sc gc` / `sc push` work on such a clone, that `sc backfill +# ` repairs the gap, that `--lazy` is accepted with only a size filter, +# and that a prefix filter combines with a size limit. +# +# Self-checking: every claim is an assertion; any failure exits non-zero +# before the RESULT line. +set -euo pipefail + +cd "$(dirname "$0")/.." +cargo build --quiet --bin sc +SC="$PWD/target/debug/sc" + +fail() { echo "FAIL: $1"; exit 1; } + +W="$(mktemp -d)" +trap 'rm -rf "$W"' EXIT + +objects() { find "$1/.sc/objects" -type f | wc -l | tr -d ' '; } + +ORIGIN="$W/origin" +mkdir -p "$ORIGIN" +( + cd "$ORIGIN" + "$SC" init >/dev/null + mkdir -p src + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > assets.bin # v1, ~128 KiB + printf 'fn main() {}\n' > src/a.txt + "$SC" commit -m "v1" --author demo >/dev/null + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > assets.bin # v2 + printf 'fn main() { 2 }\n' > src/a.txt + "$SC" commit -m "v2" --author demo >/dev/null +) +FULL=$(objects "$ORIGIN") +echo "origin: two versions of a ~128 KiB assets.bin; $FULL object(s) ✔" + +echo +echo "=== 1: blob:limit=16KiB leaves the old assets.bin out, the checkout is whole ===" +LIM="$W/limited" +"$SC" clone --filter blob:limit=16KiB "$ORIGIN" "$LIM" > "$W/clone.out" +grep -q "partial: blob:limit=16384" "$W/clone.out" || fail "clone must report the limit: $(cat "$W/clone.out")" +[ "$(objects "$LIM")" -lt "$FULL" ] || fail "the limited clone must hold fewer objects than the origin" +cmp "$LIM/assets.bin" "$ORIGIN/assets.bin" || fail "the tip's large file must be checked out byte-identically" +[ -f "$LIM/.sc/promisor-blob-limit" ] || fail "the clone must record its limit" +echo "$(objects "$LIM") of $FULL object(s); tip assets.bin materialized ✔" + +echo +echo "=== 2: verify reports the gap as expected, gc and status are happy ===" +(cd "$LIM" && "$SC" verify) > "$W/verify.out" +grep -q "partial: 1 object(s) outside filter \[blob:limit=16384\]" "$W/verify.out" || fail "verify gap line: $(cat "$W/verify.out")" +(cd "$LIM" && "$SC" gc >/dev/null && "$SC" status >/dev/null) +echo "verify: 1 gap (the old assets.bin), gc + status fine ✔" + +echo +echo "=== 3: a commit pushes back; the origin keeps the history ===" +(cd "$LIM" && printf 'fn main() { 3 }\n' > src/a.txt && "$SC" commit -m "v3" --author demo >/dev/null && "$SC" push origin >/dev/null) +(cd "$ORIGIN" && "$SC" log | grep -q "v3") || fail "the origin must have the pushed commit" +echo "push from the limited clone landed ✔" + +echo +echo "=== 4: --lazy is accepted when the only filter is blob: ===" +LAZY="$W/lazy" +"$SC" clone --filter blob:limit=16KiB --lazy "$ORIGIN" "$LAZY" >/dev/null +[ -f "$LAZY/.sc/promisor-lazy" ] || fail "--lazy must work with only a blob: filter" +echo "lazy marker set on a size-only clone ✔" + +echo +echo "=== 5: a prefix filter combines with the limit ===" +COMBO="$W/combo" +"$SC" clone --filter src/ --filter blob:limit=16KiB "$ORIGIN" "$COMBO" > "$W/combo.out" +grep -q "partial: src/, blob:limit=16384" "$W/combo.out" || fail "combo report: $(cat "$W/combo.out")" +[ ! -e "$COMBO/assets.bin" ] || fail "assets.bin is outside the prefix and must not be materialized" +echo "src/ + blob:limit: scope first, limit second ✔" + +echo +echo "=== 6: grammar errors are actionable ===" +if "$SC" clone --filter blob:none "$ORIGIN" "$W/bad" 2>"$W/bad.err"; then fail "blob:none must be rejected"; fi +grep -q "blob:limit=0" "$W/bad.err" || fail "blob:none error must point at blob:limit=0: $(cat "$W/bad.err")" +echo "blob:none -> use blob:limit=0 ✔" + +echo +echo "=== 7: backfill by the file's own path, then --all ===" +(cd "$LIM" && "$SC" backfill assets.bin >/dev/null) +(cd "$LIM" && "$SC" verify) | grep -q "partial: 0 object(s)" || fail "backfill must close the gap" +(cd "$LIM" && "$SC" backfill --all >/dev/null) +[ ! -e "$LIM/.sc/promisor" ] && [ ! -e "$LIM/.sc/promisor-blob-limit" ] || fail "--all must remove both markers" +echo "backfill assets.bin closed the gap; --all made it a full clone ✔" + +echo +echo "RESULT: blob-size clone filter proven (limit, tips complete, verify/gc/push, lazy marker, combo, repair)" diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 5d73de0d..4e25ef6e 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -522,6 +522,22 @@ too, not just the src-control-side metadata. store and working tree and, on a lazy clone, fetch more (`sc sparse set`, `sc cat`, `sc backfill`). It is not an isolation boundary (compare Secret injection). +- **A blob-size limit (`--filter blob:limit=`, ADR-0055) is request scope + like the prefix filter, and the server evaluates it.** It hides nothing: any + client that can read the repository can fetch the left-out blobs (`sc + backfill`, by-id fill), and a server that sends fewer blobs than asked can + only create a gap, which surfaces as `GapInView`/`GapBySize` or a fill error + before any write, never as wrong content (objects are hash-verified). The + size is the stored payload length, so for a protected path it is the + ciphertext size, which anyone who can fetch the object already sees; the + filter reveals nothing new. A limited `GetPack` makes the server stream-measure + each distinct candidate blob once (`Store::blob_size`, fixed buffer, hash + checked, cached by id), bounded by the stored bytes of those blobs, which an + unfiltered `GetPack` reads anyway. Reads are not budgeted (#233 budgets writes) + and the filter does not change that; the listener limits are the bound. Every + active filter, a limit alone included, still excludes private branches and is + refused by bucket remotes. The wire bump (version 6) makes a mixed-version pair + fail closed at the handshake in both directions. - **Operations that need a complete closure refuse on a partial clone, fail closed:** `sc export`, `sc rewrap`, `sc grant`/`revoke`, private branches, and `sc push` to a git remote. They refuse rather than run on partial data. diff --git a/docs/adr/0037-partial-clone.md b/docs/adr/0037-partial-clone.md index b2f29670..45f2f9ad 100644 --- a/docs/adr/0037-partial-clone.md +++ b/docs/adr/0037-partial-clone.md @@ -1,6 +1,6 @@ # ADR-0037: Partial clone (promisor store + prefix-scoped fetch) -- **Status:** Accepted; the merge, replay and session refusals are lifted by [ADR-0049](0049-merge-on-partial-clones.md) and [ADR-0050](0050-sessions-on-partial-clones.md), and the filter-based walks replaced by [ADR-0048](0048-presence-based-partial-clones.md) +- **Status:** Accepted; the merge, replay and session refusals are lifted by [ADR-0049](0049-merge-on-partial-clones.md) and [ADR-0050](0050-sessions-on-partial-clones.md), and the filter-based walks replaced by [ADR-0048](0048-presence-based-partial-clones.md); the blob-size axis it deferred is [ADR-0055](0055-blob-size-clone-filter.md) - **Date:** 2026-07-09 - **Phase:** 27 - **Builds on:** ADR-0034 (sparse checkout — the materialize half), ADR-0035 (streaming pack transfer), ADR-0015 (packfiles + reachability gc), ADR-0022/0036 (transports) @@ -73,7 +73,10 @@ independent filters). (sparse-view into un-fetched gaps with no "one slice" mental model). - **Blob-size / object-count filters** (git's `--filter=blob:limit`): prefix-only matches sparse and the path-boundary machinery already - built; size filters are a separate axis, deferred. + built; size filters are a separate axis, deferred. *Since built:* the size + filter is [ADR-0055](0055-blob-size-clone-filter.md) (`--filter + blob:limit=`, server-evaluated, `PROTOCOL_VERSION` 6); the object-count + limit stays deferred. ## Refinements discovered during the build diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index af4d692b..48383a51 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -1,6 +1,6 @@ # ADR-0048: Presence-based walks for partial clones (P37b) -- **Status:** Accepted (built) +- **Status:** Accepted (built); [ADR-0055](0055-blob-size-clone-filter.md) adds a second kind of gap (a blob left out by size) that these presence-based walks cover unchanged - **Date:** 2026-10-16 - **Phase:** 37b - **Builds on:** ADR-0047 (want-by-id fetch, 37a), ADR-0037 (partial clone), ADR-0034 (sparse), ADR-0035 (streaming transfer), ADR-0039/0041 (size limits) @@ -332,6 +332,19 @@ review established, as constraints on that work: through the export walk at the first absent object. 37b-i adds the same up-front refusal that `sc export` has. +## Addendum: a blob left out by size (ADR-0055) + +`sc clone --filter blob:limit=` leaves out blobs by size rather than by +path, and the model above covers it without a new record: the gap is an absent +blob a present tree references, the promise P is unchanged, and push, verify and +gc already ask the store. What changes: the filter now has a second dimension, +`.sc/promisor-blob-limit` records it and an explicit `Scope` (a size-only clone +is `AllPaths`, which an empty prefix list could not say), "filter active" means +`prefixes || blob_limit` everywhere, and a full sparse view must still fill on a +lazy `AllPaths` clone (`ensure_view`). Every ref tip is complete at clone time; +a later fetch can bring a tip that points at a gapped blob, which is the +`GapInView` case above with the limit named in its message. + ## Phasing - **37b-i, engine (built).** `reachable_objects_partial`; gc, push and verify on it; @@ -410,6 +423,7 @@ review established, as constraints on that work: the fetch and the record, and nothing it proves that the push-time check does not. - **Make `reachable_objects_filtered` presence-aware for everyone.** The origin - uses it to cut transfers. + uses it to cut transfers. (ADR-0055 extends that same walk with a blob-size + rule and every-tip exemption, `reachable_objects_limited`, still server-side.) - **A tree-recursive merge by id in this ADR.** Withdrawn; see decision 6. - **Server-side merge and history operations.** Still deferred (ROADMAP). diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md index f531ac84..c4c1a3ff 100644 --- a/docs/adr/0055-blob-size-clone-filter.md +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -1,45 +1,42 @@ # ADR-0055: Blob-size clone filter (`--filter blob:limit=`) -- **Status:** Proposed. Needs an owner decision: it requires a wire change (`PROTOCOL_VERSION` 5 to 6). Nothing is implemented. +- **Status:** Accepted (built). Accepted after an independent Opus plan review and a Sol review; their blockers are decisions 3, 5, 7 and 8 below. The object-count limit and a limit-aware `sc fetch` are deferred (decision 2). - **Date:** 2026-10-20 - **Phase:** 27 follow-on (issue #217) -- **Builds on:** ADR-0037 (partial clone), ADR-0047 (want-by-id fetch), ADR-0048 (presence-based walks), ADR-0049/0050 (merge and sessions on partial clones), ADR-0044 (private branches), ADR-0014 (protected paths) +- **Builds on:** ADR-0037 (partial clone), ADR-0047 (want-by-id fetch), ADR-0048 (presence-based walks), ADR-0049/0050 (merge and sessions on partial clones), ADR-0044 (private branches), ADR-0014 (protected paths), ADR-0046 (bucket remotes) ## Context -`sc clone --filter` takes path prefixes only. Issue #217 asks for git's -`blob:limit` axis (skip blobs above a size) and an object-count limit. +`sc clone --filter` took path prefixes only. Issue #217 asks for git's +`blob:limit` axis (leave out blobs above a size) and an object-count limit. ADR-0048 already made a partial clone a store plus one promise: every id a present object references and the store lacks is held by the origin. The filter records what a clone *asked for*; presence decides what exists. A blob -left out by size is the same kind of gap as a blob left out by path, so the -client side needs almost nothing new. The open questions are who learns the -size, and what a checkout does with a gapped file. +left out by size is the same kind of gap as one left out by path, so the client +side needs almost nothing new. The open questions were who learns the size, and +what a checkout does with a gapped file. ## Findings that shape the design 1. **Only the server can know a blob's size before receiving it.** A - `TreeEntry` is `{name, kind, id, mode, perms}` (`core/src/object.rs:119`); - there is no size. The wire has no size query: `HasObject` returns a bool, - `GetObject`/`GetObjects` return the object itself (`wire.rs:118`). The - by-id reply caps (ADR-0048 decision 5) refuse an over-cap *batch*, they do - not report sizes. -2. **The prefix filter cannot carry it.** `GetPack` ships `filter: - Vec` of path prefixes (`wire.rs:138`, `put_strs`). A reserved - entry such as `blob:limit=1048576` in that list would be read by a v5 - server as a path prefix and silently send almost nothing, a fail-unsafe - misparse. ADR-0047 rejected the same trick for the lazy flag. -3. **The walk that cuts a transfer never reads blobs.** - `walk_tree_filtered` decides blob inclusion from the path alone - (`reachable.rs:347`). A size rule means the server reads (or at least - sizes) each candidate blob once during the walk. -4. **Git has no object-count filter.** Its filters are `blob:none`, - `blob:limit`, `tree:`, `object:type`, `sparse:oid`, `combine`. A - count limit would also be undefined here: `reachable_objects_filtered` - collects into a `BTreeSet` ordered by id, so "the first N objects" would - be an arbitrary, server-dependent set, and a client could not tell a - cut-off object from a missing one by any rule. + `TreeEntry` is `{name, kind, id, mode, perms}` (`core/src/object.rs`); there + is no size. The wire has no size query: `HasObject` returns a bool, + `GetObject`/`GetObjects` return the object itself. The by-id reply caps + (ADR-0048 decision 5) refuse an over-cap *batch*; they do not report sizes. +2. **The prefix filter cannot carry it.** `GetPack` shipped `filter: + Vec` of path prefixes. A reserved entry such as `blob:limit=1048576` + in that list would be read by a v5 server as a path prefix and silently send + almost nothing, a fail-unsafe misparse. ADR-0047 rejected the same trick for + the lazy flag. +3. **The walk that cuts a transfer never read blobs.** `walk_tree_filtered` + decided blob inclusion from the path alone. A size rule makes the server + measure each candidate blob once per request. +4. **Git has no object-count filter** (`blob:none`, `blob:limit`, `tree:`, + `object:type`, `sparse:oid`, `combine`). A count limit is also undefined + here: `reachable_objects_filtered` collects into a `BTreeSet` ordered by id, + so "the first N objects" would be an arbitrary, server-dependent set, and a + client could not tell a cut-off object from a missing one by any rule. ## Decisions @@ -48,161 +45,222 @@ size, and what a checkout does with a gapped file. `sc clone --filter blob:limit= `, with `` parsed by `scl_core::parse_byte_size` (`1048576`, `1MiB`, `512KiB`). It combines with prefix filters in the same repeatable flag: `--filter src/ --filter -blob:limit=1MiB` means "only `src/`, and within it no blob above 1 MiB except -as decision 4 says". `blob:limit=0` is legal and means no blobs beyond the -tips. +blob:limit=1MiB` means "only `src/`, and within it no blob of 1 MiB or more +unless decision 4 keeps it". -`--filter` values starting with `blob:` become a reserved grammar. An unknown -`blob:` form (`blob:none`, a misspelling) is an error, not a prefix. Cost: a -path prefix that literally begins with `blob:` can no longer be named in -`--filter`. Alternative: a separate `--blob-limit` flag, which has no -collision. This ADR recommends the issue's form; it is an owner choice. +**Boundary:** a blob of size >= the limit is left out (like git; `blob:limit=1` +leaves out every non-empty blob outside the tips). `blob:limit=0` leaves out +every blob no ref tip reaches. Size is the stored object's payload length. -### 2. Object-count limit: deferred, not built +**Grammar.** `blob:` is reserved in `--filter`: only `blob:limit=` is +accepted, once. A second `blob:limit` is an error. `blob:none` is an error that +points at `blob:limit=0`. Any other `blob:` form is an error naming the one +supported form. Cost: a path prefix that literally begins with `blob:` can no +longer be named in `--filter`. `sc clone --help` documents this. `--lazy` +requires `--filter`, which a lone `blob:limit=` satisfies, and works with it. +Rejected: a separate `--blob-limit` flag, which has no collision but diverges +from the issue and from git. -No stable definition (finding 4), no git precedent, no stated use case. -`ROADMAP.md` keeps a one-line deferred entry for it; the blob-size part -leaves the deferred list when this lands. +### 2. Deferred -### 3. The server evaluates the filter: wire change, `PROTOCOL_VERSION` 6 +- **Object-count limit.** No stable definition (finding 4), no git precedent, no + stated use case. `ROADMAP.md` keeps a one-line deferred entry. +- **A limit-aware `sc fetch`.** Fetch stays unfiltered (ADR-0048 Context), so new + large blobs arrive in full and the limit bounds the initial transfer, not later + growth. A limit-aware fetch needs the paths a user backfilled to ride in the + request as exemptions, which is a larger design. + +### 3. The server evaluates the filter: wire change, `PROTOCOL_VERSION` 5 to 6 Findings 1 and 2 leave no client-side design that saves transfer. -- `Request::GetPack` gains a trailing optional `blob_limit` (a tag byte, then - a `u64` when present; absent means no limit, so `blob:limit=0` stays - representable). `PROTOCOL_VERSION` 5 to 6; a v5 peer is rejected at the - handshake, as at P25, P27, P34 and P37. -- `Transport::get_pack` gains the same parameter. Every implementation - (`LocalTransport`, `WireClient`, stdio, http) forwards it; the two wrappers - forward by hand as in ADR-0047. -- `build_pack_tempfile` passes it to `reachable_objects_filtered`, which - drops a blob over the limit into `gaps` (never `included`). The `haves` - side stays unfiltered, as ADR-0037 requires. -- Bucket remotes already refuse partial clone (ADR-0046) and refuse this. - -Alternatives considered and rejected: - -- **Client-side drop on ingest, no bump.** The client downloads every blob and - discards the large ones. It saves disk and gives the same gap model, but - not one byte of transfer, which is the point of the feature. It would also - hide the missing wire support behind a flag that does not do what git users - expect. -- **A size-query verb (`ObjectSizes`) plus client-chosen fetches.** Also a - new verb and a bump. The client would walk all history level by level over - `GetObjects` to learn blob ids first, then ask sizes, then fetch: many - round trips for something the server does in one walk. -- **Sentinel in the prefix list.** Finding 2. - -Server cost: each candidate blob is read once to measure it, then read again -when written to the pack if kept. This is O(blobs under the filter) per -clone and backfill, never per fetch. A size index is a later optimization. - -### 4. Tips are complete; the limit applies to history - -The server exempts blobs reachable from the root tree of each *want* (every -ref tip the client asked for) that match the prefix filter. The limit applies -to blobs reachable only through parent snapshots. - -Reason: checkout, `sc switch` between branches, and every operation that -materializes a tip need those blobs, and the clone's own initial checkout -runs before any lazy marker can exist (`sync.rs`, the -`ensure-view-exempt: clone` note). A strict limit would make a non-lazy -clone fail at its own checkout with `GapInView`, and a lazy one would fetch -the same large files in a second round trip right after clone. The -alternative that skips large tracked files at checkout would need a third -"tracked but absent" state in status, diff, commit graft, materialize and the -merge engine (the sparse machinery covers paths, not sizes) and is not a small -change. - -What this saves: old revisions of large files, and large files only reachable -from history. That is where git's `blob:limit` pays off for binary assets -too. What it does not do: skip a large file at HEAD. The issue's phrase -"reports or lazily fills large ones" is therefore met by the history case: -operations reaching an old gapped blob fill it by id on a lazy clone and fail -with the gap error on a non-lazy one (decision 5). - -### 5. Gap record, presentation and repair +- `Request::GetPack` gains a trailing optional `blob_limit` (a tag byte, then a + big-endian `u64` when present; absent means no limit, so `blob:limit=0` stays + representable). `PROTOCOL_VERSION` 5 to 6, in the same style as P25, P27, P34 + and P37. +- `Transport::get_pack` takes a `PackFilter` instead of a prefix slice; every + implementation (`LocalTransport`, `WireClient`, stdio, http, bucket) carries + it, and the two wrappers forward by hand as in ADR-0047. +- The sender's walk (`reachable_objects_limited`) moves a blob at or over the + limit from `included` to `gaps`, unless decision 4 exempts it. The `haves` side + stays unfiltered, as ADR-0037 requires. + +**Alternatives that avoid a bump, and why they lose.** + +- *A capability-probed new verb* (say `GetPackSized`): a v5 server answers an + unknown opcode with a protocol error mid-session, so the client needs a probe + round trip first, and `Hello` carries only a version, so the probe is itself a + new verb. It buys mixed-version operation only for the size filter and leaves + every other verb at v5, two code paths to keep forever. +- *A trailing optional field on `GetPack` without a bump:* a v5 decoder ends with + `done()` and rejects trailing bytes as `trailing bytes in frame`, a mid-session + error the client cannot distinguish from corruption, with no way to fall back. +- *A size manifest plus `GetObjects`:* the client would walk all history level by + level to learn blob ids, ask for sizes through a new verb (a bump anyway), then + fetch: many round trips for what the server does in one walk. +- *Client-side drop on ingest:* the client downloads every blob and discards the + large ones. It saves disk and gives the same gap model, but not one byte of + transfer, which is the point of the feature. +- *A sentinel in the prefix list:* finding 2. + +**Cost of the bump.** The server and its clients upgrade together. Every mixed +pair fails closed at the handshake, before any repository access, in both +directions, for `sc+http(s)://` and for ssh remotes alike (an `ssh://` remote +runs the remote host's own `sc serve --stdio` binary, so its version is whatever +that host has installed). The messages are the server's own: + +- v6 client, v5 server: `wire protocol error: unsupported protocol version 6 + (server speaks 5)`. +- v5 client, v6 server: `unsupported protocol version 5 (server speaks 6)`, which + the v5 client prints as `wire protocol error: unsupported protocol version 5 + (server speaks 6)`. + +Pinned by `a_v6_client_reports_a_v5_servers_refusal`, +`serve_rejects_a_v5_client_naming_both_versions` and `handshake_rejects_v5_peer` +(the retargeted successor of `handshake_rejects_v4_peer`), plus encode and decode +round trips of `GetPack` with and without a limit and strict rejection of a bad +tag or a v5-length frame. + +### 4. Every ref tip is complete; the limit applies to history + +The walk exempts blobs reachable from the root tree of each *want* (every ref tip +the client asked for) that match the prefix filter. The limit applies to blobs +reachable only through parent snapshots. + +Reason: checkout, `sc switch` between branches and every operation that +materializes a tip need those blobs, and the clone's own initial checkout runs +before any lazy marker exists. A strict limit would make a non-lazy clone fail at +its own checkout with `GapInView`, and a lazy one would fetch the same large +files in a second round trip right after the clone. Skipping large tracked files +at checkout would need a third "tracked but absent" state in status, diff, commit +graft, materialize and the merge engine (the sparse machinery covers paths, not +sizes). + +**Trade-off.** The filter saves old revisions of large files and large files only +reachable from history, which is where git's `blob:limit` pays off for binary +assets. It does not skip a large file at HEAD or at any other branch tip, so a +clone with many branches carries each tip's large files. "Reports or lazily fills +large ones" from the issue is therefore met for the history case: an operation +that reaches an old gapped blob fills it by id on a lazy clone and fails with the +gap error otherwise. + +**Tips are complete at clone time only (D2).** A later `sc fetch` is unfiltered +and subtracts the closure of the local tips, so a new tip that points at an old, +gapped blob (a revert) arrives without it. A lazy clone heals that on the +fast-forward or switch (decision 7); a non-lazy one gets the gap error with the +limit clause (decision 6) and `sc backfill ` repairs it. + +**Ordering (D7).** `walk_snapshots` queues every tip before any parent, so a +tip's exempt walk always runs before a non-tip snapshot's limited one, even when +one tip is an ancestor of another. The `(id, path)` expansion gate then only ever +lets a limited walk skip work an exempt walk already did, never the reverse, and +a blob already `included` is kept without measuring. A tip root tree that equals +an older subtree is expanded at its own `(root, "")` key. Tests: an ancestor tip, +a tip root equal to a subtree of a non-tip, and the walk with the wants in both +orders. + +### 5. Scope is explicit; `filter active` has one definition (B1, B2) + +- **`Promisor`** holds `Scope` (`AllPaths` or `Prefixes(list)`), `blob_limit: + Option` and the origin. `AllPaths` is a clone with a limit and no prefix; + an empty prefix list means "matches nothing" and cannot say it. Every consumer + of `matches`/`should_descend`/`prefixes` handles the case (graft, materialize + bound, sparse refusals, gap hints, verify text, widen), and `widen` on + `AllPaths` is a no-op, so `backfill ` cannot narrow the scope. +- **Persistence.** `.sc/promisor` is unchanged (`origin ` then one prefix per + line). `.sc/promisor-blob-limit` (local, uncommitted) holds `limit ` and + `scope all-paths|prefixes`, written before `.sc/promisor` and removed with it + (`promisor::remove`, so `backfill --all` leaves neither). It is a separate file + for the ADR-0047 reason: any text is a legal path. A malformed file is an error, + not an unlimited clone. A limit file with no `.sc/promisor` is leftover state + and reads as a full clone. +- **An older binary** cannot talk to a v6 server, and locally it reads + `.sc/promisor` only: a size-only clone has an empty prefix list there, which it + treats as a degenerate partial clone that matches nothing (gaps everywhere, + never wrong content), and it ignores the limit file. If it ran `backfill + `, that would add a prefix to `.sc/promisor`; the current binary still + reads `scope all-paths` from the limit file and keeps the whole-tree scope. If + it ran `backfill --all`, it would remove `.sc/promisor` and leave the limit file, + which the current binary ignores. +- **`PackFilter`** is the one definition of "filter active": it exists only when + `!prefixes.is_empty() || blob_limit.is_some()`. Every `Some(&PackFilter)` + therefore means a partial transfer, in all of these places, each with a test: + the server walk and its private-branch handling (`filter_manifest_wants` drops + manifest wants), the bucket refusal (`BucketTransport::get_pack`, a served + `--store` over the wire, and a direct `sc+wal://` clone through the CLI), and + the client's clone, which skips refs whose tip did not arrive. + +### 6. Gap record, presentation and repair - **No new gap record.** Absence is the gap (ADR-0048). Push, verify and gc - already walk presence (`reachable_objects_partial`) and need no change; the - tests for them are new, the code is not. -- **Marker:** `.sc/promisor-blob-limit`, local and uncommitted, one decimal - byte count. Not a field in `.sc/promisor`, for the ADR-0047 reason (that - file is `origin` plus one path prefix per line). `Repo::promisor()` returns - a clone with the limit marker as partial even when the prefix list is empty; - an empty prefix list together with the marker means the whole tree is in - scope. `Promisor::matches`/`should_descend` and its consumers - (`graft_out_of_sparse`, the materialize old-view bound, push and verify - messages) need that case checked one by one; it is the main implementation - risk. -- **Presentation:** `sc verify` keeps `partial: N object(s) outside filter - [...]`, with `blob:limit=` printed in the bracket beside any - prefixes. `sc clone` prints one line naming the limit and the gap count. + already walk presence (`reachable_objects_partial`) and need no code change; + tests pin them on a size-limited clone, including the push gap-edge check + against a destination that lacks the history. +- **Presentation.** `sc clone` prints `partial: blob:limit=` (prefixes + first when both). `sc verify` keeps `partial: N object(s) outside filter + [...]` with the same bracket text. `GapInView` gains the limit: its message + names `blob:limit=` as a cause beside "an earlier fetch left a gap", and + `prefix` is the file's own path, not a covering prefix. Gaps found by a merge, + pick, rebase, session or `sc cat` on a clone with a limit are + `GapBySize { path, limit }`: "`` was left out of this partial clone + (blob:limit=); run `sc backfill ` to fetch it first". A prefix-only + clone keeps every old message byte for byte. - **Repair, lazy and online:** nothing to do. `ensure_view`, `sc cat` and the ADR-0049 merge fill by id. -- **Repair, otherwise:** the gap error already names `sc backfill `. - For a size gap the prefix is the gapped file's own path, which - `Promisor::matches` accepts as an exact boundary. `sc backfill ` sends - every version of that path without applying the limit (backfill passes no - `blob_limit`). `sc backfill --all` removes the markers and converts to a full - clone, as before (`promisor::remove` must also delete the limit marker). - Error text gains one clause when the limit marker exists: "(blob:limit=1MiB - left this out of the clone)". No `--min-size` form: it would be a second - size parser on the repair side with no use that `` does not cover. -- **`sc fetch` is unchanged:** it stays unfiltered (ADR-0048 Context), so new - large blobs from new commits arrive in full. A limit-aware fetch needs the - exemption list for backfilled paths in the request, which is a larger design. - Deferred; the clone's limit therefore bounds the initial transfer, not later - growth. - -### 6. Interactions - -- **Protected content.** The size is the stored object's payload length. A - protected file's blob is ciphertext, so its size is the plaintext size plus - the sealing overhead; the server cannot see plaintext. A file near the - limit may flip sides from its plaintext size by that overhead. Protected - and plain blobs are treated by the same rule. No new information leaks: - ciphertext length is already visible to anyone who can fetch the object. -- **Private branches.** Unchanged: a filtered transfer already excludes them - (ADR-0044). -- **Sealed objects, signatures, transcripts.** Never subject to the limit; - they are not path-reached blobs. -- **Sparse and prefix filters.** The prefix decides scope first, the limit - prunes inside it. `.sc/sparse` is still set to the prefixes (none for a - size-only clone, meaning a full view). -- **Push, verify, gc.** Presence-based already (ADR-0048); a size gap is - checked against the push destination like any other gap. -- **Merge, pick, rebase, ws, work** (ADR-0049/0050). A merge base or an old - version that is a size gap is "asked for" by the engine and filled by id on - a lazy, online clone, or fails with the gap error. No new engine code. -- **Wire caps.** A gapped blob over the by-id reply cap behaves as in - ADR-0048 decision 5. - -## Implementation outline (once accepted) - -1. Wire v6: `GetPack.blob_limit`, `Transport::get_pack`, all transports, a - version test, the wire doc comments. -2. Server walk: size-aware, tip-exempt `reachable_objects_filtered`; tests - that a filtered pack omits a large history blob and keeps a large tip - blob; a differential test against the unlimited walk. -3. `Repo`: clone option, `.sc/promisor-blob-limit`, whole-tree scope case in - `Promisor`, `promisor::remove`, backfill passing no limit. -4. CLI: parse `blob:limit=` in `run_clone`, reject unknown `blob:` - forms, report line, `sc verify` text. -5. Tests: push, verify and gc on a size-limited clone; the gap error text; - prefix plus size; `sc backfill ` repair; lazy fill by id; partial - clone refusals unchanged. -6. Docs: ADR-0037 and ADR-0048 pointers, CLAUDE.md capability row and bullets, - ROADMAP, ARCHITECTURE, THREAT-MODEL (one line on sizes), the - `sc-cli-reference` skill, a demo script. - -## Open questions for the owner - -1. Accept the `PROTOCOL_VERSION` 6 bump? (Recommended: yes; no useful design - avoids it.) -2. Tips complete, limit on history only (decision 4), versus a strict limit? - (Recommended: tips complete.) -3. `--filter blob:limit=...` with `blob:` reserved, or a separate - `--blob-limit` flag? (Recommended: the issue's form.) -4. Object-count limit deferred, and `sc fetch` not limit-aware in the first - version. (Recommended: both deferred; PR says `Refs #217`.) +- **Repair, otherwise:** `sc backfill ` fetches every version of that path + (backfill sends no limit), and `sc backfill --all` converts to a full clone. No + `--min-size` form: it would be a second size parser on the repair side with no + use that `` does not cover. + +### 7. A full view must still fill (B3) + +`Repo::ensure_view` returned without filling when the sparse view was full, so a +lazy size-only clone, whose view is always full, would have failed on switch, +undo, merge, replay and sessions to a revision with a gapped blob. The guard now +skips only when the clone is not `AllPaths`. For an `AllPaths` clone the fill is +bounded by the one revision being laid out. Tests: switch and undo to a gapped +revision (refused, then filled when lazy), merge, cherry-pick and rebase with a +gapped merge base (the engine fills it by id, or refuses before any write with +the file's own path), and a workspace session on a size-limited clone. + +### 8. Interactions + +- **Protected content (D6).** The size is the stored object's payload length. A + protected file's blob is ciphertext, so its size is the plaintext size plus the + sealing overhead, and a file near the limit can land on the other side of it + from its plaintext size. Protected and plain blobs follow the same rule. No new + information leaks: ciphertext length is already visible to anyone who can fetch + the object. +- **Private branches.** A filtered transfer excludes them (ADR-0044), now under + any active filter (decision 5). +- **Sealed objects, signatures, transcripts.** Not path-reached blobs; never + subject to the limit. +- **Sparse and prefix filters.** The prefix decides scope first, the limit prunes + inside it. A size-only clone leaves `.sc/sparse` empty (a full view). +- **Push, verify, gc.** Presence-based (ADR-0048). +- **Merge, pick, rebase, ws, work** (ADR-0049/0050). No new engine code; a size + gap is "asked for" like any other (decision 7). +- **Server cost and limits (D5).** The walk measures a blob with `Store::blob_size` + (streamed through a fixed buffer, hash-verified, never admitted to the object + cache) and caches each verdict by blob id, so a distinct blob is measured at + most once per request. The work is bounded by the stored bytes of the distinct + candidate blobs, which an unfiltered `GetPack` reads anyway to build its pack; + the filter changes bandwidth, not read cost. Reads are unbudgeted (#233 budgets + writes), and the filter does not change that: `--max-connections`, `--timeout` + and the listener limits remain the bounds. A size index on the server is a later + optimization. + +## Consequences + +- A clone can skip the old revisions of large files, with a complete checkout and + every ref tip complete. +- Wire version 6; servers and clients upgrade together. +- A new marker file, one new scope case in `Promisor`, one new error variant + (`GapBySize`) and a field on `GapInView`. +- `sc fetch` is not limit-aware (decision 2). + +## Open questions + +1. A limit-aware `sc fetch` with per-path exemptions for backfilled files. +2. An object-count or `tree:`-style filter, if a use case appears. +3. A server-side size index so a limited request need not stream-measure. diff --git a/docs/adr/README.md b/docs/adr/README.md index 9c220f8e..fdf19e3e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -65,7 +65,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0052](0052-secret-injection-by-fd.md) | Secret injection by file descriptor (`sc run --secrets-fd`) | Accepted (built) | — | | [0053](0053-tls-pin-management.md) | TLS pin management (`sc tls`) | Accepted (built) | — | | [0054](0054-tls-ca-validation.md) | CA-path validation for `sc+https://` | Accepted (built) | — | -| [0055](0055-blob-size-clone-filter.md) | Blob-size clone filter (`--filter blob:limit=`) | Proposed (needs wire v6 decision) | 27 | +| [0055](0055-blob-size-clone-filter.md) | Blob-size clone filter (`--filter blob:limit=`) | Accepted (built) | 27 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, From 3a2df915f5dcf89d688f05c8374668d14711e696 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:38:56 +0300 Subject: [PATCH 4/7] fix(repo): let a size-only clone restore its full view; review follow-ups (ref #217) Review found a size-only clone could narrow its view with sparse set but never return to the full one it started with (empty sparse set and sparse disable were refused on every partial clone). They are now allowed for an AllPaths scope. Also: measure nothing for limit 0 and u64::MAX, only name the limit as a gap cause for paths in scope, a stdio test that the limit reaches the server, and corrected cost wording in the threat model. Refs #217 --- CLAUDE.md | 2 +- crates/repo/src/partial_size.rs | 62 ++++++++++++++++++------- crates/repo/src/reachable.rs | 7 +++ crates/repo/src/sparse.rs | 12 ++++- crates/repo/src/stdio_transport.rs | 49 +++++++++++++++++++ crates/repo/src/worktree.rs | 7 ++- docs/THREAT-MODEL.md | 6 ++- docs/adr/0055-blob-size-clone-filter.md | 14 ++++-- 8 files changed, 132 insertions(+), 27 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index cf4481f8..633c6855 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -185,7 +185,7 @@ transport-adjacent. The rest, imperatively: renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or identity material. -- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`, +- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (both allowed on a size-only `AllPaths` clone, ADR-0055; and, on a non-lazy clone, one outside the filter), `sc rewrap`, `sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge, cherry-pick or rebase whose conflict lies outside the view (a `.theirs` sidecar sits beside its conflicted path, so it is in view whenever that path is; ADR-0049) (no ref, state or working-tree change; on every clone, a conflict stop that needs sealing outside the view is refused the same way), `sc push` to a git remote, and `sc fetch` from any native remote but the promisor origin.** `sc merge`, `sc cherry-pick` and `sc rebase` work, including conflicts inside the view (P37c). `sc ws` and `sc work` work too (ADR-0050): workspaces and agents see the clone's view only, and harvest lands through the same merge. `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs index 8cfa41b2..53d714ca 100644 --- a/crates/repo/src/partial_size.rs +++ b/crates/repo/src/partial_size.rs @@ -533,10 +533,11 @@ fn limited_walk_orders_and_none_match_the_unlimited_walk() { sz.cleanup(); } -/// B3: a lazy size-only clone has a full view, and switch/undo to a revision -/// with a gap must still fill (the `is_full()` guard used to skip it). +/// B3: a lazy size-only clone has a full view, and a switch to a revision with +/// a gap must still fill (the `is_full()` guard used to skip it). Undo has its +/// own test below. #[test] -fn lazy_full_view_switch_and_undo_fill_the_gapped_revision() { +fn lazy_full_view_switch_fills_the_gapped_revision() { let sz = Sz::new("b3"); let clone = sz.clone(&[], LIMIT, false); refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); @@ -555,12 +556,9 @@ fn lazy_full_view_switch_and_undo_fill_the_gapped_revision() { ); assert!(has(&clone, &blob_id(&huge(1))), "filled by id"); - // Undo back to main (already complete) and then forward again is a no-op - // for gaps; undo to the gapped tip on a non-lazy clone fails first. + // gc keeps the filled blobs (reachable from `old`). clone.switch("main").unwrap(); - crate::promisor::set_lazy(clone.layout(), false).unwrap(); clone.gc(Duration::from_secs(0)).unwrap(); - // gc kept the filled blobs (reachable from `old`), so nothing is missing. assert!(has(&clone, &blob_id(&huge(1)))); drop(clone); sz.cleanup(); @@ -820,15 +818,6 @@ fn partial_refusals_are_unchanged_on_a_size_limited_clone() { let clone = sz.clone(&[], LIMIT, false); let (sk, pk) = scl_crypto::generate_keypair(); let rid = pk.recipient_id(); - let err = clone.set_sparse(&[], None).unwrap_err(); - assert!( - matches!(err, Error::InvalidArgument(ref m) if m.contains("empty sparse spec")), - "got {err:?}" - ); - assert!( - err.to_string().contains(&format!("blob:limit={LIMIT}")), - "{err}" - ); assert!(matches!( clone.rewrap(&sk, &[], &[], false), Err(Error::PartialCloneUnsupported(_)) @@ -841,6 +830,22 @@ fn partial_refusals_are_unchanged_on_a_size_limited_clone() { clone.revoke("src/", &rid), Err(Error::PartialCloneUnsupported(_)) )); + drop(clone); + sz.cleanup(); + + // A prefix + limit clone keeps the sparse refusals and names both filters. + let sz = Sz::new("refuse-prefix"); + let clone = sz.clone(&["src/"], LIMIT, false); + let err = clone.set_sparse(&[], None).unwrap_err(); + assert!( + matches!(err, Error::InvalidArgument(ref m) if m.contains("empty sparse spec")), + "got {err:?}" + ); + assert!( + err.to_string() + .contains(&format!("src/, blob:limit={LIMIT}")), + "{err}" + ); assert!( clone.disable_sparse(None).is_err(), "sparse disable stays refused" @@ -849,6 +854,31 @@ fn partial_refusals_are_unchanged_on_a_size_limited_clone() { sz.cleanup(); } +/// A size-only clone can narrow its view and return to the full one it started +/// with: `sparse set` with no prefixes and `sparse disable` are allowed on an +/// `AllPaths` scope (the closure is filled when lazy, `GapInView` otherwise). +#[test] +fn a_size_only_clone_can_narrow_and_restore_its_full_view() { + let sz = Sz::new("restore"); + let clone = sz.clone(&[], LIMIT, false); + clone.set_sparse(&["src/".to_string()], None).unwrap(); + assert!(!sz.clone_root.join("big.txt").exists(), "narrowed"); + clone.set_sparse(&[], None).unwrap(); + assert!(clone.sparse_spec().unwrap().is_full()); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "restored" + ); + clone.set_sparse(&["src/".to_string()], None).unwrap(); + clone.disable_sparse(None).unwrap(); + assert!(clone.sparse_spec().unwrap().is_full()); + assert!(sz.clone_root.join("big.txt").exists()); + assert!(clone.promisor().unwrap().unwrap().is_all_paths()); + drop(clone); + sz.cleanup(); +} + /// B3 for undo: restoring a ref to a revision with a size gap fills it first on /// a lazy full-view clone, and refuses before moving anything otherwise. #[test] diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index 44f6efe8..9976a2ff 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -437,6 +437,13 @@ impl SizeGate<'_> { if self.exempt { return Ok(true); } + // Decided by the limit alone: nothing to measure. + if self.limit == 0 { + return Ok(false); + } + if self.limit == u64::MAX { + return Ok(true); + } if let Some(keep) = self.verdicts.get(id) { return Ok(*keep); } diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index bc3b0c1c..97375058 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -337,8 +337,11 @@ impl Repo { // tree) — so a refused widen leaves nothing partially written. if let Some(promisor) = self.promisor()? { // No prefixes means a full checkout, which is what `sparse - // disable` is, and that is refused on a partial clone. - if prefixes.is_empty() { + // disable` is, and that is refused on a partial clone. A size-only + // clone (ADR-0055, `AllPaths`) is the exception: a full view is its + // starting state, so a narrowed view must be able to return to it. + // The view closure is filled (lazy) or refused (`GapInView`) below. + if prefixes.is_empty() && !promisor.is_all_paths() { return Err(Error::InvalidArgument(format!( "cannot set an empty sparse spec (a full checkout) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full \ @@ -743,6 +746,11 @@ impl Repo { // the origin). Refuse up front, before any disk write, pointing at // backfill — same preflight discipline as `set_sparse` above. if let Some(promisor) = self.promisor()? { + // A size-only clone's scope is every path, so disabling sparse only + // restores its full view: the same operation as an empty spec. + if promisor.is_all_paths() { + return self.set_sparse(&[], identity); + } return Err(Error::InvalidArgument(format!( "cannot disable sparse checkout (full materialization) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full clone \ diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 3b171187..19c51c2b 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -681,6 +681,55 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + /// ADR-0055: the client's `blob_limit` reaches the server and is applied + /// there. A tip blob over the limit is kept, an old one over it is not, an + /// old small one is kept. + #[test] + fn blob_limit_get_pack_over_wire() { + let root = tmp_repo("blob-limit"); + let repo = crate::repo::Repo::open(&root).unwrap(); + let old_big = vec![b'o'; 4096]; + let new_big = vec![b'n'; 4096]; + std::fs::write(root.join("big"), &old_big).unwrap(); + std::fs::write(root.join("small"), b"s1").unwrap(); + repo.commit("t", "c1").unwrap(); + std::fs::write(root.join("big"), &new_big).unwrap(); + std::fs::write(root.join("small"), b"s2").unwrap(); + let tip = repo.commit("t", "c2").unwrap(); + drop(repo); + let id = |b: &[u8]| Object::blob(b.to_vec()).id(); + + let (client, server) = connect(root.clone()); + for (limit, old_big_sent, small_old_sent) in [ + (1024u64, false, true), + (0, false, false), + (1 << 20, true, true), + ] { + let mut pack = Vec::new(); + client + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&[], Some(limit)).as_ref(), + &mut pack, + ) + .unwrap(); + let ids: Vec<_> = scl_core::pack::parse_pack(&pack) + .unwrap() + .into_iter() + .map(|(i, _)| i) + .collect(); + assert!(ids.contains(&id(&new_big)), "limit {limit}: tip blob"); + assert!(ids.contains(&id(b"s2")), "limit {limit}: tip small blob"); + assert_eq!(ids.contains(&id(&old_big)), old_big_sent, "limit {limit}"); + assert_eq!(ids.contains(&id(b"s1")), small_old_sent, "limit {limit}"); + } + client.bye().unwrap(); + drop(client); + server.join().unwrap().unwrap(); + std::fs::remove_dir_all(&root).unwrap(); + } + #[test] fn handshake_surfaces_not_a_repo_as_typed_error() { let root = std::env::temp_dir().join(format!("scl-stdio-norepo-{}", std::process::id())); diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 6e171dc6..1f793784 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -737,7 +737,10 @@ pub(crate) fn preflight_view( let gap = |path: &str| { // With a blob limit the missing object is a file, and `sc backfill` // repairs it by its own path rather than a covering prefix (ADR-0055). - let prefix = if promisor.blob_limit().is_some() && !path.is_empty() { + // The limit is the cause only for a path in the clone's scope; a path + // outside it (a view widened past the filter) was left out by the filter. + let sized = promisor.blob_limit().filter(|_| promisor.matches(path)); + let prefix = if sized.is_some() && !path.is_empty() { path.to_string() } else { promisor @@ -762,7 +765,7 @@ pub(crate) fn preflight_view( path.to_string() }, prefix, - blob_limit: promisor.blob_limit(), + blob_limit: sized, } }; if !store.contains(&root) { diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 4e25ef6e..e6343510 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -532,8 +532,10 @@ too, not just the src-control-side metadata. ciphertext size, which anyone who can fetch the object already sees; the filter reveals nothing new. A limited `GetPack` makes the server stream-measure each distinct candidate blob once (`Store::blob_size`, fixed buffer, hash - checked, cached by id), bounded by the stored bytes of those blobs, which an - unfiltered `GetPack` reads anyway. Reads are not budgeted (#233 budgets writes) + checked, cached by id; a limit of 0 or `u64::MAX` measures nothing), bounded by + the stored bytes of those blobs. A blob that is kept is read twice, once to + measure and once to write the pack, so the read cost is at most twice an + unfiltered `GetPack`'s, which already reads every blob into its pack. Reads are not budgeted (#233 budgets writes) and the filter does not change that; the listener limits are the bound. Every active filter, a limit alone included, still excludes private branches and is refused by bucket remotes. The wire bump (version 6) makes a mixed-version pair diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md index c4c1a3ff..8d05b3a4 100644 --- a/docs/adr/0055-blob-size-clone-filter.md +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -236,16 +236,22 @@ the file's own path), and a workspace session on a size-limited clone. - **Sealed objects, signatures, transcripts.** Not path-reached blobs; never subject to the limit. - **Sparse and prefix filters.** The prefix decides scope first, the limit prunes - inside it. A size-only clone leaves `.sc/sparse` empty (a full view). + inside it. A size-only clone leaves `.sc/sparse` empty (a full view). It may + narrow that view with `sc sparse set` and return to it with `sc sparse set` + (no prefixes) or `sc sparse disable`: a full view is its starting state, so + those two refusals apply to prefix scopes only, and the closure is filled + (lazy) or refused with `GapInView` before anything changes. - **Push, verify, gc.** Presence-based (ADR-0048). - **Merge, pick, rebase, ws, work** (ADR-0049/0050). No new engine code; a size gap is "asked for" like any other (decision 7). - **Server cost and limits (D5).** The walk measures a blob with `Store::blob_size` (streamed through a fixed buffer, hash-verified, never admitted to the object cache) and caches each verdict by blob id, so a distinct blob is measured at - most once per request. The work is bounded by the stored bytes of the distinct - candidate blobs, which an unfiltered `GetPack` reads anyway to build its pack; - the filter changes bandwidth, not read cost. Reads are unbudgeted (#233 budgets + most once per request (a limit of 0 or `u64::MAX` measures nothing). The work + is bounded by the stored bytes of the distinct candidate blobs; a kept blob is + read twice (measure, then pack), so the read cost is at most twice that of an + unfiltered `GetPack`, which already reads every blob into its pack. The filter + changes bandwidth, not the order of magnitude of the work. Reads are unbudgeted (#233 budgets writes), and the filter does not change that: `--max-connections`, `--timeout` and the listener limits remain the bounds. A size index on the server is a later optimization. From 6345f12b40fb1ba4c4657ebe936c050c68dcfe54 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 21:06:12 +0300 Subject: [PATCH 5/7] test,docs: assert fixture cleanup, split the CLAUDE.md refusal clauses (ref #217) Addresses CodeRabbit review on #262. --- CLAUDE.md | 2 +- crates/cli/tests/blob_limit_filter.rs | 23 ++++++++++++++--------- crates/repo/src/partial_size.rs | 26 +++++++++++++++----------- 3 files changed, 30 insertions(+), 21 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 633c6855..89c9d1c8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -185,7 +185,7 @@ transport-adjacent. The rest, imperatively: renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or identity material. -- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (both allowed on a size-only `AllPaths` clone, ADR-0055; and, on a non-lazy clone, one outside the filter), `sc rewrap`, +- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and `sparse disable`; both are allowed on a size-only `AllPaths` clone, ADR-0055), on a non-lazy clone a `sparse set` outside the filter, `sc rewrap`, `sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge, cherry-pick or rebase whose conflict lies outside the view (a `.theirs` sidecar sits beside its conflicted path, so it is in view whenever that path is; ADR-0049) (no ref, state or working-tree change; on every clone, a conflict stop that needs sealing outside the view is refused the same way), `sc push` to a git remote, and `sc fetch` from any native remote but the promisor origin.** `sc merge`, `sc cherry-pick` and `sc rebase` work, including conflicts inside the view (P37c). `sc ws` and `sc work` work too (ADR-0050): workspaces and agents see the clone's view only, and harvest lands through the same merge. `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what diff --git a/crates/cli/tests/blob_limit_filter.rs b/crates/cli/tests/blob_limit_filter.rs index b3bc0c49..d4129dc6 100644 --- a/crates/cli/tests/blob_limit_filter.rs +++ b/crates/cli/tests/blob_limit_filter.rs @@ -25,6 +25,16 @@ fn tmp(tag: &str) -> PathBuf { d } +/// Remove fixture directories and assert they are gone. +fn remove_all(dirs: &[&Path]) { + for d in dirs { + if d.exists() { + std::fs::remove_dir_all(d).unwrap(); + } + assert!(!d.exists(), "fixture {} must be removed", d.display()); + } +} + const OLD: &[u8] = &[b'o'; 4096]; const NEW: &[u8] = &[b'n'; 4096]; @@ -76,8 +86,7 @@ fn size_limited_clone_checks_out_complete_and_reports_the_gap() { ); assert!(sc(&dst, &["gc"]).status.success()); assert!(sc(&dst, &["status"]).status.success()); - std::fs::remove_dir_all(&src).unwrap(); - std::fs::remove_dir_all(&dst).unwrap(); + remove_all(&[&src, &dst]); } #[test] @@ -129,9 +138,7 @@ fn lazy_and_backfill_repair_a_size_gap() { assert!(all.status.success(), "{}", stderr(&all)); assert!(!dst2.join(".sc/promisor").exists()); assert!(!dst2.join(".sc/promisor-blob-limit").exists()); - for d in [&src, &dst, &dst2] { - std::fs::remove_dir_all(d).unwrap(); - } + remove_all(&[&src, &dst, &dst2]); } #[test] @@ -162,7 +169,7 @@ fn filter_grammar_errors_are_actionable() { "{}", stdout(&help) ); - std::fs::remove_dir_all(&src).unwrap(); + remove_all(&[&src]); } #[test] @@ -190,7 +197,5 @@ fn a_bucket_remote_refuses_a_size_only_clone() { "{}", stderr(&out) ); - for d in [&src, &bucket, &dst] { - let _ = std::fs::remove_dir_all(d); - } + remove_all(&[&src, &bucket, &dst]); } diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs index 53d714ca..20ba431a 100644 --- a/crates/repo/src/partial_size.rs +++ b/crates/repo/src/partial_size.rs @@ -38,6 +38,16 @@ fn text(edits: &[(usize, &str)]) -> Vec { out.into_bytes() } +/// Remove fixture directories and assert they are gone. +fn remove_all(dirs: &[&std::path::Path]) { + for d in dirs { + if d.exists() { + std::fs::remove_dir_all(d).unwrap(); + } + assert!(!d.exists(), "fixture {} must be removed", d.display()); + } +} + fn huge(tag: u8) -> Vec { vec![tag; 8192] } @@ -200,9 +210,7 @@ fn boundary_is_inclusive_and_zero_keeps_only_tip_blobs() { ); drop(zero); drop(origin); - for r in [&origin_root, &clone_root, &zero_root] { - std::fs::remove_dir_all(r).unwrap(); - } + remove_all(&[&origin_root, &clone_root, &zero_root]); } /// A clone that combines a prefix with a limit: the prefix decides scope, the @@ -312,7 +320,7 @@ fn scope_and_limit_round_trip_through_the_markers() { crate::promisor::store(&layout, &crate::Promisor::new("o", vec![])).unwrap(); std::fs::write(layout.promisor_blob_limit_path(), "limit nope\n").unwrap(); assert!(crate::promisor::load(&layout).is_err()); - std::fs::remove_dir_all(&root).unwrap(); + remove_all(&[&root]); } /// Widening an all-paths scope is a no-op, and widening a prefix scope keeps @@ -399,9 +407,7 @@ fn gc_verify_and_push_work_on_a_size_limited_clone() { assert_eq!(origin.head_tip().unwrap(), Some(tip)); assert!(origin.snapshot(&tip).is_ok()); drop(origin); - for r in [&origin_root, &clone_root] { - std::fs::remove_dir_all(r).unwrap(); - } + remove_all(&[&origin_root, &clone_root]); } /// A destination that is not a mirror lacks the gapped history, so the push is @@ -427,7 +433,7 @@ fn push_to_a_non_mirror_refuses_on_a_size_gap() { other => panic!("expected PushGap, got {other:?}"), } drop(clone); - std::fs::remove_dir_all(&other_root).unwrap(); + remove_all(&[&other_root]); sz.cleanup(); } @@ -494,9 +500,7 @@ fn a_tip_root_equal_to_an_older_subtree_is_still_expanded() { ); drop(clone); drop(origin); - for r in [&origin_root, &clone_root] { - std::fs::remove_dir_all(r).unwrap(); - } + remove_all(&[&origin_root, &clone_root]); } /// The walk itself: limit semantics at the `reachable` level, both orderings of From a34eafb8ba07242c39f43ef6e5523f29cf7d745f Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 21:54:42 +0300 Subject: [PATCH 6/7] fix(repo): harden blob filter parsing and gap messages (ref #217) - blob: is reserved case-insensitively and untrimmed, so ' Blob:limit=1KiB' is the filter, not a path prefix that matches nothing - a limit above 1 GiB filters nothing and is an error - reachable_objects_limited refuses a limit without a path filter - promisor::remove deletes .sc/promisor before the limit marker - gap errors blame blob:limit only for a missing file in scope; a missing directory is named by path, a missing root tree points at backfill --all Refs #217 --- .claude/skills/sc-cli-reference/SKILL.md | 3 +- crates/repo/src/error.rs | 12 ++- crates/repo/src/fill.rs | 24 ++--- crates/repo/src/merge_tree.rs | 63 ++++++++---- crates/repo/src/partial_merge.rs | 9 +- crates/repo/src/partial_size.rs | 121 +++++++++++++++++++++++ crates/repo/src/promisor.rs | 32 ++++-- crates/repo/src/reachable.rs | 7 ++ crates/repo/src/worktree.rs | 22 +++-- docs/adr/0055-blob-size-clone-filter.md | 12 ++- 10 files changed, 252 insertions(+), 53 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 7b707e28..13973059 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -382,7 +382,8 @@ cargo run --bin sc -- clone --filter blob:limit= [--filter ] [- # left out (old revisions of large files), every tip # and the checkout stay complete; blob:limit=0 = no # blobs beyond the tips. `blob:` is reserved in - # --filter (blob:none is an error), once only. + # --filter (blob:none is an error), once only; a + # limit above 1 GiB is an error. # Server-evaluated: wire v6, upgrade both sides. # Repair: `sc backfill `, a lazy fill, or # `backfill --all`; fetch is not limit-aware diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index fa4aea62..3fff3442 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -53,7 +53,7 @@ pub enum Error { /// out of the reply. Raised before anything is written, so the working /// tree and refs are untouched. `prefix` is the filter prefix that /// `sc backfill` can act on. - #[error("{path} is missing from this partial clone's local objects ({}); nothing was changed. Run `sc backfill {prefix}` to repair it", gap_cause(*.blob_limit))] + #[error("{path} is missing from this partial clone's local objects ({}); nothing was changed. Run {} to repair it", gap_cause(*.blob_limit), repair_command(.prefix))] GapInView { path: String, prefix: String, @@ -212,3 +212,13 @@ fn gap_cause(blob_limit: Option) -> String { None => "an earlier fetch left a gap inside the checked-out view".to_string(), } } + +/// The `sc backfill` command that repairs a gap: a prefix names a path; a gap +/// at the root tree has none, and only a full backfill covers it. +fn repair_command(prefix: &str) -> String { + if prefix.is_empty() { + "`sc backfill --all`".to_string() + } else { + format!("`sc backfill {prefix}`") + } +} diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index 76581c69..a452d0b0 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -235,7 +235,7 @@ impl Repo { let mut filler = Filler::new(self, offline)?; let mut cur = root; for (i, comp) in comps.iter().enumerate() { - filler.ensure_present(&cur, path)?; + filler.ensure_present(&cur, path, true)?; let entry = { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); @@ -258,7 +258,7 @@ impl Repo { "{path} is a protected path; sc cat prints public content only" ))); } - filler.ensure_present(&entry.id, path)?; + filler.ensure_present(&entry.id, path, false)?; let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); return match store.get(&entry.id)? { @@ -286,13 +286,13 @@ impl NeedsFiller<'_> { /// can repair. Call it with the store lock released: the fill takes it. /// Fetched objects stay (they are immutable CAS additions), so a retry /// resumes from current presence. Returns what this call cost. - pub(crate) fn fill(&mut self, needs: &[(ObjectId, String)]) -> Result { - let Some((_, first)) = needs.first() else { + pub(crate) fn fill(&mut self, needs: &[crate::merge_tree::Need]) -> Result { + let Some(first) = needs.first() else { return Ok(FillStats::default()); }; let before = self.filler.stats; - let ids: BTreeSet = needs.iter().map(|(id, _)| *id).collect(); - if let Err(e) = self.filler.ensure_many(&ids, first) { + let ids: BTreeSet = needs.iter().map(|n| n.id).collect(); + if let Err(e) = self.filler.ensure_many(&ids, &first.path, first.tree) { return Err(match e { Error::GapOutsideFilter(p) if p.is_empty() => { crate::promisor::partial_clone_unsupported("merge") @@ -342,14 +342,14 @@ impl<'a> Filler<'a> { /// Ensure `id` is in the local store. `path` names what the caller is /// reading, for the gap error. - fn ensure_present(&mut self, id: &ObjectId, path: &str) -> Result<()> { + fn ensure_present(&mut self, id: &ObjectId, path: &str, tree: bool) -> Result<()> { let present = self.repo.vfs.store().lock().unwrap().contains(id); if present { return Ok(()); } let Some(origin) = self.origin.clone() else { return Err(match self.repo.promisor()? { - Some(p) => crate::promisor::gap_error(Some(&p), path), + Some(p) => crate::promisor::gap_error(Some(&p), path, tree), None => Error::Core(scl_core::Error::NotFound(*id)), }); }; @@ -366,7 +366,7 @@ impl<'a> Filler<'a> { /// Ensure every id in `ids` is in the local store, fetching the absent /// ones in batches. `path` names what the caller is laying out, for the /// gap error when filling is not allowed. - fn ensure_many(&mut self, ids: &BTreeSet, path: &str) -> Result<()> { + fn ensure_many(&mut self, ids: &BTreeSet, path: &str, tree: bool) -> Result<()> { let absent: Vec = { let store_arc = self.repo.vfs.store(); let store = store_arc.lock().unwrap(); @@ -380,7 +380,7 @@ impl<'a> Filler<'a> { }; let Some(origin) = self.origin.clone() else { return Err(match self.repo.promisor()? { - Some(p) => crate::promisor::gap_error(Some(&p), path), + Some(p) => crate::promisor::gap_error(Some(&p), path, tree), None => Error::Core(scl_core::Error::NotFound(*first)), }); }; @@ -400,7 +400,7 @@ impl<'a> Filler<'a> { /// subtree is read once per path. Progress is by presence, so an /// interrupted fill resumes where it stopped. fn fill_view(&mut self, root: ObjectId, sparse: &Sparse) -> Result<()> { - self.ensure_many(&BTreeSet::from([root]), "")?; + self.ensure_many(&BTreeSet::from([root]), "", true)?; let mut frontier: Vec<(ObjectId, String)> = vec![(root, String::new())]; // Under a path the view matches, every descendant is in view whatever // its path, so a repeated subtree is read once by id. Above that (the @@ -440,7 +440,7 @@ impl<'a> Filler<'a> { } } let hint = frontier.first().map(|(_, p)| p.clone()).unwrap_or_default(); - self.ensure_many(&want, &hint)?; + self.ensure_many(&want, &hint, true)?; frontier = next; } Ok(()) diff --git a/crates/repo/src/merge_tree.rs b/crates/repo/src/merge_tree.rs index f2186659..8174fb74 100644 --- a/crates/repo/src/merge_tree.rs +++ b/crates/repo/src/merge_tree.rs @@ -94,12 +94,22 @@ impl TreeMergeDone { } } +/// An object a pass could not read: where the traversal found it and whether +/// it is a directory (so a gap error does not blame a blob-size limit for a +/// missing tree). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct Need { + pub(crate) id: ObjectId, + pub(crate) path: String, + pub(crate) tree: bool, +} + /// The outcome of one [`merge_trees`] pass. #[derive(Debug)] pub(crate) enum TreeMerge { /// The pass needs these objects (subtrees or blobs the store lacks), each /// with the path at which the traversal found it. Fill them and call again. - Needs(Vec<(ObjectId, String)>), + Needs(Vec), Done(TreeMergeDone), } @@ -110,14 +120,18 @@ struct Walk<'a> { acc: MergeAcc, carried: Vec<(String, TreeEntry)>, carried_root: Option, - needs: Vec<(ObjectId, String)>, + needs: Vec, needs_seen: BTreeSet, } impl Walk<'_> { - fn need(&mut self, id: ObjectId, path: &str) { + fn need(&mut self, id: ObjectId, path: &str, tree: bool) { if self.needs_seen.insert(id) { - self.needs.push((id, path.to_string())); + self.needs.push(Need { + id, + path: path.to_string(), + tree, + }); } } @@ -260,7 +274,7 @@ impl crate::repo::Repo { /// store whose index disagrees with itself. fn fill_loop( mut pass: impl FnMut() -> Result, - mut fill: impl FnMut(&[(ObjectId, String)]) -> Result, + mut fill: impl FnMut(&[Need]) -> Result, ) -> Result<(TreeMergeDone, FillStats)> { let mut stats = FillStats::default(); let mut asked: BTreeSet = BTreeSet::new(); @@ -269,13 +283,13 @@ fn fill_loop( TreeMerge::Done(done) => return Ok((done, stats)), TreeMerge::Needs(needs) => needs, }; - if let Some((id, path)) = needs.iter().find(|(id, _)| asked.contains(id)) { + if let Some(Need { id, path, .. }) = needs.iter().find(|n| asked.contains(&n.id)) { return Err(Error::Protocol(format!( "merge asked again for object {id} (at {path:?}) after it was fetched; \ the object store and its index disagree" ))); } - asked.extend(needs.iter().map(|(id, _)| *id)); + asked.extend(needs.iter().map(|n| n.id)); let round = fill(&needs)?; stats.requests += round.requests; stats.objects += round.objects; @@ -300,7 +314,7 @@ fn visit(store: &mut Store, w: &mut Walk, path: &str, ents: [Option; Ok(()) => Ok(()), // A blob the leaf must read is absent: ask for it. Err(Error::Core(scl_core::Error::NotFound(id))) => { - w.need(id, path); + w.need(id, path, false); Ok(()) } Err(e) => Err(e), @@ -352,7 +366,7 @@ fn tree_node( let mut missing = false; for e in ents.iter().flatten() { if !store.contains(&e.id) { - w.need(e.id, path); + w.need(e.id, path, e.kind == EntryKind::Tree); missing = true; } } @@ -411,7 +425,7 @@ fn enumerate_mixed( let [b, o, t] = [&maps[0], &maps[1], &maps[2]].map(|m| m.get(&p).copied()); match merge_path(store, p.clone(), b, o, t, &w.prots, w.identity, &mut w.acc) { Ok(()) => {} - Err(Error::Core(scl_core::Error::NotFound(id))) => w.need(id, &p), + Err(Error::Core(scl_core::Error::NotFound(id))) => w.need(id, &p, false), Err(e) => return Err(e), } } @@ -429,7 +443,7 @@ fn flatten_present( let mut stack = vec![(root, prefix.to_string())]; while let Some((id, p)) = stack.pop() { if !store.contains(&id) { - w.need(id, &p); + w.need(id, &p, true); continue; } let tree = store.get_tree(&id)?; @@ -903,7 +917,7 @@ mod tests { cov.needed_fill += 1; } assert!(!list.is_empty()); - for (id, path) in list { + for Need { id, path, .. } in list { assert!( allowed.contains(&id), "seed {seed}: asked for {id} under a carried subtree" @@ -1120,7 +1134,14 @@ mod tests { let d_ours = tree_at(&mut s, ours, "d").unwrap(); s.delete(&d_ours).unwrap(); match merge_trees(&mut s, Some((base, &p)), (ours, &p), (theirs, &p), None).unwrap() { - TreeMerge::Needs(n) => assert_eq!(n, vec![(d_ours, "d".to_string())]), + TreeMerge::Needs(n) => assert_eq!( + n, + vec![Need { + id: d_ours, + path: "d".to_string(), + tree: true + }] + ), other => panic!("expected Needs, got {other:?}"), } } @@ -1152,13 +1173,21 @@ mod tests { }) } + fn blob_need(id: ObjectId, path: &str) -> Need { + Need { + id, + path: path.into(), + tree: false, + } + } + #[test] fn fill_loop_fills_then_finishes_and_sums_the_cost() { let a = scl_core::Object::blob(b"a".to_vec()).id(); let b = scl_core::Object::blob(b"b".to_vec()).id(); let mut passes = vec![ - TreeMerge::Needs(vec![(a, "x".into())]), - TreeMerge::Needs(vec![(b, "x/y".into())]), + TreeMerge::Needs(vec![blob_need(a, "x")]), + TreeMerge::Needs(vec![blob_need(b, "x/y")]), empty_done(), ] .into_iter(); @@ -1166,7 +1195,7 @@ mod tests { let (_, stats) = fill_loop( || Ok(passes.next().unwrap()), |needs| { - filled.extend(needs.iter().map(|(id, _)| *id)); + filled.extend(needs.iter().map(|n| n.id)); Ok(FillStats { requests: 1, objects: needs.len(), @@ -1191,7 +1220,7 @@ mod tests { let a = scl_core::Object::blob(b"a".to_vec()).id(); let mut rounds = 0; let err = fill_loop( - || Ok(TreeMerge::Needs(vec![(a, "x".into())])), + || Ok(TreeMerge::Needs(vec![blob_need(a, "x")])), |_| { rounds += 1; Ok(FillStats::default()) diff --git a/crates/repo/src/partial_merge.rs b/crates/repo/src/partial_merge.rs index 02082422..6037bdbe 100644 --- a/crates/repo/src/partial_merge.rs +++ b/crates/repo/src/partial_merge.rs @@ -859,7 +859,14 @@ fn an_interrupted_fill_changes_nothing_and_the_retry_succeeds() { // Round one by hand: fetch the docs trees of both sides, then lose the origin // before the blobs arrive. let docs = [&b1, &b2].map(|tip| dir_id(&fx.origin, tip, "docs")); - let needs: Vec<(ObjectId, String)> = docs.iter().map(|id| (*id, "docs".to_string())).collect(); + let needs: Vec = docs + .iter() + .map(|id| crate::merge_tree::Need { + id: *id, + path: "docs".to_string(), + tree: true, + }) + .collect(); clone.needs_filler().unwrap().fill(&needs).unwrap(); assert!(docs.iter().all(|id| has(&clone, id)), "trees arrived"); let gone = fx.origin_root.with_extension("gone"); diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs index 20ba431a..2aa6bb67 100644 --- a/crates/repo/src/partial_size.rs +++ b/crates/repo/src/partial_size.rs @@ -14,6 +14,7 @@ use crate::promisor::{PackFilter, Scope}; use crate::refs; use crate::replay::{PickResult, RebaseResult}; use crate::repo::Repo; +use crate::sparse::Sparse; const LIMIT: u64 = 1024; @@ -375,6 +376,126 @@ fn filter_grammar_rejects_duplicates_and_unknown_blob_forms() { assert!(other.to_string().contains("reserved"), "{other}"); assert!(parse(&v(&["blob:limit=lots"])).is_err()); assert!(parse(&v(&["blob:limit="])).is_err()); + + // The reserved grammar is matched case-insensitively and untrimmed, so a + // variant spelling is the size filter or an error, never a path prefix. + for spelling in ["Blob:limit=1KiB", " blob:limit=1KiB", "BLOB:LIMIT=1KiB"] { + assert_eq!( + parse(&v(&[spelling])).unwrap(), + (vec![], Some(1024)), + "{spelling:?}" + ); + } + for bad in ["Blob:none", " blob:oops", "BLOB:", "Blob:limit"] { + let e = parse(&v(&[bad])).unwrap_err(); + assert!(e.to_string().contains("blob:"), "{bad:?}: {e}"); + } + // Names that merely start with `blob` stay ordinary prefixes. + assert_eq!( + parse(&v(&["blobs/", "blob"])).unwrap(), + (vec!["blobs/".to_string(), "blob".to_string()], None) + ); + + // A limit above the largest transferable object filters nothing. + for huge in ["18446744073709551615", "2GiB", "1073741825"] { + let e = parse(&v(&[&format!("blob:limit={huge}")])).unwrap_err(); + assert!( + e.to_string().contains("would filter nothing"), + "{huge}: {e}" + ); + } + assert_eq!(parse(&v(&["blob:limit=1GiB"])).unwrap().1, Some(1 << 30)); +} + +/// A size limit with no path filter is a caller bug: the walk refuses instead +/// of silently ignoring the limit. +#[test] +fn limited_walk_without_a_filter_is_refused() { + let sz = Sz::new("nofilter"); + let arc = sz.origin.vfs().store(); + let mut store = arc.lock().unwrap(); + let err = crate::reachable::reachable_objects_limited(&mut *store, &[sz.c2], None, Some(1)) + .unwrap_err(); + assert!(matches!(err, Error::InvalidArgument(_)), "{err:?}"); + drop(store); + drop(arc); + sz.cleanup(); +} + +/// `promisor::remove` deletes `.sc/promisor` before the limit marker, so an +/// interrupted removal leaves a stray limit file that `load` ignores. +#[test] +fn remove_deletes_the_marker_before_the_limit_file() { + let root = std::env::temp_dir().join(format!("scl-p27sz-rm-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let layout = crate::layout::Layout::at(&root); + std::fs::create_dir_all(&layout.dot_sc).unwrap(); + crate::promisor::store(&layout, &crate::Promisor::with_blob_limit("o", vec![], 5)).unwrap(); + crate::promisor::remove(&layout).unwrap(); + assert!(!layout.promisor_path().exists()); + assert!(!layout.promisor_blob_limit_path().exists()); + // The state a crash between the two removals leaves: only the stray file. + std::fs::write( + layout.promisor_blob_limit_path(), + "limit 5\nscope all-paths\n", + ) + .unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), None); + remove_all(&[&root]); +} + +/// A missing directory (a fetch left it out) is not blamed on the blob limit +/// and is named by its path; a missing root tree points at `--all`, not at an +/// empty `sc backfill `. +#[test] +fn directory_and_root_gaps_do_not_blame_the_limit() { + let sz = Sz::new("dirgap"); + let clone = sz.clone(&[], LIMIT, false); + let src_tree = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + let root = s.get_snapshot(&sz.c2).unwrap().root; + s.get_tree(&root).unwrap().get("src").unwrap().id + }; + let root = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + s.delete(&src_tree).unwrap(); + s.get_snapshot(&sz.c2).unwrap().root + }; + let err = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + crate::worktree::preflight_view(clone.layout(), &mut s, root, &Sparse::default()) + .unwrap_err() + }; + match &err { + Error::GapInView { + path, + prefix, + blob_limit, + } => { + assert_eq!(path, "src"); + assert_eq!(prefix, "src", "names the path"); + assert_eq!(*blob_limit, None, "a directory is not a size gap"); + } + other => panic!("expected GapInView, got {other:?}"), + } + assert!(!err.to_string().contains("blob:limit"), "{err}"); + + let root_gap = Error::GapInView { + path: "(root tree)".into(), + prefix: String::new(), + blob_limit: None, + }; + assert!( + root_gap + .to_string() + .ends_with("Run `sc backfill --all` to repair it"), + "{root_gap}" + ); + drop(clone); + sz.cleanup(); } /// gc, verify and push on a size-limited clone (presence-based, ADR-0048). diff --git a/crates/repo/src/promisor.rs b/crates/repo/src/promisor.rs index 47d2ee7f..02e2b20c 100644 --- a/crates/repo/src/promisor.rs +++ b/crates/repo/src/promisor.rs @@ -212,7 +212,10 @@ pub fn parse_clone_filters(values: &[String]) -> Result<(Vec, Option Result<(Vec, Option scl_core::MAX_MAX_OBJECT_SIZE as u64 { + return Err(Error::InvalidArgument(format!( + "`blob:limit={size}` is above the largest transferable object ({} bytes, 1 GiB) \ + and would filter nothing; give a smaller limit, or omit the filter", + scl_core::MAX_MAX_OBJECT_SIZE + ))); + } + limit = Some(n); } Ok((prefixes, limit)) } @@ -249,11 +262,12 @@ pub fn partial_gap_hint(path: &str) -> Error { } /// The gap error for `path` on this partial clone: [`Error::GapBySize`] when -/// the clone has a blob-size limit and `path` is in its scope (the file's own -/// path is what `sc backfill` repairs), otherwise the prefix hint. -pub fn gap_error(promisor: Option<&Promisor>, path: &str) -> Error { +/// the clone has a blob-size limit, `path` is in its scope and the missing +/// object is a file (`tree` false; a missing directory was left out by a fetch, +/// not by the limit), otherwise the prefix hint naming the path. +pub fn gap_error(promisor: Option<&Promisor>, path: &str, tree: bool) -> Error { match promisor { - Some(p) if !path.is_empty() && p.matches(path) => match p.blob_limit() { + Some(p) if !tree && !path.is_empty() && p.matches(path) => match p.blob_limit() { Some(limit) => Error::GapBySize { path: path.to_string(), limit, @@ -388,9 +402,11 @@ fn remove_file_if_exists(path: &std::path::Path) -> Result<()> { /// after independently confirming the object closure is actually /// complete — this function does not itself verify anything. pub fn remove(layout: &Layout) -> Result<()> { - set_lazy(layout, false)?; + // `.sc/promisor` first: a crash after it leaves only a stray limit file, + // which `load` ignores, never a marker that lost its limit and matches nothing. + remove_file_if_exists(&layout.promisor_path())?; remove_file_if_exists(&layout.promisor_blob_limit_path())?; - remove_file_if_exists(&layout.promisor_path()) + set_lazy(layout, false) } /// True when this partial clone has opted in to filling gaps by object id diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index 9976a2ff..73d13bc5 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -161,6 +161,13 @@ pub fn reachable_objects_limited( filter: Option<&dyn PrefixFilter>, blob_limit: Option, ) -> Result { + // A limit needs a scope to apply to (an all-paths filter for a size-only + // walk). Failing here is safer than silently sending everything. + if blob_limit.is_some() && filter.is_none() { + return Err(Error::InvalidArgument( + "blob limit without a path filter".into(), + )); + } let mut gaps = BTreeSet::new(); let tip_set: HashSet = tips.iter().copied().collect(); // Size verdicts by blob id: a blob shared by many paths or versions is diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index 1f793784..c6336ffd 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -734,13 +734,15 @@ pub(crate) fn preflight_view( let Some(promisor) = crate::promisor::load(layout)? else { return Ok(()); }; - let gap = |path: &str| { - // With a blob limit the missing object is a file, and `sc backfill` - // repairs it by its own path rather than a covering prefix (ADR-0055). - // The limit is the cause only for a path in the clone's scope; a path - // outside it (a view widened past the filter) was left out by the filter. - let sized = promisor.blob_limit().filter(|_| promisor.matches(path)); - let prefix = if sized.is_some() && !path.is_empty() { + let gap = |path: &str, is_blob: bool| { + // With a blob limit `sc backfill` repairs a gap by its own path rather + // than a covering prefix (ADR-0055). The limit is named as a cause only + // for a missing file in the clone's scope: a missing directory, or a path + // outside the scope, was left out by a fetch or by the filter. + let sized = promisor + .blob_limit() + .filter(|_| is_blob && promisor.matches(path)); + let prefix = if promisor.blob_limit().is_some() && !path.is_empty() { path.to_string() } else { promisor @@ -769,7 +771,7 @@ pub(crate) fn preflight_view( } }; if !store.contains(&root) { - return Err(gap("")); + return Err(gap("", false)); } let mut stack = vec![(root, String::new())]; while let Some((tree_id, path)) = stack.pop() { @@ -783,13 +785,13 @@ pub(crate) fn preflight_view( match e.kind { EntryKind::Blob => { if sparse.matches(&child) && !store.contains(&e.id) { - return Err(gap(&child)); + return Err(gap(&child, true)); } } EntryKind::Tree => { if sparse.should_descend(&child) { if !store.contains(&e.id) { - return Err(gap(&child)); + return Err(gap(&child, false)); } stack.push((e.id, child)); } diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md index 8d05b3a4..b331d97c 100644 --- a/docs/adr/0055-blob-size-clone-filter.md +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -52,13 +52,16 @@ unless decision 4 keeps it". leaves out every non-empty blob outside the tips). `blob:limit=0` leaves out every blob no ref tip reaches. Size is the stored object's payload length. -**Grammar.** `blob:` is reserved in `--filter`: only `blob:limit=` is +**Grammar.** `blob:` is reserved in `--filter`, matched after trimming and +lower-casing (so `Blob:limit=1KiB` is the filter, never a path prefix that +matches nothing): only `blob:limit=` is accepted, once. A second `blob:limit` is an error. `blob:none` is an error that points at `blob:limit=0`. Any other `blob:` form is an error naming the one supported form. Cost: a path prefix that literally begins with `blob:` can no longer be named in `--filter`. `sc clone --help` documents this. `--lazy` requires `--filter`, which a lone `blob:limit=` satisfies, and works with it. -Rejected: a separate `--blob-limit` flag, which has no collision but diverges +A limit above the largest transferable object (1 GiB, `MAX_MAX_OBJECT_SIZE`) +filters nothing and is an error. Rejected: a separate `--blob-limit` flag, which has no collision but diverges from the issue and from git. ### 2. Deferred @@ -204,7 +207,10 @@ orders. pick, rebase, session or `sc cat` on a clone with a limit are `GapBySize { path, limit }`: "`` was left out of this partial clone (blob:limit=); run `sc backfill ` to fetch it first". A prefix-only - clone keeps every old message byte for byte. + clone keeps every old message byte for byte. The limit is named only for a + missing file in scope; a missing directory (a fetch left it out) is named by its + path, and a missing root tree points at `sc backfill --all` (merge needs carry + a tree/blob flag for this). - **Repair, lazy and online:** nothing to do. `ensure_view`, `sc cat` and the ADR-0049 merge fill by id. - **Repair, otherwise:** `sc backfill ` fetches every version of that path From 1be9f91b3554d8c18ed47cf3641a19c784b5a7ed Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 22:12:26 +0300 Subject: [PATCH 7/7] test,docs: named sessions on a size-limited clone (ref #217) Adapts the session tests to the named-session API, adds a concurrent named sessions test on a size-only clone, and notes ADR-0056 in ADR-0055. --- crates/repo/src/partial_size.rs | 57 +++++++++++++++++++++++-- docs/adr/0055-blob-size-clone-filter.md | 4 ++ 2 files changed, 57 insertions(+), 4 deletions(-) diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs index 2aa6bb67..a75f355b 100644 --- a/crates/repo/src/partial_size.rs +++ b/crates/repo/src/partial_size.rs @@ -874,10 +874,10 @@ fn replay_and_merge_with_a_size_gap_as_base() { #[test] fn workspace_session_on_a_size_limited_clone_lands() { use crate::ws::WsHarvestOutcome; - for lazy in [false, true] { - let sz = Sz::new(if lazy { "ws-lazy" } else { "ws" }); + for (lazy, name) in [(false, None), (true, None), (false, Some("named"))] { + let sz = Sz::new(&format!("ws-{lazy}-{}", name.unwrap_or("default"))); let clone = sz.clone(&[], LIMIT, lazy); - let session = clone.ws_fork(1, "t", None).unwrap(); + let session = clone.ws_fork(name, 1, "t", None).unwrap(); let ws = &session.workspaces[0].dir; assert_eq!( std::fs::read(ws.join("src/huge.bin")).unwrap(), @@ -891,7 +891,7 @@ fn workspace_session_on_a_size_limited_clone_lands() { ) .unwrap(); clone.commit("t", "host edit").unwrap(); - let outcomes = clone.ws_harvest(None, "t", None).unwrap(); + let outcomes = clone.ws_harvest(name, None, "t", None).unwrap(); assert!( matches!(outcomes.as_slice(), [WsHarvestOutcome::Landed { .. }]), "got {outcomes:?}" @@ -1047,3 +1047,52 @@ fn undo_to_a_gapped_revision_fills_when_lazy_and_refuses_otherwise() { sz.cleanup(); } } + +/// Named, concurrent sessions (ADR-0056) on a size-only clone: each records the +/// host's full view at fork, lays out the large tip file, and lands on its own; +/// the second harvest merges over the first. A fork always starts at a ref tip, +/// which is complete, so a size gap cannot be in a new session's base. +#[test] +fn named_sessions_on_a_size_limited_clone() { + use crate::ws::WsHarvestOutcome; + let sz = Sz::new("named-ws"); + let clone = sz.clone(&[], LIMIT, false); + let a = clone.ws_fork(Some("a"), 1, "t", None).unwrap(); + let b = clone.ws_fork(Some("b"), 1, "t", None).unwrap(); + assert!( + a.sparse.is_empty() && b.sparse.is_empty(), + "full view recorded" + ); + for sess in [&a, &b] { + assert_eq!( + std::fs::read(sess.workspaces[0].dir.join("src/huge.bin")).unwrap(), + huge(2) + ); + } + std::fs::write( + a.workspaces[0].dir.join("big.txt"), + text(&[(0, "ours"), (100, "from a")]), + ) + .unwrap(); + std::fs::write( + b.workspaces[0].dir.join("big.txt"), + text(&[(0, "ours"), (200, "from b")]), + ) + .unwrap(); + for name in ["a", "b"] { + let out = clone.ws_harvest(Some(name), None, "t", None).unwrap(); + assert!( + matches!(out.as_slice(), [WsHarvestOutcome::Landed { .. }]), + "{name}: {out:?}" + ); + } + let landed = String::from_utf8(std::fs::read(sz.clone_root.join("big.txt")).unwrap()).unwrap(); + assert!(landed.contains("from a\n") && landed.contains("from b\n")); + + assert!( + clone.ws_sessions().unwrap().is_empty(), + "both sessions ended" + ); + drop(clone); + sz.cleanup(); +} diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md index b331d97c..fd12b357 100644 --- a/docs/adr/0055-blob-size-clone-filter.md +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -250,6 +250,10 @@ the file's own path), and a workspace session on a size-limited clone. - **Push, verify, gc.** Presence-based (ADR-0048). - **Merge, pick, rebase, ws, work** (ADR-0049/0050). No new engine code; a size gap is "asked for" like any other (decision 7). + Named and concurrent sessions (ADR-0056) hold too: a session records the + host's view at fork, which for a size-only clone is the full view (`sparse` + empty), a fork starts at a ref tip so its base is complete, and each harvest + lands through the same merge. `named_sessions_on_a_size_limited_clone` pins it. - **Server cost and limits (D5).** The walk measures a blob with `Store::blob_size` (streamed through a fixed buffer, hash-verified, never admitted to the object cache) and caches each verdict by blob id, so a distinct blob is measured at