diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 202f35d4..c57a0672 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -382,6 +382,18 @@ cargo run --bin sc -- clone --filter # partial clone ( # prefixes; writes .sc/promisor + .sc/sparse to # the same prefixes; not supported over a # git-bridge remote +cargo run --bin sc -- clone --filter blob:limit= [--filter ] [--lazy] + # blob-size partial clone (ADR-0055, #217): + # is bytes or a KiB/MiB/GiB suffix; blobs of + # AT LEAST that size that no ref tip reaches are + # left out (old revisions of large files), every tip + # and the checkout stay complete; blob:limit=0 = no + # blobs beyond the tips. `blob:` is reserved in + # --filter (blob:none is an error), once only; a + # limit above 1 GiB is an error. + # Server-evaluated: wire v6, upgrade both sides. + # Repair: `sc backfill `, a lazy fill, or + # `backfill --all`; fetch is not limit-aware cargo run --bin sc -- backfill # widen a partial clone: fetch objects under # these prefixes from the promisor origin and # widen .sc/promisor; errors if this repo isn't @@ -408,7 +420,10 @@ bash demo/run_lazy_fetch_demo.sh # by-id fetch proof (P37a): cat of # exactly 3 objects (2 trees + blob), the sibling # stays unfetched, offline/non-lazy keep the gap # error, and a read-only sc+http server answers - # GetObjects (protocol v5) + # GetObjects +bash demo/run_blob_limit_demo.sh # blob-size filter proof (ADR-0055): old large + # blob left out, tip checkout whole, verify/gc/ + # push fine, --lazy, prefix+limit, backfill repair bash demo/run_partial_clone_demo.sh # partial clone proof (P27): sc clone --filter # src/ fetches fewer objects than a full clone # (object-store count + sc verify's gap report); diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 080920dc..35efef20 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -338,7 +338,8 @@ phases but the shape is stable: `transcript-drops` — the derived set of dropped transcripts (`sc transcript drop`), rebuilt from the store on clone. - `sparse` — the local, uncommitted sparse-checkout prefix spec (P24); - `promisor` — a partial clone's fetch-filter + origin (P27). + `promisor` — a partial clone's fetch-filter + origin (P27); + `promisor-blob-limit` — its blob-size limit and whole-tree-or-prefix scope (ADR-0055). - `serve-tokens.toml` — bearer-token access control for `sc serve --http` (P29); tokens may carry an expiry and be rotated in place (`sc serve token add --expires` / `rotate`). `serve-tls/` — the auto-minted `sc+https://` server @@ -1234,6 +1235,24 @@ by id (`ensure_view`) before switch, fast-forward, undo and `sparse set`; object cap; the client reads at most 4 GiB) and split when refused. `sc export` and `sparse disable` stay refused on a partial clone. See ADR-0048. +## Phase 27 follow-on — blob-size clone filter (built) + +`sc clone --filter blob:limit=` (alone or with prefixes) leaves out blobs +of at least `` bytes that no ref tip's root tree reaches. Only the server +knows a blob's size before sending it (tree entries carry none), so +`GetPack` gained a trailing optional `blob_limit` and `PROTOCOL_VERSION` is 6. +The sender's walk, `reachable_objects_limited`, walks every want's own tree +first (every ref tip stays complete, so the initial checkout is whole), then +moves over-limit blobs reached only through history from `included` to `gaps`, +measuring each distinct blob once with `Store::blob_size`. The client side adds +nothing to the gap model: absence is the gap (37b). `.sc/promisor-blob-limit` +records the limit and an explicit `AllPaths`/`Prefixes` scope (a size-only clone +is `AllPaths`, with a full sparse view); `PackFilter` is the single definition of +a filtered transfer (a prefix or a limit). A lazy size-only clone fills a gapped +revision by id even though its view is full; otherwise the gap error names the +file and `sc backfill ` repairs it. `sc fetch` is unfiltered, and the +object-count limit is deferred. See ADR-0055. + ## Phase 37c — merge, cherry-pick and rebase on a partial clone (built) `merge_trees` is a tree-recursive three-way merge that carries a subtree **by diff --git a/CLAUDE.md b/CLAUDE.md index fc9f471e..ae7ae088 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -142,7 +142,7 @@ the code, those win. | P24 | Sparse checkouts: `.sc/sparse` local prefix spec; checkout-only (CAS keeps everything) | [0034](docs/adr/0034-sparse-checkouts.md) | | P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) | | P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) | -| P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | +| P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3. Blob-size filter (#217): `--filter blob:limit=` (combinable with prefixes; `blob:` is reserved) leaves out blobs of at least that size unless a ref tip reaches them, evaluated by the server (`GetPack.blob_limit`, wire v6); the gap is absence as in ADR-0048, `sc backfill ` or a lazy fill repairs it, `.sc/promisor-blob-limit` records limit and scope | [0037](docs/adr/0037-partial-clone.md), [0055](docs/adr/0055-blob-size-clone-filter.md) | | P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | | P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; retained signed tombstones ride a transfer when their snapshot is in it or was collected by the sender, only verifying signatures are indexed, and held-but-untrusted tombstones are re-judged from `.sc/transcript-pending` when the trusted-signer set changes (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) | @@ -163,8 +163,11 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in `docs/THREAT-MODEL.md` — read it before touching anything crypto- or transport-adjacent. The rest, imperatively: -- **Wire protocol is version 5** (bumped from 4 in P37 for the additive - `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB; +- **Wire protocol is version 6** (bumped from 5 in #217 for `GetPack.blob_limit`, + ADR-0055; 4 to 5 in P37 for the additive `GetObjects` verb; 3 to 4 in P34 for + the sealed-branch object kinds). Servers and clients upgrade together: every + mixed pair fails closed at the handshake, ssh remotes included (they run the + remote host's own `sc` binary). The max object size (`scl_core::max_object_size()`, default 256 MiB; `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every @@ -184,7 +187,7 @@ transport-adjacent. The rest, imperatively: identity material. Opening a repository can write the derived transcript-drop indexes (`.sc/transcript-drops`, `.sc/transcript-pending`) when the trusted signer set changed; no key material is involved (#230). -- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`, +- **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and `sparse disable`; both are allowed on a size-only `AllPaths` clone, ADR-0055), on a non-lazy clone a `sparse set` outside the filter, `sc rewrap`, `sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge, cherry-pick or rebase whose conflict lies outside the view (a `.theirs` sidecar sits beside its conflicted path, so it is in view whenever that path is; ADR-0049) (no ref, state or working-tree change; on every clone, a conflict stop that needs sealing outside the view is refused the same way), `sc push` to a git remote, and `sc fetch` from any native remote but the promisor origin.** `sc merge`, `sc cherry-pick` and `sc rebase` work, including conflicts inside the view (P37c). `sc ws` and `sc work` work too (ADR-0050): workspaces and agents see the clone's view only, and harvest lands through the same merge. `sc backfill --all` converts to a genuine full clone and re-enables them. The filter records what @@ -193,6 +196,13 @@ transport-adjacent. The rest, imperatively: or `sparse set` fetches what its view needs by id first; on a non-lazy or offline clone it fails with `GapInView` (nothing changed) and names the `sc backfill ` that repairs it. + A `--filter blob:limit=` clone (ADR-0055) is partial too: a blob of at + least that size is left out unless a ref tip reaches it, tips are complete at + clone time only, a later fetch can bring one that points at a gapped blob, and + with a limit the gap errors name the file's own path (`GapBySize`, or + `GapInView` with the limit as a cause). A size-only clone has an `AllPaths` + scope and a full sparse view, and a lazy one still fills on switch, undo, merge + and replay. `sc fetch` is not limit-aware, and the object-count limit is deferred. - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. @@ -313,7 +323,7 @@ transport-adjacent. The rest, imperatively: Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of truth — do not re-list items here); entries that name an issue mirror the open GitHub issues. Standing entries: `sc ws` workspace refresh (#215), -blob-size clone filters (#217), bucket compaction and gc (#216), the non-unix +a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), bucket compaction and gc (#216), the non-unix `.sc/local-key` ACL (#214), and the open sparse, private-branch and abort edge cases (#241, #246, #247, #248). diff --git a/README.md b/README.md index 7f478bee..01ed372b 100644 --- a/README.md +++ b/README.md @@ -169,8 +169,8 @@ via `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system`), bucket (`sc+wal://`, `sc+s3://`), and Git (including hosted GitHub) network transports, plus private branches sealed to a recipient set until publish and a read-only Tauri desktop browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds -27 independent end-to-end proofs, one per major capability. Remaining +28 independent end-to-end proofs, one per major capability. Remaining follow-ons (transcript drop delivery gaps, bucket compaction and gc, `sc ws` -workspace refresh, size/count clone filters, and similar hardening +workspace refresh, a limit-aware fetch and an object-count clone limit, and similar hardening items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open GitHub issues rather than being open MVP scope. diff --git a/ROADMAP.md b/ROADMAP.md index f09c39b5..af478562 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -593,6 +593,16 @@ across every phase. carried even under a rule); conflicts inside the view are marked and resolved as on a full clone, conflicts outside it are refused before any write. (ADR-0049.) +- **Phase 27 follow-on — Blob-size clone filter (#217).** `sc clone --filter + blob:limit=` (combinable with prefix filters; `blob:` is reserved) leaves + out blobs of at least that size that no ref tip's root tree reaches, so old + revisions of large files stay on the server while every tip and the checkout + are complete. The server evaluates it (`GetPack.blob_limit`, `PROTOCOL_VERSION` + 6; client and server upgrade together). The gap is absence (ADR-0048), recorded + by `.sc/promisor-blob-limit` with an explicit `AllPaths`/`Prefixes` scope; a lazy + clone fills by id (a full view included), and otherwise the gap error names the + file and `sc backfill ` repairs it. Proven by `demo/run_blob_limit_demo.sh`. + (ADR-0055.) - **Phase 37d — Agent sessions on a partial clone.** `sc ws` and `sc work` work on a partial clone: a workspace sees the host's sparse view, and harvest lands through the ADR-0049 merge. (ADR-0050.) @@ -666,6 +676,7 @@ under Deferred below. | **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) | | **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) | +| **P27 follow-on — Blob-size clone filter (#217)** | Clone a repo with large binary history without fetching old revisions | `sc clone --filter blob:limit=1MiB ` (with or without prefixes) leaves out old large blobs, the tip checkout is whole; verify/gc/push work; `--lazy` fills by id, `sc backfill ` repairs; wire v6; proven by `demo/run_blob_limit_demo.sh` | [0055](docs/adr/0055-blob-size-clone-filter.md) | | **P37d — Sessions on partial clones** | Agent workspaces on a partial clone | `sc ws fork/run/harvest/abandon` and `sc work` see the clone's sparse view; harvest lands through the ADR-0049 merge, conflicts fall back to `work-` | [0050](docs/adr/0050-sessions-on-partial-clones.md) | > **Prior art.** Phases P5–P9 adapt decisions from the sibling project @@ -790,10 +801,11 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in everything else reports a gap with the `sc backfill` repair. Dialing the promisor origin from any read path is still deferred (offline breakage, surprise latency, fetch-during-gc). -- **Blob-size and object-count clone filters (#217, P27).** `sc clone --filter` - is prefix-only; git's `--filter=blob:limit` size axis needs a decision on the - repair path for a size gap, where the filter is evaluated, and protected - content first. +- **Remainder of #217 (P27): a limit-aware `sc fetch` and an object-count clone + limit.** The blob-size filter shipped (ADR-0055); `sc fetch` stays unfiltered, + so the limit bounds the initial transfer, not later growth, and a limit-aware + fetch needs per-path exemptions for backfilled files. An object-count limit has + no stable definition (the transfer set is id-ordered) and no git precedent. - **`sc rewrap` on partial clones — refused (P37).** `Repo::rewrap` calls `refuse_on_partial`; a paths-only/streaming rebuild that would lift the refusal remains a possible follow-on. diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index f172d982..faf9e224 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -310,8 +310,13 @@ enum Cmd { git: bool, /// Partial clone (P27): only fetch objects under this path prefix /// (repeatable). Writes `.sc/promisor` + `.sc/sparse`; missing - /// objects can later be pulled in with `sc backfill`. Not supported - /// over a git-bridge remote. + /// objects can later be pulled in with `sc backfill`. Also accepts + /// `blob:limit=` (bytes, or a KiB/MiB/GiB suffix; ADR-0055), + /// once: blobs of at least that size are left out unless a ref tip + /// reaches them, so old versions of large files stay on the server. + /// `blob:` is reserved (a prefix cannot start with it); `blob:limit=0` + /// leaves out every blob no tip reaches. Not supported over a + /// git-bridge remote. #[arg(long)] filter: Vec, /// Mark the partial clone lazy (P37): `sc cat` may fetch the trees @@ -3009,10 +3014,7 @@ fn run_verify(refname: Option, require: bool) -> Result<()> { // EXPECTED to be absent on a partial clone, never missing/corrupt — print // a count instead of treating them as a verify failure. if let Some(gaps) = repo.partial_gap_count(&[tip])? { - let prefixes = repo - .promisor()? - .map(|p| p.prefixes().join(", ")) - .unwrap_or_default(); + let prefixes = repo.promisor()?.map(|p| p.describe()).unwrap_or_default(); print_line(&format!( "partial: {gaps} object(s) outside filter [{prefixes}]" )); @@ -3769,19 +3771,20 @@ fn run_clone( return run_clone_git(&src, &dst); } let src = absolute_local_source(src); - let filter_opt = (!filter.is_empty()).then_some(filter.as_slice()); - let repo = scl_repo::Repo::clone_url_opts(&src, &dst, filter_opt, no_transcripts)?; + let (prefixes, blob_limit) = scl_repo::promisor::parse_clone_filters(&filter)?; + let repo = + scl_repo::Repo::clone_url_limited(&src, &dst, &prefixes, blob_limit, no_transcripts)?; if lazy { scl_repo::promisor::set_lazy(repo.layout(), true)?; } let n = repo.branches()?.len(); - if filter_opt.is_some() { + if let Some(promisor) = repo.promisor()? { println!( "cloned {} into {} ({} branch(es), partial: {})", src, dst.display(), n, - filter.join(", ") + promisor.describe() ); } else { println!("cloned {} into {} ({} branch(es))", src, dst.display(), n); diff --git a/crates/cli/tests/blob_limit_filter.rs b/crates/cli/tests/blob_limit_filter.rs new file mode 100644 index 00000000..d4129dc6 --- /dev/null +++ b/crates/cli/tests/blob_limit_filter.rs @@ -0,0 +1,201 @@ +//! ADR-0055: `sc clone --filter blob:limit=` through the real binary. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn sc(dir: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_sc")) + .args(args) + .current_dir(dir) + .output() + .expect("sc runs") +} + +fn stdout(o: &Output) -> String { + String::from_utf8_lossy(&o.stdout).into_owned() +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-blob-limit-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + d +} + +/// Remove fixture directories and assert they are gone. +fn remove_all(dirs: &[&Path]) { + for d in dirs { + if d.exists() { + std::fs::remove_dir_all(d).unwrap(); + } + assert!(!d.exists(), "fixture {} must be removed", d.display()); + } +} + +const OLD: &[u8] = &[b'o'; 4096]; +const NEW: &[u8] = &[b'n'; 4096]; + +/// An origin whose first commit holds a 4 KiB `big.bin` that the second +/// replaced, plus a small file. +fn origin(tag: &str) -> PathBuf { + let root = tmp(&format!("{tag}-origin")); + std::fs::create_dir_all(&root).unwrap(); + assert!(sc(&root, &["init"]).status.success()); + std::fs::write(root.join("big.bin"), OLD).unwrap(); + std::fs::write(root.join("small.txt"), "one\n").unwrap(); + assert!(sc(&root, &["commit", "-m", "c1"]).status.success()); + std::fs::write(root.join("big.bin"), NEW).unwrap(); + assert!(sc(&root, &["commit", "-m", "c2"]).status.success()); + root +} + +#[test] +fn size_limited_clone_checks_out_complete_and_reports_the_gap() { + let src = origin("clone"); + let dst = tmp("clone-dst"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(out.status.success(), "{}", stderr(&out)); + assert!( + stdout(&out).contains("partial: blob:limit=1024"), + "{}", + stdout(&out) + ); + assert_eq!(std::fs::read(dst.join("big.bin")).unwrap(), NEW); + assert_eq!( + std::fs::read_to_string(dst.join(".sc/promisor-blob-limit")).unwrap(), + "limit 1024\nscope all-paths\n" + ); + let v = sc(&dst, &["verify"]); + assert!(v.status.success(), "{}", stderr(&v)); + let text = stdout(&v); + assert!( + text.contains("partial: 1 object(s) outside filter [blob:limit=1024]"), + "{text}" + ); + assert!(sc(&dst, &["gc"]).status.success()); + assert!(sc(&dst, &["status"]).status.success()); + remove_all(&[&src, &dst]); +} + +#[test] +fn lazy_and_backfill_repair_a_size_gap() { + let src = origin("lazy"); + let dst = tmp("lazy-dst"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + "--lazy", + ], + ); + assert!( + out.status.success(), + "--lazy works with only blob: {}", + stderr(&out) + ); + assert!(dst.join(".sc/promisor-lazy").exists()); + + // Non-lazy sibling: backfill by the file's own path repairs the gap. + let dst2 = tmp("lazy-dst2"); + let out = sc( + &src, + &[ + "clone", + src.to_str().unwrap(), + dst2.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(out.status.success(), "{}", stderr(&out)); + let b = sc(&dst2, &["backfill", "big.bin"]); + assert!(b.status.success(), "{}", stderr(&b)); + let v = stdout(&sc(&dst2, &["verify"])); + assert!(v.contains("partial: 0 object(s)"), "{v}"); + assert!( + std::fs::read_to_string(dst2.join(".sc/promisor-blob-limit")) + .unwrap() + .contains("scope all-paths"), + "backfill keeps the whole-tree scope" + ); + let all = sc(&dst2, &["backfill", "--all"]); + assert!(all.status.success(), "{}", stderr(&all)); + assert!(!dst2.join(".sc/promisor").exists()); + assert!(!dst2.join(".sc/promisor-blob-limit").exists()); + remove_all(&[&src, &dst, &dst2]); +} + +#[test] +fn filter_grammar_errors_are_actionable() { + let src = origin("grammar"); + let dst = tmp("grammar-dst"); + let run = |filters: &[&str]| { + let mut args = vec!["clone", src.to_str().unwrap(), dst.to_str().unwrap()]; + for f in filters { + args.push("--filter"); + args.push(f); + } + sc(&src, &args) + }; + let none = run(&["blob:none"]); + assert!(!none.status.success()); + assert!(stderr(&none).contains("blob:limit=0"), "{}", stderr(&none)); + let dup = run(&["blob:limit=1", "blob:limit=2"]); + assert!(!dup.status.success()); + assert!(stderr(&dup).contains("more than once"), "{}", stderr(&dup)); + let bad = run(&["blob:limit=lots"]); + assert!(!bad.status.success()); + assert!(!dst.exists(), "a rejected filter leaves no destination"); + // `sc clone --help` documents the grammar. + let help = sc(&src, &["clone", "--help"]); + assert!( + stdout(&help).contains("blob:limit="), + "{}", + stdout(&help) + ); + remove_all(&[&src]); +} + +#[test] +fn a_bucket_remote_refuses_a_size_only_clone() { + let src = origin("bucket"); + let bucket = tmp("bucket-wal"); + std::fs::create_dir_all(&bucket).unwrap(); + let url = format!("sc+wal://{}", bucket.display()); + assert!(sc(&src, &["remote", "add", "bk", &url]).status.success()); + assert!(sc(&src, &["push", "bk"]).status.success()); + let dst = tmp("bucket-dst"); + let out = sc( + &src, + &[ + "clone", + &url, + dst.to_str().unwrap(), + "--filter", + "blob:limit=1KiB", + ], + ); + assert!(!out.status.success(), "a bucket remote must refuse"); + assert!( + stderr(&out).contains("partial clone from bucket remotes"), + "{}", + stderr(&out) + ); + remove_all(&[&src, &bucket, &dst]); +} diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index b16c9043..a913311b 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -608,7 +608,7 @@ impl Transport for BucketTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { if filter.is_some() { @@ -922,7 +922,22 @@ mod tests { // filter is refused loudly, not ignored let filt = vec!["src/".to_string()]; assert!(t - .get_pack(&[tip], &[], Some(&filt), &mut Vec::new()) + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&filt, None).as_ref(), + &mut Vec::new() + ) + .is_err()); + // ... and so is a size-only filter (ADR-0055): "filter active" is + // `prefixes || blob_limit`, not "prefixes non-empty". + assert!(t + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&[], Some(1024)).as_ref(), + &mut Vec::new() + ) .is_err()); drop(t); std::fs::remove_dir_all(&broot).unwrap(); diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index 42af300c..3fff3442 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -53,8 +53,20 @@ pub enum Error { /// out of the reply. Raised before anything is written, so the working /// tree and refs are untouched. `prefix` is the filter prefix that /// `sc backfill` can act on. - #[error("{path} is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill {prefix}` to repair it")] - GapInView { path: String, prefix: String }, + #[error("{path} is missing from this partial clone's local objects ({}); nothing was changed. Run {} to repair it", gap_cause(*.blob_limit), repair_command(.prefix))] + GapInView { + path: String, + prefix: String, + /// The clone's `blob:limit` (ADR-0055), when it has one: then `prefix` + /// is the file's own path, and the message names the limit as a cause. + blob_limit: Option, + }, + /// A path inside a partial clone's scope is absent and the clone was made + /// with a blob-size limit (ADR-0055): the blob is a large old version the + /// clone left out, or a fetch brought a tip that points at one. `sc + /// backfill ` fetches every version of that file. + #[error("{path} was left out of this partial clone (blob:limit={limit}); run `sc backfill {path}` to fetch it first")] + GapBySize { path: String, limit: u64 }, /// A push found an object the local store lacks (referenced by `referrer`) /// that the destination does not hold either (ADR-0048). Pushing the new /// tip would leave the destination with a tree that names an object it @@ -190,3 +202,23 @@ pub enum Error { } pub type Result = std::result::Result; + +/// The cause clause of [`Error::GapInView`]. +fn gap_cause(blob_limit: Option) -> String { + match blob_limit { + Some(limit) => format!( + "blob:limit={limit} left this out of the clone, or an earlier fetch left a gap inside the checked-out view" + ), + None => "an earlier fetch left a gap inside the checked-out view".to_string(), + } +} + +/// The `sc backfill` command that repairs a gap: a prefix names a path; a gap +/// at the root tree has none, and only a full backfill covers it. +fn repair_command(prefix: &str) -> String { + if prefix.is_empty() { + "`sc backfill --all`".to_string() + } else { + format!("`sc backfill {prefix}`") + } +} diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs index af05851d..a452d0b0 100644 --- a/crates/repo/src/fill.rs +++ b/crates/repo/src/fill.rs @@ -157,8 +157,11 @@ impl Repo { offline: bool, ) -> Result { // An empty spec is a full checkout: never fill the whole tree on a - // partial clone's behalf (the preflight reports the gap instead). - if sparse.is_full() { + // partial clone's behalf (the preflight reports the gap instead). A + // size-only clone (ADR-0055) is the exception: its view is always full + // and what is missing under one root is exactly the large old blobs + // the operation needs, so the fill is bounded by that one revision. + if sparse.is_full() && !self.promisor()?.is_some_and(|p| p.is_all_paths()) { return Ok(FillStats::default()); } let mut filler = Filler::new(self, offline)?; @@ -232,7 +235,7 @@ impl Repo { let mut filler = Filler::new(self, offline)?; let mut cur = root; for (i, comp) in comps.iter().enumerate() { - filler.ensure_present(&cur, path)?; + filler.ensure_present(&cur, path, true)?; let entry = { let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); @@ -255,7 +258,7 @@ impl Repo { "{path} is a protected path; sc cat prints public content only" ))); } - filler.ensure_present(&entry.id, path)?; + filler.ensure_present(&entry.id, path, false)?; let store_arc = self.vfs.store(); let mut store = store_arc.lock().unwrap(); return match store.get(&entry.id)? { @@ -283,13 +286,13 @@ impl NeedsFiller<'_> { /// can repair. Call it with the store lock released: the fill takes it. /// Fetched objects stay (they are immutable CAS additions), so a retry /// resumes from current presence. Returns what this call cost. - pub(crate) fn fill(&mut self, needs: &[(ObjectId, String)]) -> Result { - let Some((_, first)) = needs.first() else { + pub(crate) fn fill(&mut self, needs: &[crate::merge_tree::Need]) -> Result { + let Some(first) = needs.first() else { return Ok(FillStats::default()); }; let before = self.filler.stats; - let ids: BTreeSet = needs.iter().map(|(id, _)| *id).collect(); - if let Err(e) = self.filler.ensure_many(&ids, first) { + let ids: BTreeSet = needs.iter().map(|n| n.id).collect(); + if let Err(e) = self.filler.ensure_many(&ids, &first.path, first.tree) { return Err(match e { Error::GapOutsideFilter(p) if p.is_empty() => { crate::promisor::partial_clone_unsupported("merge") @@ -339,16 +342,15 @@ impl<'a> Filler<'a> { /// Ensure `id` is in the local store. `path` names what the caller is /// reading, for the gap error. - fn ensure_present(&mut self, id: &ObjectId, path: &str) -> Result<()> { + fn ensure_present(&mut self, id: &ObjectId, path: &str, tree: bool) -> Result<()> { let present = self.repo.vfs.store().lock().unwrap().contains(id); if present { return Ok(()); } let Some(origin) = self.origin.clone() else { - return Err(if self.repo.promisor()?.is_some() { - crate::promisor::partial_gap_hint(path) - } else { - Error::Core(scl_core::Error::NotFound(*id)) + return Err(match self.repo.promisor()? { + Some(p) => crate::promisor::gap_error(Some(&p), path, tree), + None => Error::Core(scl_core::Error::NotFound(*id)), }); }; if self.transport.is_none() { @@ -364,7 +366,7 @@ impl<'a> Filler<'a> { /// Ensure every id in `ids` is in the local store, fetching the absent /// ones in batches. `path` names what the caller is laying out, for the /// gap error when filling is not allowed. - fn ensure_many(&mut self, ids: &BTreeSet, path: &str) -> Result<()> { + fn ensure_many(&mut self, ids: &BTreeSet, path: &str, tree: bool) -> Result<()> { let absent: Vec = { let store_arc = self.repo.vfs.store(); let store = store_arc.lock().unwrap(); @@ -377,10 +379,9 @@ impl<'a> Filler<'a> { return Ok(()); }; let Some(origin) = self.origin.clone() else { - return Err(if self.repo.promisor()?.is_some() { - crate::promisor::partial_gap_hint(path) - } else { - Error::Core(scl_core::Error::NotFound(*first)) + return Err(match self.repo.promisor()? { + Some(p) => crate::promisor::gap_error(Some(&p), path, tree), + None => Error::Core(scl_core::Error::NotFound(*first)), }); }; if self.transport.is_none() { @@ -399,7 +400,7 @@ impl<'a> Filler<'a> { /// subtree is read once per path. Progress is by presence, so an /// interrupted fill resumes where it stopped. fn fill_view(&mut self, root: ObjectId, sparse: &Sparse) -> Result<()> { - self.ensure_many(&BTreeSet::from([root]), "")?; + self.ensure_many(&BTreeSet::from([root]), "", true)?; let mut frontier: Vec<(ObjectId, String)> = vec![(root, String::new())]; // Under a path the view matches, every descendant is in view whatever // its path, so a repeated subtree is read once by id. Above that (the @@ -439,7 +440,7 @@ impl<'a> Filler<'a> { } } let hint = frontier.first().map(|(_, p)| p.clone()).unwrap_or_default(); - self.ensure_many(&want, &hint)?; + self.ensure_many(&want, &hint, true)?; frontier = next; } Ok(()) @@ -759,7 +760,7 @@ mod tests { &self, _: &[ObjectId], _: &[ObjectId], - _: Option<&[String]>, + _: Option<&crate::promisor::PackFilter>, _: &mut dyn Write, ) -> Result<()> { unimplemented!() @@ -866,7 +867,7 @@ mod tests { &self, w: &[ObjectId], h: &[ObjectId], - f: Option<&[String]>, + f: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.inner.get_pack(w, h, f, out) @@ -951,7 +952,7 @@ mod tests { &self, _: &[ObjectId], _: &[ObjectId], - _: Option<&[String]>, + _: Option<&crate::promisor::PackFilter>, _: &mut dyn Write, ) -> Result<()> { unimplemented!() diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index f0f853af..58680f9e 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -646,7 +646,7 @@ impl Transport for HttpTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.client.get_pack(wants, haves, filter, out) @@ -2784,6 +2784,7 @@ mod tests { wants: vec![tip], haves: vec![], filter: vec![], + blob_limit: None, } .encode(), ) diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index 90e82fe3..a19318d4 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -122,6 +122,13 @@ impl Layout { pub fn promisor_path(&self) -> PathBuf { self.dot_sc.join("promisor") } + /// `.sc/promisor-blob-limit` — the blob-size limit a partial clone was + /// made with and whether its scope is every path (ADR-0055). A separate + /// file for the same reason as `promisor-lazy`: a line in `.sc/promisor` + /// is a path prefix, and any text is a legal path. + pub fn promisor_blob_limit_path(&self) -> PathBuf { + self.dot_sc.join("promisor-blob-limit") + } /// `.sc/promisor-lazy` — empty marker (P37): this partial clone may fill /// gaps from its promisor remote by object id (ADR-0047). A separate file, /// not a field in `.sc/promisor`, because that file's lines after diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index c3819dd3..9edff190 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -27,6 +27,8 @@ mod partial_replay; #[cfg(test)] mod partial_sessions; #[cfg(test)] +mod partial_size; +#[cfg(test)] mod partial_walks; pub mod pick_state; pub mod private; @@ -71,7 +73,7 @@ pub use gc::GcStats; pub use git_marks::MarksStore; pub use http_transport::ScHttpUrl; pub use oplog::{OpRecord, UndoOutcome}; -pub use promisor::Promisor; +pub use promisor::{PackFilter, Promisor, Scope}; pub use protect_ops::PrefixRecipient; pub use remote::{RemoteConfig, RemoteKind}; pub use replay::{PickResult, RebaseResult}; diff --git a/crates/repo/src/merge_tree.rs b/crates/repo/src/merge_tree.rs index f2186659..8174fb74 100644 --- a/crates/repo/src/merge_tree.rs +++ b/crates/repo/src/merge_tree.rs @@ -94,12 +94,22 @@ impl TreeMergeDone { } } +/// An object a pass could not read: where the traversal found it and whether +/// it is a directory (so a gap error does not blame a blob-size limit for a +/// missing tree). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct Need { + pub(crate) id: ObjectId, + pub(crate) path: String, + pub(crate) tree: bool, +} + /// The outcome of one [`merge_trees`] pass. #[derive(Debug)] pub(crate) enum TreeMerge { /// The pass needs these objects (subtrees or blobs the store lacks), each /// with the path at which the traversal found it. Fill them and call again. - Needs(Vec<(ObjectId, String)>), + Needs(Vec), Done(TreeMergeDone), } @@ -110,14 +120,18 @@ struct Walk<'a> { acc: MergeAcc, carried: Vec<(String, TreeEntry)>, carried_root: Option, - needs: Vec<(ObjectId, String)>, + needs: Vec, needs_seen: BTreeSet, } impl Walk<'_> { - fn need(&mut self, id: ObjectId, path: &str) { + fn need(&mut self, id: ObjectId, path: &str, tree: bool) { if self.needs_seen.insert(id) { - self.needs.push((id, path.to_string())); + self.needs.push(Need { + id, + path: path.to_string(), + tree, + }); } } @@ -260,7 +274,7 @@ impl crate::repo::Repo { /// store whose index disagrees with itself. fn fill_loop( mut pass: impl FnMut() -> Result, - mut fill: impl FnMut(&[(ObjectId, String)]) -> Result, + mut fill: impl FnMut(&[Need]) -> Result, ) -> Result<(TreeMergeDone, FillStats)> { let mut stats = FillStats::default(); let mut asked: BTreeSet = BTreeSet::new(); @@ -269,13 +283,13 @@ fn fill_loop( TreeMerge::Done(done) => return Ok((done, stats)), TreeMerge::Needs(needs) => needs, }; - if let Some((id, path)) = needs.iter().find(|(id, _)| asked.contains(id)) { + if let Some(Need { id, path, .. }) = needs.iter().find(|n| asked.contains(&n.id)) { return Err(Error::Protocol(format!( "merge asked again for object {id} (at {path:?}) after it was fetched; \ the object store and its index disagree" ))); } - asked.extend(needs.iter().map(|(id, _)| *id)); + asked.extend(needs.iter().map(|n| n.id)); let round = fill(&needs)?; stats.requests += round.requests; stats.objects += round.objects; @@ -300,7 +314,7 @@ fn visit(store: &mut Store, w: &mut Walk, path: &str, ents: [Option; Ok(()) => Ok(()), // A blob the leaf must read is absent: ask for it. Err(Error::Core(scl_core::Error::NotFound(id))) => { - w.need(id, path); + w.need(id, path, false); Ok(()) } Err(e) => Err(e), @@ -352,7 +366,7 @@ fn tree_node( let mut missing = false; for e in ents.iter().flatten() { if !store.contains(&e.id) { - w.need(e.id, path); + w.need(e.id, path, e.kind == EntryKind::Tree); missing = true; } } @@ -411,7 +425,7 @@ fn enumerate_mixed( let [b, o, t] = [&maps[0], &maps[1], &maps[2]].map(|m| m.get(&p).copied()); match merge_path(store, p.clone(), b, o, t, &w.prots, w.identity, &mut w.acc) { Ok(()) => {} - Err(Error::Core(scl_core::Error::NotFound(id))) => w.need(id, &p), + Err(Error::Core(scl_core::Error::NotFound(id))) => w.need(id, &p, false), Err(e) => return Err(e), } } @@ -429,7 +443,7 @@ fn flatten_present( let mut stack = vec![(root, prefix.to_string())]; while let Some((id, p)) = stack.pop() { if !store.contains(&id) { - w.need(id, &p); + w.need(id, &p, true); continue; } let tree = store.get_tree(&id)?; @@ -903,7 +917,7 @@ mod tests { cov.needed_fill += 1; } assert!(!list.is_empty()); - for (id, path) in list { + for Need { id, path, .. } in list { assert!( allowed.contains(&id), "seed {seed}: asked for {id} under a carried subtree" @@ -1120,7 +1134,14 @@ mod tests { let d_ours = tree_at(&mut s, ours, "d").unwrap(); s.delete(&d_ours).unwrap(); match merge_trees(&mut s, Some((base, &p)), (ours, &p), (theirs, &p), None).unwrap() { - TreeMerge::Needs(n) => assert_eq!(n, vec![(d_ours, "d".to_string())]), + TreeMerge::Needs(n) => assert_eq!( + n, + vec![Need { + id: d_ours, + path: "d".to_string(), + tree: true + }] + ), other => panic!("expected Needs, got {other:?}"), } } @@ -1152,13 +1173,21 @@ mod tests { }) } + fn blob_need(id: ObjectId, path: &str) -> Need { + Need { + id, + path: path.into(), + tree: false, + } + } + #[test] fn fill_loop_fills_then_finishes_and_sums_the_cost() { let a = scl_core::Object::blob(b"a".to_vec()).id(); let b = scl_core::Object::blob(b"b".to_vec()).id(); let mut passes = vec![ - TreeMerge::Needs(vec![(a, "x".into())]), - TreeMerge::Needs(vec![(b, "x/y".into())]), + TreeMerge::Needs(vec![blob_need(a, "x")]), + TreeMerge::Needs(vec![blob_need(b, "x/y")]), empty_done(), ] .into_iter(); @@ -1166,7 +1195,7 @@ mod tests { let (_, stats) = fill_loop( || Ok(passes.next().unwrap()), |needs| { - filled.extend(needs.iter().map(|(id, _)| *id)); + filled.extend(needs.iter().map(|n| n.id)); Ok(FillStats { requests: 1, objects: needs.len(), @@ -1191,7 +1220,7 @@ mod tests { let a = scl_core::Object::blob(b"a".to_vec()).id(); let mut rounds = 0; let err = fill_loop( - || Ok(TreeMerge::Needs(vec![(a, "x".into())])), + || Ok(TreeMerge::Needs(vec![blob_need(a, "x")])), |_| { rounds += 1; Ok(FillStats::default()) diff --git a/crates/repo/src/partial_merge.rs b/crates/repo/src/partial_merge.rs index 02082422..6037bdbe 100644 --- a/crates/repo/src/partial_merge.rs +++ b/crates/repo/src/partial_merge.rs @@ -859,7 +859,14 @@ fn an_interrupted_fill_changes_nothing_and_the_retry_succeeds() { // Round one by hand: fetch the docs trees of both sides, then lose the origin // before the blobs arrive. let docs = [&b1, &b2].map(|tip| dir_id(&fx.origin, tip, "docs")); - let needs: Vec<(ObjectId, String)> = docs.iter().map(|id| (*id, "docs".to_string())).collect(); + let needs: Vec = docs + .iter() + .map(|id| crate::merge_tree::Need { + id: *id, + path: "docs".to_string(), + tree: true, + }) + .collect(); clone.needs_filler().unwrap().fill(&needs).unwrap(); assert!(docs.iter().all(|id| has(&clone, id)), "trees arrived"); let gone = fx.origin_root.with_extension("gone"); diff --git a/crates/repo/src/partial_size.rs b/crates/repo/src/partial_size.rs new file mode 100644 index 00000000..a75f355b --- /dev/null +++ b/crates/repo/src/partial_size.rs @@ -0,0 +1,1098 @@ +//! Tests for the blob-size clone filter (ADR-0055, `--filter blob:limit=`). +//! +//! The fixture is an origin whose history has large blobs that only an old +//! commit references, so a size-limited clone leaves them out while every ref +//! tip stays complete. + +use std::path::PathBuf; +use std::time::Duration; + +use scl_core::{Object, ObjectId}; + +use crate::error::Error; +use crate::promisor::{PackFilter, Scope}; +use crate::refs; +use crate::replay::{PickResult, RebaseResult}; +use crate::repo::Repo; +use crate::sparse::Sparse; + +const LIMIT: u64 = 1024; + +fn blob_id(bytes: &[u8]) -> ObjectId { + Object::blob(bytes.to_vec()).id() +} + +fn has(repo: &Repo, id: &ObjectId) -> bool { + repo.vfs().store().lock().unwrap().contains(id) +} + +/// 300 numbered lines (about 2.7 KB, over `LIMIT`) with `edits` replacing +/// lines by index, so two edits to different lines merge cleanly. +fn text(edits: &[(usize, &str)]) -> Vec { + let mut out = String::new(); + for n in 0..300 { + match edits.iter().find(|(i, _)| *i == n) { + Some((_, replacement)) => out.push_str(&format!("{replacement}\n")), + None => out.push_str(&format!("line {n}\n")), + } + } + out.into_bytes() +} + +/// Remove fixture directories and assert they are gone. +fn remove_all(dirs: &[&std::path::Path]) { + for d in dirs { + if d.exists() { + std::fs::remove_dir_all(d).unwrap(); + } + assert!(!d.exists(), "fixture {} must be removed", d.display()); + } +} + +fn huge(tag: u8) -> Vec { + vec![tag; 8192] +} + +struct Sz { + origin_root: PathBuf, + clone_root: PathBuf, + origin: Repo, + c1: ObjectId, + c2: ObjectId, +} + +impl Sz { + /// An origin with `big.txt` (a text over the limit), `small.txt` and + /// `src/huge.bin` at `c1`, each rewritten at `c2` (`main`). Everything the + /// first commit holds that the second replaced is history only. + fn new(tag: &str) -> Sz { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-{tag}-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-{tag}-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(origin_root.join("src")).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("big.txt"), text(&[])).unwrap(); + std::fs::write(origin_root.join("small.txt"), b"s1\n").unwrap(); + std::fs::write(origin_root.join("src/huge.bin"), huge(1)).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + std::fs::write(origin_root.join("big.txt"), text(&[(0, "ours")])).unwrap(); + std::fs::write(origin_root.join("small.txt"), b"s2\n").unwrap(); + std::fs::write(origin_root.join("src/huge.bin"), huge(2)).unwrap(); + let c2 = origin.commit("t", "c2").unwrap(); + Sz { + origin_root, + clone_root, + origin, + c1, + c2, + } + } + + fn clone(&self, prefixes: &[&str], limit: u64, lazy: bool) -> Repo { + let prefixes: Vec = prefixes.iter().map(|s| s.to_string()).collect(); + let clone = Repo::clone_url_limited( + self.origin_root.to_str().unwrap(), + &self.clone_root, + &prefixes, + Some(limit), + false, + ) + .unwrap(); + crate::promisor::set_lazy(clone.layout(), lazy).unwrap(); + clone + } + + fn cleanup(self) { + drop(self.origin); + for r in [&self.origin_root, &self.clone_root] { + if r.exists() { + std::fs::remove_dir_all(r).unwrap(); + } + assert!(!r.exists(), "fixture {} must be removed", r.display()); + } + } +} + +/// The core promise: the history-only large blobs are left out, every tip blob +/// (large or small) is present, the checkout is complete, and the clone knows +/// its scope and limit across a reload. +#[test] +fn size_only_clone_leaves_out_history_blobs_and_keeps_every_tip_complete() { + let sz = Sz::new("core"); + let clone = sz.clone(&[], LIMIT, false); + + assert!(!has(&clone, &blob_id(&text(&[]))), "old big.txt is a gap"); + assert!(!has(&clone, &blob_id(&huge(1))), "old huge.bin is a gap"); + assert!(has(&clone, &blob_id(&text(&[(0, "ours")]))), "tip big.txt"); + assert!(has(&clone, &blob_id(&huge(2))), "tip huge.bin"); + assert!(has(&clone, &blob_id(b"s1\n")), "small history blob is kept"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "the checkout materializes the large tip file" + ); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(2) + ); + + let promisor = clone.promisor().unwrap().unwrap(); + assert!(promisor.is_all_paths()); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert!(promisor.prefixes().is_empty()); + assert_eq!(promisor.describe(), format!("blob:limit={LIMIT}")); + assert_eq!( + std::fs::read_to_string(clone.layout().promisor_blob_limit_path()).unwrap(), + "limit 1024\nscope all-paths\n" + ); + assert!(clone.sparse_spec().unwrap().is_full(), "the view is full"); + // verify counts the two large old blobs as the gap frontier. + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(2)); + drop(clone); + sz.cleanup(); +} + +/// A blob of exactly `limit` bytes is left out (size >= limit, like git); one +/// byte less is kept. `blob:limit=0` leaves out every blob no tip reaches. +#[test] +fn boundary_is_inclusive_and_zero_keeps_only_tip_blobs() { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-edge-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-edge-clone-{pid}")); + let zero_root = base.join(format!("scl-p27sz-edge-zero-{pid}")); + for r in [&origin_root, &clone_root, &zero_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(&origin_root).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + let eq = vec![b'e'; LIMIT as usize]; + let lt = vec![b'l'; LIMIT as usize - 1]; + std::fs::write(origin_root.join("eq.bin"), &eq).unwrap(); + std::fs::write(origin_root.join("lt.bin"), <).unwrap(); + origin.commit("t", "c1").unwrap(); + std::fs::write(origin_root.join("eq.bin"), b"gone").unwrap(); + std::fs::write(origin_root.join("lt.bin"), b"gone too").unwrap(); + origin.commit("t", "c2").unwrap(); + + let clone = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &clone_root, + &[], + Some(LIMIT), + false, + ) + .unwrap(); + assert!(!has(&clone, &blob_id(&eq)), "size == limit is left out"); + assert!(has(&clone, &blob_id(<)), "size == limit - 1 is kept"); + drop(clone); + + let zero = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &zero_root, + &[], + Some(0), + false, + ) + .unwrap(); + assert!( + !has(&zero, &blob_id(<)), + "limit 0 leaves out every old blob" + ); + assert!(has(&zero, &blob_id(b"gone")), "but every tip blob is kept"); + assert_eq!( + std::fs::read(zero_root.join("eq.bin")).unwrap(), + b"gone", + "the checkout is complete" + ); + drop(zero); + drop(origin); + remove_all(&[&origin_root, &clone_root, &zero_root]); +} + +/// A clone that combines a prefix with a limit: the prefix decides scope, the +/// limit prunes inside it, and the promisor keeps both. +#[test] +fn prefix_plus_limit_scopes_first_then_prunes() { + let sz = Sz::new("combo"); + let clone = sz.clone(&["src/"], LIMIT, false); + assert!( + !has(&clone, &blob_id(&huge(1))), + "in scope, over limit, old" + ); + assert!(has(&clone, &blob_id(&huge(2))), "in scope, over limit, tip"); + assert!( + !has(&clone, &blob_id(&text(&[(0, "ours")]))), + "outside the prefix, even at the tip" + ); + let promisor = clone.promisor().unwrap().unwrap(); + assert!(!promisor.is_all_paths()); + assert_eq!(promisor.prefixes(), &["src/".to_string()]); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert_eq!( + promisor.describe(), + format!("src/, blob:limit={LIMIT}"), + "verify names both" + ); + assert_eq!( + std::fs::read_to_string(clone.layout().promisor_blob_limit_path()).unwrap(), + "limit 1024\nscope prefixes\n" + ); + assert!(!sz.clone_root.join("big.txt").exists()); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(2) + ); + drop(clone); + sz.cleanup(); +} + +/// `sc backfill ` on a size-only clone fetches every version of the path, +/// and the whole-tree scope and the limit survive it and a reload (B1). +#[test] +fn backfill_keeps_the_whole_tree_scope_and_limit() { + let sz = Sz::new("backfill"); + let clone = sz.clone(&[], LIMIT, false); + clone.backfill(&["src/huge.bin".to_string()]).unwrap(); + assert!(has(&clone, &blob_id(&huge(1))), "the old version arrived"); + assert!( + !has(&clone, &blob_id(&text(&[]))), + "an unrelated path is untouched" + ); + let promisor = clone.promisor().unwrap().unwrap(); + assert_eq!(promisor.blob_limit(), Some(LIMIT)); + assert!(promisor.is_all_paths(), "still every path"); + assert!(promisor.prefixes().is_empty(), "widen did not add a prefix"); + assert!(matches!( + crate::promisor::load(clone.layout()).unwrap().unwrap(), + p if p == promisor + )); + // The gap count falls by the repaired blob. + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(1)); + + clone.backfill_all().unwrap(); + assert!(clone.promisor().unwrap().is_none(), "now a full clone"); + assert!( + !clone.layout().promisor_blob_limit_path().exists(), + "the limit marker went with it" + ); + assert!(has(&clone, &blob_id(&text(&[])))); + drop(clone); + sz.cleanup(); +} + +/// Marker persistence: the explicit scope round-trips, a stray limit file with +/// no `.sc/promisor` is ignored, and a prefix scope with a limit reloads as such. +#[test] +fn scope_and_limit_round_trip_through_the_markers() { + let root = std::env::temp_dir().join(format!("scl-p27sz-marker-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let layout = crate::layout::Layout::at(&root); + std::fs::create_dir_all(&layout.dot_sc).unwrap(); + + let all = crate::Promisor::with_blob_limit("o", vec![], 5); + crate::promisor::store(&layout, &all).unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), Some(all)); + + let pref = crate::Promisor::with_blob_limit("o", vec!["a/".into(), "b".into()], 7); + crate::promisor::store(&layout, &pref).unwrap(); + let back = crate::promisor::load(&layout).unwrap().unwrap(); + assert_eq!(back, pref); + assert!(matches!(back.prefixes(), [a, b] if a == "a/" && b == "b")); + + // Storing a limit-less marker removes the limit file. + crate::promisor::store(&layout, &crate::Promisor::new("o", vec!["a/".into()])).unwrap(); + assert!(!layout.promisor_blob_limit_path().exists()); + + // A limit file without `.sc/promisor` is leftover state: a full clone. + std::fs::write( + layout.promisor_blob_limit_path(), + "limit 9\nscope all-paths\n", + ) + .unwrap(); + std::fs::remove_file(layout.promisor_path()).unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), None); + + // Malformed limit file is an error, not a silent unlimited clone. + crate::promisor::store(&layout, &crate::Promisor::new("o", vec![])).unwrap(); + std::fs::write(layout.promisor_blob_limit_path(), "limit nope\n").unwrap(); + assert!(crate::promisor::load(&layout).is_err()); + remove_all(&[&root]); +} + +/// Widening an all-paths scope is a no-op, and widening a prefix scope keeps +/// its limit. +#[test] +fn widen_cannot_narrow_an_all_paths_scope() { + let mut p = crate::Promisor::with_blob_limit("o", vec![], 5); + p.widen(&["x/".into()]); + assert!(p.is_all_paths()); + assert!(p.matches("anything/at/all")); + assert!(p.should_descend("a/b")); + assert!(matches!( + Scope::AllPaths, + ref s if *s == Scope::AllPaths + )); + let mut q = crate::Promisor::with_blob_limit("o", vec!["a/".into()], 5); + q.widen(&["b/".into()]); + assert_eq!(q.prefixes(), &["a/".to_string(), "b/".to_string()]); + assert_eq!(q.blob_limit(), Some(5)); +} + +/// `PackFilter` is the one definition of "filter active": a limit alone counts. +#[test] +fn pack_filter_is_active_for_a_prefix_or_a_limit() { + assert!(PackFilter::new(&[], None).is_none()); + assert!(PackFilter::new(&["a/".to_string()], None).is_some()); + let f = PackFilter::new(&[], Some(0)).expect("a limit of zero is still a filter"); + assert_eq!(f.blob_limit(), Some(0)); + assert!(f.prefixes().is_empty()); +} + +/// `--filter` grammar (decision 1): one limit, no `blob:none`, `blob:` reserved. +#[test] +fn filter_grammar_rejects_duplicates_and_unknown_blob_forms() { + use crate::promisor::parse_clone_filters as parse; + let v = |xs: &[&str]| xs.iter().map(|s| s.to_string()).collect::>(); + assert_eq!( + parse(&v(&["src/", "blob:limit=1MiB"])).unwrap(), + (vec!["src/".to_string()], Some(1 << 20)) + ); + assert_eq!(parse(&v(&["blob:limit=0"])).unwrap(), (vec![], Some(0))); + assert_eq!( + parse(&v(&["blob:limit=512KiB"])).unwrap().1, + Some(512 << 10) + ); + assert_eq!(parse(&v(&["blob:limit=2048"])).unwrap().1, Some(2048)); + let dup = parse(&v(&["blob:limit=1", "blob:limit=2"])).unwrap_err(); + assert!(dup.to_string().contains("more than once"), "{dup}"); + let none = parse(&v(&["blob:none"])).unwrap_err(); + assert!(none.to_string().contains("blob:limit=0"), "{none}"); + let other = parse(&v(&["blob:oops"])).unwrap_err(); + assert!(other.to_string().contains("reserved"), "{other}"); + assert!(parse(&v(&["blob:limit=lots"])).is_err()); + assert!(parse(&v(&["blob:limit="])).is_err()); + + // The reserved grammar is matched case-insensitively and untrimmed, so a + // variant spelling is the size filter or an error, never a path prefix. + for spelling in ["Blob:limit=1KiB", " blob:limit=1KiB", "BLOB:LIMIT=1KiB"] { + assert_eq!( + parse(&v(&[spelling])).unwrap(), + (vec![], Some(1024)), + "{spelling:?}" + ); + } + for bad in ["Blob:none", " blob:oops", "BLOB:", "Blob:limit"] { + let e = parse(&v(&[bad])).unwrap_err(); + assert!(e.to_string().contains("blob:"), "{bad:?}: {e}"); + } + // Names that merely start with `blob` stay ordinary prefixes. + assert_eq!( + parse(&v(&["blobs/", "blob"])).unwrap(), + (vec!["blobs/".to_string(), "blob".to_string()], None) + ); + + // A limit above the largest transferable object filters nothing. + for huge in ["18446744073709551615", "2GiB", "1073741825"] { + let e = parse(&v(&[&format!("blob:limit={huge}")])).unwrap_err(); + assert!( + e.to_string().contains("would filter nothing"), + "{huge}: {e}" + ); + } + assert_eq!(parse(&v(&["blob:limit=1GiB"])).unwrap().1, Some(1 << 30)); +} + +/// A size limit with no path filter is a caller bug: the walk refuses instead +/// of silently ignoring the limit. +#[test] +fn limited_walk_without_a_filter_is_refused() { + let sz = Sz::new("nofilter"); + let arc = sz.origin.vfs().store(); + let mut store = arc.lock().unwrap(); + let err = crate::reachable::reachable_objects_limited(&mut *store, &[sz.c2], None, Some(1)) + .unwrap_err(); + assert!(matches!(err, Error::InvalidArgument(_)), "{err:?}"); + drop(store); + drop(arc); + sz.cleanup(); +} + +/// `promisor::remove` deletes `.sc/promisor` before the limit marker, so an +/// interrupted removal leaves a stray limit file that `load` ignores. +#[test] +fn remove_deletes_the_marker_before_the_limit_file() { + let root = std::env::temp_dir().join(format!("scl-p27sz-rm-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let layout = crate::layout::Layout::at(&root); + std::fs::create_dir_all(&layout.dot_sc).unwrap(); + crate::promisor::store(&layout, &crate::Promisor::with_blob_limit("o", vec![], 5)).unwrap(); + crate::promisor::remove(&layout).unwrap(); + assert!(!layout.promisor_path().exists()); + assert!(!layout.promisor_blob_limit_path().exists()); + // The state a crash between the two removals leaves: only the stray file. + std::fs::write( + layout.promisor_blob_limit_path(), + "limit 5\nscope all-paths\n", + ) + .unwrap(); + assert_eq!(crate::promisor::load(&layout).unwrap(), None); + remove_all(&[&root]); +} + +/// A missing directory (a fetch left it out) is not blamed on the blob limit +/// and is named by its path; a missing root tree points at `--all`, not at an +/// empty `sc backfill `. +#[test] +fn directory_and_root_gaps_do_not_blame_the_limit() { + let sz = Sz::new("dirgap"); + let clone = sz.clone(&[], LIMIT, false); + let src_tree = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + let root = s.get_snapshot(&sz.c2).unwrap().root; + s.get_tree(&root).unwrap().get("src").unwrap().id + }; + let root = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + s.delete(&src_tree).unwrap(); + s.get_snapshot(&sz.c2).unwrap().root + }; + let err = { + let arc = clone.vfs().store(); + let mut s = arc.lock().unwrap(); + crate::worktree::preflight_view(clone.layout(), &mut s, root, &Sparse::default()) + .unwrap_err() + }; + match &err { + Error::GapInView { + path, + prefix, + blob_limit, + } => { + assert_eq!(path, "src"); + assert_eq!(prefix, "src", "names the path"); + assert_eq!(*blob_limit, None, "a directory is not a size gap"); + } + other => panic!("expected GapInView, got {other:?}"), + } + assert!(!err.to_string().contains("blob:limit"), "{err}"); + + let root_gap = Error::GapInView { + path: "(root tree)".into(), + prefix: String::new(), + blob_limit: None, + }; + assert!( + root_gap + .to_string() + .ends_with("Run `sc backfill --all` to repair it"), + "{root_gap}" + ); + drop(clone); + sz.cleanup(); +} + +/// gc, verify and push on a size-limited clone (presence-based, ADR-0048). +#[test] +fn gc_verify_and_push_work_on_a_size_limited_clone() { + let sz = Sz::new("gcpush"); + let clone = sz.clone(&[], LIMIT, false); + let old_big = blob_id(&text(&[])); + + clone.gc(Duration::from_secs(0)).unwrap(); + assert!(has(&clone, &blob_id(&huge(2))), "gc keeps the tip"); + assert!(!has(&clone, &old_big), "and does not invent the gap"); + assert_eq!(clone.partial_gap_count(&[sz.c2]).unwrap(), Some(2)); + + // A commit on the clone carries the gapped history by id and pushes. + std::fs::write(sz.clone_root.join("small.txt"), b"s3\n").unwrap(); + let tip = clone.commit("t", "local").unwrap(); + // The origin handle holds the repo lock; release it for the push. + let Sz { + origin_root, + clone_root, + origin, + .. + } = sz; + drop(origin); + assert_eq!(clone.push("origin").unwrap(), tip); + drop(clone); + let origin = Repo::open(&origin_root).unwrap(); + assert!(has(&origin, &old_big), "the origin still holds the history"); + assert_eq!(origin.head_tip().unwrap(), Some(tip)); + assert!(origin.snapshot(&tip).is_ok()); + drop(origin); + remove_all(&[&origin_root, &clone_root]); +} + +/// A destination that is not a mirror lacks the gapped history, so the push is +/// refused before sending a snapshot that points at it. +#[test] +fn push_to_a_non_mirror_refuses_on_a_size_gap() { + let sz = Sz::new("pushother"); + let clone = sz.clone(&[], LIMIT, false); + let other_root = std::env::temp_dir().join(format!("scl-p27sz-other-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&other_root); + std::fs::create_dir_all(&other_root).unwrap(); + drop(Repo::init(&other_root).unwrap()); + clone + .remote_add("other", other_root.to_str().unwrap()) + .unwrap(); + std::fs::write(sz.clone_root.join("small.txt"), b"s3\n").unwrap(); + clone.commit("t", "local").unwrap(); + let err = clone.push("other").unwrap_err(); + match err { + Error::PushGap { id, .. } => { + assert!(id == blob_id(&text(&[])) || id == blob_id(&huge(1)), "{id}") + } + other => panic!("expected PushGap, got {other:?}"), + } + drop(clone); + remove_all(&[&other_root]); + sz.cleanup(); +} + +/// Tip-exemption ordering (D7): a tip that is an ancestor of another tip keeps +/// its own blobs, so switching to it works on a non-lazy clone. +#[test] +fn an_ancestor_tip_is_exempt_even_though_another_tip_reaches_it() { + let sz = Sz::new("anc"); + // `old` is a ref at c1, which is also main's parent. + sz.origin.branch("old").unwrap(); + refs::write_branch_tip(sz.origin.layout(), "old", &sz.c1).unwrap(); + let clone = sz.clone(&[], LIMIT, false); + assert!( + has(&clone, &blob_id(&text(&[]))), + "old tip's big.txt is kept" + ); + assert!(has(&clone, &blob_id(&huge(1)))); + clone.fetch("origin").unwrap(); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + clone.switch("old").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(1) + ); + drop(clone); + sz.cleanup(); +} + +/// A tip whose root tree equals an older subtree: `side` is the first commit +/// (root R), and `main` moved everything into `x/` and then deleted it, so R +/// also appears at `x` in a non-tip. The tip's blob must be kept. +#[test] +fn a_tip_root_equal_to_an_older_subtree_is_still_expanded() { + let pid = std::process::id(); + let base = std::env::temp_dir(); + let origin_root = base.join(format!("scl-p27sz-eq-origin-{pid}")); + let clone_root = base.join(format!("scl-p27sz-eq-clone-{pid}")); + for r in [&origin_root, &clone_root] { + let _ = std::fs::remove_dir_all(r); + } + std::fs::create_dir_all(&origin_root).unwrap(); + let origin = Repo::init(&origin_root).unwrap(); + std::fs::write(origin_root.join("big.bin"), huge(7)).unwrap(); + let c1 = origin.commit("t", "c1").unwrap(); + std::fs::create_dir_all(origin_root.join("x")).unwrap(); + std::fs::rename(origin_root.join("big.bin"), origin_root.join("x/big.bin")).unwrap(); + origin.commit("t", "move").unwrap(); + std::fs::remove_dir_all(origin_root.join("x")).unwrap(); + std::fs::write(origin_root.join("tiny.txt"), b"t").unwrap(); + origin.commit("t", "drop").unwrap(); + refs::write_branch_tip(origin.layout(), "side", &c1).unwrap(); + + let clone = Repo::clone_url_limited( + origin_root.to_str().unwrap(), + &clone_root, + &[], + Some(LIMIT), + false, + ) + .unwrap(); + assert!( + has(&clone, &blob_id(&huge(7))), + "side's tip reaches the blob; the older x/ copy must not hide it" + ); + drop(clone); + drop(origin); + remove_all(&[&origin_root, &clone_root]); +} + +/// The walk itself: limit semantics at the `reachable` level, both orderings of +/// the wants, and `None` equals the unlimited walk. +#[test] +fn limited_walk_orders_and_none_match_the_unlimited_walk() { + let sz = Sz::new("walk"); + let arc = sz.origin.vfs().store(); + let mut store = arc.lock().unwrap(); + let all = crate::promisor::Promisor::with_blob_limit("", vec![], LIMIT); + let f: &dyn crate::reachable::PrefixFilter = &all; + let full = crate::reachable::reachable_objects(&mut *store, &[sz.c2]).unwrap(); + + let none = + crate::reachable::reachable_objects_limited(&mut *store, &[sz.c2], Some(f), None).unwrap(); + assert_eq!(none.included, full, "no limit keeps everything"); + assert!(none.gaps.is_empty()); + + for tips in [vec![sz.c2], vec![sz.c2, sz.c1], vec![sz.c1, sz.c2]] { + let r = + crate::reachable::reachable_objects_limited(&mut *store, &tips, Some(f), Some(LIMIT)) + .unwrap(); + let old_in_tips = tips.contains(&sz.c1); + assert_eq!( + r.included.contains(&blob_id(&huge(1))), + old_in_tips, + "old huge.bin is kept exactly when c1 is itself a tip ({tips:?})" + ); + assert!(r.included.contains(&blob_id(&huge(2)))); + assert!(r.included.is_disjoint(&r.gaps)); + } + drop(store); + drop(arc); + sz.cleanup(); +} + +/// B3: a lazy size-only clone has a full view, and a switch to a revision with +/// a gap must still fill (the `is_full()` guard used to skip it). Undo has its +/// own test below. +#[test] +fn lazy_full_view_switch_fills_the_gapped_revision() { + let sz = Sz::new("b3"); + let clone = sz.clone(&[], LIMIT, false); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + + // Non-lazy: refused before writing, naming the file. + let err = clone.switch("old").unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "main"); + + crate::promisor::set_lazy(clone.layout(), true).unwrap(); + clone.switch("old").unwrap(); + assert_eq!(refs::current_branch(clone.layout()).unwrap(), "old"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + assert!(has(&clone, &blob_id(&huge(1))), "filled by id"); + + // gc keeps the filled blobs (reachable from `old`). + clone.switch("main").unwrap(); + clone.gc(Duration::from_secs(0)).unwrap(); + assert!(has(&clone, &blob_id(&huge(1)))); + drop(clone); + sz.cleanup(); +} + +/// D2: tips are complete at clone time only. A fetch can bring a tip that +/// points at a gapped old blob; the non-lazy ff fails before writing with the +/// new clause and the file's own path, `backfill ` repairs it, and a +/// lazy clone heals it by id. +#[test] +fn a_fetch_that_reintroduces_a_gapped_blob_is_a_gap_in_view() { + for lazy in [false, true] { + let sz = Sz::new(if lazy { "d2-lazy" } else { "d2" }); + let clone = sz.clone(&[], LIMIT, lazy); + // Revert big.txt to its c1 content: the blob the clone left out. + std::fs::write(sz.origin_root.join("big.txt"), text(&[])).unwrap(); + sz.origin.commit("t", "revert").unwrap(); + clone.fetch("origin").unwrap(); + assert!( + !has(&clone, &blob_id(&text(&[]))), + "the fetch did not send it" + ); + + if lazy { + clone.merge("origin/main", "t").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + } else { + let err = clone.merge("origin/main", "t").unwrap_err(); + match &err { + Error::GapInView { + path, + prefix, + blob_limit, + } => { + assert_eq!(path, "big.txt"); + assert_eq!(prefix, "big.txt", "the file's own path"); + assert_eq!(*blob_limit, Some(LIMIT)); + } + other => panic!("expected GapInView, got {other:?}"), + } + let msg = err.to_string(); + assert!(msg.contains("blob:limit=1024"), "{msg}"); + assert!(msg.contains("`sc backfill big.txt`"), "{msg}"); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "nothing was written" + ); + clone.backfill(&["big.txt".to_string()]).unwrap(); + clone.merge("origin/main", "t").unwrap(); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[]) + ); + } + drop(clone); + sz.cleanup(); + } +} + +/// D3 on a prefix + limit clone: the error names the file, not the covering +/// prefix `src/`. +#[test] +fn gap_error_names_the_file_on_a_prefix_plus_limit_clone() { + let sz = Sz::new("d3"); + let clone = sz.clone(&["src/"], LIMIT, false); + std::fs::write(sz.origin_root.join("src/huge.bin"), huge(1)).unwrap(); + sz.origin.commit("t", "revert").unwrap(); + clone.fetch("origin").unwrap(); + let err = clone.merge("origin/main", "t").unwrap_err(); + match &err { + Error::GapInView { path, prefix, .. } => { + assert_eq!(path, "src/huge.bin"); + assert_eq!(prefix, "src/huge.bin", "not the covering prefix"); + } + other => panic!("expected GapInView, got {other:?}"), + } + assert!( + err.to_string().contains("`sc backfill src/huge.bin`"), + "{err}" + ); + drop(clone); + sz.cleanup(); +} + +/// A prefix-only partial clone keeps its old error text (no limit clause). +#[test] +fn prefix_only_gap_error_is_unchanged() { + let err = Error::GapInView { + path: "src/a.txt".into(), + prefix: "src/".into(), + blob_limit: None, + }; + assert_eq!( + err.to_string(), + "src/a.txt is missing from this partial clone's local objects (an earlier fetch left a gap inside the checked-out view); nothing was changed. Run `sc backfill src/` to repair it" + ); +} + +/// Merge / cherry-pick / rebase with a size gap as the merge base (D3 + the +/// ADR-0049 engine): a non-lazy clone is refused before any write with the +/// file's own path, a lazy one fills the base by id and lands the merge. +#[test] +fn replay_and_merge_with_a_size_gap_as_base() { + struct Case(&'static str); + for case in [Case("merge"), Case("pick"), Case("rebase")] { + for lazy in [false, true] { + let sz = Sz::new(&format!("base-{}-{lazy}", case.0)); + // `b1` forks at c1 and edits another line of big.txt, so the base + // (c1's big.txt) is reachable from no tip and is a gap. + refs::write_branch_tip(sz.origin.layout(), "b1", &sz.c1).unwrap(); + sz.origin.switch("b1").unwrap(); + std::fs::write(sz.origin_root.join("big.txt"), text(&[(299, "theirs")])).unwrap(); + std::fs::write(sz.origin_root.join("src/huge.bin"), huge(1)).unwrap(); + sz.origin.commit("t", "b1 edit").unwrap(); + sz.origin.switch("main").unwrap(); + + let clone = sz.clone(&[], LIMIT, lazy); + clone.fetch("origin").unwrap(); + let base_blob = blob_id(&text(&[])); + assert!(!has(&clone, &base_blob), "premise: the base is a gap"); + let before = refs::read_branch_tip(clone.layout(), "main").unwrap(); + + let run = |c: &Repo| -> Result<(), Error> { + match case.0 { + "merge" => c.merge("origin/b1", "t").map(|_| ()), + "pick" => c + .cherry_pick("origin/b1", "t", None, None) + .map(|r| assert!(matches!(r, PickResult::Picked(_)))), + _ => c.rebase("origin/b1", "t", None).map(|r| { + assert!(matches!( + r, + RebaseResult::Rebased { .. } | RebaseResult::FastForwarded(_) + )) + }), + } + }; + if lazy { + run(&clone).unwrap_or_else(|e| panic!("{} lazy: {e:?}", case.0)); + assert!(has(&clone, &base_blob), "the base was filled by id"); + let merged = std::fs::read(sz.clone_root.join("big.txt")).unwrap(); + let merged = String::from_utf8(merged).unwrap(); + if case.0 != "rebase" { + assert!(merged.starts_with("ours\n"), "{}", &merged[..20]); + assert!(merged.contains("theirs\n")); + } + } else { + let err = run(&clone) + .err() + .unwrap_or_else(|| panic!("{}: non-lazy run succeeded", case.0)); + match &err { + Error::GapBySize { path, limit } => { + assert_eq!(path, "big.txt", "{} names the file", case.0); + assert_eq!(*limit, LIMIT); + } + Error::GapInView { prefix, .. } => { + assert_eq!(prefix, "big.txt", "{} names the file", case.0) + } + other => panic!("{}: unexpected {other:?}", case.0), + } + assert!( + err.to_string().contains("sc backfill big.txt"), + "{}: {err}", + case.0 + ); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + before, + "refused before any ref moved" + ); + assert!(!clone.merge_in_progress() && !clone.pick_in_progress()); + // The repair the message names makes the retry succeed. + clone.backfill(&["big.txt".to_string()]).unwrap(); + run(&clone).unwrap(); + } + drop(clone); + sz.cleanup(); + } + } +} + +/// Sessions on a size-only clone: a non-lazy workspace forks the complete tip +/// view and harvests a clean merge, and a lazy clone's fork after a healed +/// fast-forward does too. +#[test] +fn workspace_session_on_a_size_limited_clone_lands() { + use crate::ws::WsHarvestOutcome; + for (lazy, name) in [(false, None), (true, None), (false, Some("named"))] { + let sz = Sz::new(&format!("ws-{lazy}-{}", name.unwrap_or("default"))); + let clone = sz.clone(&[], LIMIT, lazy); + let session = clone.ws_fork(name, 1, "t", None).unwrap(); + let ws = &session.workspaces[0].dir; + assert_eq!( + std::fs::read(ws.join("src/huge.bin")).unwrap(), + huge(2), + "the large tip file is laid out in the workspace" + ); + std::fs::write(ws.join("big.txt"), text(&[(0, "ours"), (150, "ws")])).unwrap(); + std::fs::write( + sz.clone_root.join("big.txt"), + text(&[(0, "ours"), (299, "host")]), + ) + .unwrap(); + clone.commit("t", "host edit").unwrap(); + let outcomes = clone.ws_harvest(name, None, "t", None).unwrap(); + assert!( + matches!(outcomes.as_slice(), [WsHarvestOutcome::Landed { .. }]), + "got {outcomes:?}" + ); + let landed = + String::from_utf8(std::fs::read(sz.clone_root.join("big.txt")).unwrap()).unwrap(); + assert!(landed.contains("ws\n") && landed.contains("host\n")); + drop(clone); + sz.cleanup(); + } +} + +/// `--lazy` works when the only filter is `blob:`: the marker is set after a +/// size-only clone and `sc cat` fills an old version by id. +#[test] +fn lazy_works_when_the_only_filter_is_a_size_limit() { + let sz = Sz::new("lazycat"); + let clone = sz.clone(&[], LIMIT, true); + assert!(crate::promisor::is_lazy(clone.layout())); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + let bytes = clone.cat_at("old", "src/huge.bin").unwrap(); + assert_eq!(&bytes[..], &huge(1)[..]); + assert!(has(&clone, &blob_id(&huge(1)))); + drop(clone); + sz.cleanup(); +} + +/// A non-lazy clone's `sc cat` of an old version gets the size-aware error. +#[test] +fn cat_of_a_gapped_blob_names_the_limit_when_not_lazy() { + let sz = Sz::new("catgap"); + let clone = sz.clone(&[], LIMIT, false); + refs::write_branch_tip(clone.layout(), "old", &sz.c1).unwrap(); + let err = clone.cat_at("old", "src/huge.bin").unwrap_err(); + assert!(matches!(err, Error::GapBySize { .. }), "got {err:?}"); + assert!(err.to_string().contains("blob:limit=1024"), "{err}"); + assert!( + err.to_string().contains("sc backfill src/huge.bin"), + "{err}" + ); + drop(clone); + sz.cleanup(); +} + +/// The partial-clone refusals apply to a size-only clone exactly as before. +#[test] +fn partial_refusals_are_unchanged_on_a_size_limited_clone() { + let sz = Sz::new("refuse"); + let clone = sz.clone(&[], LIMIT, false); + let (sk, pk) = scl_crypto::generate_keypair(); + let rid = pk.recipient_id(); + assert!(matches!( + clone.rewrap(&sk, &[], &[], false), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.grant("src/", &sk, &pk), + Err(Error::PartialCloneUnsupported(_)) + )); + assert!(matches!( + clone.revoke("src/", &rid), + Err(Error::PartialCloneUnsupported(_)) + )); + drop(clone); + sz.cleanup(); + + // A prefix + limit clone keeps the sparse refusals and names both filters. + let sz = Sz::new("refuse-prefix"); + let clone = sz.clone(&["src/"], LIMIT, false); + let err = clone.set_sparse(&[], None).unwrap_err(); + assert!( + matches!(err, Error::InvalidArgument(ref m) if m.contains("empty sparse spec")), + "got {err:?}" + ); + assert!( + err.to_string() + .contains(&format!("src/, blob:limit={LIMIT}")), + "{err}" + ); + assert!( + clone.disable_sparse(None).is_err(), + "sparse disable stays refused" + ); + drop(clone); + sz.cleanup(); +} + +/// A size-only clone can narrow its view and return to the full one it started +/// with: `sparse set` with no prefixes and `sparse disable` are allowed on an +/// `AllPaths` scope (the closure is filled when lazy, `GapInView` otherwise). +#[test] +fn a_size_only_clone_can_narrow_and_restore_its_full_view() { + let sz = Sz::new("restore"); + let clone = sz.clone(&[], LIMIT, false); + clone.set_sparse(&["src/".to_string()], None).unwrap(); + assert!(!sz.clone_root.join("big.txt").exists(), "narrowed"); + clone.set_sparse(&[], None).unwrap(); + assert!(clone.sparse_spec().unwrap().is_full()); + assert_eq!( + std::fs::read(sz.clone_root.join("big.txt")).unwrap(), + text(&[(0, "ours")]), + "restored" + ); + clone.set_sparse(&["src/".to_string()], None).unwrap(); + clone.disable_sparse(None).unwrap(); + assert!(clone.sparse_spec().unwrap().is_full()); + assert!(sz.clone_root.join("big.txt").exists()); + assert!(clone.promisor().unwrap().unwrap().is_all_paths()); + drop(clone); + sz.cleanup(); +} + +/// B3 for undo: restoring a ref to a revision with a size gap fills it first on +/// a lazy full-view clone, and refuses before moving anything otherwise. +#[test] +fn undo_to_a_gapped_revision_fills_when_lazy_and_refuses_otherwise() { + for lazy in [false, true] { + let sz = Sz::new(if lazy { "undo-lazy" } else { "undo" }); + let clone = sz.clone(&[], LIMIT, lazy); + let head = refs::current_branch(clone.layout()).unwrap(); + crate::oplog::record( + clone.layout(), + "synthetic", + &head, + &head, + &[("main".to_string(), Some(sz.c1), Some(sz.c2))], + ) + .unwrap(); + if lazy { + clone.undo().unwrap(); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(sz.c1) + ); + assert!( + has(&clone, &blob_id(&huge(1))), + "filled before the ref moved" + ); + assert_eq!( + std::fs::read(sz.clone_root.join("src/huge.bin")).unwrap(), + huge(1) + ); + } else { + let err = clone.undo().unwrap_err(); + assert!(matches!(err, Error::GapInView { .. }), "got {err:?}"); + assert_eq!( + refs::read_branch_tip(clone.layout(), "main").unwrap(), + Some(sz.c2), + "no ref moved" + ); + } + drop(clone); + sz.cleanup(); + } +} + +/// Named, concurrent sessions (ADR-0056) on a size-only clone: each records the +/// host's full view at fork, lays out the large tip file, and lands on its own; +/// the second harvest merges over the first. A fork always starts at a ref tip, +/// which is complete, so a size gap cannot be in a new session's base. +#[test] +fn named_sessions_on_a_size_limited_clone() { + use crate::ws::WsHarvestOutcome; + let sz = Sz::new("named-ws"); + let clone = sz.clone(&[], LIMIT, false); + let a = clone.ws_fork(Some("a"), 1, "t", None).unwrap(); + let b = clone.ws_fork(Some("b"), 1, "t", None).unwrap(); + assert!( + a.sparse.is_empty() && b.sparse.is_empty(), + "full view recorded" + ); + for sess in [&a, &b] { + assert_eq!( + std::fs::read(sess.workspaces[0].dir.join("src/huge.bin")).unwrap(), + huge(2) + ); + } + std::fs::write( + a.workspaces[0].dir.join("big.txt"), + text(&[(0, "ours"), (100, "from a")]), + ) + .unwrap(); + std::fs::write( + b.workspaces[0].dir.join("big.txt"), + text(&[(0, "ours"), (200, "from b")]), + ) + .unwrap(); + for name in ["a", "b"] { + let out = clone.ws_harvest(Some(name), None, "t", None).unwrap(); + assert!( + matches!(out.as_slice(), [WsHarvestOutcome::Landed { .. }]), + "{name}: {out:?}" + ); + } + let landed = String::from_utf8(std::fs::read(sz.clone_root.join("big.txt")).unwrap()).unwrap(); + assert!(landed.contains("from a\n") && landed.contains("from b\n")); + + assert!( + clone.ws_sessions().unwrap().is_empty(), + "both sessions ended" + ); + drop(clone); + sz.cleanup(); +} diff --git a/crates/repo/src/partial_walks.rs b/crates/repo/src/partial_walks.rs index 01e77601..d5cb6f13 100644 --- a/crates/repo/src/partial_walks.rs +++ b/crates/repo/src/partial_walks.rs @@ -142,7 +142,7 @@ fn fetch_leaves_in_filter_gap_gc_works_and_ff_fails_before_writing() { let head_before = refs::read_branch_tip(clone.layout(), "main").unwrap(); let err = clone.merge("origin/main", "t").unwrap_err(); match &err { - Error::GapInView { path, prefix } => { + Error::GapInView { path, prefix, .. } => { assert_eq!(path, "src/readme.txt"); assert_eq!(prefix, "src/"); } diff --git a/crates/repo/src/private.rs b/crates/repo/src/private.rs index aba30038..19cdb717 100644 --- a/crates/repo/src/private.rs +++ b/crates/repo/src/private.rs @@ -2545,6 +2545,44 @@ mod tests { std::fs::remove_dir_all(&part_dst).unwrap(); } + /// ADR-0055 B2: "filter active" is `prefixes || blob_limit`, so a clone + /// whose only filter is a size limit also leaves the private ref out. + #[test] + fn size_only_clone_excludes_private_branches() { + let (root, repo, (alice_sk, _), _bob) = setup("clonesize"); + repo.branch_private("fix", &alice_sk, &[], &[]).unwrap(); + repo.switch_with_identity("fix", Some(&alice_sk)).unwrap(); + write(&root, "src/private.rs", "sealed\n"); + repo.commit_private("alice", "c1", Some(&alice_sk)).unwrap(); + repo.switch_with_identity("main", Some(&alice_sk)).unwrap(); + drop(repo); + + let part_dst = tmp_root("clonesize-part"); + let part = Repo::clone_url_limited( + &root.display().to_string(), + &part_dst, + &[], + Some(1024), + false, + ) + .unwrap(); + let names: Vec = part + .branches() + .unwrap() + .into_iter() + .map(|(n, _)| n) + .collect(); + assert!( + !names.contains(&"fix".to_string()), + "a size-only clone must exclude private refs: {names:?}" + ); + assert!(names.contains(&"main".to_string())); + drop(part); + + std::fs::remove_dir_all(&root).unwrap(); + std::fs::remove_dir_all(&part_dst).unwrap(); + } + #[test] fn wire_pack_for_a_private_branch_carries_ciphertext_only() { let (root, repo, (alice_sk, _), _bob) = setup("wire"); diff --git a/crates/repo/src/promisor.rs b/crates/repo/src/promisor.rs index 754d67cf..02e2b20c 100644 --- a/crates/repo/src/promisor.rs +++ b/crates/repo/src/promisor.rs @@ -27,35 +27,89 @@ use crate::error::{Error, Result}; use crate::layout::Layout; use crate::repo::Repo; -/// A partial clone's durable marker: the fetch-filter prefixes + the -/// promisor remote (origin URL) that can fill in objects outside the -/// filter on demand. Absent file = a full clone. -#[derive(Clone, Debug, Default, PartialEq, Eq)] +/// Which paths a partial clone asked for (ADR-0055 B1). A size-only clone +/// (`--filter blob:limit=` with no prefix) is in scope everywhere, which +/// an empty prefix list cannot say: that reads as "matches nothing". +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Scope { + /// Every path (a size-only clone). + AllPaths, + /// Only paths under these prefixes. + Prefixes(Vec), +} + +/// A partial clone's durable marker: the fetch-filter scope + the optional +/// blob-size limit + the promisor remote (origin URL) that can fill in objects +/// outside the filter on demand. Absent file = a full clone. +#[derive(Clone, Debug, PartialEq, Eq)] pub struct Promisor { - prefixes: Vec, + scope: Scope, + blob_limit: Option, pub origin: String, } +impl Default for Promisor { + fn default() -> Self { + Promisor { + scope: Scope::Prefixes(Vec::new()), + blob_limit: None, + origin: String::new(), + } + } +} + impl Promisor { /// Build a marker from an explicit origin + prefix list. pub fn new(origin: impl Into, prefixes: Vec) -> Self { Promisor { - prefixes, + scope: Scope::Prefixes(prefixes), + blob_limit: None, origin: origin.into(), } } + /// Build a marker for a clone made with a blob-size limit (ADR-0055). An + /// empty `prefixes` means every path is in scope ([`Scope::AllPaths`]). + pub fn with_blob_limit( + origin: impl Into, + prefixes: Vec, + blob_limit: u64, + ) -> Self { + Promisor { + scope: if prefixes.is_empty() { + Scope::AllPaths + } else { + Scope::Prefixes(prefixes) + }, + blob_limit: Some(blob_limit), + origin: origin.into(), + } + } + + /// The blob-size limit the clone was made with, if any: blobs of at least + /// this many bytes, reachable only through history, were left out. + pub fn blob_limit(&self) -> Option { + self.blob_limit + } + + /// True when every path is in scope (a size-only clone). + pub fn is_all_paths(&self) -> bool { + matches!(self.scope, Scope::AllPaths) + } + /// Whether `path` falls under any recorded prefix, at a `/` boundary — /// mirrors [`crate::sparse::Sparse::matches`] / `protect::matching_prefix`. - /// Unlike `Sparse`, an empty prefix list matches nothing (a promisor - /// marker with no prefixes would be a degenerate/empty partial clone, - /// not "everything" — callers gate on `Repo::promisor()` returning - /// `None` for "no filter at all"). + /// An empty prefix list matches nothing (a degenerate partial clone); + /// [`Scope::AllPaths`] matches everything. Callers gate on + /// `Repo::promisor()` returning `None` for "no filter at all". pub fn matches(&self, path: &str) -> bool { - self.prefixes.iter().any(|p| { - let bare = p.trim_end_matches('/'); - path == bare || path.starts_with(&format!("{bare}/")) - }) + match &self.scope { + Scope::AllPaths => true, + Scope::Prefixes(prefixes) => prefixes.iter().any(|p| { + let bare = p.trim_end_matches('/'); + path == bare || path.starts_with(&format!("{bare}/")) + }), + } } /// Should a tree walk descend into `path`? True if `path` is itself @@ -64,30 +118,141 @@ impl Promisor { /// content deeper in the tree). The empty path (the root) always /// descends, since every prefix lies under it. pub fn should_descend(&self, path: &str) -> bool { - path.is_empty() - || self.matches(path) - || self - .prefixes - .iter() - .any(|p| p.trim_end_matches('/').starts_with(&format!("{path}/"))) + match &self.scope { + Scope::AllPaths => true, + Scope::Prefixes(prefixes) => { + path.is_empty() + || self.matches(path) + || prefixes + .iter() + .any(|p| p.trim_end_matches('/').starts_with(&format!("{path}/"))) + } + } } - /// The recorded prefixes, in the order they were set. + /// The recorded prefixes, in the order they were set (empty for + /// [`Scope::AllPaths`]). pub fn prefixes(&self) -> &[String] { - &self.prefixes + match &self.scope { + Scope::AllPaths => &[], + Scope::Prefixes(prefixes) => prefixes, + } + } + + /// The filter as one line for messages: the prefixes, then `blob:limit=` + /// when the clone has a limit (`sc verify`, refusals). + pub fn describe(&self) -> String { + let mut parts: Vec = self.prefixes().to_vec(); + if let Some(limit) = self.blob_limit { + parts.push(format!("blob:limit={limit}")); + } + parts.join(", ") } /// Widen the filter to also include `prefixes` (backfill, Task 4). - /// Dedups against the existing set; order is existing-then-new. + /// Dedups against the existing set; order is existing-then-new. An + /// all-paths scope stays all-paths (nothing is wider). pub fn widen(&mut self, prefixes: &[String]) { - for p in prefixes { - if !self.prefixes.iter().any(|existing| existing == p) { - self.prefixes.push(p.clone()); + if let Scope::Prefixes(existing) = &mut self.scope { + for p in prefixes { + if !existing.iter().any(|e| e == p) { + existing.push(p.clone()); + } } } } } +/// The filter a `GetPack` carries (ADR-0037, ADR-0055): path prefixes and an +/// optional blob-size limit. The one place that decides whether a transfer is +/// filtered: a value exists only when it is active (`!prefixes.is_empty() || +/// blob_limit.is_some()`), so every `Option<&PackFilter>` that is `Some` means +/// "partial transfer" on the sender, the bucket refusal and the client's +/// private-ref handling alike. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PackFilter { + prefixes: Vec, + blob_limit: Option, +} + +impl PackFilter { + /// `None` when neither a prefix nor a limit is given (a full transfer). + pub fn new(prefixes: &[String], blob_limit: Option) -> Option { + (!prefixes.is_empty() || blob_limit.is_some()).then(|| PackFilter { + prefixes: prefixes.to_vec(), + blob_limit, + }) + } + + /// The path prefixes; empty means every path. + pub fn prefixes(&self) -> &[String] { + &self.prefixes + } + + /// Blobs of at least this many bytes that are not reachable from a want's + /// own root tree are left out. + pub fn blob_limit(&self) -> Option { + self.blob_limit + } + + /// The path scope as a [`Promisor`] the reachability walk can ask. + pub(crate) fn scope(&self) -> Promisor { + match self.blob_limit { + Some(limit) => Promisor::with_blob_limit(String::new(), self.prefixes.clone(), limit), + None => Promisor::new(String::new(), self.prefixes.clone()), + } + } +} + +/// Split `sc clone --filter` values into path prefixes and the optional blob +/// limit (ADR-0055). A value starting with `blob:` is the reserved size +/// grammar, never a prefix: only `blob:limit=` is accepted, once. A +/// path prefix that literally begins with `blob:` cannot be named here. +pub fn parse_clone_filters(values: &[String]) -> Result<(Vec, Option)> { + let mut prefixes = Vec::new(); + let mut limit = None; + for v in values { + // Reserved however it is spelled: ` Blob:limit=1KiB` must not slip + // through as a path prefix that matches nothing. + let norm = v.trim().to_ascii_lowercase(); + let Some(spec) = norm.strip_prefix("blob:") else { + prefixes.push(v.clone()); + continue; + }; + if spec == "none" { + return Err(Error::InvalidArgument( + "unsupported filter `blob:none`; use `blob:limit=0` to leave out every blob \ + that no ref tip reaches" + .into(), + )); + } + let Some(size) = spec.strip_prefix("limit=") else { + return Err(Error::InvalidArgument(format!( + "unsupported filter `{v}`; the only blob filter is `blob:limit=` \ + (`blob:` is reserved and cannot start a path prefix)" + ))); + }; + if limit.is_some() { + return Err(Error::InvalidArgument( + "`blob:limit` given more than once; pass a single `--filter blob:limit=`" + .into(), + )); + } + let n = scl_core::parse_byte_size(size).map_err(Error::InvalidArgument)?; + // No object above the largest configurable cap can be transferred, so + // a limit above it filters nothing and is almost certainly a mistake. + if n > scl_core::MAX_MAX_OBJECT_SIZE as u64 { + return Err(Error::InvalidArgument(format!( + "`blob:limit={size}` is above the largest transferable object ({} bytes, 1 GiB) \ + and would filter nothing; give a smaller limit, or omit the filter", + scl_core::MAX_MAX_OBJECT_SIZE + ))); + } + limit = Some(n); + } + Ok((prefixes, limit)) +} + /// Build the "outside this partial clone's fetch filter" error for `path` /// (P27 Task 5) — used by the sparse-widen preflight /// (`set_sparse`/`disable_sparse`), where `path` is a real prefix `sc @@ -96,6 +261,23 @@ pub fn partial_gap_hint(path: &str) -> Error { Error::GapOutsideFilter(path.to_string()) } +/// The gap error for `path` on this partial clone: [`Error::GapBySize`] when +/// the clone has a blob-size limit, `path` is in its scope and the missing +/// object is a file (`tree` false; a missing directory was left out by a fetch, +/// not by the limit), otherwise the prefix hint naming the path. +pub fn gap_error(promisor: Option<&Promisor>, path: &str, tree: bool) -> Error { + match promisor { + Some(p) if !tree && !path.is_empty() && p.matches(path) => match p.blob_limit() { + Some(limit) => Error::GapBySize { + path: path.to_string(), + limit, + }, + None => partial_gap_hint(path), + }, + _ => partial_gap_hint(path), + } +} + /// Build the "this operation isn't supported on a partial clone at all" /// error (P27 Task 5 review fix): merge, cherry-pick/rebase replay, `sc ws /// harvest`, and `sc work` all need the FULL tree of whatever they touch, @@ -109,7 +291,9 @@ pub fn partial_clone_unsupported(op: &str) -> Error { } /// Load the repo's promisor marker. An absent `.sc/promisor` file reads as -/// `None` — a full clone, not an empty filter. +/// `None` — a full clone, not an empty filter. `.sc/promisor-blob-limit` +/// (ADR-0055) is read only when `.sc/promisor` exists: a stray limit file +/// without it is leftover state and is ignored. pub fn load(layout: &Layout) -> Result> { match std::fs::read_to_string(layout.promisor_path()) { Ok(text) => { @@ -129,18 +313,74 @@ pub fn load(layout: &Layout) -> Result> { )) } }; - let prefixes = lines.map(|l| l.to_string()).collect(); - Ok(Some(Promisor { prefixes, origin })) + let prefixes: Vec = lines.map(|l| l.to_string()).collect(); + let (blob_limit, all_paths) = load_limit(layout)?; + let scope = if all_paths { + Scope::AllPaths + } else { + Scope::Prefixes(prefixes) + }; + Ok(Some(Promisor { + scope, + blob_limit, + origin, + })) } Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), Err(e) => Err(e.into()), } } -/// Persist the repo's promisor marker, atomically. +/// Parse `.sc/promisor-blob-limit`: `limit ` then `scope all-paths` or +/// `scope prefixes`. Absent file = no limit and prefix scope. +fn load_limit(layout: &Layout) -> Result<(Option, bool)> { + let text = match std::fs::read_to_string(layout.promisor_blob_limit_path()) { + Ok(t) => t, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((None, false)), + Err(e) => return Err(e.into()), + }; + let bad = || { + Error::InvalidArgument( + "malformed .sc/promisor-blob-limit: expected 'limit ' then \ + 'scope all-paths|prefixes'" + .into(), + ) + }; + let mut lines = text.lines(); + let limit = lines + .next() + .and_then(|l| l.strip_prefix("limit ")) + .and_then(|n| n.parse::().ok()) + .ok_or_else(bad)?; + let all_paths = match lines.next() { + Some("scope all-paths") => true, + Some("scope prefixes") => false, + _ => return Err(bad()), + }; + Ok((Some(limit), all_paths)) +} + +/// Persist the repo's promisor marker, atomically. The limit file goes first: +/// a crash between the two writes leaves a limit file with no marker (ignored +/// by [`load`]) rather than a marker that has lost its limit. pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { + match p.blob_limit { + Some(limit) => { + let scope = if p.is_all_paths() { + "all-paths" + } else { + "prefixes" + }; + let text = format!("limit {limit}\nscope {scope}\n"); + scl_core::fsutil::atomic_write_durable( + &layout.promisor_blob_limit_path(), + text.as_bytes(), + )?; + } + None => remove_file_if_exists(&layout.promisor_blob_limit_path())?, + } let mut text = format!("origin {}\n", p.origin); - for prefix in &p.prefixes { + for prefix in p.prefixes() { text.push_str(prefix); text.push('\n'); } @@ -148,18 +388,25 @@ pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { Ok(()) } +fn remove_file_if_exists(path: &std::path::Path) -> Result<()> { + match std::fs::remove_file(path) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), + } +} + /// Remove the repo's promisor marker: this repo becomes a genuine full /// clone (P27 final review I2, `Repo::backfill_all`'s escape hatch). A /// no-op if the marker is already absent. Callers MUST only call this /// after independently confirming the object closure is actually /// complete — this function does not itself verify anything. pub fn remove(layout: &Layout) -> Result<()> { - set_lazy(layout, false)?; - match std::fs::remove_file(layout.promisor_path()) { - Ok(()) => Ok(()), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), - Err(e) => Err(e.into()), - } + // `.sc/promisor` first: a crash after it leaves only a stray limit file, + // which `load` ignores, never a marker that lost its limit and matches nothing. + remove_file_if_exists(&layout.promisor_path())?; + remove_file_if_exists(&layout.promisor_blob_limit_path())?; + set_lazy(layout, false) } /// True when this partial clone has opted in to filling gaps by object id diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index 9cf965f9..73d13bc5 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -2,7 +2,7 @@ //! snapshot tips. Parameterized by an `ObjectSource` so it walks either the //! local `Store` (push) or a remote `Transport` (clone/fetch). Reused by P8 gc. -use std::collections::{BTreeMap, BTreeSet, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}; use scl_core::{EntryKind, Object, ObjectId, Store}; @@ -12,12 +12,28 @@ use crate::transport::Transport; /// Minimal read access the walk needs: fetch a decoded object by id. pub trait ObjectSource { fn get(&mut self, id: &ObjectId) -> Result; + + /// Byte length of the blob `id`. The default loads the object; a source + /// that can measure without admitting the blob to a cache overrides it. + fn blob_size(&mut self, id: &ObjectId) -> Result { + Ok(match self.get(id)? { + Object::Blob(b) => b.len() as u64, + _ => return Err(Error::Core(scl_core::Error::WrongKind(*id, "blob"))), + }) + } } impl ObjectSource for Store { fn get(&mut self, id: &ObjectId) -> Result { Ok(Store::get(self, id)?) } + + /// Streams the payload through a fixed buffer and checks its hash, so + /// measuring a large blob neither fills the object cache nor trusts + /// unverified bytes (ADR-0055). + fn blob_size(&mut self, id: &ObjectId) -> Result { + Ok(Store::blob_size(self, id)? as u64) + } } /// An `ObjectSource` backed by a remote `Transport`. @@ -122,7 +138,41 @@ pub fn reachable_objects_filtered( tips: &[ObjectId], filter: Option<&dyn PrefixFilter>, ) -> Result { + reachable_objects_limited(src, tips, filter, None) +} + +/// [`reachable_objects_filtered`] plus the blob-size limit (ADR-0055). With +/// `blob_limit = Some(n)`, a blob of at least `n` bytes goes to `gaps` instead +/// of `included`, unless it is reachable from the root tree of one of `tips` +/// (the wants): every tip is complete, so the limit only prunes blobs that are +/// reachable through parent snapshots alone. `blob_limit` needs a `filter` to +/// say which paths are in scope (use an all-paths one for a size-only walk). +/// With `None` this is `reachable_objects_filtered` exactly. +/// +/// Ordering: `walk_snapshots` queues every tip before any parent, so a tip's +/// exempt walk always runs before a non-tip snapshot's limited one, even when +/// one tip is an ancestor of another. The `(id, path)` expansion gate then only +/// ever lets a limited walk skip work an exempt walk already did (a superset), +/// never the reverse; and a tip root tree that equals an older subtree is still +/// expanded at its own `(root, "")` key. +pub fn reachable_objects_limited( + src: &mut impl ObjectSource, + tips: &[ObjectId], + filter: Option<&dyn PrefixFilter>, + blob_limit: Option, +) -> Result { + // A limit needs a scope to apply to (an all-paths filter for a size-only + // walk). Failing here is safer than silently sending everything. + if blob_limit.is_some() && filter.is_none() { + return Err(Error::InvalidArgument( + "blob limit without a path filter".into(), + )); + } let mut gaps = BTreeSet::new(); + let tip_set: HashSet = tips.iter().copied().collect(); + // Size verdicts by blob id: a blob shared by many paths or versions is + // measured once per request (ADR-0055). + let mut sizes: HashMap = HashMap::new(); // (id, path) pairs already pushed for expansion, under a filter only. // Hoisted to span EVERY snapshot's walk in this call (not just one), so // a later snapshot whose ROOT tree dedups to an already-expanded @@ -131,8 +181,21 @@ pub fn reachable_objects_filtered( // comment for why a bare-id gate on the root is wrong under a filter // (C1, P27 final review). let mut expanded: HashSet<(ObjectId, String)> = HashSet::new(); - let mut included = walk_snapshots(src, tips, |src, _snapshot, root, included| { - walk_tree_filtered(src, root, included, &mut gaps, filter, &mut expanded) + let mut included = walk_snapshots(src, tips, |src, snapshot, root, included| { + let mut gate = blob_limit.map(|limit| SizeGate { + limit, + exempt: tip_set.contains(&snapshot), + verdicts: &mut sizes, + }); + walk_tree_sized( + src, + root, + included, + &mut gaps, + filter, + &mut expanded, + gate.as_mut(), + ) })?; // A byte-identical subtree can be gapped at one path and included at // another (same id, different filter verdict per path) — included @@ -363,6 +426,48 @@ fn walk_tree_filtered( gaps: &mut BTreeSet, filter: Option<&dyn PrefixFilter>, expanded: &mut HashSet<(ObjectId, String)>, +) -> Result<()> { + walk_tree_sized(src, root, included, gaps, filter, expanded, None) +} + +/// The blob-size rule for one snapshot's tree walk (ADR-0055): `exempt` is true +/// for a want's own walk, where every in-scope blob is kept; otherwise a blob +/// of `limit` bytes or more is a gap. `verdicts` caches "keep?" by blob id. +struct SizeGate<'a> { + limit: u64, + exempt: bool, + verdicts: &'a mut HashMap, +} + +impl SizeGate<'_> { + fn keeps(&mut self, src: &mut impl ObjectSource, id: &ObjectId) -> Result { + if self.exempt { + return Ok(true); + } + // Decided by the limit alone: nothing to measure. + if self.limit == 0 { + return Ok(false); + } + if self.limit == u64::MAX { + return Ok(true); + } + if let Some(keep) = self.verdicts.get(id) { + return Ok(*keep); + } + let keep = src.blob_size(id)? < self.limit; + self.verdicts.insert(*id, keep); + Ok(keep) + } +} + +fn walk_tree_sized( + src: &mut impl ObjectSource, + root: ObjectId, + included: &mut BTreeSet, + gaps: &mut BTreeSet, + filter: Option<&dyn PrefixFilter>, + expanded: &mut HashSet<(ObjectId, String)>, + mut size: Option<&mut SizeGate<'_>>, ) -> Result<()> { let mut stack: Vec<(ObjectId, String)> = Vec::new(); match filter { @@ -395,7 +500,15 @@ fn walk_tree_filtered( included.insert(e.id); } Some(f) => { - if f.matches(&child_path) { + let keep = f.matches(&child_path) + && match size.as_deref_mut() { + // Already kept via an exempt walk: no need to measure. + Some(gate) if !included.contains(&e.id) => { + gate.keeps(src, &e.id)? + } + _ => true, + }; + if keep { included.insert(e.id); } else { gaps.insert(e.id); diff --git a/crates/repo/src/sparse.rs b/crates/repo/src/sparse.rs index f6bee3a2..97375058 100644 --- a/crates/repo/src/sparse.rs +++ b/crates/repo/src/sparse.rs @@ -337,13 +337,16 @@ impl Repo { // tree) — so a refused widen leaves nothing partially written. if let Some(promisor) = self.promisor()? { // No prefixes means a full checkout, which is what `sparse - // disable` is, and that is refused on a partial clone. - if prefixes.is_empty() { + // disable` is, and that is refused on a partial clone. A size-only + // clone (ADR-0055, `AllPaths`) is the exception: a full view is its + // starting state, so a narrowed view must be able to return to it. + // The view closure is filled (lazy) or refused (`GapInView`) below. + if prefixes.is_empty() && !promisor.is_all_paths() { return Err(Error::InvalidArgument(format!( "cannot set an empty sparse spec (a full checkout) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full \ clone first, or name a prefix within it", - promisor.prefixes().join(", ") + promisor.describe() ))); } // A lazy clone may widen the view past its fetch filter: the @@ -743,11 +746,16 @@ impl Repo { // the origin). Refuse up front, before any disk write, pointing at // backfill — same preflight discipline as `set_sparse` above. if let Some(promisor) = self.promisor()? { + // A size-only clone's scope is every path, so disabling sparse only + // restores its full view: the same operation as an empty spec. + if promisor.is_all_paths() { + return self.set_sparse(&[], identity); + } return Err(Error::InvalidArgument(format!( "cannot disable sparse checkout (full materialization) on a partial clone: \ only [{}] was fetched; run `sc backfill --all` to convert to a full clone \ first, or `sc sparse set` to a prefix within it", - promisor.prefixes().join(", ") + promisor.describe() ))); } let dirty = self.status()?; diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 965caf0e..19c51c2b 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -160,7 +160,7 @@ impl Transport for WireClient { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { let mut rw = self.rw.borrow_mut(); @@ -169,7 +169,8 @@ impl Transport for WireClient { &Request::GetPack { wants: wants.to_vec(), haves: haves.to_vec(), - filter: filter.map(|f| f.to_vec()).unwrap_or_default(), + filter: filter.map(|f| f.prefixes().to_vec()).unwrap_or_default(), + blob_limit: filter.and_then(|f| f.blob_limit()), } .encode(), )?; @@ -346,7 +347,7 @@ impl Transport for StdioTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { self.client.get_pack(wants, haves, filter, out) @@ -655,7 +656,12 @@ mod tests { let mut pack = Vec::new(); client - .get_pack(&[tip], &[], Some(&["src/".to_string()]), &mut pack) + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&["src/".to_string()], None).as_ref(), + &mut pack, + ) .unwrap(); let ids: Vec<_> = scl_core::pack::parse_pack(&pack) .unwrap() @@ -675,6 +681,55 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + /// ADR-0055: the client's `blob_limit` reaches the server and is applied + /// there. A tip blob over the limit is kept, an old one over it is not, an + /// old small one is kept. + #[test] + fn blob_limit_get_pack_over_wire() { + let root = tmp_repo("blob-limit"); + let repo = crate::repo::Repo::open(&root).unwrap(); + let old_big = vec![b'o'; 4096]; + let new_big = vec![b'n'; 4096]; + std::fs::write(root.join("big"), &old_big).unwrap(); + std::fs::write(root.join("small"), b"s1").unwrap(); + repo.commit("t", "c1").unwrap(); + std::fs::write(root.join("big"), &new_big).unwrap(); + std::fs::write(root.join("small"), b"s2").unwrap(); + let tip = repo.commit("t", "c2").unwrap(); + drop(repo); + let id = |b: &[u8]| Object::blob(b.to_vec()).id(); + + let (client, server) = connect(root.clone()); + for (limit, old_big_sent, small_old_sent) in [ + (1024u64, false, true), + (0, false, false), + (1 << 20, true, true), + ] { + let mut pack = Vec::new(); + client + .get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&[], Some(limit)).as_ref(), + &mut pack, + ) + .unwrap(); + let ids: Vec<_> = scl_core::pack::parse_pack(&pack) + .unwrap() + .into_iter() + .map(|(i, _)| i) + .collect(); + assert!(ids.contains(&id(&new_big)), "limit {limit}: tip blob"); + assert!(ids.contains(&id(b"s2")), "limit {limit}: tip small blob"); + assert_eq!(ids.contains(&id(&old_big)), old_big_sent, "limit {limit}"); + assert_eq!(ids.contains(&id(b"s1")), small_old_sent, "limit {limit}"); + } + client.bye().unwrap(); + drop(client); + server.join().unwrap().unwrap(); + std::fs::remove_dir_all(&root).unwrap(); + } + #[test] fn handshake_surfaces_not_a_repo_as_typed_error() { let root = std::env::temp_dir().join(format!("scl-stdio-norepo-{}", std::process::id())); @@ -693,21 +748,28 @@ mod tests { } #[test] - fn handshake_rejects_v4_peer() { - // P37 bumped PROTOCOL_VERSION 4 -> 5 for the additive `GetObjects` - // verb: a v4 server cannot decode it, so refuse at the handshake. + fn handshake_rejects_v5_peer() { + // ADR-0055 bumped PROTOCOL_VERSION 5 -> 6: `GetPack` gained the + // trailing `blob_limit`, so a v5 server cannot decode it. Refuse at + // the handshake and name both versions. let (client_read, mut server_write) = std::io::pipe().unwrap(); let (mut server_read, client_write) = std::io::pipe().unwrap(); let server = std::thread::spawn(move || { let f = wire::read_frame(&mut server_read).unwrap(); assert!(matches!( wire::Request::decode(&f).unwrap(), - wire::Request::Hello { version: 5 } + wire::Request::Hello { version: 6 } )); - wire::write_ok(&mut server_write, &wire::u32_body(4)).unwrap(); + wire::write_ok(&mut server_write, &wire::u32_body(5)).unwrap(); }); let err = WireClient::handshake(client_read, client_write).unwrap_err(); - assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + match &err { + Error::Protocol(m) => assert_eq!( + m, "server speaks protocol 5, this client speaks 6", + "got {m}" + ), + other => panic!("expected a protocol error, got {other:?}"), + } server.join().unwrap(); } @@ -1002,6 +1064,7 @@ mod tests { wants: vec![tip], haves: vec![], filter: vec![], + blob_limit: None, } .encode(), ) diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 3346aad7..64f2e780 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -96,12 +96,28 @@ impl Repo { dst: impl AsRef, filter: Option<&[String]>, no_transcripts: bool, + ) -> Result { + Self::clone_url_limited(src_url, dst, filter.unwrap_or(&[]), None, no_transcripts) + } + + /// [`Repo::clone_url_opts`] with the blob-size limit (ADR-0055): the + /// clone is partial when `prefixes` is non-empty or `blob_limit` is set. + /// With a limit and no prefix every path is in scope (`Scope::AllPaths`) + /// and the sparse view stays full. The server leaves out blobs of at least + /// `blob_limit` bytes that no ref tip's root tree reaches, so the initial + /// checkout is always complete. + pub fn clone_url_limited( + src_url: &str, + dst: impl AsRef, + prefixes: &[String], + blob_limit: Option, + no_transcripts: bool, ) -> Result { // Before the network or the destination directory: a blank prefix // would be persisted as a sparse line that reloads as a full view. - if let Some(f) = filter { - crate::sparse::validate_prefixes(f)?; - } + crate::sparse::validate_prefixes(prefixes)?; + let pack_filter = crate::promisor::PackFilter::new(prefixes, blob_limit); + let filter = pack_filter.as_ref(); let transport = open_transport(src_url)?; let remote_refs = transport.list_refs()?; let head_branch = transport.head_branch()?; @@ -177,9 +193,17 @@ impl Repo { // doesn't transfer a pre-existing `.sc/sparse` from the source (out // of P24 scope, sparse config is local like `.scignore`). let sparse_spec = match filter { - Some(prefixes) => { - let promisor = crate::promisor::Promisor::new(src_url, prefixes.to_vec()); + Some(_) => { + let promisor = match blob_limit { + Some(limit) => crate::promisor::Promisor::with_blob_limit( + src_url, + prefixes.to_vec(), + limit, + ), + None => crate::promisor::Promisor::new(src_url, prefixes.to_vec()), + }; crate::promisor::store(&dst_repo.layout, &promisor)?; + // A size-only clone has no prefixes: the view stays full. let sparse = crate::sparse::Sparse::new(prefixes.to_vec()); crate::sparse::store(&dst_repo.layout, &sparse)?; sparse @@ -284,7 +308,9 @@ impl Repo { &mut store, &tips, &[], - Some(prefixes), + // Backfill never applies the clone's blob limit: it fetches + // every version of the named paths (ADR-0055). + crate::promisor::PackFilter::new(prefixes, None).as_ref(), )?; } @@ -613,7 +639,7 @@ fn transfer_objects( store: &mut Store, tips: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result<()> { transfer_objects_opts(layout, transport, store, tips, haves, filter, false) } @@ -627,7 +653,7 @@ fn transfer_objects_opts( store: &mut Store, tips: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, no_transcripts: bool, ) -> Result<()> { let guard = crate::transport::TempPackGuard::new(layout)?; diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index ce1d1ee1..cec8d419 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -47,16 +47,19 @@ pub trait Transport { /// `filter`, when `Some`, narrows the sender's `wants`-side walk to the /// given path prefixes (P27 Task 3) — the wire form of a partial clone's /// `.sc/promisor` prefix list, matched with the same `/`-boundary - /// discipline as `Promisor::matches`/`should_descend`. The `haves`-side - /// walk is always unfiltered (the receiver's full closure is excluded - /// regardless of filter, since a filter only narrows what's sent, never - /// widens what's assumed already held). `filter = None` reproduces the - /// pre-P27 full-transfer behavior byte-for-byte. + /// discipline as `Promisor::matches`/`should_descend` — and/or leaves out + /// blobs of at least `blob_limit` bytes that no want's own root tree + /// reaches (ADR-0055; no prefixes means every path). A `PackFilter` exists + /// only when it is active, so `Some` always means a partial transfer. The + /// `haves`-side walk is always unfiltered (the receiver's full closure is + /// excluded regardless of filter, since a filter only narrows what's sent, + /// never widens what's assumed already held). `filter = None` reproduces + /// the pre-P27 full-transfer behavior byte-for-byte. fn get_pack( &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()>; @@ -250,14 +253,13 @@ impl LocalTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result { use std::collections::BTreeSet; let mut store = self.store.borrow_mut(); // Reachable-from-wants minus reachable-from-haves, computed on this // (the remote) store. `haves` the remote doesn't have are skipped. - let promisor_filter = - filter.map(|prefixes| crate::promisor::Promisor::new(String::new(), prefixes.to_vec())); + let promisor_filter = filter.map(|f| f.scope()); // Filtered transfers exclude private branches entirely (ADR-0044): // a want that resolves to a branch manifest is dropped before the // walk. Serves both transports — the wire server's GetPack delegates @@ -266,10 +268,11 @@ impl LocalTransport { crate::reachable::filter_manifest_wants(&mut *store, wants, promisor_filter.is_some())?; let mut want_set = match &promisor_filter { Some(pf) => { - crate::reachable::reachable_objects_filtered( + crate::reachable::reachable_objects_limited( &mut *store, &wants, Some(pf as &dyn crate::reachable::PrefixFilter), + filter.and_then(|f| f.blob_limit()), )? .included } @@ -748,7 +751,7 @@ impl Transport for LocalTransport { &self, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, out: &mut dyn Write, ) -> Result<()> { let guard = self.build_pack_tempfile(wants, haves, filter)?; @@ -1138,8 +1141,13 @@ mod tests { other => panic!("expected snapshot, got {other:?}"), }; let mut pack = Vec::new(); - t.get_pack(&[tip], &[], Some(&["src/".to_string()]), &mut pack) - .unwrap(); + t.get_pack( + &[tip], + &[], + crate::promisor::PackFilter::new(&["src/".to_string()], None).as_ref(), + &mut pack, + ) + .unwrap(); let _ = crate::repo::Repo::init(&dst_root).unwrap(); let t2 = LocalTransport::open(&dst_root).unwrap(); diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index a6c454cb..83eb78de 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -28,8 +28,13 @@ use crate::error::{Error, Result}; /// Bumped 4 -> 5 in P37 (ADR-0047): the additive `GetObjects` verb (batched /// by-id object fetch) — a v4 server would answer its opcode with an /// "unknown opcode" protocol error mid-session, so the mismatch is refused at -/// the handshake instead. -pub const PROTOCOL_VERSION: u32 = 5; +/// the handshake instead. Bumped 5 -> 6 in ADR-0055: `Request::GetPack` gains a +/// trailing optional `blob_limit` (the `--filter blob:limit=` clone +/// filter, evaluated by the server because only it knows a blob's size). A v5 +/// peer would reject the longer frame as trailing bytes, or worse, a v6 limit +/// riding in the prefix list would read as a path, so the mismatch is refused +/// at the handshake in both directions instead. +pub const PROTOCOL_VERSION: u32 = 6; // Request opcodes (one per Transport verb, plus session control). const OP_HELLO: u8 = 0x01; @@ -133,12 +138,16 @@ pub enum Request { id: ObjectId, expected_old: Option, }, - /// `filter`: partial-clone prefix list (P27 Task 3), empty = no filter - /// (full transfer, matching pre-P27 behavior byte-for-byte). + /// `filter`: partial-clone prefix list (P27 Task 3), empty = no prefix + /// filter. `blob_limit` (ADR-0055): leave out blobs of at least that many + /// bytes that no want's own root tree reaches. The transfer is filtered + /// when `filter` is non-empty or `blob_limit` is set; with neither it is a + /// full transfer, matching pre-P27 behavior byte-for-byte. GetPack { wants: Vec, haves: Vec, filter: Vec, + blob_limit: Option, }, /// Marker only (P25) — the pack body is no longer embedded in this /// request's frame. Immediately after this request frame, the sender @@ -218,6 +227,9 @@ impl<'a> Cur<'a> { fn u32(&mut self) -> Result { Ok(u32::from_be_bytes(self.take(4)?.try_into().unwrap())) } + fn u64(&mut self) -> Result { + Ok(u64::from_be_bytes(self.take(8)?.try_into().unwrap())) + } fn id(&mut self) -> Result { Ok(ObjectId::from_bytes(self.take(32)?.try_into().unwrap())) } @@ -319,11 +331,19 @@ impl Request { wants, haves, filter, + blob_limit, } => { out.push(OP_GET_PACK); put_ids(&mut out, wants); put_ids(&mut out, haves); put_strs(&mut out, filter); + match blob_limit { + Some(limit) => { + out.push(1); + out.extend_from_slice(&limit.to_be_bytes()); + } + None => out.push(0), + } } Request::PutPack => out.push(OP_PUT_PACK), Request::GetObjects { ids } => { @@ -366,6 +386,11 @@ impl Request { wants: c.ids()?, haves: c.ids()?, filter: c.strs()?, + blob_limit: match c.u8()? { + 0 => None, + 1 => Some(c.u64()?), + t => return Err(Error::Protocol(format!("bad blob_limit tag {t}"))), + }, }, OP_PUT_PACK => Request::PutPack, OP_GET_OBJECTS => Request::GetObjects { @@ -1012,7 +1037,7 @@ fn bucket_get_pack_tempfile( tmp_dir: &std::path::Path, wants: &[ObjectId], haves: &[ObjectId], - filter: Option<&[String]>, + filter: Option<&crate::promisor::PackFilter>, ) -> Result { let guard = TempPackGuard::new_in(tmp_dir)?; let mut f = std::fs::File::create(guard.path())?; @@ -1211,16 +1236,15 @@ fn serve_session( wants, haves, filter, + blob_limit, } => { - let filter_opt = if filter.is_empty() { - None - } else { - Some(filter.as_slice()) - }; + let filter_opt = crate::promisor::PackFilter::new(&filter, blob_limit); let result = match &transport { - ServeTransport::Local(t) => t.build_pack_tempfile(&wants, &haves, filter_opt), + ServeTransport::Local(t) => { + t.build_pack_tempfile(&wants, &haves, filter_opt.as_ref()) + } ServeTransport::Bucket { transport: bt, tmp } => { - bucket_get_pack_tempfile(bt, &tmp.0, &wants, &haves, filter_opt) + bucket_get_pack_tempfile(bt, &tmp.0, &wants, &haves, filter_opt.as_ref()) } }; match result { @@ -1529,16 +1553,37 @@ mod tests { wants: vec![some_id(7)], haves: vec![some_id(8), some_id(9)], filter: vec![], + blob_limit: None, }, Request::GetPack { wants: vec![some_id(7)], haves: vec![], filter: vec!["src/".to_string(), "docs/".to_string()], + blob_limit: None, + }, + Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: None, + }, + Request::GetPack { + wants: vec![some_id(7)], + haves: vec![], + filter: vec!["src/".to_string()], + blob_limit: Some(1 << 20), + }, + Request::GetPack { + wants: vec![some_id(7)], + haves: vec![], + filter: vec![], + blob_limit: Some(0), }, Request::GetPack { wants: vec![], haves: vec![], filter: vec![], + blob_limit: Some(u64::MAX), }, Request::PutPack, Request::GetObjects { ids: vec![] }, @@ -1851,6 +1896,93 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + /// ADR-0055: a v5 client (no `blob_limit` field) is refused at the + /// handshake with a message naming both versions, so it never sends a + /// `GetPack` the v6 decoder would misparse. + #[test] + fn serve_rejects_a_v5_client_naming_both_versions() { + let root = tmp_repo("v5client"); + let r = run_session(&root, &[Request::Hello { version: 5 }]); + match &r[0] { + Err(Error::Protocol(m)) => assert_eq!( + m, "unsupported protocol version 5 (server speaks 6)", + "got {m}" + ), + other => panic!("expected a protocol error, got {other:?}"), + } + std::fs::remove_dir_all(&root).unwrap(); + } + + /// ADR-0055, the other direction: a v6 client against a v5 server sees the + /// v5 server's own refusal (its `handshake_hello` text with its version), + /// and no `GetPack` is ever sent. + #[test] + fn a_v6_client_reports_a_v5_servers_refusal() { + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let f = read_frame(&mut server_read).unwrap(); + assert!(matches!( + Request::decode(&f).unwrap(), + Request::Hello { version: 6 } + )); + write_err( + &mut server_write, + EC_PROTOCOL, + "unsupported protocol version 6 (server speaks 5)", + ) + .unwrap(); + }); + let Err(err) = crate::stdio_transport::WireClient::handshake(client_read, client_write) + else { + panic!("the handshake must fail"); + }; + assert_eq!( + err.to_string(), + "wire protocol error: unsupported protocol version 6 (server speaks 5)" + ); + server.join().unwrap(); + } + + /// The trailing `blob_limit` tag is strict: an unknown tag or a truncated + /// limit is a protocol error, not a silently unfiltered pack. + #[test] + fn get_pack_decode_rejects_a_bad_blob_limit_tag() { + let mut ok = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: Some(7), + } + .encode(); + assert!(Request::decode(&ok).is_ok()); + let tag_at = ok.len() - 9; + ok[tag_at] = 9; + assert!(matches!(Request::decode(&ok), Err(Error::Protocol(_)))); + let mut truncated = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: Some(7), + } + .encode(); + truncated.pop(); + assert!(matches!( + Request::decode(&truncated), + Err(Error::Protocol(_)) + )); + // A v5 frame (no trailing field) no longer decodes. + let mut v5 = Request::GetPack { + wants: vec![], + haves: vec![], + filter: vec![], + blob_limit: None, + } + .encode(); + v5.pop(); + assert!(matches!(Request::decode(&v5), Err(Error::Protocol(_)))); + } + #[test] fn serve_reports_not_a_repo_as_typed_error() { let root = std::env::temp_dir().join(format!("scl-wire-norepo-{}", std::process::id())); @@ -2954,6 +3086,22 @@ mod tests { let got = scl_core::pack::parse_pack(&out).unwrap(); assert_eq!(got.len(), objects.len()); + // A served `--store` bucket refuses a partial transfer of either kind + // (ADR-0046, ADR-0055 B2): a size-only request is "filtered" too. + for filter in [ + crate::promisor::PackFilter::new(&["src/".to_string()], None), + crate::promisor::PackFilter::new(&[], Some(1024)), + ] { + let err = client + .get_pack(&[tip], &[], filter.as_ref(), &mut Vec::new()) + .unwrap_err(); + assert!( + err.to_string() + .contains("partial clone from bucket remotes"), + "{err}" + ); + } + // PutPack + UpdateRef land a second commit through the same session. let (tip2, objects2) = tiny_history_for_bucket("seed2", b"second commit"); let (pack2, _idx2) = scl_core::pack::build_pack(&objects2).unwrap(); diff --git a/crates/repo/src/worktree.rs b/crates/repo/src/worktree.rs index d1c31169..c6336ffd 100644 --- a/crates/repo/src/worktree.rs +++ b/crates/repo/src/worktree.rs @@ -734,21 +734,32 @@ pub(crate) fn preflight_view( let Some(promisor) = crate::promisor::load(layout)? else { return Ok(()); }; - let gap = |path: &str| { - let prefix = promisor - .prefixes() - .iter() - .find(|p| { - let bare = p.trim_end_matches('/'); - path == bare || path.starts_with(&format!("{bare}/")) - }) - .cloned() - // Outside the filter (a view widened past it): name the path - // itself, which `sc backfill` accepts as a prefix. For the root - // tree there is no path, so fall back to the first filter prefix. - .or_else(|| (!path.is_empty()).then(|| path.to_string())) - .or_else(|| promisor.prefixes().first().cloned()) - .unwrap_or_default(); + let gap = |path: &str, is_blob: bool| { + // With a blob limit `sc backfill` repairs a gap by its own path rather + // than a covering prefix (ADR-0055). The limit is named as a cause only + // for a missing file in the clone's scope: a missing directory, or a path + // outside the scope, was left out by a fetch or by the filter. + let sized = promisor + .blob_limit() + .filter(|_| is_blob && promisor.matches(path)); + let prefix = if promisor.blob_limit().is_some() && !path.is_empty() { + path.to_string() + } else { + promisor + .prefixes() + .iter() + .find(|p| { + let bare = p.trim_end_matches('/'); + path == bare || path.starts_with(&format!("{bare}/")) + }) + .cloned() + // Outside the filter (a view widened past it): name the path + // itself, which `sc backfill` accepts as a prefix. For the root + // tree there is no path, so fall back to the first filter prefix. + .or_else(|| (!path.is_empty()).then(|| path.to_string())) + .or_else(|| promisor.prefixes().first().cloned()) + .unwrap_or_default() + }; Error::GapInView { path: if path.is_empty() { "(root tree)".to_string() @@ -756,10 +767,11 @@ pub(crate) fn preflight_view( path.to_string() }, prefix, + blob_limit: sized, } }; if !store.contains(&root) { - return Err(gap("")); + return Err(gap("", false)); } let mut stack = vec![(root, String::new())]; while let Some((tree_id, path)) = stack.pop() { @@ -773,13 +785,13 @@ pub(crate) fn preflight_view( match e.kind { EntryKind::Blob => { if sparse.matches(&child) && !store.contains(&e.id) { - return Err(gap(&child)); + return Err(gap(&child, true)); } } EntryKind::Tree => { if sparse.should_descend(&child) { if !store.contains(&e.id) { - return Err(gap(&child)); + return Err(gap(&child, false)); } stack.push((e.id, child)); } diff --git a/demo/run_blob_limit_demo.sh b/demo/run_blob_limit_demo.sh new file mode 100755 index 00000000..b99472ba --- /dev/null +++ b/demo/run_blob_limit_demo.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# P27 follow-on demo: blob-size clone filter (ADR-0055, #217). Proves that +# `sc clone --filter blob:limit=` leaves large OLD blobs out of the +# transfer while every ref tip stays complete (the checkout is whole), that +# `sc verify` / `sc gc` / `sc push` work on such a clone, that `sc backfill +# ` repairs the gap, that `--lazy` is accepted with only a size filter, +# and that a prefix filter combines with a size limit. +# +# Self-checking: every claim is an assertion; any failure exits non-zero +# before the RESULT line. +set -euo pipefail + +cd "$(dirname "$0")/.." +cargo build --quiet --bin sc +SC="$PWD/target/debug/sc" + +fail() { echo "FAIL: $1"; exit 1; } + +W="$(mktemp -d)" +trap 'rm -rf "$W"' EXIT + +objects() { find "$1/.sc/objects" -type f | wc -l | tr -d ' '; } + +ORIGIN="$W/origin" +mkdir -p "$ORIGIN" +( + cd "$ORIGIN" + "$SC" init >/dev/null + mkdir -p src + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > assets.bin # v1, ~128 KiB + printf 'fn main() {}\n' > src/a.txt + "$SC" commit -m "v1" --author demo >/dev/null + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > assets.bin # v2 + printf 'fn main() { 2 }\n' > src/a.txt + "$SC" commit -m "v2" --author demo >/dev/null +) +FULL=$(objects "$ORIGIN") +echo "origin: two versions of a ~128 KiB assets.bin; $FULL object(s) ✔" + +echo +echo "=== 1: blob:limit=16KiB leaves the old assets.bin out, the checkout is whole ===" +LIM="$W/limited" +"$SC" clone --filter blob:limit=16KiB "$ORIGIN" "$LIM" > "$W/clone.out" +grep -q "partial: blob:limit=16384" "$W/clone.out" || fail "clone must report the limit: $(cat "$W/clone.out")" +[ "$(objects "$LIM")" -lt "$FULL" ] || fail "the limited clone must hold fewer objects than the origin" +cmp "$LIM/assets.bin" "$ORIGIN/assets.bin" || fail "the tip's large file must be checked out byte-identically" +[ -f "$LIM/.sc/promisor-blob-limit" ] || fail "the clone must record its limit" +echo "$(objects "$LIM") of $FULL object(s); tip assets.bin materialized ✔" + +echo +echo "=== 2: verify reports the gap as expected, gc and status are happy ===" +(cd "$LIM" && "$SC" verify) > "$W/verify.out" +grep -q "partial: 1 object(s) outside filter \[blob:limit=16384\]" "$W/verify.out" || fail "verify gap line: $(cat "$W/verify.out")" +(cd "$LIM" && "$SC" gc >/dev/null && "$SC" status >/dev/null) +echo "verify: 1 gap (the old assets.bin), gc + status fine ✔" + +echo +echo "=== 3: a commit pushes back; the origin keeps the history ===" +(cd "$LIM" && printf 'fn main() { 3 }\n' > src/a.txt && "$SC" commit -m "v3" --author demo >/dev/null && "$SC" push origin >/dev/null) +(cd "$ORIGIN" && "$SC" log | grep -q "v3") || fail "the origin must have the pushed commit" +echo "push from the limited clone landed ✔" + +echo +echo "=== 4: --lazy is accepted when the only filter is blob: ===" +LAZY="$W/lazy" +"$SC" clone --filter blob:limit=16KiB --lazy "$ORIGIN" "$LAZY" >/dev/null +[ -f "$LAZY/.sc/promisor-lazy" ] || fail "--lazy must work with only a blob: filter" +echo "lazy marker set on a size-only clone ✔" + +echo +echo "=== 5: a prefix filter combines with the limit ===" +COMBO="$W/combo" +"$SC" clone --filter src/ --filter blob:limit=16KiB "$ORIGIN" "$COMBO" > "$W/combo.out" +grep -q "partial: src/, blob:limit=16384" "$W/combo.out" || fail "combo report: $(cat "$W/combo.out")" +[ ! -e "$COMBO/assets.bin" ] || fail "assets.bin is outside the prefix and must not be materialized" +echo "src/ + blob:limit: scope first, limit second ✔" + +echo +echo "=== 6: grammar errors are actionable ===" +if "$SC" clone --filter blob:none "$ORIGIN" "$W/bad" 2>"$W/bad.err"; then fail "blob:none must be rejected"; fi +grep -q "blob:limit=0" "$W/bad.err" || fail "blob:none error must point at blob:limit=0: $(cat "$W/bad.err")" +echo "blob:none -> use blob:limit=0 ✔" + +echo +echo "=== 7: backfill by the file's own path, then --all ===" +(cd "$LIM" && "$SC" backfill assets.bin >/dev/null) +(cd "$LIM" && "$SC" verify) | grep -q "partial: 0 object(s)" || fail "backfill must close the gap" +(cd "$LIM" && "$SC" backfill --all >/dev/null) +[ ! -e "$LIM/.sc/promisor" ] && [ ! -e "$LIM/.sc/promisor-blob-limit" ] || fail "--all must remove both markers" +echo "backfill assets.bin closed the gap; --all made it a full clone ✔" + +echo +echo "RESULT: blob-size clone filter proven (limit, tips complete, verify/gc/push, lazy marker, combo, repair)" diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 7fdeb542..5091a62d 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -541,6 +541,24 @@ too, not just the src-control-side metadata. no longer holds the repo lock (ADR-0056), so an `sc` it runs from inside the workspace finds the host repo and can change it, as a hand-started agent always could. +- **A blob-size limit (`--filter blob:limit=`, ADR-0055) is request scope + like the prefix filter, and the server evaluates it.** It hides nothing: any + client that can read the repository can fetch the left-out blobs (`sc + backfill`, by-id fill), and a server that sends fewer blobs than asked can + only create a gap, which surfaces as `GapInView`/`GapBySize` or a fill error + before any write, never as wrong content (objects are hash-verified). The + size is the stored payload length, so for a protected path it is the + ciphertext size, which anyone who can fetch the object already sees; the + filter reveals nothing new. A limited `GetPack` makes the server stream-measure + each distinct candidate blob once (`Store::blob_size`, fixed buffer, hash + checked, cached by id; a limit of 0 or `u64::MAX` measures nothing), bounded by + the stored bytes of those blobs. A blob that is kept is read twice, once to + measure and once to write the pack, so the read cost is at most twice an + unfiltered `GetPack`'s, which already reads every blob into its pack. Reads are not budgeted (#233 budgets writes) + and the filter does not change that; the listener limits are the bound. Every + active filter, a limit alone included, still excludes private branches and is + refused by bucket remotes. The wire bump (version 6) makes a mixed-version pair + fail closed at the handshake in both directions. - **Operations that need a complete closure refuse on a partial clone, fail closed:** `sc export`, `sc rewrap`, `sc grant`/`revoke`, private branches, and `sc push` to a git remote. They refuse rather than run on partial data. diff --git a/docs/adr/0037-partial-clone.md b/docs/adr/0037-partial-clone.md index b2f29670..45f2f9ad 100644 --- a/docs/adr/0037-partial-clone.md +++ b/docs/adr/0037-partial-clone.md @@ -1,6 +1,6 @@ # ADR-0037: Partial clone (promisor store + prefix-scoped fetch) -- **Status:** Accepted; the merge, replay and session refusals are lifted by [ADR-0049](0049-merge-on-partial-clones.md) and [ADR-0050](0050-sessions-on-partial-clones.md), and the filter-based walks replaced by [ADR-0048](0048-presence-based-partial-clones.md) +- **Status:** Accepted; the merge, replay and session refusals are lifted by [ADR-0049](0049-merge-on-partial-clones.md) and [ADR-0050](0050-sessions-on-partial-clones.md), and the filter-based walks replaced by [ADR-0048](0048-presence-based-partial-clones.md); the blob-size axis it deferred is [ADR-0055](0055-blob-size-clone-filter.md) - **Date:** 2026-07-09 - **Phase:** 27 - **Builds on:** ADR-0034 (sparse checkout — the materialize half), ADR-0035 (streaming pack transfer), ADR-0015 (packfiles + reachability gc), ADR-0022/0036 (transports) @@ -73,7 +73,10 @@ independent filters). (sparse-view into un-fetched gaps with no "one slice" mental model). - **Blob-size / object-count filters** (git's `--filter=blob:limit`): prefix-only matches sparse and the path-boundary machinery already - built; size filters are a separate axis, deferred. + built; size filters are a separate axis, deferred. *Since built:* the size + filter is [ADR-0055](0055-blob-size-clone-filter.md) (`--filter + blob:limit=`, server-evaluated, `PROTOCOL_VERSION` 6); the object-count + limit stays deferred. ## Refinements discovered during the build diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index af4d692b..48383a51 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -1,6 +1,6 @@ # ADR-0048: Presence-based walks for partial clones (P37b) -- **Status:** Accepted (built) +- **Status:** Accepted (built); [ADR-0055](0055-blob-size-clone-filter.md) adds a second kind of gap (a blob left out by size) that these presence-based walks cover unchanged - **Date:** 2026-10-16 - **Phase:** 37b - **Builds on:** ADR-0047 (want-by-id fetch, 37a), ADR-0037 (partial clone), ADR-0034 (sparse), ADR-0035 (streaming transfer), ADR-0039/0041 (size limits) @@ -332,6 +332,19 @@ review established, as constraints on that work: through the export walk at the first absent object. 37b-i adds the same up-front refusal that `sc export` has. +## Addendum: a blob left out by size (ADR-0055) + +`sc clone --filter blob:limit=` leaves out blobs by size rather than by +path, and the model above covers it without a new record: the gap is an absent +blob a present tree references, the promise P is unchanged, and push, verify and +gc already ask the store. What changes: the filter now has a second dimension, +`.sc/promisor-blob-limit` records it and an explicit `Scope` (a size-only clone +is `AllPaths`, which an empty prefix list could not say), "filter active" means +`prefixes || blob_limit` everywhere, and a full sparse view must still fill on a +lazy `AllPaths` clone (`ensure_view`). Every ref tip is complete at clone time; +a later fetch can bring a tip that points at a gapped blob, which is the +`GapInView` case above with the limit named in its message. + ## Phasing - **37b-i, engine (built).** `reachable_objects_partial`; gc, push and verify on it; @@ -410,6 +423,7 @@ review established, as constraints on that work: the fetch and the record, and nothing it proves that the push-time check does not. - **Make `reachable_objects_filtered` presence-aware for everyone.** The origin - uses it to cut transfers. + uses it to cut transfers. (ADR-0055 extends that same walk with a blob-size + rule and every-tip exemption, `reachable_objects_limited`, still server-side.) - **A tree-recursive merge by id in this ADR.** Withdrawn; see decision 6. - **Server-side merge and history operations.** Still deferred (ROADMAP). diff --git a/docs/adr/0055-blob-size-clone-filter.md b/docs/adr/0055-blob-size-clone-filter.md new file mode 100644 index 00000000..fd12b357 --- /dev/null +++ b/docs/adr/0055-blob-size-clone-filter.md @@ -0,0 +1,282 @@ +# ADR-0055: Blob-size clone filter (`--filter blob:limit=`) + +- **Status:** Accepted (built). Accepted after an independent Opus plan review and a Sol review; their blockers are decisions 3, 5, 7 and 8 below. The object-count limit and a limit-aware `sc fetch` are deferred (decision 2). +- **Date:** 2026-10-20 +- **Phase:** 27 follow-on (issue #217) +- **Builds on:** ADR-0037 (partial clone), ADR-0047 (want-by-id fetch), ADR-0048 (presence-based walks), ADR-0049/0050 (merge and sessions on partial clones), ADR-0044 (private branches), ADR-0014 (protected paths), ADR-0046 (bucket remotes) + +## Context + +`sc clone --filter` took path prefixes only. Issue #217 asks for git's +`blob:limit` axis (leave out blobs above a size) and an object-count limit. + +ADR-0048 already made a partial clone a store plus one promise: every id a +present object references and the store lacks is held by the origin. The +filter records what a clone *asked for*; presence decides what exists. A blob +left out by size is the same kind of gap as one left out by path, so the client +side needs almost nothing new. The open questions were who learns the size, and +what a checkout does with a gapped file. + +## Findings that shape the design + +1. **Only the server can know a blob's size before receiving it.** A + `TreeEntry` is `{name, kind, id, mode, perms}` (`core/src/object.rs`); there + is no size. The wire has no size query: `HasObject` returns a bool, + `GetObject`/`GetObjects` return the object itself. The by-id reply caps + (ADR-0048 decision 5) refuse an over-cap *batch*; they do not report sizes. +2. **The prefix filter cannot carry it.** `GetPack` shipped `filter: + Vec` of path prefixes. A reserved entry such as `blob:limit=1048576` + in that list would be read by a v5 server as a path prefix and silently send + almost nothing, a fail-unsafe misparse. ADR-0047 rejected the same trick for + the lazy flag. +3. **The walk that cuts a transfer never read blobs.** `walk_tree_filtered` + decided blob inclusion from the path alone. A size rule makes the server + measure each candidate blob once per request. +4. **Git has no object-count filter** (`blob:none`, `blob:limit`, `tree:`, + `object:type`, `sparse:oid`, `combine`). A count limit is also undefined + here: `reachable_objects_filtered` collects into a `BTreeSet` ordered by id, + so "the first N objects" would be an arbitrary, server-dependent set, and a + client could not tell a cut-off object from a missing one by any rule. + +## Decisions + +### 1. User-facing form + +`sc clone --filter blob:limit= `, with `` parsed by +`scl_core::parse_byte_size` (`1048576`, `1MiB`, `512KiB`). It combines with +prefix filters in the same repeatable flag: `--filter src/ --filter +blob:limit=1MiB` means "only `src/`, and within it no blob of 1 MiB or more +unless decision 4 keeps it". + +**Boundary:** a blob of size >= the limit is left out (like git; `blob:limit=1` +leaves out every non-empty blob outside the tips). `blob:limit=0` leaves out +every blob no ref tip reaches. Size is the stored object's payload length. + +**Grammar.** `blob:` is reserved in `--filter`, matched after trimming and +lower-casing (so `Blob:limit=1KiB` is the filter, never a path prefix that +matches nothing): only `blob:limit=` is +accepted, once. A second `blob:limit` is an error. `blob:none` is an error that +points at `blob:limit=0`. Any other `blob:` form is an error naming the one +supported form. Cost: a path prefix that literally begins with `blob:` can no +longer be named in `--filter`. `sc clone --help` documents this. `--lazy` +requires `--filter`, which a lone `blob:limit=` satisfies, and works with it. +A limit above the largest transferable object (1 GiB, `MAX_MAX_OBJECT_SIZE`) +filters nothing and is an error. Rejected: a separate `--blob-limit` flag, which has no collision but diverges +from the issue and from git. + +### 2. Deferred + +- **Object-count limit.** No stable definition (finding 4), no git precedent, no + stated use case. `ROADMAP.md` keeps a one-line deferred entry. +- **A limit-aware `sc fetch`.** Fetch stays unfiltered (ADR-0048 Context), so new + large blobs arrive in full and the limit bounds the initial transfer, not later + growth. A limit-aware fetch needs the paths a user backfilled to ride in the + request as exemptions, which is a larger design. + +### 3. The server evaluates the filter: wire change, `PROTOCOL_VERSION` 5 to 6 + +Findings 1 and 2 leave no client-side design that saves transfer. + +- `Request::GetPack` gains a trailing optional `blob_limit` (a tag byte, then a + big-endian `u64` when present; absent means no limit, so `blob:limit=0` stays + representable). `PROTOCOL_VERSION` 5 to 6, in the same style as P25, P27, P34 + and P37. +- `Transport::get_pack` takes a `PackFilter` instead of a prefix slice; every + implementation (`LocalTransport`, `WireClient`, stdio, http, bucket) carries + it, and the two wrappers forward by hand as in ADR-0047. +- The sender's walk (`reachable_objects_limited`) moves a blob at or over the + limit from `included` to `gaps`, unless decision 4 exempts it. The `haves` side + stays unfiltered, as ADR-0037 requires. + +**Alternatives that avoid a bump, and why they lose.** + +- *A capability-probed new verb* (say `GetPackSized`): a v5 server answers an + unknown opcode with a protocol error mid-session, so the client needs a probe + round trip first, and `Hello` carries only a version, so the probe is itself a + new verb. It buys mixed-version operation only for the size filter and leaves + every other verb at v5, two code paths to keep forever. +- *A trailing optional field on `GetPack` without a bump:* a v5 decoder ends with + `done()` and rejects trailing bytes as `trailing bytes in frame`, a mid-session + error the client cannot distinguish from corruption, with no way to fall back. +- *A size manifest plus `GetObjects`:* the client would walk all history level by + level to learn blob ids, ask for sizes through a new verb (a bump anyway), then + fetch: many round trips for what the server does in one walk. +- *Client-side drop on ingest:* the client downloads every blob and discards the + large ones. It saves disk and gives the same gap model, but not one byte of + transfer, which is the point of the feature. +- *A sentinel in the prefix list:* finding 2. + +**Cost of the bump.** The server and its clients upgrade together. Every mixed +pair fails closed at the handshake, before any repository access, in both +directions, for `sc+http(s)://` and for ssh remotes alike (an `ssh://` remote +runs the remote host's own `sc serve --stdio` binary, so its version is whatever +that host has installed). The messages are the server's own: + +- v6 client, v5 server: `wire protocol error: unsupported protocol version 6 + (server speaks 5)`. +- v5 client, v6 server: `unsupported protocol version 5 (server speaks 6)`, which + the v5 client prints as `wire protocol error: unsupported protocol version 5 + (server speaks 6)`. + +Pinned by `a_v6_client_reports_a_v5_servers_refusal`, +`serve_rejects_a_v5_client_naming_both_versions` and `handshake_rejects_v5_peer` +(the retargeted successor of `handshake_rejects_v4_peer`), plus encode and decode +round trips of `GetPack` with and without a limit and strict rejection of a bad +tag or a v5-length frame. + +### 4. Every ref tip is complete; the limit applies to history + +The walk exempts blobs reachable from the root tree of each *want* (every ref tip +the client asked for) that match the prefix filter. The limit applies to blobs +reachable only through parent snapshots. + +Reason: checkout, `sc switch` between branches and every operation that +materializes a tip need those blobs, and the clone's own initial checkout runs +before any lazy marker exists. A strict limit would make a non-lazy clone fail at +its own checkout with `GapInView`, and a lazy one would fetch the same large +files in a second round trip right after the clone. Skipping large tracked files +at checkout would need a third "tracked but absent" state in status, diff, commit +graft, materialize and the merge engine (the sparse machinery covers paths, not +sizes). + +**Trade-off.** The filter saves old revisions of large files and large files only +reachable from history, which is where git's `blob:limit` pays off for binary +assets. It does not skip a large file at HEAD or at any other branch tip, so a +clone with many branches carries each tip's large files. "Reports or lazily fills +large ones" from the issue is therefore met for the history case: an operation +that reaches an old gapped blob fills it by id on a lazy clone and fails with the +gap error otherwise. + +**Tips are complete at clone time only (D2).** A later `sc fetch` is unfiltered +and subtracts the closure of the local tips, so a new tip that points at an old, +gapped blob (a revert) arrives without it. A lazy clone heals that on the +fast-forward or switch (decision 7); a non-lazy one gets the gap error with the +limit clause (decision 6) and `sc backfill ` repairs it. + +**Ordering (D7).** `walk_snapshots` queues every tip before any parent, so a +tip's exempt walk always runs before a non-tip snapshot's limited one, even when +one tip is an ancestor of another. The `(id, path)` expansion gate then only ever +lets a limited walk skip work an exempt walk already did, never the reverse, and +a blob already `included` is kept without measuring. A tip root tree that equals +an older subtree is expanded at its own `(root, "")` key. Tests: an ancestor tip, +a tip root equal to a subtree of a non-tip, and the walk with the wants in both +orders. + +### 5. Scope is explicit; `filter active` has one definition (B1, B2) + +- **`Promisor`** holds `Scope` (`AllPaths` or `Prefixes(list)`), `blob_limit: + Option` and the origin. `AllPaths` is a clone with a limit and no prefix; + an empty prefix list means "matches nothing" and cannot say it. Every consumer + of `matches`/`should_descend`/`prefixes` handles the case (graft, materialize + bound, sparse refusals, gap hints, verify text, widen), and `widen` on + `AllPaths` is a no-op, so `backfill ` cannot narrow the scope. +- **Persistence.** `.sc/promisor` is unchanged (`origin ` then one prefix per + line). `.sc/promisor-blob-limit` (local, uncommitted) holds `limit ` and + `scope all-paths|prefixes`, written before `.sc/promisor` and removed with it + (`promisor::remove`, so `backfill --all` leaves neither). It is a separate file + for the ADR-0047 reason: any text is a legal path. A malformed file is an error, + not an unlimited clone. A limit file with no `.sc/promisor` is leftover state + and reads as a full clone. +- **An older binary** cannot talk to a v6 server, and locally it reads + `.sc/promisor` only: a size-only clone has an empty prefix list there, which it + treats as a degenerate partial clone that matches nothing (gaps everywhere, + never wrong content), and it ignores the limit file. If it ran `backfill + `, that would add a prefix to `.sc/promisor`; the current binary still + reads `scope all-paths` from the limit file and keeps the whole-tree scope. If + it ran `backfill --all`, it would remove `.sc/promisor` and leave the limit file, + which the current binary ignores. +- **`PackFilter`** is the one definition of "filter active": it exists only when + `!prefixes.is_empty() || blob_limit.is_some()`. Every `Some(&PackFilter)` + therefore means a partial transfer, in all of these places, each with a test: + the server walk and its private-branch handling (`filter_manifest_wants` drops + manifest wants), the bucket refusal (`BucketTransport::get_pack`, a served + `--store` over the wire, and a direct `sc+wal://` clone through the CLI), and + the client's clone, which skips refs whose tip did not arrive. + +### 6. Gap record, presentation and repair + +- **No new gap record.** Absence is the gap (ADR-0048). Push, verify and gc + already walk presence (`reachable_objects_partial`) and need no code change; + tests pin them on a size-limited clone, including the push gap-edge check + against a destination that lacks the history. +- **Presentation.** `sc clone` prints `partial: blob:limit=` (prefixes + first when both). `sc verify` keeps `partial: N object(s) outside filter + [...]` with the same bracket text. `GapInView` gains the limit: its message + names `blob:limit=` as a cause beside "an earlier fetch left a gap", and + `prefix` is the file's own path, not a covering prefix. Gaps found by a merge, + pick, rebase, session or `sc cat` on a clone with a limit are + `GapBySize { path, limit }`: "`` was left out of this partial clone + (blob:limit=); run `sc backfill ` to fetch it first". A prefix-only + clone keeps every old message byte for byte. The limit is named only for a + missing file in scope; a missing directory (a fetch left it out) is named by its + path, and a missing root tree points at `sc backfill --all` (merge needs carry + a tree/blob flag for this). +- **Repair, lazy and online:** nothing to do. `ensure_view`, `sc cat` and the + ADR-0049 merge fill by id. +- **Repair, otherwise:** `sc backfill ` fetches every version of that path + (backfill sends no limit), and `sc backfill --all` converts to a full clone. No + `--min-size` form: it would be a second size parser on the repair side with no + use that `` does not cover. + +### 7. A full view must still fill (B3) + +`Repo::ensure_view` returned without filling when the sparse view was full, so a +lazy size-only clone, whose view is always full, would have failed on switch, +undo, merge, replay and sessions to a revision with a gapped blob. The guard now +skips only when the clone is not `AllPaths`. For an `AllPaths` clone the fill is +bounded by the one revision being laid out. Tests: switch and undo to a gapped +revision (refused, then filled when lazy), merge, cherry-pick and rebase with a +gapped merge base (the engine fills it by id, or refuses before any write with +the file's own path), and a workspace session on a size-limited clone. + +### 8. Interactions + +- **Protected content (D6).** The size is the stored object's payload length. A + protected file's blob is ciphertext, so its size is the plaintext size plus the + sealing overhead, and a file near the limit can land on the other side of it + from its plaintext size. Protected and plain blobs follow the same rule. No new + information leaks: ciphertext length is already visible to anyone who can fetch + the object. +- **Private branches.** A filtered transfer excludes them (ADR-0044), now under + any active filter (decision 5). +- **Sealed objects, signatures, transcripts.** Not path-reached blobs; never + subject to the limit. +- **Sparse and prefix filters.** The prefix decides scope first, the limit prunes + inside it. A size-only clone leaves `.sc/sparse` empty (a full view). It may + narrow that view with `sc sparse set` and return to it with `sc sparse set` + (no prefixes) or `sc sparse disable`: a full view is its starting state, so + those two refusals apply to prefix scopes only, and the closure is filled + (lazy) or refused with `GapInView` before anything changes. +- **Push, verify, gc.** Presence-based (ADR-0048). +- **Merge, pick, rebase, ws, work** (ADR-0049/0050). No new engine code; a size + gap is "asked for" like any other (decision 7). + Named and concurrent sessions (ADR-0056) hold too: a session records the + host's view at fork, which for a size-only clone is the full view (`sparse` + empty), a fork starts at a ref tip so its base is complete, and each harvest + lands through the same merge. `named_sessions_on_a_size_limited_clone` pins it. +- **Server cost and limits (D5).** The walk measures a blob with `Store::blob_size` + (streamed through a fixed buffer, hash-verified, never admitted to the object + cache) and caches each verdict by blob id, so a distinct blob is measured at + most once per request (a limit of 0 or `u64::MAX` measures nothing). The work + is bounded by the stored bytes of the distinct candidate blobs; a kept blob is + read twice (measure, then pack), so the read cost is at most twice that of an + unfiltered `GetPack`, which already reads every blob into its pack. The filter + changes bandwidth, not the order of magnitude of the work. Reads are unbudgeted (#233 budgets + writes), and the filter does not change that: `--max-connections`, `--timeout` + and the listener limits remain the bounds. A size index on the server is a later + optimization. + +## Consequences + +- A clone can skip the old revisions of large files, with a complete checkout and + every ref tip complete. +- Wire version 6; servers and clients upgrade together. +- A new marker file, one new scope case in `Promisor`, one new error variant + (`GapBySize`) and a field on `GapInView`. +- `sc fetch` is not limit-aware (decision 2). + +## Open questions + +1. A limit-aware `sc fetch` with per-path exemptions for backfilled files. +2. An object-count or `tree:`-style filter, if a use case appears. +3. A server-side size index so a limited request need not stream-measure. diff --git a/docs/adr/README.md b/docs/adr/README.md index 76205858..6a5eb47d 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -65,6 +65,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0052](0052-secret-injection-by-fd.md) | Secret injection by file descriptor (`sc run --secrets-fd`) | Accepted (built) | — | | [0053](0053-tls-pin-management.md) | TLS pin management (`sc tls`) | Accepted (built) | — | | [0054](0054-tls-ca-validation.md) | CA-path validation for `sc+https://` | Accepted (built) | — | +| [0055](0055-blob-size-clone-filter.md) | Blob-size clone filter (`--filter blob:limit=`) | Accepted (built) | 27 | | [0056](0056-named-concurrent-ws-sessions.md) | Named and concurrent `sc ws` sessions | Accepted (built; refresh designed, deferred) | 20 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are