diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 9cbeea8a..c27a5048 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -296,6 +296,23 @@ cargo run --bin sc -- backfill --all # fetch EVERY remaining object (no # genuine full clone and merge/pick/rebase/ws # fork/harvest/sc work/export/sparse disable all # work again (P27 final review I2) +cargo run --bin sc -- clone --filter --lazy + # lazy partial clone (P37): also writes + # .sc/promisor-lazy so `sc cat` may fetch missing + # trees/blobs by object id; requires --filter +cargo run --bin sc -- backfill --lazy [prefix…] + # mark an existing partial clone lazy (P37); with + # prefixes, also backfills them; conflicts --all +cargo run --bin sc -- cat : # print one file as of a ref (P37). On a lazy + # partial clone fetches only the absent trees and + # the blob on that path (one GetObjects round trip + # per level). SC_OFFLINE=1 forces the gap error. + # Refuses protected paths and private branches +bash demo/run_lazy_fetch_demo.sh # by-id fetch proof (P37a): cat of one path adds + # exactly 3 objects (2 trees + blob), the sibling + # stays unfetched, offline/non-lazy keep the gap + # error, and a read-only sc+http server answers + # GetObjects (protocol v5) bash demo/run_partial_clone_demo.sh # partial clone proof (P27): sc clone --filter # src/ fetches fewer objects than a full clone # (object-store count + sc verify's gap report); diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 346d05d7..de70bc0e 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1060,3 +1060,18 @@ at accepted `BranchManifest` metadata and becomes `private_opaque`; authors, messages, paths, timestamps, and inner topology never cross IPC. The WebView has no generic dialog, filesystem, URL, process, or object-store command. See ADR-0045. + +## Phase 37a — want-by-id fetch (built) + +A partial clone marked lazy (`.sc/promisor-lazy`) can read a path outside its +fetch filter without `sc backfill`. `sc cat :` walks from the ref's +root tree; each absent tree, then the blob, is fetched with one `GetObjects` +round trip (`PROTOCOL_VERSION` 5): the client names ids, the server answers +with the objects it can serve plus the ids it will not send, with no haves and +no reachability walk. `Store::get` stays network-free; the only code that dials +the promisor after clone is `Repo::fetch_by_id_from` in `crates/repo/src/ +fill.rs`, which verifies the reply against the request before writing. A fill +only adds objects, so push, gc and verify (filter-scoped walks) are unchanged. +Sparse widening over a filled path and the lifted P27 refusals need +presence-based walks and are phase 37b. See ADR-0047 and +`docs/research/scion-evaluation.md`. diff --git a/CLAUDE.md b/CLAUDE.md index 0f05fa0b..2919495d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,7 @@ the project tree if desired. ## Capability map (what's built, by phase) -All 36 phases are built and tested. One line of current fact per phase; the +All 36 phases and P37a are built and tested. One line of current fact per phase; the authoritative rationale and full semantics live in the linked ADR, the design in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not @@ -151,6 +151,7 @@ the code, those win. | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) | | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | ## Standing boundaries & gotchas @@ -158,8 +159,8 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in `docs/THREAT-MODEL.md` — read it before touching anything crypto- or transport-adjacent. The rest, imperatively: -- **Wire protocol is version 4** (bumped from 3 in P34 for the two additive - sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the +- **Wire protocol is version 5** (bumped from 4 in P37 for the additive + `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the transfer path only — a larger blob commits locally but fails every subsequent push/fetch/clone at the receiver. - **The P35 desktop app is a keyless, read-only top-level adapter.** Its diff --git a/ROADMAP.md b/ROADMAP.md index 80311282..dd28108c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -548,12 +548,12 @@ across every phase. ## Active -**P37 — Want-by-id fetch and gap fill (proposed, not started).** Batched -`GetObjects` verb + an explicit `fill_gaps` seam so partial clones can run -merge/rebase/ws/work/export without `Store::get` ever touching the network; -design in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md), prior art -in `docs/research/scion-evaluation.md`. Phases: 37a (verb + read-only entry -points), 37b (lift the P27 refusals). +**P37b — presence-based walks and the lifted P27 refusals (proposed).** +P37a (built) added the batched `GetObjects` verb and `sc cat`; the remaining +work needs push, verify and gc to ask "is this object present?" instead of "is +this path in the filter?", then sparse widen over a filled prefix, then merge, +rebase, `ws`, `work` and `export` on partial clones. Design and open questions +in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md). ## Completed phases (usability-first ordering) @@ -591,6 +591,7 @@ points), 37b (lift the P27 refusals). | **P33 — Randomized protected encryption** | Close equality confirmation for new protected content | fresh DEK + nonce per seal, dual-read/randomized-write, keyed local stat cache, rewrap upgrade | [0043](docs/adr/0043-randomized-protected-encryption.md) | | **P34 — Private branches** | Keep an entire branch opaque before publication | sealed object closure behind a manifest; O(1) grant, KEK-rotating revoke, scanner-gated publish | [0044](docs/adr/0044-per-branch-access-control.md) | | **P35 — Native desktop browser** | Prove native `.sc` objects can power a polished visual tool | Tauri app opens a repository and browses refs, snapshot DAG/provenance, public trees, and first-parent diffs while locked/private states remain safe | [0045](docs/adr/0045-native-desktop-read-model.md) | +| **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | > **Prior art.** Phases P5–P9 adapt decisions from the sibling project > [git.agentic](https://github.com/git-agentic/git.agentic) (same BLAKE3 @@ -743,9 +744,9 @@ scale-&-reach horizon): origin automatically on any gap, matching git's promisor default. Real scope/risk (offline breakage, surprise latency, fetch-during-gc), so it stays deferred, not accidentally missing. - **Now scoped as P37 (ADR-0047, Proposed):** an opt-in explicit `fill_gaps` - seam over a batched by-id `GetObjects` verb, keeping `Store::get` - network-free. + **Now scoped as P37 (ADR-0047):** an opt-in `.sc/promisor-lazy` marker over a + batched by-id `GetObjects` verb, keeping `Store::get` network-free. P37a + (`sc cat`) is built; the rest is P37b. - **Per-case gap-tolerant merge/rebase/ws-harvest/sc-work on a partial clone (P27).** These are refused entirely today (`Error::PartialCloneUnsupported`) rather than gap-tolerantly @@ -754,7 +755,7 @@ scale-&-reach horizon): individually is the follow-on if the "backfill to a full clone first" workaround proves too coarse in practice. **Now scoped as P37b (ADR-0047, Proposed):** lifted one operation at a - time on top of the P37a `fill_gaps` seam. + time once push, verify and gc are presence-based. - **Blob-size/object-count clone filters (P27).** `sc clone --filter` is prefix-only, matching the sparse path-boundary machinery already built; git's `--filter=blob:limit` size-based filtering is a separate axis, diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 3a576e9e..ffc7c21c 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -303,6 +303,12 @@ enum Cmd { /// over a git-bridge remote. #[arg(long)] filter: Vec, + /// Mark the partial clone lazy (P37): `sc cat` may fetch the trees + /// and blobs it needs from the promisor remote by object id, instead + /// of failing with a gap error. `SC_OFFLINE=1` turns it off per + /// process. Requires `--filter`. + #[arg(long, requires = "filter")] + lazy: bool, }, /// Serve a repo to a remote `sc` client, either over stdin/stdout /// (invoked by `ssh` for ssh:// remotes) or over TCP (`sc+http://` @@ -499,8 +505,22 @@ enum Cmd { /// and merge/pick/rebase/`ws harvest`/`sc work`/export/sparse /// disable all work again. Mutually exclusive with `prefixes` /// (P27 final review I2). - #[arg(long)] + #[arg(long, conflicts_with = "lazy")] all: bool, + /// Mark this partial clone lazy (P37): `sc cat` may fetch missing + /// trees and blobs by object id (`SC_OFFLINE=1` disables it per + /// process). May be given with no prefixes. + #[arg(long)] + lazy: bool, + }, + /// Print one file as of a ref: `sc cat :` (P37). On a lazy + /// partial clone, missing trees and the blob are fetched by object id. + /// Public content only: protected paths and private branches are refused. + /// `` is a branch or remote-tracking ref (`main`, `origin/main`); + /// `HEAD` is not accepted. + Cat { + /// `:`, e.g. `main:docs/api/guide.md`. + target: String, }, /// Session transcript operations (P30): attach a sealed agent-session /// transcript to a commit, list/inspect them, and sign them. @@ -942,7 +962,9 @@ fn main() -> Result<()> { dst, git, filter, - } => run_clone(src, dst, git, filter), + lazy, + } => run_clone(src, dst, git, filter, lazy), + Cmd::Cat { target } => run_cat(&target), Cmd::Serve { sub, stdio, @@ -1022,7 +1044,11 @@ fn main() -> Result<()> { Cmd::Sign { r#ref, identity } => run_sign(&r#ref, identity), Cmd::Verify { r#ref, require } => run_verify(r#ref, require), Cmd::Sparse { op } => run_sparse(op), - Cmd::Backfill { prefixes, all } => run_backfill(prefixes, all), + Cmd::Backfill { + prefixes, + all, + lazy, + } => run_backfill(prefixes, all, lazy), Cmd::Transcript { op } => run_transcript(op), } } @@ -3395,7 +3421,20 @@ fn resolve_ids_to_pubkeys( Ok(out) } -fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec) -> Result<()> { +/// A relative local clone source is recorded as the remote and promisor +/// origin; resolved from inside the clone it would name the wrong directory, +/// so persist the absolute path. URLs (`ssh://`, `sc+http://`, ...) and paths +/// that do not resolve are returned unchanged. +fn absolute_local_source(src: String) -> String { + if src.contains("://") { + return src; + } + std::fs::canonicalize(&src) + .map(|p| p.display().to_string()) + .unwrap_or(src) +} + +fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec, lazy: bool) -> Result<()> { // Auto-detect unambiguous git URL forms (https/http, scp-style, file://): // those can never be sc-native, so no flag is needed. Bare `ssh://` is // ambiguous — it means an sc-native remote (ADR-0022) unless `--git` @@ -3407,8 +3446,12 @@ fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec) -> Resul } return run_clone_git(&src, &dst); } + let src = absolute_local_source(src); let filter_opt = (!filter.is_empty()).then_some(filter.as_slice()); let repo = scl_repo::Repo::clone_url_filtered(&src, &dst, filter_opt)?; + if lazy { + scl_repo::promisor::set_lazy(repo.layout(), true)?; + } let n = repo.branches()?.len(); if filter_opt.is_some() { println!( @@ -3428,8 +3471,20 @@ fn run_clone(src: String, dst: PathBuf, git: bool, filter: Vec) -> Resul /// origin and widen `.sc/promisor`. Errors if this repo is not a partial /// clone. `--all` ignores `prefixes` and instead fetches every remaining /// object, then removes `.sc/promisor` entirely (P27 final review I2). -fn run_backfill(prefixes: Vec, all: bool) -> Result<()> { +fn run_backfill(prefixes: Vec, all: bool, lazy: bool) -> Result<()> { let repo = open_repo()?; + if lazy { + if repo.promisor()?.is_none() { + anyhow::bail!("not a partial clone (.sc/promisor absent); --lazy has nothing to mark"); + } + if !prefixes.is_empty() { + repo.backfill(&prefixes)?; + println!("backfilled {} prefix(es)", prefixes.len()); + } + scl_repo::promisor::set_lazy(repo.layout(), true)?; + println!("marked lazy: `sc cat` may fetch missing objects by id"); + return Ok(()); + } if all { if !prefixes.is_empty() { anyhow::bail!("--all fetches every remaining object; pass either --all or explicit prefixes, not both"); @@ -3445,6 +3500,20 @@ fn run_backfill(prefixes: Vec, all: bool) -> Result<()> { Ok(()) } +/// `sc cat :`: write one file's bytes to stdout. Branch names are +/// flat (no `:`), so the first `:` splits ref from path. +fn run_cat(target: &str) -> Result<()> { + let (refname, path) = target + .split_once(':') + .filter(|(r, p)| !r.is_empty() && !p.is_empty()) + .ok_or_else(|| anyhow::anyhow!("expected :, got '{target}'"))?; + let repo = open_repo()?; + let bytes = repo.cat_at(refname, path)?; + use std::io::Write; + std::io::stdout().write_all(&bytes)?; + Ok(()) +} + /// Clone from a hosted Git URL: init + remote add origin --git + fetch + /// adopt the remote's default branch (P10's unborn fast-forward adoption). /// Reached by auto-detect for unambiguous git URL forms, or by `--git` for @@ -4586,6 +4655,7 @@ mod tests { dst.clone(), false, Vec::new(), + false, ) .unwrap(); let repo = scl_repo::Repo::open(&dst).unwrap(); @@ -4602,6 +4672,21 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + #[test] + fn clone_source_is_recorded_as_an_absolute_path_but_urls_are_untouched() { + // A relative source would resolve against the clone's own directory + // once the user cd's into it, so the recorded origin must be absolute. + let abs = absolute_local_source(".".to_string()); + assert!(std::path::Path::new(&abs).is_absolute(), "got {abs}"); + for url in ["ssh://host/srv/repo", "sc+http://127.0.0.1:1/x"] { + assert_eq!(absolute_local_source(url.to_string()), url); + } + assert_eq!( + absolute_local_source("no/such/dir/anywhere".to_string()), + "no/such/dir/anywhere" + ); + } + #[test] fn clone_bare_ssh_url_without_git_flag_stays_sc_native() { // Env mutation (SC_SSH) — serialize with the shared git-env lock. @@ -4620,6 +4705,7 @@ mod tests { root.join("cloned-ssh"), false, Vec::new(), + false, ) .unwrap_err(); std::env::remove_var("SC_SSH"); diff --git a/crates/repo/src/fill.rs b/crates/repo/src/fill.rs new file mode 100644 index 00000000..4865bfa4 --- /dev/null +++ b/crates/repo/src/fill.rs @@ -0,0 +1,619 @@ +//! By-id gap filling for lazy partial clones (P37, ADR-0047). +//! +//! A partial clone (ADR-0037) never fetched anything outside its filter. A +//! clone marked lazy (`.sc/promisor-lazy`) may ask its promisor remote for +//! exactly the trees and blobs one read needs, by id, one tree depth per round +//! trip. This module is the ONLY place that dials the remote after clone: +//! `Store::get` stays network-free, so gc, verify, materialize and the desktop +//! read model can never block on the network. +//! +//! Filling only ADDS objects the filter did not cover. The promisor filter is +//! unchanged, so push, gc and verify (all filter-scoped walks) behave exactly +//! as they did before the fill; they simply find a few extra present objects. + +use std::collections::BTreeSet; +use std::sync::Arc; + +use scl_core::{EntryKind, Object, ObjectId}; + +use crate::error::{Error, Result}; +use crate::repo::Repo; +use crate::stdio_transport::open_transport; +use crate::transport::{ingest_pack_file_checked, TempPackGuard, Transport}; + +/// What a fill cost: network round trips and objects ingested. Lets tests and +/// the demo assert that a read fetched only what it needed. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub struct FillStats { + /// `GetObjects` round trips made. + pub requests: usize, + /// Objects written to the local store. + pub objects: usize, +} + +impl Repo { + /// Fetch `ids` from `transport` by id, verify them, and write them to the + /// local store. One round trip; `ids` must not exceed + /// [`crate::wire::MAX_GET_OBJECTS_IDS`]. + /// + /// Every returned object is hash-verified before any is written, and the + /// reply must partition the request exactly: nothing unrequested, nothing + /// both sent and refused, nothing silently omitted. An id the remote + /// reports as not sent is a terminal `NotFound` (after the objects it did + /// send are ingested: they are immutable, so keeping them is harmless and + /// a retry resumes from current presence). + pub(crate) fn fetch_by_id_from( + &self, + transport: &dyn Transport, + ids: &[ObjectId], + ) -> Result { + let requested: BTreeSet = ids.iter().copied().collect(); + let guard = TempPackGuard::new(&self.layout)?; + let not_sent = { + let mut f = std::fs::File::create(guard.path())?; + transport.get_objects(ids, &mut f)? + }; + let not_sent_set: BTreeSet = not_sent.iter().copied().collect(); + + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); + let ingested = ingest_pack_file_checked( + &self.layout, + &mut store, + guard.path(), + Some(requested.len()), + |found| { + if let Some(id) = found.difference(&requested).next() { + return Err(Error::Protocol(format!( + "remote sent unrequested object {id}" + ))); + } + if let Some(id) = not_sent_set.difference(&requested).next() { + return Err(Error::Protocol(format!( + "remote listed unrequested object {id} as not sent" + ))); + } + if let Some(id) = found.intersection(¬_sent_set).next() { + return Err(Error::Protocol(format!( + "remote both sent and refused object {id}" + ))); + } + if let Some(id) = requested + .iter() + .find(|id| !found.contains(id) && !not_sent_set.contains(id)) + { + return Err(Error::Protocol(format!( + "remote neither sent nor refused object {id}" + ))); + } + Ok(()) + }, + )?; + if let Some(id) = not_sent.first() { + return Err(Error::Core(scl_core::Error::NotFound(*id))); + } + Ok(ingested.len()) + } + + /// The bytes of the file at `path` as of `refname`'s tip (`sc cat + /// :`). On a lazy partial clone, trees and the blob missing + /// locally are fetched by id on the way down (one round trip per tree + /// depth plus one for the blob); otherwise a missing object is the usual + /// gap error pointing at `sc backfill`. + /// + /// Refuses private branches and protected paths: this phase prints + /// public plaintext only. + pub fn cat_at(&self, refname: &str, path: &str) -> Result> { + Ok(self.cat_at_with_stats(refname, path)?.0) + } + + /// [`Repo::cat_at`], also reporting what the fill cost. + pub fn cat_at_with_stats(&self, refname: &str, path: &str) -> Result<(Arc<[u8]>, FillStats)> { + self.cat_at_impl(refname, path, crate::promisor::offline()) + } + + /// `offline` is a parameter so tests can exercise the `SC_OFFLINE` path + /// without mutating the process environment. + fn cat_at_impl( + &self, + refname: &str, + path: &str, + offline: bool, + ) -> Result<(Arc<[u8]>, FillStats)> { + let comps: Vec<&str> = path.split('/').filter(|c| !c.is_empty()).collect(); + if comps.is_empty() { + return Err(Error::InvalidArgument("sc cat needs a file path".into())); + } + let tip = crate::refs::resolve_tip(&self.layout, refname)? + .ok_or_else(|| Error::InvalidArgument(format!("unknown ref '{refname}'")))?; + let root = { + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); + match store.get(&tip)? { + Object::Snapshot(s) => s.root, + Object::Manifest(_) => return Err(Error::PrivateUnsupported("sc cat".into())), + _ => { + return Err(Error::InvalidArgument(format!( + "'{refname}' does not name a snapshot" + ))) + } + } + }; + + let mut filler = Filler::new(self, offline)?; + let mut cur = root; + for (i, comp) in comps.iter().enumerate() { + filler.ensure_present(&cur, path)?; + let entry = { + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); + let tree = store.get_tree(&cur)?; + tree.get(comp).cloned() + } + .ok_or_else(|| Error::Core(scl_core::Error::PathNotFound(path.to_string())))?; + let last = i + 1 == comps.len(); + match (entry.kind, last) { + (EntryKind::Tree, false) => cur = entry.id, + (EntryKind::Tree, true) => { + return Err(Error::InvalidArgument(format!("{path} is a directory"))) + } + (EntryKind::Blob, false) => { + return Err(Error::InvalidArgument(format!("{comp} is not a directory"))) + } + (EntryKind::Blob, true) => { + if entry.perms & scl_core::PROTECTED != 0 { + return Err(Error::InvalidArgument(format!( + "{path} is a protected path; sc cat prints public content only" + ))); + } + filler.ensure_present(&entry.id, path)?; + let store_arc = self.vfs.store(); + let mut store = store_arc.lock().unwrap(); + return match store.get(&entry.id)? { + Object::Blob(b) => Ok((b, filler.stats)), + _ => Err(Error::InvalidArgument(format!("{path} is not a file"))), + }; + } + } + } + unreachable!("the loop returns on the last path component") + } +} + +/// Makes objects present, dialing the promisor remote only when the clone is +/// lazy and not offline. The connection opens on first need. +struct Filler<'a> { + repo: &'a Repo, + /// The promisor origin URL when filling is allowed, else `None`. + origin: Option, + transport: Option>, + stats: FillStats, +} + +impl<'a> Filler<'a> { + fn new(repo: &'a Repo, offline: bool) -> Result> { + let origin = match repo.promisor()? { + Some(p) if crate::promisor::is_lazy(&repo.layout) && !offline => Some(p.origin), + _ => None, + }; + Ok(Filler { + repo, + origin, + transport: None, + stats: FillStats::default(), + }) + } + + /// Ensure `id` is in the local store. `path` names what the caller is + /// reading, for the gap error. + fn ensure_present(&mut self, id: &ObjectId, path: &str) -> Result<()> { + let present = self.repo.vfs.store().lock().unwrap().contains(id); + if present { + return Ok(()); + } + let Some(origin) = self.origin.clone() else { + return Err(if self.repo.promisor()?.is_some() { + crate::promisor::partial_gap_hint(path) + } else { + Error::Core(scl_core::Error::NotFound(*id)) + }); + }; + if self.transport.is_none() { + self.transport = Some(open_transport(&origin)?); + } + let transport = self.transport.as_deref().expect("opened above"); + let n = self.repo.fetch_by_id_from(transport, &[*id])?; + self.stats.requests += 1; + self.stats.objects += n; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Read, Write}; + + /// Source repo with `src/a.txt`, `docs/readme.md` and `docs/api/guide.md`. + /// Returns the source root, its tip, and the ids of the `docs` tree, + /// `docs/api` tree, `guide.md` blob and `readme.md` blob. + struct Fixture { + src_root: std::path::PathBuf, + dst_root: std::path::PathBuf, + tip: ObjectId, + docs_tree: ObjectId, + api_tree: ObjectId, + guide_blob: ObjectId, + readme_blob: ObjectId, + } + + impl Fixture { + fn new(tag: &str) -> Fixture { + let pid = std::process::id(); + let src_root = std::env::temp_dir().join(format!("scl-fill-{tag}-src-{pid}")); + let dst_root = std::env::temp_dir().join(format!("scl-fill-{tag}-dst-{pid}")); + let _ = std::fs::remove_dir_all(&src_root); + let _ = std::fs::remove_dir_all(&dst_root); + std::fs::create_dir_all(src_root.join("src")).unwrap(); + std::fs::create_dir_all(src_root.join("docs/api")).unwrap(); + let src = Repo::init(&src_root).unwrap(); + std::fs::write(src_root.join("src/a.txt"), b"src-one").unwrap(); + std::fs::write(src_root.join("docs/readme.md"), b"readme body").unwrap(); + std::fs::write(src_root.join("docs/api/guide.md"), b"guide body").unwrap(); + let tip = src.commit("t", "c1").unwrap(); + let store_arc = src.vfs().store(); + let mut store = store_arc.lock().unwrap(); + let root = store.get_snapshot(&tip).unwrap().root; + let docs_tree = store.get_tree(&root).unwrap().get("docs").unwrap().id; + let docs = store.get_tree(&docs_tree).unwrap(); + let api_tree = docs.get("api").unwrap().id; + let readme_blob = docs.get("readme.md").unwrap().id; + let guide_blob = store + .get_tree(&api_tree) + .unwrap() + .get("guide.md") + .unwrap() + .id; + drop(store); + drop(src); + Fixture { + src_root, + dst_root, + tip, + docs_tree, + api_tree, + guide_blob, + readme_blob, + } + } + + /// A partial clone filtered to `src/`. + fn clone(&self, lazy: bool) -> Repo { + let dst = Repo::clone_url_filtered( + self.src_root.to_str().unwrap(), + &self.dst_root, + Some(&["src/".to_string()]), + ) + .unwrap(); + crate::promisor::set_lazy(dst.layout(), lazy).unwrap(); + dst + } + + fn has(&self, repo: &Repo, id: &ObjectId) -> bool { + repo.vfs().store().lock().unwrap().contains(id) + } + + fn cleanup(self) { + for root in [&self.src_root, &self.dst_root] { + if root.exists() { + std::fs::remove_dir_all(root).unwrap(); + } + assert!(!root.exists(), "fixture {} must be removed", root.display()); + } + } + } + + #[test] + fn cat_fills_only_the_path_it_reads() { + let fx = Fixture::new("path"); + let dst = fx.clone(true); + // The read path itself never touches the network: before any fill the + // gapped objects are simply absent. + assert!(dst + .vfs() + .store() + .lock() + .unwrap() + .get_tree(&fx.docs_tree) + .is_err()); + + let (bytes, stats) = dst.cat_at_with_stats("main", "docs/api/guide.md").unwrap(); + assert_eq!(&bytes[..], b"guide body"); + // docs tree, docs/api tree, guide blob: one round trip each, nothing else. + assert_eq!( + stats, + FillStats { + requests: 3, + objects: 3 + } + ); + assert!(fx.has(&dst, &fx.docs_tree)); + assert!(fx.has(&dst, &fx.api_tree)); + assert!(fx.has(&dst, &fx.guide_blob)); + assert!( + !fx.has(&dst, &fx.readme_blob), + "a sibling file must not be fetched" + ); + + // A second read of the same path is free. + let (_, again) = dst.cat_at_with_stats("main", "docs/api/guide.md").unwrap(); + assert_eq!(again, FillStats::default()); + fx.cleanup(); + } + + #[test] + fn cat_inside_the_filter_makes_no_request() { + let fx = Fixture::new("infilter"); + let dst = fx.clone(true); + let (bytes, stats) = dst.cat_at_with_stats("main", "src/a.txt").unwrap(); + assert_eq!(&bytes[..], b"src-one"); + assert_eq!(stats, FillStats::default()); + fx.cleanup(); + } + + #[test] + fn non_lazy_clone_keeps_the_gap_error() { + let fx = Fixture::new("nonlazy"); + let dst = fx.clone(false); + let err = dst.cat_at("main", "docs/readme.md").unwrap_err(); + assert!(matches!(err, Error::GapOutsideFilter(_)), "got {err:?}"); + assert!(!fx.has(&dst, &fx.docs_tree), "nothing may be fetched"); + fx.cleanup(); + } + + #[test] + fn offline_forces_the_gap_error_even_when_lazy() { + let fx = Fixture::new("offline"); + let dst = fx.clone(true); + let err = dst.cat_at_impl("main", "docs/readme.md", true).unwrap_err(); + assert!(matches!(err, Error::GapOutsideFilter(_)), "got {err:?}"); + assert!(!fx.has(&dst, &fx.docs_tree)); + fx.cleanup(); + } + + #[test] + fn a_retry_resumes_from_current_presence() { + let fx = Fixture::new("resume"); + let dst = fx.clone(true); + // Simulate a fill that died after the first level. + let t = open_transport(fx.src_root.to_str().unwrap()).unwrap(); + dst.fetch_by_id_from(t.as_ref(), &[fx.docs_tree]).unwrap(); + assert!(fx.has(&dst, &fx.docs_tree)); + + let (bytes, stats) = dst.cat_at_with_stats("main", "docs/api/guide.md").unwrap(); + assert_eq!(&bytes[..], b"guide body"); + assert_eq!( + stats, + FillStats { + requests: 2, + objects: 2 + } + ); + fx.cleanup(); + } + + #[test] + fn gc_keeps_filled_objects_and_push_still_composes() { + let fx = Fixture::new("gc"); + let dst = fx.clone(true); + dst.cat_at("main", "docs/api/guide.md").unwrap(); + dst.gc(std::time::Duration::ZERO).unwrap(); + assert!(fx.has(&dst, &fx.guide_blob), "gc must keep filled objects"); + let (_, again) = dst.cat_at_with_stats("main", "docs/api/guide.md").unwrap(); + assert_eq!(again, FillStats::default()); + + // A commit inside the filter after a fill pushes cleanly: the filter + // is unchanged, so push sends in-filter objects only. + std::fs::write(fx.dst_root.join("src/a.txt"), b"src-two").unwrap(); + dst.commit("t", "edit src").unwrap(); + dst.push("origin").unwrap(); + fx.cleanup(); + } + + #[test] + fn cat_refuses_directories_missing_paths_and_unknown_refs() { + let fx = Fixture::new("errors"); + let dst = fx.clone(true); + assert!(matches!( + dst.cat_at("main", "docs/api").unwrap_err(), + Error::InvalidArgument(_) + )); + assert!(matches!( + dst.cat_at("main", "docs/nope.md").unwrap_err(), + Error::Core(scl_core::Error::PathNotFound(_)) + )); + assert!(matches!( + dst.cat_at("no-such-branch", "src/a.txt").unwrap_err(), + Error::InvalidArgument(_) + )); + assert!(matches!( + dst.cat_at("main", "/").unwrap_err(), + Error::InvalidArgument(_) + )); + let _ = fx.tip; + fx.cleanup(); + } + + #[test] + fn verify_gap_count_is_unchanged_by_a_fill() { + let fx = Fixture::new("verify"); + let dst = fx.clone(true); + let before = dst.partial_gap_count(&[fx.tip]).unwrap(); + assert!( + before.is_some_and(|n| n > 0), + "docs/ is a gap before the fill" + ); + dst.cat_at("main", "docs/api/guide.md").unwrap(); + assert_eq!( + dst.partial_gap_count(&[fx.tip]).unwrap(), + before, + "a fill adds objects but never changes the filter, so the filter-scoped count is stable" + ); + fx.cleanup(); + } + + #[test] + fn cat_refuses_protected_paths_and_private_branches() { + let root = std::env::temp_dir().join(format!("scl-fill-refuse-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(root.join("secret")).unwrap(); + let repo = Repo::init(&root).unwrap(); + let (alice_sk, alice_pk) = scl_crypto::generate_keypair(); + repo.protect("secret/", &[alice_pk], None).unwrap(); + std::fs::write(root.join("secret/a.txt"), b"classified").unwrap(); + std::fs::write(root.join("open.txt"), b"public").unwrap(); + repo.commit("t", "base").unwrap(); + + let err = repo.cat_at("main", "secret/a.txt").unwrap_err(); + assert!( + matches!(&err, Error::InvalidArgument(m) if m.contains("protected")), + "got {err:?}" + ); + assert_eq!(&repo.cat_at("main", "open.txt").unwrap()[..], b"public"); + + repo.branch_private("fix", &alice_sk, &[], &[]).unwrap(); + let err = repo.cat_at("fix", "open.txt").unwrap_err(); + assert!(matches!(err, Error::PrivateUnsupported(_)), "got {err:?}"); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// A transport that repeats one requested object many times, to prove the + /// receiver stops at the requested record count instead of decompressing + /// and hashing every duplicate. + #[test] + fn fetch_rejects_duplicate_records_before_processing_them_all() { + let fx = Fixture::new("dups"); + let dst = fx.clone(true); + let want = Object::blob(b"wanted".to_vec()); + let flood = Scripted { + send: vec![want.clone(); 50], + not_sent: vec![], + }; + let err = dst.fetch_by_id_from(&flood, &[want.id()]).unwrap_err(); + assert!( + matches!(&err, Error::Core(scl_core::Error::PackCorrupt(m)) if m.contains("requested")), + "got {err:?}" + ); + assert!(!fx.has(&dst, &want.id()), "a rejected pack lands nothing"); + fx.cleanup(); + } + + /// A transport whose `get_objects` replies with a scripted pack and + /// not-sent list, to exercise the receiver's reply validation. + struct Scripted { + send: Vec, + not_sent: Vec, + } + + impl Transport for Scripted { + fn list_refs(&self) -> Result> { + unimplemented!() + } + fn head_branch(&self) -> Result { + unimplemented!() + } + fn has_object(&self, _: &ObjectId) -> Result { + unimplemented!() + } + fn get_object(&self, _: &ObjectId) -> Result> { + unimplemented!() + } + fn put_object(&self, _: &ObjectId, _: &[u8]) -> Result<()> { + unimplemented!() + } + fn update_ref(&self, _: &str, _: &ObjectId, _: Option<&ObjectId>) -> Result<()> { + unimplemented!() + } + fn get_pack( + &self, + _: &[ObjectId], + _: &[ObjectId], + _: Option<&[String]>, + _: &mut dyn Write, + ) -> Result<()> { + unimplemented!() + } + fn put_pack(&self, _: &mut dyn Read) -> Result> { + unimplemented!() + } + fn get_objects(&self, _: &[ObjectId], out: &mut dyn Write) -> Result> { + let mut w = scl_core::pack::PackWriter::new(&mut *out, self.send.len() as u32)?; + for o in &self.send { + w.write_object(&o.id(), &o.encode())?; + } + w.finish()?; + Ok(self.not_sent.clone()) + } + } + + #[test] + fn fetch_rejects_a_reply_that_does_not_partition_the_request() { + let fx = Fixture::new("liar"); + let dst = fx.clone(true); + let want = Object::blob(b"wanted".to_vec()); + let extra = Object::blob(b"smuggled".to_vec()); + let other = Object::blob(b"also requested".to_vec()); + let store_has = |id: &ObjectId| fx.has(&dst, id); + + // Unrequested object smuggled in, within the requested record count + // (a longer pack is caught earlier by the record bound). + let liar = Scripted { + send: vec![want.clone(), extra.clone()], + not_sent: vec![other.id()], + }; + let err = dst + .fetch_by_id_from(&liar, &[want.id(), other.id()]) + .unwrap_err(); + assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + assert!( + !store_has(&want.id()) && !store_has(&extra.id()), + "nothing lands" + ); + + // Requested id neither sent nor refused. + let liar = Scripted { + send: vec![want.clone()], + not_sent: vec![], + }; + let err = dst + .fetch_by_id_from(&liar, &[want.id(), other.id()]) + .unwrap_err(); + assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + assert!(!store_has(&want.id())); + + // Sent and refused at once. + let liar = Scripted { + send: vec![want.clone()], + not_sent: vec![want.id()], + }; + assert!(matches!( + dst.fetch_by_id_from(&liar, &[want.id()]).unwrap_err(), + Error::Protocol(_) + )); + + // A refused id is terminal, but what was sent is kept. + let honest = Scripted { + send: vec![want.clone()], + not_sent: vec![other.id()], + }; + let err = dst + .fetch_by_id_from(&honest, &[want.id(), other.id()]) + .unwrap_err(); + assert!(matches!(err, Error::Core(scl_core::Error::NotFound(id)) if id == other.id())); + assert!(store_has(&want.id()), "verified objects are retained"); + fx.cleanup(); + } +} diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index 7ae71fa9..b0854912 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -524,6 +524,9 @@ impl Transport for HttpTransport { fn put_pack(&self, src: &mut dyn Read) -> Result> { self.client.put_pack(src) } + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + self.client.get_objects(ids, out) + } } /// Bound on how long an accepted connection may take to send its HTTP diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index fecc32d0..2ba77532 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -103,6 +103,13 @@ impl Layout { pub fn promisor_path(&self) -> PathBuf { self.dot_sc.join("promisor") } + /// `.sc/promisor-lazy` — empty marker (P37): this partial clone may fill + /// gaps from its promisor remote by object id (ADR-0047). A separate file, + /// not a field in `.sc/promisor`, because that file's lines after + /// `origin` are free-form path prefixes. + pub fn promisor_lazy_path(&self) -> PathBuf { + self.dot_sc.join("promisor-lazy") + } /// `.sc/serve-tokens.toml` — server access-control tokens (P29): each entry is /// `{label, hash = BLAKE3(raw token), scope}`. Presence of ≥1 entry turns on /// bearer auth for `sc serve --http`. Distinct from `recipients.toml` diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 68b5175a..51e55fd5 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -6,6 +6,7 @@ pub(crate) mod cache; pub(crate) mod conflicts; pub mod diff3; pub mod error; +pub mod fill; pub mod gc; pub mod git_marks; pub mod http_transport; diff --git a/crates/repo/src/promisor.rs b/crates/repo/src/promisor.rs index a8430b11..0b60836e 100644 --- a/crates/repo/src/promisor.rs +++ b/crates/repo/src/promisor.rs @@ -154,6 +154,7 @@ pub fn store(layout: &Layout, p: &Promisor) -> Result<()> { /// after independently confirming the object closure is actually /// complete — this function does not itself verify anything. pub fn remove(layout: &Layout) -> Result<()> { + set_lazy(layout, false)?; match std::fs::remove_file(layout.promisor_path()) { Ok(()) => Ok(()), Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), @@ -161,6 +162,32 @@ pub fn remove(layout: &Layout) -> Result<()> { } } +/// True when this partial clone has opted in to filling gaps by object id +/// (ADR-0047). Absent marker, or no promisor at all, reads as `false`. +pub fn is_lazy(layout: &Layout) -> bool { + layout.promisor_lazy_path().is_file() +} + +/// Set or clear the lazy marker. Clearing an absent marker is a no-op. +pub fn set_lazy(layout: &Layout, lazy: bool) -> Result<()> { + if lazy { + scl_core::fsutil::atomic_write_durable(&layout.promisor_lazy_path(), b"")?; + return Ok(()); + } + match std::fs::remove_file(layout.promisor_lazy_path()) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), + } +} + +/// True when the environment forces offline behavior (`SC_OFFLINE` set to +/// anything but empty or `0`): a lazy clone then reports the gap error +/// instead of dialing the promisor remote. +pub fn offline() -> bool { + matches!(std::env::var("SC_OFFLINE"), Ok(v) if !v.is_empty() && v != "0") +} + impl Repo { /// The repo's current promisor marker, if any (thin wrapper over /// `load`). `None` means this is a full clone. @@ -256,4 +283,37 @@ mod tests { p.widen(&["src/".into(), "docs/".into(), "tests/".into()]); assert_eq!(p.prefixes().len(), 3); } + + #[test] + fn lazy_marker_is_separate_from_the_filter_file() { + let layout = tmp_layout("lazy"); + // A prefix that looks like a config line must stay a plain prefix: + // laziness lives in its own marker, never in `.sc/promisor`. + let p = Promisor::new("origin-url", vec!["lazy = true".into()]); + store(&layout, &p).unwrap(); + assert!(!is_lazy(&layout)); + assert_eq!(load(&layout).unwrap(), Some(p.clone())); + + set_lazy(&layout, true).unwrap(); + assert!(is_lazy(&layout)); + assert_eq!(load(&layout).unwrap(), Some(p)); + + set_lazy(&layout, false).unwrap(); + assert!(!is_lazy(&layout)); + // Clearing an absent marker is a no-op. + set_lazy(&layout, false).unwrap(); + + std::fs::remove_dir_all(&layout.root).unwrap(); + } + + #[test] + fn remove_clears_the_lazy_marker_too() { + let layout = tmp_layout("lazy-rm"); + store(&layout, &Promisor::new("origin-url", vec!["src/".into()])).unwrap(); + set_lazy(&layout, true).unwrap(); + remove(&layout).unwrap(); + assert!(!is_lazy(&layout)); + assert!(!layout.promisor_lazy_path().exists()); + std::fs::remove_dir_all(&layout.root).unwrap(); + } } diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index c5614d05..ee3f4fa7 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -132,14 +132,49 @@ impl Transport for WireClient { let frame = wire::read_frame(&mut rw.0)?; wire::decode_ids_body(&wire::parse_response(frame)?) } + + /// One round trip (P37, ADR-0047): the OK body lists the ids the server + /// will not send, and a chunk stream with the rest follows. The reply + /// stream is capped (`GET_OBJECTS_RESPONSE_CAP`) so a hostile server + /// cannot make a by-id fetch spool without bound. + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + if ids.len() > wire::MAX_GET_OBJECTS_IDS { + return Err(Error::InvalidArgument(format!( + "by-id fetch takes at most {} ids per call", + wire::MAX_GET_OBJECTS_IDS + ))); + } + let mut rw = self.rw.borrow_mut(); + wire::write_frame( + &mut rw.1, + &Request::GetObjects { ids: ids.to_vec() }.encode(), + )?; + let frame = wire::read_frame(&mut rw.0)?; + let not_sent = wire::decode_ids_body_capped(&wire::parse_response(frame)?, ids.len())?; + let mut seen = std::collections::BTreeSet::new(); + if let Some(dup) = not_sent.iter().find(|id| !seen.insert(**id)) { + return Err(Error::Protocol(format!( + "reply lists object {dup} as not sent more than once" + ))); + } + wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP)?; + Ok(not_sent) + } } +/// Most bytes a client accepts in one `GetObjects` reply stream (1 GiB). +const GET_OBJECTS_RESPONSE_CAP: u64 = 1 << 30; + /// A [`Transport`] whose far end is a child process speaking the wire protocol /// on its stdio — `ssh sc serve --stdio ` for real remotes. #[derive(Debug)] pub struct StdioTransport { client: WireClient, ChildStdin>, child: Child, + /// Set when a by-id reply was abandoned mid-stream. The child may be + /// blocked writing frames nobody will read, so `Drop` must kill it + /// instead of sending `Bye` and waiting. + desynced: std::cell::Cell, } impl StdioTransport { @@ -157,7 +192,11 @@ impl StdioTransport { let stdin = child.stdin.take().expect("stdin piped"); let stdout = std::io::BufReader::new(child.stdout.take().expect("stdout piped")); match WireClient::handshake(stdout, stdin) { - Ok(client) => Ok(StdioTransport { client, child }), + Ok(client) => Ok(StdioTransport { + client, + child, + desynced: std::cell::Cell::new(false), + }), Err(Error::ConnectionLost(msg)) => { let stderr = reap_with_stderr(&mut child); Err(Error::ConnectionLost(if stderr.is_empty() { @@ -176,7 +215,13 @@ impl StdioTransport { impl Drop for StdioTransport { fn drop(&mut self) { - let _ = self.client.bye(); // best-effort: the server exits its loop + if self.desynced.get() { + // The stream is out of sync and the child may be stuck writing + // into a pipe we no longer read; `Bye` would never be seen. + let _ = self.child.kill(); + } else { + let _ = self.client.bye(); // best-effort: the server exits its loop + } let _ = self.child.wait(); } } @@ -228,6 +273,14 @@ impl Transport for StdioTransport { fn put_pack(&self, src: &mut dyn Read) -> Result> { self.client.put_pack(src) } + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + let r = self.client.get_objects(ids, out); + if r.is_err() { + // Any failure may have left unread reply frames on the pipe. + self.desynced.set(true); + } + r + } } /// A parsed `ssh://[user@]host[:port]/abs/path` remote URL. @@ -413,6 +466,44 @@ mod tests { (client, handle) } + #[test] + fn wire_client_get_objects_round_trip_found_missing_and_refused() { + let root = tmp_repo("getobjs"); + let (client, server) = connect(root.clone()); + + let blob = Object::blob(b"fetched by id".to_vec()); + let sealed = Object::Sealed(scl_core::SealedObj { + payload: std::sync::Arc::from(b"ciphertext".to_vec().into_boxed_slice()), + }); + let absent = Object::blob(b"not on the server".to_vec()); + for o in [&blob, &sealed] { + client.put_object(&o.id(), &o.encode()).unwrap(); + } + + let mut out = Vec::new(); + let not_sent = client + .get_objects(&[blob.id(), sealed.id(), absent.id()], &mut out) + .unwrap(); + assert_eq!(not_sent, vec![sealed.id(), absent.id()]); + let got = scl_core::pack::parse_pack(&out).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].0, blob.id()); + + // The client refuses an oversize request without a round trip. + let too_many = vec![blob.id(); wire::MAX_GET_OBJECTS_IDS + 1]; + assert!(matches!( + client.get_objects(&too_many, &mut Vec::new()), + Err(Error::InvalidArgument(_)) + )); + // ...and the connection is still in sync afterwards. + assert!(client.has_object(&blob.id()).unwrap()); + + client.bye().unwrap(); + drop(client); + server.join().unwrap().unwrap(); + std::fs::remove_dir_all(&root).unwrap(); + } + #[test] fn wire_client_satisfies_the_transport_contract() { let root = tmp_repo("contract"); @@ -515,6 +606,112 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + #[test] + fn handshake_rejects_v4_peer() { + // P37 bumped PROTOCOL_VERSION 4 -> 5 for the additive `GetObjects` + // verb: a v4 server cannot decode it, so refuse at the handshake. + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let f = wire::read_frame(&mut server_read).unwrap(); + assert!(matches!( + wire::Request::decode(&f).unwrap(), + wire::Request::Hello { version: 5 } + )); + wire::write_ok(&mut server_write, &wire::u32_body(4)).unwrap(); + }); + let err = WireClient::handshake(client_read, client_write).unwrap_err(); + assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + server.join().unwrap(); + } + + /// A hostile server that answers a one-id `GetObjects` with `reply_ids` + /// as the not-sent list, then an empty pack stream. + fn get_objects_against_hostile_reply(reply_ids: Vec) -> Error { + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let f = wire::read_frame(&mut server_read).unwrap(); + assert!(matches!( + wire::Request::decode(&f).unwrap(), + wire::Request::Hello { .. } + )); + wire::write_ok(&mut server_write, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + let f = wire::read_frame(&mut server_read).unwrap(); + assert!(matches!( + wire::Request::decode(&f).unwrap(), + wire::Request::GetObjects { .. } + )); + wire::write_ok(&mut server_write, &wire::ids_body(&reply_ids)).unwrap(); + }); + let client = WireClient::handshake(client_read, client_write).unwrap(); + let id = Object::blob(b"wanted".to_vec()).id(); + let err = client.get_objects(&[id], &mut Vec::new()).unwrap_err(); + server.join().unwrap(); + err + } + + #[test] + fn get_objects_rejects_a_not_sent_list_longer_than_the_request() { + let id = Object::blob(b"wanted".to_vec()).id(); + let other = Object::blob(b"other".to_vec()).id(); + let err = get_objects_against_hostile_reply(vec![id, other]); + assert!(matches!(err, Error::Protocol(_)), "got {err:?}"); + } + + #[test] + fn get_objects_rejects_a_duplicated_not_sent_id() { + let id = Object::blob(b"wanted".to_vec()).id(); + // Within the one-id cap this is still a duplicate once the list is + // asked to hold two copies of the same id. + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let server = std::thread::spawn(move || { + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + let _ = wire::read_frame(&mut server_read).unwrap(); + wire::write_ok(&mut server_write, &wire::ids_body(&[id, id])).unwrap(); + }); + let client = WireClient::handshake(client_read, client_write).unwrap(); + let other = Object::blob(b"other".to_vec()).id(); + let err = client + .get_objects(&[id, other], &mut Vec::new()) + .unwrap_err(); + server.join().unwrap(); + assert!( + matches!(&err, Error::Protocol(m) if m.contains("more than once")), + "got {err:?}" + ); + } + + #[test] + fn a_desynced_stdio_transport_kills_a_child_that_never_exits_on_drop() { + // `sleep` ignores the wire protocol and never exits on its own, which + // is what a child blocked writing an abandoned reply looks like. + let mut child = Command::new("sleep") + .arg("600") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .spawn() + .unwrap(); + let stdin = child.stdin.take().unwrap(); + let stdout = std::io::BufReader::new(child.stdout.take().unwrap()); + let t = StdioTransport { + client: WireClient { + rw: std::cell::RefCell::new((stdout, stdin)), + }, + child, + desynced: std::cell::Cell::new(true), + }; + let (tx, rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + drop(t); + tx.send(()).unwrap(); + }); + rx.recv_timeout(std::time::Duration::from_secs(10)) + .expect("drop must kill the child instead of waiting for it"); + } + #[test] fn handshake_rejects_v2_peer() { // Hand-rolled "old server" that answers HELLO with the retired v2 diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 7a3dc2d1..2aa13276 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -64,6 +64,53 @@ pub trait Transport { /// each object into the store. Returns the contained ids. Refs are the /// caller's job. fn put_pack(&self, src: &mut dyn Read) -> Result>; + + /// Batched by-id fetch (P37, ADR-0047): write a pack holding every object + /// of `ids` this remote can serve into `out`, and return the ids it did + /// NOT send, in request order. "Not sent" covers both an id the remote + /// lacks and an id it refuses to serve by id (sealed-branch objects), so + /// the two cannot be told apart. At most + /// [`crate::wire::MAX_GET_OBJECTS_IDS`] ids per call. + /// + /// The default refuses: only transports that can answer without a + /// reachability walk (local path, wire clients) implement it. + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + let _ = (ids, out); + Err(Error::InvalidArgument( + "this remote does not support by-id object fetch".into(), + )) + } +} + +/// Canonical encoded size of `obj`, computed without copying a large payload: +/// a blob or sealed object encodes as one tag byte plus its payload; the small +/// structural kinds are encoded to measure them. +fn encoded_len(obj: &Object) -> u64 { + match obj { + Object::Blob(b) => b.len() as u64 + 1, + Object::Sealed(s) => s.payload.len() as u64 + 1, + other => other.encode().len() as u64, + } +} + +/// Refuse an object a receiver would reject anyway: `pack::parse_pack_reader` +/// fails any record whose canonical encoding exceeds `MAX_OBJECT_SIZE`. +fn check_servable_size(id: &ObjectId, len: u64) -> Result<()> { + if len > scl_core::MAX_OBJECT_SIZE as u64 { + return Err(Error::PackTooLarge(format!( + "object {id} exceeds MAX_OBJECT_SIZE and cannot be transferred" + ))); + } + Ok(()) +} + +/// Object kinds a by-id read never serves (ADR-0047): sealed private-branch +/// members and branch manifests. Filtered transfers already exclude them +/// (`filter_manifest_wants`); this keeps the by-id verb consistent with that +/// rule. It is a consistency rule, not a confidentiality boundary: sealed +/// objects are ciphertext and an unfiltered pack still serves them. +fn refused_by_id(obj: &Object) -> bool { + matches!(obj, Object::Sealed(_) | Object::Manifest(_)) } /// Transport over a remote `.sc/` directory on the local filesystem. @@ -91,6 +138,50 @@ impl LocalTransport { }) } + /// Resolve a by-id request against this store into a temp pack of the + /// servable objects, returning `(not_sent, pack)` (P37, ADR-0047). The + /// shared core of [`Transport::get_objects`] and `wire::serve`'s + /// `GetObjects` arm. Duplicate ids are served once. `max_bytes` (0 = + /// unlimited) bounds the summed canonical encoded size, and any single + /// object over `MAX_OBJECT_SIZE` is refused (every receiver would reject + /// it). Both fail with `PackTooLarge` before any pack byte is written. + /// Blobs and sealed objects are measured without re-encoding them. + pub(crate) fn build_objects_tempfile( + &self, + ids: &[ObjectId], + max_bytes: u64, + ) -> Result<(Vec, TempPackGuard)> { + use std::collections::BTreeSet; + let mut store = self.store.borrow_mut(); + let mut seen = BTreeSet::new(); + let mut found = Vec::new(); + let mut not_sent = Vec::new(); + let mut total: u64 = 0; + for id in ids { + if !seen.insert(*id) { + continue; + } + match store.get(id) { + Ok(obj) if refused_by_id(&obj) => not_sent.push(*id), + Ok(obj) => { + let size = encoded_len(&obj); + check_servable_size(id, size)?; + total = total.saturating_add(size); + if max_bytes > 0 && total > max_bytes { + return Err(Error::PackTooLarge(format!( + "by-id response exceeds the {max_bytes}-byte limit" + ))); + } + found.push(*id); + } + Err(scl_core::Error::NotFound(_)) => not_sent.push(*id), + Err(e) => return Err(e.into()), + } + } + let guard = write_ids_to_temp_pack(&self.layout, &mut store, &found)?; + Ok((not_sent, guard)) + } + /// This transport's layout — `wire::serve` (a sibling module) needs it to /// spill an incoming chunk stream to a temp pack file before handing the /// path to [`ingest_pack_file`] directly (P25), bypassing a second @@ -327,11 +418,41 @@ pub(crate) fn ingest_pack_file( layout: &Layout, store: &mut Store, path: &std::path::Path, +) -> Result> { + ingest_pack_file_checked(layout, store, path, None, |_| Ok(())) +} + +/// [`ingest_pack_file`] with a gate between its two passes (P37): `check` +/// sees the set of ids pass 1 verified and may veto the whole pack, so a +/// receiver that knows what it asked for (`Repo`'s by-id fill) can reject +/// unrequested objects before any of them is written. `max_records` aborts +/// pass 1 as soon as the pack holds more records than the caller asked for, +/// so a peer cannot make the receiver decompress and hash the same large +/// object over and over (a set of ids hides duplicate records). +pub(crate) fn ingest_pack_file_checked( + layout: &Layout, + store: &mut Store, + path: &std::path::Path, + max_records: Option, + check: impl FnOnce(&std::collections::BTreeSet) -> Result<()>, ) -> Result> { // Pass 1: verify every record's hash; write nothing. { let f = std::fs::File::open(path)?; - scl_core::pack::parse_pack_reader(f, |_id, _obj| Ok(()))?; + let mut seen = std::collections::BTreeSet::new(); + let mut records = 0usize; + scl_core::pack::parse_pack_reader(f, |id, _obj| { + records += 1; + if max_records.is_some_and(|max| records > max) { + return Err(scl_core::Error::PackCorrupt(format!( + "pack holds more than the {} record(s) requested", + max_records.unwrap_or(0) + ))); + } + seen.insert(id); + Ok(()) + })?; + check(&seen)?; } // Pass 2: every record verified in pass 1 — now write each into the store. let f = std::fs::File::open(path)?; @@ -392,6 +513,19 @@ impl Transport for LocalTransport { Ok(self.store.borrow_mut().get(id)?.encode()) } + fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { + if ids.len() > crate::wire::MAX_GET_OBJECTS_IDS { + return Err(Error::InvalidArgument(format!( + "by-id fetch takes at most {} ids per call", + crate::wire::MAX_GET_OBJECTS_IDS + ))); + } + let (not_sent, guard) = self.build_objects_tempfile(ids, 0)?; + let mut f = std::fs::File::open(guard.path())?; + std::io::copy(&mut f, out)?; + Ok(not_sent) + } + fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { if ObjectId::of(bytes) != *id { return Err(Error::CorruptObject(*id)); @@ -489,6 +623,130 @@ mod tests { layout } + #[test] + fn local_get_objects_serves_found_and_reports_the_rest_not_sent() { + let layout = tmp_remote("getobjs"); + let t = LocalTransport::open(&layout.root).unwrap(); + let blob = Object::blob(b"servable".to_vec()); + let sealed = Object::Sealed(scl_core::SealedObj { + payload: std::sync::Arc::from(b"ciphertext".to_vec().into_boxed_slice()), + }); + let manifest = Object::Manifest(scl_core::BranchManifest { + base: blob.id(), + prev: None, + anchors: vec![], + closure: vec![sealed.id()], + index_ct: vec![1, 2, 3], + kek_wraps: vec![], + }); + let absent = Object::blob(b"never stored".to_vec()); + for o in [&blob, &sealed, &manifest] { + t.put_object(&o.id(), &o.encode()).unwrap(); + } + + let mut out = Vec::new(); + let not_sent = t + .get_objects( + &[ + blob.id(), + sealed.id(), + manifest.id(), + absent.id(), + blob.id(), + ], + &mut out, + ) + .unwrap(); + // Sealed and Manifest are refused and absent is missing; the three + // are not distinguishable, and a duplicate request id is served once. + assert_eq!(not_sent, vec![sealed.id(), manifest.id(), absent.id()]); + let got = scl_core::pack::parse_pack(&out).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].0, blob.id()); + + // All-missing still yields a valid (empty) pack. + let mut out = Vec::new(); + let not_sent = t.get_objects(&[absent.id()], &mut out).unwrap(); + assert_eq!(not_sent, vec![absent.id()]); + assert!(scl_core::pack::parse_pack(&out).unwrap().is_empty()); + + std::fs::remove_dir_all(&layout.root).unwrap(); + } + + #[test] + fn encoded_len_matches_the_canonical_encoding_and_the_receiver_limit() { + let blob = Object::blob(vec![1u8; 300]); + let sealed = Object::Sealed(scl_core::SealedObj { + payload: std::sync::Arc::from(vec![2u8; 300].into_boxed_slice()), + }); + let tree = Object::Tree(scl_core::Tree { + entries: vec![scl_core::TreeEntry { + name: "a.txt".into(), + kind: scl_core::EntryKind::Blob, + id: blob.id(), + mode: scl_core::FileMode(0o644), + perms: 0, + }], + }); + for o in [&blob, &sealed, &tree] { + assert_eq!(encoded_len(o), o.encode().len() as u64, "{}", o.kind_name()); + } + let id = blob.id(); + // The receiver rejects a canonical encoding above MAX_OBJECT_SIZE, and + // the encoding of a blob is its payload plus one tag byte. + let max = scl_core::MAX_OBJECT_SIZE as u64; + assert!(check_servable_size(&id, max).is_ok()); + assert!(matches!( + check_servable_size(&id, max + 1), + Err(Error::PackTooLarge(_)) + )); + assert_eq!(encoded_len(&Object::blob(vec![0u8; 10])), 11); + } + + #[test] + fn small_structural_objects_count_against_the_reply_cap() { + let layout = tmp_remote("getobjs-tree-cap"); + let t = LocalTransport::open(&layout.root).unwrap(); + let blob = Object::blob(b"x".to_vec()); + let entries: Vec = (0..20) + .map(|i| scl_core::TreeEntry { + name: format!("file-{i:02}.txt"), + kind: scl_core::EntryKind::Blob, + id: blob.id(), + mode: scl_core::FileMode(0o644), + perms: 0, + }) + .collect(); + let tree = Object::Tree(scl_core::Tree { entries }); + t.put_object(&tree.id(), &tree.encode()).unwrap(); + // A tree carries no blob payload but is several hundred bytes encoded; + // a 50-byte cap must refuse it. + assert!(matches!( + t.build_objects_tempfile(&[tree.id()], 50), + Err(Error::PackTooLarge(_)) + )); + assert!(t.build_objects_tempfile(&[tree.id()], 0).is_ok()); + std::fs::remove_dir_all(&layout.root).unwrap(); + } + + #[test] + fn local_get_objects_enforces_the_size_cap_and_id_cap() { + let layout = tmp_remote("getobjs-cap"); + let t = LocalTransport::open(&layout.root).unwrap(); + let blob = Object::blob(vec![7u8; 4096]); + t.put_object(&blob.id(), &blob.encode()).unwrap(); + assert!(matches!( + t.build_objects_tempfile(&[blob.id()], 100), + Err(Error::PackTooLarge(_)) + )); + let too_many = vec![blob.id(); crate::wire::MAX_GET_OBJECTS_IDS + 1]; + assert!(matches!( + t.get_objects(&too_many, &mut Vec::new()), + Err(Error::InvalidArgument(_)) + )); + std::fs::remove_dir_all(&layout.root).unwrap(); + } + #[test] fn local_transport_objects_and_refs_roundtrip() { let layout = tmp_remote("rt"); diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index 5a4cbfa6..da5d8983 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -25,7 +25,11 @@ use crate::error::{Error, Result}; /// object kinds (sealed object, branch manifest) now cross the wire inside /// packs — a v3 peer's `Object::decode` would hit an unknown tag deep in /// pack ingest, so the mismatch is refused cleanly at the handshake instead. -pub const PROTOCOL_VERSION: u32 = 4; +/// Bumped 4 -> 5 in P37 (ADR-0047): the additive `GetObjects` verb (batched +/// by-id object fetch) — a v4 server would answer its opcode with an +/// "unknown opcode" protocol error mid-session, so the mismatch is refused at +/// the handshake instead. +pub const PROTOCOL_VERSION: u32 = 5; // Request opcodes (one per Transport verb, plus session control). const OP_HELLO: u8 = 0x01; @@ -38,6 +42,12 @@ const OP_PUT_OBJECT: u8 = 0x14; const OP_UPDATE_REF: u8 = 0x15; const OP_GET_PACK: u8 = 0x16; const OP_PUT_PACK: u8 = 0x17; +const OP_GET_OBJECTS: u8 = 0x18; + +/// Most object ids one `GetObjects` request may carry (ADR-0047). Bounds the +/// server's per-request work and the response a client must be ready to read; +/// a client with a longer frontier splits it into several requests. +pub const MAX_GET_OBJECTS_IDS: usize = 1024; // Response status bytes. const ST_OK: u8 = 0; @@ -119,6 +129,13 @@ pub enum Request { /// (`write_pack_stream`); the receiver destreams them /// (`read_pack_stream`) before replying. PutPack, + /// Batched by-id object fetch (P37, ADR-0047): the server answers with an + /// OK frame whose body lists the ids it will NOT send (absent or + /// refused), then a chunk stream of the objects it found. At most + /// [`MAX_GET_OBJECTS_IDS`] ids. + GetObjects { + ids: Vec, + }, } // --- field encoding helpers (length-prefixed, matching core's canonical style) --- @@ -191,6 +208,18 @@ impl<'a> Cur<'a> { let n = self.u32()? as usize; (0..n).map(|_| self.id()).collect() } + /// Like [`Cur::ids`], but rejects a count above `max` before decoding any + /// id, so an oversize request is refused without work proportional to its + /// claimed length. Nothing is pre-sized from the count. + fn ids_capped(&mut self, max: usize) -> Result> { + let n = self.u32()? as usize; + if n > max { + return Err(Error::PackTooLarge(format!( + "request names {n} ids, limit is {max}" + ))); + } + (0..n).map(|_| self.id()).collect() + } /// Read a length prefix used to PRE-SIZE a collection, rejecting a fabricated /// count that exceeds the bytes actually remaining in the frame. Every element /// consumes >= 1 byte, so `n > remaining` can never be satisfied and is a @@ -280,6 +309,10 @@ impl Request { put_strs(&mut out, filter); } Request::PutPack => out.push(OP_PUT_PACK), + Request::GetObjects { ids } => { + out.push(OP_GET_OBJECTS); + put_ids(&mut out, ids); + } } out } @@ -318,6 +351,9 @@ impl Request { filter: c.strs()?, }, OP_PUT_PACK => Request::PutPack, + OP_GET_OBJECTS => Request::GetObjects { + ids: c.ids_capped(MAX_GET_OBJECTS_IDS)?, + }, op => return Err(Error::Protocol(format!("unknown opcode 0x{op:02x}"))), }; c.done()?; @@ -597,6 +633,21 @@ pub fn decode_ids_body(b: &[u8]) -> Result> { Ok(ids) } +/// Decode an id-list body that may name at most `max` ids. For a `GetObjects` +/// reply `max` is the number of ids requested, so a hostile peer cannot make +/// the client decode a list sized by the frame limit instead of the request. +pub fn decode_ids_body_capped(b: &[u8], max: usize) -> Result> { + let mut c = Cur::new(b); + let ids = c.ids_capped(max).map_err(|e| match e { + Error::PackTooLarge(_) => { + Error::Protocol(format!("reply lists more than the {max} id(s) requested")) + } + e => e, + })?; + c.done()?; + Ok(ids) +} + /// Response body for ListRefs: the remote's (branch, tip) pairs. pub fn refs_body(refs: &[(String, ObjectId)]) -> Vec { let mut b = Vec::new(); @@ -1019,6 +1070,30 @@ fn serve_session( } } } + Request::GetObjects { ids } => { + // Streams like GetPack, so it is handled inline. Bucket + // remotes refuse: they refuse partial clone (ADR-0046), and + // by-id reads exist only for partial clones. + let result = match &transport { + ServeTransport::Local(t) => { + t.build_objects_tempfile(&ids, policy.max_pack_size) + } + ServeTransport::Bucket { .. } => Err(Error::InvalidArgument( + "bucket remotes do not support by-id object fetch".into(), + )), + }; + match result { + Ok((not_sent, guard)) => { + write_ok(w, &ids_body(¬_sent))?; + let mut f = std::fs::File::open(guard.path())?; + write_pack_stream(w, &mut f, pack_chunk_size())?; + } + Err(e) => { + let (code, msg) = err_to_wire(&e); + write_err(w, code, &msg)?; + } + } + } Request::PutPack => { match spill_pack_stream(r, &transport.tmp_dir(), policy.max_pack_size) { Ok(guard) => { @@ -1083,7 +1158,10 @@ fn serve_session( .as_transport() .update_ref(&branch, &id, expected_old.as_ref()) .map(|()| Vec::new()), - Request::Bye | Request::GetPack { .. } | Request::PutPack => { + Request::Bye + | Request::GetPack { .. } + | Request::PutPack + | Request::GetObjects { .. } => { unreachable!("handled above") } }; @@ -1162,6 +1240,27 @@ mod tests { root } + #[test] + fn get_objects_decode_rejects_more_than_the_id_cap() { + let ids: Vec = (0..=MAX_GET_OBJECTS_IDS) + .map(|i| ObjectId::from_bytes([(i % 251) as u8; 32])) + .collect(); + let bytes = Request::GetObjects { ids }.encode(); + assert!(matches!( + Request::decode(&bytes), + Err(Error::PackTooLarge(_)) + )); + // Exactly at the cap is fine. + let ids: Vec = (0..MAX_GET_OBJECTS_IDS) + .map(|i| ObjectId::from_bytes([(i % 251) as u8; 32])) + .collect(); + let bytes = Request::GetObjects { ids: ids.clone() }.encode(); + assert_eq!( + Request::decode(&bytes).unwrap(), + Request::GetObjects { ids } + ); + } + #[test] fn every_request_encodes_and_decodes_roundtrip() { let reqs = vec![ @@ -1203,6 +1302,10 @@ mod tests { filter: vec![], }, Request::PutPack, + Request::GetObjects { ids: vec![] }, + Request::GetObjects { + ids: vec![some_id(10), some_id(11)], + }, ]; for req in reqs { let bytes = req.encode(); @@ -1326,6 +1429,11 @@ mod tests { assert!(!decode_bool_body(&bool_body(false)).unwrap()); let ids = vec![some_id(1), some_id(2)]; assert_eq!(decode_ids_body(&ids_body(&ids)).unwrap(), ids); + assert_eq!(decode_ids_body_capped(&ids_body(&ids), 2).unwrap(), ids); + assert!(matches!( + decode_ids_body_capped(&ids_body(&ids), 1), + Err(Error::Protocol(_)) + )); let refs = vec![ ("dev".to_string(), some_id(3)), ("main".to_string(), some_id(4)), @@ -1800,6 +1908,67 @@ mod tests { let _ = std::fs::remove_dir_all(&root); } + /// `GetObjects` is a read verb: a read-only session serves it, and the + /// `max_pack_size` spool cap applies to its reply. + #[test] + fn readonly_session_serves_get_objects_and_caps_the_reply() { + let root = tmp_repo("ro-getobjs"); + let blob = scl_core::Object::blob(vec![5u8; 4096]); + crate::transport::LocalTransport::open(&root) + .unwrap() + .put_object(&blob.id(), &blob.encode()) + .unwrap(); + + let run = |max_pack_size: u64| { + let mut input = Vec::new(); + write_frame( + &mut input, + &Request::Hello { + version: PROTOCOL_VERSION, + } + .encode(), + ) + .unwrap(); + write_frame( + &mut input, + &Request::GetObjects { + ids: vec![blob.id()], + } + .encode(), + ) + .unwrap(); + // A follow-up request proves the session survives a capped reply. + write_frame(&mut input, &Request::HeadBranch.encode()).unwrap(); + let mut output = Vec::new(); + let policy = WirePolicy { + read_only: true, + max_pack_size, + ro_drain_cap: 4096, + }; + serve_with_policy(&root, &mut std::io::Cursor::new(input), &mut output, policy) + .unwrap(); + let mut r = std::io::Cursor::new(output); + let _hello = parse_response(read_frame_opt(&mut r).unwrap().unwrap()).unwrap(); + let reply = parse_response(read_frame_opt(&mut r).unwrap().unwrap()); + (reply, r) + }; + + let (reply, mut rest) = run(0); + assert!(decode_ids_body(&reply.unwrap()).unwrap().is_empty()); + let mut pack = Vec::new(); + read_pack_stream(&mut rest, &mut pack, 0).unwrap(); + assert_eq!(scl_core::pack::parse_pack(&pack).unwrap().len(), 1); + let head = parse_response(read_frame_opt(&mut rest).unwrap().unwrap()).unwrap(); + assert_eq!(decode_str_body(&head).unwrap(), "main"); + + // A cap below the object's size fails the request, not the session. + let (reply, mut rest) = run(100); + assert!(matches!(reply.unwrap_err(), Error::PackTooLarge(_))); + let head = parse_response(read_frame_opt(&mut rest).unwrap().unwrap()).unwrap(); + assert_eq!(decode_str_body(&head).unwrap(), "main"); + let _ = std::fs::remove_dir_all(&root); + } + /// Mint a one-commit object set the same way /// `bucket_transport::tests::tiny_history` does (blob -> tree -> snapshot /// via a real scratch repo), returning `(tip, objects)`. `tag` diff --git a/demo/run_lazy_fetch_demo.sh b/demo/run_lazy_fetch_demo.sh new file mode 100755 index 00000000..accdec68 --- /dev/null +++ b/demo/run_lazy_fetch_demo.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# P37 demo: want-by-id fetch on a lazy partial clone (ADR-0047). Proves that +# `sc cat :` on a `--lazy` partial clone fetches ONLY the trees and +# the blob on the path it reads (by object id, one batch per tree depth), that +# a sibling file stays unfetched, that a non-lazy clone and `SC_OFFLINE=1` +# keep the loud gap error with zero network, and that the same fetch works +# over a real `sc+http://` loopback server (protocol v5 `GetObjects`). +# +# Self-checking: every claim is an assertion; any failure exits non-zero +# before the RESULT line. +set -euo pipefail + +cd "$(dirname "$0")/.." +cargo build --quiet --bin sc +SC="$PWD/target/debug/sc" + +fail() { echo "FAIL: $1"; exit 1; } + +W="$(mktemp -d)" +SERVER_PID="" +cleanup() { + if [ -n "$SERVER_PID" ] && kill -0 "$SERVER_PID" 2>/dev/null; then + kill "$SERVER_PID" 2>/dev/null || true + wait "$SERVER_PID" 2>/dev/null || true + fi + rm -rf "$W" +} +trap cleanup EXIT + +objects() { find "$1/.sc/objects" -type f | wc -l | tr -d ' '; } + +ORIGIN="$W/origin" +mkdir -p "$ORIGIN" +( + cd "$ORIGIN" + "$SC" init >/dev/null + mkdir -p src docs/api + printf 'fn main() {}\n' > src/a.txt + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > docs/api/guide.bin + head -c 65536 /dev/urandom | od -An -tx1 | tr -d ' \n' > docs/api/other.bin + "$SC" commit -m "base" --author demo >/dev/null +) +FULL=$(objects "$ORIGIN") +echo "origin: src/ + docs/api/{guide,other}.bin committed; $FULL object(s) ✔" + +echo +echo "=== 1: lazy partial clone (--filter src/ --lazy) fetches only src/ ===" +LAZY="$W/lazy" +"$SC" clone --filter src/ --lazy "$ORIGIN" "$LAZY" >/dev/null +BEFORE=$(objects "$LAZY") +[ "$BEFORE" -lt "$FULL" ] || fail "partial clone must hold fewer objects ($BEFORE) than the origin ($FULL)" +[ -f "$LAZY/.sc/promisor-lazy" ] || fail "--lazy must write .sc/promisor-lazy" +echo "lazy clone: $BEFORE object(s) of $FULL ✔" + +echo +echo "=== 2: sc cat fetches exactly the path it reads ===" +(cd "$LAZY" && "$SC" cat main:docs/api/guide.bin) > "$W/guide.out" +cmp "$W/guide.out" "$ORIGIN/docs/api/guide.bin" || fail "cat output must be byte-identical to the origin file" +AFTER=$(objects "$LAZY") +# docs tree + docs/api tree + guide blob = 3 new objects, and not other.bin. +[ "$((AFTER - BEFORE))" -eq 3 ] || fail "expected exactly 3 new objects (2 trees + 1 blob), got $((AFTER - BEFORE))" +[ "$AFTER" -lt "$FULL" ] || fail "the sibling blob must still be unfetched ($AFTER of $FULL)" +echo "cat main:docs/api/guide.bin: byte-identical, +3 objects, sibling other.bin not fetched ✔" + +echo +echo "=== 3: a second cat of the same path fetches nothing ===" +(cd "$LAZY" && "$SC" cat main:docs/api/guide.bin) > /dev/null +[ "$(objects "$LAZY")" -eq "$AFTER" ] || fail "re-reading a filled path must not fetch" +echo "re-read: no new objects ✔" + +echo +echo "=== 4: SC_OFFLINE=1 keeps the loud gap error, with zero network ===" +BEFORE_OFF=$(objects "$LAZY") +if (cd "$LAZY" && SC_OFFLINE=1 "$SC" cat main:docs/api/other.bin) >/dev/null 2>"$W/off.err"; then + fail "offline cat of an unfetched path must fail" +fi +grep -q "backfill" "$W/off.err" || fail "offline error must point at sc backfill: $(cat "$W/off.err")" +[ "$(objects "$LAZY")" -eq "$BEFORE_OFF" ] || fail "offline cat must not fetch" +echo "SC_OFFLINE=1: gap error mentions sc backfill, nothing fetched ✔" + +echo +echo "=== 5: a non-lazy partial clone keeps the P27 gap error ===" +EAGER="$W/eager" +"$SC" clone --filter src/ "$ORIGIN" "$EAGER" >/dev/null +EAGER_BEFORE=$(objects "$EAGER") +if (cd "$EAGER" && "$SC" cat main:docs/api/guide.bin) >/dev/null 2>"$W/eager.err"; then + fail "non-lazy partial clone must not fetch on read" +fi +grep -q "backfill" "$W/eager.err" || fail "non-lazy error must point at sc backfill: $(cat "$W/eager.err")" +[ "$(objects "$EAGER")" -eq "$EAGER_BEFORE" ] || fail "non-lazy cat must not fetch" +echo "non-lazy clone: gap error, nothing fetched ✔" + +echo +echo "=== 6: sc backfill --lazy opts an existing partial clone in ===" +(cd "$EAGER" && "$SC" backfill --lazy >/dev/null) +(cd "$EAGER" && "$SC" cat main:docs/api/other.bin) > "$W/other.out" +cmp "$W/other.out" "$ORIGIN/docs/api/other.bin" || fail "lazy-after-the-fact cat must be byte-identical" +echo "backfill --lazy then cat: byte-identical ✔" + +echo +echo "=== 7: the same fetch over a real sc+http:// loopback server (protocol v5) ===" +pick_port() { + local candidate + for candidate in 8741 8742 8743 8899 18741; do + if ! nc -z 127.0.0.1 "$candidate" 2>/dev/null; then echo "$candidate"; return 0; fi + done + fail "no free port found" +} +PORT="$(pick_port)" +"$SC" serve --http "127.0.0.1:$PORT" --read-only "$ORIGIN" >/dev/null 2>&1 & +SERVER_PID=$! +tries=0 +while ! nc -z 127.0.0.1 "$PORT" 2>/dev/null; do + tries=$((tries + 1)); [ "$tries" -lt 100 ] || fail "server never became ready"; sleep 0.05 +done +HTTP="$W/http" +"$SC" clone --filter src/ --lazy "sc+http://127.0.0.1:$PORT/repo" "$HTTP" >/dev/null +HTTP_BEFORE=$(objects "$HTTP") +(cd "$HTTP" && "$SC" cat main:docs/api/guide.bin) > "$W/http.out" +cmp "$W/http.out" "$ORIGIN/docs/api/guide.bin" || fail "http cat must be byte-identical" +[ "$(($(objects "$HTTP") - HTTP_BEFORE))" -eq 3 ] || fail "http cat must fetch exactly 3 objects, got $(($(objects "$HTTP") - HTTP_BEFORE))" +[ "$(objects "$HTTP")" -lt "$FULL" ] || fail "http cat must leave the sibling blob unfetched" +echo "read-only sc+http server: lazy cat byte-identical, +3 objects, sibling not fetched ✔" + +echo +echo "RESULT: lazy by-id fetch verified (path-only fetch, offline/non-lazy gap errors, http v5)" diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 307c13cd..cd20dcb8 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -372,6 +372,33 @@ too, not just the src-control-side metadata. described above, so a hostile or corrupted bucket gets no more leverage against a serve instance than it would against a direct `sc` client. +## By-id object reads (`GetObjects`, `sc cat`) — ADR-0047 + +- **A hash is not a capability.** Anyone who can reach a served repo and + knows (or guesses) an object id can ask for it by id. Reads are scoped by + who may connect (ADR-0040 tokens, `--read-only`), not by which ids they + name. This is no wider than `GetObject`, which already returns any stored + object by id. +- **Sealed private-branch objects and manifests are refused by kind** + (`Object::Sealed`, `Object::Manifest`) and reported as "not sent", the same + reply as an absent id. This keeps `GetObjects` consistent with filtered + `GetPack`. It is **not** a confidentiality boundary: those objects are + ciphertext, an unfiltered fetch still serves them, and `HasObject` still + reveals whether an id exists. `GetObject` is unchanged and still serves any + kind; closing that is a tracked follow-on, not part of this phase. +- **The client trusts nothing in the reply.** Every record is BLAKE3-verified, + and the reply must partition the request exactly (nothing unrequested, + nothing both sent and refused, nothing omitted) before any object is + written. A hostile remote can at worst answer "not sent"; it cannot plant an + object under another id or smuggle in extras. The reply stream a client will + read is capped at 1 GiB. +- **Server cost is bounded per request:** 1024 ids (checked while decoding, + before any id is read), the server's `--max-pack-size` over the summed + encoded size, and `MAX_OBJECT_SIZE` per object. +- **Laziness is a local opt-in.** `.sc/promisor-lazy` is never committed or + transferred; without it a partial clone never dials the remote after clone, + and `SC_OFFLINE=1` forces that per process. + ## Untrusted-input hardening (DoS) — ADR-0039 - A single `MAX_OBJECT_SIZE` (256 MiB) caps every untrusted length: wire frames, diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md index 2e6a2ff6..3231fe11 100644 --- a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -1,6 +1,6 @@ # ADR-0047: Want-by-id fetch and explicit gap fill for partial clones -- **Status:** Proposed +- **Status:** Accepted for 37a (built); 37b Proposed - **Date:** 2026-10-16 - **Phase:** 37 - **Builds on:** ADR-0037 (partial clone), ADR-0035 (streaming transfer), ADR-0039/0041 (size and listener limits), ADR-0040 (HTTP access control), ADR-0044 (private branches) @@ -33,91 +33,87 @@ stream bounds RAM, and P27 gives us a gap-tolerant store plus a durable Add a **batched by-id object request** and an **explicit gap-fill seam** at operation entry points. `Store::get` stays network-free, permanently. -1. **Wire: `Request::GetObjects { ids: Vec }`.** The reply is the - requested objects as a normal P25 pack stream (`ST_PACK_CHUNK`/ - `ST_PACK_END`). No `haves`, no reachability walk: server cost is - O(requested objects), independent of history size. Ids the server lacks are - reported in a trailing missing-id list rather than failing the whole batch. - `PROTOCOL_VERSION` 4 → 5 (additive verb; a v4 peer is rejected at +1. **Wire: `Request::GetObjects { ids: Vec }`.** The reply is an OK + frame whose body lists the ids the server will not send, followed by the + found objects as a normal P25 pack stream (`ST_PACK_CHUNK`/`ST_PACK_END`). + No `haves`, no reachability walk: server cost is O(requested objects), + independent of history size. A missing id does not fail the batch. + `PROTOCOL_VERSION` 4 to 5 (additive verb; a v4 peer is rejected at the handshake, same discipline as P25/P27/P34). -2. **Transport: `Transport::get_objects(&self, ids, out)`** with a default - implementation that loops `get_object`, so local-path, ssh, http(s) and - test transports compose immediately; stdio/http override it to use the one - round trip. Bucket remotes keep refusing partial clone (ADR-0046) and are - out of scope. -3. **Client seam: `Repo::fill_gaps(plan)`**, the only code path allowed to - dial the promisor remote after clone. An operation that needs more than the - filter *plans first*: it walks the trees it already holds, collects the - absent child ids (the "frontier"), batches them into one `GetObjects`, - verifies every returned object against the requested id and - `MAX_OBJECT_SIZE`, and ingests each depth's batch verify-before-write (the - P25 two-pass: every record is checked before any is written). This is - integrity, **not** transactionality: objects are content-addressed and - immutable, so a batch that lands before a later depth fails is simply - retained, and a retry recomputes the frontier from *current presence* and - resumes where it stopped. An id the server reports missing that the plan - requires is terminal (the plan fails with the gap error; nothing dependent - runs). The dependent operation mutates only after the whole plan is - satisfied. Fetched objects are ordinary store objects, so GC reachability - treats them like any other; `fill_gaps` holds the repo lock across a plan - so a concurrent `sc gc` cannot prune a half-filled frontier. Cost: one - round trip per directory depth, not per object. -4. **Opt-in, not default.** Laziness is recorded in a **separate marker file, - `.sc/promisor-lazy`** (local, uncommitted), set by `sc clone --filter - --lazy` or `sc backfill --lazy`. It is deliberately *not* a field - in `.sc/promisor`: that file's format is `origin ` followed by one - path prefix per line (`promisor::load`), and a prefix such as `lazy = true` - is a legal path, so a new field would silently reinterpret a legacy clone's - filter. Without the marker the P27 behavior is byte-identical: the gap - error with the `sc backfill` hint. With it, the entry points below call - `fill_gaps` before proceeding. `SC_OFFLINE=1` (and `--offline`) force the - gap error even when the marker is present, so offline use stays - predictable. -5. **Entry points that may call `fill_gaps`** (an exhaustive allow-list, - pinned by a test that greps for the call). In 37a, only operations that - exist today and can request an out-of-filter path: `sparse set`/`disable` - widening and the `switch`/materialize that follows it; plus one **new** - read surface scoped into 37a, `sc cat :` (print one path at a - ref). Today's `sc diff` compares the working tree to HEAD only and `sc log` - has no patch output, so neither is an entry point until a later phase gives - them revision/path arguments. In 37b: merge, cherry-pick/rebase, `ws - fork`/`harvest`, `work` and `export`. Everything else, including gc, - verify and the desktop read model (ADR-0045), never touches the network. -6. **Gap semantics change from "out of filter" to "absent".** `sc verify` - currently counts gaps by filter membership. Once fill-gap can fetch - out-of-filter objects, a fetched object is simply present and the filter - remains only the *initial* fetch scope. `reachable_objects_filtered`'s - `gaps` set continues to be computed against presence, and gc must keep - everything reachable regardless of how it arrived. +2. **Transport: `Transport::get_objects(&self, ids, out) -> not_sent`.** The + trait default refuses. `LocalTransport` (local paths and every wire + server), `WireClient`, `StdioTransport` (ssh) and `HttpTransport` + implement it; the two wrappers forward explicitly, because a default that + looped `get_object` would cost one round trip per object. Bucket remotes + refuse (ADR-0046 already refuses partial clone there). +3. **Client seam.** `Repo::fetch_by_id_from(transport, ids)` is the only + function that dials the promisor remote after clone, called through the + `Filler` in `crates/repo/src/fill.rs`. It fetches one batch, hash-verifies + every record, then checks the reply partitions the request exactly: no + unrequested object, nothing both sent and refused, nothing silently + omitted. Only then does it write. This is verify-before-write, not a + transaction: objects are immutable and content-addressed, so objects that + landed before a later failure are kept, and a retry recomputes what is + missing from current presence. An id the server refuses is terminal + (`NotFound`). `Repo` already holds the repo lock for its lifetime, so a + fill cannot interleave with `sc gc`; the seam takes no second lock. +4. **Opt-in, not default.** Laziness is a separate marker file, + `.sc/promisor-lazy` (local, uncommitted), set by `sc clone --filter + --lazy` or `sc backfill --lazy`. It is not a field in + `.sc/promisor`: that file is `origin ` followed by one path prefix per + line, and a prefix such as `lazy = true` is a legal path, so a new field + would silently reinterpret a legacy clone's filter. Without the marker the + P27 behavior is unchanged: the gap error with the `sc backfill` hint. + `SC_OFFLINE=1` forces the gap error even when the marker is present. (A + `--offline` flag is deferred; the environment variable covers scripts and + CI.) +5. **Entry points.** 37a has exactly one: the new `sc cat :`, which + walks from the ref's root tree and fetches each absent tree and the final + blob, one round trip per level. It refuses protected paths and private + branches. Today's `sc diff` compares the working tree to HEAD only and `sc + log` has no patch output, so neither is an entry point until they gain + revision and path arguments. Everything else, including gc, verify and the + desktop read model (ADR-0045), never touches the network. +6. **A fill never widens the promisor filter, and 37a keeps it that way on + purpose.** The filter decides what `push`, `gc` and `sc verify` walk. + Extra present objects outside it are harmless to all three (gc keeps them, + push ignores them, verify counts filter-derived gaps as before). Widening + the sparse view over a filled path is different: a commit there would + create objects `push` never sends, leaving the origin with a snapshot that + points at objects it does not have. That needs presence-based push, verify + and gc, which is 37b's design work, so `sc sparse set` beyond the filter + keeps its gap error in 37a. ### Authorization of by-id reads -A hash is not a capability, and Scion delegates this to an IAM layer we don't -have. Rules for `GetObjects`, in order of preference: - -- It is **read-scope**: a `--read-only` token (ADR-0040) may call it; no - write verb is implied. -- It **must not be a bypass of private-branch exclusion** (ADR-0044). Today a - filtered `GetPack` excludes private branches only by dropping wants that - resolve to a branch *manifest* (`filter_manifest_wants`); sealed *member* - objects are not recognized by that helper, and the existing single-object - `GetObject` verb (and `LocalTransport::get_object`) returns any stored - object by id. Reusing the manifest filter would therefore **not** protect - member ids. `GetObjects` (and `GetObject`, which has the same exposure - today) need a member-level authorization decision before 37a: either the - server recognizes sealed object kinds and answers "missing" for callers - outside the recipient set, or the policy is explicitly relaxed because - sealed members are ciphertext under a per-branch KEK. That choice, applied - uniformly to both verbs and to the default/local `get_objects` - implementation, is the first open question below and gates 37a. -- Protected-path ciphertext is returned as-is, exactly as it already is in - every pack: confidentiality rests on encryption, not on withholding. -- Caps: ids per request, aggregate bytes per request (reusing - `--max-pack-size`), and the existing per-object `MAX_OBJECT_SIZE`. - Over-limit requests fail with `EC_TOO_LARGE`; the client splits and retries. - -Whether to additionally require that a requested id be *reachable from a -visible ref* (a walk, so not O(request)) is part of open question 1 below. +A hash is not a capability, and Scion delegates this to an IAM layer we do +not have. The rules for `GetObjects`: + +- It is a read verb: a `--read-only` session (ADR-0040) serves it. +- It never serves a sealed-branch manifest or a sealed member object + (`Object::Manifest`, `Object::Sealed`). The server answers "not sent", + the same reply as an absent id. This matches filtered `GetPack`, which + already excludes private branches, and a partial-clone client never needs + those kinds. It is a consistency rule, not a confidentiality boundary: + sealed objects are ciphertext, an unfiltered `GetPack` still serves them, + and `HasObject` still reveals whether an id exists, so "indistinguishable + from absent" holds for this verb only. +- It cannot enforce private-branch membership by kind alone. A private + branch keeps its fork-point snapshot and carried public trees and blobs + unsealed, but those are public content by construction. +- The existing single-object `GetObject` is unchanged in 37a. It already + returns any stored object by id; this ADR does not widen that exposure and + does not close it. Closing it needs a decision about unfiltered private + transfers and is tracked as a follow-on. +- Protected-path ciphertext is returned as-is, as in every pack: + confidentiality rests on encryption, not on withholding. +- Caps: 1024 ids per request (`MAX_GET_OBJECTS_IDS`, checked while decoding), + the server's `--max-pack-size` over the summed canonical encoded size of a reply and + `MAX_OBJECT_SIZE` per object (both `PackTooLarge`, before any pack byte is + written), and a 1 GiB cap on the stream a client will read back. A client + also stops reading a pack as soon as it holds more records than it + requested, so repeating one large object cannot multiply the receiver's + decompress-and-hash work. ## Consequences @@ -139,22 +135,30 @@ visible ref* (a walk, so not O(request)) is part of open question 1 below. ## Open questions -1. **Member-level authorization and reachability on by-id reads (gates - 37a).** Choose between kind-based refusal of sealed members, a - visible-ref reachability check, or an explicit relaxation; apply it to - `GetObject` too. O(request) is the point of the verb; - a reachability check undoes it. Candidate: cache a per-connection - "visible set" from `ListRefs` tips and check membership lazily via the - tree the id was named in. Decide with a threat-model pass before 37a lands. -2. **Does `fill_gaps` widen `.sc/promisor`'s filter?** Leaning no — the filter - stays the *initial* scope, and presence is tracked by the store — but this - interacts with `sc verify`'s `partial: N gaps` line and `sc backfill - --all`'s "convert to full clone" semantics. -3. **Merge on a partial clone** needs base/ours/theirs trees for only the - *conflicting* paths. Whether the plan can be computed from tree ids alone - (identical ids ⇒ no fetch) or needs a server-assisted diff is a 37b spike. -4. **Bucket remotes.** `sc+wal://` stores objects inside packs; by-id reads need - an object→pack index. Out of scope; partial clone from buckets stays refused. +1. **Presence-based walks (gates 37b).** Push, verify and gc currently ask + "is this path inside the filter?". Widening the sparse view, and any + operation that commits outside the filter (merge, rebase), needs them to + ask "is this object present?" instead, with the filter recording only the + initial fetch scope. Push must then send every reachable present object + the origin lacks. +2. **Frontier closure and ancestors.** `fill_closure` over a whole subtree + must traverse present trees too (a tree can land before its children, so a + retry cannot trust a present root), and the ancestors of an arbitrary + prefix may be absent: only the root and the ancestors of the original + filter are guaranteed. +3. **Merge on a partial clone** needs base, ours and theirs trees for only the + conflicting paths. Whether tree ids alone suffice (identical ids need no + fetch) or a server-assisted diff is needed is a 37b spike. +4. **Reply size caps and batching.** The client reads at most 1 GiB per reply + while the server's default `--max-pack-size` is larger, so a batch that + crosses 1 GiB fails mid-stream and leaves that connection out of sync. 37a + sends one id per call so it cannot happen; 37b must split batches by size + or close the transport on that error. +5. **`GetObject` and unfiltered private transfers.** See the authorization + section. +6. **Bucket remotes.** `sc+wal://` stores objects inside packs; by-id reads need + an object-to-pack index. Out of scope; partial clone from buckets stays + refused. ## Alternatives considered @@ -170,12 +174,37 @@ visible ref* (a walk, so not O(request)) is part of open question 1 below. - **Adopt Scion's changeset model.** Changes every object id; breaks the content-addressing invariant for no gain at our scale. +## Refinements from the 37a plan review + +Two independent reviews of the 37a plan (an Opus review and a Sol review) +found problems that changed the scope. Recorded so the next phase starts from +them. + +- The plan widened the sparse view after a fill and kept the promisor filter + as is. Both reviews showed `push` walks with the filter (`sync.rs`), so an + edit under a filled prefix would never reach the origin. 37a was cut back to + the read-only entry point (`sc cat`); the sparse widen moved to 37b. +- "Ancestors are present" was wrong for any prefix outside the original + filter, and the frontier loop could not resume after a partial fill. Both + are open question 2; `sc cat` avoids them by walking one path and checking + presence at each level. +- Reusing `filter_manifest_wants` would not have protected sealed members. The + server now refuses by kind in one helper shared by every transport. +- Ssh and http transports forward every method by hand, so the default would + have silently degraded to one round trip per object. They forward + `get_objects` explicitly. +- Reply validation needed a hook between the ingest passes + (`ingest_pack_file_checked`) so unrequested objects are rejected before any + is written. + ## Phasing -- **37a** — gated on open question 1. `GetObjects` verb, - `Transport::get_objects`, `Repo::fill_gaps`, `--lazy` (the - `.sc/promisor-lazy` marker)/`SC_OFFLINE`, entry points: sparse widen + - switch, and the new `sc cat :`. Demo: `demo/run_lazy_fetch_demo.sh` counts objects fetched and - proves `Store::get` stays offline. -- **37b** — lift the P27 refusals (merge, replay, ws, work, export), one - operation per task. +- **37a (built).** `GetObjects` verb and protocol v5, `Transport::get_objects`, + `Repo::fetch_by_id_from`, `.sc/promisor-lazy` and `SC_OFFLINE`, `sc clone + --lazy`, `sc backfill --lazy`, `sc cat :`. Proof: + `demo/run_lazy_fetch_demo.sh` (a path-only fetch of exactly three objects, + sibling untouched, offline and non-lazy gap errors, a read-only `sc+http` + server). +- **37b (proposed).** Presence-based push, verify and gc; then sparse widen + over a filled prefix; then lift the P27 refusals (merge, replay, ws, work, + export), one operation per task.