diff --git a/ROADMAP.md b/ROADMAP.md index 279e9eea..80311282 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -548,8 +548,12 @@ across every phase. ## Active -**None.** Phase 35 completed the first native desktop vertical slice; the next -horizon is TBD. +**P37 — Want-by-id fetch and gap fill (proposed, not started).** Batched +`GetObjects` verb + an explicit `fill_gaps` seam so partial clones can run +merge/rebase/ws/work/export without `Store::get` ever touching the network; +design in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md), prior art +in `docs/research/scion-evaluation.md`. Phases: 37a (verb + read-only entry +points), 37b (lift the P27 refusals). ## Completed phases (usability-first ordering) @@ -739,6 +743,9 @@ scale-&-reach horizon): origin automatically on any gap, matching git's promisor default. Real scope/risk (offline breakage, surprise latency, fetch-during-gc), so it stays deferred, not accidentally missing. + **Now scoped as P37 (ADR-0047, Proposed):** an opt-in explicit `fill_gaps` + seam over a batched by-id `GetObjects` verb, keeping `Store::get` + network-free. - **Per-case gap-tolerant merge/rebase/ws-harvest/sc-work on a partial clone (P27).** These are refused entirely today (`Error::PartialCloneUnsupported`) rather than gap-tolerantly @@ -746,6 +753,8 @@ scale-&-reach horizon): Threading gap-tolerance through `three_way`/replay/`ws::harvest` individually is the follow-on if the "backfill to a full clone first" workaround proves too coarse in practice. + **Now scoped as P37b (ADR-0047, Proposed):** lifted one operation at a + time on top of the P37a `fill_gaps` seam. - **Blob-size/object-count clone filters (P27).** `sc clone --filter` is prefix-only, matching the sparse path-boundary machinery already built; git's `--filter=blob:limit` size-based filtering is a separate axis, @@ -944,6 +953,15 @@ scale-&-reach horizon): the manifest's compare-and-swap; there is no lease/TTL primitive for operations that need to hold exclusive intent across more than one bucket round-trip (e.g. a long-running compaction). Deferred. +- **Derived-data layer (Scion-inspired, after P37).** Blame, path history, + commit-graph skip lists and copy-detection computed by one generic + recursive mechanism over snapshots, stored outside the CAS and rebuildable + (so no id/format change). Would speed merge-base/ancestry queries and feed + the desktop app. Needs its own ADR; see `docs/research/scion-evaluation.md`. +- **Server-side history ops + ref-CAS commits (Scion-inspired, after P37).** + Server RPCs for log/diff/blame/merge so agents and CI need no local repo, + and a commit-by-ref-CAS path that skips push. Builds on P29 auth, P31 + limits, and ADR-0047's by-id reads; a further wire bump. - **Partial clone from bucket remotes (P36a follow-on).** `sc clone --filter` against `sc+wal://`/`sc+s3://` is refused outright (`BucketTransport` has no per-prefix negotiation); teaching the WAL format diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md new file mode 100644 index 00000000..2e6a2ff6 --- /dev/null +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -0,0 +1,181 @@ +# ADR-0047: Want-by-id fetch and explicit gap fill for partial clones + +- **Status:** Proposed +- **Date:** 2026-10-16 +- **Phase:** 37 +- **Builds on:** ADR-0037 (partial clone), ADR-0035 (streaming transfer), ADR-0039/0041 (size and listener limits), ADR-0040 (HTTP access control), ADR-0044 (private branches) +- **Research:** `docs/research/scion-evaluation.md` + +## Context + +A partial clone (ADR-0037) fetches only in-filter objects and then refuses +every operation that needs the rest: merge, cherry-pick/rebase, `ws +fork`/`harvest`, `work`, `export`, and `sparse disable`. ADR-0037 rejected +*transparent* lazy-fetch because it would put network I/O inside `Store::get`, +which gc, verify, materialize, merge and export all call — offline breakage, +surprise latency, fetch-during-gc. It left `sc backfill ` as the +manual escape hatch and recorded lazy-fetch as a deferred follow-on. + +Anthropic's Scion write-up (see the research note) names the same failure on +git partial clones and fixes it differently: the client never describes what +it *has*. It asks for specific immutable trees and blobs *by id*, starting at +the root tree and descending, and any cache can answer because the objects are +immutable. The network is not hidden in a read path; "what do I need next" is +an explicit, cheap, batchable request. + +We already have the pieces: `Request::GetObject(id)` exists on the wire, +objects are self-authenticating (BLAKE3 of canonical encoding), the P25 chunk +stream bounds RAM, and P27 gives us a gap-tolerant store plus a durable +`.sc/promisor` marker. + +## Decision + +Add a **batched by-id object request** and an **explicit gap-fill seam** at +operation entry points. `Store::get` stays network-free, permanently. + +1. **Wire: `Request::GetObjects { ids: Vec }`.** The reply is the + requested objects as a normal P25 pack stream (`ST_PACK_CHUNK`/ + `ST_PACK_END`). No `haves`, no reachability walk: server cost is + O(requested objects), independent of history size. Ids the server lacks are + reported in a trailing missing-id list rather than failing the whole batch. + `PROTOCOL_VERSION` 4 → 5 (additive verb; a v4 peer is rejected at + handshake, same discipline as P25/P27/P34). +2. **Transport: `Transport::get_objects(&self, ids, out)`** with a default + implementation that loops `get_object`, so local-path, ssh, http(s) and + test transports compose immediately; stdio/http override it to use the one + round trip. Bucket remotes keep refusing partial clone (ADR-0046) and are + out of scope. +3. **Client seam: `Repo::fill_gaps(plan)`**, the only code path allowed to + dial the promisor remote after clone. An operation that needs more than the + filter *plans first*: it walks the trees it already holds, collects the + absent child ids (the "frontier"), batches them into one `GetObjects`, + verifies every returned object against the requested id and + `MAX_OBJECT_SIZE`, and ingests each depth's batch verify-before-write (the + P25 two-pass: every record is checked before any is written). This is + integrity, **not** transactionality: objects are content-addressed and + immutable, so a batch that lands before a later depth fails is simply + retained, and a retry recomputes the frontier from *current presence* and + resumes where it stopped. An id the server reports missing that the plan + requires is terminal (the plan fails with the gap error; nothing dependent + runs). The dependent operation mutates only after the whole plan is + satisfied. Fetched objects are ordinary store objects, so GC reachability + treats them like any other; `fill_gaps` holds the repo lock across a plan + so a concurrent `sc gc` cannot prune a half-filled frontier. Cost: one + round trip per directory depth, not per object. +4. **Opt-in, not default.** Laziness is recorded in a **separate marker file, + `.sc/promisor-lazy`** (local, uncommitted), set by `sc clone --filter + --lazy` or `sc backfill --lazy`. It is deliberately *not* a field + in `.sc/promisor`: that file's format is `origin ` followed by one + path prefix per line (`promisor::load`), and a prefix such as `lazy = true` + is a legal path, so a new field would silently reinterpret a legacy clone's + filter. Without the marker the P27 behavior is byte-identical: the gap + error with the `sc backfill` hint. With it, the entry points below call + `fill_gaps` before proceeding. `SC_OFFLINE=1` (and `--offline`) force the + gap error even when the marker is present, so offline use stays + predictable. +5. **Entry points that may call `fill_gaps`** (an exhaustive allow-list, + pinned by a test that greps for the call). In 37a, only operations that + exist today and can request an out-of-filter path: `sparse set`/`disable` + widening and the `switch`/materialize that follows it; plus one **new** + read surface scoped into 37a, `sc cat :` (print one path at a + ref). Today's `sc diff` compares the working tree to HEAD only and `sc log` + has no patch output, so neither is an entry point until a later phase gives + them revision/path arguments. In 37b: merge, cherry-pick/rebase, `ws + fork`/`harvest`, `work` and `export`. Everything else, including gc, + verify and the desktop read model (ADR-0045), never touches the network. +6. **Gap semantics change from "out of filter" to "absent".** `sc verify` + currently counts gaps by filter membership. Once fill-gap can fetch + out-of-filter objects, a fetched object is simply present and the filter + remains only the *initial* fetch scope. `reachable_objects_filtered`'s + `gaps` set continues to be computed against presence, and gc must keep + everything reachable regardless of how it arrived. + +### Authorization of by-id reads + +A hash is not a capability, and Scion delegates this to an IAM layer we don't +have. Rules for `GetObjects`, in order of preference: + +- It is **read-scope**: a `--read-only` token (ADR-0040) may call it; no + write verb is implied. +- It **must not be a bypass of private-branch exclusion** (ADR-0044). Today a + filtered `GetPack` excludes private branches only by dropping wants that + resolve to a branch *manifest* (`filter_manifest_wants`); sealed *member* + objects are not recognized by that helper, and the existing single-object + `GetObject` verb (and `LocalTransport::get_object`) returns any stored + object by id. Reusing the manifest filter would therefore **not** protect + member ids. `GetObjects` (and `GetObject`, which has the same exposure + today) need a member-level authorization decision before 37a: either the + server recognizes sealed object kinds and answers "missing" for callers + outside the recipient set, or the policy is explicitly relaxed because + sealed members are ciphertext under a per-branch KEK. That choice, applied + uniformly to both verbs and to the default/local `get_objects` + implementation, is the first open question below and gates 37a. +- Protected-path ciphertext is returned as-is, exactly as it already is in + every pack: confidentiality rests on encryption, not on withholding. +- Caps: ids per request, aggregate bytes per request (reusing + `--max-pack-size`), and the existing per-object `MAX_OBJECT_SIZE`. + Over-limit requests fail with `EC_TOO_LARGE`; the client splits and retries. + +Whether to additionally require that a requested id be *reachable from a +visible ref* (a walk, so not O(request)) is part of open question 1 below. + +## Consequences + +- Partial clones of a large repo become usable for the whole day-to-day + workflow, not just read-and-commit within the filter; phase 37b retires the + P27 refusals one operation at a time, each with a test that the old + `PartialCloneUnsupported` is gone. +- The network is entered at a handful of named call sites, which keeps + ADR-0037's "fail loudly, no surprise latency" grain while delivering the + lazy-fetch UX where the user opted in. +- Every received object is verified against its requested id before ingest, so + a malicious or buggy remote can at worst return `missing`, never wrong + content. No new dependency. Core invariants (content addressing, `Arc<[u8]>` + blobs, mode-scoped disk invariant) are untouched; `core` learns nothing. +- Fetch granularity is per tree level; a pathological deep path costs one + round trip per level. Mitigation is batching and, later, a server-side + "subtree closure to depth N" hint — only if measurement shows it matters. +- `PROTOCOL_VERSION` 5. THREAT-MODEL.md gets a by-id-read section. + +## Open questions + +1. **Member-level authorization and reachability on by-id reads (gates + 37a).** Choose between kind-based refusal of sealed members, a + visible-ref reachability check, or an explicit relaxation; apply it to + `GetObject` too. O(request) is the point of the verb; + a reachability check undoes it. Candidate: cache a per-connection + "visible set" from `ListRefs` tips and check membership lazily via the + tree the id was named in. Decide with a threat-model pass before 37a lands. +2. **Does `fill_gaps` widen `.sc/promisor`'s filter?** Leaning no — the filter + stays the *initial* scope, and presence is tracked by the store — but this + interacts with `sc verify`'s `partial: N gaps` line and `sc backfill + --all`'s "convert to full clone" semantics. +3. **Merge on a partial clone** needs base/ours/theirs trees for only the + *conflicting* paths. Whether the plan can be computed from tree ids alone + (identical ids ⇒ no fetch) or needs a server-assisted diff is a 37b spike. +4. **Bucket remotes.** `sc+wal://` stores objects inside packs; by-id reads need + an object→pack index. Out of scope; partial clone from buckets stays refused. + +## Alternatives considered + +- **Transparent lazy-fetch inside `Store::get`** (git's promisor default). + Rejected again for the ADR-0037 reasons; the seam in decision 3 delivers the + same UX without the blast radius. +- **Keep `sc backfill` only.** Safe, but leaves merge/rebase/ws/work/export + unusable on partial clones, which is exactly where agents and CI want them. +- **Server-side history ops (Scion's blame/merge RPCs) first.** More powerful + — a client with no history at all — but a much larger surface (new wire + verbs per op, server CPU, authorization per query). Better built on top of + by-id fetch than instead of it; tracked in ROADMAP → Deferred. +- **Adopt Scion's changeset model.** Changes every object id; breaks the + content-addressing invariant for no gain at our scale. + +## Phasing + +- **37a** — gated on open question 1. `GetObjects` verb, + `Transport::get_objects`, `Repo::fill_gaps`, `--lazy` (the + `.sc/promisor-lazy` marker)/`SC_OFFLINE`, entry points: sparse widen + + switch, and the new `sc cat :`. Demo: `demo/run_lazy_fetch_demo.sh` counts objects fetched and + proves `Store::get` stays offline. +- **37b** — lift the P27 refusals (merge, replay, ws, work, export), one + operation per task. diff --git a/docs/adr/README.md b/docs/adr/README.md index df379b4a..4a1de3bf 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,6 +56,8 @@ supersedes the old one and update the **Status** line of the superseded record. | [0043](0043-randomized-protected-encryption.md) | Randomized protected-path encryption (dual-read, randomized-write) | Accepted | 33 | | [0044](0044-per-branch-access-control.md) | Per-branch access control with opaque private branches | Accepted | 34 | | [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 | +| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | +| [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Proposed | 37 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, diff --git a/docs/research/scion-evaluation.md b/docs/research/scion-evaluation.md new file mode 100644 index 00000000..f886710a --- /dev/null +++ b/docs/research/scion-evaluation.md @@ -0,0 +1,65 @@ +# Scion evaluation — what should src-control take from Anthropic's source-control redesign? + +Research date: 2026-10-16 +Source: + +## Recommendation (summary) + +Scion is Anthropic's server-side source control for an internal monorepo +(30k+ merges/day, ~100 GB history, ~25M CI checkouts/day). It is **not** a +git replacement and, by its authors' own account, is too bespoke to package. +It is useful to us as a **design source only**, in the same way walgit was +(`docs/research/walgit-evaluation.md`). + +- **Take: want-by-id fetch (the `scn` negotiation).** The client names the + trees/blobs it wants and never describes what it has; requests target + immutable objects, so any cache can answer and transfers resume. This is the + mechanism that retires our "transparent lazy-fetch" Deferred item and the + P27 refusals on partial clones. → ADR-0047 (Proposed). +- **Take later: derived data.** Blame, path history, commit-graph skip lists + and copy-detection built by one generic recursive mechanism over changesets, + stored separately and rebuildable. We have none of these. Needs its own ADR + after ADR-0047; recorded in ROADMAP → Deferred. +- **Take later: server-side ops as an API + ref-CAS commits.** Query + history/diff/blame/merge on the server so agents and CI need no local repo; + commit by compare-and-swapping a ref rather than pushing. Recorded in + ROADMAP → Deferred. +- **Skip: changeset data model.** A changeset lists files changed relative to + its first parent. Adopting it changes every object id — a format break + against the content-addressing invariant for no benefit at our scale. +- **Skip: tiered daemon/cache/server and the no-merge-queue workflow.** These + matter at monorepo scale; our wedge is parallel agents and secrets. + +## Where we already line up + +| Scion | src-control today | +|---|---| +| Immutable content-addressed objects, no packfile maintenance | Content-addressed (BLAKE3); packs exist (P8) but P36 bucket remotes use immutable packs + CAS'd manifest | +| Sparse/virtual checkouts | P1 in-RAM worktrees, P24 sparse — but bolted on, not protocol-native | +| Refs as compare-and-swap | P36 manifest CAS; `UpdateRef { expected_old }` on the wire | +| Per-directory permissions | P7/P34 — ours are cryptographic, Scion's are IAM | +| Disposable server instances over shared storage | P36 `sc serve --store` | + +## Why partial clone is the right first target + +Scion's article names the failure we hit in P27: "history-dependent operations +like diff and blame became fragile on partial clones." ADR-0037 refuses merge, +cherry-pick/rebase, `ws fork`/`harvest`, `work`, `export` and `sparse disable` +on a partial clone, and rejected transparent lazy-fetch because it "threads +network I/O into the deepest read path (`store.get`)". Scion's answer is not +to hide the network in `get`; it is to make *what to fetch* a cheap, explicit, +by-id request the client plans in advance. That keeps ADR-0037's +fail-loudly grain while removing the refusals. + +## Caveats worth carrying into the design + +- Scion trusts an IAM layer to authorize by-id reads. We have no such layer; + see ADR-0047 "Authorization of by-id reads". +- Round trips: a recursive tree walk is one request per depth level, not per + object. Scion hides this behind a >99% hit-rate cache tier; we have none, so + ADR-0047 batches per level and measures. +- The article reports that Claude-written code sometimes violated core + constraints (scanning the log instead of using the designated API). The same + discipline applies to us: the seam in ADR-0047 is the *only* place the + network may be entered, and a test pins that `Store::get` stays + network-free.