Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,8 +548,12 @@ across every phase.

## Active

**None.** Phase 35 completed the first native desktop vertical slice; the next
horizon is TBD.
**P37 — Want-by-id fetch and gap fill (proposed, not started).** Batched
`GetObjects` verb + an explicit `fill_gaps` seam so partial clones can run
merge/rebase/ws/work/export without `Store::get` ever touching the network;
design in [ADR-0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md), prior art
in `docs/research/scion-evaluation.md`. Phases: 37a (verb + read-only entry
points), 37b (lift the P27 refusals).

## Completed phases (usability-first ordering)

Expand Down Expand Up @@ -739,13 +743,18 @@ scale-&-reach horizon):
origin automatically on any gap, matching git's promisor default. Real
scope/risk (offline breakage, surprise latency, fetch-during-gc), so it
stays deferred, not accidentally missing.
**Now scoped as P37 (ADR-0047, Proposed):** an opt-in explicit `fill_gaps`
seam over a batched by-id `GetObjects` verb, keeping `Store::get`
network-free.
Comment on lines +746 to +748

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire the duplicate deferred scope now assigned to P37

The immediately following Deferred item still describes gap-tolerant merge/rebase/ws-harvest/work as a future follow-on, even though the new Active section and ADR assign those same operations to phase 37b. Because ROADMAP is the phase and deferred-work source of truth, leaving both entries gives conflicting status and ownership; mark that deferred item as scoped to 37b or remove it alongside this update.

AGENTS.md reference: AGENTS.md:L11-L13

Useful? React with 👍 / 👎.

- **Per-case gap-tolerant merge/rebase/ws-harvest/sc-work on a partial
clone (P27).** These are refused entirely today
(`Error::PartialCloneUnsupported`) rather than gap-tolerantly
reimplemented — a deliberate MVP coarsening, not a technical ceiling.
Threading gap-tolerance through `three_way`/replay/`ws::harvest`
individually is the follow-on if the "backfill to a full clone first"
workaround proves too coarse in practice.
**Now scoped as P37b (ADR-0047, Proposed):** lifted one operation at a
time on top of the P37a `fill_gaps` seam.
- **Blob-size/object-count clone filters (P27).** `sc clone --filter` is
prefix-only, matching the sparse path-boundary machinery already built;
git's `--filter=blob:limit` size-based filtering is a separate axis,
Expand Down Expand Up @@ -944,6 +953,15 @@ scale-&-reach horizon):
the manifest's compare-and-swap; there is no lease/TTL primitive for
operations that need to hold exclusive intent across more than one bucket
round-trip (e.g. a long-running compaction). Deferred.
- **Derived-data layer (Scion-inspired, after P37).** Blame, path history,
commit-graph skip lists and copy-detection computed by one generic
recursive mechanism over snapshots, stored outside the CAS and rebuildable
(so no id/format change). Would speed merge-base/ancestry queries and feed
the desktop app. Needs its own ADR; see `docs/research/scion-evaluation.md`.
- **Server-side history ops + ref-CAS commits (Scion-inspired, after P37).**
Server RPCs for log/diff/blame/merge so agents and CI need no local repo,
and a commit-by-ref-CAS path that skips push. Builds on P29 auth, P31
limits, and ADR-0047's by-id reads; a further wire bump.
- **Partial clone from bucket remotes (P36a follow-on).** `sc clone
--filter` against `sc+wal://`/`sc+s3://` is refused outright
(`BucketTransport` has no per-prefix negotiation); teaching the WAL format
Expand Down
181 changes: 181 additions & 0 deletions docs/adr/0047-want-by-id-fetch-and-gap-fill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
# ADR-0047: Want-by-id fetch and explicit gap fill for partial clones

- **Status:** Proposed
- **Date:** 2026-10-16
- **Phase:** 37
- **Builds on:** ADR-0037 (partial clone), ADR-0035 (streaming transfer), ADR-0039/0041 (size and listener limits), ADR-0040 (HTTP access control), ADR-0044 (private branches)
- **Research:** `docs/research/scion-evaluation.md`

## Context

A partial clone (ADR-0037) fetches only in-filter objects and then refuses
every operation that needs the rest: merge, cherry-pick/rebase, `ws
fork`/`harvest`, `work`, `export`, and `sparse disable`. ADR-0037 rejected
*transparent* lazy-fetch because it would put network I/O inside `Store::get`,
which gc, verify, materialize, merge and export all call — offline breakage,
surprise latency, fetch-during-gc. It left `sc backfill <prefix…>` as the
manual escape hatch and recorded lazy-fetch as a deferred follow-on.

Anthropic's Scion write-up (see the research note) names the same failure on
git partial clones and fixes it differently: the client never describes what
it *has*. It asks for specific immutable trees and blobs *by id*, starting at
the root tree and descending, and any cache can answer because the objects are
immutable. The network is not hidden in a read path; "what do I need next" is
an explicit, cheap, batchable request.

We already have the pieces: `Request::GetObject(id)` exists on the wire,
objects are self-authenticating (BLAKE3 of canonical encoding), the P25 chunk
stream bounds RAM, and P27 gives us a gap-tolerant store plus a durable
`.sc/promisor` marker.

## Decision

Add a **batched by-id object request** and an **explicit gap-fill seam** at
operation entry points. `Store::get` stays network-free, permanently.

1. **Wire: `Request::GetObjects { ids: Vec<ObjectId> }`.** The reply is the
requested objects as a normal P25 pack stream (`ST_PACK_CHUNK`/
`ST_PACK_END`). No `haves`, no reachability walk: server cost is
O(requested objects), independent of history size. Ids the server lacks are
reported in a trailing missing-id list rather than failing the whole batch.
`PROTOCOL_VERSION` 4 → 5 (additive verb; a v4 peer is rejected at
handshake, same discipline as P25/P27/P34).
2. **Transport: `Transport::get_objects(&self, ids, out)`** with a default
implementation that loops `get_object`, so local-path, ssh, http(s) and
test transports compose immediately; stdio/http override it to use the one
round trip. Bucket remotes keep refusing partial clone (ADR-0046) and are
out of scope.
3. **Client seam: `Repo::fill_gaps(plan)`**, the only code path allowed to
dial the promisor remote after clone. An operation that needs more than the
filter *plans first*: it walks the trees it already holds, collects the
absent child ids (the "frontier"), batches them into one `GetObjects`,
verifies every returned object against the requested id and
`MAX_OBJECT_SIZE`, and ingests each depth's batch verify-before-write (the
P25 two-pass: every record is checked before any is written). This is
integrity, **not** transactionality: objects are content-addressed and
immutable, so a batch that lands before a later depth fails is simply
retained, and a retry recomputes the frontier from *current presence* and
resumes where it stopped. An id the server reports missing that the plan
requires is terminal (the plan fails with the gap error; nothing dependent
runs). The dependent operation mutates only after the whole plan is
satisfied. Fetched objects are ordinary store objects, so GC reachability
treats them like any other; `fill_gaps` holds the repo lock across a plan
so a concurrent `sc gc` cannot prune a half-filled frontier. Cost: one
round trip per directory depth, not per object.
4. **Opt-in, not default.** Laziness is recorded in a **separate marker file,
`.sc/promisor-lazy`** (local, uncommitted), set by `sc clone --filter
<prefix…> --lazy` or `sc backfill --lazy`. It is deliberately *not* a field
in `.sc/promisor`: that file's format is `origin <url>` followed by one
path prefix per line (`promisor::load`), and a prefix such as `lazy = true`
is a legal path, so a new field would silently reinterpret a legacy clone's
filter. Without the marker the P27 behavior is byte-identical: the gap
error with the `sc backfill` hint. With it, the entry points below call
`fill_gaps` before proceeding. `SC_OFFLINE=1` (and `--offline`) force the
gap error even when the marker is present, so offline use stays
predictable.
5. **Entry points that may call `fill_gaps`** (an exhaustive allow-list,
pinned by a test that greps for the call). In 37a, only operations that
exist today and can request an out-of-filter path: `sparse set`/`disable`
widening and the `switch`/materialize that follows it; plus one **new**
read surface scoped into 37a, `sc cat <ref>:<path>` (print one path at a
ref). Today's `sc diff` compares the working tree to HEAD only and `sc log`
has no patch output, so neither is an entry point until a later phase gives
them revision/path arguments. In 37b: merge, cherry-pick/rebase, `ws
fork`/`harvest`, `work` and `export`. Everything else, including gc,
verify and the desktop read model (ADR-0045), never touches the network.
6. **Gap semantics change from "out of filter" to "absent".** `sc verify`
currently counts gaps by filter membership. Once fill-gap can fetch
out-of-filter objects, a fetched object is simply present and the filter
remains only the *initial* fetch scope. `reachable_objects_filtered`'s
`gaps` set continues to be computed against presence, and gc must keep
everything reachable regardless of how it arrived.

### Authorization of by-id reads

A hash is not a capability, and Scion delegates this to an IAM layer we don't
have. Rules for `GetObjects`, in order of preference:

- It is **read-scope**: a `--read-only` token (ADR-0040) may call it; no
write verb is implied.
- It **must not be a bypass of private-branch exclusion** (ADR-0044). Today a
filtered `GetPack` excludes private branches only by dropping wants that
resolve to a branch *manifest* (`filter_manifest_wants`); sealed *member*
objects are not recognized by that helper, and the existing single-object
`GetObject` verb (and `LocalTransport::get_object`) returns any stored
object by id. Reusing the manifest filter would therefore **not** protect
member ids. `GetObjects` (and `GetObject`, which has the same exposure
today) need a member-level authorization decision before 37a: either the
server recognizes sealed object kinds and answers "missing" for callers
outside the recipient set, or the policy is explicitly relaxed because
sealed members are ciphertext under a per-branch KEK. That choice, applied
uniformly to both verbs and to the default/local `get_objects`
implementation, is the first open question below and gates 37a.
- Protected-path ciphertext is returned as-is, exactly as it already is in
every pack: confidentiality rests on encryption, not on withholding.
- Caps: ids per request, aggregate bytes per request (reusing
`--max-pack-size`), and the existing per-object `MAX_OBJECT_SIZE`.
Over-limit requests fail with `EC_TOO_LARGE`; the client splits and retries.

Whether to additionally require that a requested id be *reachable from a
visible ref* (a walk, so not O(request)) is part of open question 1 below.

## Consequences

- Partial clones of a large repo become usable for the whole day-to-day
workflow, not just read-and-commit within the filter; phase 37b retires the
P27 refusals one operation at a time, each with a test that the old
`PartialCloneUnsupported` is gone.
- The network is entered at a handful of named call sites, which keeps
ADR-0037's "fail loudly, no surprise latency" grain while delivering the
lazy-fetch UX where the user opted in.
- Every received object is verified against its requested id before ingest, so
a malicious or buggy remote can at worst return `missing`, never wrong
content. No new dependency. Core invariants (content addressing, `Arc<[u8]>`
blobs, mode-scoped disk invariant) are untouched; `core` learns nothing.
- Fetch granularity is per tree level; a pathological deep path costs one
round trip per level. Mitigation is batching and, later, a server-side
"subtree closure to depth N" hint — only if measurement shows it matters.
- `PROTOCOL_VERSION` 5. THREAT-MODEL.md gets a by-id-read section.

## Open questions

1. **Member-level authorization and reachability on by-id reads (gates
37a).** Choose between kind-based refusal of sealed members, a
visible-ref reachability check, or an explicit relaxation; apply it to
`GetObject` too. O(request) is the point of the verb;
a reachability check undoes it. Candidate: cache a per-connection
"visible set" from `ListRefs` tips and check membership lazily via the
tree the id was named in. Decide with a threat-model pass before 37a lands.
2. **Does `fill_gaps` widen `.sc/promisor`'s filter?** Leaning no — the filter
stays the *initial* scope, and presence is tracked by the store — but this
interacts with `sc verify`'s `partial: N gaps` line and `sc backfill
--all`'s "convert to full clone" semantics.
3. **Merge on a partial clone** needs base/ours/theirs trees for only the
*conflicting* paths. Whether the plan can be computed from tree ids alone
(identical ids ⇒ no fetch) or needs a server-assisted diff is a 37b spike.
4. **Bucket remotes.** `sc+wal://` stores objects inside packs; by-id reads need
an object→pack index. Out of scope; partial clone from buckets stays refused.

## Alternatives considered

- **Transparent lazy-fetch inside `Store::get`** (git's promisor default).
Rejected again for the ADR-0037 reasons; the seam in decision 3 delivers the
same UX without the blast radius.
- **Keep `sc backfill` only.** Safe, but leaves merge/rebase/ws/work/export
unusable on partial clones, which is exactly where agents and CI want them.
- **Server-side history ops (Scion's blame/merge RPCs) first.** More powerful
— a client with no history at all — but a much larger surface (new wire
verbs per op, server CPU, authorization per query). Better built on top of
by-id fetch than instead of it; tracked in ROADMAP → Deferred.
- **Adopt Scion's changeset model.** Changes every object id; breaks the
content-addressing invariant for no gain at our scale.

## Phasing

- **37a** — gated on open question 1. `GetObjects` verb,
`Transport::get_objects`, `Repo::fill_gaps`, `--lazy` (the
`.sc/promisor-lazy` marker)/`SC_OFFLINE`, entry points: sparse widen +
switch, and the new `sc cat <ref>:<path>`. Demo: `demo/run_lazy_fetch_demo.sh` counts objects fetched and
proves `Store::get` stays offline.
- **37b** — lift the P27 refusals (merge, replay, ws, work, export), one
operation per task.
2 changes: 2 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ supersedes the old one and update the **Status** line of the superseded record.
| [0043](0043-randomized-protected-encryption.md) | Randomized protected-path encryption (dual-read, randomized-write) | Accepted | 33 |
| [0044](0044-per-branch-access-control.md) | Per-branch access control with opaque private branches | Accepted | 34 |
| [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 |
| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 |
| [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Proposed | 37 |

See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are
assigned in creation order, so 0017 — the secret scanner — sequences as phase P5,
Expand Down
65 changes: 65 additions & 0 deletions docs/research/scion-evaluation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Scion evaluation — what should src-control take from Anthropic's source-control redesign?

Research date: 2026-10-16
Source: <https://www.anthropic.com/engineering/redesigning-source-control>

## Recommendation (summary)

Scion is Anthropic's server-side source control for an internal monorepo
(30k+ merges/day, ~100 GB history, ~25M CI checkouts/day). It is **not** a
git replacement and, by its authors' own account, is too bespoke to package.
It is useful to us as a **design source only**, in the same way walgit was
(`docs/research/walgit-evaluation.md`).

- **Take: want-by-id fetch (the `scn` negotiation).** The client names the
trees/blobs it wants and never describes what it has; requests target
immutable objects, so any cache can answer and transfers resume. This is the
mechanism that retires our "transparent lazy-fetch" Deferred item and the
P27 refusals on partial clones. → ADR-0047 (Proposed).
- **Take later: derived data.** Blame, path history, commit-graph skip lists
and copy-detection built by one generic recursive mechanism over changesets,
stored separately and rebuildable. We have none of these. Needs its own ADR
after ADR-0047; recorded in ROADMAP → Deferred.
- **Take later: server-side ops as an API + ref-CAS commits.** Query
history/diff/blame/merge on the server so agents and CI need no local repo;
commit by compare-and-swapping a ref rather than pushing. Recorded in
ROADMAP → Deferred.
- **Skip: changeset data model.** A changeset lists files changed relative to
its first parent. Adopting it changes every object id — a format break
against the content-addressing invariant for no benefit at our scale.
- **Skip: tiered daemon/cache/server and the no-merge-queue workflow.** These
matter at monorepo scale; our wedge is parallel agents and secrets.

## Where we already line up

| Scion | src-control today |
|---|---|
| Immutable content-addressed objects, no packfile maintenance | Content-addressed (BLAKE3); packs exist (P8) but P36 bucket remotes use immutable packs + CAS'd manifest |
| Sparse/virtual checkouts | P1 in-RAM worktrees, P24 sparse — but bolted on, not protocol-native |
| Refs as compare-and-swap | P36 manifest CAS; `UpdateRef { expected_old }` on the wire |
| Per-directory permissions | P7/P34 — ours are cryptographic, Scion's are IAM |
| Disposable server instances over shared storage | P36 `sc serve --store` |

## Why partial clone is the right first target

Scion's article names the failure we hit in P27: "history-dependent operations
like diff and blame became fragile on partial clones." ADR-0037 refuses merge,
cherry-pick/rebase, `ws fork`/`harvest`, `work`, `export` and `sparse disable`
on a partial clone, and rejected transparent lazy-fetch because it "threads
network I/O into the deepest read path (`store.get`)". Scion's answer is not
to hide the network in `get`; it is to make *what to fetch* a cheap, explicit,
by-id request the client plans in advance. That keeps ADR-0037's
fail-loudly grain while removing the refusals.

## Caveats worth carrying into the design

- Scion trusts an IAM layer to authorize by-id reads. We have no such layer;
see ADR-0047 "Authorization of by-id reads".
- Round trips: a recursive tree walk is one request per depth level, not per
object. Scion hides this behind a >99% hit-rate cache tier; we have none, so
ADR-0047 batches per level and measures.
- The article reports that Claude-written code sometimes violated core
constraints (scanning the log instead of using the designated API). The same
discipline applies to us: the seam in ADR-0047 is the *only* place the
network may be entered, and a test pins that `Store::get` stays
network-free.
Loading