diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index feaa97bd..7d1cd135 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -3,8 +3,8 @@ A next-generation version control system built around a snapshot-and-tag model (Jujutsu-inspired) with per-file permissions, native committed secrets, and in-memory clones. This document covers the architecture as built — the two -founding wedges and the full collaborative VCS grown around them across 30 -vertical-slice phases. The rationale behind each major decision is recorded as an +founding wedges and the full collaborative VCS grown around them across 36 +vertical-slice phases plus the P37 partial-clone series (37a-37d). The rationale behind each major decision is recorded as an ADR in [`docs/adr/`](docs/adr/), and a compact per-phase appendix closes this doc. ## Thesis and MVP scope @@ -33,10 +33,12 @@ Everything in Phase 1 was designed so Phase 2 dropped in without re-architecting secrets are just a special object kind in the same content-addressed store. That composability held across every phase since. **All three long-term-thesis pillars now ship** — in-RAM worktrees (P1), committed secrets (P2), and per-file -permissions (P7, encrypted paths) — atop a full collaborative VCS: three-way -merge and history editing, packfiles + GC, ssh:// / sc+http:// / Git network -transports, sparse and partial clone, signed commits, and intent-deep provenance -via sealed agent-session transcripts. The phase-by-phase build log is in +permissions (P7, encrypted paths; per-branch in P34) — atop a full collaborative +VCS: three-way merge and history editing, packfiles + GC, ssh:// / sc+http(s):// / +bucket (sc+wal://, sc+s3://) / Git network transports, sparse and partial clone +(with lazy fetch, merge and agent sessions on a partial clone), signed commits, +intent-deep provenance via sealed agent-session transcripts, and a read-only +native desktop browser. The phase-by-phase build log is in [`ROADMAP.md`](ROADMAP.md) and [`CLAUDE.md`](CLAUDE.md); the sections below cover the architecture, with a compact per-phase appendix at the end. @@ -61,9 +63,12 @@ to reason about — which is exactly the property Phase 1 is meant to demonstrat ## System overview -The codebase is a Cargo workspace of eight Rust crates with a strict dependency -direction (top-level adapters `{cli, desktop} → repo → {vfs, gitio, crypto} -→ core`, with the separate leaf edge `repo → tlsio`): +The codebase is a Cargo workspace of eight library/binary crates plus the +desktop app, with a strict dependency direction: the top-level adapters +`{cli, desktop}` sit on `repo`; `repo → {vfs, crypto} → core`, plus the separate +leaf edges `repo → tlsio` and `repo → objio` (`tlsio` and `objio` depend on no +other workspace crate, not even `core`). `gitio → core` is linked by `cli`, never +by `repo`: ``` src-control/ @@ -75,20 +80,28 @@ src-control/ │ │ plus a system-git mirror bridge for hosted Git (the only crate linking gix) │ ├── crypto/ envelope encryption (secrets + protected paths) AND Ed25519 signing + │ │ v2 dual-key identities (scl-crypto; the only crate linking RustCrypto) +│ ├── tlsio/ rustls/rcgen wrappers for sc+https:// (P32); a leaf, the only crate +│ │ linking rustls +│ ├── objio/ `Bucket` trait + directory and S3 backends for sc+wal:// / sc+s3:// (P36); +│ │ a leaf, the only crate linking the S3 SDK │ ├── repo/ durable .sc/ repo: refs/branches/working tree, three-way merge, history │ │ editing (cherry-pick/rebase/amend/undo), remotes + wire transport (ssh://, -│ │ sc+http://), protection + secret lifecycle, GC, sparse & partial clone, -│ │ signatures & session transcripts, agent workspaces -│ └── cli/ `sc` binary — the full ~42-command surface (see CLAUDE.md) -├── apps/desktop/ Tauri v2 top-level adapter + bundled TypeScript renderer; -│ native read model only, with no UI dependency in domain crates +│ │ sc+http(s)://, bucket WAL), protection + secret lifecycle, GC, sparse & +│ │ partial clone, signatures & session transcripts, agent workspaces +│ └── cli/ `sc` binary — the full ~43-command surface (`sc --help`; the annotated +│ reference is the sc-cli-reference skill) +├── apps/desktop/ Tauri v2 top-level adapter (`scl-desktop`) + bundled React/TypeScript +│ renderer; native read model only, with no UI dependency in domain crates +├── demo/ 27 end-to-end proof scripts (`run_*.sh`) +├── fuzz/ cargo-fuzz target for the wire decoder (`wire_decode`) └── ARCHITECTURE.md ``` `core` knows nothing about Git, worktrees, or cryptography. `gitio` is the only crate that links `gix`, keeping the Git dependency quarantined behind one boundary. `crypto` is the only crate that links the RustCrypto stack, keeping -the cryptographic dependency quarantined behind another. `repo` owns the `.sc/` +the cryptographic dependency quarantined behind another; `tlsio` does the same +for rustls and `objio` for the S3 SDK. `repo` owns the `.sc/` on-disk layout and all repo orchestration; it is deliberately **Git-agnostic** (it never links `gitio`) — `cli` links both `repo` and `gitio` and passes imported snapshots down. This matters because the long-term plan is to own the @@ -205,8 +218,9 @@ first-class remote, and Phase 18 reaches *hosted* Git (GitHub over https/ssh) through a spawned-system-`git` mirror bridge (`gix` can fetch but not push, so a lazily-built bare `mirror.git` beside the marks map is the transport). All of it keeps `gix` quarantined in `gitio`. This is distinct from sc's **own** network -transport: `repo` speaks a framed wire protocol over ssh:// (P12) and sc+http:// -(P26) that mirrors its `Transport` trait — no `gix` involved (see the appendix). +transport: `repo` speaks a framed wire protocol over ssh:// (P12), sc+http:// +(P26) and sc+https:// (P32) that mirrors its `Transport` trait, and treats a +bucket as a remote (P36) — no `gix` involved (see the appendix). ## Cryptography and key management @@ -228,7 +242,10 @@ a `BTreeMap` so canonical encoding is insertion-order-independent), kept separat from the file tree so `checkout` never materializes them. An authorized context decrypts in memory and injects the value into a child process environment; `sc secret-demo` proves the authorize/deny/grant flow end-to-end with the same -zero-residue teardown as Phase 1. See ADR-0010. +zero-residue teardown as Phase 1. `sc run` injects through the child's +environment by default; `sc run --secrets-fd N` (ADR-0052, unix) hands the +values over a pipe on descriptor N instead, which keeps them out of env dumps +but is still not isolation. See ADR-0010. The consequence is that a clone in an **unauthorized** context — one whose private key is not among the recipients — receives the secret object intact but @@ -318,16 +335,31 @@ phases but the shape is stable: so `sc undo`/redo can invert it (P14). Local-only, like a reflog. - `signatures` — snapshot→signature-object index (P22); `transcripts` — snapshot→transcript-object index, one-to-many (P30). Both gc-rooted, local-only. + `transcript-drops` — the derived set of dropped transcripts (`sc transcript + drop`), rebuilt from the store on clone. - `sparse` — the local, uncommitted sparse-checkout prefix spec (P24); `promisor` — a partial clone's fetch-filter + origin (P27). -- `serve-tokens.toml` — bearer-token access control for `sc serve --http` (P29). +- `serve-tokens.toml` — bearer-token access control for `sc serve --http` (P29); + tokens may carry an expiry and be rotated in place (`sc serve token add + --expires` / `rotate`). `serve-tls/` — the auto-minted `sc+https://` server + identity (P32). +- `promisor-lazy` — empty marker: this partial clone may fill gaps by object id + (P37a); `sparse-prev` — the previous sparse spec while a `sparse set` is in + flight (P37b). +- `local-key`, `protected-cache` (and `ws/cache-`) — the P33 keyed stat cache + for protected paths; never committed, never transferred. - `tmp/` — guard-cleaned scratch for streaming pack transfer (P25). - `ws/` — durable agent-session workspaces: `session.toml` manifest + per-agent checkouts (P20). `git-remotes//` — the `git_oid ↔ sc_id` marks map and a lazily-built bare `mirror.git` for network Git (P10/P18). - Transient in-progress state (removed on completion/abort): `MERGE_HEAD`, - `PICK_HEAD`, `REBASE_STATE`, and their `*_DECIDED_ROOT` records — an - interrupted merge/pick/rebase resumes or aborts cleanly from these. + `PICK_HEAD`, `REBASE_STATE`, and their `*_DECIDED_ROOT` records, plus + `CONFLICT_PLAINTEXT` / `CONFLICT_SIDECARS` listing what a stopped operation + wrote into the working tree — an interrupted merge/pick/rebase resumes or + aborts cleanly from these. +- The `sc+https://` client's pin file is user-level, not part of `.sc/` + (`$SC_HTTPS_KNOWN_HOSTS`, else `$XDG_CONFIG_HOME/sc/known_hosts`, else + `~/.config/sc/known_hosts`; managed with `sc tls`). ### Git-like working tree @@ -398,11 +430,10 @@ Phase 8 adds three tightly coupled capabilities to the persistent store: transport read path resolves packed, sharded, and compressed objects from the remote store. This replaces the prior object-at-a-time transfer. -Remaining follow-ons: network Git remotes, HTTP transport, streaming (>4 GiB) -frames, bulk re-wrap, multiple escrow keys, interactive workspace sessions -and auto-merge of clean workspace results (merge shipped as Phase 4; -break-glass escrow shipped as Phase 11; ssh-native network transport shipped -as Phase 12; agent workspaces shipped as Phase 13). +Follow-ons named when this phase shipped have since landed as their own phases: +network Git remotes (P18), the HTTP transport (P26), streaming frames (P25), +bulk re-wrap and multiple escrow keys (P17), and interactive workspace sessions +with auto-merge (P20). ## Phase 9 — Git export (built) @@ -877,13 +908,13 @@ explicitly and offline — no lazy-fetch from inside a read path. Building a *new* commit on a partial clone needed real machinery beyond carry-by-id: `worktree::graft_out_of_sparse` splices the tip's untouched out-of-filter subtrees back into a freshly built root by id, never reading -their content, so push's filtered reachability walk still sends only the -client's new in-filter objects. `commit` refuses (`Error::GappedPathContent`) +their content, so push's filtered reachability walk (until P37b-i made it +presence-based) still sent only the client's new in-filter objects. `commit` refuses (`Error::GappedPathContent`) rather than silently drop content under an unfetched subtree. `sc backfill --all` is the genuine full-clone escape hatch: fetch everything, verify the -closure, then remove `.sc/promisor`. Merge, replay, `sc ws fork/harvest`, and -`sc work` are all refused outright on a partial clone as a deliberate MVP -coarsening rather than per-case gap tolerance. See ADR-0037. +closure, then remove `.sc/promisor`. P27 refused merge, replay, `sc ws +fork/harvest`, and `sc work` outright on a partial clone as a deliberate MVP +coarsening; P37b-P37d lifted those refusals (see below). See ADR-0037. ## Phase 28 — security hardening sweep (built) @@ -900,16 +931,20 @@ ADR-0039. ## Phase 29 — sc+http access control (built) -`sc serve --http` gains three composed access-control gates at the -connection opening, closing P26's stated "unauthenticated" boundary: a -read-only mode that rejects any write-shaped request before it reaches -`wire::serve`, a fail-closed bind check, and bearer-token authentication -against a `.sc/serve-tokens.toml` allowlist. `EC_READONLY` (or equivalent -server flag) forces the read-only gate independent of tokens, so a -public-read mirror needs no token management at all. Each gate runs before -the P25/P26 wire handshake begins, matching P26's own status-line-before- -handshake discipline, so a rejected connection never reaches -`LocalTransport`. See ADR-0040. +`sc serve --http` gains three composed access-control gates, closing P26's +stated "unauthenticated" boundary: a fail-closed bind check and bearer-token +authentication against a `.sc/serve-tokens.toml` allowlist, both at the +connection opening, and a read-only mode that rejects each mutating request +inside `wire::serve_with_policy` before any store write. `--read-only` forces the +read-only gate independent of tokens (a rejected write gets the typed +`EC_READONLY` wire error), so a public-read mirror needs no token management at +all. The bind and token gates +run before the P25/P26 wire handshake begins, matching P26's own +status-line-before-handshake discipline, so a connection they reject never +reaches `LocalTransport`; the read-only gate is enforced per request after the +handshake. Tokens later gained an optional expiry (`sc serve token add +--expires`; an expired token gets the same 401 as an unknown one) and `sc serve +token rotate` (ADR-0040 addendum). See ADR-0040. ## Phase 30 — agent session transcripts (built) @@ -933,7 +968,28 @@ runs, so a signed transcript survives exactly as long as the transcript itself does. `sc export --to ` drops transcripts (no Git-native form exists) and reports a count. A bucket remote (P36) has no index, so its `get_pack` rebuilds the same over-send from the packs the manifest chain names -(#229, no format change). See ADR-0038. +(no format change), at the cost of downloading and decoding every on-chain pack +on each fresh fetch or clone. + +`sc transcript drop [--identity ]` removes a transcript from view +without a new object tag or wire change: a drop is a **tombstone**, a +`Transcript` object with an empty ciphertext and wrap list that names the +dropped id, recorded in a local `.sc/transcript-drops` index and treated as a +grow-only set (no undrop; re-attach to restore). Reads, `sc log` markers and +the over-send filter through the drop set, and `sc gc` reclaims the body. A +tombstone travels with fetch and push only when it is signed (a push that finds +the remote already at the same branch tip sends nothing, so a drop made +afterwards propagates once a push moves the branch or the remote fetches), and +a receiving peer honors it only if the signature comes from a key in its `[signers] +trusted` list; an unsigned drop is local-only, and an untrusted tombstone stays +inert. Drop is not erasure: a peer that never receives an honored tombstone +keeps the sealed body, and a remote that received a tombstone before trusting +its signer keeps it inert (#230). `sc push --no-transcripts` leaves transcripts and their signatures out of the +pack (saving bandwidth); `sc clone|fetch --no-transcripts` cannot, because the +server still sends the full pack, so the receiver hash-checks every record and +then discards transcripts and their signatures, writing and indexing none +(saving disk and exposure, not bandwidth). It is per command, not remembered, +and has no effect on git remotes, which carry none. See ADR-0038 and its addendum. ## Phase 31 — listener resource limits (built) @@ -959,7 +1015,7 @@ Proven by `demo/run_limits_demo.sh`. See ADR-0041. A new leaf crate, `crates/tlsio` (`scl-tlsio`), is the only crate linking rustls/rcgen/`ring` — extending the dependency rule with the separate leaf edge -`repo → tlsio` while leaving `repo → {vfs, gitio, crypto} → core` and the RustCrypto quarantine in +`repo → tlsio` while leaving `repo → {vfs, crypto} → core` and the RustCrypto quarantine in `crates/crypto` untouched. Two seam functions grow a TLS wrap and nothing else changes: the client wraps the `TcpStream` before the opening write when the URL is `sc+https://`; the server wraps before the opening read when @@ -967,12 +1023,19 @@ the URL is `sc+https://`; the server wraps before the opening read when byte-for-byte unchanged; `PROTOCOL_VERSION` stays 3. Trust is accept-new TOFU, the SSH `known_hosts` shape: the client pins -`SHA-256(SPKI DER)` per host in `~/.config/sc/known_hosts` -(`SC_HTTPS_KNOWN_HOSTS` overrides; `SC_HTTPS_FINGERPRINT` pre-pins for CI; +`SHA-256(SPKI DER)` per host in the pin file (`$SC_HTTPS_KNOWN_HOSTS`, else +`$XDG_CONFIG_HOME/sc/known_hosts`, else `~/.config/sc/known_hosts`; +`SC_HTTPS_FINGERPRINT` pre-pins for CI; `SC_HTTPS_STRICT=1` refuses unknown hosts). A pin mismatch always hard-fails. The server auto-mints a self-signed identity into `.sc/serve-tls/` (key `0600`) unless PEM is supplied; `sc serve fingerprint` -prints the pin an operator distributes out-of-band. **Gate change:** a +prints the pin an operator distributes out-of-band. Follow-ons: `sc tls +list/forget/pin/path` manages the pin file (ADR-0053), and a client can opt into +CA validation instead of pinning with `SC_HTTPS_CA_FILE=` or +`SC_HTTPS_CA=system` (ADR-0054; fail-closed, no revocation checking, the pin +file is neither read nor written). Client connect, TLS handshake and opening +are each held to a 30 s inactivity bound, and bracketed IPv6 hosts parse in `sc+http(s)://` URLs. +**Gate change:** a non-loopback bind now requires `--read-only`, `--allow-public`, or (`--tls` AND ≥1 token) — P29's "tokens alone justify a public bind" is narrowed, since a token protecting only a plaintext channel was never the guarantee @@ -1065,6 +1128,45 @@ messages, paths, timestamps, and inner topology never cross IPC. The WebView has no generic dialog, filesystem, URL, process, or object-store command. See ADR-0045. +## Phase 36 — bucket WAL remotes: sc+wal:// and sc+s3:// (built) + +A remote that needs no server process. `BucketTransport` +(`crates/repo/src/bucket_transport.rs`) implements the same `Transport` trait as +the local, ssh and http remotes on top of a flat key/value bucket, structured +as a write-ahead log of **immutable, content-addressed packs** plus **one +compare-and-swapped manifest**. A push uploads a pack, appends a parent-linked +`log/` entry carrying its ref updates, then CASes the manifest's +`head_seq` forward (`Bucket::put_if_tag`). A lost CAS refreshes and retries (up +to 16 times), so pushes to different branches both land; a push whose target +branch no longer matches its expected old tip fails with `NonFastForward`, +exactly as on any other transport, and a loser's orphaned log entry is invisible +to readers because they walk `parent_seq` back from the manifest. There is no +leader, lock file or lease. The new leaf crate `objio` holds the `Bucket` trait +and its two backends (`DirBucket` for `sc+wal://`, `S3Bucket` for +`sc+s3:///`), quarantining the S3 SDK the way `gix` and +RustCrypto are quarantined. + +Reads treat the bucket as untrusted input: WAL metadata (manifest, log +entries, checkpoints, idx) is capped at a fixed 256 MiB, pack records follow the +live max object size (`scl_core::max_object_size()`), and every WAL record +decodes strictly and fails closed. P36b adds **checkpoint folding**: once the log tail passes 64 +entries a push opportunistically writes `checkpoints/` (branch tips plus a +cumulative pack list) and points the manifest at it, so a cold reader seeds from +the checkpoint and walks only the tail. P36c adds `sc serve --store `: an +`sc serve --http`/`--stdio` instance whose `.sc/` home supplies tokens, the TLS +identity and scratch space while all served content lives in the bucket, so +instances are disposable. Public content sits in the bucket as plaintext at +rest (the bucket ACL is the perimeter; sealed content stays ciphertext), and +`sc clone --filter` is refused against a bucket remote. Bucket compaction and gc +are not built (open issue #216). A bucket keeps no signature or transcript +index, so its `get_pack` rebuilds the over-send from the packs the manifest +chain names; signatures, transcripts and signed drop tombstones therefore +travel through a bucket like any other remote, but each fresh fetch or clone +downloads and decodes every on-chain pack to do it. The bucket cannot judge a +tombstone, so a receiver honors it only for a trusted signer, and until then +may see a transcript the bucket still holds. See ADR-0038's bucket addendum, +ADR-0046 and `docs/research/walgit-evaluation.md`. + ## Phase 37a — want-by-id fetch (built) A partial clone marked lazy (`.sc/promisor-lazy`) can read a path outside its @@ -1075,7 +1177,54 @@ with the objects it can serve plus the ids it will not send, with no haves and no reachability walk. `Store::get` stays network-free; the only code that dials the promisor after clone is `Repo::fetch_by_id_from` in `crates/repo/src/ fill.rs`, which verifies the reply against the request before writing. A fill -only adds objects, so push, gc and verify (filter-scoped walks) are unchanged. -Sparse widening over a filled path and the lifted P27 refusals need -presence-based walks and are phase 37b. See ADR-0047 and +only adds objects; push, gc and verify were filter-scoped walks when P37a +shipped and became presence-based in P37b-i (below). +Sparse widening over a filled path and the lifted P27 refusals needed +presence-based walks and are P37b-P37d (below). See ADR-0047 and `docs/research/scion-evaluation.md`. + +## Phase 37b — presence-based partial clones (built) + +The filter records what a clone *asked for*, not what is present: an ordinary +`sc fetch` on a partial clone sends no filter and advertises its local tips as +haves, so an object new at an in-filter path can be omitted when it equals an +out-of-filter object reachable from a have. 37b-i therefore makes push, verify +and gc walk what the store **holds** (`reachable_objects_partial`) instead of +the fetch filter; push checks every gap id against the destination before +sending, a view gap fails with `GapInView` before any write (a preflight at the +top of `materialize`, plus `undo` and `sparse set`), and a partial clone fetches +from its origin only. 37b-ii lets a lazy, online clone fill the view's closure +by id (`ensure_view`) before switch, fast-forward, undo and `sparse set`; +`sparse set` may widen past the filter; and a by-id reply is capped at +`min(--max-pack-size, 2 GiB)` (the 2 GiB ceiling is twice the largest settable +object cap; the client reads at most 4 GiB) and split when refused. `sc export` and `sparse +disable` stay refused on a partial clone. See ADR-0048. + +## Phase 37c — merge, cherry-pick and rebase on a partial clone (built) + +`merge_trees` is a tree-recursive three-way merge that carries a subtree **by +id** when the sides agree and no protection rule applies, reads only where the +sides differ or a rule intersects (a subtree with the same id on all three sides +is carried unread even under a protection rule), and on a lazy, online clone fills what it +asks for by id outside the store lock (a non-lazy clone gets the gap error with +the `sc backfill ` hint). A carried protected subtree keeps the complete +input wrap union. A conflict inside the view is marked and resolved like on a +full clone (`sc conflicts`, `sc resolve`, `merge --abort`, `sc commit`; the +out-of-view part is grafted from the decided root); a conflict outside the view, +or a result that must be sealed outside it, is refused before any ref, +merge-state or working-tree change. Cherry-pick and rebase share the engine +through `replay_commit`, and a stopped pick or rebase keeps its carried subtrees +in the decided root. Content and mode now merge as separate dimensions, so a +mode-only edit merges and a file/directory collision fails before any write on +full clones too. See ADR-0049. + +## Phase 37d — agent sessions on a partial clone (built) + +`sc ws fork/run/harvest/abandon` and `sc work` work on a partial clone, with the +host's sparse view as the agent's whole world (`sc work` no longer hands agents +a complete tree there). The view closure is filled (lazy, online) or refused with +the gap error before anything is written; a workspace commit grafts the +out-of-view subtrees from the base without reading them; and harvest lands +through the ADR-0049 merge, with any conflict falling back to a `work-` +branch as on a full clone. `sc backfill --all` remains the way to a genuine full +clone. See ADR-0050. diff --git a/CLAUDE.md b/CLAUDE.md index 8c6d20ff..c83ffbf2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ recorded as an ADR in `docs/adr/`. Keep them in sync when the design changes. Crate roles are what `ls crates/` + each `Cargo.toml` say; the rules are what matters. Strict dependency direction: top-level adapters `{cli, desktop} → repo → -{vfs, gitio, crypto} → core`, with the separate leaf edges `repo → tlsio` +{vfs, crypto} → core` (`gitio → core` is linked by `cli`, never by `repo`), with the separate leaf edges `repo → tlsio` **and `repo → objio`** (`tlsio` and `objio` are leaves — each depends on no other workspace crate, not even `core`). **`core` must never depend on Git, worktrees, or crypto.** **`gix` must stay quarantined in `gitio`** — if you diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8dec37a9..4270df19 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,24 +25,40 @@ cargo fmt --all # format (CI enforces --check) cargo clippy --workspace --all-targets -- -D warnings # lint (CI enforces this) ``` +The workspace includes the Tauri desktop crate (`apps/desktop/src-tauri`), so on +Linux `cargo test --workspace` needs the Tauri v2 system packages (WebKitGTK +and friends; the exact list is in the `Install Tauri Linux prerequisites` step +of [`.github/workflows/ci.yml`](.github/workflows/ci.yml)). If you touch the +desktop frontend, also run `npm ci`, `npm run typecheck` and `npm test` in +`apps/desktop` (see [`apps/desktop/README.md`](apps/desktop/README.md)). + The demos under `demo/` are independent end-to-end proofs and double as integration tests — if you touch a subsystem, run the relevant `demo/run_*.sh`. ## The bar for a change -CI runs `fmt --check`, `clippy -D warnings`, `cargo test --workspace`, and -`cargo doc`. Your PR must be green on all of them. In addition: +CI (`ci.yml`) runs `fmt --check`, `clippy -D warnings`, `cargo test --workspace +--all-targets` (Linux and macOS) and `cargo doc` for the Rust workspace, plus a +`desktop` job (`npm ci`, `npm audit --audit-level=high`, typecheck, test). +Your PR must be green on all of them. Dependency changes also run the RustSec +`audit` workflow, and CodeQL, Scorecard, the SBOM generation, the +`vendor-provenance` check and a nightly `wire_decode` fuzz run live in their own +workflows. In addition: 1. **Every new behavior ships with a test.** Tests live in `#[cfg(test)] mod tests` next to the code. A test that materializes to disk must clean up after itself and assert the path is gone. 2. **Respect the dependency rule and quarantines** (enforced socially, not just by - the compiler): `cli → repo → {vfs, gitio, crypto, tlsio} → core` (`tlsio` is a - leaf with no workspace deps). `core` never depends on Git, worktrees, or - crypto. **`gix` stays in `gitio` only**; **RustCrypto stays in `crypto` - only**; **rustls/rcgen stay in `tlsio` only**. If you find yourself reaching - for `gix`, a RustCrypto type, or TLS elsewhere, add a function to - `gitio`/`crypto`/`tlsio` instead. + the compiler): the adapters `cli` and `desktop` sit on top of `repo`, and + `repo → {vfs, crypto} → core` plus the leaf edges `repo → tlsio` and + `repo → objio` (`tlsio` and `objio` have no workspace deps). `gitio` depends + only on `core`; `repo` never links it (`cli` links both and passes imported + snapshots down). `core` never depends on Git, worktrees, or crypto. **`gix` + stays in `gitio` only**; **RustCrypto stays in `crypto` only**; + **rustls/rcgen stay in `tlsio` only**; **the S3 SDK stays in `objio` only**. + If you find yourself reaching for `gix`, a RustCrypto type, TLS, or an + object-store SDK elsewhere, add a function to `gitio`/`crypto`/`tlsio`/`objio` + instead. 3. **Do not break the [core invariants in `CLAUDE.md`](CLAUDE.md#core-invariants-do-not-break)** — content addressing (`BLAKE3(canonical_encoding)`), `Arc<[u8]>`-shared blobs, the mode-scoped disk invariant (ephemeral = zero residue), and "never silently drop @@ -57,7 +73,7 @@ Some changes get a security pass before merge as standing policy, not ad-hoc judgment. If your PR does any of the following, say so in the PR description and walk the checklist: -- **Touches `crates/crypto`, `crates/tlsio`, or any transport/wire code** → +- **Touches `crates/crypto`, `crates/tlsio`, `crates/objio`, or any transport/wire code** → re-read the relevant [`docs/THREAT-MODEL.md`](docs/THREAT-MODEL.md) section; state in the PR whether any documented boundary moves. If one does, the PR must also update `THREAT-MODEL.md` (and add an ADR if the decision changes). diff --git a/README.md b/README.md index eb41d1aa..edb031f2 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,9 @@ A next-generation version control system built around a snapshot-and-tag model (Jujutsu-inspired), with per-file permissions, native committed secrets, and -in-memory clones as the long-term thesis. Across 34 shipped phases, this -repository now proves that thesis end to end: +in-memory clones as the long-term thesis. Across 36 shipped phases plus the +P37 partial-clone series (37a-37d), this repository now proves that thesis end +to end: 1. **In-memory virtual worktrees** (the agent wedge) — fork N parallel worktrees of a repo entirely in RAM, run and check out against each, and tear them down @@ -22,8 +23,9 @@ repository now proves that thesis end to end: durable `.sc/` repos, branches, three-way merge, history editing (cherry-pick/rebase/amend/undo), durable multi-invocation agent sessions, signed commits with session-transcript provenance, sparse and partial - checkouts, and local/ssh/HTTP+TLS/Git network transports. **Implemented - (Phases 3-34.)** + checkouts (including merge and agent sessions on a partial clone), local/ + ssh/HTTP+TLS/bucket/Git network transports, and a read-only native desktop + browser. **Implemented (Phases 3-37.)** The system builds on / interoperates with Git rather than replacing it: it imports an existing Git repo's `HEAD` in-process (via `gix`), exports history @@ -31,9 +33,10 @@ back to Git commits, and reaches hosted Git (e.g. GitHub) through a spawned system-git mirror bridge. See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design, -[docs/adr/](docs/adr/) for the architecture decision records covering every -phase, and [CLAUDE.md](CLAUDE.md) for project conventions and the per-phase -capability map. +[docs/adr/](docs/adr/) for the architecture decision records (0001-0054) behind +every phase, and [CLAUDE.md](CLAUDE.md) for project conventions and the +per-phase capability map. [CONTRIBUTING.md](CONTRIBUTING.md) covers the +development workflow. > ⚠️ **Pre-1.0, not independently audited — don't trust production secrets to it > yet.** src-control implements real cryptography (committed-secret envelope @@ -41,9 +44,10 @@ capability map. > TLS), but these are MVP implementations that have **not had an independent > security audit**. There are deliberate boundaries you should understand first — > e.g. pre-Phase-33 convergent ciphertext in history stays equality-confirmable -> forever, `sc+https://` trust is TOFU pinning (the first connection to a host is -> unverified), and rotation cuts off future reads but cannot erase ciphertext -> already in history. They are consolidated in +> forever, `sc+https://` trust defaults to TOFU pinning (the first connection to +> a host is unverified unless you pre-pin it or opt into CA validation), and +> rotation cuts off future reads but cannot erase ciphertext already in +> history. They are consolidated in > [docs/THREAT-MODEL.md](docs/THREAT-MODEL.md). Report vulnerabilities privately > per [SECURITY.md](SECURITY.md). @@ -60,12 +64,17 @@ crates/ Ed25519 signing (the ONLY crate that links RustCrypto) tlsio/ TLS for the sc+https:// transport (the ONLY crate that links rustls/rcgen; a leaf — no workspace deps) + objio/ object-store buckets for sc+wal:// and sc+s3:// remotes (the ONLY crate + that links the S3 SDK; a leaf — no workspace deps) repo/ durable .sc repo — refs, branches, working tree, merge, history editing, remotes, protection, GC, sparse/partial clone, transcripts, - transport + transport (ssh, sc+http(s), bucket WAL) cli/ `sc` binary +apps/ + desktop/ read-only Tauri v2 + React/TypeScript repository browser (see + apps/desktop/README.md); calls `repo` directly, never the `sc` binary demo/ - 25 end-to-end proof scripts covering everything below, e.g.: + 27 end-to-end proof scripts covering everything below, e.g.: run_demo.sh independent before/after filesystem diff, zero residue run_repo_demo.sh persistent repo end-to-end demo run_ssh_remote_demo.sh ssh-native transport round trip @@ -73,10 +82,13 @@ demo/ run_transcript_demo.sh sealed agent-session transcript proof ``` -Dependency direction is strict: `cli → repo → {vfs, gitio, crypto, tlsio} → core` -(`tlsio` is a leaf). Only `gitio` links `gix`, only `crypto` links RustCrypto, -and only `tlsio` links rustls, keeping the rest of the system Git-, crypto-, -and TLS-library-agnostic. +Dependency direction is strict: the adapters `cli` and `desktop` sit on top of +`repo`; `repo → {vfs, crypto} → core`, plus the leaf edges `repo → tlsio` and +`repo → objio` (`tlsio` and `objio` depend on no other workspace crate). `gitio` +depends only on `core`, and `repo` never links it: `cli` links both and passes +imported snapshots down. Only `gitio` links `gix`, only `crypto` links +RustCrypto, only `tlsio` links rustls, and only `objio` links the S3 SDK, +keeping the rest of the system Git-, crypto-, TLS- and object-store-agnostic. ## Quick start @@ -113,9 +125,14 @@ cargo run --bin sc -- export --to /path/to/git/repo bash demo/run_demo.sh ``` -See `CLAUDE.md`'s Commands section for the full command surface (merge, -history editing, secrets lifecycle, sparse/partial clone, ssh/HTTP remotes, -session transcripts, and more). +Run `sc --help` (and `sc --help`) for the full command surface: +merge, history editing, secrets lifecycle, sparse/partial clone, ssh/HTTP(S)/ +bucket remotes, session transcripts, and more. The annotated reference lives in +the `sc-cli-reference` skill (`.claude/skills/sc-cli-reference/SKILL.md`), which +does not yet cover bucket remotes. Those take `sc remote add +sc+wal://` or `sc+s3:///` (see ADR-0046), and `sc serve +--http 127.0.0.1:8730 --store ` serves a bucket from a +disposable instance (`--stdio` also works; `sc serve --help`). ## How Phase 1 works @@ -142,13 +159,18 @@ permissions via encrypted paths (Phase 7, randomized sealing since Phase 33) all ship alongside a full persistent, branchable, content-addressed VCS — merge and history editing, durable multi-invocation agent sessions with auto-merge, a commit-time secret scanner, secret/permission lifecycle -(rotation, escrow, revocation tombstones, bulk rewrap), signed commits with -sealed session-transcript provenance, sparse and partial checkouts, and local, -ssh-native, sc-native HTTP/TLS (with bearer-token auth and listener resource -limits), and Git (including hosted GitHub) network transports, plus private -branches sealed to a recipient set until publish. All 34 phases are -implemented and tested; `demo/` holds 26 independent end-to-end proofs, one -per major capability. Remaining follow-ons (transparent lazy-fetch, -per-case gap-tolerant merge on partial clones, CA-path validation alongside -TOFU pinning, rotate-for-paths, and similar hardening items) are tracked in -[ROADMAP.md](ROADMAP.md)'s Deferred section rather than being open MVP scope. +(rotation, escrow, revocation tombstones, bulk rewrap, `sc rewrap --rotate` +for protected paths), signed commits with sealed session-transcript +provenance (droppable through signed tombstones), sparse and partial checkouts (lazy want-by-id fetch, merge, +cherry-pick, rebase and agent sessions on a partial clone), and local, +ssh-native, sc-native HTTP/TLS (bearer tokens with expiry and rotation, +listener resource limits, TOFU pinning managed by `sc tls`, or CA validation +via `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system`), bucket (`sc+wal://`, `sc+s3://`), +and Git (including hosted GitHub) network transports, plus private branches +sealed to a recipient set until publish and a read-only Tauri desktop +browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds +27 independent end-to-end proofs, one per major capability. Remaining +follow-ons (transcript drop delivery gaps, bucket compaction and gc, named +concurrent `sc ws` sessions, size/count clone filters, and similar hardening +items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open +GitHub issues rather than being open MVP scope. diff --git a/SECURITY.md b/SECURITY.md index 85546793..7a0b9932 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -17,16 +17,30 @@ it are consolidated in [`docs/THREAT-MODEL.md`](docs/THREAT-MODEL.md). In brief: nonce per seal). Content sealed *before* P33 used convergent encryption and remains **equality-confirmable forever** — rotation cannot erase ciphertext already in history. -- **Two confidential transports ship:** `sc+https://` (in-binary TLS with - accept-new TOFU pinning; the *first* connection to a host is unverified by - construction, and `SC_HTTPS_FINGERPRINT` / `SC_HTTPS_STRICT=1` close that - window) and `ssh://`. Plain `sc serve --http` remains **plaintext** and its - bearer tokens cross the wire in the clear — use it only on loopback or behind - the documented stream-mode proxy setup; prefer `sc+https://` for any public - bind. +- **Two confidential sc-native transports ship:** `sc+https://` (in-binary + TLS; by default accept-new TOFU pinning, so the *first* connection to a host + is unverified by construction, and `SC_HTTPS_FINGERPRINT`, `sc tls pin` or + `SC_HTTPS_STRICT=1` close that window; `SC_HTTPS_CA_FILE` / `SC_HTTPS_CA=system` + switch a client to CA validation instead, with no revocation checking) and + `ssh://`. Plain `sc serve --http` remains **plaintext** and its bearer + tokens (which can carry an expiry and be rotated) cross the wire in the + clear — use it only on loopback or behind the documented stream-mode proxy + setup; prefer `sc+https://` for any public bind. +- **Bucket remotes (`sc+wal://`, `sc+s3://`) put the bucket's ACL on the + perimeter.** Public content sits in the bucket as plaintext at rest and a + writer with raw bucket access can move refs outside the `sc` client; sealed + content stays ciphertext. A bucket cannot judge a transcript drop tombstone, so + it serves a dropped transcript's body next to the tombstone; the receiver + honors the drop only for a signer in its `[signers] trusted` list. - Committed secrets injected by `sc run` live in an **authorized local process context, not strong isolation** — same-user processes can observe the child - environment. + environment. `sc run --secrets-fd N` (unix only; other platforms refuse it) keeps them + out of the environment but is still not isolation. +- **`sc transcript drop` is not erasure either.** A drop travels as a signed + tombstone that a peer honors only when the signature comes from a key in its + `[signers] trusted` list; an unsigned drop is local-only, and any trusted + signer can hide transcripts on every peer that trusts them. Peers without + the tombstone keep the sealed body. - **Rotation/revocation ≠ erasure.** They cut off *future* reads through the current registry but cannot erase ciphertext already in history and copied to other clones; real cutover of a leaked credential always means rotating the diff --git a/docs/agents/ACCESS.md b/docs/agents/ACCESS.md index d9d3bf26..85f014fa 100644 --- a/docs/agents/ACCESS.md +++ b/docs/agents/ACCESS.md @@ -29,6 +29,7 @@ it as the revocation checklist. Written as OSTIF-audit follow-up T-14 (G-012). | `GITHUB_TOKEN` — `ci.yml` | `contents: read` | Ephemeral per-run. | | `GITHUB_TOKEN` — `codeql.yml` | `contents: read` top-level; analyze job adds `security-events: write`, `actions: read` | Ephemeral per-run. | | `GITHUB_TOKEN` — `audit.yml` | `contents: read`, `checks: write`, `issues: write` (job-scoped as of audit T-20) | Files RustSec findings as public issues. | +| `GITHUB_TOKEN` — `fuzz.yml`, `sbom.yml`, `vendor-provenance.yml` | `contents: read` | Ephemeral per-run. Nightly `wire_decode` fuzz, weekly CycloneDX SBOMs, and the vendored-glib provenance check; none holds a write scope. | | `GITHUB_TOKEN` — `scorecard_analysis.yml` | `read-all` top-level; job adds `security-events: write`, `id-token: write` | Publishes Scorecard results. | | AI agents (Claude Code et al.) | `gh` CLI under the maintainer's auth: issue/PR create, comment, label, close; local commit + push | Operate the public triage pipeline (`issue-tracker.md`). **Excluded** from draft security advisories and private reports; security fixes need distinct human review (see vulnerability-response.md). Compromise playbook: incident-response class 2. | | Dependabot | Version + security update PRs (cargo, npm, github-actions) | PRs merge only through the standing required checks. | diff --git a/docs/agents/domain.md b/docs/agents/domain.md index 90d4de6b..afa6de45 100644 --- a/docs/agents/domain.md +++ b/docs/agents/domain.md @@ -14,7 +14,7 @@ currently live in two root files — treat them as the glossary/context source: pointer to it. The full per-phase narrative log is archived at `docs/archive/claude-md-phase-log-2026-07.md`. - **`ARCHITECTURE.md`** at the repo root — the full design and rationale. -- **`docs/adr/`** — 43+ ADRs (0001…) recording every significant decision; +- **`docs/adr/`** — 54 ADRs (0001…0054) recording every significant decision; read the ADRs that touch the area you're about to work in. This is a single-context repo, so there are no per-crate `crates//docs/adr/` dirs — all decisions live in the one root `docs/adr/`. @@ -33,11 +33,12 @@ Single-context repo: / ├── CLAUDE.md ← working guide + capability map (AGENTS.md points here) ├── ARCHITECTURE.md ← full design + rationale -├── docs/adr/ ← 0001…0043+ architectural decisions (one root collection) -└── crates/ ← core, vfs, gitio, crypto, tlsio, repo, cli (one cohesive VCS domain) +├── docs/adr/ ← 0001…0054 architectural decisions (one root collection) +├── crates/ ← core, vfs, gitio, crypto, tlsio, objio, repo, cli (one cohesive VCS domain) +└── apps/desktop/ ← Tauri v2 read-only browser (a top-level adapter over repo) ``` -(This is a seven-crate Rust workspace but **one** domain — a next-gen VCS — so a +(This is an eight-crate Rust workspace plus a desktop app but **one** domain — a next-gen VCS — so a single context, not a multi-context `CONTEXT-MAP.md` layout.) ## Use the project's vocabulary