From 45d2adc19baa9e808cc550dd8c74069e84aa6f14 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 11:44:14 +0300 Subject: [PATCH] fix(repo): serve signatures, transcripts and drop tombstones from bucket remotes (ref #229) BucketTransport::get_pack built its pack from reachability only, so a fetch or clone from sc+wal:// / sc+s3:// lost snapshot signatures as well as transcripts and their signatures. A bucket has no .sc/signatures or .sc/transcripts index, so get_pack now rebuilds the sender seam by decoding the on-chain packs (cached per pack hash) and over-sends the same set LocalTransport does. No bucket format change, no PROTOCOL_VERSION bump. The receiver trust gate for tombstones and the --no-transcripts skip are unchanged. Docs: ADR-0038 and ADR-0046 addenda, THREAT-MODEL, CLAUDE.md, ROADMAP, sc-cli-reference skill and --no-transcripts help text. --- .claude/skills/sc-cli-reference/SKILL.md | 5 +- ARCHITECTURE.md | 4 +- CLAUDE.md | 2 +- ROADMAP.md | 6 + crates/cli/src/main.rs | 10 +- crates/cli/tests/bucket_remote.rs | 59 ++++ crates/repo/src/bucket_transport.rs | 361 ++++++++++++++++++++- crates/repo/src/sync.rs | 2 +- docs/THREAT-MODEL.md | 12 + docs/adr/0038-agent-session-transcripts.md | 59 +++- docs/adr/0046-wal-bucket-remotes.md | 13 + 11 files changed, 513 insertions(+), 20 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 29094d6c..6d4d35b9 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -228,8 +228,9 @@ cargo run --bin sc -- clone|fetch|push --no-transcripts ... # leave transcript # a later plain fetch brings them in; a plain push # carries them only once the branch moves again. # Snapshots transfer and verify as usual; no-op for - # git and bucket (sc+wal://, sc+s3://) remotes - # (push to a bucket still skips the upload) + # git remotes. Bucket remotes (sc+wal://, sc+s3://) + # serve transcripts and signatures (#229), so the + # flag works there like on any native remote cargo run --bin sc -- transcript drop [--identity ] # drop a transcript by id/hex prefix (see `list`): writes # a tombstone that travels with fetch/clone so a peer # that still has it cannot restore it; gc reclaims the diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 22b4e23f..951f1785 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -931,7 +931,9 @@ a transfer-relevant snapshot, and the receiver reindexes idempotently. `sc gc` roots live transcript ids into reachability before signature pruning runs, so a signed transcript survives exactly as long as the transcript itself does. `sc export --to ` drops transcripts (no Git-native -form exists) and reports a count. See ADR-0038. +form exists) and reports a count. A bucket remote (P36) has no index, so its +`get_pack` rebuilds the same over-send from the packs the manifest chain names +(#229, no format change). See ADR-0038. ## Phase 31 — listener resource limits (built) diff --git a/CLAUDE.md b/CLAUDE.md index 497aee0c..6706cd62 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -150,7 +150,7 @@ the code, those win. | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) | | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | -| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change) | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | diff --git a/ROADMAP.md b/ROADMAP.md index 230deb65..f3b06c45 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -877,6 +877,12 @@ scale-&-reach horizon): rewrap/grant/revoke surface — sealed once, to the recipient set at attach time, permanently. Extending P17's rewrap and P16's grant/revoke machinery to transcripts is deferred. +- **Bucket leaf sidecar (#229 follow-on).** Bucket `get_pack` finds signatures, + transcripts and tombstones by decoding every on-chain pack (cached per pack + hash within one transport). A per-pack sidecar object listing those leaves, + written at push time and ignored by older readers, would remove the decode; + deferred until fetch egress or CPU on a real bucket shows the scan (up to a + second download of each pack per fetch) is too costly. - **Sender-side `--no-transcripts` for fetch/clone (P30).** The #211 opt-out is client-side: `fetch`/`clone` still receive transcripts and discard them, so only `push` saves bandwidth. Having the server skip diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 7d762df7..25ed58de 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -321,8 +321,7 @@ enum Cmd { /// still sends them, the client hash-checks and discards them, so /// this transfer writes none and indexes none. Per command, not /// remembered by the clone; a later plain fetch brings them in. - /// Does nothing for git remotes and bucket remotes (sc+wal://, - /// sc+s3://), which carry no transcripts. + /// Does nothing for git remotes, which carry no transcripts. #[arg(long)] no_transcripts: bool, }, @@ -406,8 +405,7 @@ enum Cmd { /// this fetch writes none and indexes none (a new signature over a /// transcript you already hold is dropped too). Per command, not /// remembered; a later plain fetch brings them in. Does nothing for - /// git remotes and bucket remotes (sc+wal://, sc+s3://), which - /// carry no transcripts. + /// git remotes, which carry no transcripts. #[arg(long)] no_transcripts: bool, }, @@ -420,9 +418,7 @@ enum Cmd { include_encrypted: bool, /// Do not send transcript objects (P30) or their signatures. Sealed /// bodies stay sealed either way; snapshots still transfer. Does - /// nothing for git remotes, which never carry transcripts. A bucket - /// remote (sc+wal://, sc+s3://) is stored but never served them, so - /// there the flag only spares the upload. + /// nothing for git remotes, which never carry transcripts. #[arg(long)] no_transcripts: bool, }, diff --git a/crates/cli/tests/bucket_remote.rs b/crates/cli/tests/bucket_remote.rs index 5b10364e..4a984bab 100644 --- a/crates/cli/tests/bucket_remote.rs +++ b/crates/cli/tests/bucket_remote.rs @@ -86,6 +86,65 @@ fn bucket_clone_push_fetch_round_trip_and_url_validation() { } } +/// #229: a signed snapshot pushed to a bucket keeps its signature on a fresh +/// clone, so `sc verify --require` passes there once the signer is trusted +/// (trust is local config, written into the clone). +#[test] +fn verify_require_passes_after_a_bucket_clone_of_a_signed_snapshot() { + let a = tmp("sign-a"); + let bucket = tmp("sign-bucket"); + let parent = tmp("sign-parent"); + let url = format!("sc+wal://{}", bucket.display()); + + // Keys live outside the work tree (the scanner flags key material in one). + let id = parent.join("alice.id"); + let out = sc(&parent, &["keygen", "--out", id.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + let text = String::from_utf8_lossy(&out.stdout).into_owned(); + let sig_pk = text + .lines() + .find_map(|l| l.split_whitespace().find(|w| w.starts_with("scl-sig-"))) + .expect("keygen prints a signing public key") + .to_string(); + + assert!(sc(&a, &["init"]).status.success()); + std::fs::write(a.join("f.txt"), b"signed").unwrap(); + let out = sc( + &a, + &[ + "commit", + "-m", + "c1", + "--sign", + "--identity", + id.to_str().unwrap(), + ], + ); + assert!(out.status.success(), "{out:?}"); + assert!(sc(&a, &["remote", "add", "origin", &url]).status.success()); + assert!(sc(&a, &["push", "origin"]).status.success()); + + let b = parent.join("b"); + let out = sc(&parent, &["clone", &url, b.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + std::fs::write( + b.join(".sc/recipients.toml"), + format!("[signing]\nalice = \"{sig_pk}\"\n\n[signers]\ntrusted = [\"alice\"]\n"), + ) + .unwrap(); + let out = sc(&b, &["verify", "--require"]); + assert!( + out.status.success(), + "verify --require after a bucket clone: {out:?}" + ); + assert!(String::from_utf8_lossy(&out.stdout).contains("1 trusted")); + + for d in [&a, &bucket, &parent] { + std::fs::remove_dir_all(d).unwrap(); + assert!(!d.exists()); + } +} + /// `sc serve --http --store ` serves a bucket instead of the /// serve-home's own object store (P36c): a repo pushed straight to the /// bucket is clonable through the server, and a push through the server is diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index 306928fd..d4977641 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -66,6 +66,26 @@ pub struct BucketTransport { /// (possibly long-lived, e.g. `wire::serve`-hosted) transport regardless /// of whether this call succeeds or fails. pending_packs: RefCell>, + /// Signature and transcript objects found in each scanned pack, keyed by + /// pack hash (#229). Packs are immutable, so an entry never goes stale; + /// it spares a long-lived `serve --store` session from re-reading every + /// pack body on each `get_pack`. + leaf_cache: RefCell>>, +} + +/// A leaf object a pack holds that no tree or snapshot references, so the +/// reachability walk cannot find it (#229). Only the fields the sender seam +/// needs are kept. +#[derive(Clone)] +enum Leaf { + /// A signature `id` over `target` (a snapshot, transcript or tombstone). + Signature { id: ObjectId, target: ObjectId }, + /// A transcript or drop tombstone `id` attached to `snapshot`. + Transcript { + id: ObjectId, + snapshot: ObjectId, + tombstone: bool, + }, } /// Untrusted-length guard (P28 parity): refuse any WAL metadata value — @@ -188,6 +208,7 @@ impl BucketTransport { pack_cache: RefCell::new(None), staged: RefCell::new(Vec::new()), pending_packs: RefCell::new(Vec::new()), + leaf_cache: RefCell::new(BTreeMap::new()), }; t.refresh()?; Ok(t) @@ -325,6 +346,89 @@ impl BucketTransport { Ok(read_object_at_bounded(pack, offset, id, scl_core::MAX_OBJECT_SIZE)?.encode()) } + /// Signature and transcript objects of one on-chain pack, found by + /// decoding its records (the idx carries no object kind). Reads through + /// `parse_pack_reader`, so it keeps the untrusted-source guards (hash + /// check, `MAX_OBJECT_SIZE` caps); peak RAM is the pack body plus one + /// decoded object. Does not touch `pack_cache`, which the reachability + /// walk and the pack writer rely on for locality. + fn pack_leaves(&self, hash: &str) -> Result> { + if let Some(hit) = self.leaf_cache.borrow().get(hash) { + return Ok(hit.clone()); + } + let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { + return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + }; + let mut leaves = Vec::new(); + scl_core::pack::parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { + match obj { + Object::Signature(s) => leaves.push(Leaf::Signature { + id, + target: s.snapshot, + }), + Object::Transcript(t) => leaves.push(Leaf::Transcript { + id, + snapshot: t.snapshot, + tombstone: t.dropped_id().is_some(), + }), + _ => {} + } + Ok(()) + })?; + self.leaf_cache + .borrow_mut() + .insert(hash.to_string(), leaves.clone()); + Ok(leaves) + } + + /// The bucket-side sender seam (#229, ADR-0038/ADR-0046): the ids of + /// every transcript and signature `LocalTransport::build_pack_tempfile` + /// would over-send for a transfer whose snapshots are `snaps` — transcripts + /// and signed drop tombstones attached to them, and signatures over + /// snapshots, transcripts and tombstones. A bucket keeps no + /// `.sc/transcripts` or `.sc/signatures` index, so it is rebuilt from the + /// packs the manifest chain names; whatever a pusher uploaded is served. + /// An unsigned tombstone is withheld, as on `LocalTransport`. Whether a + /// tombstone is honored is the receiver's call (`index_incoming`). + fn leaf_ids_for(&self, snaps: &std::collections::BTreeSet) -> Result> { + let packs = match self.view.borrow().as_ref() { + Some(v) => v.packs.clone(), + None => return Ok(Vec::new()), + }; + // target -> signature ids; (snapshot, id, tombstone) per transcript. + // Drop the walk's cached pack body first so the scan never holds two. + *self.pack_cache.borrow_mut() = None; + let mut sigs: BTreeMap> = BTreeMap::new(); + let mut transcripts: Vec<(ObjectId, ObjectId, bool)> = Vec::new(); + for hash in &packs { + for leaf in self.pack_leaves(hash)? { + match leaf { + Leaf::Signature { id, target } => sigs.entry(target).or_default().push(id), + Leaf::Transcript { + id, + snapshot, + tombstone, + } => transcripts.push((snapshot, id, tombstone)), + } + } + } + let mut out: std::collections::BTreeSet = std::collections::BTreeSet::new(); + for (snapshot, id, tombstone) in transcripts { + if snaps.contains(&snapshot) && (!tombstone || sigs.contains_key(&id)) { + out.insert(id); + } + } + // A transcript's own signature is keyed by the transcript id, so the + // targets are the snapshots plus the transcripts chosen above. + let chosen = out.clone(); + for (target, ids) in &sigs { + if snaps.contains(target) || chosen.contains(target) { + out.extend(ids.iter().copied()); + } + } + Ok(out.into_iter().collect()) + } + /// Upload one pack (+ its idx) content-addressed by BLAKE3 of the pack /// bytes, so two callers who happen to pack the identical object set /// converge on the same key. `put_new`'s "key already exists" outcome is @@ -517,7 +621,16 @@ impl Transport for BucketTransport { }; let have_set = crate::reachable::reachable_objects(&mut src, &known_haves)?; let want_set = crate::reachable::reachable_objects(&mut src, wants)?; - let ids: Vec = want_set.difference(&have_set).copied().collect(); + // Sender seam (#229): signatures and transcripts are leaves no walk + // reaches. Over-send them for every snapshot in play, never + // subtracting any via `have_set`, exactly as `LocalTransport` does. + let snaps: std::collections::BTreeSet = + want_set.union(&have_set).copied().collect(); + let leaf_ids = self.leaf_ids_for(&snaps)?; + let mut ids: Vec = want_set.difference(&have_set).copied().collect(); + ids.extend(leaf_ids.into_iter().filter(|id| !have_set.contains(id))); + ids.sort(); + ids.dedup(); let mut writer = PackWriter::new(out, ids.len() as u32)?; for id in &ids { let bytes = self.object_bytes(id)?; @@ -1707,4 +1820,250 @@ mod tests { std::fs::remove_dir_all(&broot).unwrap(); assert!(!broot.exists()); } + + // ---- #229: signatures, transcripts and drop tombstones round-trip ---- + + /// An origin repo pushed to a fresh `DirBucket`: one snapshot signed by + /// `identity`, carrying a transcript that is signed too. Returns + /// (origin, url, snapshot, transcript id, identity, scratch dir). + fn signed_bucket_origin( + tag: &str, + ) -> ( + crate::repo::Repo, + String, + ObjectId, + ObjectId, + scl_crypto::Identity, + std::path::PathBuf, + ) { + let base = std::env::temp_dir().join(format!("scl-bt-229-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&base); + std::fs::create_dir_all(base.join("a")).unwrap(); + let url = format!("sc+wal://{}", base.join("bucket").display()); + let a = crate::repo::Repo::init(base.join("a")).unwrap(); + std::fs::write(base.join("a/f1"), b"one").unwrap(); + let snap = a.commit("t", "c1").unwrap(); + let (_s, identity) = scl_crypto::generate_identity_v2(); + a.sign_snapshot(snap, &identity).unwrap(); + let tid = a + .attach_transcript(snap, "claude-code", "s1", b"body", &[identity.enc.public()]) + .unwrap(); + a.sign_transcript(tid, &identity).unwrap(); + a.remote_add("origin", &url).unwrap(); + a.push("origin").unwrap(); + (a, url, snap, tid, identity, base) + } + + fn trust_map(identity: &scl_crypto::Identity) -> std::collections::HashMap<[u8; 32], String> { + let mut m = std::collections::HashMap::new(); + m.insert( + identity.signing.as_ref().unwrap().public().to_bytes(), + "alice".to_string(), + ); + m + } + + #[test] + fn clone_from_a_bucket_delivers_snapshot_and_transcript_signatures_and_transcripts() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("clone"); + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + + assert_eq!( + b.sig_status(&snap, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()), + "a signed snapshot keeps its signature through a bucket" + ); + assert!(b + .transcripts_for(&snap) + .unwrap() + .iter() + .any(|(t, _)| *t == tid)); + assert_eq!(b.signatures_for(&tid).unwrap().len(), 1); + assert_eq!( + b.open_transcript(&tid, &identity.enc).unwrap().as_slice(), + b"body" + ); + + drop(a); + drop(b); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + + #[test] + fn fetch_from_a_bucket_delivers_a_later_transcript_and_signature() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("fetch"); + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + + // Origin signs a second commit and attaches a signed transcript to it. + std::fs::write(base.join("a/f2"), b"two").unwrap(); + let snap2 = a.commit("t", "c2").unwrap(); + a.sign_snapshot(snap2, &identity).unwrap(); + let tid2 = a + .attach_transcript(snap2, "claude-code", "s2", b"two", &[identity.enc.public()]) + .unwrap(); + a.sign_transcript(tid2, &identity).unwrap(); + a.push("origin").unwrap(); + + b.fetch("origin").unwrap(); + assert_eq!( + b.sig_status(&snap2, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()) + ); + assert!(b + .transcripts_for(&snap2) + .unwrap() + .iter() + .any(|(t, _)| *t == tid2)); + assert_eq!(b.signatures_for(&tid2).unwrap().len(), 1); + assert!(b + .transcripts_for(&snap) + .unwrap() + .iter() + .any(|(t, _)| *t == tid)); + + drop(a); + drop(b); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + + #[test] + fn no_transcripts_clone_and_fetch_from_a_bucket_skip_transcripts_but_keep_signatures() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("notrans"); + let b = crate::repo::Repo::clone_url_opts(&url, base.join("b"), None, true).unwrap(); + + assert_eq!( + b.sig_status(&snap, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()) + ); + assert!(!b.store_arc().lock().unwrap().contains(&tid)); + assert!(b.transcripts_for(&snap).unwrap().is_empty()); + assert!(b.signatures_for(&tid).unwrap().is_empty()); + + // Per command, not persisted: a plain fetch brings it in. + b.fetch("origin").unwrap(); + assert!(b + .transcripts_for(&snap) + .unwrap() + .iter() + .any(|(t, _)| *t == tid)); + + drop(a); + drop(b); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + + #[test] + fn signed_drop_tombstone_propagates_through_a_bucket_only_to_trusted_signers() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("drop"); + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + let c = crate::repo::Repo::clone_url(&url, base.join("c")).unwrap(); + assert_eq!(b.transcripts_for(&snap).unwrap().len(), 1); + + // Origin drops the transcript (signed tombstone) and pushes. + crate::transcripts::tests::trust_signer(&a, &identity); + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(base.join("a/f2"), b"two").unwrap(); + a.commit("t", "c2").unwrap(); + a.push("origin").unwrap(); + + // B does not trust the signer: the tombstone is inert, transcript stays. + b.fetch("origin").unwrap(); + assert_eq!(b.transcripts_for(&snap).unwrap().len(), 1); + // Once trusted, the next fetch honors the (re-served) tombstone. + crate::transcripts::tests::trust_signer(&b, &identity); + b.fetch("origin").unwrap(); + assert!(b.transcripts_for(&snap).unwrap().is_empty()); + + // A peer that trusts the signer up front ends in the same state. + crate::transcripts::tests::trust_signer(&c, &identity); + c.fetch("origin").unwrap(); + assert!(c.transcripts_for(&snap).unwrap().is_empty()); + + // --no-transcripts leaves the tombstone out like any transcript. + let d = crate::repo::Repo::clone_url_opts(&url, base.join("d"), None, true).unwrap(); + assert!(d.transcripts_for(&snap).unwrap().is_empty()); + assert!(crate::transcripts::load(d.layout()).unwrap().is_empty()); + let tomb = ObjectId::of( + &Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)).encode(), + ); + assert!( + !d.store_arc().lock().unwrap().contains(&tomb), + "the tombstone object itself is skipped" + ); + assert!( + c.store_arc().lock().unwrap().contains(&tomb), + "a plain fetch from the bucket does receive the tombstone" + ); + + drop(a); + drop(b); + drop(c); + drop(d); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + + #[test] + fn a_signature_added_later_to_a_held_snapshot_rides_a_bucket_fetch() { + // The have side of the over-send: B already holds `snap`, so it sits in + // the have closure; its retroactive signature must still be served. + let base = std::env::temp_dir().join(format!("scl-bt-229-retro-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&base); + std::fs::create_dir_all(base.join("a")).unwrap(); + let url = format!("sc+wal://{}", base.join("bucket").display()); + let a = crate::repo::Repo::init(base.join("a")).unwrap(); + std::fs::write(base.join("a/f1"), b"one").unwrap(); + let snap = a.commit("t", "c1").unwrap(); + a.remote_add("origin", &url).unwrap(); + a.push("origin").unwrap(); + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + + let (_s, identity) = scl_crypto::generate_identity_v2(); + a.sign_snapshot(snap, &identity).unwrap(); + std::fs::write(base.join("a/f2"), b"two").unwrap(); + a.commit("t", "c2").unwrap(); + a.push("origin").unwrap(); + + b.fetch("origin").unwrap(); + assert_eq!( + b.sig_status(&snap, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()) + ); + + drop(a); + drop(b); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + + #[test] + fn an_unsigned_drop_tombstone_in_the_bucket_is_not_served() { + let (a, url, snap, tid, _identity, base) = signed_bucket_origin("unsigned-tomb"); + // A local-only drop (no identity) makes an unsigned tombstone; `push` + // would never send it, so put it into the bucket by hand. + let tomb_obj = Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)); + let tomb = ObjectId::of(&tomb_obj.encode()); + std::fs::write(base.join("a/f2"), b"two").unwrap(); + let tip2 = a.commit("t", "c2").unwrap(); + a.push("origin").unwrap(); + let t = BucketTransport::open(&url).unwrap(); + t.put_object(&tomb, &tomb_obj.encode()).unwrap(); + t.update_ref("side", &tip2, None).unwrap(); + assert!(t.has_object(&tomb).unwrap(), "the bucket holds it"); + + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + assert!( + !b.store_arc().lock().unwrap().contains(&tomb), + "an unsigned tombstone is withheld, as LocalTransport does" + ); + assert_eq!(b.transcripts_for(&snap).unwrap().len(), 1); + + drop(a); + drop(b); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } } diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 8c5913c5..75cdc350 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -87,7 +87,7 @@ impl Repo { /// (ADR-0038 addendum, #211). With `no_transcripts`, transcript objects /// (and signatures over them) in the received pack are hash-checked but /// not written, so this clone holds no transcript object and no index - /// entry. Bucket remotes never serve transcripts, so the flag does + /// entry. Git remotes never carry transcripts, so the flag does /// nothing there. /// Snapshots, trees and blobs transfer unchanged. Per command, not /// persisted: a later plain `fetch` brings the transcripts in. diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index dd77a844..20bfb3fb 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -410,6 +410,18 @@ too, not just the src-control-side metadata. described above, so a hostile or corrupted bucket gets no more leverage against a serve instance than it would against a direct `sc` client. +- **Signatures, transcripts and drop tombstones are served from what the + bucket holds (#229).** The bucket does not judge them: it returns every + signature over a snapshot in the transfer and every transcript or signed + tombstone attached to one. A writer with raw bucket access can therefore add + a tombstone or a signature, but the receiver's checks are unchanged: a + signature is verified against `[signers] trusted` when read, and a tombstone + is honored only with a valid drop-digest signature from a trusted signer. + A hostile bucket can also make `get_pack` read every pack on the chain (the + scan is not limited to packs holding wanted objects); each pack stays under + the `MAX_OBJECT_SIZE` guards, so this costs bandwidth and CPU, not memory + beyond one pack body. + ## By-id object reads (`GetObjects`, `sc cat`) — ADR-0047 - **A hash is not a capability.** Anyone who can reach a served repo and diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index 792ee7ac..2b0184de 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -182,13 +182,10 @@ pipe-safety discipline). objects, so no `.sc/transcripts` or `.sc/signatures` entry created by this transfer points at a transcript the repo does not hold. 4. **Remotes where the flag does nothing.** Git remotes never carry - transcripts. Bucket remotes (`sc+wal://`, `sc+s3://`) never serve them: - `BucketTransport::get_pack` walks reachability only, so a fetch or clone - from a bucket delivers no transcripts or transcript signatures whether or - not the flag is given, and a plain fetch does not change that. For `push` - the flag still works: a normal bucket push uploads and stores the - transcripts (they are simply never served back), and `--no-transcripts` - skips that upload. The serving gap is tracked in #229. + transcripts, so the flag is a no-op there. Bucket remotes (`sc+wal://`, + `sc+s3://`) serve transcripts since #229 (see the bucket addendum below), + so the flag works on them like on any native remote: fetch and clone + discard what the bucket over-sends, and push skips the upload. 5. **Interaction with transcript-drop tombstones (second addendum below, #212).** A drop tombstone is a `Transcript` object with an empty envelope, so `--no-transcripts` treats it like any other transcript: the pack is @@ -370,3 +367,51 @@ reported anywhere (it is pruned by gc, hidden before that). digest, so it never authorizes a drop. - `sc export` still counts only transcripts that are not tombstones; a dropped body that gc has not swept yet is still counted as dropped history. + +## Addendum: bucket remotes serve transcripts and signatures (issue #229) + +Before this change `BucketTransport::get_pack` built its pack from +reachability only. A push to a bucket uploaded signatures, transcripts and +signed drop tombstones (the sender seam runs on the pushing side), but a +fetch or clone never got them back: snapshot signatures were lost too, so a +signed snapshot cloned from a bucket was `Unsigned` and `sc verify --require` +failed. (Confirmed by test before the fix; the issue left this open.) + +- **No format change, no `PROTOCOL_VERSION` bump.** A bucket keeps no + `.sc/signatures` or `.sc/transcripts` index, and its pack idx carries no + object kind. `get_pack` therefore rebuilds the sender seam from the packs + the manifest chain already names: it decodes each on-chain pack once + (`parse_pack_reader`, same hash and `MAX_OBJECT_SIZE` guards as any + untrusted pack), keeps only `{id, target}` for signatures and + `{id, snapshot, is-tombstone}` for transcripts, and applies the same rule as + `LocalTransport::build_pack_tempfile`: over-send every transcript attached + to a snapshot in the want or have closure, plus signatures over those + snapshots and over the chosen transcripts and tombstones, never subtracting + them via the have set. A tombstone is sent only if a signature covers it + (an unsigned drop is never pushed anyway). +- **Trust stays on the receiver.** The bucket cannot judge a tombstone, so it + serves what it holds; `index_incoming` and clone `reindex` still honor a + tombstone only for a signer under `[signers] trusted`. A transcript the + bucket still holds next to its tombstone is served too, and a bucket cannot + filter it the way `LocalTransport` filters through its drop set. So unlike a + clone of a local origin that honored the drop (which never receives the + body), a clone from a bucket receives the body and the tombstone together; + the receiver never indexes a dropped id once it trusts the signer, and until + then shows the transcript. Withholding the body whenever a signed tombstone + exists would let anyone with bucket write access hide transcripts from + peers. +- **`--no-transcripts`** needs nothing bucket-specific: the receiver-side skip + in `ingest_pack_file` applies to the over-sent transcripts, tombstones and + their signatures. Snapshot signatures still arrive. +- **Cost.** The scan downloads and decodes every on-chain pack body once per + `BucketTransport` (leaf results cached per pack hash; packs are immutable), + including packs the transfer does not otherwise touch, and every fetch + (also `--no-transcripts`, which still needs the snapshot signatures) pays + it. That is the same order as the existing `get_pack`, whose have/want + reachability walks already fetch every pack that holds a reachable commit + or tree, but it is up to a second download of each pack per fresh transport + (each CLI fetch). Peak RAM is one pack body plus one decoded object (the + walk's cached body is released before the scan). A persisted per-pack leaf sidecar + would avoid the decode; deferred in `ROADMAP.md`. +- `sc serve --store` uses the same `get_pack`, so served-bucket fetch and + clone get the same behavior. diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index f1bffe14..d80621d3 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -203,3 +203,16 @@ separately-spawned `sc` **processes**). This supersedes the two now-stale Consequences bullets above about checkpoint folding and `sc serve` being unable to host a bucket as its backing store — both are built as of P36b/P36c. + +## As built (#229, 2026-10): signatures and transcripts on `get_pack` + +`BucketTransport::get_pack` now over-sends signatures, transcripts and signed +drop tombstones the way `LocalTransport` does, so a bucket round trip no +longer strips snapshot signatures (a signed snapshot used to clone as +`Unsigned`). The bucket format is unchanged: pushes already upload these +objects inside ordinary packs, and `get_pack` recovers which snapshot or +transcript each one covers by decoding the on-chain packs (cached per pack +hash). No manifest, log, checkpoint or idx change and no `PROTOCOL_VERSION` +bump. The receiver's trust gate for tombstones and the `--no-transcripts` +skip are unchanged. Rationale and cost in ADR-0038's #229 addendum. +