From 720637688eb59f6a85c956b5c52c67637f30f8d5 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 17:44:01 +0300 Subject: [PATCH 01/13] feat(repo): deliver drop tombstones after gc and after late signer trust (ref #230) Send every retained signed tombstone on every transfer instead of selecting them through the transfer set's snapshots, so a stale peer is told after the dropper's gc collected the snapshot. Remember held, signed, unhonored tombstones in a capped .sc/transcript-pending ring and re-evaluate them on Repo::open and after each ingest, so a push-only remote honors a drop once the operator trusts the signer. No PROTOCOL_VERSION bump; the trust gate is unchanged. ADR-0038 addendum records cost and limits. --- .claude/skills/sc-cli-reference/SKILL.md | 5 +- ARCHITECTURE.md | 13 +- CLAUDE.md | 4 +- ROADMAP.md | 12 +- crates/repo/src/bucket_transport.rs | 35 +++- crates/repo/src/layout.rs | 7 + crates/repo/src/repo.rs | 5 +- crates/repo/src/sync.rs | 190 +++++++++++++++++ crates/repo/src/transcripts.rs | 231 ++++++++++++++++++++- crates/repo/src/transport.rs | 6 + docs/THREAT-MODEL.md | 10 +- docs/adr/0038-agent-session-transcripts.md | 106 ++++++++-- 12 files changed, 579 insertions(+), 45 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 250caf76..a37f6bde 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -248,7 +248,10 @@ cargo run --bin sc -- transcript drop [--identity ] # drop a transcr # body (loose: after the grace window; already packed: next gc). Not erasure. --identity # signs the tombstone; peers honor it only if that key # is under [signers] trusted (unsigned = local-only). - # Propagates on the peer's next fetch (ADR-0038 addendum) + # Propagates on the peer's next fetch (ADR-0038 addendum); + # every retained signed tombstone rides every transfer, and + # a remote holding one from before it trusted the signer + # honors it on its next open/ingest (#230) cargo run --bin sc -- ws harvest --transcript [--sign] [--identity ] # attach the # file to each harvested workspace's landed snapshot # (P30; landing status prints before the attach) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index df4e403d..e379641f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -986,14 +986,17 @@ without a new object tag or wire change: a drop is a **tombstone**, a dropped id, recorded in a local `.sc/transcript-drops` index and treated as a grow-only set (no undrop; re-attach to restore). Reads, `sc log` markers and the over-send filter through the drop set, and `sc gc` reclaims the body. A -tombstone travels with fetch and push only when it is signed (a push that finds -the remote already at the same branch tip sends nothing, so a drop made -afterwards propagates once a push moves the branch or the remote fetches), and +tombstone travels with fetch and push only when it is signed, and every +retained signed tombstone rides every transfer, so a peer is told even after +the dropper's gc collected the snapshot (a push that finds the remote already at +the same branch tip sends nothing, so a drop made afterwards propagates once a +push moves the branch or the remote fetches), and a receiving peer honors it only if the signature comes from a key in its `[signers] trusted` list; an unsigned drop is local-only, and an untrusted tombstone stays inert. Drop is not erasure: a peer that never receives an honored tombstone -keeps the sealed body, and a remote that received a tombstone before trusting -its signer keeps it inert (#230). `sc push --no-transcripts` leaves transcripts and their signatures out of the +keeps the sealed body. A tombstone held before its signer was trusted is +remembered in `.sc/transcript-pending` (capped at 1024, newest kept) and +honored on the next open or ingest once the signer is trusted (#230). `sc push --no-transcripts` leaves transcripts and their signatures out of the pack (saving bandwidth); `sc clone|fetch --no-transcripts` cannot, because the server still sends the full pack, so the receiver hash-checks every record and then discards transcripts and their signatures, writing and indexing none diff --git a/CLAUDE.md b/CLAUDE.md index 0515a765..252f8834 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,7 +145,7 @@ the code, those win. | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | | P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | -| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex | [0038](docs/adr/0038-agent-session-transcripts.md) | +| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; every retained signed tombstone rides every transfer, and held-but-untrusted ones are re-evaluated from `.sc/transcript-pending` on open/ingest (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff; later `--max-conn-bytes` per-connection write budget (#233) | [0041](docs/adr/0041-listener-resource-limits.md) | | P32 | In-binary TLS: `sc+https://` via leaf crate `tlsio`; accept-new TOFU pinning (`sc tls list/forget/pin/path` manages the pin file); `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system` validate against a CA instead; client connect/handshake bounded; public plaintext bind no longer justified by tokens alone | [0042](docs/adr/0042-in-binary-tls-sc-https.md), [0053](docs/adr/0053-tls-pin-management.md), [0054](docs/adr/0054-tls-ca-validation.md) | | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | @@ -299,7 +299,7 @@ Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of truth — do not re-list items here); entries that name an issue mirror the open GitHub issues. Standing entries: named/concurrent `sc ws` sessions (#215), blob-size clone filters (#217), bucket compaction and gc (#216), the non-unix -`.sc/local-key` ACL (#214), transcript-drop delivery gaps (#230), and the open +`.sc/local-key` ACL (#214), and the open sparse, private-branch and abort edge cases (#241, #246, #247, #248, #243). ## Agent skills diff --git a/ROADMAP.md b/ROADMAP.md index 3e1700d6..f12a4410 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -480,8 +480,12 @@ across every phase. grow-only drop tombstone (a bare `Transcript` object, so no wire change) hides the body from `list`/`show`, travels with fetch/clone, blocks resurrection in either arrival order, and is kept by gc while the dropped - body is swept at gc (after the grace window if loose). (ADR-0038 addendum; - delivery gaps tracked in #230.) + body is swept at gc (after the grace window if loose). (ADR-0038 addendum.) + Issue #230 closed the delivery gaps: every retained signed tombstone rides + every transfer (so a stale peer is told after the dropper's gc collected the + snapshot), and a remote that held a tombstone before trusting the signer + re-evaluates it from a capped `.sc/transcript-pending` index on open and + ingest. - **Phase 32 — In-binary TLS (`sc+https://`).** Closes the audit's High #1 (plaintext bearer tokens/traffic on `sc+http://`) and ADR-0036's "no TLS" @@ -751,10 +755,6 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in - **Named/multiple concurrent ws sessions (#215, P20).** P20 ships one unnamed session per repo; named or concurrent sessions and workspace re-fork/refresh from a newer tip need a design decision first. -- **Transcript drop delivery gaps (#230, ADR-0038 addendum).** A tombstone is - not sent after the dropper's gc removed its snapshot, and a tombstone a - push-only remote received before trusting the signer is not re-evaluated - later; both are documented limits, not regressions. - **Sparse untracked-file guard edge cases.** #241: on a case-insensitive filesystem the untracked-file guard can alias an old-view path with a newly in-view one. #248: an untracked file at an out-of-view path that the other side adds wins diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index b16c9043..a8640383 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -423,7 +423,13 @@ impl BucketTransport { } let mut out: std::collections::BTreeSet = std::collections::BTreeSet::new(); for (snapshot, id, tombstone) in transcripts { - if snaps.contains(&snapshot) && (!tombstone || sigs.contains_key(&id)) { + // A signed tombstone goes out whatever its snapshot (#230). + let send = if tombstone { + sigs.contains_key(&id) + } else { + snaps.contains(&snapshot) + }; + if send { out.insert(id); } } @@ -2094,4 +2100,31 @@ mod tests { std::fs::remove_dir_all(&base).unwrap(); assert!(!base.exists()); } + + #[test] + fn a_signed_tombstone_is_served_whatever_the_transfer_set() { + // #230: the sender no longer selects tombstones through the snapshots + // of the transfer set, a bucket included. + let (a, url, snap, tid, identity, base) = signed_bucket_origin("tomb-any"); + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(base.join("a/f2"), b"two").unwrap(); + a.commit("t", "c2").unwrap(); + a.push("origin").unwrap(); + let tomb = ObjectId::of( + &Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)).encode(), + ); + + let t = BucketTransport::open(&url).unwrap(); + t.list_refs().unwrap(); + let served = t.leaf_ids_for(&std::collections::BTreeSet::new()).unwrap(); + assert!( + served.contains(&tomb), + "signed tombstone, empty transfer set" + ); + assert!(!served.contains(&tid), "the body still needs its snapshot"); + + drop(a); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } } diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index 607a63f0..61b26901 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -90,6 +90,13 @@ impl Layout { pub fn transcript_drops_path(&self) -> PathBuf { self.dot_sc.join("transcript-drops") } + /// `.sc/transcript-pending` — tombstone object ids this repo holds, has a + /// signature for, and has not honored yet (untrusted signer, or snapshot + /// not held), one hex id per line, newest last, capped. Re-evaluated on + /// every open and ingest; gc roots them (ADR-0038, #230). + pub fn transcript_pending_path(&self) -> PathBuf { + self.dot_sc.join("transcript-pending") + } /// `.sc/tmp` — scratch space for transient files (P25 streaming pack /// transfer: spilled sender/receiver pack bodies). Never durable state — /// safe to delete at any time; callers that write here are responsible diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 63a8965f..c71ed5f5 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -75,7 +75,10 @@ impl Repo { fn open_layout(layout: Layout, budget_bytes: usize) -> Result { let lock = RepoLock::acquire(&layout)?; - let store = Store::open_persistent(layout.objects_dir(), budget_bytes)?; + let mut store = Store::open_persistent(layout.objects_dir(), budget_bytes)?; + // Honor held tombstones whose signer was trusted since (#230). Best + // effort: a failure must not stop an open, and the next open retries. + let _ = crate::transcripts::reevaluate_pending(&layout, &mut store); Ok(Repo { layout, vfs: VfsRepo::new(store), diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 75cdc350..e63f1116 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -1280,6 +1280,196 @@ mod tests { std::fs::remove_dir_all(&b_root).unwrap(); } + fn tombstone_id(snap: ObjectId, tid: ObjectId) -> ObjectId { + ObjectId::of( + &scl_core::Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)) + .encode(), + ) + } + + #[test] + fn stale_peer_learns_a_drop_after_the_droppers_gc_collected_the_snapshot() { + // #230 gap 1: the tombstone outlives its snapshot on the dropper, so + // the snapshot is in no transfer set, yet a peer that still holds the + // body and the snapshot must be told. + let pid = std::process::id(); + let a_root = std::env::temp_dir().join(format!("scl-tomb-gc-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-tomb-gc-b-{pid}")); + let _ = std::fs::remove_dir_all(&a_root); + let _ = std::fs::remove_dir_all(&b_root); + std::fs::create_dir_all(&a_root).unwrap(); + let a = Repo::init(&a_root).unwrap(); + std::fs::write(a_root.join("f0"), b"zero").unwrap(); + let c0 = a.commit("t", "c0").unwrap(); + std::fs::write(a_root.join("f1"), b"one").unwrap(); + let c1 = a.commit("t", "c1").unwrap(); + let (_s, identity) = scl_crypto::generate_identity_v2(); + let tid = a + .attach_transcript(c1, "claude-code", "s1", b"body", &[identity.enc.public()]) + .unwrap(); + let b = Repo::clone_to(&a_root, &b_root).unwrap(); + assert_eq!(b.transcripts_for(&c1).unwrap().len(), 1); + crate::transcripts::tests::trust_signer(&b, &identity); + + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + // Rewind past c1 and forget the oplog roots, so gc collects it. + crate::refs::write_branch_tip(a.layout(), "main", &c0).unwrap(); + std::fs::remove_file(a.layout().oplog_path()).unwrap(); + std::fs::write(a_root.join("f2"), b"two").unwrap(); + a.commit("t", "c2").unwrap(); + a.gc(std::time::Duration::from_secs(0)).unwrap(); + assert!( + !a.store_arc().lock().unwrap().contains(&c1), + "gc collected the dropped transcript's snapshot" + ); + assert!(a + .store_arc() + .lock() + .unwrap() + .contains(&tombstone_id(c1, tid))); + drop(a); + + b.fetch("origin").unwrap(); + assert!(b.transcripts_for(&c1).unwrap().is_empty()); + + drop(b); + std::fs::remove_dir_all(&a_root).unwrap(); + std::fs::remove_dir_all(&b_root).unwrap(); + } + + #[test] + fn push_only_remote_honors_a_held_tombstone_once_the_signer_is_trusted() { + // #230 gap 2: the remote got the tombstone by push while it did not + // trust the signer. Trusting later must be enough: push never + // re-sends an object the remote holds. + let pid = std::process::id(); + let a_root = std::env::temp_dir().join(format!("scl-tomb-late-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-tomb-late-b-{pid}")); + let _ = std::fs::remove_dir_all(&b_root); + let (a, snap, tid, identity) = origin_with_signed_transcript(&a_root); + let b = Repo::clone_to(&a_root, &b_root).unwrap(); + drop(a); // push takes the origin's repo lock + + b.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(b_root.join("f2"), b"two").unwrap(); + b.commit("t", "c2").unwrap(); + b.push("origin").unwrap(); + + let a = Repo::open(&a_root).unwrap(); + assert_eq!( + a.transcripts_for(&snap).unwrap().len(), + 1, + "untrusted signer: the tombstone is held but inert" + ); + assert!(a.layout().transcript_pending_path().exists()); + crate::transcripts::tests::trust_signer(&a, &identity); + drop(a); + + // No re-push, no operator workaround: the next open re-evaluates. + let a = Repo::open(&a_root).unwrap(); + assert!(a.transcripts_for(&snap).unwrap().is_empty()); + assert!(!a.layout().transcript_pending_path().exists()); + drop(a); + + // A push that sends other objects re-evaluates too (long-lived server). + let c_root = std::env::temp_dir().join(format!("scl-tomb-late-c-{pid}")); + let _ = std::fs::remove_dir_all(&c_root); + let (c, snap_c, tid_c, id_c) = origin_with_signed_transcript(&c_root); + let d_root = std::env::temp_dir().join(format!("scl-tomb-late-d-{pid}")); + let _ = std::fs::remove_dir_all(&d_root); + let d = Repo::clone_to(&c_root, &d_root).unwrap(); + drop(c); + d.drop_transcript(&tid_c.to_hex(), Some(&id_c)).unwrap(); + std::fs::write(d_root.join("f2"), b"two").unwrap(); + d.commit("t", "c2").unwrap(); + d.push("origin").unwrap(); + let c = Repo::open(&c_root).unwrap(); + crate::transcripts::tests::trust_signer(&c, &id_c); + drop(c); + std::fs::write(d_root.join("f3"), b"three").unwrap(); + d.commit("t", "c3").unwrap(); + d.push("origin").unwrap(); + let c = Repo::open(&c_root).unwrap(); + assert!(c.transcripts_for(&snap_c).unwrap().is_empty()); + + drop((b, c, d)); + for root in [&a_root, &b_root, &c_root, &d_root] { + std::fs::remove_dir_all(root).unwrap(); + } + } + + #[test] + fn a_peer_that_never_trusts_the_signer_never_honors_the_tombstone() { + let pid = std::process::id(); + let a_root = std::env::temp_dir().join(format!("scl-tomb-never-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-tomb-never-b-{pid}")); + let _ = std::fs::remove_dir_all(&b_root); + let (a, snap, tid, identity) = origin_with_signed_transcript(&a_root); + let b = Repo::clone_to(&a_root, &b_root).unwrap(); + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + let (_s, other) = scl_crypto::generate_identity_v2(); + crate::transcripts::tests::trust_signer(&b, &other); + drop(a); + + for _ in 0..2 { + b.fetch("origin").unwrap(); + } + drop(b); + let b = Repo::open(&b_root).unwrap(); + assert_eq!(b.transcripts_for(&snap).unwrap().len(), 1); + assert!(crate::transcripts::tests::pending_count(&b) <= 1); + + drop(b); + std::fs::remove_dir_all(&a_root).unwrap(); + std::fs::remove_dir_all(&b_root).unwrap(); + } + + #[test] + fn no_transcripts_transfers_neither_hold_nor_remember_a_tombstone() { + let pid = std::process::id(); + let a_root = std::env::temp_dir().join(format!("scl-tomb-nt-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-tomb-nt-b-{pid}")); + let _ = std::fs::remove_dir_all(&b_root); + let (a, snap, tid, identity) = origin_with_signed_transcript(&a_root); + let b = Repo::clone_to(&a_root, &b_root).unwrap(); + let tomb = tombstone_id(snap, tid); + + // Fetch side: the dropper's tombstone is not stored or remembered. + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + drop(a); + b.fetch_opts("origin", true).unwrap(); + assert!(!b.store_arc().lock().unwrap().contains(&tomb)); + assert!(!b.layout().transcript_pending_path().exists()); + b.fetch("origin").unwrap(); + assert!(b.layout().transcript_pending_path().exists()); + assert_eq!(b.transcripts_for(&snap).unwrap().len(), 1); + + // Push side: `--no-transcripts` leaves the tombstone at home. + let c_root = std::env::temp_dir().join(format!("scl-tomb-nt-c-{pid}")); + let _ = std::fs::remove_dir_all(&c_root); + let (c, snap_c, tid_c, id_c) = origin_with_signed_transcript(&c_root); + let d_root = std::env::temp_dir().join(format!("scl-tomb-nt-d-{pid}")); + let _ = std::fs::remove_dir_all(&d_root); + let d = Repo::clone_to(&c_root, &d_root).unwrap(); + drop(c); + d.drop_transcript(&tid_c.to_hex(), Some(&id_c)).unwrap(); + std::fs::write(d_root.join("f2"), b"two").unwrap(); + d.commit("t", "c2").unwrap(); + d.push_opts("origin", true).unwrap(); + let c = Repo::open(&c_root).unwrap(); + assert!(!c + .store_arc() + .lock() + .unwrap() + .contains(&tombstone_id(snap_c, tid_c))); + assert!(!c.layout().transcript_pending_path().exists()); + + drop((b, c, d)); + for root in [&a_root, &b_root, &c_root, &d_root] { + std::fs::remove_dir_all(root).unwrap(); + } + } + #[test] fn signatures_ride_ssh_transport() { // The wire-protocol pipe harness `WireClient` <-> `wire::serve` IS diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 113b8cd4..a9d64a4c 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -116,6 +116,49 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { Ok(()) } +/// Most held-but-unhonored tombstones remembered (#230). A peer can push +/// arbitrarily many, so the set is a FIFO ring: the newest win, and an evicted +/// one is only lost to a push-only remote (a fetching peer is re-sent every +/// retained tombstone on each fetch). +const MAX_PENDING: usize = 1024; + +fn read_pending(layout: &Layout) -> Vec { + std::fs::read_to_string(layout.transcript_pending_path()) + .map(|s| s.lines().filter_map(|l| l.trim().parse().ok()).collect()) + .unwrap_or_default() +} + +fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { + let path = layout.transcript_pending_path(); + if ids.is_empty() { + return match std::fs::remove_file(&path) { + Err(e) if e.kind() != std::io::ErrorKind::NotFound => Err(Error::Io(e)), + _ => Ok(()), + }; + } + let out: String = ids.iter().map(|i| format!("{}\n", i.to_hex())).collect(); + scl_core::fsutil::atomic_write_durable(&path, out.as_bytes())?; + Ok(()) +} + +/// Remember `new` tombstones for re-evaluation, deduped, newest last, capped +/// at [`MAX_PENDING`]. Writes only when the set changes. +fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { + let mut all = read_pending(layout); + let before = all.len(); + for id in new { + if !all.contains(id) { + all.push(*id); + } + } + if all.len() == before { + return Ok(()); + } + let excess = all.len().saturating_sub(MAX_PENDING); + all.drain(..excess); + write_pending(layout, &all) +} + /// Rewrite the whole index file atomically (temp file + rename), so a /// concurrent reader never observes a half-written index. fn write_index(layout: &Layout, entries: &[(ObjectId, ObjectId)]) -> Result<()> { @@ -252,6 +295,8 @@ pub(crate) fn gc_prune(layout: &Layout, reachable: &mut BTreeSet) -> R // itself is never rooted, so it is swept after the grace window. let drops = read_drops(layout)?; reachable.extend(drops.iter().map(|(_, tomb, _)| *tomb)); + // Held-but-unhonored tombstones stay until trusted or evicted (#230). + reachable.extend(read_pending(layout)); let dropped: BTreeSet = drops.iter().map(|(_, _, d)| *d).collect(); let entries = read_raw(layout)?; let mut kept = Vec::with_capacity(entries.len()); @@ -291,11 +336,17 @@ pub(crate) fn index_incoming( let mut count = 0; let trusted = trusted_signers(layout); let sigs = crate::signatures::index_by_target(layout)?; + let mut held = Vec::new(); for id in ids { if let Object::Transcript(t) = store.get(id)? { match t.dropped_id() { Some(dropped) => { if !tombstone_honored(store, &trusted, &sigs, *id, &t, dropped)? { + // Held but inert. Remember a signed one: a push-only + // remote is never sent it again (#230). + if sigs.contains_key(id) { + held.push(*id); + } continue; } append_drop(layout, (t.snapshot, *id, dropped))? @@ -305,9 +356,47 @@ pub(crate) fn index_incoming( count += 1; } } + note_pending(layout, &held)?; Ok(count) } +/// Re-run the honoring checks on every remembered tombstone (#230): a signer +/// trusted since, or a snapshot or signature that arrived since, now lets the +/// drop take effect without the sender re-sending it. Idempotent; cost is +/// bounded by [`MAX_PENDING`] and zero when nothing is pending. An entry whose +/// object is gone, or that no longer decodes as a tombstone, is forgotten. +/// Returns how many drops were honored. +pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) -> Result { + let pending = read_pending(layout); + if pending.is_empty() { + return Ok(0); + } + let trusted = trusted_signers(layout); + let sigs = crate::signatures::index_by_target(layout)?; + let (mut keep, mut honored) = (Vec::new(), 0); + for id in &pending { + if !store.contains(id) { + continue; + } + let Ok(Object::Transcript(t)) = store.get(id) else { + continue; + }; + let Some(dropped) = t.dropped_id() else { + continue; + }; + if tombstone_honored(store, &trusted, &sigs, *id, &t, dropped)? { + append_drop(layout, (t.snapshot, *id, dropped))?; + honored += 1; + } else { + keep.push(*id); + } + } + if keep.len() != pending.len() { + write_pending(layout, &keep)?; + } + Ok(honored) +} + /// Full-store rebuild of the transcript index (clone path): scan every /// object the store can resolve (`Store::all_ids`, not a reachability walk /// — a `Transcript` is a leaf no tree/snapshot references), keep the ones @@ -361,17 +450,22 @@ pub(crate) fn indexed_transcript_ids_for( out.insert(tid); } } - // Signed tombstones travel with their snapshot so a drop propagates. An - // unsigned (local-only) drop is never sent: peers would ignore it, and an - // older peer would index it as a live transcript. + // Every retained signed tombstone travels on every transfer, not only + // those of `snapshots`: gc keeps a tombstone after it collects its + // snapshot, and a stale peer that still lists the body must still be told + // (#230). Cost: one tombstone plus one signature per drop, which receivers + // dedupe and ignore when they cannot use it. An unsigned (local-only) drop + // is never sent: peers would ignore it, and an older peer would index it + // as a live transcript. let drops = read_drops(layout)?; if !drops.is_empty() { let sigs = crate::signatures::index_by_target(layout)?; - for (snap, tomb, _) in drops { - if wanted.contains(&snap) && sigs.contains_key(&tomb) { - out.insert(tomb); - } - } + out.extend( + drops + .into_iter() + .filter(|(_, tomb, _)| sigs.contains_key(tomb)) + .map(|(_, tomb, _)| tomb), + ); } Ok(out.into_iter().collect()) } @@ -728,6 +822,11 @@ pub(crate) mod tests { .unwrap(); } + /// Number of tombstones `repo` is remembering for re-evaluation. + pub(crate) fn pending_count(repo: &Repo) -> usize { + read_pending(repo.layout()).len() + } + #[test] fn drop_hides_transcript_and_blocks_resurrection() { let root = tmp_root("drop"); @@ -995,4 +1094,120 @@ pub(crate) mod tests { drop(repo); std::fs::remove_dir_all(&root).unwrap(); } + + #[test] + fn sender_offers_every_retained_signed_tombstone_whatever_the_transfer_set() { + let root = tmp_root("send-all"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let signed = repo + .attach_transcript(snap, "a", "s", b"signed", &[owner.enc.public()]) + .unwrap(); + let unsigned = repo + .attach_transcript(snap, "a", "u", b"unsigned", &[owner.enc.public()]) + .unwrap(); + repo.drop_transcript(&signed.to_hex(), Some(&owner)) + .unwrap(); + repo.drop_transcript(&unsigned.to_hex(), None).unwrap(); + let drops = read_drops(repo.layout()).unwrap(); + let tomb = drops.iter().find(|d| d.2 == signed).unwrap().1; + + // `snap` is not in the transfer set (gc may have collected it). + let sent = indexed_transcript_ids_for(repo.layout(), &[]).unwrap(); + assert_eq!(sent, vec![tomb], "the signed tombstone only"); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + } + + #[test] + fn held_tombstone_is_honored_after_the_signer_is_trusted_and_never_before() { + let root = tmp_root("pending"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let (_s, stranger) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let arc = repo.store_arc(); + let put_tomb = |signer: Option<&scl_crypto::Identity>| { + let tomb = Transcript::drop_tombstone(snap, &body); + let id = arc.lock().unwrap().put(Object::Transcript(tomb)).unwrap(); + if let Some(who) = signer { + repo.sign_drop(id, who.signing.as_ref().unwrap()).unwrap(); + } + id + }; + + // Unsigned: inert and not remembered. Signed by nobody trusted: held. + let unsigned = put_tomb(None); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[unsigned]).unwrap(); + assert!(read_pending(repo.layout()).is_empty()); + drop(store); + let signed = put_tomb(Some(&owner)); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[signed]).unwrap(); + assert_eq!(read_pending(repo.layout()), vec![signed]); + assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 0); + assert!(read_drops(repo.layout()).unwrap().is_empty()); + drop(store); + + // A trusted stranger does not make the owner's tombstone count. + trust_signer(&repo, &stranger); + let mut store = arc.lock().unwrap(); + assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 0); + drop(store); + assert_eq!(load(repo.layout()).unwrap().len(), 1, "still visible"); + + // Gc keeps a held tombstone and its signature. + let mut reachable = BTreeSet::new(); + gc_prune(repo.layout(), &mut reachable).unwrap(); + assert!(reachable.contains(&signed)); + + trust_signer(&repo, &owner); + let mut store = arc.lock().unwrap(); + assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 1); + assert_eq!( + reevaluate_pending(repo.layout(), &mut store).unwrap(), + 0, + "idempotent: nothing is pending any more" + ); + drop(store); + assert!(load(repo.layout()).unwrap().is_empty(), "now hidden"); + assert!(!repo.layout().transcript_pending_path().exists()); + assert_eq!(read_drops(repo.layout()).unwrap().len(), 1); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + } + + #[test] + fn pending_set_is_a_bounded_fifo_and_forgets_missing_objects() { + let root = tmp_root("pending-cap"); + let repo = Repo::init(&root).unwrap(); + let ids: Vec = (0..MAX_PENDING + 5) + .map(|i| ObjectId::of(format!("t{i}").as_bytes())) + .collect(); + note_pending(repo.layout(), &ids).unwrap(); + note_pending(repo.layout(), &ids[10..13]).unwrap(); // dedup, no growth + let held = read_pending(repo.layout()); + assert_eq!(held.len(), MAX_PENDING); + assert_eq!(held[0], ids[5], "oldest evicted"); + assert_eq!(held[MAX_PENDING - 1], ids[MAX_PENDING + 4]); + + // None of these objects exist: a re-evaluation forgets them all. + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 0); + drop(store); + assert!(!repo.layout().transcript_pending_path().exists()); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + } } diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 1c4e6eaf..63f56ac8 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -609,6 +609,12 @@ fn ingest_pack_file_impl( // Receiver seam (P30 Task 4): same "ids just written" contract, for // Transcript objects. crate::transcripts::index_incoming(layout, store, &ids)?; + // Tombstones held from earlier transfers whose signer is trusted by now + // (#230). A `--no-transcripts` transfer indexes no transcript state. Best + // effort: the pack is already stored, so a failure here must not fail it. + if !skip_transcripts { + let _ = crate::transcripts::reevaluate_pending(layout, store); + } Ok(ids) } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 5d73de0d..8322abd4 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -207,9 +207,13 @@ too, not just the src-control-side metadata. transcripts on every peer that trusts it and make those peers' gc reclaim the bodies. Drop is not erasure: peers without the tombstone keep the body, `GetObjects` serves it until gc sweeps it, and a peer that still holds it can - re-send it. A tombstone ignored on arrival stays ignored on a push-only or - served remote, so configure `[signers] trusted` there before the first drop - push. Trust is checked at ingest and is not retroactive. + re-send it. A tombstone held while its signer is untrusted is remembered + (`.sc/transcript-pending`, capped at 1024, FIFO) and re-evaluated on open and + after each ingest, so trusting the signer later is enough on a push-only or + served remote. Every retained signed tombstone is sent on every transfer, so + a peer is told even after gc collected the snapshot; a peer that pushes junk + can only evict pending entries, not get one honored. Trust is checked at + evaluation and is not retroactive. ## Network transport (`sc serve --http` / `sc+http(s)://` / `ssh://`) — ADR-0022/0036/0040/0041/0042 diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index 2b0184de..ed639743 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -259,10 +259,11 @@ over-send, whatever order the lines landed in. adds the signer to the trust list and fetches). `push` never re-sends an object the remote already holds, so a remote that received the tombstone by push before trusting the signer keeps it inert. -- **sender:** the over-send for a transfer set includes the tombstones of its - snapshots that have an indexed signature, with those signatures. An - unsigned (local-only) drop is never sent. Fetching from the dropper (or a - push that moves the ref) propagates the drop. A peer that still holds the +- **sender:** the over-send of every transfer includes each retained + tombstone that has an indexed signature, with its signatures (selection + widened by #230, see the delivery addendum below; it was the tombstones of + the transfer set's snapshots). An unsigned (local-only) drop is never sent. + Fetching from the dropper (or a push that moves the ref) propagates the drop. A peer that still holds the transcript object hides it once it ingests an honored tombstone. `push` still skips all I/O when the remote tip already equals the local tip, so a drop alone does not push, the same pre-existing behavior as a retroactive @@ -303,12 +304,9 @@ machinery instead. cannot hide a transcript and gc never reclaims a body because of them. Operational consequences. A tombstone that arrives while its signer is -untrusted is ignored and not recorded. A fetching peer retries it on its next -fetch. A push-only or served remote has no retry: it already holds the object, -so a later push does not send it again and nothing re-runs the check, and the -drop stays inert there even after the signer is trusted (tracked in #230). -Configure -`[signers] trusted` on a remote before the first drop is pushed to it. Any key under `[signers] trusted` can hide transcripts on every peer that +untrusted is not recorded as a drop. A fetching peer retries it on its next +fetch. A push-only or served remote re-evaluates what it holds (delivery +addendum below, #230), so trusting the signer later is enough. Any key under `[signers] trusted` can hide transcripts on every peer that trusts it, and make that peer's gc reclaim the bodies. `GetObjects` still serves a dropped body until gc sweeps it, and a peer that still holds the body re-sends it on a later transfer, which restarts its grace window. All of @@ -338,14 +336,8 @@ dead**, as P16 keeps revocation tombstones forever. They are load-bearing against a later fetch of the same history from a stale peer, and cost about 100 bytes each. This is the one place the drop index outlives its snapshot; no transcript content does. -Retention is local protection only. The sender selects tombstones through the -snapshots of the transfer set, and a snapshot gc has removed is never in it, -so a stale peer that still lists the body is not told about a drop whose -snapshot the dropper has since collected. Sending every retained tombstone on -every transfer would close that, at the cost of one object per drop per fetch -for receivers that cannot use it; deferred to #230, together with the -push-only remote limit above (a tombstone rejected on arrival is never -re-evaluated there). +Retention alone protects only the dropper. Delivery of a tombstone whose +snapshot gc has collected is the #230 addendum below. ### `sc verify` @@ -415,3 +407,81 @@ failed. (Confirmed by test before the fix; the issue left this open.) would avoid the decode; deferred in `ROADMAP.md`. - `sc serve --store` uses the same `get_pack`, so served-bucket fetch and clone get the same behavior. + +## Addendum: tombstone delivery after gc and late trust (issue #230) + +Two gaps in the drop addendum above, closed with no `PROTOCOL_VERSION` bump, +no new object kind and no change to the trust gate. A tombstone is still +honored only when a drop-digest signature verifies under a key in the +receiver's `[signers] trusted`; unsigned drops still never leave the dropper. + +### 1. Send every retained signed tombstone + +`indexed_transcript_ids_for` (used by `LocalTransport::build_pack_tempfile` +for fetch/clone/serve and by `push_opts`) no longer selects tombstones +through the transfer set's snapshots. It returns every line of +`.sc/transcript-drops` that has an indexed signature, plus (as before) the +signatures over them. The bucket sender (`leaf_ids_for`) does the same. Why +not match against the advertised haves: the stale peer's snapshot is exactly +what the dropper's gc removed, so the dropper cannot resolve a have to it, +and an id-only match would need the dropper to keep the snapshot id forever. + +Cost and limits: + +- One tombstone (about 100 bytes) plus one signature per retained drop, per + transfer, regardless of branch. Nothing is capped: capping would silently + withhold a drop from a peer, and the set only grows by explicit drops (the + dropper's own and ones it honored from trusted signers). It is never + subtracted via the have set, like transcripts and signatures. +- A receiver that has the drop recorded dedups (`append_drop` writes only a + new line); one that cannot use it (snapshot unknown, signer untrusted) + stores the small object and ignores it. +- A push sends only tombstones the remote lacks (`has_object`), so repeat + pushes cost one existence check per drop. `--no-transcripts` still leaves + tombstones and their signatures out of both directions (#228). +- A fresh clone made after the dropper's gc receives retained tombstones + whose snapshots it never has. They stay inert and are remembered as + pending (below, capped), so they occupy ring slots without ever being + honored. + +### 2. Remember and re-evaluate held, unhonored tombstones + +A push-only or served remote holds the tombstone object from an earlier +push, so `push` (has-object) never re-sends it and `index_incoming` never +re-runs. Fix: a small local index plus re-evaluation. + +- `.sc/transcript-pending`: one tombstone object id per line, newest last. + `index_incoming` appends a tombstone that is held, **has an indexed + signature**, and failed `tombstone_honored`. Unsigned tombstones are never + remembered. A derived cache like `.sc/transcript-drops`: losing it only + loses the retry. +- `reevaluate_pending` re-runs the unchanged `tombstone_honored` on each + entry with the current `[signers] trusted` and signature index, records the + ones that now pass as drops and removes them from the file, forgets entries + whose object is gone or no longer a tombstone, and leaves the rest. It is + idempotent and does nothing (one failed file read) when no pending file + exists. It runs at `Repo::open` (holding the repo lock `open` takes) and at + the end of every ingest except a `--no-transcripts` one. Both are best + effort: a failure never fails an open or a transfer, and the next trigger + retries. The open trigger covers `sc serve` (per connection), `sc gc`, + `sc transcript list`, and every other command, so an operator who edits + `recipients.toml` needs no extra step; the ingest trigger covers a + long-lived server process. +- **Bounded.** The file is a FIFO ring of 1024 ids. A peer that pushes many + signed junk tombstones evicts older entries, so the work per trigger is at + most 1024 object reads and signature checks, and the file stays under + about 70 KiB. An evicted legitimate entry is lost only for a push-only + remote; a fetching peer is re-sent every retained tombstone (part 1). A + writer with push access can still push out a genuine pending entry; it + cannot make one honored. +- **gc** roots pending tombstones like drop lines, so they survive the + loose-object sweep and the signature index keeps their signatures + (`transcripts::gc_prune` runs before `signatures::gc_prune`). +- Trust is still checked at evaluation time and still not retroactive: + removing a signer later does not undo a recorded drop. + +### Not changed + +`push` still returns early when the remote tip equals the local tip, so a +drop alone does not push (unchanged). Clone `reindex` is unchanged: it +rebuilds drops from the store and does not touch the pending file. From fd9494c90747c0bd5fbc956720da9d6b8fd7dced Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 17:52:47 +0300 Subject: [PATCH 02/13] fix(repo): keep a pending tombstone on an unreadable object; record upgrade limit (ref #230) --- crates/repo/src/transcripts.rs | 4 +++- docs/THREAT-MODEL.md | 4 +++- docs/adr/0038-agent-session-transcripts.md | 9 +++++++++ docs/adr/0045-native-desktop-read-model.md | 5 ++++- 4 files changed, 19 insertions(+), 3 deletions(-) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index a9d64a4c..5d2e08b9 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -384,7 +384,9 @@ pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) - let Some(dropped) = t.dropped_id() else { continue; }; - if tombstone_honored(store, &trusted, &sigs, *id, &t, dropped)? { + // An unreadable object keeps the entry for the next trigger rather + // than aborting the rest. + if tombstone_honored(store, &trusted, &sigs, *id, &t, dropped).unwrap_or(false) { append_drop(layout, (t.snapshot, *id, dropped))?; honored += 1; } else { diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 8322abd4..a894c502 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -213,7 +213,9 @@ too, not just the src-control-side metadata. served remote. Every retained signed tombstone is sent on every transfer, so a peer is told even after gc collected the snapshot; a peer that pushes junk can only evict pending entries, not get one honored. Trust is checked at - evaluation and is not retroactive. + evaluation and is not retroactive. A tombstone a remote already held before + upgrading to a build with the pending index is not remembered: trust the + signer before the drop is pushed, or have a fetching peer re-send it. ## Network transport (`sc serve --http` / `sc+http(s)://` / `ssh://`) — ADR-0022/0036/0040/0041/0042 diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index ed639743..3e9c84d8 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -482,6 +482,15 @@ re-runs. Fix: a small local index plus re-evaluation. ### Not changed +A tombstone a repo already held when it upgraded to this build has no pending +entry, so a push-only remote in that state still needs the signer trusted +before the drop is pushed (a fetching peer is re-sent it and recovers). A +one-time full-store scan would fix it but is unbounded, so it is not done. +`Repo::open` now may update the derived drop indexes under the lock it already +takes; failures there are ignored, not reported. A pending entry also costs +re-evaluation work on each ingest, including a lazy by-id gap fill, up to the +1024 cap. + `push` still returns early when the remote tip equals the local tip, so a drop alone does not push (unchanged). Clone `reindex` is unchanged: it rebuilds drops from the store and does not touch the pending file. diff --git a/docs/adr/0045-native-desktop-read-model.md b/docs/adr/0045-native-desktop-read-model.md index a3a1e014..8ebdbf39 100644 --- a/docs/adr/0045-native-desktop-read-model.md +++ b/docs/adr/0045-native-desktop-read-model.md @@ -66,7 +66,10 @@ run on the same machine. - Protected/private plaintext cannot enter frontend state in this phase. - Oversized public blobs remain native objects but do not enter frontend state. - Opening a query briefly acquires the repository's existing single-writer - lock; a future long-lived read handle needs its own concurrency ADR. + lock; a future long-lived read handle needs its own concurrency ADR. That + open can also update the derived `.sc/transcript-drops` and + `.sc/transcript-pending` indexes (ADR-0038, #230), best effort; the app + still writes no refs, objects or working-tree files. - Mutation and authorized decryption remain future slices with separate threat modeling. From 4fdc63c341f900e0b5b9ca7400fd1e50c35f1101 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 17:56:38 +0300 Subject: [PATCH 03/13] docs,test: sync the drop addendum with #230; assert test roots are gone (ref #230) --- crates/repo/src/sync.rs | 6 ++++++ crates/repo/src/transcripts.rs | 3 +++ docs/adr/0038-agent-session-transcripts.md | 7 ++++--- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index e63f1116..96011689 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -1334,7 +1334,9 @@ mod tests { drop(b); std::fs::remove_dir_all(&a_root).unwrap(); + assert!(!a_root.exists()); std::fs::remove_dir_all(&b_root).unwrap(); + assert!(!b_root.exists()); } #[test] @@ -1395,6 +1397,7 @@ mod tests { drop((b, c, d)); for root in [&a_root, &b_root, &c_root, &d_root] { std::fs::remove_dir_all(root).unwrap(); + assert!(!root.exists()); } } @@ -1421,7 +1424,9 @@ mod tests { drop(b); std::fs::remove_dir_all(&a_root).unwrap(); + assert!(!a_root.exists()); std::fs::remove_dir_all(&b_root).unwrap(); + assert!(!b_root.exists()); } #[test] @@ -1467,6 +1472,7 @@ mod tests { drop((b, c, d)); for root in [&a_root, &b_root, &c_root, &d_root] { std::fs::remove_dir_all(root).unwrap(); + assert!(!root.exists()); } } diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 5d2e08b9..d71460c3 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -1122,6 +1122,7 @@ pub(crate) mod tests { drop(repo); std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); } #[test] @@ -1186,6 +1187,7 @@ pub(crate) mod tests { drop(repo); std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); } #[test] @@ -1211,5 +1213,6 @@ pub(crate) mod tests { drop(repo); std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); } } diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index 3e9c84d8..affa4d50 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -257,8 +257,9 @@ over-send, whatever order the lines landed in. A tombstone that fails the checks stays an inert object. A fetching peer re-evaluates it whenever it is sent again (for example after the operator adds the signer to the trust list and fetches). `push` never re-sends an - object the remote already holds, so a remote that received the tombstone by - push before trusting the signer keeps it inert. + object the remote already holds; a remote that received the tombstone by + push before trusting the signer re-evaluates it from + `.sc/transcript-pending` (#230, delivery addendum below). - **sender:** the over-send of every transfer includes each retained tombstone that has an indexed signature, with its signatures (selection widened by #230, see the delivery addendum below; it was the tombstones of @@ -315,7 +316,7 @@ this follows from drop not being erasure. Trade-offs. Trust is local configuration, not repository content, so a fresh clone ignores tombstones until its operator trusts the signer and fetches again (a clone of a store that already honored the drop never receives the -body in the first place). Trust is checked at ingest, not retroactively, so removing a signer +body in the first place). Trust is checked when a tombstone is evaluated (at ingest, or on a later open or ingest for a held one, #230), not retroactively, so removing a signer later does not undo drops already recorded. "Trusted signer" is the same project-level authority that can sign commits; it is deliberately not narrowed to the transcript's recipients or its own signer, because From 20eeae7e2a5554aa40e7eda96be56ef185648473 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 18:14:26 +0300 Subject: [PATCH 04/13] fix(repo): address #230 review: serve-path trigger, late signatures, bounded batch and signature work, keep unreadable entries --- crates/repo/src/sync.rs | 33 ++++++ crates/repo/src/transcripts.rs | 112 +++++++++++++++++---- crates/repo/src/transport.rs | 12 ++- docs/adr/0038-agent-session-transcripts.md | 22 ++-- 4 files changed, 153 insertions(+), 26 deletions(-) diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index 96011689..eb7300a5 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -1401,6 +1401,39 @@ mod tests { } } + #[test] + fn served_remote_honors_a_held_tombstone_when_a_transport_opens_it() { + // A `sc serve` or local-path remote never goes through `Repo::open`. + let pid = std::process::id(); + let a_root = std::env::temp_dir().join(format!("scl-tomb-serve-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-tomb-serve-b-{pid}")); + let _ = std::fs::remove_dir_all(&b_root); + let (a, snap, tid, identity) = origin_with_signed_transcript(&a_root); + let b = Repo::clone_to(&a_root, &b_root).unwrap(); + drop(a); + b.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(b_root.join("f2"), b"two").unwrap(); + b.commit("t", "c2").unwrap(); + b.push("origin").unwrap(); + + let a = Repo::open(&a_root).unwrap(); + crate::transcripts::tests::trust_signer(&a, &identity); + drop(a); + let _served = crate::transport::LocalTransport::open(&a_root).unwrap(); + assert!(!crate::layout::Layout::at(&a_root) + .transcript_pending_path() + .exists()); + drop(_served); + let a = Repo::open(&a_root).unwrap(); + assert!(a.transcripts_for(&snap).unwrap().is_empty()); + + drop((a, b)); + for root in [&a_root, &b_root] { + std::fs::remove_dir_all(root).unwrap(); + assert!(!root.exists()); + } + } + #[test] fn a_peer_that_never_trusts_the_signer_never_honors_the_tombstone() { let pid = std::process::id(); diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index d71460c3..a78535c0 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -122,6 +122,11 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { /// retained tombstone on each fetch). const MAX_PENDING: usize = 1024; +/// Signatures read per pending tombstone on re-evaluation. A peer can attach +/// any number of junk signature objects to a target, and this runs on every +/// open, so the work is capped. A real tombstone carries one or two. +const MAX_PENDING_SIGS: usize = 16; + fn read_pending(layout: &Layout) -> Vec { std::fs::read_to_string(layout.transcript_pending_path()) .map(|s| s.lines().filter_map(|l| l.trim().parse().ok()).collect()) @@ -144,14 +149,18 @@ fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { /// Remember `new` tombstones for re-evaluation, deduped, newest last, capped /// at [`MAX_PENDING`]. Writes only when the set changes. fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { + // Only the newest MAX_PENDING of a (peer-sized) batch can survive. + let new = &new[new.len().saturating_sub(MAX_PENDING)..]; let mut all = read_pending(layout); - let before = all.len(); + let mut seen: BTreeSet = all.iter().copied().collect(); + let mut added = false; for id in new { - if !all.contains(id) { + if seen.insert(*id) { all.push(*id); + added = true; } } - if all.len() == before { + if !added { return Ok(()); } let excess = all.len().saturating_sub(MAX_PENDING); @@ -337,23 +346,35 @@ pub(crate) fn index_incoming( let trusted = trusted_signers(layout); let sigs = crate::signatures::index_by_target(layout)?; let mut held = Vec::new(); + // Honor a tombstone, or remember a signed one that is held but inert: a + // push-only remote is never sent it again (#230). + let mut consider = |store: &mut scl_core::Store, id: ObjectId, t: &Transcript, d: ObjectId| { + if tombstone_honored(store, &trusted, &sigs, id, t, d)? { + append_drop(layout, (t.snapshot, id, d))?; + } else if sigs.contains_key(&id) { + held.push(id); + } + Ok::<_, Error>(()) + }; for id in ids { - if let Object::Transcript(t) = store.get(id)? { - match t.dropped_id() { - Some(dropped) => { - if !tombstone_honored(store, &trusted, &sigs, *id, &t, dropped)? { - // Held but inert. Remember a signed one: a push-only - // remote is never sent it again (#230). - if sigs.contains_key(id) { - held.push(*id); - } - continue; + match store.get(id)? { + Object::Transcript(t) => match t.dropped_id() { + Some(dropped) => consider(store, *id, &t, dropped)?, + None => { + append_index(layout, t.snapshot, *id)?; + count += 1; + } + }, + // A signature that lands after the tombstone it covers (separate + // transfers) is a reason to look at that tombstone again. + Object::Signature(s) if store.contains(&s.snapshot) => { + if let Object::Transcript(t) = store.get(&s.snapshot)? { + if let Some(dropped) = t.dropped_id() { + consider(store, s.snapshot, &t, dropped)?; } - append_drop(layout, (t.snapshot, *id, dropped))? } - None => append_index(layout, t.snapshot, *id)?, } - count += 1; + _ => {} } } note_pending(layout, &held)?; @@ -372,14 +393,25 @@ pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) - return Ok(0); } let trusted = trusted_signers(layout); - let sigs = crate::signatures::index_by_target(layout)?; + let mut sigs = crate::signatures::index_by_target(layout)?; + let wanted: BTreeSet = pending.iter().copied().collect(); + sigs.retain(|target, ids| { + ids.truncate(MAX_PENDING_SIGS); + wanted.contains(target) + }); let (mut keep, mut honored) = (Vec::new(), 0); for id in &pending { if !store.contains(id) { continue; } - let Ok(Object::Transcript(t)) = store.get(id) else { - continue; + let t = match store.get(id) { + Ok(Object::Transcript(t)) => t, + Ok(_) => continue, + // Unreadable now, maybe fine later: keep it. + Err(_) => { + keep.push(*id); + continue; + } }; let Some(dropped) = t.dropped_id() else { continue; @@ -1190,6 +1222,48 @@ pub(crate) mod tests { assert!(!root.exists()); } + #[test] + fn a_signature_that_lands_after_its_held_tombstone_makes_it_pending() { + let root = tmp_root("late-sig"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let tomb = repo + .store_arc() + .lock() + .unwrap() + .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .unwrap(); + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + assert!(read_pending(repo.layout()).is_empty(), "unsigned: not kept"); + drop(store); + + // The signature arrives in a later transfer, without the tombstone. + repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) + .unwrap(); + let sig = crate::signatures::index_by_target(repo.layout()).unwrap()[&tomb][0]; + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[sig]).unwrap(); + assert_eq!(read_pending(repo.layout()), vec![tomb]); + drop(store); + + trust_signer(&repo, &owner); + let mut store = arc.lock().unwrap(); + assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 1); + drop(store); + assert!(load(repo.layout()).unwrap().is_empty()); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + #[test] fn pending_set_is_a_bounded_fifo_and_forgets_missing_objects() { let root = tmp_root("pending-cap"); diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 63f56ac8..b87f0dd7 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -156,7 +156,17 @@ impl LocalTransport { } // Match the repo's store budget so a single large object never fails to resolve // (a blob bigger than the whole budget would BudgetExceed). - let store = Store::open_persistent(layout.objects_dir(), crate::repo::default_budget())?; + let mut store = + Store::open_persistent(layout.objects_dir(), crate::repo::default_budget())?; + // A served or local-path remote never passes through `Repo::open`: + // honor held tombstones whose signer was trusted since (#230). Best + // effort under the repo lock; a busy lock means a `Repo` is open and + // ran it already. + if layout.transcript_pending_path().exists() { + if let Ok(_lock) = RepoLock::acquire(&layout) { + let _ = crate::transcripts::reevaluate_pending(&layout, &mut store); + } + } Ok(LocalTransport { layout, store: RefCell::new(store), diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index affa4d50..d4d8a766 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -454,15 +454,20 @@ re-runs. Fix: a small local index plus re-evaluation. - `.sc/transcript-pending`: one tombstone object id per line, newest last. `index_incoming` appends a tombstone that is held, **has an indexed signature**, and failed `tombstone_honored`. Unsigned tombstones are never - remembered. A derived cache like `.sc/transcript-drops`: losing it only + remembered. A signature that arrives later, in a transfer without the + tombstone, is also a trigger: `index_incoming` looks up the tombstone it + covers. A derived cache like `.sc/transcript-drops`: losing it only loses the retry. - `reevaluate_pending` re-runs the unchanged `tombstone_honored` on each entry with the current `[signers] trusted` and signature index, records the ones that now pass as drops and removes them from the file, forgets entries - whose object is gone or no longer a tombstone, and leaves the rest. It is + whose object is gone or no longer a tombstone, and leaves the rest (an + object that fails to read stays, to be tried again). It is idempotent and does nothing (one failed file read) when no pending file - exists. It runs at `Repo::open` (holding the repo lock `open` takes) and at - the end of every ingest except a `--no-transcripts` one. Both are best + exists. It runs at `Repo::open` (holding the repo lock `open` takes), at + `LocalTransport::open` (what `sc serve` and a local-path remote use instead + of `Repo::open`; it takes the repo lock itself and skips if it is busy), and + at the end of every ingest except a `--no-transcripts` one. Both are best effort: a failure never fails an open or a transfer, and the next trigger retries. The open trigger covers `sc serve` (per connection), `sc gc`, `sc transcript list`, and every other command, so an operator who edits @@ -470,8 +475,13 @@ re-runs. Fix: a small local index plus re-evaluation. long-lived server process. - **Bounded.** The file is a FIFO ring of 1024 ids. A peer that pushes many signed junk tombstones evicts older entries, so the work per trigger is at - most 1024 object reads and signature checks, and the file stays under - about 70 KiB. An evicted legitimate entry is lost only for a push-only + most 1024 tombstone reads and, per tombstone, its first 16 indexed + signatures (`MAX_PENDING_SIGS`; a peer can attach any number of junk + signature objects to a target, and this runs on every open), and the file + stays under about 70 KiB. The cap means 16 junk signatures indexed before + the real one keep that drop inert here; a fetch from the dropper still + evaluates it uncapped through `index_incoming`. A pack's batch is + deduplicated and cut to the newest 1024 before it touches the file. An evicted legitimate entry is lost only for a push-only remote; a fetching peer is re-sent every retained tombstone (part 1). A writer with push access can still push out a genuine pending entry; it cannot make one honored. From 722c475b6495d401a90221e03bd98abf231d22b9 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 18:20:55 +0300 Subject: [PATCH 05/13] fix(repo): cap signature work inside tombstone_honored after selecting trusted signers; dedupe signature-driven lookups (ref #230) --- crates/repo/src/transcripts.rs | 162 +++++++++++++++++++-- docs/adr/0038-agent-session-transcripts.md | 29 ++-- 2 files changed, 167 insertions(+), 24 deletions(-) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index a78535c0..d08b733f 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -122,10 +122,17 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { /// retained tombstone on each fetch). const MAX_PENDING: usize = 1024; -/// Signatures read per pending tombstone on re-evaluation. A peer can attach -/// any number of junk signature objects to a target, and this runs on every -/// open, so the work is capped. A real tombstone carries one or two. -const MAX_PENDING_SIGS: usize = 16; +/// Signature objects `tombstone_honored` reads per tombstone, and how many of +/// those that claim a trusted key it verifies. A peer can attach any number of +/// junk signature objects to a target and this runs on every open, so the work +/// is capped; a real tombstone carries one or two. Signatures by an untrusted +/// key cost one read and never count against the verification cap, so they +/// cannot push a trusted one out of it. +const MAX_SIG_READS: usize = 256; +const MAX_SIG_VERIFIES: usize = 8; + +/// Distinct signature targets `index_incoming` looks up per batch. +const MAX_SIG_LOOKUPS: usize = 1024; fn read_pending(layout: &Layout) -> Vec { std::fs::read_to_string(layout.transcript_pending_path()) @@ -267,13 +274,25 @@ fn tombstone_honored( _ => return Ok(false), } } - for sid in sigs.get(&tomb).map(Vec::as_slice).unwrap_or_default() { + let mut verifies = 0; + for sid in sigs + .get(&tomb) + .map(Vec::as_slice) + .unwrap_or_default() + .iter() + .take(MAX_SIG_READS) + { if let Object::Signature(sig) = store.get(sid)? { - if trusted.contains_key(&sig.signer) - && scl_crypto::verify_transcript_sig(&sig.signer, &drop_digest(&tomb), &sig.sig) - { + if !trusted.contains_key(&sig.signer) { + continue; + } + if scl_crypto::verify_transcript_sig(&sig.signer, &drop_digest(&tomb), &sig.sig) { return Ok(true); } + verifies += 1; + if verifies >= MAX_SIG_VERIFIES { + break; + } } } Ok(false) @@ -356,10 +375,18 @@ pub(crate) fn index_incoming( } Ok::<_, Error>(()) }; + // Each tombstone is considered once per batch, and the signature-driven + // lookups (one object read per distinct target) are capped. + let mut considered = BTreeSet::new(); + let mut lookups = 0; for id in ids { match store.get(id)? { Object::Transcript(t) => match t.dropped_id() { - Some(dropped) => consider(store, *id, &t, dropped)?, + Some(dropped) => { + if considered.insert(*id) { + consider(store, *id, &t, dropped)? + } + } None => { append_index(layout, t.snapshot, *id)?; count += 1; @@ -367,9 +394,15 @@ pub(crate) fn index_incoming( }, // A signature that lands after the tombstone it covers (separate // transfers) is a reason to look at that tombstone again. - Object::Signature(s) if store.contains(&s.snapshot) => { + Object::Signature(s) + if lookups < MAX_SIG_LOOKUPS + && !considered.contains(&s.snapshot) + && store.contains(&s.snapshot) => + { + lookups += 1; if let Object::Transcript(t) = store.get(&s.snapshot)? { if let Some(dropped) = t.dropped_id() { + considered.insert(s.snapshot); consider(store, s.snapshot, &t, dropped)?; } } @@ -395,10 +428,7 @@ pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) - let trusted = trusted_signers(layout); let mut sigs = crate::signatures::index_by_target(layout)?; let wanted: BTreeSet = pending.iter().copied().collect(); - sigs.retain(|target, ids| { - ids.truncate(MAX_PENDING_SIGS); - wanted.contains(target) - }); + sigs.retain(|target, _| wanted.contains(target)); let (mut keep, mut honored) = (Vec::new(), 0); for id in &pending { if !store.contains(id) { @@ -1264,6 +1294,110 @@ pub(crate) mod tests { assert!(!root.exists()); } + #[test] + fn a_flood_of_untrusted_signatures_cannot_keep_a_trusted_one_from_honoring() { + let root = tmp_root("sig-flood"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let tomb = repo + .store_arc() + .lock() + .unwrap() + .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .unwrap(); + // Far more junk signatures by untrusted keys than any cap, indexed + // before the real one. + let arc = repo.store_arc(); + let mut junk_ids = Vec::new(); + for i in 0..(MAX_SIG_READS + 40) { + let mut signer = [0u8; 32]; + signer[..8].copy_from_slice(&(i as u64 + 1).to_le_bytes()); + let junk = SignatureObj { + snapshot: tomb, + signer, + sig: [7u8; 64], + }; + junk_ids.push(arc.lock().unwrap().put(Object::Signature(junk)).unwrap()); + } + let mut store = arc.lock().unwrap(); + crate::signatures::index_incoming(repo.layout(), &mut store, &junk_ids).unwrap(); + drop(store); + repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) + .unwrap(); + trust_signer(&repo, &owner); + let sigs = crate::signatures::index_by_target(repo.layout()).unwrap(); + assert!( + sigs[&tomb].len() > MAX_SIG_READS, + "real one sits past the read cap" + ); + + // Reading is capped, so put the real one first to model the normal + // case (it arrives with the tombstone): junk afterwards is harmless. + let mut ordered = sigs.clone(); + let last = ordered.get_mut(&tomb).unwrap().pop().unwrap(); + ordered.get_mut(&tomb).unwrap().insert(0, last); + let mut store = arc.lock().unwrap(); + let trusted = trusted_signers(repo.layout()); + let t = match store.get(&tomb).unwrap() { + Object::Transcript(t) => t, + _ => unreachable!(), + }; + assert!(tombstone_honored(&mut store, &trusted, &ordered, tomb, &t, body).unwrap()); + + // Untrusted junk ahead of a trusted signature is skipped for free: + // 20 junk signatures put the real one at index 20 (past the 8 + // verification cap, inside the read cap). + let mut mid = sigs.clone(); + let v = mid.get_mut(&tomb).unwrap(); + let last = v.pop().unwrap(); + v.truncate(20); + v.push(last); + assert!(tombstone_honored(&mut store, &trusted, &mid, tomb, &t, body).unwrap()); + drop(store); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn signature_lookups_per_batch_are_deduplicated_and_capped() { + let root = tmp_root("sig-batch"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let arc = repo.store_arc(); + // Many signatures over one snapshot: one lookup, not one per signature. + let ids: Vec = (0..50u64) + .map(|i| { + let mut signer = [0u8; 32]; + signer[..8].copy_from_slice(&(i + 1).to_le_bytes()); + arc.lock() + .unwrap() + .put(Object::Signature(SignatureObj { + snapshot: snap, + signer, + sig: [1u8; 64], + })) + .unwrap() + }) + .collect(); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &ids).unwrap(); + drop(store); + assert!(read_pending(repo.layout()).is_empty()); + assert!(read_drops(repo.layout()).unwrap().is_empty()); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + #[test] fn pending_set_is_a_bounded_fifo_and_forgets_missing_objects() { let root = tmp_root("pending-cap"); diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index d4d8a766..e5f8b809 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -475,16 +475,25 @@ re-runs. Fix: a small local index plus re-evaluation. long-lived server process. - **Bounded.** The file is a FIFO ring of 1024 ids. A peer that pushes many signed junk tombstones evicts older entries, so the work per trigger is at - most 1024 tombstone reads and, per tombstone, its first 16 indexed - signatures (`MAX_PENDING_SIGS`; a peer can attach any number of junk - signature objects to a target, and this runs on every open), and the file - stays under about 70 KiB. The cap means 16 junk signatures indexed before - the real one keep that drop inert here; a fetch from the dropper still - evaluates it uncapped through `index_incoming`. A pack's batch is - deduplicated and cut to the newest 1024 before it touches the file. An evicted legitimate entry is lost only for a push-only - remote; a fetching peer is re-sent every retained tombstone (part 1). A - writer with push access can still push out a genuine pending entry; it - cannot make one honored. + most 1024 tombstone reads plus the per-tombstone signature work below. + A pack's batch is deduplicated and cut to the newest 1024 before it + touches the file; the file stays under about 70 KiB. An evicted legitimate + entry is lost only for a push-only remote; a fetching peer is re-sent every + retained tombstone (part 1). A writer with push access can still push out a + genuine pending entry; it cannot make one honored. +- **Signature work is capped inside `tombstone_honored`** (so it bounds + ingest and re-evaluation alike): at most `MAX_SIG_READS` (256) indexed + signatures are read per tombstone, and at most `MAX_SIG_VERIFIES` (8) of + those that claim a trusted key are verified. A signature by an untrusted + key costs one read and does not count against the verification cap, so a + flood from untrusted keys cannot push a trusted signature out of it. Limit: + a writer who indexes more than 256 junk signatures, or 8 junk ones claiming + a trusted key (public, since it appears on every signature), ahead of the + real one keeps the drop inert on that peer. That is a denial of the drop, + never an honored one, and the real signature normally arrives with its + tombstone, before any junk. Signature-driven lookups in `index_incoming` + (a signature that lands after its tombstone) read each distinct target at + most once per batch and at most 1024 targets per batch. - **gc** roots pending tombstones like drop lines, so they survive the loose-object sweep and the signature index keeps their signatures (`transcripts::gc_prune` runs before `signatures::gc_prune`). From 6fc997bccdb3623a15984f145606aabf1910feb9 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 18:59:45 +0300 Subject: [PATCH 06/13] fix(repo): skip recorded drops on ingest, dedupe pending batches, fail gc on unreadable pending list (ref #230) --- crates/repo/src/signatures.rs | 20 +++++ crates/repo/src/transcripts.rs | 91 +++++++++++++++++----- docs/adr/0038-agent-session-transcripts.md | 14 ++++ 3 files changed, 107 insertions(+), 18 deletions(-) diff --git a/crates/repo/src/signatures.rs b/crates/repo/src/signatures.rs index 5ee033db..b0c90436 100644 --- a/crates/repo/src/signatures.rs +++ b/crates/repo/src/signatures.rs @@ -195,6 +195,26 @@ pub(crate) fn index_by_target( Ok(out) } +/// [`index_by_target`] restricted to `targets`, keeping at most `per_target` +/// signature ids each (index order). Only the wanted entries are retained, so +/// a large index costs a scan, not memory. +pub(crate) fn index_for_targets( + layout: &Layout, + targets: &BTreeSet, + per_target: usize, +) -> Result>> { + let mut out: std::collections::HashMap> = Default::default(); + for (target, sig_id) in read_index(layout)? { + if targets.contains(&target) { + let v = out.entry(target).or_default(); + if v.len() < per_target { + v.push(sig_id); + } + } + } + Ok(out) +} + /// Sender-side seam (Task 3): the indexed signature object ids covering any /// of `snapshots`, deduped and sorted — the set a push/export needs to /// carry alongside the snapshots themselves so the receiving repo's index diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index d08b733f..150fc6d7 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -134,10 +134,14 @@ const MAX_SIG_VERIFIES: usize = 8; /// Distinct signature targets `index_incoming` looks up per batch. const MAX_SIG_LOOKUPS: usize = 1024; -fn read_pending(layout: &Layout) -> Vec { - std::fs::read_to_string(layout.transcript_pending_path()) - .map(|s| s.lines().filter_map(|l| l.trim().parse().ok()).collect()) - .unwrap_or_default() +/// A missing file is empty; any other read failure is an error, so gc never +/// treats an unreadable pending list as empty and sweeps what it roots. +fn read_pending(layout: &Layout) -> Result> { + match std::fs::read_to_string(layout.transcript_pending_path()) { + Ok(s) => Ok(s.lines().filter_map(|l| l.trim().parse().ok()).collect()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(Vec::new()), + Err(e) => Err(Error::Io(e)), + } } fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { @@ -156,9 +160,12 @@ fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { /// Remember `new` tombstones for re-evaluation, deduped, newest last, capped /// at [`MAX_PENDING`]. Writes only when the set changes. fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { - // Only the newest MAX_PENDING of a (peer-sized) batch can survive. + // Dedupe the batch (order kept), then only its newest MAX_PENDING can + // survive. + let mut once = BTreeSet::new(); + let new: Vec = new.iter().copied().filter(|i| once.insert(*i)).collect(); let new = &new[new.len().saturating_sub(MAX_PENDING)..]; - let mut all = read_pending(layout); + let mut all = read_pending(layout)?; let mut seen: BTreeSet = all.iter().copied().collect(); let mut added = false; for id in new { @@ -324,7 +331,7 @@ pub(crate) fn gc_prune(layout: &Layout, reachable: &mut BTreeSet) -> R let drops = read_drops(layout)?; reachable.extend(drops.iter().map(|(_, tomb, _)| *tomb)); // Held-but-unhonored tombstones stay until trusted or evicted (#230). - reachable.extend(read_pending(layout)); + reachable.extend(read_pending(layout)?); let dropped: BTreeSet = drops.iter().map(|(_, _, d)| *d).collect(); let entries = read_raw(layout)?; let mut kept = Vec::with_capacity(entries.len()); @@ -365,19 +372,30 @@ pub(crate) fn index_incoming( let trusted = trusted_signers(layout); let sigs = crate::signatures::index_by_target(layout)?; let mut held = Vec::new(); + // Tombstones already recorded as drops need nothing: every fetch re-sends + // them all, so skipping them keeps a resend O(new) rather than O(drops^2). + let mut recorded: BTreeSet = read_drops(layout)? + .into_iter() + .map(|(_, tomb, _)| tomb) + .collect(); // Honor a tombstone, or remember a signed one that is held but inert: a // push-only remote is never sent it again (#230). let mut consider = |store: &mut scl_core::Store, id: ObjectId, t: &Transcript, d: ObjectId| { + if recorded.contains(&id) { + return Ok::<_, Error>(()); + } if tombstone_honored(store, &trusted, &sigs, id, t, d)? { append_drop(layout, (t.snapshot, id, d))?; + recorded.insert(id); } else if sigs.contains_key(&id) { held.push(id); } - Ok::<_, Error>(()) + Ok(()) }; // Each tombstone is considered once per batch, and the signature-driven // lookups (one object read per distinct target) are capped. let mut considered = BTreeSet::new(); + let mut looked_up = BTreeSet::new(); let mut lookups = 0; for id in ids { match store.get(id)? { @@ -397,6 +415,7 @@ pub(crate) fn index_incoming( Object::Signature(s) if lookups < MAX_SIG_LOOKUPS && !considered.contains(&s.snapshot) + && looked_up.insert(s.snapshot) && store.contains(&s.snapshot) => { lookups += 1; @@ -421,14 +440,13 @@ pub(crate) fn index_incoming( /// object is gone, or that no longer decodes as a tombstone, is forgotten. /// Returns how many drops were honored. pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) -> Result { - let pending = read_pending(layout); + let pending = read_pending(layout)?; if pending.is_empty() { return Ok(0); } let trusted = trusted_signers(layout); - let mut sigs = crate::signatures::index_by_target(layout)?; let wanted: BTreeSet = pending.iter().copied().collect(); - sigs.retain(|target, _| wanted.contains(target)); + let sigs = crate::signatures::index_for_targets(layout, &wanted, MAX_SIG_READS)?; let (mut keep, mut honored) = (Vec::new(), 0); for id in &pending { if !store.contains(id) { @@ -888,7 +906,7 @@ pub(crate) mod tests { /// Number of tombstones `repo` is remembering for re-evaluation. pub(crate) fn pending_count(repo: &Repo) -> usize { - read_pending(repo.layout()).len() + read_pending(repo.layout()).unwrap().len() } #[test] @@ -1212,12 +1230,12 @@ pub(crate) mod tests { let unsigned = put_tomb(None); let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[unsigned]).unwrap(); - assert!(read_pending(repo.layout()).is_empty()); + assert!(read_pending(repo.layout()).unwrap().is_empty()); drop(store); let signed = put_tomb(Some(&owner)); let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[signed]).unwrap(); - assert_eq!(read_pending(repo.layout()), vec![signed]); + assert_eq!(read_pending(repo.layout()).unwrap(), vec![signed]); assert_eq!(reevaluate_pending(repo.layout(), &mut store).unwrap(), 0); assert!(read_drops(repo.layout()).unwrap().is_empty()); drop(store); @@ -1271,7 +1289,10 @@ pub(crate) mod tests { let arc = repo.store_arc(); let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); - assert!(read_pending(repo.layout()).is_empty(), "unsigned: not kept"); + assert!( + read_pending(repo.layout()).unwrap().is_empty(), + "unsigned: not kept" + ); drop(store); // The signature arrives in a later transfer, without the tombstone. @@ -1280,7 +1301,7 @@ pub(crate) mod tests { let sig = crate::signatures::index_by_target(repo.layout()).unwrap()[&tomb][0]; let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[sig]).unwrap(); - assert_eq!(read_pending(repo.layout()), vec![tomb]); + assert_eq!(read_pending(repo.layout()).unwrap(), vec![tomb]); drop(store); trust_signer(&repo, &owner); @@ -1390,7 +1411,7 @@ pub(crate) mod tests { let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &ids).unwrap(); drop(store); - assert!(read_pending(repo.layout()).is_empty()); + assert!(read_pending(repo.layout()).unwrap().is_empty()); assert!(read_drops(repo.layout()).unwrap().is_empty()); drop(repo); @@ -1398,6 +1419,40 @@ pub(crate) mod tests { assert!(!root.exists()); } + #[test] + fn a_recorded_drop_is_not_requeued_and_a_batch_is_deduped_before_the_cap() { + let root = tmp_root("recorded"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + repo.drop_transcript(&body.to_hex(), Some(&owner)).unwrap(); + let tomb = read_drops(repo.layout()).unwrap()[0].1; + + // Signer no longer trusted, tombstone re-sent: already recorded, so + // it is not queued again. + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + drop(store); + assert!(read_pending(repo.layout()).unwrap().is_empty()); + + // The same id repeated fills no more than one slot. + let a = ObjectId::of(b"a"); + let b = ObjectId::of(b"b"); + let mut batch = vec![a]; + batch.extend(std::iter::repeat_n(b, MAX_PENDING + 10)); + note_pending(repo.layout(), &batch).unwrap(); + assert_eq!(read_pending(repo.layout()).unwrap(), vec![a, b]); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + #[test] fn pending_set_is_a_bounded_fifo_and_forgets_missing_objects() { let root = tmp_root("pending-cap"); @@ -1407,7 +1462,7 @@ pub(crate) mod tests { .collect(); note_pending(repo.layout(), &ids).unwrap(); note_pending(repo.layout(), &ids[10..13]).unwrap(); // dedup, no growth - let held = read_pending(repo.layout()); + let held = read_pending(repo.layout()).unwrap(); assert_eq!(held.len(), MAX_PENDING); assert_eq!(held[0], ids[5], "oldest evicted"); assert_eq!(held[MAX_PENDING - 1], ids[MAX_PENDING + 4]); diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index e5f8b809..2302f0a9 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -500,6 +500,20 @@ re-runs. Fix: a small local index plus re-evaluation. - Trust is still checked at evaluation time and still not retroactive: removing a signer later does not undo a recorded drop. +### Review notes (accepted limits) + +- A receiver skips a tombstone it has already recorded as a drop, so the + all-drops resend costs the recorded set one read per ingest, not a check and + a file rewrite per tombstone, and a recorded drop is never queued as pending. +- Re-evaluation scans `.sc/signatures` once per trigger (time proportional to + the file, memory proportional to the pending targets, at most 256 ids each). +- Index files are written without a lock on the serve ingest path, as + `.sc/signatures` and `.sc/transcript-drops` already are; two concurrent + pushes can lose one pending entry. A fetching peer is re-sent the tombstone + and recovers; a push-only remote needs the signer trusted before the push. +- A bucket serves every signed tombstone-shaped object it holds, and its + contents are whatever writers uploaded; receivers still judge each one. + ### Not changed A tombstone a repo already held when it upgraded to this build has no pending From 364c74c77e478f3cd07685149ea5815b83b585e0 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:15:20 +0300 Subject: [PATCH 07/13] fix(repo): do not read the pending list for an empty batch (ref #230) --- crates/repo/src/transcripts.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 150fc6d7..485eaa5b 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -160,6 +160,9 @@ fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { /// Remember `new` tombstones for re-evaluation, deduped, newest last, capped /// at [`MAX_PENDING`]. Writes only when the set changes. fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { + if new.is_empty() { + return Ok(()); + } // Dedupe the batch (order kept), then only its newest MAX_PENDING can // survive. let mut once = BTreeSet::new(); From acb618d8d568e755568b7a6d45d5af61a65f1226 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:34:20 +0300 Subject: [PATCH 08/13] fix(repo): verify signatures before indexing, bound the serve-path trigger, append drop lines, restrict tombstone resend (ref #230) --- .claude/skills/sc-cli-reference/SKILL.md | 7 +- ARCHITECTURE.md | 6 +- CLAUDE.md | 6 +- ROADMAP.md | 11 +- crates/repo/src/bucket_transport.rs | 35 +-- crates/repo/src/layout.rs | 6 + crates/repo/src/repo.rs | 4 +- crates/repo/src/signatures.rs | 38 ++- crates/repo/src/sync.rs | 4 +- crates/repo/src/transcripts.rs | 337 ++++++++++++++++----- crates/repo/src/transport.rs | 24 +- docs/THREAT-MODEL.md | 20 +- docs/adr/0038-agent-session-transcripts.md | 208 +++++++------ 13 files changed, 473 insertions(+), 233 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 7c559bcf..f40d30ad 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -252,9 +252,10 @@ cargo run --bin sc -- transcript drop [--identity ] # drop a transcr # signs the tombstone; peers honor it only if that key # is under [signers] trusted (unsigned = local-only). # Propagates on the peer's next fetch (ADR-0038 addendum); - # every retained signed tombstone rides every transfer, and - # a remote holding one from before it trusted the signer - # honors it on its next open/ingest (#230) + # a signed tombstone also rides when the dropper's gc + # collected its snapshot, and a remote holding one from + # before it trusted the signer honors it once its + # trusted-signer set changes (#230) cargo run --bin sc -- ws harvest --transcript [--sign] [--identity ] # attach the # file to each harvested workspace's landed snapshot # (P30; landing status prints before the attach) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f12af800..197dda93 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -988,9 +988,9 @@ without a new object tag or wire change: a drop is a **tombstone**, a dropped id, recorded in a local `.sc/transcript-drops` index and treated as a grow-only set (no undrop; re-attach to restore). Reads, `sc log` markers and the over-send filter through the drop set, and `sc gc` reclaims the body. A -tombstone travels with fetch and push only when it is signed, and every -retained signed tombstone rides every transfer, so a peer is told even after -the dropper's gc collected the snapshot (a push that finds the remote already at +tombstone travels with fetch and push only when it is signed, and a +retained signed tombstone rides a transfer when its snapshot is in it or was +collected by the sender's gc, so a stale peer is told (a push that finds the remote already at the same branch tip sends nothing, so a drop made afterwards propagates once a push moves the branch or the remote fetches), and a receiving peer honors it only if the signature comes from a key in its `[signers] diff --git a/CLAUDE.md b/CLAUDE.md index 0478f4b6..60231fe3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,7 +145,7 @@ the code, those win. | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | | P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | -| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; every retained signed tombstone rides every transfer, and held-but-untrusted ones are re-evaluated from `.sc/transcript-pending` on open/ingest (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) | +| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; retained signed tombstones ride a transfer when their snapshot is in it or was collected by the sender, only verifying signatures are indexed, and held-but-untrusted tombstones are re-judged from `.sc/transcript-pending` when the trusted-signer set changes (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff; later `--max-conn-bytes` per-connection write budget (#233) | [0041](docs/adr/0041-listener-resource-limits.md) | | P32 | In-binary TLS: `sc+https://` via leaf crate `tlsio`; accept-new TOFU pinning (`sc tls list/forget/pin/path` manages the pin file); `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system` validate against a CA instead; client connect/handshake bounded; public plaintext bind no longer justified by tokens alone | [0042](docs/adr/0042-in-binary-tls-sc-https.md), [0053](docs/adr/0053-tls-pin-management.md), [0054](docs/adr/0054-tls-ca-validation.md) | | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | @@ -181,7 +181,9 @@ transport-adjacent. The rest, imperatively: - **The P35 desktop app is a keyless, read-only top-level adapter.** Its renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or - identity material. + identity material. Opening a repository can write the derived transcript-drop + indexes (`.sc/transcript-drops`, `.sc/transcript-pending`) when the trusted + signer set changed; no key material is involved (#230). - **Partial clones refuse `sc export`, `sparse disable`, an empty `sparse set` (and, on a non-lazy clone, one outside the filter), `sc rewrap`, `sc grant`/`revoke`, creating, switching to or leaving a private branch, a merge, cherry-pick or rebase whose conflict lies outside the view (a `.theirs` sidecar sits beside its conflicted path, so it is in view whenever that path is; ADR-0049) (no ref, state or working-tree change; on every clone, a conflict stop that needs sealing outside the view is refused the same way), `sc push` to a git remote, and `sc fetch` from any native remote but the promisor origin.** `sc merge`, `sc cherry-pick` and `sc rebase` work, including conflicts inside the view (P37c). `sc ws` and `sc work` work too (ADR-0050): workspaces and agents see the clone's view only, and harvest lands through the same merge. `sc backfill --all` diff --git a/ROADMAP.md b/ROADMAP.md index 47041b83..30a615f7 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -481,11 +481,12 @@ across every phase. hides the body from `list`/`show`, travels with fetch/clone, blocks resurrection in either arrival order, and is kept by gc while the dropped body is swept at gc (after the grace window if loose). (ADR-0038 addendum.) - Issue #230 closed the delivery gaps: every retained signed tombstone rides - every transfer (so a stale peer is told after the dropper's gc collected the - snapshot), and a remote that held a tombstone before trusting the signer - re-evaluates it from a capped `.sc/transcript-pending` index on open and - ingest. + Issue #230 closed the delivery gaps: a retained signed tombstone rides + a transfer when its snapshot is in it or was collected by the sender's gc (so + a stale peer is told), only verifying signatures enter the signature index, + and a remote that held a tombstone before trusting the signer re-judges it + from a capped `.sc/transcript-pending` index when the trusted-signer set + changes. Crowding by valid signatures from attacker keys is tracked in #257. - **Phase 32 — In-binary TLS (`sc+https://`).** Closes the audit's High #1 (plaintext bearer tokens/traffic on `sc+http://`) and ADR-0036's "no TLS" diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index a8640383..b16c9043 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -423,13 +423,7 @@ impl BucketTransport { } let mut out: std::collections::BTreeSet = std::collections::BTreeSet::new(); for (snapshot, id, tombstone) in transcripts { - // A signed tombstone goes out whatever its snapshot (#230). - let send = if tombstone { - sigs.contains_key(&id) - } else { - snaps.contains(&snapshot) - }; - if send { + if snaps.contains(&snapshot) && (!tombstone || sigs.contains_key(&id)) { out.insert(id); } } @@ -2100,31 +2094,4 @@ mod tests { std::fs::remove_dir_all(&base).unwrap(); assert!(!base.exists()); } - - #[test] - fn a_signed_tombstone_is_served_whatever_the_transfer_set() { - // #230: the sender no longer selects tombstones through the snapshots - // of the transfer set, a bucket included. - let (a, url, snap, tid, identity, base) = signed_bucket_origin("tomb-any"); - a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); - std::fs::write(base.join("a/f2"), b"two").unwrap(); - a.commit("t", "c2").unwrap(); - a.push("origin").unwrap(); - let tomb = ObjectId::of( - &Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)).encode(), - ); - - let t = BucketTransport::open(&url).unwrap(); - t.list_refs().unwrap(); - let served = t.leaf_ids_for(&std::collections::BTreeSet::new()).unwrap(); - assert!( - served.contains(&tomb), - "signed tombstone, empty transfer set" - ); - assert!(!served.contains(&tid), "the body still needs its snapshot"); - - drop(a); - std::fs::remove_dir_all(&base).unwrap(); - assert!(!base.exists()); - } } diff --git a/crates/repo/src/layout.rs b/crates/repo/src/layout.rs index 61b26901..a8857b1a 100644 --- a/crates/repo/src/layout.rs +++ b/crates/repo/src/layout.rs @@ -97,6 +97,12 @@ impl Layout { pub fn transcript_pending_path(&self) -> PathBuf { self.dot_sc.join("transcript-pending") } + /// `.sc/transcript-pending-trust` — fingerprint of the trusted-signer set + /// the pending tombstones were last judged under; a different set is the + /// trigger to judge them again. + pub fn transcript_pending_trust_path(&self) -> PathBuf { + self.dot_sc.join("transcript-pending-trust") + } /// `.sc/tmp` — scratch space for transient files (P25 streaming pack /// transfer: spilled sender/receiver pack bodies). Never durable state — /// safe to delete at any time; callers that write here are responsible diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index c550e1ee..cfbd5c57 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -76,9 +76,9 @@ impl Repo { fn open_layout(layout: Layout, budget_bytes: usize) -> Result { let lock = RepoLock::acquire(&layout)?; let mut store = Store::open_persistent(layout.objects_dir(), budget_bytes)?; - // Honor held tombstones whose signer was trusted since (#230). Best + // Honor held tombstones once the trusted-signer set changed (#230). Best // effort: a failure must not stop an open, and the next open retries. - let _ = crate::transcripts::reevaluate_pending(&layout, &mut store); + let _ = crate::transcripts::reevaluate_if_trust_changed(&layout, &mut store); Ok(Repo { layout, vfs: VfsRepo::new(store), diff --git a/crates/repo/src/signatures.rs b/crates/repo/src/signatures.rs index b0c90436..69e3261e 100644 --- a/crates/repo/src/signatures.rs +++ b/crates/repo/src/signatures.rs @@ -145,13 +145,45 @@ pub(crate) fn index_incoming( let mut count = 0; for id in ids { if let Object::Signature(sig) = store.get(id)? { - append_index(layout, sig.snapshot, *id)?; - count += 1; + // Only a signature that checks out for a target this store holds + // is indexed (and so forwarded and kept by gc): nothing forged or + // dangling can crowd out a real one (#230, ADR-0038). + if acceptable(store, &sig)? { + append_index(layout, sig.snapshot, *id)?; + count += 1; + } } } Ok(count) } +/// Whether `sig` may enter the index: its target is held, and the signature +/// verifies under the key it names in the domain its target kind uses. A +/// snapshot takes the snapshot domain; a transcript the transcript domain; a +/// drop tombstone only the domain-separated drop digest (a plain signature +/// over a tombstone id never authorizes a drop). Any other target kind is +/// refused. Trust is not judged here: that is the reader's call. +pub(crate) fn acceptable(store: &mut scl_core::Store, sig: &SignatureObj) -> Result { + if !store.contains(&sig.snapshot) { + return Ok(false); + } + let target = sig.snapshot; + Ok(match store.get(&target)? { + Object::Snapshot(_) => { + scl_crypto::verify_snapshot_sig(&sig.signer, target.as_bytes(), &sig.sig) + } + Object::Transcript(t) if t.dropped_id().is_some() => scl_crypto::verify_transcript_sig( + &sig.signer, + &crate::transcripts::drop_digest(&target), + &sig.sig, + ), + Object::Transcript(_) => { + scl_crypto::verify_transcript_sig(&sig.signer, target.as_bytes(), &sig.sig) + } + _ => false, + }) +} + /// Full-store rebuild of the signature index (Task 3): scan every object the /// store can resolve (loose + packed — `Store::all_ids`, not a reachability /// walk, since a `SignatureObj` is a leaf no tree/snapshot references), keep @@ -171,7 +203,7 @@ pub(crate) fn reindex(layout: &Layout, store: &mut scl_core::Store) -> Result = Vec::new(); for id in store.all_ids()? { if let Object::Signature(sig) = store.get(&id)? { - if store.contains(&sig.snapshot) { + if acceptable(store, &sig)? { entries.push((sig.snapshot, id)); } } diff --git a/crates/repo/src/sync.rs b/crates/repo/src/sync.rs index eb7300a5..3346aad7 100644 --- a/crates/repo/src/sync.rs +++ b/crates/repo/src/sync.rs @@ -531,7 +531,9 @@ impl Repo { let transcript_ids = if no_transcripts { Vec::new() } else { - crate::transcripts::indexed_transcript_ids_for(&self.layout, &snaps)? + crate::transcripts::indexed_transcript_ids_for(&self.layout, &snaps, &|id| { + store.contains(id) + })? }; let sig_targets: Vec = snaps.iter().chain(transcript_ids.iter()).copied().collect(); diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 485eaa5b..5fcd6a3a 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -106,13 +106,28 @@ fn dropped_ids(layout: &Layout) -> Result> { Ok(read_drops(layout)?.into_iter().map(|(_, _, d)| d).collect()) } -/// Record one tombstone, deduping an identical existing line. +/// Record one tombstone, deduping an identical existing line. The line is +/// appended (one short `O_APPEND` write, atomic on a local filesystem), not +/// merged into a rewrite of the file, so concurrent ingests (the HTTP serve +/// path holds no repo lock) cannot lose each other's drop line. A racing +/// duplicate line is harmless: readers treat the file as a set. fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { - let mut entries = read_drops(layout)?; - if !entries.contains(&entry) { - entries.push(entry); - write_drops(layout, &entries)?; + use std::io::Write; + if read_drops(layout)?.contains(&entry) { + return Ok(()); } + let line = format!( + "{} {} {}\n", + entry.0.to_hex(), + entry.1.to_hex(), + entry.2.to_hex() + ); + let mut f = std::fs::OpenOptions::new() + .append(true) + .create(true) + .open(layout.transcript_drops_path())?; + f.write_all(line.as_bytes())?; + f.sync_data()?; Ok(()) } @@ -147,10 +162,13 @@ fn read_pending(layout: &Layout) -> Result> { fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { let path = layout.transcript_pending_path(); if ids.is_empty() { - return match std::fs::remove_file(&path) { - Err(e) if e.kind() != std::io::ErrorKind::NotFound => Err(Error::Io(e)), - _ => Ok(()), - }; + for p in [&path, &layout.transcript_pending_trust_path()] { + match std::fs::remove_file(p) { + Err(e) if e.kind() != std::io::ErrorKind::NotFound => return Err(Error::Io(e)), + _ => {} + } + } + return Ok(()); } let out: String = ids.iter().map(|i| format!("{}\n", i.to_hex())).collect(); scl_core::fsutil::atomic_write_durable(&path, out.as_bytes())?; @@ -182,7 +200,58 @@ fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { } let excess = all.len().saturating_sub(MAX_PENDING); all.drain(..excess); - write_pending(layout, &all) + write_pending(layout, &all)?; + // Entries are judged under the trust in force now; remember which. + if !layout.transcript_pending_trust_path().exists() { + save_trust_fp(layout)?; + } + Ok(()) +} + +/// Fingerprint of the trusted-signer key set (the only configuration the +/// honoring check reads), so an unrelated edit to `recipients.toml` does not +/// count as a trust change. +fn trust_fp(layout: &Layout) -> String { + let mut keys: Vec<[u8; 32]> = trusted_signers(layout).into_keys().collect(); + keys.sort(); + let mut h = blake3::Hasher::new(); + for k in keys { + h.update(&k); + } + h.finalize().to_hex().to_string() +} + +fn save_trust_fp(layout: &Layout) -> Result<()> { + scl_core::fsutil::atomic_write_durable( + &layout.transcript_pending_trust_path(), + trust_fp(layout).as_bytes(), + )?; + Ok(()) +} + +/// Whether tombstones are pending and the trusted-signer set differs from the +/// one they were last judged under. One `exists` check when nothing is +/// pending; no lock and no store access, so a server can call it per +/// connection. +pub(crate) fn trust_changed(layout: &Layout) -> bool { + layout.transcript_pending_path().exists() + && std::fs::read_to_string(layout.transcript_pending_trust_path()).ok() + != Some(trust_fp(layout)) +} + +/// [`reevaluate_pending`] when [`trust_changed`], then record the new trust. +/// The re-check triggers (open, transport open, ingest) all go through here +/// so none of them does work or takes a lock while the trust is unchanged. +pub(crate) fn reevaluate_if_trust_changed( + layout: &Layout, + store: &mut scl_core::Store, +) -> Result { + if !trust_changed(layout) { + return Ok(0); + } + let honored = reevaluate_pending(layout, store)?; + save_trust_fp(layout)?; + Ok(honored) } /// Rewrite the whole index file atomically (temp file + rename), so a @@ -254,7 +323,7 @@ fn trusted_signers(layout: &Layout) -> std::collections::HashMap<[u8; 32], Strin /// tombstone id, NOT the bare id. A plain `sign_transcript` signature over the /// tombstone id (e.g. from an older binary's bulk `sc transcript sign`, which /// indexes tombstones as live transcripts) must never authorize a drop. -fn drop_digest(tomb: &ObjectId) -> [u8; 32] { +pub(crate) fn drop_digest(tomb: &ObjectId) -> [u8; 32] { let mut h = blake3::Hasher::new(); h.update(Transcript::DROP_AGENT.as_bytes()); h.update(tomb.as_bytes()); @@ -381,8 +450,10 @@ pub(crate) fn index_incoming( .into_iter() .map(|(_, tomb, _)| tomb) .collect(); - // Honor a tombstone, or remember a signed one that is held but inert: a - // push-only remote is never sent it again (#230). + // Honor a tombstone, or remember a signed one whose snapshot is held but + // that is inert: a push-only remote is never sent it again (#230). One + // whose snapshot is absent is not kept: it can only hide a body that + // arrives later, and a fetching peer is re-sent it anyway. let mut consider = |store: &mut scl_core::Store, id: ObjectId, t: &Transcript, d: ObjectId| { if recorded.contains(&id) { return Ok::<_, Error>(()); @@ -390,7 +461,7 @@ pub(crate) fn index_incoming( if tombstone_honored(store, &trusted, &sigs, id, t, d)? { append_drop(layout, (t.snapshot, id, d))?; recorded.insert(id); - } else if sigs.contains_key(&id) { + } else if sigs.contains_key(&id) && store.contains(&t.snapshot) { held.push(id); } Ok(()) @@ -527,6 +598,7 @@ pub(crate) fn reindex(layout: &Layout, store: &mut scl_core::Store) -> Result bool, ) -> Result> { let wanted: BTreeSet = snapshots.iter().copied().collect(); let mut out: BTreeSet = BTreeSet::new(); @@ -535,20 +607,24 @@ pub(crate) fn indexed_transcript_ids_for( out.insert(tid); } } - // Every retained signed tombstone travels on every transfer, not only - // those of `snapshots`: gc keeps a tombstone after it collects its + // A retained signed tombstone travels when its snapshot is in `snapshots` + // or is no longer held here: gc keeps a tombstone after it collects its // snapshot, and a stale peer that still lists the body must still be told - // (#230). Cost: one tombstone plus one signature per drop, which receivers - // dedupe and ignore when they cannot use it. An unsigned (local-only) drop - // is never sent: peers would ignore it, and an older peer would index it - // as a live transcript. + // (#230). A tombstone whose snapshot is still held but outside this + // transfer (an unpublished branch) stays home, so its ids do not leak to a + // peer that never had the snapshot. Cost: one tombstone plus one signature + // per such drop, which receivers dedupe and ignore when they cannot use + // it. An unsigned (local-only) drop is never sent: peers would ignore it, + // and an older peer would index it as a live transcript. let drops = read_drops(layout)?; if !drops.is_empty() { let sigs = crate::signatures::index_by_target(layout)?; out.extend( drops .into_iter() - .filter(|(_, tomb, _)| sigs.contains_key(tomb)) + .filter(|(snap, tomb, _)| { + sigs.contains_key(tomb) && (wanted.contains(snap) || !snapshot_held(snap)) + }) .map(|(_, tomb, _)| tomb), ); } @@ -1039,7 +1115,7 @@ pub(crate) mod tests { assert!(repo.transcripts_for(&snap).unwrap().is_empty()); // The sender ships the tombstone with its snapshot. assert_eq!( - indexed_transcript_ids_for(repo.layout(), &[snap]) + indexed_transcript_ids_for(repo.layout(), &[snap], &|_| true) .unwrap() .len(), 1 @@ -1139,7 +1215,7 @@ pub(crate) mod tests { .attach_transcript(snap, "a", "k", b"keep", &[owner.enc.public()]) .unwrap(); repo.drop_transcript(&keep.to_hex(), None).unwrap(); - let sent = indexed_transcript_ids_for(repo.layout(), &[snap]).unwrap(); + let sent = indexed_transcript_ids_for(repo.layout(), &[snap], &|_| true).unwrap(); let drops = read_drops(repo.layout()).unwrap(); assert_eq!(sent.len(), 1); assert!(drops.iter().any(|d| sent.contains(&d.1) && d.2 == body)); @@ -1181,7 +1257,7 @@ pub(crate) mod tests { } #[test] - fn sender_offers_every_retained_signed_tombstone_whatever_the_transfer_set() { + fn sender_offers_a_signed_tombstone_for_its_set_or_a_collected_snapshot() { let root = tmp_root("send-all"); let repo = Repo::init(&root).unwrap(); std::fs::write(root.join("f"), b"x").unwrap(); @@ -1199,8 +1275,18 @@ pub(crate) mod tests { let drops = read_drops(repo.layout()).unwrap(); let tomb = drops.iter().find(|d| d.2 == signed).unwrap().1; - // `snap` is not in the transfer set (gc may have collected it). - let sent = indexed_transcript_ids_for(repo.layout(), &[]).unwrap(); + // `snap` is held here but outside the transfer set (an unpublished + // branch): the tombstone stays home, so its ids do not leak. + let held = |_: &ObjectId| true; + assert!(indexed_transcript_ids_for(repo.layout(), &[], &held) + .unwrap() + .is_empty()); + // In the transfer set: sent, signed one only. + let sent = indexed_transcript_ids_for(repo.layout(), &[snap], &held).unwrap(); + assert!(sent.contains(&tomb) && !sent.contains(&signed)); + // gc collected the snapshot: still sent, so a stale peer is told. + let gone = |_: &ObjectId| false; + let sent = indexed_transcript_ids_for(repo.layout(), &[], &gone).unwrap(); assert_eq!(sent, vec![tomb], "the signed tombstone only"); drop(repo); @@ -1319,70 +1405,179 @@ pub(crate) mod tests { } #[test] - fn a_flood_of_untrusted_signatures_cannot_keep_a_trusted_one_from_honoring() { - let root = tmp_root("sig-flood"); + fn junk_signatures_sorting_ahead_of_the_real_one_cannot_keep_a_drop_inert() { + // B1: a pack is ordered by object id, so a writer can place junk + // signatures (claiming the trusted key) ahead of the real one. They + // must not reach the index, whatever the cap. + let root = tmp_root("sig-ahead"); let repo = Repo::init(&root).unwrap(); std::fs::write(root.join("f"), b"x").unwrap(); let snap = repo.commit("t", "c1").unwrap(); let (_s, owner) = scl_crypto::generate_identity_v2(); + let key = owner.signing.as_ref().unwrap(); let body = repo .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) .unwrap(); - let tomb = repo + trust_signer(&repo, &owner); + let tomb_obj = Object::Transcript(Transcript::drop_tombstone(snap, &body)); + let tomb = tomb_obj.id(); + let real = Object::Signature(SignatureObj { + snapshot: tomb, + signer: key.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(key, &drop_digest(&tomb)), + }); + let mut objs = vec![tomb_obj, real.clone()]; + // Junk claiming the trusted key, and valid signatures by other keys, + // all with ids below the real signature's. + let mut i = 0u64; + let mut junk = 0; + while junk < MAX_SIG_VERIFIES + 12 { + i += 1; + let mut sig = [0u8; 64]; + sig[..8].copy_from_slice(&i.to_le_bytes()); + let o = Object::Signature(SignatureObj { + snapshot: tomb, + signer: key.public().to_bytes(), + sig, + }); + if o.id() < real.id() { + objs.push(o); + junk += 1; + } + } + let junk_ids: Vec = objs[2..].iter().map(Object::id).collect(); + let (_s2, other) = scl_crypto::generate_identity_v2(); + let other_key = other.signing.as_ref().unwrap(); + // One valid-but-untrusted signature: indexed, never honored. + objs.push(Object::Signature(SignatureObj { + snapshot: tomb, + signer: other_key.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(other_key, &drop_digest(&tomb)), + })); + objs.sort_by_key(Object::id); + let encoded: Vec<(ObjectId, Vec)> = objs.iter().map(|o| (o.id(), o.encode())).collect(); + let (pack, _idx) = scl_core::pack::build_pack(&encoded).unwrap(); + let path = root.join("incoming.pack"); + std::fs::write(&path, pack).unwrap(); + + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + crate::transport::ingest_pack_file(repo.layout(), &mut store, &path).unwrap(); + drop(store); + + let sigs = crate::signatures::index_by_target(repo.layout()).unwrap(); + let indexed = &sigs[&tomb]; + assert!(indexed.contains(&real.id())); + assert!( + junk_ids.iter().all(|j| !indexed.contains(j)), + "forged signatures are never indexed" + ); + assert_eq!(read_drops(repo.layout()).unwrap().len(), 1, "drop honored"); + assert!(load(repo.layout()).unwrap().is_empty()); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn a_signature_over_an_absent_target_is_not_indexed() { + let root = tmp_root("sig-dangling"); + let repo = Repo::init(&root).unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let key = owner.signing.as_ref().unwrap(); + let ghost = ObjectId::of(b"nothing here"); + let id = repo .store_arc() .lock() .unwrap() - .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .put(Object::Signature(SignatureObj { + snapshot: ghost, + signer: key.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(key, &drop_digest(&ghost)), + })) .unwrap(); - // Far more junk signatures by untrusted keys than any cap, indexed - // before the real one. let arc = repo.store_arc(); - let mut junk_ids = Vec::new(); - for i in 0..(MAX_SIG_READS + 40) { - let mut signer = [0u8; 32]; - signer[..8].copy_from_slice(&(i as u64 + 1).to_le_bytes()); - let junk = SignatureObj { - snapshot: tomb, - signer, - sig: [7u8; 64], - }; - junk_ids.push(arc.lock().unwrap().put(Object::Signature(junk)).unwrap()); - } let mut store = arc.lock().unwrap(); - crate::signatures::index_incoming(repo.layout(), &mut store, &junk_ids).unwrap(); + assert_eq!( + crate::signatures::index_incoming(repo.layout(), &mut store, &[id]).unwrap(), + 0 + ); drop(store); + assert!(crate::signatures::index_by_target(repo.layout()) + .unwrap() + .is_empty()); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn a_pending_tombstones_signature_survives_gc_and_trust_change_is_detected_once() { + let root = tmp_root("pending-gc"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let arc = repo.store_arc(); + let tomb = arc + .lock() + .unwrap() + .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .unwrap(); repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) .unwrap(); - trust_signer(&repo, &owner); - let sigs = crate::signatures::index_by_target(repo.layout()).unwrap(); - assert!( - sigs[&tomb].len() > MAX_SIG_READS, - "real one sits past the read cap" - ); + let sig = crate::signatures::index_by_target(repo.layout()).unwrap()[&tomb][0]; + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + drop(store); + assert_eq!(read_pending(repo.layout()).unwrap(), vec![tomb]); - // Reading is capped, so put the real one first to model the normal - // case (it arrives with the tombstone): junk afterwards is harmless. - let mut ordered = sigs.clone(); - let last = ordered.get_mut(&tomb).unwrap().pop().unwrap(); - ordered.get_mut(&tomb).unwrap().insert(0, last); + // Nothing changed since it was judged: no trigger, no work. + assert!(!trust_changed(repo.layout())); + repo.gc(std::time::Duration::from_secs(0)).unwrap(); + let store = arc.lock().unwrap(); + assert!(store.contains(&tomb) && store.contains(&sig)); + drop(store); + assert_eq!(repo.signatures_for(&tomb).unwrap().len(), 1); + + trust_signer(&repo, &owner); + assert!(trust_changed(repo.layout())); let mut store = arc.lock().unwrap(); - let trusted = trusted_signers(repo.layout()); - let t = match store.get(&tomb).unwrap() { - Object::Transcript(t) => t, - _ => unreachable!(), - }; - assert!(tombstone_honored(&mut store, &trusted, &ordered, tomb, &t, body).unwrap()); - - // Untrusted junk ahead of a trusted signature is skipped for free: - // 20 junk signatures put the real one at index 20 (past the 8 - // verification cap, inside the read cap). - let mut mid = sigs.clone(); - let v = mid.get_mut(&tomb).unwrap(); - let last = v.pop().unwrap(); - v.truncate(20); - v.push(last); - assert!(tombstone_honored(&mut store, &trusted, &mid, tomb, &t, body).unwrap()); + assert_eq!( + reevaluate_if_trust_changed(repo.layout(), &mut store).unwrap(), + 1 + ); drop(store); + assert!(!trust_changed(repo.layout())); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn concurrent_drop_appends_lose_no_line() { + let root = tmp_root("append-race"); + let repo = Repo::init(&root).unwrap(); + let layout = repo.layout().clone(); + let handles: Vec<_> = (0..16u32) + .map(|i| { + let layout = layout.clone(); + std::thread::spawn(move || { + let e = |tag: &str| ObjectId::of(format!("{tag}{i}").as_bytes()); + append_drop(&layout, (e("s"), e("t"), e("d"))).unwrap(); + }) + }) + .collect(); + for h in handles { + h.join().unwrap(); + } + assert_eq!(read_drops(&layout).unwrap().len(), 16); drop(repo); std::fs::remove_dir_all(&root).unwrap(); diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index b87f0dd7..ce1d1ee1 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -162,9 +162,11 @@ impl LocalTransport { // honor held tombstones whose signer was trusted since (#230). Best // effort under the repo lock; a busy lock means a `Repo` is open and // ran it already. - if layout.transcript_pending_path().exists() { + // Only when the trusted-signer set changed since they were last + // judged: serving must not take the exclusive lock per connection. + if crate::transcripts::trust_changed(&layout) { if let Ok(_lock) = RepoLock::acquire(&layout) { - let _ = crate::transcripts::reevaluate_pending(&layout, &mut store); + let _ = crate::transcripts::reevaluate_if_trust_changed(&layout, &mut store); } } Ok(LocalTransport { @@ -304,7 +306,9 @@ impl LocalTransport { // `sc transcript attach --sign` survived clone/fetch/push as an // object but landed `Unsigned` on the far side). let transcript_ids = - crate::transcripts::indexed_transcript_ids_for(&self.layout, &all_snaps)?; + crate::transcripts::indexed_transcript_ids_for(&self.layout, &all_snaps, &|id| { + store.contains(id) + })?; let sig_targets: Vec = all_snaps .iter() .chain(transcript_ids.iter()) @@ -613,18 +617,20 @@ fn ingest_pack_file_impl( ids.push(id); Ok(()) })?; + // Tombstones held from earlier transfers are judged again when the + // trusted-signer set changed (#230), before this pack's own tombstones are + // noted under the current set. A `--no-transcripts` transfer indexes no + // transcript state. Best effort: the pack is already stored, so a failure + // here must not fail it. + if !skip_transcripts { + let _ = crate::transcripts::reevaluate_if_trust_changed(layout, store); + } // Receiver seam (P22 Task 3): every id above was just written to this // store, so `index_incoming`'s "ids just written" contract holds. crate::signatures::index_incoming(layout, store, &ids)?; // Receiver seam (P30 Task 4): same "ids just written" contract, for // Transcript objects. crate::transcripts::index_incoming(layout, store, &ids)?; - // Tombstones held from earlier transfers whose signer is trusted by now - // (#230). A `--no-transcripts` transfer indexes no transcript state. Best - // effort: the pack is already stored, so a failure here must not fail it. - if !skip_transcripts { - let _ = crate::transcripts::reevaluate_pending(layout, store); - } Ok(ids) } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index a894c502..7b7fc7ec 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -208,12 +208,20 @@ too, not just the src-control-side metadata. bodies. Drop is not erasure: peers without the tombstone keep the body, `GetObjects` serves it until gc sweeps it, and a peer that still holds it can re-send it. A tombstone held while its signer is untrusted is remembered - (`.sc/transcript-pending`, capped at 1024, FIFO) and re-evaluated on open and - after each ingest, so trusting the signer later is enough on a push-only or - served remote. Every retained signed tombstone is sent on every transfer, so - a peer is told even after gc collected the snapshot; a peer that pushes junk - can only evict pending entries, not get one honored. Trust is checked at - evaluation and is not retroactive. A tombstone a remote already held before + (`.sc/transcript-pending`, capped at 1024, FIFO) and judged again when the + trusted-signer set changes (on open, transport open and ingest), so trusting + the signer later is enough on a push-only or served remote. A retained signed + tombstone is also sent when its snapshot was collected by the sender's gc, so + a stale peer is told; this discloses the snapshot id, dropped transcript id + and signer public key of such drops to any peer the dropper transfers with, + including one that never had the snapshot (ids are hashes, bodies stay + sealed). Only signatures that verify for a held target are indexed, so + forged signatures are never stored or forwarded. A writer to a shared remote + who signs with keys of their own can still crowd more than 256 valid + signatures ahead of the real one, which keeps the drop inert on every peer + that ingests them (a denial of the drop, never an honored one; #257). Writers + can also evict pending entries from a push-only remote (trust the signer + before the push). Trust is checked at evaluation and is not retroactive. A tombstone a remote already held before upgrading to a build with the pending index is not remembered: trust the signer before the drop is pushed, or have a fetching peer re-send it. diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index 2302f0a9..bc17db0c 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -416,115 +416,135 @@ no new object kind and no change to the trust gate. A tombstone is still honored only when a drop-digest signature verifies under a key in the receiver's `[signers] trusted`; unsigned drops still never leave the dropper. -### 1. Send every retained signed tombstone +### 1. Send retained signed tombstones whose snapshot is in the set or gone `indexed_transcript_ids_for` (used by `LocalTransport::build_pack_tempfile` -for fetch/clone/serve and by `push_opts`) no longer selects tombstones -through the transfer set's snapshots. It returns every line of -`.sc/transcript-drops` that has an indexed signature, plus (as before) the -signatures over them. The bucket sender (`leaf_ids_for`) does the same. Why -not match against the advertised haves: the stale peer's snapshot is exactly -what the dropper's gc removed, so the dropper cannot resolve a have to it, -and an id-only match would need the dropper to keep the snapshot id forever. +for fetch/clone/serve and by `push_opts`) no longer selects tombstones only +through the transfer set's snapshots. A retained tombstone that has an indexed +signature is also sent when its snapshot is **no longer held** by the sender, +which is exactly the gc case: the dropper cannot resolve a stale peer's have +to a collected snapshot, so no have-based match can find it. A tombstone whose +snapshot is still held but outside the transfer set (an unpublished branch) +stays home. The bucket sender keeps its rule (snapshot in the want or have +closure): a bucket never collects snapshots, so the gap cannot arise there. Cost and limits: -- One tombstone (about 100 bytes) plus one signature per retained drop, per - transfer, regardless of branch. Nothing is capped: capping would silently - withhold a drop from a peer, and the set only grows by explicit drops (the - dropper's own and ones it honored from trusted signers). It is never - subtracted via the have set, like transcripts and signatures. -- A receiver that has the drop recorded dedups (`append_drop` writes only a - new line); one that cannot use it (snapshot unknown, signer untrusted) +- One tombstone (about 100 bytes) plus its signatures per qualifying drop, per + transfer. Nothing is capped: capping would silently withhold a drop from a + peer, and the set only grows by explicit drops (the dropper's own and ones + it honored from trusted signers). It is never subtracted via the have set, + like transcripts and signatures. +- **Disclosure.** A tombstone names its snapshot id, the dropped transcript id + and (through its signature) the signer's public key. Any peer that fetches + from or is pushed to by the dropper now learns these for every drop whose + snapshot the dropper has since collected, including a snapshot that peer + never had. Before this change the same peer received them only for drops on + snapshots in the transfer. Ids are hashes; the body stays sealed. Drops on + snapshots still held but unpublished are not sent. +- A receiver that has the drop recorded skips it (one read of the drop file + per ingest). One whose snapshot is unknown or whose signer is untrusted stores the small object and ignores it. - A push sends only tombstones the remote lacks (`has_object`), so repeat pushes cost one existence check per drop. `--no-transcripts` still leaves tombstones and their signatures out of both directions (#228). -- A fresh clone made after the dropper's gc receives retained tombstones - whose snapshots it never has. They stay inert and are remembered as - pending (below, capped), so they occupy ring slots without ever being - honored. -### 2. Remember and re-evaluate held, unhonored tombstones - -A push-only or served remote holds the tombstone object from an earlier -push, so `push` (has-object) never re-sends it and `index_incoming` never -re-runs. Fix: a small local index plus re-evaluation. +### 2. Index only signatures that verify, and remember held tombstones + +**Signature index hygiene (the crowding fix).** A tombstone id depends only on +its snapshot and transcript ids, so a writer can target it before it exists, +and a pack is ordered by object id, so a writer can grind junk signature ids +that sort ahead of the real one. `signatures::index_incoming` and `reindex` +therefore index a signature only if its target is held and it verifies under +the key it names, in the domain of its target kind (snapshot domain for a +snapshot, transcript domain for a transcript, the drop digest for a +tombstone). Forged and dangling signatures are never indexed, so never +forwarded to a downstream peer and never kept by gc. Side effect: a received +signature that fails to verify no longer turns a snapshot `Invalid`; it is +simply absent (`Unsigned`, which `--require` also rejects). A signature +indexed before this change is read as before. + +**Residual limit (#257).** A signature that verifies under a key the attacker +controls is still indexed. A writer can add more than `MAX_SIG_READS` (256) of +them over a predictable tombstone id, ahead of the real signature, and +`tombstone_honored` stops reading at 256 per tombstone, so the drop stays +inert on every peer that ingests them. This is a denial of the drop on shared +remotes (not on a single peer), never an honored one. The fix is an index +keyed by (target, signer) so honoring is a lookup over the trusted signers; +it is tracked in #257. The caps (`MAX_SIG_READS` 256 reads and +`MAX_SIG_VERIFIES` 8 failed checks per tombstone) bound the work meanwhile; +the verification cap is unreachable for signatures indexed by this version. + +**Pending index.** A push-only or served remote holds the tombstone from an +earlier push, so `push` (has-object) never re-sends it and `index_incoming` +never re-runs. - `.sc/transcript-pending`: one tombstone object id per line, newest last. - `index_incoming` appends a tombstone that is held, **has an indexed - signature**, and failed `tombstone_honored`. Unsigned tombstones are never - remembered. A signature that arrives later, in a transfer without the - tombstone, is also a trigger: `index_incoming` looks up the tombstone it - covers. A derived cache like `.sc/transcript-drops`: losing it only - loses the retry. -- `reevaluate_pending` re-runs the unchanged `tombstone_honored` on each - entry with the current `[signers] trusted` and signature index, records the - ones that now pass as drops and removes them from the file, forgets entries - whose object is gone or no longer a tombstone, and leaves the rest (an - object that fails to read stays, to be tried again). It is - idempotent and does nothing (one failed file read) when no pending file - exists. It runs at `Repo::open` (holding the repo lock `open` takes), at - `LocalTransport::open` (what `sc serve` and a local-path remote use instead - of `Repo::open`; it takes the repo lock itself and skips if it is busy), and - at the end of every ingest except a `--no-transcripts` one. Both are best - effort: a failure never fails an open or a transfer, and the next trigger - retries. The open trigger covers `sc serve` (per connection), `sc gc`, - `sc transcript list`, and every other command, so an operator who edits - `recipients.toml` needs no extra step; the ingest trigger covers a - long-lived server process. -- **Bounded.** The file is a FIFO ring of 1024 ids. A peer that pushes many - signed junk tombstones evicts older entries, so the work per trigger is at - most 1024 tombstone reads plus the per-tombstone signature work below. - A pack's batch is deduplicated and cut to the newest 1024 before it - touches the file; the file stays under about 70 KiB. An evicted legitimate - entry is lost only for a push-only remote; a fetching peer is re-sent every - retained tombstone (part 1). A writer with push access can still push out a - genuine pending entry; it cannot make one honored. -- **Signature work is capped inside `tombstone_honored`** (so it bounds - ingest and re-evaluation alike): at most `MAX_SIG_READS` (256) indexed - signatures are read per tombstone, and at most `MAX_SIG_VERIFIES` (8) of - those that claim a trusted key are verified. A signature by an untrusted - key costs one read and does not count against the verification cap, so a - flood from untrusted keys cannot push a trusted signature out of it. Limit: - a writer who indexes more than 256 junk signatures, or 8 junk ones claiming - a trusted key (public, since it appears on every signature), ahead of the - real one keeps the drop inert on that peer. That is a denial of the drop, - never an honored one, and the real signature normally arrives with its - tombstone, before any junk. Signature-driven lookups in `index_incoming` - (a signature that lands after its tombstone) read each distinct target at - most once per batch and at most 1024 targets per batch. + `index_incoming` appends a tombstone that **has an indexed signature**, whose + **snapshot is held**, and that failed `tombstone_honored`. Unsigned + tombstones and ones whose snapshot is absent are not kept: the latter can + only hide a body that arrives later, and a fetching peer is re-sent it + anyway. A signature that arrives later, in a transfer without the tombstone, + is also a trigger: `index_incoming` looks up the tombstone it covers (each + distinct target once per batch, at most 1024 per batch). A derived cache + like `.sc/transcript-drops`: losing it only loses the retry. +- `.sc/transcript-pending-trust`: a fingerprint of the trusted-signer key set + the pending entries were last judged under. The re-check runs only when the + current set differs (`trust_changed`: one `exists` check and one config read + when nothing is pending or nothing changed, no lock, no store access). A + snapshot arriving later is not a trigger, because only entries whose + snapshot is held are kept. +- `reevaluate_if_trust_changed` runs `reevaluate_pending`, which re-runs the + unchanged `tombstone_honored` on each entry, records the ones that now pass + as drops and removes them from the file, forgets entries whose object is + gone or no longer a tombstone, and keeps the rest (an unreadable object + stays). It runs at `Repo::open` (under the lock `open` holds), at + `LocalTransport::open` (what `sc serve` and local-path remotes use; it takes + the repo lock itself, only after `trust_changed` is true, and skips if the + lock is busy, so a served remote does not take the exclusive lock per + connection) and before the indexing step of every ingest except a + `--no-transcripts` one. All are best effort: a failure never fails an open + or a transfer, and the fingerprint is saved only after a successful pass, + so the next trigger retries. +- **Lost-update safety.** `append_drop` appends one short line with + `O_APPEND` and `fsync` instead of rewriting the file, so concurrent + ingests (the HTTP serve path holds no repo lock) cannot lose a drop line. A + pending entry is removed only after its drop line is appended. The pending + file itself is still a read-modify-write: two concurrent ingests can lose a + pending entry. That costs a push-only remote the retry for that tombstone + (trust the signer before the push, or have a fetching peer re-send it), but + never a recorded drop and never the rooting of one: gc roots every drop + line. +- **Bounded.** The file is a FIFO ring of 1024 ids. A writer who pushes many + signed tombstones evicts older entries, so a genuine pending entry can be + lost on a push-only remote; a fetching peer is re-sent every qualifying + tombstone (part 1). The work per trigger is at most 1024 tombstone reads + plus the per-tombstone signature work above. A pack's batch is + deduplicated and cut to the newest 1024 before it touches the file; the + file stays under about 70 KiB. Eviction never makes a drop honored. - **gc** roots pending tombstones like drop lines, so they survive the - loose-object sweep and the signature index keeps their signatures - (`transcripts::gc_prune` runs before `signatures::gc_prune`). -- Trust is still checked at evaluation time and still not retroactive: - removing a signer later does not undo a recorded drop. + loose-object sweep, and the signature index keeps their (verified) + signatures (`transcripts::gc_prune` runs before `signatures::gc_prune`; a + test pins it). An unreadable pending file fails gc rather than reading as + empty. +- Trust is checked at evaluation time and is not retroactive: removing a + signer later does not undo a recorded drop. -### Review notes (accepted limits) +### Other accepted limits -- A receiver skips a tombstone it has already recorded as a drop, so the - all-drops resend costs the recorded set one read per ingest, not a check and - a file rewrite per tombstone, and a recorded drop is never queued as pending. - Re-evaluation scans `.sc/signatures` once per trigger (time proportional to - the file, memory proportional to the pending targets, at most 256 ids each). -- Index files are written without a lock on the serve ingest path, as - `.sc/signatures` and `.sc/transcript-drops` already are; two concurrent - pushes can lose one pending entry. A fetching peer is re-sent the tombstone - and recovers; a push-only remote needs the signer trusted before the push. + the file, memory proportional to the pending targets). - A bucket serves every signed tombstone-shaped object it holds, and its contents are whatever writers uploaded; receivers still judge each one. - -### Not changed - -A tombstone a repo already held when it upgraded to this build has no pending -entry, so a push-only remote in that state still needs the signer trusted -before the drop is pushed (a fetching peer is re-sent it and recovers). A -one-time full-store scan would fix it but is unbounded, so it is not done. -`Repo::open` now may update the derived drop indexes under the lock it already -takes; failures there are ignored, not reported. A pending entry also costs -re-evaluation work on each ingest, including a lazy by-id gap fill, up to the -1024 cap. - -`push` still returns early when the remote tip equals the local tip, so a -drop alone does not push (unchanged). Clone `reindex` is unchanged: it -rebuilds drops from the store and does not touch the pending file. +- A tombstone a repo already held when it upgraded to this build has no + pending entry, so a push-only remote in that state still needs the signer + trusted before the drop is pushed (a fetching peer is re-sent it and + recovers). A full-store scan would fix it but is unbounded, so it is not + done. +- `Repo::open` and `LocalTransport::open` may update the derived drop indexes + when the trusted-signer set changed; failures there are ignored, not + reported. No key material is involved. +- `push` still returns early when the remote tip equals the local tip, so a + drop alone does not push (unchanged). Clone `reindex` rebuilds drops from + the store (verifying signatures as above) and does not touch the pending + file. From d066dbbadbae52d9a3c417a458c0145c721f69c0 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:51:14 +0300 Subject: [PATCH 09/13] fix(repo): keep the trust fingerprint when a pending entry cannot be read (ref #230) --- crates/repo/src/transcripts.rs | 35 ++++++++++++++++------ docs/adr/0038-agent-session-transcripts.md | 9 +++++- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 5fcd6a3a..b3171e8a 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -249,8 +249,12 @@ pub(crate) fn reevaluate_if_trust_changed( if !trust_changed(layout) { return Ok(0); } - let honored = reevaluate_pending(layout, store)?; - save_trust_fp(layout)?; + let (honored, incomplete) = reevaluate_inner(layout, store)?; + // An entry that could not be read keeps the old fingerprint, so the next + // trigger tries again once the object is repaired. + if !incomplete { + save_trust_fp(layout)?; + } Ok(honored) } @@ -514,10 +518,17 @@ pub(crate) fn index_incoming( /// object is gone, or that no longer decodes as a tombstone, is forgotten. /// Returns how many drops were honored. pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) -> Result { + Ok(reevaluate_inner(layout, store)?.0) +} + +/// [`reevaluate_pending`] plus whether some entry could not be judged because +/// an object failed to read. +fn reevaluate_inner(layout: &Layout, store: &mut scl_core::Store) -> Result<(usize, bool)> { let pending = read_pending(layout)?; if pending.is_empty() { - return Ok(0); + return Ok((0, false)); } + let mut incomplete = false; let trusted = trusted_signers(layout); let wanted: BTreeSet = pending.iter().copied().collect(); let sigs = crate::signatures::index_for_targets(layout, &wanted, MAX_SIG_READS)?; @@ -531,6 +542,7 @@ pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) - Ok(_) => continue, // Unreadable now, maybe fine later: keep it. Err(_) => { + incomplete = true; keep.push(*id); continue; } @@ -540,17 +552,22 @@ pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) - }; // An unreadable object keeps the entry for the next trigger rather // than aborting the rest. - if tombstone_honored(store, &trusted, &sigs, *id, &t, dropped).unwrap_or(false) { - append_drop(layout, (t.snapshot, *id, dropped))?; - honored += 1; - } else { - keep.push(*id); + match tombstone_honored(store, &trusted, &sigs, *id, &t, dropped) { + Ok(true) => { + append_drop(layout, (t.snapshot, *id, dropped))?; + honored += 1; + } + Ok(false) => keep.push(*id), + Err(_) => { + incomplete = true; + keep.push(*id); + } } } if keep.len() != pending.len() { write_pending(layout, &keep)?; } - Ok(honored) + Ok((honored, incomplete)) } /// Full-store rebuild of the transcript index (clone path): scan every diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index bc17db0c..f9c67488 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -505,7 +505,8 @@ never re-runs. connection) and before the indexing step of every ingest except a `--no-transcripts` one. All are best effort: a failure never fails an open or a transfer, and the fingerprint is saved only after a successful pass, - so the next trigger retries. + so the next trigger retries (this includes an object that fails to + read: the fingerprint stays old until a pass completes). - **Lost-update safety.** `append_drop` appends one short line with `O_APPEND` and `fsync` instead of rewriting the file, so concurrent ingests (the HTTP serve path holds no repo lock) cannot lose a drop line. A @@ -532,6 +533,12 @@ never re-runs. ### Other accepted limits +- Verifying a signature before indexing costs one Ed25519 check per incoming + signature object, linear in the pack size like hash-checking its records. A + per-ingest verification budget was considered and refused: it would drop + legitimate signatures from a large fetch (a long signed history), which is a + worse failure than the cost. + - Re-evaluation scans `.sc/signatures` once per trigger (time proportional to the file, memory proportional to the pending targets). - A bucket serves every signed tombstone-shaped object it holds, and its From 05b5114d6ec17d1c173694cff9fb19383ff97dd4 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:55:48 +0300 Subject: [PATCH 10/13] fix(repo): reevaluate_pending is test-only now (ref #230) --- crates/repo/src/transcripts.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index b3171e8a..a564796f 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -517,6 +517,7 @@ pub(crate) fn index_incoming( /// bounded by [`MAX_PENDING`] and zero when nothing is pending. An entry whose /// object is gone, or that no longer decodes as a tombstone, is forgotten. /// Returns how many drops were honored. +#[cfg(test)] pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) -> Result { Ok(reevaluate_inner(layout, store)?.0) } From 9019ff483d7a5b5255d05c288e1206fffa5ce9fb Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:07:08 +0300 Subject: [PATCH 11/13] fix(repo): fsync the .sc dir when the drop index is created; record ordering limits (ref #230) --- crates/repo/src/transcripts.rs | 8 +++++++- docs/adr/0038-agent-session-transcripts.md | 10 ++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index a564796f..4e5333f2 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -122,12 +122,18 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { entry.1.to_hex(), entry.2.to_hex() ); + let path = layout.transcript_drops_path(); + let is_new = !path.exists(); let mut f = std::fs::OpenOptions::new() .append(true) .create(true) - .open(layout.transcript_drops_path())?; + .open(&path)?; f.write_all(line.as_bytes())?; f.sync_data()?; + if is_new { + // Make the new directory entry durable too. + std::fs::File::open(&layout.dot_sc)?.sync_all()?; + } Ok(()) } diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index f9c67488..e033039f 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -533,6 +533,16 @@ never re-runs. ### Other accepted limits +- Ordering across separate pushes on a push-only remote. A tombstone that + arrives before its snapshot is not kept as pending, and a signature that + arrives before its target is not indexed (the target must be held); a later + push resends neither, because the remote already has the object. Within one + pack (the normal case: snapshot, transcript, tombstone and signature travel + together) the order does not matter, and a fetching peer is re-sent + everything. The signature-driven lookup of a late signature is capped at + 1024 distinct held targets per batch, so a pack that signs more held objects + than that before the tombstone's signature can skip it. + - Verifying a signature before indexing costs one Ed25519 check per incoming signature object, linear in the pack size like hash-checking its records. A per-ingest verification budget was considered and refused: it would drop From e9aae444e0f4f91c39fecc65796ed8cc809640f0 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:29:40 +0300 Subject: [PATCH 12/13] fix(repo): one trust snapshot per pass, bounded retries, torn-tail-safe drop append, ordering and forgery tests; docs (ref #230) --- ARCHITECTURE.md | 4 +- crates/repo/src/signatures.rs | 84 +++++++ crates/repo/src/transcripts.rs | 243 +++++++++++++++++---- docs/adr/0032-signed-commits-provenance.md | 14 +- docs/adr/0038-agent-session-transcripts.md | 25 ++- 5 files changed, 320 insertions(+), 50 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 197dda93..fe2b2d60 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -809,7 +809,9 @@ key and an Ed25519 signing key in a single `scl-id-` file (a v1 `scl-sk-` identity still parses and encrypts, but can't sign). Verification (`sig_status`) is strict four-state — `Invalid` beats `Trusted` beats `Untrusted` beats `Unsigned` — and `sc verify [] [--require]` walks -every parent, not just the mainline. +every parent, not just the mainline. A received signature is verified before +it is indexed and a forged one is dropped (ADR-0038 addendum, #230), so a +transfer cannot make a snapshot `Invalid`. Transfer needed zero wire changes: signatures ride the existing pack as ordinary objects, reindexed by id on receipt. `recipients.toml` gains diff --git a/crates/repo/src/signatures.rs b/crates/repo/src/signatures.rs index 69e3261e..69c75dbd 100644 --- a/crates/repo/src/signatures.rs +++ b/crates/repo/src/signatures.rs @@ -723,4 +723,88 @@ mod tests { drop(repo); std::fs::remove_dir_all(&root).unwrap(); } + + /// Write `objs` as a pack under `repo` and ingest it like a transfer. + fn ingest(repo: &Repo, objs: &[Object]) { + let enc: Vec<(ObjectId, Vec)> = objs.iter().map(|o| (o.id(), o.encode())).collect(); + let (pack, _idx) = scl_core::pack::build_pack(&enc).unwrap(); + let path = repo.layout().dot_sc.join("incoming.pack"); + std::fs::write(&path, pack).unwrap(); + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + crate::transport::ingest_pack_file(repo.layout(), &mut store, &path).unwrap(); + std::fs::remove_file(&path).unwrap(); + } + + #[test] + fn a_signature_sorting_below_its_snapshot_in_the_pack_is_still_indexed() { + // The pack is ordered by id, so the signature can come first; every + // object is written before indexing, so its target is held anyway. + let a_root = tmp_root("order-a"); + let b_root = tmp_root("order-b"); + let a = Repo::init(&a_root).unwrap(); + let (_s, identity) = scl_crypto::generate_identity_v2(); + let key = identity.signing.as_ref().unwrap(); + let (snap, snap_obj, sig_obj) = loop { + std::fs::write(a_root.join("f"), format!("{:?}", std::time::Instant::now())).unwrap(); + let snap = a.commit("t", "c").unwrap(); + let sig_obj = Object::Signature(SignatureObj { + snapshot: snap, + signer: key.public().to_bytes(), + sig: scl_crypto::sign_snapshot_id(key, snap.as_bytes()), + }); + if sig_obj.id() < snap { + let snap_obj = a.store_arc().lock().unwrap().get(&snap).unwrap(); + break (snap, snap_obj, sig_obj); + } + }; + let b = Repo::init(&b_root).unwrap(); + let mut objs = [snap_obj, sig_obj.clone()]; + objs.sort_by_key(Object::id); + assert_eq!(objs[0].id(), sig_obj.id(), "the signature leads the pack"); + ingest(&b, &objs); + let mut trust = HashMap::new(); + trust.insert(key.public().to_bytes(), "alice".to_string()); + assert_eq!( + b.sig_status(&snap, &trust).unwrap(), + SigStatus::Trusted("alice".to_string()) + ); + drop((a, b)); + std::fs::remove_dir_all(&a_root).unwrap(); + std::fs::remove_dir_all(&b_root).unwrap(); + assert!(!a_root.exists() && !b_root.exists()); + } + + #[test] + fn a_forged_snapshot_signature_over_the_wire_leaves_the_snapshot_unsigned() { + // Pins the ingest filter: a received forgery is not indexed, so it + // cannot turn a snapshot Invalid (ADR-0038, #230). + let a_root = tmp_root("forge-a"); + let b_root = tmp_root("forge-b"); + let a = Repo::init(&a_root).unwrap(); + std::fs::write(a_root.join("f"), b"x").unwrap(); + let snap = a.commit("t", "c").unwrap(); + let snap_obj = a.store_arc().lock().unwrap().get(&snap).unwrap(); + let (_s, identity) = scl_crypto::generate_identity_v2(); + let signer = identity.signing.as_ref().unwrap().public().to_bytes(); + let forged = Object::Signature(SignatureObj { + snapshot: snap, + signer, + sig: [9u8; 64], + }); + let b = Repo::init(&b_root).unwrap(); + ingest(&b, &[snap_obj, forged.clone()]); + let mut trust = HashMap::new(); + trust.insert(signer, "alice".to_string()); + assert_eq!(b.sig_status(&snap, &trust).unwrap(), SigStatus::Unsigned); + assert!(b.signatures_for(&snap).unwrap().is_empty()); + assert!( + b.store_arc().lock().unwrap().contains(&forged.id()), + "the object is stored, only the index refuses it" + ); + drop((a, b)); + std::fs::remove_dir_all(&a_root).unwrap(); + std::fs::remove_dir_all(&b_root).unwrap(); + assert!(!a_root.exists() && !b_root.exists()); + } } diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 4e5333f2..56a879fb 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -107,7 +107,8 @@ fn dropped_ids(layout: &Layout) -> Result> { } /// Record one tombstone, deduping an identical existing line. The line is -/// appended (one short `O_APPEND` write, atomic on a local filesystem), not +/// appended (one short `O_APPEND` write; atomic on a local filesystem only, NFS +/// gives no such guarantee), not /// merged into a rewrite of the file, so concurrent ingests (the HTTP serve /// path holds no repo lock) cannot lose each other's drop line. A racing /// duplicate line is harmless: readers treat the file as a set. @@ -116,14 +117,18 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { if read_drops(layout)?.contains(&entry) { return Ok(()); } + let path = layout.transcript_drops_path(); + let is_new = !path.exists(); + // A crash or ENOSPC mid-write can leave a line without its newline; start + // on a fresh line so this one is not merged into the torn tail. + let torn = std::fs::read(&path).is_ok_and(|b| b.last().is_some_and(|c| *c != b'\n')); let line = format!( - "{} {} {}\n", + "{}{} {} {}\n", + if torn { "\n" } else { "" }, entry.0.to_hex(), entry.1.to_hex(), entry.2.to_hex() ); - let path = layout.transcript_drops_path(); - let is_new = !path.exists(); let mut f = std::fs::OpenOptions::new() .append(true) .create(true) @@ -182,8 +187,9 @@ fn write_pending(layout: &Layout, ids: &[ObjectId]) -> Result<()> { } /// Remember `new` tombstones for re-evaluation, deduped, newest last, capped -/// at [`MAX_PENDING`]. Writes only when the set changes. -fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { +/// at [`MAX_PENDING`]. Writes only when the set changes. `fp` is the trusted +/// set they were just judged under. +fn note_pending(layout: &Layout, new: &[ObjectId], fp: &str) -> Result<()> { if new.is_empty() { return Ok(()); } @@ -208,59 +214,90 @@ fn note_pending(layout: &Layout, new: &[ObjectId]) -> Result<()> { all.drain(..excess); write_pending(layout, &all)?; // Entries are judged under the trust in force now; remember which. - if !layout.transcript_pending_trust_path().exists() { - save_trust_fp(layout)?; + if read_trust_state(layout).is_none() { + save_trust_state(layout, fp, 0)?; } Ok(()) } +type Trusted = std::collections::HashMap<[u8; 32], String>; + +/// Failed passes (an object would not read) after which the same trusted set +/// is not retried again, so an unreadable object cannot make every open or +/// served connection take the exclusive lock forever. A changed set retries. +const MAX_RETRIES: u32 = 3; + /// Fingerprint of the trusted-signer key set (the only configuration the /// honoring check reads), so an unrelated edit to `recipients.toml` does not -/// count as a trust change. -fn trust_fp(layout: &Layout) -> String { - let mut keys: Vec<[u8; 32]> = trusted_signers(layout).into_keys().collect(); +/// count as a trust change. Computed from the same map used for judging, so +/// the value saved is the set the entries were judged under. +fn fp_of(trusted: &Trusted) -> String { + let mut keys: Vec<&[u8; 32]> = trusted.keys().collect(); keys.sort(); let mut h = blake3::Hasher::new(); for k in keys { - h.update(&k); + h.update(k.as_slice()); } h.finalize().to_hex().to_string() } -fn save_trust_fp(layout: &Layout) -> Result<()> { +/// `.sc/transcript-pending-trust`: ` `. +fn read_trust_state(layout: &Layout) -> Option<(String, u32)> { + let s = std::fs::read_to_string(layout.transcript_pending_trust_path()).ok()?; + let mut it = s.split_whitespace(); + let fp = it.next()?.to_string(); + Some((fp, it.next().and_then(|n| n.parse().ok()).unwrap_or(0))) +} + +fn save_trust_state(layout: &Layout, fp: &str, failed: u32) -> Result<()> { scl_core::fsutil::atomic_write_durable( &layout.transcript_pending_trust_path(), - trust_fp(layout).as_bytes(), + format!("{fp} {failed}").as_bytes(), )?; Ok(()) } -/// Whether tombstones are pending and the trusted-signer set differs from the -/// one they were last judged under. One `exists` check when nothing is -/// pending; no lock and no store access, so a server can call it per -/// connection. +/// Pending tombstones exist and were last judged under another trusted set, +/// or a pass failed and retries remain. +fn needs_judging(layout: &Layout, fp: &str) -> bool { + layout.transcript_pending_path().exists() + && match read_trust_state(layout) { + None => true, + Some((saved, failed)) => saved != fp || (failed > 0 && failed < MAX_RETRIES), + } +} + +/// Whether [`reevaluate_if_trust_changed`] has work. One `exists` check when +/// nothing is pending; no lock and no store access, so a server can call it +/// per connection. pub(crate) fn trust_changed(layout: &Layout) -> bool { layout.transcript_pending_path().exists() - && std::fs::read_to_string(layout.transcript_pending_trust_path()).ok() - != Some(trust_fp(layout)) + && needs_judging(layout, &fp_of(&trusted_signers(layout))) } -/// [`reevaluate_pending`] when [`trust_changed`], then record the new trust. -/// The re-check triggers (open, transport open, ingest) all go through here -/// so none of them does work or takes a lock while the trust is unchanged. +/// Judge the pending tombstones again when [`trust_changed`], then record the +/// trust they were judged under. The re-check triggers (open, transport open, +/// ingest) all go through here so none of them does work or takes a lock +/// while the trust is unchanged. An entry that could not be read keeps the +/// pass "failed" (up to [`MAX_RETRIES`] times per trusted set) instead of +/// advancing the fingerprint, so repairing the object lets a later trigger +/// try again. pub(crate) fn reevaluate_if_trust_changed( layout: &Layout, store: &mut scl_core::Store, ) -> Result { - if !trust_changed(layout) { + let trusted = trusted_signers(layout); + let fp = fp_of(&trusted); + if !needs_judging(layout, &fp) { return Ok(0); } - let (honored, incomplete) = reevaluate_inner(layout, store)?; - // An entry that could not be read keeps the old fingerprint, so the next - // trigger tries again once the object is repaired. - if !incomplete { - save_trust_fp(layout)?; - } + let (honored, incomplete) = reevaluate_inner(layout, store, &trusted)?; + let failed = match (incomplete, read_trust_state(layout)) { + (false, _) => 0, + (true, Some((saved, n))) if saved == fp => n + 1, + (true, _) => 1, + }; + save_trust_state(layout, &fp, failed)?; Ok(honored) } @@ -513,7 +550,7 @@ pub(crate) fn index_incoming( _ => {} } } - note_pending(layout, &held)?; + note_pending(layout, &held, &fp_of(&trusted))?; Ok(count) } @@ -525,18 +562,21 @@ pub(crate) fn index_incoming( /// Returns how many drops were honored. #[cfg(test)] pub(crate) fn reevaluate_pending(layout: &Layout, store: &mut scl_core::Store) -> Result { - Ok(reevaluate_inner(layout, store)?.0) + Ok(reevaluate_inner(layout, store, &trusted_signers(layout))?.0) } /// [`reevaluate_pending`] plus whether some entry could not be judged because /// an object failed to read. -fn reevaluate_inner(layout: &Layout, store: &mut scl_core::Store) -> Result<(usize, bool)> { +fn reevaluate_inner( + layout: &Layout, + store: &mut scl_core::Store, + trusted: &Trusted, +) -> Result<(usize, bool)> { let pending = read_pending(layout)?; if pending.is_empty() { return Ok((0, false)); } let mut incomplete = false; - let trusted = trusted_signers(layout); let wanted: BTreeSet = pending.iter().copied().collect(); let sigs = crate::signatures::index_for_targets(layout, &wanted, MAX_SIG_READS)?; let (mut keep, mut honored) = (Vec::new(), 0); @@ -559,7 +599,7 @@ fn reevaluate_inner(layout: &Layout, store: &mut scl_core::Store) -> Result<(usi }; // An unreadable object keeps the entry for the next trigger rather // than aborting the rest. - match tombstone_honored(store, &trusted, &sigs, *id, &t, dropped) { + match tombstone_honored(store, trusted, &sigs, *id, &t, dropped) { Ok(true) => { append_drop(layout, (t.snapshot, *id, dropped))?; honored += 1; @@ -1667,7 +1707,7 @@ pub(crate) mod tests { let b = ObjectId::of(b"b"); let mut batch = vec![a]; batch.extend(std::iter::repeat_n(b, MAX_PENDING + 10)); - note_pending(repo.layout(), &batch).unwrap(); + note_pending(repo.layout(), &batch, "fp").unwrap(); assert_eq!(read_pending(repo.layout()).unwrap(), vec![a, b]); drop(repo); @@ -1682,8 +1722,8 @@ pub(crate) mod tests { let ids: Vec = (0..MAX_PENDING + 5) .map(|i| ObjectId::of(format!("t{i}").as_bytes())) .collect(); - note_pending(repo.layout(), &ids).unwrap(); - note_pending(repo.layout(), &ids[10..13]).unwrap(); // dedup, no growth + note_pending(repo.layout(), &ids, "fp").unwrap(); + note_pending(repo.layout(), &ids[10..13], "fp").unwrap(); // dedup, no growth let held = read_pending(repo.layout()).unwrap(); assert_eq!(held.len(), MAX_PENDING); assert_eq!(held[0], ids[5], "oldest evicted"); @@ -1700,4 +1740,131 @@ pub(crate) mod tests { std::fs::remove_dir_all(&root).unwrap(); assert!(!root.exists()); } + + #[test] + fn a_drop_signature_sorting_below_its_tombstone_in_the_pack_is_indexed_and_honored() { + let root = tmp_root("sig-below-tomb"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let key = owner.signing.as_ref().unwrap(); + trust_signer(&repo, &owner); + // Sealing is randomized, so each attach mints a new body (and tombstone + // id) until the signature's id sorts below the tombstone's. + let (body, tomb_obj, sig_obj) = loop { + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let tomb_obj = Object::Transcript(Transcript::drop_tombstone(snap, &body)); + let tomb = tomb_obj.id(); + let sig_obj = Object::Signature(SignatureObj { + snapshot: tomb, + signer: key.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(key, &drop_digest(&tomb)), + }); + if sig_obj.id() < tomb { + break (body, tomb_obj, sig_obj); + } + }; + let _ = body; + let mut objs = [tomb_obj, sig_obj.clone()]; + objs.sort_by_key(Object::id); + assert_eq!(objs[0].id(), sig_obj.id()); + let enc: Vec<(ObjectId, Vec)> = objs.iter().map(|o| (o.id(), o.encode())).collect(); + let (pack, _idx) = scl_core::pack::build_pack(&enc).unwrap(); + let path = root.join("in.pack"); + std::fs::write(&path, pack).unwrap(); + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + crate::transport::ingest_pack_file(repo.layout(), &mut store, &path).unwrap(); + drop(store); + assert_eq!(read_drops(repo.layout()).unwrap().len(), 1); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn a_torn_drop_line_does_not_swallow_the_next_append() { + let root = tmp_root("torn"); + let repo = Repo::init(&root).unwrap(); + let e = |t: &str| ObjectId::of(t.as_bytes()); + append_drop(repo.layout(), (e("s1"), e("t1"), e("d1"))).unwrap(); + // A crash left half a line with no newline. + let path = repo.layout().transcript_drops_path(); + let mut f = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .unwrap(); + std::io::Write::write_all(&mut f, &e("s2").to_hex().as_bytes()[..20]).unwrap(); + drop(f); + append_drop(repo.layout(), (e("s3"), e("t3"), e("d3"))).unwrap(); + let drops = read_drops(repo.layout()).unwrap(); + assert_eq!(drops.len(), 2, "the torn line is skipped, the next is kept"); + assert!(drops.contains(&(e("s3"), e("t3"), e("d3")))); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn an_unreadable_pending_object_is_retried_a_bounded_number_of_times() { + let root = tmp_root("retries"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let arc = repo.store_arc(); + let tomb = arc + .lock() + .unwrap() + .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .unwrap(); + repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) + .unwrap(); + let mut store = arc.lock().unwrap(); + index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + assert_eq!(read_pending(repo.layout()).unwrap(), vec![tomb]); + drop(store); + + // Make the tombstone unreadable but still "contained". + let path = std::fs::read_dir(repo.layout().objects_dir()) + .unwrap() + .flat_map(|d| std::fs::read_dir(d.unwrap().path()).into_iter().flatten()) + .map(|f| f.unwrap().path()) + .find(|p| { + p.file_name() + .unwrap() + .to_string_lossy() + .starts_with(&tomb.to_hex()[2..]) + }); + let path = path.expect("the tombstone is a loose object"); + let saved = std::fs::read(&path).unwrap(); + std::fs::write(&path, b"garbage").unwrap(); + // A fresh store, so no cached copy hides the damage. + let mut fresh = + scl_core::Store::open_persistent(repo.layout().objects_dir(), 1 << 20).unwrap(); + trust_signer(&repo, &owner); + for _ in 0..MAX_RETRIES { + assert!(trust_changed(repo.layout())); + reevaluate_if_trust_changed(repo.layout(), &mut fresh).unwrap(); + } + assert!(!trust_changed(repo.layout()), "retries are bounded"); + assert_eq!( + read_pending(repo.layout()).unwrap(), + vec![tomb], + "entry kept" + ); + + std::fs::write(&path, saved).unwrap(); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } } diff --git a/docs/adr/0032-signed-commits-provenance.md b/docs/adr/0032-signed-commits-provenance.md index 57a583c9..d8756c0f 100644 --- a/docs/adr/0032-signed-commits-provenance.md +++ b/docs/adr/0032-signed-commits-provenance.md @@ -86,15 +86,19 @@ Code-verified against `crates/crypto/src/signing.rs`, found, regardless of position — a snapshot with one invalid signature and one valid-and-trusted signature is `Invalid`, full stop. Only once every signature verifies does trust ranking apply: `Trusted` beats - `Untrusted` beats `Unsigned` (no signatures at all). + `Untrusted` beats `Unsigned` (no signatures at all). Since #230 a + *received* signature is verified before it is indexed and a forged one is + dropped, so ingest never produces `Invalid` (ADR-0038, "Index only + signatures that verify"). - **Transfer needs zero wire changes.** Signatures are ordinary CAS objects, so they ride the existing pack format unmodified. The receiver side runs `put_pack` then `index_incoming` over the newly-written ids; - `index_incoming` treats a missing snapshot for an incoming signature as - a **hard error** (`NotFound`), not a skip — the seam's contract is that + `index_incoming` treats an incoming signature id that is not in the + store as a **hard error** (`NotFound`) — the seam's contract is that every id just written by `put_pack` is retrievable, so a `NotFound` - here means the transfer itself is broken, not that the signature is - stale. + here means the transfer itself is broken. A signature whose *target* is + not held, or that does not verify, is silently not indexed (#230, + ADR-0038). - **Review-caught Critical:** the first cut of `fetch` conflated "already have the snapshot" with "already have its signatures" — a re-fetch of an already-possessed commit skipped resending its diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index e033039f..ad6469ef 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -458,8 +458,11 @@ that sort ahead of the real one. `signatures::index_incoming` and `reindex` therefore index a signature only if its target is held and it verifies under the key it names, in the domain of its target kind (snapshot domain for a snapshot, transcript domain for a transcript, the drop digest for a -tombstone). Forged and dangling signatures are never indexed, so never -forwarded to a downstream peer and never kept by gc. Side effect: a received +tombstone). Forged and dangling signatures are never indexed, so a +`LocalTransport` or `sc serve` sender never forwards them (it sends indexed +signatures) and gc never keeps them. A bucket is the exception: `get_pack` +forwards the on-chain signatures it finds unverified, and the receiver's +ingest filters them. Side effect: a received signature that fails to verify no longer turns a snapshot `Invalid`; it is simply absent (`Unsigned`, which `--require` also rejects). A signature indexed before this change is read as before. @@ -504,9 +507,13 @@ never re-runs. lock is busy, so a served remote does not take the exclusive lock per connection) and before the indexing step of every ingest except a `--no-transcripts` one. All are best effort: a failure never fails an open - or a transfer, and the fingerprint is saved only after a successful pass, - so the next trigger retries (this includes an object that fails to - read: the fingerprint stays old until a pass completes). + or a transfer, and the fingerprint is saved with the pass's outcome + (` `), computed from the same trusted-signer map + the entries were judged with. A pass in which an object failed to read does + not count as done and is retried by the next trigger, at most + `MAX_RETRIES` (3) times per trusted set, so an unreadable object cannot make + every open or served connection take the exclusive lock forever; a changed + trusted set resets the count. - **Lost-update safety.** `append_drop` appends one short line with `O_APPEND` and `fsync` instead of rewriting the file, so concurrent ingests (the HTTP serve path holds no repo lock) cannot lose a drop line. A @@ -515,7 +522,13 @@ never re-runs. pending entry. That costs a push-only remote the retry for that tombstone (trust the signer before the push, or have a fetching peer re-send it), but never a recorded drop and never the rooting of one: gc roots every drop - line. + line. The same unlocked read-modify-write applies to `.sc/signatures`, + `.sc/transcripts` and the pending rewrite on the serve path; taking the + repo lock there changes `Locked` behaviour for concurrent connections and + for a local `sc` command that holds the lock for its lifetime, so it is + tracked separately as #260 rather than decided here. `O_APPEND` atomicity is + a local-filesystem guarantee (none on NFS). `append_drop` starts on a new + line when the file ends in a torn line. - **Bounded.** The file is a FIFO ring of 1024 ids. A writer who pushes many signed tombstones evicts older entries, so a genuine pending entry can be lost on a push-only remote; a fetching peer is re-sent every qualifying From b19cabfb23d8ed4f83282a0cdbc567fe468e2183 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:47:18 +0300 Subject: [PATCH 13/13] docs: forged signatures stay in the CAS but are never indexed (ref #230) --- docs/THREAT-MODEL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 7b7fc7ec..a069de7f 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -216,7 +216,9 @@ too, not just the src-control-side metadata. and signer public key of such drops to any peer the dropper transfers with, including one that never had the snapshot (ids are hashes, bodies stay sealed). Only signatures that verify for a held target are indexed, so - forged signatures are never stored or forwarded. A writer to a shared remote + forged signatures stay in the object store (until gc sweeps them) but are + never indexed, so a `sc serve` or local sender never forwards them and gc + never keeps them (a bucket forwards what it holds; receivers filter). A writer to a shared remote who signs with keys of their own can still crowd more than 256 valid signatures ahead of the real one, which keeps the drop inert on every peer that ingests them (a denial of the drop, never an honored one; #257). Writers