diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 12a47e87..c3ec84c0 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -142,7 +142,7 @@ a retired pre-P16 snapshot encoding, refused with a clear error): other object, so it flows through fork/checkout/clone untouched and stays ciphertext until an authorized context decrypts it. - **Signature** (P22) — a bytes-only object over a domain-separated snapshot id - (`Ed25519`), indexed on the side (`.sc/signatures`); the crypto stays + (`Ed25519`), indexed on the side (`.sc/signatures.v2`); the crypto stays quarantined in `crates/crypto`, so `core` sees only opaque bytes. - **Transcript** (P30) — a sealed agent-session record attached to the snapshot it motivated: the same envelope shape as a `Secret` (fresh DEK wrapped per @@ -291,7 +291,7 @@ recipient/escrow set. See ADR-0019/0027. HKDF-derived into *both* an X25519 encryption key and an Ed25519 signing key, written as a single `scl-id-` file (a pre-P22 v1 `scl-sk-` identity still parses and encrypts but cannot sign). Signatures are bytes-only CAS objects over -a domain-separated snapshot id, indexed in `.sc/signatures`; `sc verify` walks +a domain-separated snapshot id, indexed in `.sc/signatures.v2`; `sc verify` walks history and reports a four-state status (trusted ✓ / untrusted ? / invalid ✗ / unsigned). Session transcripts (P30) reuse the same signing machinery under a distinct domain. Signing binds identity to a *snapshot id*, detecting history @@ -816,7 +816,7 @@ generic `"abandoned"` a manual `ws abandon` still shows. See ADR-0031. Signatures are ordinary CAS objects (`TAG_SIGNATURE`, bytes-only in `crates/core` — no crypto crosses the quarantine) over the domain-separated -snapshot id; a local `.sc/signatures` index maps snapshot → signature ids, +snapshot id; a local `.sc/signatures.v2` index maps (snapshot, signer) → signature id, gc-rooted and pruned alongside dead snapshots. `sc keygen` now emits a **v2 identity**: one random seed, HKDF-derived into *both* an X25519 encryption key and an Ed25519 signing key in a single `scl-id-` file (a v1 @@ -983,7 +983,7 @@ a keyless clone gets ciphertext only. A one-to-many `.sc/transcripts` index (snapshot → transcript ids) means history-editing ops start each fresh snapshot with none attached, so there is no stale-provenance carry-forward to reason about. Signing is opt-in and reuses P22's machinery outright — -same `SignatureObj` type, same `.sc/signatures` index, no second index — +same `SignatureObj` type, same `.sc/signatures.v2` index, no second index — under a distinct signature domain string. Because transcripts are ordinary content-addressed CAS objects, transfer diff --git a/CLAUDE.md b/CLAUDE.md index d67f140f..290474d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,7 +145,7 @@ the code, those win. | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3. Blob-size filter (#217): `--filter blob:limit=` (combinable with prefixes; `blob:` is reserved) leaves out blobs of at least that size unless a ref tip reaches them, evaluated by the server (`GetPack.blob_limit`, wire v6); the gap is absence as in ADR-0048, `sc backfill ` or a lazy fill repairs it, `.sc/promisor-blob-limit` records limit and scope; `sc fetch` from the promisor origin sends the recorded limit again (`--no-blob-limit` opts out for one fetch); no object-count limit (rejected) | [0037](docs/adr/0037-partial-clone.md), [0055](docs/adr/0055-blob-size-clone-filter.md) | | P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token add [--expires]`/`rotate`/`remove`/`list`; an expired token gets the same 401 as an unknown one), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | -| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; retained signed tombstones ride a transfer when their snapshot is in it or was collected by the sender, only verifying signatures are indexed, and held-but-untrusted tombstones are re-judged from `.sc/transcript-pending` when the trusted-signer set changes (#230) | [0038](docs/adr/0038-agent-session-transcripts.md) | +| P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex; retained signed tombstones ride a transfer when their snapshot is in it or was collected by the sender, only verifying signatures are indexed, and held-but-untrusted tombstones are re-judged from `.sc/transcript-pending` when the trusted-signer set changes (#230); the signature index (`.sc/signatures.v2`) holds one slot per (target, signer) so attacker-made keys cannot crowd out a drop (#257), and serve-path index updates take a short `.sc/index.lock` distinct from the repo lock (#260) | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff; later `--max-conn-bytes` per-connection write budget (#233) | [0041](docs/adr/0041-listener-resource-limits.md) | | P32 | In-binary TLS: `sc+https://` via leaf crate `tlsio`; accept-new TOFU pinning (`sc tls list/forget/pin/path` manages the pin file); `SC_HTTPS_CA_FILE`/`SC_HTTPS_CA=system` validate against a CA instead; client connect/handshake bounded; public plaintext bind no longer justified by tokens alone | [0042](docs/adr/0042-in-binary-tls-sc-https.md), [0053](docs/adr/0053-tls-pin-management.md), [0054](docs/adr/0054-tls-ca-validation.md) | | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | @@ -243,6 +243,10 @@ transport-adjacent. The rest, imperatively: `sc rewrap --rotate` at the tip (fresh key per protected blob; a plain `sc rewrap` only replaces wrap lists) plus rotating the underlying credential. History keeps the old ciphertext (rotation is not erasure). +- **The signature index is `.sc/signatures.v2` (#257).** `.sc/signatures` is the + pre-#257 file, kept only for the one-time import. Do not run an older binary's + `sc gc` on a repo this build has signed in or ingested into: it roots only the + legacy file and sweeps the signatures v2 lists (ADR-0038). - **The oplog is local-only** (like a reflog) — it never travels over fetch/push/clone. `sc undo` twice = redo; undoing the initial commit is refused. diff --git a/ROADMAP.md b/ROADMAP.md index b1c97b45..d31518d9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -494,7 +494,9 @@ across every phase. a stale peer is told), only verifying signatures enter the signature index, and a remote that held a tombstone before trusting the signer re-judges it from a capped `.sc/transcript-pending` index when the trusted-signer set - changes. Crowding by valid signatures from attacker keys is tracked in #257. + changes. Valid signatures from attacker keys cannot crowd out a drop (#257: the + signature index holds one slot per (target, signer)), and serve-path index + updates take a short index lock (#260). - **Phase 32 — In-binary TLS (`sc+https://`).** Closes the audit's High #1 (plaintext bearer tokens/traffic on `sc+http://`) and ADR-0036's "no TLS" diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index 3fff3442..a5c82a7c 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -10,6 +10,10 @@ pub enum Error { RepoExists(String), #[error("repo is locked by another process (remove {0} if stale)")] Locked(String), + /// The short index-update lock (`.sc/index.lock`) stayed busy past its wait. + /// Never remove that file: it is an OS lock on a file that stays in place. + #[error("the index update lock {0} stayed busy; retry (do not remove the file)")] + IndexBusy(String), #[error("object {0} is corrupt (failed hash verification on read)")] CorruptObject(ObjectId), #[error("malformed ref: {0}")] diff --git a/crates/repo/src/gc.rs b/crates/repo/src/gc.rs index 41b7f5ba..9ebda621 100644 --- a/crates/repo/src/gc.rs +++ b/crates/repo/src/gc.rs @@ -102,6 +102,10 @@ fn walk_decided_root( pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result { let mut stats = GcStats::default(); + // A pre-#257 signature index must be imported before it is pruned: an + // unread legacy file would otherwise root nothing and lose its objects. + signatures::migrate_legacy(layout, store)?; + // Trim oplog records past the grace window BEFORE computing roots, so a // just-trimmed record's snapshot doesn't linger as a root this pass (it // always keeps the newest record regardless of cutoff). @@ -164,18 +168,27 @@ pub fn run(layout: &Layout, store: &mut Store, grace: Duration) -> Result PathBuf { self.dot_sc.join("oplog") } - /// `.sc/signatures` — the append-only snapshot-signature index (P22): - /// one ` ` line per indexed signature. + /// `.sc/signatures.v2` — the signature index (P22, keyed per + /// `(target, signer)` since #257): one ` + /// ` line per indexed signature. pub fn signatures_path(&self) -> PathBuf { + self.dot_sc.join("signatures.v2") + } + /// `.sc/signatures` — the pre-#257 index (` ` lines). Read + /// once by the migration into [`Self::signatures_path`], then left alone + /// so an older binary keeps a file it understands. + pub fn signatures_legacy_path(&self) -> PathBuf { self.dot_sc.join("signatures") } /// `.sc/sparse` — the sparse-checkout prefix spec (P24): one prefix per diff --git a/crates/repo/src/lock.rs b/crates/repo/src/lock.rs index 5b2c667c..da78bb42 100644 --- a/crates/repo/src/lock.rs +++ b/crates/repo/src/lock.rs @@ -54,6 +54,77 @@ impl RepoLock { } } +thread_local! { + /// Index locks this thread holds (by path), so a nested critical section + /// on the same thread reuses the held lock instead of deadlocking on it. + static HELD_INDEX_LOCKS: std::cell::RefCell> = const { std::cell::RefCell::new(Vec::new()) }; +} + +/// How long [`with_index_lock`] waits for another holder before giving up. +/// Tests get a long wait: their threads queue behind durable (fsynced) writes +/// on a loaded machine; the timeout path is tested with `with_index_lock_for`. +const INDEX_LOCK_WAIT: std::time::Duration = + std::time::Duration::from_secs(if cfg!(test) { 300 } else { 30 }); + +/// Run `f` holding the short-lived index lock (`.sc/index.lock`, ADR-0038 +/// "Index lock", #260). It serializes the read-modify-write of the derived +/// index files (`signatures.v2`, `transcripts`, `transcript-pending`...) +/// between concurrent serve-path ingests and local commands, WITHOUT being +/// [`RepoLock`]: it is held only around one index update (or one coupled +/// group of them), never for a command's lifetime, so it never turns a busy repo into `Locked` for a +/// local `sc` command. +/// +/// An advisory `flock` on a file that is never removed: the OS drops it when +/// the holder dies, so there is no stale-lock breaking (and none of its +/// races). Polls up to [`INDEX_LOCK_WAIT`], then `Error::IndexBusy`. Reentrant per +/// thread. Lock order: this lock is a leaf. Never acquire [`RepoLock`] (or +/// wait on anything else) while holding it; taking it under `RepoLock` is fine. +pub(crate) fn with_index_lock(layout: &Layout, f: impl FnOnce() -> Result) -> Result { + with_index_lock_for(layout, INDEX_LOCK_WAIT, f) +} + +fn with_index_lock_for( + layout: &Layout, + wait: std::time::Duration, + f: impl FnOnce() -> Result, +) -> Result { + let path = layout.dot_sc.join("index.lock"); + if HELD_INDEX_LOCKS.with(|h| h.borrow().contains(&path)) { + return f(); + } + let file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&path)?; + let deadline = std::time::Instant::now() + wait; + loop { + match file.try_lock() { + Ok(()) => break, + Err(std::fs::TryLockError::WouldBlock) if std::time::Instant::now() < deadline => { + std::thread::sleep(std::time::Duration::from_millis(2)); + } + Err(std::fs::TryLockError::WouldBlock) => { + return Err(Error::IndexBusy(path.display().to_string())) + } + Err(std::fs::TryLockError::Error(e)) => return Err(e.into()), + } + } + HELD_INDEX_LOCKS.with(|h| h.borrow_mut().push(path.clone())); + // Pops the entry (and unlocks via `file`'s drop) even if `f` panics. + struct Release(PathBuf); + impl Drop for Release { + fn drop(&mut self) { + HELD_INDEX_LOCKS.with(|h| h.borrow_mut().retain(|p| p != &self.0)); + } + } + let _release = Release(path); + let out = f(); + drop(file); + out +} + /// True only when the lock file names a PID we can *prove* is not running. /// Unreadable file, no parseable PID, or an inconclusive probe all return /// false (respect the lock). @@ -143,4 +214,53 @@ mod tests { assert_eq!(content.trim(), std::process::id().to_string()); std::fs::remove_dir_all(&layout.root).unwrap(); } + + #[test] + fn index_lock_is_not_the_repo_lock() { + // Holding either never makes the other fail: a local command keeps + // `RepoLock` for its lifetime while a serve-path ingest takes the short + // index lock, and vice versa (#260). + let layout = tmp_layout("idx-indep"); + let repo_lock = RepoLock::acquire(&layout).unwrap(); + with_index_lock(&layout, || Ok(())).unwrap(); + assert!(matches!(RepoLock::acquire(&layout), Err(Error::Locked(_)))); + drop(repo_lock); + with_index_lock(&layout, || { + // Reentrant on the same thread, and RepoLock is still free. + with_index_lock(&layout, || Ok(()))?; + drop(RepoLock::acquire(&layout)?); + Ok(()) + }) + .unwrap(); + std::fs::remove_dir_all(&layout.root).unwrap(); + assert!(!layout.root.exists()); + } + + #[test] + fn index_lock_serializes_threads_and_times_out_with_locked() { + let layout = tmp_layout("idx-wait"); + let (held_tx, held_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let l2 = layout.clone(); + let holder = std::thread::spawn(move || { + with_index_lock(&l2, || { + held_tx.send(()).unwrap(); + release_rx.recv().unwrap(); + Ok(()) + }) + .unwrap(); + }); + held_rx.recv().unwrap(); + // Another thread of the same process waits, then gives up. + assert!(matches!( + with_index_lock_for(&layout, std::time::Duration::from_millis(100), || Ok(())), + Err(Error::IndexBusy(_)) + )); + release_tx.send(()).unwrap(); + holder.join().unwrap(); + // Released: free again, and the failed waiter left no held entry. + with_index_lock(&layout, || Ok(())).unwrap(); + std::fs::remove_dir_all(&layout.root).unwrap(); + assert!(!layout.root.exists()); + } } diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 5923944b..ec5a4893 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -99,6 +99,11 @@ impl Repo { fn open_layout(layout: Layout, budget_bytes: usize) -> Result { let lock = RepoLock::acquire(&layout)?; let mut store = Store::open_persistent(layout.objects_dir(), budget_bytes)?; + // Import a pre-#257 signature index first (best effort; every writer + // and gc retry it and fail rather than start a v2 file without it): the + // re-evaluation below reads that index, and a pass that saw it empty + // would record the trust as judged. + let _ = crate::signatures::migrate_legacy(&layout, &mut store); // Honor held tombstones once the trusted-signer set changed (#230). Best // effort: a failure must not stop an open, and the next open retries. let _ = crate::transcripts::reevaluate_if_trust_changed(&layout, &mut store); diff --git a/crates/repo/src/signatures.rs b/crates/repo/src/signatures.rs index 69c75dbd..fc993972 100644 --- a/crates/repo/src/signatures.rs +++ b/crates/repo/src/signatures.rs @@ -6,14 +6,20 @@ //! producing a signature (`sign_snapshot`), storing it in the CAS, indexing //! it, and verifying it (`sig_status`). //! -//! Index file `.sc/signatures`: append-only, one ` -//! ` line per indexed signature. Multiple signers can each -//! index their own signature over the same snapshot (multiple lines sharing -//! a snapshot hex). Appends dedup (a signer signing the same snapshot twice -//! is a no-op, both because the object id is identical — Ed25519 signing is -//! deterministic — and because the index line would be identical). `gc` -//! rewrites the file atomically when it prunes dead entries; every other -//! writer treats the file as append-only. +//! Index file `.sc/signatures.v2`: one ` +//! ` line per indexed signature, keyed by `(target, signer)`: a +//! signer occupies ONE slot per target (#257), whatever it signs or sends, so +//! the entries over a target are bounded by the distinct signers and honoring +//! a drop is a lookup over the TRUSTED signers only (`SigIndex`), never a +//! scan over signatures from keys an attacker made. Appends dedup on +//! `(target, signer)`. `gc` rewrites the file atomically when it prunes dead +//! entries; every other writer appends. All writers run under the short +//! index lock (`lock::with_index_lock`, ADR-0038 "Index lock", #260). +//! +//! The pre-#257 file `.sc/signatures` (` ` lines, no signer) is +//! left alone, because an older binary reading three columns would skip the +//! lines and drop them on its next rewrite. `migrate_legacy` imports it once +//! into the v2 file, re-verifying every entry. use std::collections::BTreeSet; @@ -21,6 +27,7 @@ use scl_core::{Object, ObjectId, SignatureObj}; use crate::error::{Error, Result}; use crate::layout::Layout; +use crate::lock::with_index_lock; use crate::repo::Repo; /// Verification status of one snapshot's signatures against a trust map. @@ -41,21 +48,25 @@ pub enum SigStatus { Unsigned, } -/// Parse one ` ` index line. Returns `None` for a -/// malformed line (wrong field count or bad hex) rather than erroring — the -/// index is a best-effort cache, not the source of truth (the CAS objects -/// are), so a corrupt line is skipped, not fatal. -fn parse_line(line: &str) -> Option<(ObjectId, ObjectId)> { +/// One index entry: `(target, signature object, signer)`. +type Entry = (ObjectId, ObjectId, [u8; 32]); + +/// Parse one ` ` index line. Returns `None` +/// for a malformed line rather than erroring: the index is a best-effort +/// cache (the CAS objects are the source of truth), so a corrupt line is +/// skipped, not fatal. +fn parse_line(line: &str) -> Option { let mut parts = line.split_whitespace(); - let snap = parts.next()?.parse::().ok()?; + let target = parts.next()?.parse::().ok()?; let sig = parts.next()?.parse::().ok()?; + let signer: [u8; 32] = hex::decode(parts.next()?).ok()?.try_into().ok()?; if parts.next().is_some() { return None; } - Some((snap, sig)) + Some((target, sig, signer)) } -fn read_index(layout: &Layout) -> Result> { +fn read_index(layout: &Layout) -> Result> { let path = layout.signatures_path(); let contents = match std::fs::read_to_string(&path) { Ok(s) => s, @@ -66,25 +77,183 @@ fn read_index(layout: &Layout) -> Result> { } /// Rewrite the whole index file atomically (temp file + rename), so a -/// concurrent reader never observes a half-written index. -fn write_index(layout: &Layout, entries: &[(ObjectId, ObjectId)]) -> Result<()> { +/// concurrent reader never observes a half-written index. Caller holds the +/// index lock. +fn write_index(layout: &Layout, entries: &[Entry]) -> Result<()> { let mut out = String::new(); - for (snap, sig) in entries { - out.push_str(&format!("{} {}\n", snap.to_hex(), sig.to_hex())); + for (target, sig, signer) in entries { + out.push_str(&format!( + "{} {} {}\n", + target.to_hex(), + sig.to_hex(), + hex::encode(signer) + )); } scl_core::fsutil::atomic_write_durable(&layout.signatures_path(), out.as_bytes())?; Ok(()) } -/// Append one `(snapshot, sig_id)` entry, deduping against an identical -/// existing entry. -pub(crate) fn append_index(layout: &Layout, snapshot: ObjectId, sig_id: ObjectId) -> Result<()> { - let mut entries = read_index(layout)?; - if entries.iter().any(|(s, g)| *s == snapshot && *g == sig_id) { +/// Index `new`, skipping any `(target, signer)` already present (the first +/// entry keeps the slot, and Ed25519 is deterministic, so a signer has one +/// valid signature per target anyway). One locked read and one appended write +/// for the whole batch: the file is never rewritten here, so a large index +/// (one line per distinct signer a writer made up) costs a read, not a +/// rewrite and fsync of everything, while the lock is held. Imports a +/// pre-#257 index first, so a v2 file is never started while legacy entries +/// are unread. +fn append_entries(layout: &Layout, store: &mut scl_core::Store, new: &[Entry]) -> Result<()> { + use std::io::Write; + if new.is_empty() { + return Ok(()); + } + migrate_legacy(layout, store)?; + with_index_lock(layout, || { + let path = layout.signatures_path(); + let text = match std::fs::read_to_string(&path) { + Ok(t) => Some(t), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(Error::Io(e)), + }; + let mut slots: BTreeSet<(ObjectId, [u8; 32])> = text + .iter() + .flat_map(|t| t.lines().filter_map(parse_line)) + .map(|(t, _, s)| (t, s)) + .collect(); + let mut out = String::new(); + // A crash or ENOSPC mid-write can leave a line without its newline; + // start on a fresh line so this one is not merged into the torn tail. + if text + .as_ref() + .is_some_and(|t| !t.is_empty() && !t.ends_with('\n')) + { + out.push('\n'); + } + let mut added = false; + for (target, sig, signer) in new { + if slots.insert((*target, *signer)) { + added = true; + out.push_str(&format!( + "{} {} {}\n", + target.to_hex(), + sig.to_hex(), + hex::encode(signer) + )); + } + } + if !added { + return Ok(()); + } + let mut f = std::fs::OpenOptions::new() + .append(true) + .create(true) + .open(&path)?; + f.write_all(out.as_bytes())?; + f.sync_data()?; + if text.is_none() { + // Make the new directory entry durable too. + scl_core::fsutil::fsync_dir(&layout.dot_sc)?; + } + Ok(()) + }) +} + +/// Index the signature `sig_id`, just put in `store`, if it is acceptable +/// (see [`acceptable`]): the local signing paths hold the same bar as ingest, +/// so a signature in the wrong domain for its target (a plain transcript +/// signature over a drop tombstone) can never take its signer's slot. +pub(crate) fn append_index( + layout: &Layout, + store: &mut scl_core::Store, + sig_id: ObjectId, +) -> Result<()> { + let Object::Signature(sig) = store.get(&sig_id)? else { + return Err(Error::InvalidArgument(format!( + "{sig_id} is not a signature object" + ))); + }; + if !acceptable(store, &sig)? { + return Err(Error::InvalidArgument(format!( + "signature {sig_id} does not verify for its target {} in its signing domain", + sig.snapshot + ))); + } + append_entries(layout, store, &[(sig.snapshot, sig_id, sig.signer)]) +} + +/// A failure to read or decode one object, as opposed to an I/O problem that +/// may pass: the object will not become readable by retrying. +fn permanent(e: &Error) -> bool { + matches!( + e, + Error::CorruptObject(_) + | Error::Core( + scl_core::Error::NotFound(_) + | scl_core::Error::Malformed(_) + | scl_core::Error::WrongKind(..) + | scl_core::Error::PackCorrupt(_) + | scl_core::Error::BadPackIndex(_) + ) + ) +} + +/// Whether a pre-#257 index is waiting to be imported. Anything that judges +/// tombstones against the index must not run meanwhile: it would see no +/// signatures and record the trust as judged. +pub(crate) fn migration_pending(layout: &Layout) -> bool { + layout.signatures_legacy_path().exists() && !layout.signatures_path().exists() +} + +/// Import a pre-#257 `.sc/signatures` (no signer column) into the v2 index, +/// once. The signer comes from the signature object, and each entry goes +/// through [`acceptable`] again: an index written before #230 may hold +/// forgeries, and one naming a trusted key would take that signer's slot. +/// Entries whose object is gone or fails are dropped. No-op when there is no +/// legacy file or the v2 file exists. The legacy file stays in place. +pub(crate) fn migrate_legacy(layout: &Layout, store: &mut scl_core::Store) -> Result<()> { + let legacy = layout.signatures_legacy_path(); + if layout.signatures_path().exists() || !legacy.exists() { return Ok(()); } - entries.push((snapshot, sig_id)); - write_index(layout, &entries) + with_index_lock(layout, || { + if layout.signatures_path().exists() { + return Ok(()); + } + let text = match std::fs::read_to_string(&legacy) { + Ok(t) => t, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(Error::Io(e)), + }; + let mut slots = BTreeSet::new(); + let mut entries: Vec = Vec::new(); + for line in text.lines() { + let mut p = line.split_whitespace(); + let (Some(target), Some(id), None) = ( + p.next().and_then(|t| t.parse::().ok()), + p.next().and_then(|t| t.parse::().ok()), + p.next(), + ) else { + continue; + }; + // An object that is gone or will not decode is dropped like any + // entry that does not verify: one bad object must not stop every + // index write. A transient I/O error still fails the import. + let checked = (|| -> Result> { + let Object::Signature(sig) = store.get(&id)? else { + return Ok(None); + }; + Ok((sig.snapshot == target && acceptable(store, &sig)?).then_some(sig.signer)) + })(); + match checked { + Ok(Some(signer)) if slots.insert((target, signer)) => { + entries.push((target, id, signer)); + } + Ok(_) => {} + Err(e) if permanent(&e) => {} + Err(e) => return Err(e), + } + } + write_index(layout, &entries) + }) } /// gc integration: given the reachable object set already computed by @@ -101,22 +270,24 @@ pub(crate) fn append_index(layout: &Layout, snapshot: ObjectId, sig_id: ObjectId /// Returns the number of dropped entries (0 when nothing needed pruning, in /// which case the index file is left untouched rather than rewritten). pub(crate) fn gc_prune(layout: &Layout, reachable: &mut BTreeSet) -> Result { - let entries = read_index(layout)?; - if entries.is_empty() { - return Ok(0); - } - let mut kept = Vec::with_capacity(entries.len()); - for (snap, sig_id) in &entries { - if reachable.contains(snap) { - reachable.insert(*sig_id); - kept.push((*snap, *sig_id)); + with_index_lock(layout, || { + let entries = read_index(layout)?; + if entries.is_empty() { + return Ok(0); } - } - let dropped = entries.len() - kept.len(); - if dropped > 0 { - write_index(layout, &kept)?; - } - Ok(dropped) + let mut kept = Vec::with_capacity(entries.len()); + for e in &entries { + if reachable.contains(&e.0) { + reachable.insert(e.1); + kept.push(*e); + } + } + let dropped = entries.len() - kept.len(); + if dropped > 0 { + write_index(layout, &kept)?; + } + Ok(dropped) + }) } /// Receiver-side seam (Task 3): given a batch of object ids just written to @@ -137,24 +308,50 @@ pub(crate) fn gc_prune(layout: &Layout, reachable: &mut BTreeSet) -> R /// so it now propagates like any other store error. Callers with a looser /// contract (ids that may or may not resolve) should filter before calling, /// not rely on this function to do it silently. +/// +/// Production ingest calls [`verify_incoming`] and [`append_accepted`] +/// separately (see `transport::ingest_pack_file_impl`); this is the two in +/// one for tests. +#[cfg(test)] pub(crate) fn index_incoming( layout: &Layout, store: &mut scl_core::Store, ids: &[ObjectId], ) -> Result { - let mut count = 0; + let accepted = verify_incoming(store, ids)?; + append_entries(layout, store, &accepted)?; + Ok(accepted.len()) +} + +/// The verifying half of [`index_incoming`], with no lock and no index write, +/// so a caller can hold the index lock only around the file updates +/// ([`append_accepted`]) and not around one Ed25519 check per signature in a +/// pack a writer chose. +pub(crate) fn verify_incoming( + store: &mut scl_core::Store, + ids: &[ObjectId], +) -> Result> { + let mut accepted = Vec::new(); for id in ids { if let Object::Signature(sig) = store.get(id)? { // Only a signature that checks out for a target this store holds // is indexed (and so forwarded and kept by gc): nothing forged or - // dangling can crowd out a real one (#230, ADR-0038). + // dangling can take a slot (#230, ADR-0038). if acceptable(store, &sig)? { - append_index(layout, sig.snapshot, *id)?; - count += 1; + accepted.push((sig.snapshot, *id, sig.signer)); } } } - Ok(count) + Ok(accepted) +} + +/// Index what [`verify_incoming`] accepted. +pub(crate) fn append_accepted( + layout: &Layout, + store: &mut scl_core::Store, + accepted: &[(ObjectId, ObjectId, [u8; 32])], +) -> Result<()> { + append_entries(layout, store, accepted) } /// Whether `sig` may enter the index: its target is held, and the signature @@ -200,48 +397,65 @@ pub(crate) fn acceptable(store: &mut scl_core::Store, sig: &SignatureObj) -> Res /// bookkeeping the right ids, only on the objects actually landing on disk. /// O(store), which is fine for a one-shot clone of a store that size. pub(crate) fn reindex(layout: &Layout, store: &mut scl_core::Store) -> Result { - let mut entries: Vec<(ObjectId, ObjectId)> = Vec::new(); + let mut entries: Vec = Vec::new(); for id in store.all_ids()? { if let Object::Signature(sig) = store.get(&id)? { if acceptable(store, &sig)? { - entries.push((sig.snapshot, id)); + entries.push((sig.snapshot, id, sig.signer)); } } } - entries.sort(); - entries.dedup(); + // One slot per (target, signer): the lowest signature id keeps it, so the + // rebuild is deterministic. (Sorted by slot first: equal slots must be + // adjacent for `dedup_by_key`.) + entries.sort_by_key(|e| (e.0, e.2, e.1)); + entries.dedup_by_key(|e| (e.0, e.2)); let count = entries.len(); - write_index(layout, &entries)?; + with_index_lock(layout, || write_index(layout, &entries))?; Ok(count) } -/// The whole signature index as `signed target -> signature object ids`, read -/// once so a caller checking many targets does not rescan the file per target. -pub(crate) fn index_by_target( - layout: &Layout, -) -> Result>> { - let mut out: std::collections::HashMap> = Default::default(); - for (target, sig_id) in read_index(layout)? { - out.entry(target).or_default().push(sig_id); +/// What honoring a drop needs from the index (#257): which targets carry any +/// signature, and, for the signers it asks about, the signature object each +/// holds over a target. Entries by other signers are not retained, so a flood +/// of signatures from keys an attacker made costs a file scan, not memory or +/// verification work. +pub(crate) struct SigIndex { + signed: std::collections::HashSet, + by_signer: std::collections::HashMap<(ObjectId, [u8; 32]), ObjectId>, +} + +impl SigIndex { + /// Whether any signature (by any signer) is indexed over `target`. + pub(crate) fn is_signed(&self, target: &ObjectId) -> bool { + self.signed.contains(target) + } + + /// The signature object `signer` holds over `target`, if it asked about + /// that signer when loading. + pub(crate) fn by(&self, target: &ObjectId, signer: &[u8; 32]) -> Option { + self.by_signer.get(&(*target, *signer)).copied() } - Ok(out) } -/// [`index_by_target`] restricted to `targets`, keeping at most `per_target` -/// signature ids each (index order). Only the wanted entries are retained, so -/// a large index costs a scan, not memory. -pub(crate) fn index_for_targets( +/// Read the index once for `targets` (every target when `None`), keeping the +/// per-signer entries of `signers` only. +pub(crate) fn load_sig_index( layout: &Layout, - targets: &BTreeSet, - per_target: usize, -) -> Result>> { - let mut out: std::collections::HashMap> = Default::default(); - for (target, sig_id) in read_index(layout)? { - if targets.contains(&target) { - let v = out.entry(target).or_default(); - if v.len() < per_target { - v.push(sig_id); - } + targets: Option<&BTreeSet>, + signers: &std::collections::HashMap<[u8; 32], String>, +) -> Result { + let mut out = SigIndex { + signed: Default::default(), + by_signer: Default::default(), + }; + for (target, sig_id, signer) in read_index(layout)? { + if targets.is_some_and(|t| !t.contains(&target)) { + continue; + } + out.signed.insert(target); + if signers.contains_key(&signer) { + out.by_signer.insert((target, signer), sig_id); } } Ok(out) @@ -250,14 +464,15 @@ pub(crate) fn index_for_targets( /// Sender-side seam (Task 3): the indexed signature object ids covering any /// of `snapshots`, deduped and sorted — the set a push/export needs to /// carry alongside the snapshots themselves so the receiving repo's index -/// stays complete. +/// stays complete. The index holds one signature per (target, signer), so a +/// target is sent at most one per signer. pub(crate) fn indexed_signature_ids_for( layout: &Layout, snapshots: &[ObjectId], ) -> Result> { let wanted: BTreeSet = snapshots.iter().copied().collect(); let mut out: BTreeSet = BTreeSet::new(); - for (snap, sig_id) in read_index(layout)? { + for (snap, sig_id, _) in read_index(layout)? { if wanted.contains(&snap) { out.insert(sig_id); } @@ -343,12 +558,10 @@ impl Repo { signer, sig, }; - let id = { - let arc = self.store_arc(); - let i = arc.lock().unwrap().put(Object::Signature(sig_obj))?; - i - }; - append_index(self.layout(), snapshot, id)?; + let arc = self.store_arc(); + let mut store = arc.lock().unwrap(); + let id = store.put(Object::Signature(sig_obj))?; + append_index(self.layout(), &mut store, id)?; Ok(id) } @@ -358,7 +571,7 @@ impl Repo { let arc = self.store_arc(); let mut store = arc.lock().unwrap(); let mut out = Vec::new(); - for (snap, sig_id) in entries { + for (snap, sig_id, _) in entries { if snap != *snapshot { continue; } @@ -516,7 +729,12 @@ mod tests { // replicated signature) rather than going through `sign_snapshot`. { let raw = std::fs::read_to_string(repo.layout().signatures_path()).unwrap(); - let extra = format!("{} {}\n", snap.to_hex(), bad_id.to_hex()); + let extra = format!( + "{} {} {}\n", + snap.to_hex(), + bad_id.to_hex(), + hex::encode([7u8; 32]) + ); std::fs::write(repo.layout().signatures_path(), raw + &extra).unwrap(); } assert_eq!( @@ -807,4 +1025,198 @@ mod tests { std::fs::remove_dir_all(&b_root).unwrap(); assert!(!a_root.exists() && !b_root.exists()); } + + /// A signature object by a fresh key over `target`, stored but not indexed. + fn put_sig(repo: &Repo, target: ObjectId, forged: bool) -> (ObjectId, [u8; 32]) { + let (_s, identity) = scl_crypto::generate_identity_v2(); + let key = identity.signing.as_ref().unwrap(); + let signer = key.public().to_bytes(); + let sig = if forged { + [9u8; 64] + } else { + scl_crypto::sign_snapshot_id(key, target.as_bytes()) + }; + let id = repo + .store_arc() + .lock() + .unwrap() + .put(Object::Signature(SignatureObj { + snapshot: target, + signer, + sig, + })) + .unwrap(); + (id, signer) + } + + #[test] + fn a_signer_holds_one_slot_per_target() { + let root = tmp_root("one-slot"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("a.txt"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (first, signer) = put_sig(&repo, snap, false); + let second = ObjectId::of(b"another signature object by the same signer"); + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + append_entries(repo.layout(), &mut store, &[(snap, first, signer)]).unwrap(); + append_entries(repo.layout(), &mut store, &[(snap, second, signer)]).unwrap(); + append_entries(repo.layout(), &mut store, &[(snap, first, signer)]).unwrap(); + drop(store); + assert_eq!( + indexed_signature_ids_for(repo.layout(), &[snap]).unwrap(), + vec![first], + "the first entry keeps the slot" + ); + // Another signer over the same target, and the same signer over another target. + let (other, _) = put_sig(&repo, snap, false); + let mut store = arc.lock().unwrap(); + append_index(repo.layout(), &mut store, other).unwrap(); + append_entries(repo.layout(), &mut store, &[(first, second, signer)]).unwrap(); + drop(store); + assert_eq!( + std::fs::read_to_string(repo.layout().signatures_path()) + .unwrap() + .lines() + .count(), + 3 + ); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn a_legacy_index_is_imported_once_and_verified_on_open() { + let root = tmp_root("legacy"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("a.txt"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (good, signer) = put_sig(&repo, snap, false); + let (forged, _) = put_sig(&repo, snap, true); + let missing = ObjectId::of(b"no such signature object"); + let layout = repo.layout().clone(); + drop(repo); + // A pre-#257 index: no signer column, possibly with entries that + // would not be indexed today. + let legacy = format!( + "{s} {g}\n{s} {f}\n{s} {m}\nnot a line\n", + s = snap.to_hex(), + g = good.to_hex(), + f = forged.to_hex(), + m = missing.to_hex() + ); + std::fs::write(layout.signatures_legacy_path(), &legacy).unwrap(); + assert!(!layout.signatures_path().exists()); + + let repo = Repo::open(&root).unwrap(); + let v2 = std::fs::read_to_string(layout.signatures_path()).unwrap(); + assert_eq!( + v2, + format!( + "{} {} {}\n", + snap.to_hex(), + good.to_hex(), + hex::encode(signer) + ), + "only the entry that verifies survives, with its signer" + ); + assert_eq!( + std::fs::read_to_string(layout.signatures_legacy_path()).unwrap(), + legacy, + "the old file is left for an older binary" + ); + let mut trust = HashMap::new(); + trust.insert(signer, "alice".to_string()); + assert_eq!( + repo.sig_status(&snap, &trust).unwrap(), + SigStatus::Trusted("alice".to_string()) + ); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn a_writer_imports_the_legacy_index_before_starting_the_new_one() { + let root = tmp_root("legacy-writer"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("a.txt"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (old, _) = put_sig(&repo, snap, false); + std::fs::write( + repo.layout().signatures_legacy_path(), + format!("{} {}\n", snap.to_hex(), old.to_hex()), + ) + .unwrap(); + // Open already ran, so the v2 file does not exist yet: signing must + // not start it without the legacy entry. + let (_s, identity) = scl_crypto::generate_identity_v2(); + let new = repo.sign_snapshot(snap, &identity).unwrap(); + let mut ids = indexed_signature_ids_for(repo.layout(), &[snap]).unwrap(); + ids.sort(); + let mut want = vec![old, new]; + want.sort(); + assert_eq!(ids, want); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn open_imports_the_legacy_index_before_re_judging_pending_tombstones() { + // The re-evaluation reads the signature index; run against an empty + // not-yet-imported one it would record the trust as judged and the + // drop would stay inert (the sender never re-sends a held tombstone). + let root = tmp_root("legacy-pending"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + let arc = repo.store_arc(); + let tomb = arc + .lock() + .unwrap() + .put(Object::Transcript(scl_core::Transcript::drop_tombstone( + snap, &body, + ))) + .unwrap(); + repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) + .unwrap(); + let mut store = arc.lock().unwrap(); + crate::transcripts::index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + drop(store); + assert_eq!(crate::transcripts::tests::pending_count(&repo), 1); + // Turn the v2 index back into a legacy one, then trust the signer. + let layout = repo.layout().clone(); + let v2 = std::fs::read_to_string(layout.signatures_path()).unwrap(); + let legacy: String = v2 + .lines() + .map(|l| l.split(' ').take(2).collect::>().join(" ") + "\n") + .collect(); + std::fs::remove_file(layout.signatures_path()).unwrap(); + std::fs::write(layout.signatures_legacy_path(), legacy).unwrap(); + crate::transcripts::tests::trust_signer(&repo, &owner); + // While the import is pending (e.g. it failed), a trust pass must + // neither honor nor record the trust as judged. + { + let arc = repo.store_arc(); + let mut store = arc.lock().unwrap(); + crate::transcripts::reevaluate_if_trust_changed(&layout, &mut store).unwrap(); + } + assert!(crate::transcripts::trust_changed(&layout)); + drop(repo); + + let repo = Repo::open(&root).unwrap(); + assert!( + repo.transcripts_for(&snap).unwrap().is_empty(), + "the drop was honored on open" + ); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } } diff --git a/crates/repo/src/transcripts.rs b/crates/repo/src/transcripts.rs index 56a879fb..f4e6965e 100644 --- a/crates/repo/src/transcripts.rs +++ b/crates/repo/src/transcripts.rs @@ -31,6 +31,7 @@ use scl_core::{Object, ObjectId, Secret, SignatureObj, Transcript}; use crate::error::{Error, Result}; use crate::layout::Layout; +use crate::lock::with_index_lock; use crate::repo::Repo; use crate::signatures::SigStatus; @@ -113,6 +114,12 @@ fn dropped_ids(layout: &Layout) -> Result> { /// path holds no repo lock) cannot lose each other's drop line. A racing /// duplicate line is harmless: readers treat the file as a set. fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { + // Under the index lock only so `reindex`'s whole-file rewrite cannot drop + // a line appended meanwhile. + with_index_lock(layout, || append_drop_locked(layout, entry)) +} + +fn append_drop_locked(layout: &Layout, entry: DropEntry) -> Result<()> { use std::io::Write; if read_drops(layout)?.contains(&entry) { return Ok(()); @@ -137,7 +144,7 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { f.sync_data()?; if is_new { // Make the new directory entry durable too. - std::fs::File::open(&layout.dot_sc)?.sync_all()?; + scl_core::fsutil::fsync_dir(&layout.dot_sc)?; } Ok(()) } @@ -148,15 +155,6 @@ fn append_drop(layout: &Layout, entry: DropEntry) -> Result<()> { /// retained tombstone on each fetch). const MAX_PENDING: usize = 1024; -/// Signature objects `tombstone_honored` reads per tombstone, and how many of -/// those that claim a trusted key it verifies. A peer can attach any number of -/// junk signature objects to a target and this runs on every open, so the work -/// is capped; a real tombstone carries one or two. Signatures by an untrusted -/// key cost one read and never count against the verification cap, so they -/// cannot push a trusted one out of it. -const MAX_SIG_READS: usize = 256; -const MAX_SIG_VERIFIES: usize = 8; - /// Distinct signature targets `index_incoming` looks up per batch. const MAX_SIG_LOOKUPS: usize = 1024; @@ -198,26 +196,28 @@ fn note_pending(layout: &Layout, new: &[ObjectId], fp: &str) -> Result<()> { let mut once = BTreeSet::new(); let new: Vec = new.iter().copied().filter(|i| once.insert(*i)).collect(); let new = &new[new.len().saturating_sub(MAX_PENDING)..]; - let mut all = read_pending(layout)?; - let mut seen: BTreeSet = all.iter().copied().collect(); - let mut added = false; - for id in new { - if seen.insert(*id) { - all.push(*id); - added = true; + with_index_lock(layout, || { + let mut all = read_pending(layout)?; + let mut seen: BTreeSet = all.iter().copied().collect(); + let mut added = false; + for id in new { + if seen.insert(*id) { + all.push(*id); + added = true; + } } - } - if !added { - return Ok(()); - } - let excess = all.len().saturating_sub(MAX_PENDING); - all.drain(..excess); - write_pending(layout, &all)?; - // Entries are judged under the trust in force now; remember which. - if read_trust_state(layout).is_none() { - save_trust_state(layout, fp, 0)?; - } - Ok(()) + if !added { + return Ok(()); + } + let excess = all.len().saturating_sub(MAX_PENDING); + all.drain(..excess); + write_pending(layout, &all)?; + // Entries are judged under the trust in force now; remember which. + if read_trust_state(layout).is_none() { + save_trust_state(layout, fp, 0)?; + } + Ok(()) + }) } type Trusted = std::collections::HashMap<[u8; 32], String>; @@ -288,17 +288,24 @@ pub(crate) fn reevaluate_if_trust_changed( ) -> Result { let trusted = trusted_signers(layout); let fp = fp_of(&trusted); - if !needs_judging(layout, &fp) { + if !needs_judging(layout, &fp) || crate::signatures::migration_pending(layout) { return Ok(0); } - let (honored, incomplete) = reevaluate_inner(layout, store, &trusted)?; - let failed = match (incomplete, read_trust_state(layout)) { - (false, _) => 0, - (true, Some((saved, n))) if saved == fp => n + 1, - (true, _) => 1, - }; - save_trust_state(layout, &fp, failed)?; - Ok(honored) + // The pass rewrites the pending list, so it runs under the index lock. + // Judged again inside it: a racing pass may have just finished the work. + with_index_lock(layout, || { + if !needs_judging(layout, &fp) { + return Ok(0); + } + let (honored, incomplete) = reevaluate_inner(layout, store, &trusted)?; + let failed = match (incomplete, read_trust_state(layout)) { + (false, _) => 0, + (true, Some((saved, n))) if saved == fp => n + 1, + (true, _) => 1, + }; + save_trust_state(layout, &fp, failed)?; + Ok(honored) + }) } /// Rewrite the whole index file atomically (temp file + rename), so a @@ -319,18 +326,20 @@ pub(crate) fn append_index( snapshot: ObjectId, transcript_id: ObjectId, ) -> Result<()> { - if dropped_ids(layout)?.contains(&transcript_id) { - return Ok(()); - } - let mut entries = read_index(layout)?; - if entries - .iter() - .any(|(s, t)| *s == snapshot && *t == transcript_id) - { - return Ok(()); - } - entries.push((snapshot, transcript_id)); - write_index(layout, &entries) + with_index_lock(layout, || { + if dropped_ids(layout)?.contains(&transcript_id) { + return Ok(()); + } + let mut entries = read_index(layout)?; + if entries + .iter() + .any(|(s, t)| *s == snapshot && *t == transcript_id) + { + return Ok(()); + } + entries.push((snapshot, transcript_id)); + write_index(layout, &entries) + }) } /// `[signing]` keys listed under `[signers] trusted` in `.sc/recipients.toml`: @@ -384,8 +393,8 @@ pub(crate) fn drop_digest(tomb: &ObjectId) -> [u8; 32] { /// `trusted`. An unsigned or untrusted tombstone stays an inert object. fn tombstone_honored( store: &mut scl_core::Store, - trusted: &std::collections::HashMap<[u8; 32], String>, - sigs: &std::collections::HashMap>, + trusted: &Trusted, + sigs: &crate::signatures::SigIndex, tomb: ObjectId, t: &Transcript, dropped: ObjectId, @@ -400,25 +409,19 @@ fn tombstone_honored( _ => return Ok(false), } } - let mut verifies = 0; - for sid in sigs - .get(&tomb) - .map(Vec::as_slice) - .unwrap_or_default() - .iter() - .take(MAX_SIG_READS) - { - if let Object::Signature(sig) = store.get(sid)? { - if !trusted.contains_key(&sig.signer) { - continue; - } - if scl_crypto::verify_transcript_sig(&sig.signer, &drop_digest(&tomb), &sig.sig) { + // The index holds one signature per (target, signer), so only the trusted + // signers are looked up: at most one read and one verification each, and + // signatures from keys anyone else made are never reached (#257). + for signer in trusted.keys() { + let Some(sid) = sigs.by(&tomb, signer) else { + continue; + }; + if let Object::Signature(sig) = store.get(&sid)? { + if sig.signer == *signer + && scl_crypto::verify_transcript_sig(signer, &drop_digest(&tomb), &sig.sig) + { return Ok(true); } - verifies += 1; - if verifies >= MAX_SIG_VERIFIES { - break; - } } } Ok(false) @@ -444,6 +447,10 @@ pub fn load(layout: &Layout) -> Result> { /// Returns the number of dropped entries (0 when nothing needed pruning, in /// which case the index file is left untouched rather than rewritten). pub(crate) fn gc_prune(layout: &Layout, reachable: &mut BTreeSet) -> Result { + with_index_lock(layout, || gc_prune_locked(layout, reachable)) +} + +fn gc_prune_locked(layout: &Layout, reachable: &mut BTreeSet) -> Result { // Tombstones are load-bearing against resurrection: root them (and keep // their lines) even when their snapshot is dead. The dropped transcript // itself is never rooted, so it is swept after the grace window. @@ -489,7 +496,7 @@ pub(crate) fn index_incoming( ) -> Result { let mut count = 0; let trusted = trusted_signers(layout); - let sigs = crate::signatures::index_by_target(layout)?; + let sigs = crate::signatures::load_sig_index(layout, None, &trusted)?; let mut held = Vec::new(); // Tombstones already recorded as drops need nothing: every fetch re-sends // them all, so skipping them keeps a resend O(new) rather than O(drops^2). @@ -508,7 +515,7 @@ pub(crate) fn index_incoming( if tombstone_honored(store, &trusted, &sigs, id, t, d)? { append_drop(layout, (t.snapshot, id, d))?; recorded.insert(id); - } else if sigs.contains_key(&id) && store.contains(&t.snapshot) { + } else if sigs.is_signed(&id) && store.contains(&t.snapshot) { held.push(id); } Ok(()) @@ -578,7 +585,7 @@ fn reevaluate_inner( } let mut incomplete = false; let wanted: BTreeSet = pending.iter().copied().collect(); - let sigs = crate::signatures::index_for_targets(layout, &wanted, MAX_SIG_READS)?; + let sigs = crate::signatures::load_sig_index(layout, Some(&wanted), trusted)?; let (mut keep, mut honored) = (Vec::new(), 0); for id in &pending { if !store.contains(id) { @@ -627,7 +634,7 @@ pub(crate) fn reindex(layout: &Layout, store: &mut scl_core::Store) -> Result = Vec::new(); let mut drops: Vec = Vec::new(); let trusted = trusted_signers(layout); - let sigs = crate::signatures::index_by_target(layout)?; + let sigs = crate::signatures::load_sig_index(layout, None, &trusted)?; for id in store.all_ids()? { if let Object::Transcript(t) = store.get(&id)? { if !store.contains(&t.snapshot) { @@ -650,8 +657,10 @@ pub(crate) fn reindex(layout: &Layout, store: &mut scl_core::Store) -> Result = drops.iter().map(|(_, tomb, _)| *tomb).collect(); + let sigs = crate::signatures::load_sig_index(layout, Some(&tombs), &Default::default())?; out.extend( drops .into_iter() .filter(|(snap, tomb, _)| { - sigs.contains_key(tomb) && (wanted.contains(snap) || !snapshot_held(snap)) + sigs.is_signed(tomb) && (wanted.contains(snap) || !snapshot_held(snap)) }) .map(|(_, tomb, _)| tomb), ); @@ -809,12 +819,10 @@ impl Repo { signer: key.public().to_bytes(), sig: scl_crypto::sign_transcript_id(key, &drop_digest(&tomb)), }; - let id = { - let arc = self.store_arc(); - let i = arc.lock().unwrap().put(Object::Signature(sig_obj))?; - i - }; - crate::signatures::append_index(self.layout(), tomb, id) + let arc = self.store_arc(); + let mut store = arc.lock().unwrap(); + let id = store.put(Object::Signature(sig_obj))?; + crate::signatures::append_index(self.layout(), &mut store, id) } /// All indexed transcripts for `snapshot`, loaded from the CAS (id + @@ -899,12 +907,10 @@ impl Repo { signer, sig, }; - let id = { - let arc = self.store_arc(); - let i = arc.lock().unwrap().put(Object::Signature(sig_obj))?; - i - }; - crate::signatures::append_index(self.layout(), tid, id)?; + let arc = self.store_arc(); + let mut store = arc.lock().unwrap(); + let id = store.put(Object::Signature(sig_obj))?; + crate::signatures::append_index(self.layout(), &mut store, id)?; Ok(id) } @@ -1154,7 +1160,9 @@ pub(crate) mod tests { assert_eq!(repo.transcripts_for(&snap).unwrap().len(), 1); // A plain transcript signature over the bare tombstone id (what an - // older binary's bulk `sc transcript sign` would produce) is not a drop. + // older binary's bulk `sc transcript sign` would produce) is not a + // drop, and is not indexed either, so it cannot take the owner's slot + // from the real drop signature below. let plain = { let id = arc .lock() @@ -1163,7 +1171,7 @@ pub(crate) mod tests { snap, &victim, ))) .unwrap(); - repo.sign_transcript(id, &owner).unwrap(); + assert!(repo.sign_transcript(id, &owner).is_err()); let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[id]).unwrap(); id @@ -1451,7 +1459,7 @@ pub(crate) mod tests { // The signature arrives in a later transfer, without the tombstone. repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) .unwrap(); - let sig = crate::signatures::index_by_target(repo.layout()).unwrap()[&tomb][0]; + let sig = crate::signatures::indexed_signature_ids_for(repo.layout(), &[tomb]).unwrap()[0]; let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[sig]).unwrap(); assert_eq!(read_pending(repo.layout()).unwrap(), vec![tomb]); @@ -1495,7 +1503,7 @@ pub(crate) mod tests { // all with ids below the real signature's. let mut i = 0u64; let mut junk = 0; - while junk < MAX_SIG_VERIFIES + 12 { + while junk < 20 { i += 1; let mut sig = [0u8; 64]; sig[..8].copy_from_slice(&i.to_le_bytes()); @@ -1529,8 +1537,7 @@ pub(crate) mod tests { crate::transport::ingest_pack_file(repo.layout(), &mut store, &path).unwrap(); drop(store); - let sigs = crate::signatures::index_by_target(repo.layout()).unwrap(); - let indexed = &sigs[&tomb]; + let indexed = crate::signatures::indexed_signature_ids_for(repo.layout(), &[tomb]).unwrap(); assert!(indexed.contains(&real.id())); assert!( junk_ids.iter().all(|j| !indexed.contains(j)), @@ -1568,9 +1575,11 @@ pub(crate) mod tests { 0 ); drop(store); - assert!(crate::signatures::index_by_target(repo.layout()) - .unwrap() - .is_empty()); + assert!( + crate::signatures::indexed_signature_ids_for(repo.layout(), &[ghost]) + .unwrap() + .is_empty() + ); drop(repo); std::fs::remove_dir_all(&root).unwrap(); @@ -1595,7 +1604,7 @@ pub(crate) mod tests { .unwrap(); repo.sign_drop(tomb, owner.signing.as_ref().unwrap()) .unwrap(); - let sig = crate::signatures::index_by_target(repo.layout()).unwrap()[&tomb][0]; + let sig = crate::signatures::indexed_signature_ids_for(repo.layout(), &[tomb]).unwrap()[0]; let mut store = arc.lock().unwrap(); index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); drop(store); @@ -1867,4 +1876,199 @@ pub(crate) mod tests { std::fs::remove_dir_all(&root).unwrap(); assert!(!root.exists()); } + + #[test] + fn valid_signatures_from_hundreds_of_other_keys_cannot_keep_a_drop_inert() { + // #257: a writer who signs the predictable tombstone id with many keys + // of their own, ahead of the real signature, used to fill the + // per-tombstone read cap. Honoring now looks only at trusted signers. + let root = tmp_root("sig-flood"); + let repo = Repo::init(&root).unwrap(); + std::fs::write(root.join("f"), b"x").unwrap(); + let snap = repo.commit("t", "c1").unwrap(); + let (_s, owner) = scl_crypto::generate_identity_v2(); + let key = owner.signing.as_ref().unwrap(); + let body = repo + .attach_transcript(snap, "a", "s", b"body", &[owner.enc.public()]) + .unwrap(); + trust_signer(&repo, &owner); + let arc = repo.store_arc(); + let tomb = arc + .lock() + .unwrap() + .put(Object::Transcript(Transcript::drop_tombstone(snap, &body))) + .unwrap(); + let junk: Vec = (0..300) + .map(|_| { + let (_s, who) = scl_crypto::generate_identity_v2(); + let k = who.signing.as_ref().unwrap(); + arc.lock() + .unwrap() + .put(Object::Signature(SignatureObj { + snapshot: tomb, + signer: k.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(k, &drop_digest(&tomb)), + })) + .unwrap() + }) + .collect(); + let real = arc + .lock() + .unwrap() + .put(Object::Signature(SignatureObj { + snapshot: tomb, + signer: key.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(key, &drop_digest(&tomb)), + })) + .unwrap(); + let mut store = arc.lock().unwrap(); + assert_eq!( + crate::signatures::index_incoming(repo.layout(), &mut store, &junk).unwrap(), + 300, + "every valid junk signature is indexed, one slot per key" + ); + crate::signatures::index_incoming(repo.layout(), &mut store, &[real]).unwrap(); + index_incoming(repo.layout(), &mut store, &[tomb]).unwrap(); + drop(store); + assert_eq!(read_drops(repo.layout()).unwrap().len(), 1, "drop honored"); + assert!(load(repo.layout()).unwrap().is_empty()); + + // A resend of the junk adds nothing: the slots are taken. + let mut store = arc.lock().unwrap(); + crate::signatures::index_incoming(repo.layout(), &mut store, &junk).unwrap(); + drop(store); + assert_eq!( + crate::signatures::indexed_signature_ids_for(repo.layout(), &[tomb]) + .unwrap() + .len(), + 301 + ); + + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// `n` threads each push a pack (a snapshot signature, a transcript, a + /// tombstone for it and the tombstone's untrusted signature) at one served + /// directory; every derived index entry must survive (#260). With + /// `repo_open` a `Repo` holds the repo lock throughout, like a local `sc` + /// command; otherwise each thread also moves a ref, taking the repo lock. + fn concurrent_ingests_lose_nothing(tag: &str, repo_open: bool) { + let n = 8usize; + let src_root = tmp_root(&format!("{tag}-src")); + let dst_root = tmp_root(&format!("{tag}-dst")); + let src = Repo::init(&src_root).unwrap(); + std::fs::write(src_root.join("f"), b"x").unwrap(); + let snap = src.commit("t", "c1").unwrap(); + let src_arc = src.store_arc(); + let mut packs = Vec::new(); + for i in 0..n { + let (_s, who) = scl_crypto::generate_identity_v2(); + let k = who.signing.as_ref().unwrap(); + let tid = src + .attach_transcript(snap, "a", &format!("s{i}"), b"body", &[who.enc.public()]) + .unwrap(); + let tomb = Object::Transcript(Transcript::drop_tombstone(snap, &tid)); + let tomb_id = tomb.id(); + let objs = [ + Object::Signature(SignatureObj { + snapshot: snap, + signer: k.public().to_bytes(), + sig: scl_crypto::sign_snapshot_id(k, snap.as_bytes()), + }), + src_arc.lock().unwrap().get(&tid).unwrap(), + tomb, + Object::Signature(SignatureObj { + snapshot: tomb_id, + signer: k.public().to_bytes(), + sig: scl_crypto::sign_transcript_id(k, &drop_digest(&tomb_id)), + }), + ]; + let enc: Vec<(ObjectId, Vec)> = objs.iter().map(|o| (o.id(), o.encode())).collect(); + let path = dst_root.join(format!("in-{i}.pack")); + // dst is created below; the packs live beside it. + packs.push((path, enc, tomb_id)); + } + let dst = Repo::init(&dst_root).unwrap(); + dst.store_arc() + .lock() + .unwrap() + .put(src_arc.lock().unwrap().get(&snap).unwrap()) + .unwrap(); + for (path, enc, _) in &packs { + std::fs::write(path, scl_core::pack::build_pack(enc).unwrap().0).unwrap(); + } + let layout = dst.layout().clone(); + let held = if repo_open { + Some(dst) + } else { + drop(dst); + None + }; + + let handles: Vec<_> = packs + .iter() + .enumerate() + .map(|(i, (path, _, _))| { + let (root, path) = (dst_root.clone(), path.clone()); + std::thread::spawn(move || { + let t = crate::transport::LocalTransport::open(&root).unwrap(); + t.ingest_from(&path).unwrap(); + if !repo_open { + use crate::transport::Transport; + // The repo lock is try-once: retry while another + // connection holds it. Finishing at all shows the + // index lock never waits on it. + let started = std::time::Instant::now(); + while let Err(e) = t.update_ref(&format!("b{i}"), &snap, None) { + assert!(matches!(e, Error::Locked(_)), "{e:?}"); + assert!(started.elapsed().as_secs() < 30, "update_ref starved"); + std::thread::sleep(std::time::Duration::from_millis(1)); + } + } + }) + }) + .collect(); + for h in handles { + h.join().unwrap(); + } + + let tombs: Vec = packs.iter().map(|(_, _, t)| *t).collect(); + let mut targets = tombs.clone(); + targets.push(snap); + assert_eq!( + crate::signatures::indexed_signature_ids_for(&layout, &targets) + .unwrap() + .len(), + 2 * n, + "a signature index entry was lost" + ); + assert_eq!( + load(&layout).unwrap().len(), + n, + "a transcript entry was lost" + ); + let mut pending = read_pending(&layout).unwrap(); + pending.sort(); + let mut want = tombs; + want.sort(); + assert_eq!(pending, want, "a pending entry was lost"); + + drop(held); + drop(src); + std::fs::remove_dir_all(&src_root).unwrap(); + std::fs::remove_dir_all(&dst_root).unwrap(); + assert!(!src_root.exists() && !dst_root.exists()); + } + + #[test] + fn concurrent_serve_ingests_lose_no_index_entry_beside_a_local_command() { + concurrent_ingests_lose_nothing("conc-open", true); + } + + #[test] + fn concurrent_serve_ingests_lose_no_index_entry_and_do_not_deadlock_with_update_ref() { + concurrent_ingests_lose_nothing("conc-ref", false); + } } diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 027f5b5b..a669c800 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -167,6 +167,7 @@ impl LocalTransport { // ran it already. // Only when the trusted-signer set changed since they were last // judged: serving must not take the exclusive lock per connection. + let _ = crate::signatures::migrate_legacy(&layout, &mut store); if crate::transcripts::trust_changed(&layout) { if let Ok(_lock) = RepoLock::acquire(&layout) { let _ = crate::transcripts::reevaluate_if_trust_changed(&layout, &mut store); @@ -630,12 +631,18 @@ fn ingest_pack_file_impl( if !skip_transcripts { let _ = crate::transcripts::reevaluate_if_trust_changed(layout, store); } - // Receiver seam (P22 Task 3): every id above was just written to this - // store, so `index_incoming`'s "ids just written" contract holds. - crate::signatures::index_incoming(layout, store, &ids)?; - // Receiver seam (P30 Task 4): same "ids just written" contract, for - // Transcript objects. - crate::transcripts::index_incoming(layout, store, &ids)?; + // Receiver seams (P22 Task 3 signatures, P30 Task 4 transcripts): every id + // above was just written to this store, so `index_incoming`'s "ids just + // written" contract holds. Both passes run under one index lock: gc prunes + // the transcript and signature indexes together, and between two separate + // locks it would drop a signature whose transcript was not yet indexed. + // The Ed25519 checks run before the lock is taken; it covers the file + // updates and the (bounded) tombstone judging only. + let accepted = crate::signatures::verify_incoming(store, &ids)?; + crate::lock::with_index_lock(layout, || { + crate::signatures::append_accepted(layout, store, &accepted)?; + crate::transcripts::index_incoming(layout, store, &ids) + })?; Ok(ids) } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 47a54336..7f97b8ac 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -218,10 +218,13 @@ too, not just the src-control-side metadata. sealed). Only signatures that verify for a held target are indexed, so forged signatures stay in the object store (until gc sweeps them) but are never indexed, so a `sc serve` or local sender never forwards them and gc - never keeps them (a bucket forwards what it holds; receivers filter). A writer to a shared remote - who signs with keys of their own can still crowd more than 256 valid - signatures ahead of the real one, which keeps the drop inert on every peer - that ingests them (a denial of the drop, never an honored one; #257). Writers + never keeps them (a bucket forwards what it holds; receivers filter). The signature + index keeps one signature per (target, signer) and honoring looks up only the + trusted signers, so signatures valid under keys a writer made themselves + cannot crowd out the real one (resolved, #257); they cost a line and a + check each. Index updates on the serve path take a short `.sc/index.lock`, so + concurrent pushes and local commands do not lose index or pending entries + (resolved, #260). Writers can also evict pending entries from a push-only remote (trust the signer before the push). Trust is checked at evaluation and is not retroactive. A tombstone a remote already held before upgrading to a build with the pending index is not remembered: trust the diff --git a/docs/adr/0038-agent-session-transcripts.md b/docs/adr/0038-agent-session-transcripts.md index ad6469ef..10985131 100644 --- a/docs/adr/0038-agent-session-transcripts.md +++ b/docs/adr/0038-agent-session-transcripts.md @@ -467,16 +467,13 @@ signature that fails to verify no longer turns a snapshot `Invalid`; it is simply absent (`Unsigned`, which `--require` also rejects). A signature indexed before this change is read as before. -**Residual limit (#257).** A signature that verifies under a key the attacker -controls is still indexed. A writer can add more than `MAX_SIG_READS` (256) of -them over a predictable tombstone id, ahead of the real signature, and -`tombstone_honored` stops reading at 256 per tombstone, so the drop stays -inert on every peer that ingests them. This is a denial of the drop on shared -remotes (not on a single peer), never an honored one. The fix is an index -keyed by (target, signer) so honoring is a lookup over the trusted signers; -it is tracked in #257. The caps (`MAX_SIG_READS` 256 reads and -`MAX_SIG_VERIFIES` 8 failed checks per tombstone) bound the work meanwhile; -the verification cap is unreachable for signatures indexed by this version. +**Former residual limit (#257, resolved).** A signature that verified under a +key the attacker controlled was still indexed, so a writer could add more +than 256 of them over a predictable tombstone id, ahead of the real one, and +the capped read in `tombstone_honored` never reached the trusted signer's +signature (a denial of the drop on every peer that ingested them, never an +honored one). Resolved by keying the index per (target, signer); see the #257 +and #260 addendum below. **Pending index.** A push-only or served remote holds the tombstone from an earlier push, so `push` (has-object) never re-sends it and `index_incoming` @@ -522,11 +519,9 @@ never re-runs. pending entry. That costs a push-only remote the retry for that tombstone (trust the signer before the push, or have a fetching peer re-send it), but never a recorded drop and never the rooting of one: gc roots every drop - line. The same unlocked read-modify-write applies to `.sc/signatures`, - `.sc/transcripts` and the pending rewrite on the serve path; taking the - repo lock there changes `Locked` behaviour for concurrent connections and - for a local `sc` command that holds the lock for its lifetime, so it is - tracked separately as #260 rather than decided here. `O_APPEND` atomicity is + line. (The pending file and the other indexes were unlocked + read-modify-writes when this addendum was written; the index lock below, + #260, serializes them.) `O_APPEND` atomicity is a local-filesystem guarantee (none on NFS). `append_drop` starts on a new line when the file ends in a torn line. - **Bounded.** The file is a FIFO ring of 1024 ids. A writer who pushes many @@ -562,7 +557,7 @@ never re-runs. legitimate signatures from a large fetch (a long signed history), which is a worse failure than the cost. -- Re-evaluation scans `.sc/signatures` once per trigger (time proportional to +- Re-evaluation scans `.sc/signatures.v2` once per trigger (time proportional to the file, memory proportional to the pending targets). - A bucket serves every signed tombstone-shaped object it holds, and its contents are whatever writers uploaded; receivers still judge each one. @@ -578,3 +573,137 @@ never re-runs. drop alone does not push (unchanged). Clone `reindex` rebuilds drops from the store (verifying signatures as above) and does not touch the pending file. + +## Addendum: signature index per signer, and the index lock (issues #257, #260) + +Two hardening follow-ups to the #230 delivery work, with no `PROTOCOL_VERSION` +bump, no new object kind and no change to the trust gate. + +### 1. The signature index holds one slot per (target, signer) (#257) + +`.sc/signatures` kept every verified signature over a target. A signature +made by a key the attacker controls verifies, so a writer could index 256 or +more of them over a tombstone id ahead of the trusted signer's, and the capped +scan in `tombstone_honored` never reached it. Now: + +- **Format.** `.sc/signatures.v2`, one ` ` line + per signature (signer = 32-byte Ed25519 key, hex). An append is a no-op when + the `(target, signer)` slot is taken: the first entry keeps it. This loses + nothing: Ed25519 is deterministic, so a signer has one valid signature per + target, and for any target kind only one signing domain is acceptable (see + `signatures::acceptable`), so a signature in the wrong domain is never + indexed and cannot occupy a slot. The local signing paths hold the same + bar: `append_index` refuses a signature that is not acceptable, so a plain + transcript signature over a drop tombstone cannot take the owner's slot + ahead of the real drop signature. Appends write only the new lines + (`O_APPEND`, under the lock) and never rewrite the file, so a large index + costs a read, not a rewrite and fsync, while the lock is held. Entries are bounded by the distinct + signers, not by the number of signature objects. +- **Honoring.** `tombstone_honored` iterates the trusted signers, looks up the + one signature each holds over the tombstone (`signatures::SigIndex`), reads + that object and verifies the drop digest. At most one read and one + verification per trusted signer; signatures by other keys are never read. + `MAX_SIG_READS` and `MAX_SIG_VERIFIES` are gone. The index loader keeps + per-signer entries only for trusted signers, so a flood costs a file scan, + not memory. +- **Why a new file name.** An older binary parses exactly two columns and + skips any other line, then rewrites the file on its next append from the + lines it understood, so appending a third column to `.sc/signatures` would + make an older binary drop every new entry. The new binary leaves the old + file alone and uses `.sc/signatures.v2`. +- **Migration.** `signatures::migrate_legacy` imports `.sc/signatures` into + the v2 file once (when v2 is absent): it reads each signature object for + its signer and runs `acceptable` again, because an index written before #230 + may hold forgeries, and one naming a trusted key would take that signer's + slot. Entries that do not verify, or whose object is gone, are dropped, so a + snapshot that was `Invalid` only through such an entry reads `Unsigned` + (the #230 behavior). It runs at `Repo::open` and `LocalTransport::open` + (best effort), and is mandatory before any v2 write (`append_entries`) and + before gc prunes, which fail rather than start a v2 file, or root nothing, + while legacy entries are unread. The old file is not deleted or rewritten. + `reindex` rebuilds v2 from the store and ignores the legacy file. +- **Downgrade.** Do not run an older binary's `sc gc` on a repo this build + has signed in or ingested into. An older gc roots only the legacy + `.sc/signatures`, so it sweeps the signature objects that only + `.sc/signatures.v2` lists, and the v2 entries then dangle. Otherwise an older + binary keeps using `.sc/signatures`: signatures it adds after the upgrade are + in the object store but not in v2 (`sc clone` rebuilds the index; there is no + standalone reindex command), and signatures made by this build are not + visible to it. +- **Failed migration.** `Repo::open` and `LocalTransport::open` ignore a + failed import (for example a read-only `.sc/`), and until it succeeds the + index reads as empty: signatures read `Unsigned` and a served repo forwards + none. The trust re-evaluation does not run while an import is pending, so it + cannot record the trust as judged against an empty index. Every writer and + gc retry the import and fail rather than proceed. + An object that is gone or will not decode is dropped from the import like + any entry that does not verify, so one bad object does not block it; only a + transient I/O error fails it. +- **Forwarding.** `indexed_signature_ids_for` (sender seam for fetch, clone, + serve and push) returns the indexed signatures, which are now one per + (target, signer), so a sender forwards at most one signature per signer per + target. `sc verify` / `sig_status` read all entries of a snapshot, as + before: the index is no longer growable past one per key. +- **Limit.** Distinct attacker keys still get one slot each. Their cost is one + line, one object and one Ed25519 check per key at ingest (as in #230); they + can no longer stop a drop from being honored, because honoring never reads + them. + +### 2. The index lock (#260) + +Serve-path ingest (`LocalTransport::put_pack` and the wire/HTTP `ingest_from`) +updated the derived indexes without the repo lock, so two connections, or a +connection racing a local `sc` command, could lose a signature, transcript or +pending entry to a read-modify-write race. + +Decision: **a dedicated short-lived index lock, `.sc/index.lock`** +(`lock::with_index_lock`), distinct from `RepoLock`. Alternatives considered: + +- *`RepoLock` around the post-ingest updates.* `RepoLock` is try-once and a + local `sc` command holds it for its whole life, so a push would fail + `Locked` for the duration of any open command, and two concurrent pushes + would fail each other. Rejected: it changes `Locked` behavior for serving. +- *Append-only indexes with `O_APPEND` lines.* Appends do not lose each other, + but gc prune, `reindex` and the pending ring (removal, FIFO eviction) are + rewrites that still race an append, so a lock is needed anyway. Rejected as + not smaller. + +Design: + +- An advisory `flock` (`File::try_lock`) on a file that is never removed. The + OS releases it when the holder dies, so there is no stale-lock breaking. + Threads of one process conflict too (separate open files). Reentrant per + thread, so a nested critical section reuses the held lock. +- **Bounded wait**: polls up to 30 seconds, then `Error::IndexBusy` (a distinct + error: its message must not tell anyone to delete the lock file, which would + let two holders lock different inodes). The sections it guards are single file updates, not command + lifetimes. +- **Lock order**: the index lock is a leaf. Code may take it while holding + `RepoLock` (a local command, `LocalTransport::open`'s re-evaluation) and + never takes `RepoLock` or waits on anything else while holding it. + `update_ref` takes `RepoLock` after the ingest has returned, so the two + cannot deadlock; tests drive `put_pack`-path ingests and `update_ref` from + threads together. +- **Atomic groups**: an ingest takes the lock once around its signature and + transcript index updates (its Ed25519 checks run before the lock, so a large + pack does not lengthen the hold), and gc once around its transcript and signature prunes + (the signature prune relies on the transcript prune having rooted the + transcript ids), so neither can interleave with the other's half. +- **Covered**: every read-modify-write or rewrite of `.sc/signatures.v2`, + `.sc/transcripts`, `.sc/transcript-drops` (the `O_APPEND` line and + `reindex`'s rewrite), `.sc/transcript-pending` and its trust state: the + appends, `gc` prune, `reindex`, `note_pending` and the whole + trust-change re-evaluation pass. Readers take no lock (writes are atomic + renames or single appended lines). +- **Unchanged**: `RepoLock` and its `Locked` behavior for local commands. A + local command never fails because an ingest holds the index lock, nor the + reverse. +- A failure to get the lock (a holder busy for more than 30 seconds) fails + that ingest after its objects are stored. The client sees an error, but a + retried push sends only objects the remote lacks, so the objects of that + ingest stay unindexed on that remote until a reindex (`sc clone` of it). This + is the old loss window, now reachable only after the wait; the longest + holders are the one-time legacy import and the trust-change re-evaluation + (at most 1024 tombstones). +- `flock` is advisory and only as good as the filesystem's support for it + (like the `O_APPEND` note above, not guaranteed on NFS).