diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 64e18d0e..57f63c6f 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -290,6 +290,20 @@ cargo run --bin sc -- serve --http [--read-only] [--allow-public] # cap, both --http and --stdio, default 16 GiB, # 0=unlimited, floor = the max object size; # counted mid-stream abort -> EC_TOO_LARGE (P31) + # --max-conn-bytes : per-connection + # write budget, both --http and --stdio: every + # PutObject's bytes + per PutPack the larger of + # its spool bytes and the decoded bytes held + # (structural records; all records on --store); + # over it -> typed "connection write budget + # exceeded" error, then the connection closes. + # Default = --max-pack-size (16 GiB if 0) + the + # max object size; must be > 0, >= the max object + # size and >= a nonzero --max-pack-size, else + # startup aborts. Reads unbudgeted. The default is + # ~16.25 GiB and does NOT bound RAM: lower it with + # --max-pack-size; --http warns when a writable + # non-loopback listener uses the default (#233) # --max-object-size : largest # single object accepted from a peer, default # 256MiB, range 2MiB..1GiB, out of range aborts diff --git a/CLAUDE.md b/CLAUDE.md index c83ffbf2..4b994096 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -169,6 +169,14 @@ transport-adjacent. The rest, imperatively: transfer path only — a larger blob commits locally but fails every subsequent push/fetch/clone at the receiver. There is no negotiation: each receiver applies its own cap, so it must admit the objects it receives. Never inline a size in a new length guard; read the accessor. +- **Writes are budgeted per connection (#233).** `sc serve --max-conn-bytes` + (default `--max-pack-size` plus the max object size; must be > 0 and at least + both) counts `PutObject` bytes (structural objects included, `blob_size` 0 or + not) and, per `PutPack`, the larger of its spool bytes and the decoded bytes + held (structural records; every record on a `--store` bucket); the default + (~16.25 GiB) does not bound RAM, lower it with `--max-pack-size`; exceeding it replies `ConnectionBudgetExceeded` (`EC_BUDGET`, no + version bump) and closes the connection. Reads are not budgeted. A new + object-write verb must charge `ConnBudget` in `serve_session`. - **The P35 desktop app is a keyless, read-only top-level adapter.** Its renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or diff --git a/ROADMAP.md b/ROADMAP.md index f486f7f2..3a1d2378 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -804,6 +804,12 @@ scale-&-reach horizon): `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap (default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire negotiation: each receiver applies its own cap. +- **Per-connection write budget — shipped (#233, ADR-0039 second addendum).** + `sc serve --max-conn-bytes` bounds the sum of `PutObject` bytes and `PutPack` + bytes (per pack, the larger of spool bytes and decoded bytes held) one connection can + make the server retain; over it the server replies + `ConnectionBudgetExceeded` and closes. No `PROTOCOL_VERSION` bump. Not built: + a request-count or CPU budget, and a per-peer (cross-connection) budget. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the minimum configurable object cap minus one (2 MiB - 1, so every chunk frame fits any valid receiver) (a chunk frame carries a tag byte too; an oversized value used to make every transfer fail with a confusing receiver diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 58cbc2e0..d470de1b 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -367,6 +367,17 @@ enum Cmd { /// size (default 256 MiB) when non-zero. (P31) #[arg(long)] max_pack_size: Option, + /// Per-connection budget over everything a peer makes the server + /// write (every PutObject, structural objects included, plus per + /// PutPack the larger of its spool bytes and the decoded bytes held): + /// bytes, or a number with a KiB/MiB/GiB suffix (both --http and + /// --stdio). Exceeding it replies an error and closes the + /// connection; reads are not budgeted. Default: --max-pack-size (16 + /// GiB if unlimited) plus the max object size. Must be greater than + /// zero, at least the max object size, and at least --max-pack-size + /// (a nonzero one). (#233) + #[arg(long, value_name = "SIZE")] + max_conn_bytes: Option, /// Largest single object accepted from a peer: bytes, or a number /// with a KiB/MiB/GiB suffix. Default 256MiB, range 2MiB to 1GiB; /// out-of-range values are refused at startup, never clamped. Clients @@ -1099,6 +1110,7 @@ fn main() -> Result<()> { max_connections, timeout, max_pack_size, + max_conn_bytes, max_object_size: _, tls, tls_cert, @@ -1129,6 +1141,7 @@ fn main() -> Result<()> { max_connections, timeout, max_pack_size, + max_conn_bytes, tls, tls_cert, tls_key, @@ -3801,6 +3814,13 @@ fn run_clone_git(url: &str, dst: &std::path::Path) -> Result<()> { Ok(()) } +/// Does the listener run on the default connection budget with no explicit +/// bound on either knob? A nonzero `--max-pack-size` already lowers the default +/// budget with it; `0` (unlimited) still falls back to the 16 GiB default. +fn runs_default_budget(conn_bytes_given: bool, max_pack_size: Option) -> bool { + !conn_bytes_given && matches!(max_pack_size, None | Some(0)) +} + /// Serve a repo over the wire protocol: either on stdin/stdout (invoked as /// the remote-side command by `ssh` for `ssh://` remotes), or over TCP for /// `sc+http://` clients (P26, `--http `; blocks until the listener is @@ -3808,7 +3828,8 @@ fn run_clone_git(url: &str, dst: &std::path::Path) -> Result<()> { /// `--read-only`/`--allow-public` are `--http`-only (P29): `--stdio` /// delegates auth/access entirely to ssh, so combining them is refused /// rather than silently ignored. `--max-connections`/`--timeout` are also -/// `--http`-only (P31). `--max-pack-size` applies to both (P31). `--store` +/// `--http`-only (P31). `--max-pack-size` and `--max-conn-bytes` apply +/// to both (P31, #233). `--store` /// (P36c) redirects served content to a bucket WAL remote; `path` remains /// the serve home (tokens/TLS identity/scratch) either way. A malformed /// `--store` URL is validated up front, before any bind — same fail-fast @@ -3821,12 +3842,22 @@ fn run_serve( max_connections: Option, timeout: Option, max_pack_size: Option, + max_conn_bytes: Option, tls: bool, tls_cert: Option, tls_key: Option, store: Option, path: PathBuf, ) -> Result<()> { + let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE); + let default_budget = runs_default_budget(max_conn_bytes.is_some(), max_pack_size); + let conn_bytes = match max_conn_bytes { + Some(raw) => { + scl_core::parse_byte_size(&raw).map_err(|e| anyhow::anyhow!("--max-conn-bytes: {e}"))? + } + None => scl_repo::wire::default_max_conn_bytes(max_pack), + }; + scl_repo::wire::validate_max_conn_bytes(conn_bytes, max_pack, scl_core::max_object_size())?; if let Some(url) = &store { scl_repo::BucketUrl::parse(url)?; // fail fast on malformed URLs, before any bind } @@ -3863,12 +3894,12 @@ fn run_serve( path.display() ); } - let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE); scl_repo::wire::validate_max_pack_size(max_pack, scl_core::max_object_size())?; let policy = scl_repo::wire::WirePolicy { read_only: false, max_pack_size: max_pack, ro_drain_cap: scl_repo::wire::RO_DRAIN_CAP, + max_conn_bytes: conn_bytes, }; let mut stdin = std::io::stdin().lock(); let mut stdout = std::io::stdout().lock(); @@ -3895,6 +3926,15 @@ fn run_serve( if let Some(m) = max_pack_size { limits.max_pack_size = m; } + limits.max_conn_bytes = conn_bytes; + if default_budget && !read_only && !scl_repo::http_transport::addr_is_loopback(&addr) { + eprintln!( + "warning: writable listener on {addr} runs with the default --max-conn-bytes \ + ({conn_bytes} bytes per connection, times --max-connections); this does not \ + bound server memory or disk tightly. Lower --max-conn-bytes and \ + --max-pack-size together." + ); + } let tls_mode = match (tls, tls_cert, tls_key) { (false, None, None) => scl_repo::http_transport::TlsMode::Off, (false, _, _) => anyhow::bail!("--tls-cert/--tls-key require --tls"), @@ -4615,6 +4655,17 @@ mod tests { use super::*; use std::time::Duration; + #[test] + fn default_budget_warning_logic() { + assert!(runs_default_budget(false, None)); + assert!( + runs_default_budget(false, Some(0)), + "0 = unlimited pack cap" + ); + assert!(!runs_default_budget(false, Some(1 << 30))); + assert!(!runs_default_budget(true, None)); + } + #[test] fn parses_suffixed_durations() { assert_eq!( diff --git a/crates/cli/tests/max_conn_bytes.rs b/crates/cli/tests/max_conn_bytes.rs new file mode 100644 index 00000000..fb37ff85 --- /dev/null +++ b/crates/cli/tests/max_conn_bytes.rs @@ -0,0 +1,163 @@ +//! `sc serve --max-conn-bytes` (#233, ADR-0039/0041 addenda): flag +//! parsing and bounds are checked at startup, and a real push over +//! `sc+http://` that exceeds the connection's write budget is refused with the +//! typed error while a push under it works. + +use std::io::BufRead; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Output, Stdio}; + +fn sc(dir: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_sc")) + .args(args) + .current_dir(dir) + .env_remove("SC_MAX_OBJECT_SIZE") + .env("SC_AUTHOR", "t") + .output() + .expect("sc runs") +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-connbytes-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +struct Server(Child); +impl Drop for Server { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn spawn_http_server(root: &Path, extra: &[&str]) -> (Server, String) { + let mut args = vec!["serve", "--http", "127.0.0.1:0"]; + args.extend_from_slice(extra); + args.push(root.to_str().unwrap()); + let mut child = Command::new(env!("CARGO_BIN_EXE_sc")) + .args(&args) + .env_remove("SC_MAX_OBJECT_SIZE") + .stdout(Stdio::piped()) + .spawn() + .expect("spawn sc serve --http"); + let mut reader = std::io::BufReader::new(child.stdout.take().unwrap()); + let mut line = String::new(); + assert!(reader.read_line(&mut line).unwrap() > 0, "server announced"); + let addr = line + .trim() + .strip_prefix("listening on ") + .unwrap_or_else(|| panic!("unexpected startup line: {line:?}")) + .to_string(); + std::thread::spawn(move || for _ in reader.lines() {}); + (Server(child), addr) +} + +#[test] +fn flag_is_parsed_and_bounds_checked_at_startup() { + let root = tmp("flag"); + assert!(sc(&root, &["init"]).status.success()); + // Unparsable, below the object cap (default 256 MiB), and below an + // explicit pack cap are all refused before serving. + for (args, needle) in [ + (vec!["--max-conn-bytes", "x"], "--max-conn-bytes"), + (vec!["--max-conn-bytes", "1.5MiB"], "--max-conn-bytes"), + (vec!["--max-conn-bytes", "0"], "greater than zero"), + ( + vec!["--max-conn-bytes", "1MiB"], + "below the max object size (268435456)", + ), + ( + vec![ + "--max-object-size", + "2MiB", + "--max-pack-size", + "8388608", + "--max-conn-bytes", + "4MiB", + ], + "below --max-pack-size", + ), + ] { + let mut a = vec!["serve", "--stdio"]; + a.extend(&args); + a.push("."); + let o = sc(&root, &a); + assert!(!o.status.success(), "{args:?} must be rejected"); + assert!(stderr(&o).contains(needle), "{args:?}: {}", stderr(&o)); + } + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn a_push_over_the_budget_is_refused_and_one_under_it_works() { + // Unlimited pack cap so the connection budget is the binding limit. + let server_dir = tmp("srv"); + assert!(sc(&server_dir, &["init"]).status.success()); + let (server, addr) = spawn_http_server( + &server_dir, + &[ + "--max-object-size", + "2MiB", + "--max-pack-size", + "0", + "--max-conn-bytes", + "2MiB", + ], + ); + let url = format!("sc+http://{addr}"); + + let client = tmp("cli"); + assert!(sc(&client, &["init"]).status.success()); + assert!(sc(&client, &["remote", "add", "origin", &url]) + .status + .success()); + + // Incompressible content so the spooled pack is about as big as the file. + let noise = |len: usize| -> Vec { + let mut x = 0x9E37_79B9_7F4A_7C15u64; + (0..len) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect() + }; + std::fs::write(client.join("big.bin"), noise(3 * 1024 * 1024)).unwrap(); + assert!(sc(&client, &["commit", "-m", "big"]).status.success()); + let o = sc(&client, &["push", "origin"]); + assert!(!o.status.success(), "3 MiB push exceeds a 2 MiB budget"); + let err = stderr(&o); + assert!( + err.contains("connection write budget exceeded") && err.contains("--max-conn-bytes"), + "typed refusal names the knob: {err}" + ); + + let small = tmp("small"); + assert!(sc(&small, &["init"]).status.success()); + assert!(sc(&small, &["remote", "add", "origin", &url]) + .status + .success()); + std::fs::write(small.join("a.txt"), "hello\n").unwrap(); + assert!(sc(&small, &["commit", "-m", "small"]).status.success()); + let o = sc(&small, &["push", "origin"]); + assert!( + o.status.success(), + "a push under the budget works: {}", + stderr(&o) + ); + + drop(server); // stop the listener before deleting its root + for d in [server_dir, client, small] { + std::fs::remove_dir_all(&d).unwrap(); + assert!(!d.exists()); + } +} diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index ed28b3e8..fadf5822 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -70,14 +70,14 @@ fn check_object_size_bounds(n: usize) -> std::result::Result<(), String> { } } -/// Parse an object-size setting: plain bytes, or a number with a `KiB`, -/// `MiB` or `GiB` suffix (case-insensitive). Applies the same hard bounds as -/// [`set_max_object_size`]; an out-of-range value is an error, not clamped. -pub fn parse_object_size(s: &str) -> std::result::Result { +/// Parse a byte-size setting: plain bytes, or a number with a `KiB`, `MiB` or +/// `GiB` suffix (case-insensitive). No range check; callers apply their own +/// bounds (see [`parse_object_size`]). +pub fn parse_byte_size(s: &str) -> std::result::Result { let t = s.trim(); let lower = t.to_ascii_lowercase(); let (digits, mult) = if let Some(d) = lower.strip_suffix("kib") { - (d, 1usize << 10) + (d, 1u64 << 10) } else if let Some(d) = lower.strip_suffix("mib") { (d, 1 << 20) } else if let Some(d) = lower.strip_suffix("gib") { @@ -85,14 +85,22 @@ pub fn parse_object_size(s: &str) -> std::result::Result { } else { (lower.as_str(), 1) }; - let n = digits + digits .trim() - .parse::() + .parse::() .ok() .and_then(|n| n.checked_mul(mult)) .ok_or_else(|| { - format!("invalid object size {s:?}: expected bytes or a KiB/MiB/GiB suffix, e.g. 64MiB") - })?; + format!("invalid size {s:?}: expected bytes or a KiB/MiB/GiB suffix, e.g. 64MiB") + }) +} + +/// Parse an object-size setting: [`parse_byte_size`] plus the same hard +/// bounds as [`set_max_object_size`]; an out-of-range value is an error, not +/// clamped. +pub fn parse_object_size(s: &str) -> std::result::Result { + let n = usize::try_from(parse_byte_size(s)?) + .map_err(|_| format!("invalid object size {s:?}: too large"))?; check_object_size_bounds(n)?; Ok(n) } @@ -208,6 +216,17 @@ mod tests { .contains("allowed range")); } + #[test] + fn parse_byte_size_accepts_suffixes_without_bounds() { + assert_eq!(parse_byte_size("0"), Ok(0)); + assert_eq!(parse_byte_size(" 3kib "), Ok(3 << 10)); + assert_eq!(parse_byte_size("16GiB"), Ok(16 << 30)); + assert!(parse_byte_size("").is_err()); + assert!(parse_byte_size("-1").is_err()); + assert!(parse_byte_size("1.5MiB").is_err()); + assert!(parse_byte_size("99999999999999999999GiB").is_err()); + } + #[test] fn set_max_object_size_rejects_out_of_range_values() { assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index 83c125d6..42af300c 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -170,6 +170,12 @@ pub enum Error { /// human-readable limit text, carried verbatim across the wire. #[error("pack exceeds the server's --max-pack-size limit: {0}")] PackTooLarge(String), + /// The server closed the connection because the bytes it wrote for this + /// peer (`PutObject` + `PutPack`) exceeded the operator's + /// `--max-conn-bytes` budget (#233). Payload is the server's own + /// human-readable limit text, carried verbatim across the wire. + #[error("connection write budget exceeded: {0}")] + ConnectionBudgetExceeded(String), /// The server refused the connection at accept time because /// `--max-connections` was reached (P31). Retryable. #[error("server busy (connection limit reached); retry later")] diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index 318bcd8e..f0f853af 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -836,6 +836,10 @@ pub struct ServeLimits { /// Forwarded to [`crate::wire::WirePolicy::max_pack_size`]. `0` = /// unlimited. pub max_pack_size: u64, + /// Forwarded to [`crate::wire::WirePolicy::max_conn_bytes`] (#233). + /// The CLI refuses `0`. The default is + /// [`crate::wire::default_max_conn_bytes`] of the default pack cap. + pub max_conn_bytes: u64, } impl Default for ServeLimits { fn default() -> Self { @@ -843,6 +847,7 @@ impl Default for ServeLimits { max_connections: 32, timeout_secs: 300, max_pack_size: crate::wire::DEFAULT_MAX_PACK_SIZE, + max_conn_bytes: crate::wire::default_max_conn_bytes(crate::wire::DEFAULT_MAX_PACK_SIZE), } } } @@ -872,6 +877,13 @@ fn is_loopback_host(host: &str) -> bool { } } +/// Does the `host:port` listen address bind loopback only? (Used by the CLI's +/// default-budget startup warning, #233.) +pub fn addr_is_loopback(addr: &str) -> bool { + let host = addr.rsplit_once(':').map(|(h, _)| h).unwrap_or(addr); + is_loopback_host(host.trim_start_matches('[').trim_end_matches(']')) +} + /// The fail-closed bind decision (P29, tightened by P32/ADR-0042): a /// non-loopback `addr` is allowed only if justified by `--read-only`, /// `--allow-public`, or — once TLS is available — `--tls` plus ≥1 configured @@ -887,10 +899,7 @@ fn bind_is_allowed( allow_public: bool, tls: bool, ) -> Result { - let host = addr.rsplit_once(':').map(|(h, _)| h).unwrap_or(addr); - // Strip optional [..] brackets around an IPv6 literal. - let host = host.trim_start_matches('[').trim_end_matches(']'); - if is_loopback_host(host) { + if addr_is_loopback(addr) { return Ok(true); } if read_only || allow_public { @@ -1019,6 +1028,14 @@ pub fn serve_http( store: Option<&str>, ) -> Result<()> { crate::wire::validate_max_pack_size(limits.max_pack_size, scl_core::max_object_size())?; + // `0` is the embedder opt-out (unlimited); the CLI refuses it before here. + if limits.max_conn_bytes != 0 { + crate::wire::validate_max_conn_bytes( + limits.max_conn_bytes, + limits.max_pack_size, + scl_core::max_object_size(), + )?; + } let tls_config = resolve_tls(root, &tls)?; if !bind_is_allowed(addr, root, read_only, allow_public, tls_config.is_some())? { return Err(Error::InvalidArgument(format!( @@ -1350,6 +1367,7 @@ fn handle_http_connection( read_only, max_pack_size: limits.max_pack_size, ro_drain_cap: crate::wire::RO_DRAIN_CAP, + max_conn_bytes: limits.max_conn_bytes, }; match store { Some(url) => { diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 89a54870..965caf0e 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -186,7 +186,23 @@ impl Transport for WireClient { fn put_pack(&self, src: &mut dyn Read) -> Result> { let mut rw = self.rw.borrow_mut(); wire::write_frame(&mut rw.1, &Request::PutPack.encode())?; - wire::write_pack_stream(&mut rw.1, src, wire::pack_chunk_size())?; + let mut sock = SocketWrites { + inner: &mut rw.1, + dead: false, + }; + if let Err(e) = wire::write_pack_stream(&mut sock, src, wire::pack_chunk_size()) { + // A server that refused mid-stream (pack or connection budget) + // replies and closes; surface its typed error instead of the + // broken pipe. Only when the *socket write* failed on a dead + // connection: an error reading `src` is local and the server is + // still waiting for us, so a reply read could block. + if sock.dead { + if let Ok(frame) = wire::read_frame(&mut rw.0) { + wire::parse_response(frame)?; + } + } + return Err(e); + } let frame = wire::read_frame(&mut rw.0)?; wire::decode_ids_body(&wire::parse_response(frame)?) } @@ -200,6 +216,29 @@ impl Transport for WireClient { } } +/// Write adapter that records whether a write to the peer failed because the +/// connection is gone (not a timeout), to tell it from a local `src` error. +struct SocketWrites<'a, W: Write> { + inner: &'a mut W, + dead: bool, +} + +impl Write for SocketWrites<'_, W> { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.inner.write(buf).inspect_err(|e| self.note(e)) + } + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush().inspect_err(|e| self.note(e)) + } +} + +impl SocketWrites<'_, W> { + fn note(&mut self, e: &std::io::Error) { + use std::io::ErrorKind::{BrokenPipe, ConnectionAborted, ConnectionReset}; + self.dead |= matches!(e.kind(), BrokenPipe | ConnectionReset | ConnectionAborted); + } +} + /// Most bytes a client accepts in one `GetObjects` reply stream: twice the /// server's by-id reply ceiling, so pack framing never trips it on a /// legitimate reply. A constant, not derived from the local object cap: it diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index cd5e3833..1c4e6eaf 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -325,6 +325,68 @@ impl LocalTransport { let mut store = self.store.borrow_mut(); ingest_pack_file(&self.layout, &mut store, path) } + + /// [`Self::ingest_from`] for a budgeted serve connection (#233): pass 1 + /// sums the encoded size of the pack's structural records into `tally` + /// and refuses with [`Error::ConnectionBudgetExceeded`] before any write + /// when that exceeds `tally.limit`. + pub(crate) fn ingest_from_tallied( + &self, + path: &std::path::Path, + tally: &mut PackTally, + ) -> Result> { + let mut store = self.store.borrow_mut(); + ingest_pack_file_impl( + &self.layout, + &mut store, + path, + None, + |_| Ok(()), + false, + Some(tally), + ) + } +} + +/// Decoded bytes a pack makes the receiver hold (#233). Trees, snapshots, +/// secrets and transcripts report `blob_size` 0, so the store budget never +/// sees them, and a pack's compressed size can understate them by the zstd +/// ratio. A local store counts only those; a bucket-backed receiver decodes +/// every record into memory before upload, so it sets `count_blobs`. `limit` +/// of `None` is unlimited. +#[derive(Default)] +pub(crate) struct PackTally { + pub(crate) limit: Option, + pub(crate) count_blobs: bool, + pub(crate) seen: u64, +} + +impl PackTally { + /// Add one record; true once `seen` is over `limit`. + fn add(&mut self, obj: &scl_core::Object) -> bool { + if self.count_blobs || obj.blob_size() == 0 { + self.seen = self.seen.saturating_add(obj.encode().len() as u64); + } + self.limit.is_some_and(|l| self.seen > l) + } +} + +/// Verify-only pass over a spooled pack that fills `tally` (the bucket serve +/// path, which has no store pass 1 to piggyback on). Refuses with +/// [`Error::ConnectionBudgetExceeded`] as soon as `tally.limit` is crossed. +pub(crate) fn scan_pack_tally(path: &std::path::Path, tally: &mut PackTally) -> Result<()> { + let mut over = false; + let parsed = scl_core::pack::parse_pack_reader(std::fs::File::open(path)?, |_, obj| { + if tally.add(&obj) { + over = true; + return Err(scl_core::Error::PackCorrupt("connection budget".into())); + } + Ok(()) + }); + if over { + return Err(Error::ConnectionBudgetExceeded(String::new())); + } + Ok(parsed?) } /// RAII guard for a per-transfer temporary pack file under `.sc/tmp/` @@ -461,7 +523,7 @@ pub(crate) fn ingest_pack_file_checked( max_records: Option, check: impl FnOnce(&std::collections::BTreeSet) -> Result<()>, ) -> Result> { - ingest_pack_file_impl(layout, store, path, max_records, check, false) + ingest_pack_file_impl(layout, store, path, max_records, check, false, None) } /// [`ingest_pack_file`] for a `--no-transcripts` receiver (ADR-0038 addendum, @@ -476,7 +538,7 @@ pub(crate) fn ingest_pack_file_without_transcripts( store: &mut Store, path: &std::path::Path, ) -> Result> { - ingest_pack_file_impl(layout, store, path, None, |_| Ok(()), true) + ingest_pack_file_impl(layout, store, path, None, |_| Ok(()), true, None) } fn ingest_pack_file_impl( @@ -486,6 +548,7 @@ fn ingest_pack_file_impl( max_records: Option, check: impl FnOnce(&std::collections::BTreeSet) -> Result<()>, skip_transcripts: bool, + mut tally: Option<&mut PackTally>, ) -> Result> { // Transcript ids seen in pass 1; pass 2 uses them to also drop their signatures. let mut dropped = std::collections::BTreeSet::new(); @@ -494,8 +557,13 @@ fn ingest_pack_file_impl( let f = std::fs::File::open(path)?; let mut seen = std::collections::BTreeSet::new(); let mut records = 0usize; - scl_core::pack::parse_pack_reader(f, |id, obj| { + let mut over_budget = false; + let parsed = scl_core::pack::parse_pack_reader(f, |id, obj| { records += 1; + if tally.as_deref_mut().is_some_and(|t| t.add(&obj)) { + over_budget = true; + return Err(scl_core::Error::PackCorrupt("connection budget".into())); + } if max_records.is_some_and(|max| records > max) { return Err(scl_core::Error::PackCorrupt(format!( "pack holds more than the {} record(s) requested", @@ -507,7 +575,12 @@ fn ingest_pack_file_impl( } seen.insert(id); Ok(()) - })?; + }); + if over_budget { + // The wire layer swaps in its own limit text. + return Err(Error::ConnectionBudgetExceeded(String::new())); + } + parsed?; check(&seen)?; } // Pass 2: every record verified in pass 1 — now write each into the store. diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index 52071366..a6c454cb 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -106,6 +106,10 @@ const EC_PROTOCOL: u8 = 4; /// mutating verb before any store write. See [`serve_with_policy`]. const EC_READONLY: u8 = 5; pub(crate) const EC_TOO_LARGE: u8 = 6; +/// #233: the connection's `--max-conn-bytes` write budget ran out. An +/// older peer degrades an unknown code to `Error::Remote(msg)`, so no +/// `PROTOCOL_VERSION` bump is needed. +const EC_BUDGET: u8 = 7; const EC_OTHER: u8 = 255; /// A client request — the wire mirror of one [`crate::transport::Transport`] @@ -616,10 +620,11 @@ pub fn err_to_wire(e: &Error) -> (u8, String) { Error::Protocol(_) => EC_PROTOCOL, Error::ReadOnly => EC_READONLY, Error::PackTooLarge(_) => EC_TOO_LARGE, + Error::ConnectionBudgetExceeded(_) => EC_BUDGET, _ => EC_OTHER, }; let msg = match e { - Error::PackTooLarge(inner) => inner.clone(), + Error::PackTooLarge(inner) | Error::ConnectionBudgetExceeded(inner) => inner.clone(), _ => e.to_string(), }; (code, msg) @@ -634,6 +639,7 @@ pub fn wire_to_err(code: u8, msg: String) -> Error { EC_PROTOCOL => Error::Protocol(msg), EC_READONLY => Error::ReadOnly, EC_TOO_LARGE => Error::PackTooLarge(msg), + EC_BUDGET => Error::ConnectionBudgetExceeded(msg), _ => Error::Remote(msg), // EC_NOT_FOUND + EC_OTHER: typed enough as text } } @@ -738,7 +744,7 @@ pub fn decode_refs_body(b: &[u8]) -> Result> { } use crate::bucket_transport::BucketTransport; -use crate::transport::{LocalTransport, TempPackGuard, Transport}; +use crate::transport::{scan_pack_tally, LocalTransport, PackTally, TempPackGuard, Transport}; /// Default cap on an incoming `PutPack` spool when no operator override is /// configured (P31): 16 GiB. Threaded through [`WirePolicy::max_pack_size`]; @@ -753,6 +759,21 @@ pub const DEFAULT_MAX_PACK_SIZE: u64 = 16 * 1024 * 1024 * 1024; /// [`WirePolicy::max_pack_size`] because that pack might actually be used. pub const RO_DRAIN_CAP: u64 = 8 * 1024 * 1024; +/// Default `--max-conn-bytes` for a server whose `--max-pack-size` is +/// `max_pack_size` (#233): one maximal pack plus one maximal object +/// ([`scl_core::max_object_size`]), so a legitimate push that fills the pack +/// cap and adds a trailing object still fits on one connection, while the +/// total a peer can make the server write stays finite. An unlimited +/// (`0`) pack cap falls back to [`DEFAULT_MAX_PACK_SIZE`] for this sum. +pub fn default_max_conn_bytes(max_pack_size: u64) -> u64 { + let pack = if max_pack_size == 0 { + DEFAULT_MAX_PACK_SIZE + } else { + max_pack_size + }; + pack.saturating_add(scl_core::max_object_size() as u64) +} + /// Per-connection policy for [`serve_with_policy`] (P31): read-only gating /// (P29) plus the two spool caps above. `Default` matches every pre-P31 /// caller's prior unbounded behavior for `read_only`/`ro_drain_cap` sizing @@ -767,6 +788,14 @@ pub struct WirePolicy { /// Cap on how much of a read-only connection's rejected `PutPack` stream /// is drained before closing the connection, in bytes. pub ro_drain_cap: u64, + /// Per-connection budget over every object-write verb (`PutObject`'s + /// decoded length, structural objects included; per `PutPack` the larger of its + /// spooled bytes and the decoded bytes held: structural records on a local + /// store, every record on a bucket one), in bytes (#233). Exceeding it replies + /// [`Error::ConnectionBudgetExceeded`] and closes the connection. `0` + /// means unlimited (embedders and tests; `sc serve` refuses it). Reads + /// are not budgeted. + pub max_conn_bytes: u64, } impl Default for WirePolicy { @@ -775,6 +804,81 @@ impl Default for WirePolicy { read_only: false, max_pack_size: DEFAULT_MAX_PACK_SIZE, ro_drain_cap: RO_DRAIN_CAP, + max_conn_bytes: default_max_conn_bytes(DEFAULT_MAX_PACK_SIZE), + } + } +} + +/// Validate an operator-supplied `--max-conn-bytes`. Zero is refused (there +/// is no unlimited setting from `sc serve`; `0` in [`WirePolicy`] or +/// `ServeLimits` is for embedders and tests); a budget below the object cap could never admit one +/// object, and one below a nonzero `max_pack_size` would silently lower the +/// pack cap, so all three are refused at startup rather than clamped. +pub fn validate_max_conn_bytes(budget: u64, max_pack_size: u64, object_cap: usize) -> Result<()> { + if budget == 0 { + return Err(Error::InvalidArgument( + "--max-conn-bytes must be greater than zero".into(), + )); + } + if budget < object_cap as u64 { + return Err(Error::InvalidArgument(format!( + "--max-conn-bytes {budget} is below the max object size ({object_cap}); a budget that cannot fit one object is a misconfiguration" + ))); + } + if max_pack_size != 0 && budget < max_pack_size { + return Err(Error::InvalidArgument(format!( + "--max-conn-bytes {budget} is below --max-pack-size ({max_pack_size}); a budget below the pack cap is a misconfiguration (a pack is charged by its decoded size, which can still exceed the budget)" + ))); + } + Ok(()) +} + +/// Admit a `PutObject` body of `len` bytes under object cap `cap`: `Err` is a +/// budget refusal (close the connection), `Ok(Some(e))` an over-cap reply that +/// keeps the session open, `Ok(None)` admitted. The bytes are charged first, +/// so a frame that passes the frame limit but not the cap (it may exceed the +/// cap by `FRAME_OVERHEAD - 37`) still uses up the budget. +fn gate_put_object(budget: &mut ConnBudget, len: usize, cap: usize) -> Result> { + budget.charge(len as u64)?; + Ok(check_object_len(len, cap).err()) +} + +/// Running count of bytes a connection has made the server write (#233). +struct ConnBudget { + limit: u64, + used: u64, +} + +impl ConnBudget { + fn refusal(&self) -> Error { + Error::ConnectionBudgetExceeded(format!( + "server cap is {} bytes of writes per connection; raise --max-conn-bytes on the server or split the push across connections", + self.limit + )) + } + + /// Account `n` incoming bytes, refusing before they are written when the + /// budget would be exceeded. Attempts count, accepted or not. + fn charge(&mut self, n: u64) -> Result<()> { + self.used = self.used.saturating_add(n); + if self.limit != 0 && self.used > self.limit { + return Err(self.refusal()); + } + Ok(()) + } + + /// Spool cap for the next pack: `max_pack_size` tightened to what is left + /// of the budget, and whether the budget is the binding one. + fn pack_cap(&self, max_pack_size: u64) -> (u64, bool) { + if self.limit == 0 { + return (max_pack_size, false); + } + // `max(1)`: a cap of 0 would mean unlimited to the spool reader. + let left = self.limit.saturating_sub(self.used).max(1); + if max_pack_size == 0 || left < max_pack_size { + (left, true) + } else { + (max_pack_size, false) } } } @@ -1033,6 +1137,10 @@ fn serve_session( w: &mut impl Write, policy: WirePolicy, ) -> Result<()> { + let mut budget = ConnBudget { + limit: policy.max_conn_bytes, + used: 0, + }; loop { let frame = match read_frame_opt(r)? { Some(f) => f, @@ -1070,7 +1178,7 @@ fn serve_session( // push is discarded regardless, so there's no reason to // spool an attacker-sized pack just to reject it. match spill_pack_stream(r, &transport.tmp_dir(), policy.ro_drain_cap) { - Ok(guard) => { + Ok((guard, _)) => { drop(guard); let (code, msg) = err_to_wire(&Error::ReadOnly); write_err(w, code, &msg)?; @@ -1162,16 +1270,46 @@ fn serve_session( } } Request::PutPack => { - match spill_pack_stream(r, &transport.tmp_dir(), policy.max_pack_size) { - Ok(guard) => { + let (cap, by_budget) = budget.pack_cap(policy.max_pack_size); + match spill_pack_stream(r, &transport.tmp_dir(), cap) { + Ok((guard, spooled)) => { + // A pack costs max(spool bytes, decoded bytes the + // receiver holds), not their sum: the spool is the + // compressed form of the same records. The decoded + // tally (structural records on a local store, every + // record on a bucket one) is checked against the + // whole remaining budget before any store write. + let left = budget.limit.saturating_sub(budget.used); + if budget.limit != 0 && spooled > left { + let (code, msg) = err_to_wire(&budget.refusal()); + let _ = write_err(w, code, &msg); + return Ok(()); + } + let mut tally = PackTally { + limit: (budget.limit != 0).then_some(left), + count_blobs: matches!(transport, ServeTransport::Bucket { .. }), + seen: 0, + }; let ingest_result = match &transport { - ServeTransport::Local(t) => t.ingest_from(guard.path()), + ServeTransport::Local(t) => { + t.ingest_from_tallied(guard.path(), &mut tally) + } ServeTransport::Bucket { transport: bt, .. } => { - bucket_put_pack_from_file(bt, guard.path()) + scan_pack_tally(guard.path(), &mut tally) + .and_then(|()| bucket_put_pack_from_file(bt, guard.path())) } }; + // Charge before inspecting the result: an error after + // pass 2 began (e.g. an index write) can leave records + // stored, and the connection stays open. + budget.used = budget.used.saturating_add(spooled.max(tally.seen)); match ingest_result { Ok(ids) => write_ok(w, &ids_body(&ids))?, + Err(Error::ConnectionBudgetExceeded(_)) => { + let (code, msg) = err_to_wire(&budget.refusal()); + let _ = write_err(w, code, &msg); + return Ok(()); + } Err(e) => { let (code, msg) = err_to_wire(&e); write_err(w, code, &msg)?; @@ -1187,11 +1325,51 @@ fn serve_session( // nor a subsequent request. A best-effort reply is // the most we can do; then close the connection // rather than serve further requests off a - // desynced stream. + // desynced stream. When the remaining connection + // budget was the binding cap, say so (#233). + let over_cap = matches!(e, Error::PackTooLarge(_)); + let e = match e { + Error::PackTooLarge(_) if by_budget => budget.refusal(), + e => e, + }; + let (code, msg) = err_to_wire(&e); + let _ = write_err(w, code, &msg); + if over_cap { + // The stream was well-formed, just too big. Closing + // with unread bytes makes the OS reset the socket + // and can destroy the reply above, so drain a + // bounded amount first (best-effort). Reuses the + // read-only drain cap: same job, same bound. + let _ = read_pack_stream(r, &mut std::io::sink(), policy.ro_drain_cap); + } + return Ok(()); + } + } + } + Request::PutObject { id, bytes } => { + // Distinct structural objects report `blob_size` 0, so the + // budget counts the encoded length every time (#233). + match gate_put_object(&mut budget, bytes.len(), scl_core::max_object_size()) { + Err(e) => { + // Refuse before the store write, then close: a peer + // that exhausted its budget gets no further requests. let (code, msg) = err_to_wire(&e); let _ = write_err(w, code, &msg); return Ok(()); } + Ok(Some(e)) => { + let (code, msg) = err_to_wire(&e); + write_err(w, code, &msg)?; + continue; + } + Ok(None) => {} + } + match transport.as_transport().put_object(&id, &bytes) { + Ok(()) => write_ok(w, &[])?, + Err(e) => { + let (code, msg) = err_to_wire(&e); + write_err(w, code, &msg)?; + } } } other => { @@ -1210,14 +1388,6 @@ fn serve_session( transport.as_transport().has_object(&id).map(bool_body) } Request::GetObject(id) => transport.as_transport().get_object(&id), - Request::PutObject { id, bytes } => { - check_object_len(bytes.len(), scl_core::max_object_size()).and_then(|()| { - transport - .as_transport() - .put_object(&id, &bytes) - .map(|()| Vec::new()) - }) - } Request::UpdateRef { branch, id, @@ -1229,6 +1399,7 @@ fn serve_session( Request::Bye | Request::GetPack { .. } | Request::PutPack + | Request::PutObject { .. } | Request::GetObjects { .. } => { unreachable!("handled above") } @@ -1266,11 +1437,11 @@ fn spill_pack_stream( r: &mut impl Read, tmp_dir: &std::path::Path, max_bytes: u64, -) -> Result { +) -> Result<(TempPackGuard, u64)> { let guard = TempPackGuard::new_in(tmp_dir)?; let mut f = std::fs::File::create(guard.path())?; - read_pack_stream(r, &mut f, max_bytes)?; - Ok(guard) + let n = read_pack_stream(r, &mut f, max_bytes)?; + Ok((guard, n)) } #[cfg(test)] @@ -1999,6 +2170,7 @@ mod tests { read_only: false, max_pack_size: 4096, ro_drain_cap: RO_DRAIN_CAP, + max_conn_bytes: 0, }; serve_with_policy(&root, &mut reader, &mut output, policy).unwrap(); @@ -2084,6 +2256,7 @@ mod tests { read_only: true, max_pack_size: 0, ro_drain_cap: 4096, + max_conn_bytes: 0, }; serve_with_policy(&root, &mut reader, &mut output, policy).unwrap(); @@ -2171,6 +2344,7 @@ mod tests { read_only: true, max_pack_size, ro_drain_cap: 4096, + max_conn_bytes: 0, }; serve_with_policy(&root, &mut std::io::Cursor::new(input), &mut output, policy) .unwrap(); @@ -2196,6 +2370,505 @@ mod tests { let _ = std::fs::remove_dir_all(&root); } + /// A distinct tree whose encoding is ~`len` bytes (one long entry name): + /// structural, so its `blob_size` is 0 and only its encoded length counts. + fn big_tree(seed: u8, len: usize) -> (ObjectId, Vec) { + use scl_core::object::{EntryKind, FileMode, Tree, TreeEntry}; + let name = format!("{seed:03}{}", "n".repeat(len)); + let obj = scl_core::Object::Tree(Tree::new(vec![TreeEntry { + name, + kind: EntryKind::Blob, + id: some_id(seed), + mode: FileMode::FILE, + perms: 0, + }])); + assert_eq!(obj.blob_size(), 0); + (obj.id(), obj.encode()) + } + + /// Run `reqs` (and any raw `PutPack` streams, keyed by request position) + /// through `serve_with_policy` after a HELLO; returns the reply frames + /// after the hello reply. + fn run_budget_session( + root: &std::path::Path, + policy: WirePolicy, + reqs: &[(Request, Option>)], + ) -> Vec>> { + let mut input = Vec::new(); + let hello = Request::Hello { + version: PROTOCOL_VERSION, + }; + write_frame(&mut input, &hello.encode()).unwrap(); + for (req, pack) in reqs { + write_frame(&mut input, &req.encode()).unwrap(); + if let Some(pack) = pack { + write_pack_stream(&mut input, &mut std::io::Cursor::new(pack), 1024).unwrap(); + } + } + let mut output = Vec::new(); + serve_with_policy(root, &mut std::io::Cursor::new(input), &mut output, policy).unwrap(); + let mut r = std::io::Cursor::new(output); + let mut frames = Vec::new(); + while let Some(f) = read_frame_opt(&mut r).unwrap() { + frames.push(parse_response(f)); + } + assert!(frames.remove(0).is_ok(), "hello"); + frames + } + + fn put_tree(seed: u8, len: usize) -> (Request, Option>) { + let (id, bytes) = big_tree(seed, len); + (Request::PutObject { id, bytes }, None) + } + + fn put_pack_of(seed: u8, len: usize) -> (Request, Option>) { + let (id, bytes) = big_tree(seed, len); + let (pack, _idx) = scl_core::pack::build_pack(&[(id, bytes)]).unwrap(); + (Request::PutPack, Some(pack)) + } + + fn no_spool_residue(root: &std::path::Path) { + let tmp = root.join(".sc").join("tmp"); + assert!(!tmp.exists() || std::fs::read_dir(&tmp).unwrap().next().is_none()); + } + + /// #233: repeated near-cap `PutObject` of distinct structural objects is + /// refused once the connection budget is spent, before the store write; + /// the connection then closes; a fresh connection starts at zero. + #[test] + fn put_object_budget_refuses_structural_floods_and_closes() { + let root = tmp_repo("budget-obj"); + let policy = WirePolicy { + max_conn_bytes: 2_500_000, + ..Default::default() + }; + let reqs = [ + put_tree(1, 1_000_000), + put_tree(2, 1_000_000), + put_tree(3, 1_000_000), + (Request::HeadBranch, None), // must never be answered + ]; + let frames = run_budget_session(&root, policy, &reqs); + assert_eq!( + frames.len(), + 3, + "two ok, one refusal, then closed: {frames:?}" + ); + assert!(frames[0].is_ok() && frames[1].is_ok()); + assert!(matches!( + frames[2].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(m) if m.contains("--max-conn-bytes") + )); + let t = crate::transport::LocalTransport::open(&root).unwrap(); + assert!(t.has_object(&big_tree(2, 1_000_000).0).unwrap()); + assert!( + !t.has_object(&big_tree(3, 1_000_000).0).unwrap(), + "the refused object never reached the store" + ); + + // The budget is per connection: a new session accepts the next object. + let frames = run_budget_session(&root, policy, &[put_tree(4, 1_000_000)]); + assert!(frames[0].is_ok(), "{frames:?}"); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// A legitimate single push under the default budget (one pack plus an + /// object) is untouched, and a policy of `0` (embedders only) is unlimited. + #[test] + fn legit_push_under_default_budget_and_zero_opt_out() { + let root = tmp_repo("budget-ok"); + let frames = run_budget_session( + &root, + WirePolicy::default(), + &[put_pack_of(10, 1_500_000), put_tree(11, 1_500_000)], + ); + assert!(frames.iter().all(|f| f.is_ok()), "{frames:?}"); + + let policy = WirePolicy { + max_conn_bytes: 0, + ..Default::default() + }; + let reqs: Vec<_> = (20..24).map(|i| put_tree(i, 500_000)).collect(); + let frames = run_budget_session(&root, policy, &reqs); + assert_eq!(frames.len(), 4); + assert!(frames.iter().all(|f| f.is_ok()), "{frames:?}"); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// Objects and packs draw on one budget. A pack of compressible structural + /// records is charged by its encoded size (the spool is tiny), so the + /// third request is refused by the structural tally, before ingest: the + /// connection closes, nothing from the refused pack lands, no spool + /// residue is left. + #[test] + fn put_pack_structural_bytes_share_the_budget() { + let root = tmp_repo("budget-pack"); + let policy = WirePolicy { + max_pack_size: 0, + max_conn_bytes: 1_500_000, + ..Default::default() + }; + // 500k (pack) + 500k (object) leaves 500k; the 800k pack does not fit. + let frames = run_budget_session( + &root, + policy, + &[ + put_pack_of(30, 500_000), + put_tree(31, 500_000), + put_pack_of(32, 800_000), + (Request::HeadBranch, None), // must never be answered + ], + ); + assert_eq!(frames.len(), 3, "{frames:?}"); + assert!(frames[0].is_ok() && frames[1].is_ok()); + assert!(matches!( + frames[2].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + )); + let t = crate::transport::LocalTransport::open(&root).unwrap(); + assert!(!t.has_object(&big_tree(32, 800_000).0).unwrap()); + no_spool_residue(&root); + + // Same for a pack after a large object: 3M + 2M > 4M. + let policy = WirePolicy { + max_pack_size: 4_000_000, + max_conn_bytes: 4_000_000, + ..Default::default() + }; + let frames = run_budget_session( + &root, + policy, + &[put_tree(33, 3_000_000), put_pack_of(34, 2_000_000)], + ); + assert!(frames[0].is_ok()); + assert!(matches!( + frames[1].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + )); + no_spool_residue(&root); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// Incompressible noise as one blob pack: the spool is the full size and + /// the structural tally sees nothing, so only the spool cutoff can trip. + fn put_blob_pack(seed: u8, len: usize) -> (Request, Option>) { + let mut x = 0x9E37_79B9_7F4A_7C15u64 ^ u64::from(seed); + let noise: Vec = (0..len) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + let obj = scl_core::Object::blob(noise); + let (pack, _) = scl_core::pack::build_pack(&[(obj.id(), obj.encode())]).unwrap(); + (Request::PutPack, Some(pack)) + } + + /// The spool cutoff: a pack whose bytes outrun what is left of the budget + /// is stopped mid-stream and reported as a budget refusal (not a + /// pack-size one), leaves no spool residue and closes the connection. With + /// no remaining budget conflict a same-size pack passes. + #[test] + fn pack_spool_cutoff_is_a_budget_refusal() { + let root = tmp_repo("budget-spool"); + let policy = WirePolicy { + max_pack_size: 0, + max_conn_bytes: 1_000_000, + ..Default::default() + }; + let frames = run_budget_session( + &root, + policy, + &[ + put_tree(35, 400_000), + put_blob_pack(36, 700_000), // 400k used, ~700k spool > 600k left + (Request::HeadBranch, None), // must never be answered + ], + ); + assert_eq!(frames.len(), 2, "{frames:?}"); + assert!(frames[0].is_ok()); + assert!( + matches!( + frames[1].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + ), + "a budget cutoff, not PackTooLarge: {frames:?}" + ); + no_spool_residue(&root); + + // The same blob pack alone fits. + let frames = run_budget_session(&root, policy, &[put_blob_pack(36, 700_000)]); + assert!(frames[0].is_ok(), "{frames:?}"); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// An over-cap `PutObject` body (the frame limit leaves it room) is + /// answered with a protocol error and the session stays open, but its bytes + /// are charged, so repeating it cannot run for free. + #[test] + fn oversized_put_object_attempts_are_charged() { + let mut budget = ConnBudget { + limit: 100, + used: 0, + }; + for _ in 0..9 { + let e = gate_put_object(&mut budget, 11, 10).unwrap(); + assert!(matches!(e, Some(Error::Protocol(_)))); + } + assert_eq!(budget.used, 99); + assert!(matches!( + gate_put_object(&mut budget, 11, 10), + Err(Error::ConnectionBudgetExceeded(_)) + )); + assert!(gate_put_object( + &mut ConnBudget { + limit: 100, + used: 0 + }, + 10, + 10 + ) + .unwrap() + .is_none()); + } + + /// The spool half of the max charge: an incompressible blob pack has no + /// structural bytes, so only its spooled size can use up the budget for + /// the request that follows (charging the tally alone would pass this). + #[test] + fn spool_bytes_of_a_pack_count_toward_later_requests() { + let root = tmp_repo("budget-spool-count"); + let policy = WirePolicy { + max_pack_size: 0, + max_conn_bytes: 1 << 20, + ..Default::default() + }; + let frames = run_budget_session( + &root, + policy, + &[put_blob_pack(37, 600_000), put_tree(38, 500_000)], + ); + assert!(frames[0].is_ok(), "{frames:?}"); + assert!( + matches!( + frames[1].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + ), + "{frames:?}" + ); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// Trees of random ids and names: zstd barely shrinks them, so the pack's + /// spool size is close to its structural bytes. + fn noisy_tree_pack(seed: u64, entries: usize) -> (Vec, u64, ObjectId) { + use scl_core::object::{EntryKind, FileMode, Tree, TreeEntry}; + let mut x = 0x9E37_79B9_7F4A_7C15u64 ^ seed; + let mut next = move || { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x + }; + let list = (0..entries) + .map(|_| { + let mut id = [0u8; 32]; + id.chunks_mut(8) + .for_each(|c| c.copy_from_slice(&next().to_le_bytes())); + TreeEntry { + name: format!("{:016x}", next()), + kind: EntryKind::Blob, + id: ObjectId::from_bytes(id), + mode: FileMode::FILE, + perms: 0, + } + }) + .collect(); + let obj = scl_core::Object::Tree(Tree::new(list)); + let encoded = obj.encode(); + let structural = encoded.len() as u64; + let (pack, _) = scl_core::pack::build_pack(&[(obj.id(), encoded)]).unwrap(); + (pack, structural, obj.id()) + } + + /// B1: a pack is charged max(spool, structural), not their sum. A pack of + /// near-incompressible structural records fits a budget equal to its + /// largest cost and to `--max-pack-size`; one byte less is refused. (The + /// sum would refuse the exact case: the startup bounds promise that one + /// maximal pack fits.) + #[test] + fn pack_cost_is_the_larger_of_spool_and_structural_bytes() { + let root = tmp_repo("budget-max"); + let (pack, structural, id) = noisy_tree_pack(7, 20_000); + let cost = structural.max(pack.len() as u64); + assert!( + pack.len() as u64 * 10 > structural * 6, + "premise: records are not very compressible ({} vs {structural})", + pack.len() + ); + let put = |cap: u64| { + run_budget_session( + &root, + WirePolicy { + max_pack_size: cap, + max_conn_bytes: cap, + ..Default::default() + }, + &[(Request::PutPack, Some(pack.clone()))], + ) + }; + let frames = put(cost); + assert!(frames[0].is_ok(), "exact cost fits: {frames:?}"); + let t = crate::transport::LocalTransport::open(&root).unwrap(); + assert!(t.has_object(&id).unwrap()); + + let (pack2, _, id2) = noisy_tree_pack(8, 20_000); + let frames = run_budget_session( + &root, + WirePolicy { + max_pack_size: cost - 1, + max_conn_bytes: cost - 1, + ..Default::default() + }, + &[(Request::PutPack, Some(pack2.clone()))], + ); + assert!( + matches!( + frames[0].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + ), + "{frames:?}" + ); + assert!(!t.has_object(&id2).unwrap(), "refused before any write"); + no_spool_residue(&root); + + // The default policy admits it too. + let frames = run_budget_session( + &root, + WirePolicy::default(), + &[(Request::PutPack, Some(pack2))], + ); + assert!(frames[0].is_ok(), "{frames:?}"); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// The budget is exact: a total equal to it is accepted, one byte more is + /// refused, and the refused object never lands. + #[test] + fn put_object_budget_boundary_is_exact() { + let root = tmp_repo("budget-edge"); + let (_, a) = big_tree(40, 1000); + let (_, b) = big_tree(41, 1000); + let total = (a.len() + b.len()) as u64; + let at = WirePolicy { + max_conn_bytes: total, + ..Default::default() + }; + let frames = run_budget_session(&root, at, &[put_tree(40, 1000), put_tree(41, 1000)]); + assert!(frames.iter().all(|f| f.is_ok()), "exact: {frames:?}"); + + let under = WirePolicy { + max_conn_bytes: total - 1, + ..Default::default() + }; + let frames = run_budget_session(&root, under, &[put_tree(42, 1000), put_tree(43, 1000)]); + assert!(frames[0].is_ok()); + assert!(matches!( + frames[1].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + )); + let t = crate::transport::LocalTransport::open(&root).unwrap(); + assert!(!t.has_object(&big_tree(43, 1000).0).unwrap()); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// Structural records inside a pack are retained resident, so their + /// encoded size is budgeted even when zstd makes the spooled pack tiny; + /// the over-budget pack is refused before any record lands. + #[test] + fn highly_compressible_structural_pack_is_budgeted_by_encoded_size() { + let root = tmp_repo("budget-zip"); + let (id, _) = big_tree(50, 3_000_000); + let (pack, _) = scl_core::pack::build_pack(&[(id, big_tree(50, 3_000_000).1)]).unwrap(); + assert!(pack.len() < 100_000, "premise: the pack compresses well"); + let policy = WirePolicy { + max_conn_bytes: 2_000_000, + ..Default::default() + }; + let frames = run_budget_session(&root, policy, &[(Request::PutPack, Some(pack))]); + assert!(matches!( + frames[0].as_ref().unwrap_err(), + Error::ConnectionBudgetExceeded(_) + )); + let t = crate::transport::LocalTransport::open(&root).unwrap(); + assert!(!t.has_object(&id).unwrap()); + no_spool_residue(&root); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + /// A read-only listener answers `ReadOnly` and never touches the budget: + /// reads and rejected writes are unaffected even by a tiny one. + #[test] + fn read_only_listener_ignores_the_budget() { + let root = tmp_repo("budget-ro"); + let policy = WirePolicy { + read_only: true, + max_conn_bytes: 1, + ..Default::default() + }; + let frames = run_budget_session( + &root, + policy, + &[ + put_tree(60, 1000), + put_tree(61, 1000), + (Request::HeadBranch, None), + ], + ); + assert_eq!(frames.len(), 3, "{frames:?}"); + assert!(matches!(frames[0].as_ref().unwrap_err(), Error::ReadOnly)); + assert!(matches!(frames[1].as_ref().unwrap_err(), Error::ReadOnly)); + assert!(!matches!( + frames[2].as_ref().err(), + Some(Error::ConnectionBudgetExceeded(_)) + )); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } + + #[test] + fn connection_budget_error_round_trips_and_flag_bounds() { + let e = Error::ConnectionBudgetExceeded("cap is 7".into()); + let (code, msg) = err_to_wire(&e); + assert_eq!(code, EC_BUDGET); + assert!(matches!( + wire_to_err(code, msg), + Error::ConnectionBudgetExceeded(m) if m == "cap is 7" + )); + + let obj = 4 << 20; + assert!(validate_max_conn_bytes(0, 100, obj).is_err()); + assert!(validate_max_conn_bytes(obj as u64 - 1, 0, obj).is_err()); + assert!(validate_max_conn_bytes(obj as u64, 0, obj).is_ok()); + assert!(validate_max_conn_bytes(10 << 20, 20 << 20, obj).is_err()); + assert!(validate_max_conn_bytes(20 << 20, 20 << 20, obj).is_ok()); + assert_eq!( + default_max_conn_bytes(0), + default_max_conn_bytes(DEFAULT_MAX_PACK_SIZE) + ); + assert!(default_max_conn_bytes(1 << 30) > 1 << 30); + } + /// Mint a one-commit object set the same way /// `bucket_transport::tests::tiny_history` does (blob -> tree -> snapshot /// via a real scratch repo), returning `(tip, objects)`. `tag` @@ -2312,6 +2985,48 @@ mod tests { assert!(!broot.exists()); } + /// A bucket-backed receiver decodes every record (blobs too) into memory + /// before upload, so the budget counts all of them: a highly compressible + /// blob pack is cheap to spool but refused by decoded size (#233). + #[test] + fn bucket_serve_budgets_decoded_blob_bytes() { + let broot = + std::env::temp_dir().join(format!("scl-wire-bucket-bud-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let url = format!("sc+wal://{}", broot.display()); + let home = tmp_repo("bucket-budget-home"); + let (client_read, mut server_write) = std::io::pipe().unwrap(); + let (mut server_read, client_write) = std::io::pipe().unwrap(); + let (url2, home2) = (url.clone(), home.clone()); + let srv = std::thread::spawn(move || { + serve_bucket_with_policy( + &url2, + &home2, + &mut server_read, + &mut server_write, + WirePolicy { + max_conn_bytes: 2_000_000, + ..Default::default() + }, + ) + }); + let client = + crate::stdio_transport::WireClient::handshake(client_read, client_write).unwrap(); + let blob = scl_core::Object::blob(vec![0u8; 3_000_000]); + let (pack, _) = scl_core::pack::build_pack(&[(blob.id(), blob.encode())]).unwrap(); + assert!(pack.len() < 100_000, "premise: the pack compresses well"); + let err = client + .put_pack(&mut std::io::Cursor::new(pack)) + .unwrap_err(); + assert!(matches!(err, Error::ConnectionBudgetExceeded(_)), "{err:?}"); + drop(client); + srv.join().unwrap().unwrap(); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + std::fs::remove_dir_all(&home).unwrap(); + assert!(!home.exists()); + } + /// A bucket-backed serve session refuses `GetObjects` (by-id reads exist /// only for partial clones, which bucket remotes refuse, ADR-0046) with a /// typed error, and the session stays usable afterwards. diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 409e29e8..07577c90 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -509,14 +509,23 @@ too, not just the src-control-side metadata. `LocalTransport::get_object` (load plus encode) before the receiver's cap rejects it, bounded by the local store budget. Wire sources refuse in `WireClient::get_object`, before decoding. - - **Structural objects have no per-connection aggregate budget (pre-existing, - amplified by a raised cap).** `Object::blob_size` is 0 for trees, - snapshots, secrets and transcripts, so repeated `PutObject` of distinct - structural objects on a writable listener is bounded per object (the cap) - but not in aggregate per connection: only by `--max-connections` times - connection lifetime (`--timeout`). Prefer read-only or token-gated - listeners, keep `--max-object-size` low, and size `--max-connections` - down. An aggregate per-connection byte budget is tracked in #233. + - **Writes are budgeted per connection (`--max-conn-bytes`, #233), but the + default does not protect RAM.** `Object::blob_size` is 0 for trees, + snapshots, secrets and transcripts, so the store budget does not see them; + a running per-connection counter now covers every `PutObject` (decoded + length) and every `PutPack` (the larger of its spooled bytes and the + decoded bytes the receiver holds: structural records on a local store, + every record on a `--store` bucket, which decodes them all before upload). + A request that would exceed it gets `ConnectionBudgetExceeded` and the + connection closes. The default is `--max-pack-size` plus the max object + size, about 16.25 GiB per connection, times `--max-connections` (default + 32): finite, but not a meaningful memory or disk bound. On a writable + listener, lower `--max-conn-bytes` and `--max-pack-size` together (the + budget must stay at least the pack cap); `sc serve --http` warns at + startup when a writable non-loopback listener runs on the default. Objects + the store already holds still count. A client sees the typed error only if + it overshoots by less than the 8 MiB drain cap; otherwise the connection + may reset. Reads are not budgeted. See the ADR-0039 #233 addendum. - **One length is checked after allocation:** bucket WAL metadata (manifest, log entry, idx) is capped at a fixed 256 MiB after the bucket GET has buffered it, so it bounds decode work, not the download. It is not tied to diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index 88d38d26..d56e7b50 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -3,7 +3,7 @@ - **Status:** Accepted - **Date:** 2026-07-09 - **Phase:** 28 -- **Amended:** #213 (addendum at the end) made the object cap configurable; the `MAX_OBJECT_SIZE` constant and 256 MiB figures in the sections before the addendum are the original design. +- **Amended:** #213 (first addendum) made the object cap configurable; the `MAX_OBJECT_SIZE` constant and 256 MiB figures in the sections before the addendum are the original design. #233 (second addendum, at the end) added a per-connection write budget and closes the "known gap" paragraph in the #213 addendum. - **Builds on:** ADR-0021 (durability/concurrency hardening — the P21 sweep precedent), ADR-0022 (ssh wire protocol), ADR-0015 (packfiles), ADR-0014 (protected paths), ADR-0008/0009 (secrets) ## Context @@ -213,13 +213,10 @@ connection's own store budget (each served connection opens its own store, the 1 GiB ceiling about 2 GiB + 2 GiB, per connection, times `--max-connections` (THREAT-MODEL.md). -Known gaps, not addressed here. `Object::blob_size` is 0 for structural -objects (trees, snapshots, secrets, transcripts), so a writable listener has no -per-connection aggregate byte bound on repeated `PutObject` of distinct -structural objects; each is capped, the sum is bounded only by -`--max-connections` x connection lifetime. This predates #213 and a raised cap -amplifies it: prefer read-only or token-gated listeners, keep the cap low. -Tracked in #233. A local-path remote loads and encodes an object before the +Known gap, since closed by #233 (second addendum): `Object::blob_size` is 0 for +structural objects (trees, snapshots, secrets, transcripts), so a writable +listener had no per-connection aggregate byte bound on repeated `PutObject` of +distinct structural objects. A local-path remote loads and encodes an object before the receiver's cap refuses it (`TransportSource::get`), bounded by the store budget. @@ -233,3 +230,100 @@ Unchanged: the cap is still a transfer-path guard, not a local-commit limit, so lowering it can leave a repository holding committed objects it cannot push; `read_object_at` (the trusted on-disk path) stays uncapped. The desktop app does not read the env var; it does no transfers. + +## Addendum (#233): per-connection write budget + +**Problem.** `Object::blob_size` is 0 for trees, snapshots, secrets and +transcripts, so the store budget never sees them. A writable `sc serve` +connection could `PutObject` distinct structural objects, each under the +object cap, with no bound on the sum except `--max-connections` x +connection lifetime (`--timeout` is an inactivity bound, not a total). +Repeated `PutPack` frames had the same shape: `--max-pack-size` caps one +pack, not the connection. + +**Decision.** `serve_session` keeps a running byte counter per connection +(`ConnBudget`), set by `sc serve --max-conn-bytes ` (`--http` and +`--stdio`; same parser and `KiB`/`MiB`/`GiB` suffixes as `--max-object-size`, +`scl_core::parse_byte_size`). It counts, per request: + +- `PutObject`: the frame's decoded length, every time, structural or not. + An object the store already holds still counts: checking would cost a + lookup per request and a replaying peer gains nothing from it. +- `PutPack`: the larger of the pack's spooled bytes and the decoded bytes + the receiver holds for it (below). It is `max`, not a sum, because the + spool is the compressed form of the same records: summing would refuse a + maximal pack of incompressible structure on a budget the startup bounds + say fits it. + +Reads (`GetObject`, `GetObjects`, `GetPack`, ref listing) are not budgeted. + +**What a pack holds (ingest-path finding).** Reading +`ingest_pack_file_impl`: a pack is spooled to disk, verified in pass 1, then +every record is `Store::put` in pass 2. The store keeps trees, snapshots, +secrets and transcripts resident (CLAUDE.md: "small, not evicted") and only +blobs and sealed payloads are budgeted, so a pack's structural records are +retained exactly like `PutObject` ones. The spool size understates them by +the zstd ratio (a 3 MB tree with one long name packs to a few KB). Pass 1 +therefore sums `Object::encode().len()` for every record whose `blob_size` +is 0 (`PackTally`) and refuses before pass 2 if the sum exceeds the whole +remaining budget. Blobs are not added on a local store: the store budget +bounds them and the spool already covers their bytes. A bucket-backed +`--store` is different: `BucketTransport::put_pack` decodes every record, +blobs included, into a `Vec` until the upload, so there `scan_pack_tally` +(one extra verify pass over the spool) counts every record before the +upload starts. + +**Enforcement and state.** A `PutObject` that would exceed the budget is +refused before the store write; a `PutPack` is cut off mid-stream at the +smaller of `--max-pack-size` and what is left of the budget (the existing +counted abort, reported as the budget error when the budget was the +binding cap) and a pack whose structural bytes do not fit is refused +before pass 2. Either way the server replies and closes the connection, +the same path as the `PutPack` Protocol-error arm: the spool guard drops, +no later request is served, and a retry on a fresh connection starts at +zero. Bytes of the refused request are counted (the counter is not +rolled back), which is moot because the connection ends. The comparison is +`used > limit`, so a total exactly equal to the budget is accepted. A pack's +cost is charged even when ingest then fails, since an error after pass 2 +began (an index write) can leave records stored on an open connection. + +**Delivery of the typed error.** After a mid-stream cap abort the server +drains up to `ro_drain_cap` (8 MiB, reused: it is the same "keep the socket +from being reset over unread bytes" job as the read-only path) before +closing. A client therefore sees the typed error when it overshoots by less +than that; past it the OS may reset the connection and the reply is lost. +`WireClient::put_pack` also tries to read a reply after a write fails on a +dead socket (broken pipe or reset), so a refusal that did get through is +reported instead of the I/O error. ADR-0041 decision 3 (immediate teardown +on `EC_TOO_LARGE`) is amended the same way: see its #233 note. + +**Wire.** New error code `EC_BUDGET = 7` mapped to +`Error::ConnectionBudgetExceeded(String)` (payload is the server's own +text, like `PackTooLarge`). `PROTOCOL_VERSION` is unchanged: an older client +degrades an unknown code to `Error::Remote(msg)`, which still carries the +message. + +**Bounds and default.** The flag must be greater than zero (`sc serve` has no +unlimited setting), at least the max object size (a smaller +budget could not admit one object) and at least a nonzero +`--max-pack-size` (a smaller one would silently lower the pack cap); +violations abort startup, never clamped. Default: `--max-pack-size` +(16 GiB when unlimited) plus the max object size, so a legitimate push that +fills the pack cap and adds one more object still fits one connection. That +default bounds the sum at about 16.25 GiB per connection. It does **not** +protect RAM or disk in any practical sense, and with `--max-connections` +(default 32) the product is far larger. An operator who wants a real bound +lowers `--max-pack-size` and `--max-conn-bytes` together; `sc serve --http` +prints a startup warning for a writable, non-loopback listener left on the +default. +A pack is bounded by its *decoded* size, not just `--max-pack-size`: on a +bucket `--store` a compressible pack's decoded size is usually 2-4x its spool +and can exceed the default, and on a local store a tree-dominated pack can +too. Refusing those is intended; they are the retention this budget exists to +bound. A workload that legitimately pushes more than the budget over one +connection must raise `--max-conn-bytes` or push over several connections. + +**Not covered.** The budget bounds bytes accepted, not request count or +CPU. A budgeted peer can still use `--max-connections` x the budget, and +`WirePolicy::max_conn_bytes` / `ServeLimits::max_conn_bytes` of 0 (embedders +and tests) is unlimited; only the CLI refuses it. diff --git a/docs/adr/0041-listener-resource-limits.md b/docs/adr/0041-listener-resource-limits.md index 8a4d2666..3ae3c1f4 100644 --- a/docs/adr/0041-listener-resource-limits.md +++ b/docs/adr/0041-listener-resource-limits.md @@ -198,3 +198,14 @@ scoped to the listener (`--stdio` is unaffected except where noted): cap is still trusting the OS's own thread/fd limits as the only backstop. `--max-connections 0` is an explicit opt-out of this phase's shedding behavior, not a claim that unlimited connections are safe. + +## Addendum (#233): the drain now covers `EC_TOO_LARGE` too + +Decision 3 tore the connection down right after the best-effort `EC_TOO_LARGE` +reply. Since #233 the PutPack arm, after replying to a *well-formed but too +large* stream (the `--max-pack-size` cap or the new per-connection budget, +ADR-0039 second addendum), drains up to `ro_drain_cap` (decision 4's 8 MiB) +before closing. Closing with unread bytes makes the OS reset the socket and +can destroy the reply; the drain keeps it deliverable when the client +overshoots by less than the cap. A malformed frame still tears down at once. +