diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 29094d6c..09338692 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -287,8 +287,16 @@ cargo run --bin sc -- serve --http [--read-only] [--allow-public] # trip; opening keeps its own 30s (P31) # --max-pack-size : incoming-pack aggregate # cap, both --http and --stdio, default 16 GiB, - # 0=unlimited, floor 256 MiB (MAX_OBJECT_SIZE); + # 0=unlimited, floor = the max object size; # counted mid-stream abort -> EC_TOO_LARGE (P31) + # --max-object-size : largest + # single object accepted from a peer, default + # 256MiB, range 2MiB..1GiB, out of range aborts + # startup (never clamped). Clients use env + # SC_MAX_OBJECT_SIZE (same parse, any command; empty = unset); + # flag beats env. No negotiation: each receiver + # applies its own cap; a larger object fails at + # whichever side receives it (#213) cargo run --bin sc -- serve token add --label --scope ro|rw [--expires <30m|12h|7d|2w|YYYY-MM-DD>] # mint an sct- bearer token in # .sc/serve-tokens.toml; raw value prints once on diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 22b4e23f..ab591297 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -139,7 +139,7 @@ a retired pre-P16 snapshot encoding, refused with a clear error): Objects are serialized canonically (length-prefixed, sorted tree entries) so the hash is stable across machines. Every decode length prefix is bounded by a -`MAX_OBJECT_SIZE` / remaining-bytes guard (P28) so a hostile object can't drive +max-object-size / remaining-bytes guard (P28) so a hostile object can't drive an unbounded allocation. ## In-memory VFS layer @@ -888,7 +888,7 @@ coarsening rather than per-case gap tolerance. See ADR-0037. ## Phase 28 — security hardening sweep (built) A dedicated security pass closing exposure introduced by opening the store -to untrusted network peers (P26/P27). One `MAX_OBJECT_SIZE` cap bounds any +to untrusted network peers (P26/P27). One max-object-size cap (`scl_core::max_object_size()`, ADR-0039 addendum) bounds any single object's decompressed/decoded size at every read/write boundary that takes attacker-controlled bytes, closing a decompression-bomb / unbounded- allocation class that P25's `read_frame_inner` had already named as a known @@ -943,8 +943,8 @@ limit a new connection gets an immediate busy status at the pre-handshake opening seam and closes — no queuing), `--timeout` (default 300s; read+write timeouts now persist for the whole session instead of being cleared after the opening, connection-fatal on trip since a mid-stream abort desyncs the frame -stream), and `--max-pack-size` (default 16 GiB, floor 256 MiB = -`MAX_OBJECT_SIZE`; lives in the shared `WirePolicy`, so it applies to both +stream), and `--max-pack-size` (default 16 GiB, floor = the max object +size, 256 MiB by default; lives in the shared `WirePolicy`, so it applies to both `--http` and `--stdio`; a counted mid-stream abort replies `EC_TOO_LARGE`). The read-only pre-drain is capped at 8 MiB (`RO_DRAIN_CAP`), and the accept diff --git a/CLAUDE.md b/CLAUDE.md index 497aee0c..47c06655 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -142,7 +142,7 @@ the code, those win. | P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) | | P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) | | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | -| P28 | Security hardening: `MAX_OBJECT_SIZE` (256 MiB) caps on every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | +| P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token`), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | | P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign/drop`, always-encrypted body, gc-coupled index; `--no-transcripts` on fetch/clone/push (client-side, per command); drop = grow-only tombstone (a bare `Transcript` object, no wire change), kept by gc, hides the body from fetch/reindex | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff | [0041](docs/adr/0041-listener-resource-limits.md) | @@ -152,7 +152,7 @@ the code, those win. | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | -| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | +| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 2 GiB)` (a constant since #213, independent of the object cap) and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | | P37d | Agent sessions on a partial clone: `sc ws fork`/`run`/`harvest`/`abandon` and `sc work` work; a workspace or agent sees the host's sparse view only (`sc work` no longer gets a complete tree there), the view closure is filled (lazy, online) or refused with the gap error before anything is written, a workspace commit grafts the out-of-view subtrees from the base without reading them, and harvest lands through the ADR-0049 merge (any conflict falls back to a `work-` branch as on a full clone) | [0050](docs/adr/0050-sessions-on-partial-clones.md) | @@ -163,9 +163,12 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in transport-adjacent. The rest, imperatively: - **Wire protocol is version 5** (bumped from 4 in P37 for the additive - `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the + `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB; + `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to + 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every - subsequent push/fetch/clone at the receiver. + subsequent push/fetch/clone at the receiver. There is no negotiation: each + receiver applies its own cap, so it must admit the objects it receives. Never inline a size in a new length guard; read the accessor. - **The P35 desktop app is a keyless, read-only top-level adapter.** Its renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or diff --git a/ROADMAP.md b/ROADMAP.md index 230deb65..a6acd996 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -800,11 +800,12 @@ scale-&-reach horizon): secrets to the command on a descriptor (NUL-terminated `NAME=VALUE` records) instead of its environment. Not built: the same mode for `sc ws run` and `sc work --with-secrets`, and a stdin variant. -- **`--max-object-size` operator config knob (P28).** `MAX_OBJECT_SIZE` - (256 MiB) is a compile-time constant anchoring every untrusted-length - guard; letting an operator raise or lower it per-deployment is deferred. +- **`--max-object-size` operator knob — shipped (#213, ADR-0039 addendum).** + `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap + (default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire + negotiation: each receiver applies its own cap. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run - `validate_branch_name`; `SC_PACK_CHUNK` is clamped to `MAX_OBJECT_SIZE - 1` (a chunk frame carries a tag byte too; an + `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the minimum configurable object cap minus one (2 MiB - 1, so every chunk frame fits any valid receiver) (a chunk frame carries a tag byte too; an oversized value used to make every transfer fail with a confusing receiver error); `Repo::worktree_paths` is a paths-only walk (`list_worktree_paths`, the same ignore and tracked-path rules as `read_worktree`, no file read); and diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 7d762df7..346835e3 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -364,10 +364,18 @@ enum Cmd { #[arg(long)] timeout: Option, /// Maximum incoming pack size in bytes (both --http and --stdio); - /// 0 = unlimited. Default 16 GiB. Must be ≥ 256 MiB (MAX_OBJECT_SIZE) - /// when non-zero. (P31) + /// 0 = unlimited. Default 16 GiB. Must be at least the max object + /// size (default 256 MiB) when non-zero. (P31) #[arg(long)] max_pack_size: Option, + /// Largest single object accepted from a peer: bytes, or a number + /// with a KiB/MiB/GiB suffix. Default 256MiB, range 2MiB to 1GiB; + /// out-of-range values are refused at startup, never clamped. Clients + /// set it with `SC_MAX_OBJECT_SIZE`. Each receiver applies its own + /// cap, so a larger object fails at whichever side is receiving it. + /// (#213) + #[arg(long, value_name = "SIZE")] + max_object_size: Option, /// Serve TLS (`sc+https://`): auto-mints a self-signed identity into /// `.sc/serve-tls/` unless --tls-cert/--tls-key supply a PEM pair. /// `--http` only. (P32) @@ -975,8 +983,31 @@ struct DemoArgs { checkout: bool, } +/// Fix the process-wide object cap before any command runs: `sc serve +/// --max-object-size` wins over `SC_MAX_OBJECT_SIZE` (empty = unset), which wins over the +/// default. An out-of-range or unparsable value aborts startup (#213). +fn init_max_object_size(flag: Option<&str>) -> Result<()> { + let (src, raw) = match (flag, std::env::var("SC_MAX_OBJECT_SIZE")) { + (Some(f), _) => ("--max-object-size", f.to_string()), + // An empty variable (`SC_MAX_OBJECT_SIZE=`) means unset, so exporting + // it blank in a profile does not abort every command. + (None, Ok(e)) if e.trim().is_empty() => return Ok(()), + (None, Ok(e)) => ("SC_MAX_OBJECT_SIZE", e), + (None, Err(std::env::VarError::NotPresent)) => return Ok(()), + (None, Err(e)) => anyhow::bail!("SC_MAX_OBJECT_SIZE: {e}"), + }; + let n = scl_core::parse_object_size(&raw).map_err(|e| anyhow::anyhow!("{src}: {e}"))?; + scl_core::set_max_object_size(n).map_err(|e| anyhow::anyhow!("{src}: {e}")) +} + fn main() -> Result<()> { let cli = Cli::parse(); + init_max_object_size(match &cli.cmd { + Cmd::Serve { + max_object_size, .. + } => max_object_size.as_deref(), + _ => None, + })?; match cli.cmd { Cmd::Demo(args) => run_demo(args), Cmd::Import { repo } => run_import(repo), @@ -1072,6 +1103,7 @@ fn main() -> Result<()> { max_connections, timeout, max_pack_size, + max_object_size: _, tls, tls_cert, tls_key, @@ -3836,7 +3868,7 @@ fn run_serve( ); } let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE); - scl_repo::wire::validate_max_pack_size(max_pack)?; + scl_repo::wire::validate_max_pack_size(max_pack, scl_core::max_object_size())?; let policy = scl_repo::wire::WirePolicy { read_only: false, max_pack_size: max_pack, diff --git a/crates/cli/tests/max_object_size.rs b/crates/cli/tests/max_object_size.rs new file mode 100644 index 00000000..c72a9a23 --- /dev/null +++ b/crates/cli/tests/max_object_size.rs @@ -0,0 +1,287 @@ +//! `SC_MAX_OBJECT_SIZE` / `sc serve --max-object-size` (#213, ADR-0039 +//! addendum): bounds are enforced at startup, and the cap is honoured by a +//! real object transfer. + +use std::io::BufRead; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Output, Stdio}; + +fn sc(dir: &Path, envs: &[(&str, &str)], args: &[&str]) -> Output { + let mut c = Command::new(env!("CARGO_BIN_EXE_sc")); + c.args(args) + .current_dir(dir) + .env_remove("SC_MAX_OBJECT_SIZE"); + for (k, v) in envs { + c.env(k, v); + } + c.output().expect("sc runs") +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-maxobj-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +/// A repo at `dir` with one committed `big.bin` of `len` incompressible bytes. +fn repo_with_big_file(dir: &Path, len: usize) -> Vec { + assert!(sc(dir, &[], &["init"]).status.success()); + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let big: Vec = (0..len) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + std::fs::write(dir.join("big.bin"), &big).unwrap(); + let o = sc(dir, &[("SC_AUTHOR", "t")], &["commit", "-m", "big"]); + assert!(o.status.success(), "commit is not capped: {}", stderr(&o)); + big +} + +#[test] +fn out_of_range_env_is_rejected_at_startup_not_clamped() { + let root = tmp("env-bounds"); + for bad in [ + "0", + "2097151", + "1MiB", + "2047KiB", + "1025MiB", + "2GiB", + "junk", + "99999999999999GiB", + ] { + let o = sc(&root, &[("SC_MAX_OBJECT_SIZE", bad)], &["init"]); + assert!(!o.status.success(), "{bad} must be rejected"); + assert!( + stderr(&o).contains("SC_MAX_OBJECT_SIZE"), + "{bad}: {}", + stderr(&o) + ); + assert!( + !root.join(".sc").exists(), + "{bad}: rejected before any work" + ); + } + for good in ["2097152", "2MiB", "256mib", "1GiB", "", " "] { + let o = sc(&root, &[("SC_MAX_OBJECT_SIZE", good)], &["init"]); + assert!(o.status.success(), "{good:?}: {}", stderr(&o)); + std::fs::remove_dir_all(root.join(".sc")).unwrap(); + } + std::fs::remove_dir_all(&root).unwrap(); +} + +#[test] +fn serve_flag_is_bounds_checked_and_overrides_env() { + let root = tmp("serve-flag"); + assert!(sc(&root, &[], &["init"]).status.success()); + for bad in ["512", "1MiB", "2GiB", "x"] { + let o = sc( + &root, + &[], + &["serve", "--stdio", "--max-object-size", bad, "."], + ); + assert!(!o.status.success(), "{bad} must be rejected"); + assert!( + stderr(&o).contains("--max-object-size"), + "{bad}: {}", + stderr(&o) + ); + } + // The flag wins over a bad env value (the env is never consulted); the + // pack-size floor then reports the flag's cap. + let o = sc( + &root, + &[("SC_MAX_OBJECT_SIZE", "junk")], + &[ + "serve", + "--stdio", + "--max-object-size", + "2MiB", + "--max-pack-size", + "1024", + ".", + ], + ); + assert!(!o.status.success()); + assert!( + stderr(&o).contains("below the max object size (2097152)"), + "the pack floor uses the flag's cap: {}", + stderr(&o) + ); + std::fs::remove_dir_all(&root).unwrap(); +} + +#[test] +fn serve_honours_the_env_when_the_flag_is_absent() { + let root = tmp("serve-env"); + assert!(sc(&root, &[], &["init"]).status.success()); + let o = sc( + &root, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["serve", "--stdio", "--max-pack-size", "1024", "."], + ); + assert!(!o.status.success()); + assert!( + stderr(&o).contains("below the max object size (4194304)"), + "the pack floor uses the env's cap: {}", + stderr(&o) + ); + std::fs::remove_dir_all(&root).unwrap(); +} + +#[test] +fn a_lowered_cap_refuses_a_transfer_a_raised_one_allows() { + let src = tmp("xfer-src"); + // 3 MiB: one object over a 2 MiB cap. + let big = repo_with_big_file(&src, 3 * 1024 * 1024); + + let parent = tmp("xfer-dst"); + let low = parent.join("low"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "2MiB")], + &["clone", src.to_str().unwrap(), low.to_str().unwrap()], + ); + assert!(!o.status.success(), "a 2 MiB cap must refuse a 3 MiB blob"); + assert!( + stderr(&o).contains("2097152-byte object size limit"), + "refused by the object cap, not something else: {}", + stderr(&o) + ); + + let high = parent.join("high"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["clone", src.to_str().unwrap(), high.to_str().unwrap()], + ); + assert!(o.status.success(), "a 4 MiB cap accepts it: {}", stderr(&o)); + assert_eq!(std::fs::read(high.join("big.bin")).unwrap(), big); + + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} + +/// Kills the server when dropped, so a failing assertion cannot leave it +/// holding the test's stderr open. +struct Server(Child); +impl Drop for Server { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn spawn_http_server(root: &Path, extra: &[&str]) -> (Server, String) { + let mut args = vec!["serve", "--http", "127.0.0.1:0"]; + args.extend_from_slice(extra); + args.push(root.to_str().unwrap()); + let mut child = Command::new(env!("CARGO_BIN_EXE_sc")) + .args(&args) + .env_remove("SC_MAX_OBJECT_SIZE") + .stdout(Stdio::piped()) + .spawn() + .expect("spawn sc serve --http"); + let mut reader = std::io::BufReader::new(child.stdout.take().unwrap()); + let mut line = String::new(); + assert!(reader.read_line(&mut line).unwrap() > 0, "server announced"); + let addr = line + .trim() + .strip_prefix("listening on ") + .unwrap_or_else(|| panic!("unexpected startup line: {line:?}")) + .to_string(); + // Keep draining so the server never blocks on a full stdout pipe. + std::thread::spawn(move || for _ in reader.lines() {}); + (Server(child), addr) +} + +#[test] +fn a_client_with_a_lower_cap_than_the_server_fails_with_the_mismatch_hint() { + let src = tmp("mismatch-src"); + let big = repo_with_big_file(&src, 3 * 1024 * 1024); + // The server runs at the default cap and serves the 3 MiB blob happily. + let (_server, addr) = spawn_http_server(&src, &[]); + let url = format!("sc+http://{addr}"); + + let parent = tmp("mismatch-dst"); + let low = parent.join("low"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "2MiB")], + &["clone", &url, low.to_str().unwrap()], + ); + assert!(!o.status.success(), "the lowered client must refuse"); + let err = stderr(&o); + assert!(err.contains("2097152-byte object size limit"), "{err}"); + assert!(err.contains("SC_MAX_OBJECT_SIZE"), "names the knob: {err}"); + + let ok = parent.join("ok"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["clone", &url, ok.to_str().unwrap()], + ); + assert!(o.status.success(), "an adequate cap passes: {}", stderr(&o)); + assert_eq!(std::fs::read(ok.join("big.bin")).unwrap(), big); + + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} + +#[test] +fn a_lower_cap_server_still_serves_a_by_id_fetch_to_a_higher_cap_client() { + // The cap is receiver-only: a server capped at 2 MiB that holds a + // locally committed 3 MiB blob serves it by id to a 4 MiB lazy client, + // as GetPack would. + let src = tmp("byid-src"); + assert!(sc(&src, &[], &["init"]).status.success()); + std::fs::create_dir_all(src.join("src")).unwrap(); + std::fs::create_dir_all(src.join("docs")).unwrap(); + std::fs::write(src.join("src/a.txt"), "small\n").unwrap(); + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let big: Vec = (0..3 * 1024 * 1024) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + std::fs::write(src.join("docs/big.bin"), &big).unwrap(); + let o = sc(&src, &[("SC_AUTHOR", "t")], &["commit", "-m", "c"]); + assert!(o.status.success(), "{}", stderr(&o)); + + let (_server, addr) = spawn_http_server(&src, &["--max-object-size", "2MiB"]); + let url = format!("sc+http://{addr}"); + let parent = tmp("byid-dst"); + let dst = parent.join("lazy"); + let envs = [("SC_MAX_OBJECT_SIZE", "4MiB")]; + let o = sc( + &parent, + &envs, + &[ + "clone", + &url, + dst.to_str().unwrap(), + "--filter", + "src/", + "--lazy", + ], + ); + assert!(o.status.success(), "filtered clone: {}", stderr(&o)); + let o = sc(&dst, &envs, &["cat", "main:docs/big.bin"]); + assert!(o.status.success(), "by-id fill: {}", stderr(&o)); + assert_eq!(o.stdout, big); + + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 719daef0..ed28b3e8 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -11,16 +11,154 @@ pub mod object; pub mod pack; pub mod store; -/// The largest single object (decoded/decompressed canonical bytes) this -/// system will accept from an untrusted source — a pack record's compressed -/// payload, its decompressed output, or a wire frame body. This is the -/// single anchor for every untrusted-length DoS guard added in P28: caps are -/// expressed relative to this constant rather than re-picked ad hoc at each -/// call site, so one number bounds "how big can one object get" everywhere. +/// Default for the largest single object (decoded/decompressed canonical +/// bytes) this system accepts from an untrusted source. The live value is +/// [`max_object_size`]; this is what it is when nothing configured it. /// 256 MiB comfortably exceeds any legitimate single object in this system /// (blobs, trees, snapshots, secrets) while still bounding a malicious /// peer's ability to force multi-GB allocations from a few header bytes. -pub const MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; // 256 MiB +pub const DEFAULT_MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; + +/// Hard floor for the operator-set object cap (2 MiB). A default peer sends +/// pack chunk frames of `CHUNK_SIZE` (1 MiB) plus a tag byte, which a +/// 1 MiB cap would refuse, so anything lower cannot talk to a default peer. +pub const MIN_MAX_OBJECT_SIZE: usize = 2 * 1024 * 1024; + +/// What to tell an operator whose receiver refused an object: there is no +/// negotiation, and each receiver applies its own cap to what it accepts. +pub const OBJECT_SIZE_HINT: &str = "only the receiving side's cap applies: raise SC_MAX_OBJECT_SIZE (or `sc serve --max-object-size` on a server) on the side that is refusing"; + +/// Hard ceiling for the operator-set object cap (1 GiB). Compile-time, so no +/// flag or env value can make a hostile peer's allocation claim unbounded. +pub const MAX_MAX_OBJECT_SIZE: usize = 1024 * 1024 * 1024; + +static MAX_OBJECT_SIZE: std::sync::OnceLock = std::sync::OnceLock::new(); + +/// The process-wide cap on one untrusted object (ADR-0039 addendum, #213): +/// a pack record's compressed payload and decompressed output, a wire frame +/// body, a by-id reply object, and the `SC_PACK_CHUNK` clamp. The single +/// source for every untrusted-length guard; never inline a number. Defaults +/// to [`DEFAULT_MAX_OBJECT_SIZE`]; the first call here or to +/// [`set_max_object_size`] fixes the value for the process. +pub fn max_object_size() -> usize { + // The first read fixes the value, so a later `set_max_object_size` cannot + // disagree with guards that already ran. + *MAX_OBJECT_SIZE.get_or_init(|| DEFAULT_MAX_OBJECT_SIZE) +} + +/// Set the process-wide object cap, once, at startup. Rejects (never clamps) +/// a value outside [`MIN_MAX_OBJECT_SIZE`]..=[`MAX_MAX_OBJECT_SIZE`]. Setting +/// the same value again is a no-op; a different value after one was fixed +/// is an error, since guards already running would disagree. +pub fn set_max_object_size(n: usize) -> std::result::Result<(), String> { + check_object_size_bounds(n)?; + let cur = *MAX_OBJECT_SIZE.get_or_init(|| n); + if cur == n { + Ok(()) + } else { + Err(format!("max object size is already fixed at {cur} bytes")) + } +} + +fn check_object_size_bounds(n: usize) -> std::result::Result<(), String> { + if (MIN_MAX_OBJECT_SIZE..=MAX_MAX_OBJECT_SIZE).contains(&n) { + Ok(()) + } else { + Err(format!( + "max object size {n} bytes is outside the allowed range {MIN_MAX_OBJECT_SIZE} (2 MiB) to {MAX_MAX_OBJECT_SIZE} (1 GiB)" + )) + } +} + +/// Parse an object-size setting: plain bytes, or a number with a `KiB`, +/// `MiB` or `GiB` suffix (case-insensitive). Applies the same hard bounds as +/// [`set_max_object_size`]; an out-of-range value is an error, not clamped. +pub fn parse_object_size(s: &str) -> std::result::Result { + let t = s.trim(); + let lower = t.to_ascii_lowercase(); + let (digits, mult) = if let Some(d) = lower.strip_suffix("kib") { + (d, 1usize << 10) + } else if let Some(d) = lower.strip_suffix("mib") { + (d, 1 << 20) + } else if let Some(d) = lower.strip_suffix("gib") { + (d, 1 << 30) + } else { + (lower.as_str(), 1) + }; + let n = digits + .trim() + .parse::() + .ok() + .and_then(|n| n.checked_mul(mult)) + .ok_or_else(|| { + format!("invalid object size {s:?}: expected bytes or a KiB/MiB/GiB suffix, e.g. 64MiB") + })?; + check_object_size_bounds(n)?; + Ok(n) +} + +/// Floor of a buffer growth step in the incremental readers below. Peak +/// over-allocation past the bytes actually received is at most +/// `max(READ_STEP, received / 8)`. +const READ_STEP: usize = 8 * 1024 * 1024; + +/// Size of the next growth step once `filled` bytes are in hand: geometric up +/// to [`READ_STEP`] (cheap for small objects), then `max(READ_STEP, filled/8)`, +/// so total copying stays linear even on an allocator that always copies on +/// growth, while over-allocation stays within 12.5% (or one `READ_STEP`). +fn next_step(filled: usize) -> usize { + if filled < READ_STEP { + filled.clamp(64 * 1024, READ_STEP) + } else { + READ_STEP.max(filled / 8) + } +} + +/// Read from `r` until EOF or `limit` bytes, whichever is first, growing the +/// buffer by `reserve_exact` steps (see [`next_step`]) as bytes arrive. +/// `first_step` sizes the first allocation (the caller's best guess). The +/// capacity never exceeds the bytes read by more than `max(8 MiB, 12.5%)`, so +/// a hostile length or decompression bomb cannot buy a `limit`-sized +/// allocation without supplying the bytes. +pub fn read_limited( + r: &mut impl std::io::Read, + limit: usize, + first_step: usize, +) -> std::io::Result> { + let mut buf: Vec = Vec::new(); + let mut filled = 0; + while filled < limit { + if filled == buf.len() { + let want = if filled == 0 { + first_step.clamp(1, READ_STEP) + } else { + next_step(filled) + }; + let step = want.min(limit - filled); + buf.reserve_exact(step); + buf.resize(filled + step, 0); + } + match r.read(&mut buf[filled..]) { + Ok(0) => break, + Ok(n) => filled += n, + Err(e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return Err(e), + } + } + buf.truncate(filled); + Ok(buf) +} + +/// Read exactly `len` bytes via [`read_limited`]: `len` is a peer's claim, so +/// the buffer follows the bytes that arrive. The caller still checks `len` +/// against [`max_object_size`] first. A short read is `UnexpectedEof`. +pub fn read_exact_vec(r: &mut impl std::io::Read, len: usize) -> std::io::Result> { + let buf = read_limited(r, len, len)?; + if buf.len() < len { + return Err(std::io::ErrorKind::UnexpectedEof.into()); + } + Ok(buf) +} pub use error::{Error, Result}; pub use id::ObjectId; @@ -30,3 +168,103 @@ pub use object::{ WrappedKey, PROTECTED, RANDOMIZED, }; pub use store::{Backend, SpillPolicy, Store, StoreConfig, StoreStats}; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_object_size_accepts_bytes_and_suffixes() { + assert_eq!(parse_object_size("2097152").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("2MiB").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("2048kib").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!( + parse_object_size(" 256 MiB ").unwrap(), + DEFAULT_MAX_OBJECT_SIZE + ); + assert_eq!(parse_object_size("1GiB").unwrap(), MAX_MAX_OBJECT_SIZE); + } + + #[test] + fn parse_object_size_rejects_out_of_range_and_garbage_without_clamping() { + for bad in [ + "0", + "2097151", + "2047KiB", + "1MiB", + "1025MiB", + "2GiB", + "18446744073709551615GiB", // multiplication overflow + "", + "MiB", + "-1", + "12XB", + "1.5GiB", + ] { + assert!(parse_object_size(bad).is_err(), "{bad:?} must be rejected"); + } + assert!(parse_object_size("2GiB") + .unwrap_err() + .contains("allowed range")); + } + + #[test] + fn set_max_object_size_rejects_out_of_range_values() { + assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); + assert!(set_max_object_size(MAX_MAX_OBJECT_SIZE + 1).is_err()); + // Rejection never fixes the setting, so the default still applies. + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + } + + #[test] + fn read_limited_never_over_allocates_by_more_than_one_step() { + // An endless reader (a decompression bomb's shape): stops at the + // limit, and the capacity is within one step of the bytes read. + let slack = |n: usize| READ_STEP.max(n / 8); + // Past the 12.5% regime (limit / 8 > READ_STEP). + let limit = 12 * READ_STEP + 12_345; + let v = read_limited(&mut std::io::repeat(0), limit, 4096).unwrap(); + assert_eq!(v.len(), limit); + assert!( + v.capacity() <= limit + slack(limit), + "cap {} for {limit}", + v.capacity() + ); + let limit = 3 * READ_STEP + 12_345; + let v = read_limited(&mut std::io::repeat(0), limit, 4096).unwrap(); + assert!(v.capacity() <= limit + slack(limit), "cap {}", v.capacity()); + // A claimed length with all bytes present: also within one step + // (in fact exact, since the last step is the remainder). + let src = vec![9u8; 2 * READ_STEP + 7]; + let v = read_exact_vec(&mut &src[..], src.len()).unwrap(); + assert_eq!(v, src); + assert!(v.capacity() <= src.len() + slack(src.len())); + // A small claim allocates about the claim, not a step. + let v = read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(); + assert!(v.capacity() < 1024); + } + + #[test] + fn read_limited_handles_short_reads_and_eof_before_the_limit() { + struct Dribble<'a>(&'a [u8]); + impl std::io::Read for Dribble<'_> { + fn read(&mut self, out: &mut [u8]) -> std::io::Result { + let n = self.0.len().min(out.len()).min(1000); + out[..n].copy_from_slice(&self.0[..n]); + self.0 = &self.0[n..]; + Ok(n) + } + } + let src: Vec = (0..(READ_STEP + 5000)).map(|i| i as u8).collect(); + let got = read_limited(&mut Dribble(&src), usize::MAX / 2, 10).unwrap(); + assert_eq!(got, src); + assert!(read_exact_vec(&mut Dribble(&src), src.len() + 1).is_err()); + } + + #[test] + fn read_exact_vec_reads_len_and_errors_on_short_input() { + assert_eq!(read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(), b"abcd"); + let err = read_exact_vec(&mut &b"abc"[..], 4).unwrap_err(); + assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof); + } +} diff --git a/crates/core/src/pack.rs b/crates/core/src/pack.rs index 87119bf3..386a2c04 100644 --- a/crates/core/src/pack.rs +++ b/crates/core/src/pack.rs @@ -204,7 +204,7 @@ pub fn parse_index(idx: &[u8]) -> Result> { /// untouched"). `Store::read_pack_object` is this function's one caller /// (`crates/core/src/store.rs`), reading packs `sc gc`'s own repack wrote to /// this process's `.sc/objects/`: `write_pack` performs no size validation, -/// so a locally-committed blob legitimately larger than `MAX_OBJECT_SIZE` +/// so a locally-committed blob legitimately larger than the object cap /// ("committable but not transferable", per ADR-0039's final-review /// addendum) must still round-trip through a repack, or `sc gc` would /// silently make that blob permanently unreadable. Do **not** add a length @@ -251,7 +251,8 @@ fn record_bounds(pack: &[u8], offset: u64, max: Option) -> Result<(usize, if let Some(max) = max { if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit" + "record compressed length {len} exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } } @@ -280,16 +281,14 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R Some(max) => { let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(payload)) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let mut canonical = Vec::new(); - decoder - .by_ref() - .take(max as u64 + 1) - .read_to_end(&mut canonical) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + let canonical = + crate::read_limited(&mut decoder, max + 1, payload.len().saturating_mul(4)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; if canonical.len() > max { - return Err(Error::PackCorrupt( - "decompressed object exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit".into(), - )); + return Err(Error::PackCorrupt(format!( + "decompressed object exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT + ))); } canonical } @@ -306,7 +305,7 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R /// every record. Test-convenience whole-pack helper only — its /// `zstd::decode_all` is UNBOUNDED, and every caller in this codebase is /// `#[cfg(test)]`. Untrusted or transport-received input MUST go through -/// [`parse_pack_reader`] instead, which enforces the `MAX_OBJECT_SIZE` cap +/// [`parse_pack_reader`] instead, which enforces the [`crate::max_object_size`] cap /// on both compressed length and decompressed output. pub fn parse_pack(pack: &[u8]) -> Result> { if pack.len() < 8 || &pack[..4] != PACK_MAGIC { @@ -376,7 +375,17 @@ fn read_up_to(r: &mut R, buf: &mut [u8]) -> std::io::Result { /// pack.len()`, generalized to a reader that has no total length to check /// against. pub fn parse_pack_reader( + r: R, + f: impl FnMut(ObjectId, Object) -> Result<()>, +) -> Result<()> { + parse_pack_reader_max(r, crate::max_object_size(), f) +} + +/// [`parse_pack_reader`] with an explicit per-object cap `max` (the live +/// [`crate::max_object_size`] in production). +pub fn parse_pack_reader_max( mut r: R, + max: usize, mut f: impl FnMut(ObjectId, Object) -> Result<()>, ) -> Result<()> { let mut header = [0u8; 8]; @@ -405,32 +414,34 @@ pub fn parse_pack_reader( r.read_exact(&mut len_bytes) .map_err(|e| Error::PackCorrupt(format!("truncated record length: {e}")))?; let len = u32::from_le_bytes(len_bytes) as usize; - if len > crate::MAX_OBJECT_SIZE { + if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit" + "record compressed length {len} exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } - let mut payload = vec![0u8; len]; - r.read_exact(&mut payload) + let payload = crate::read_exact_vec(&mut r, len) .map_err(|e| Error::PackCorrupt(format!("truncated record payload: {e}")))?; // Bound the decompressed output too: a small compressed payload can // still decompress to an enormous plaintext (a "zstd bomb"). Read at - // most MAX_OBJECT_SIZE + 1 bytes so we can detect and reject an + // most max + 1 bytes so we can detect and reject an // over-cap output without ever materializing it in full. - let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let mut canonical = Vec::new(); - decoder - .by_ref() - .take(crate::MAX_OBJECT_SIZE as u64 + 1) - .read_to_end(&mut canonical) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - if canonical.len() > crate::MAX_OBJECT_SIZE { - return Err(Error::PackCorrupt( - "decompressed object exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit".into(), - )); + let canonical = { + let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + crate::read_limited(&mut decoder, max + 1, payload.len().saturating_mul(4)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))? + }; + // Release the compressed copy before decoding, so peak per record is + // decompressed + decoded (about 2 x cap), not three copies. + drop(payload); + if canonical.len() > max { + return Err(Error::PackCorrupt(format!( + "decompressed object exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT + ))); } let actual_id = ObjectId::of(&canonical); if actual_id != expected_id { @@ -439,6 +450,7 @@ pub fn parse_pack_reader( ))); } let obj = Object::decode(&canonical)?; + drop(canonical); f(actual_id, obj)?; } } @@ -577,30 +589,64 @@ mod tests { #[test] fn pack_record_over_cap_rejected() { // A well-formed header, then one record whose id is arbitrary and - // whose compressed_len claims MAX_OBJECT_SIZE + 1 — parse_pack_reader - // must reject this before `vec![0u8; len]` allocates a 256+ MiB - // buffer for a length that was never actually sent. + // whose compressed_len claims cap + 1 — parse_pack_reader must + // reject this before allocating a buffer for a length that was never + // actually sent. let mut buf = Vec::new(); buf.extend_from_slice(PACK_MAGIC); buf.extend_from_slice(&FORMAT_VERSION.to_le_bytes()); buf.extend_from_slice(&[0u8; 32]); // arbitrary id - let over = (crate::MAX_OBJECT_SIZE + 1) as u32; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let over = (cap + 1) as u32; buf.extend_from_slice(&over.to_le_bytes()); // No payload bytes follow — if the implementation allocated first it // would then fail on read_exact anyway, but we want the cap check to // fire first (see zstd_bomb_rejected for the case where a payload // IS present but must still be rejected on the length check alone). - let err = parse_pack_reader(&buf[..], |_id, _obj| Ok(())).unwrap_err(); + let err = parse_pack_reader_max(&buf[..], cap, |_id, _obj| Ok(())).unwrap_err(); assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + // The same claim is not a length-cap error at a raised cap: it only + // fails later, on the (absent) payload bytes. + let err = parse_pack_reader_max(&buf[..], cap * 2, |_id, _obj| Ok(())).unwrap_err(); + assert!( + format!("{err}").contains("truncated record payload"), + "got {err:?}" + ); + } + + #[test] + fn a_legitimate_object_is_rejected_below_its_size_and_accepted_at_a_raised_cap() { + // Incompressible so the compressed record is also > 1 MiB. + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let payload: Vec = (0..crate::MIN_MAX_OBJECT_SIZE + 4096) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + let obj = Object::blob(payload); + let (pack, _idx) = build_pack(&[enc(&obj)]).unwrap(); + let low = crate::MIN_MAX_OBJECT_SIZE; + assert!(parse_pack_reader_max(&pack[..], low, |_, _| Ok(())).is_err()); + let mut n = 0; + parse_pack_reader_max(&pack[..], low * 4, |_, _| { + n += 1; + Ok(()) + }) + .unwrap(); + assert_eq!(n, 1); } #[test] fn zstd_bomb_rejected() { // A small compressed payload that decompresses to well beyond - // MAX_OBJECT_SIZE. parse_pack_reader must bound the zstd output + // the cap. parse_pack_reader must bound the zstd output // (decode-with-limit), not decode_all-then-check, or this test // would itself OOM the test process. - let bomb_plain = vec![0u8; crate::MAX_OBJECT_SIZE + 1024]; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let bomb_plain = vec![0u8; cap + 1024]; let compressed = zstd::encode_all(std::io::Cursor::new(&bomb_plain[..]), COMPRESSION_LEVEL).unwrap(); assert!( @@ -615,8 +661,15 @@ mod tests { buf.extend_from_slice(&(compressed.len() as u32).to_le_bytes()); buf.extend_from_slice(&compressed); - let err = parse_pack_reader(&buf[..], |_id, _obj| Ok(())).unwrap_err(); - assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + let err = parse_pack_reader_max(&buf[..], cap, |_id, _obj| Ok(())).unwrap_err(); + assert!( + format!("{err}").contains("decompressed object exceeds"), + "got {err:?}" + ); + // A raised cap lets the same record through the size guard (the + // all-zero payload then fails id verification, not the cap). + let err = parse_pack_reader_max(&buf[..], cap * 2, |_id, _obj| Ok(())).unwrap_err(); + assert!(format!("{err}").contains("hash mismatch"), "got {err:?}"); } #[test] @@ -628,10 +681,11 @@ mod tests { // untrusted input, unlike `read_object_at`'s one caller (`Store`'s // own self-written packs, ADR-0039's explicit unbounded carve-out). let id = ObjectId::of(b"whatever"); - let over = (crate::MAX_OBJECT_SIZE + 1) as u32; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let over = (cap + 1) as u32; let mut pack = Vec::new(); pack.extend_from_slice(&over.to_le_bytes()); // the length prefix read at offset 0 - let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); + let err = read_object_at_bounded(&pack, 0, &id, cap).unwrap_err(); assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); } @@ -644,7 +698,8 @@ mod tests { // trusted, unbounded path) deliberately does NOT get this guard — // see `read_object_at_reads_a_legitimately_oversized_local_record` // below for the regression that pins the other side of that split. - let bomb_plain = vec![0u8; crate::MAX_OBJECT_SIZE + 1024]; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let bomb_plain = vec![0u8; cap + 1024]; let compressed = zstd::encode_all(std::io::Cursor::new(&bomb_plain[..]), COMPRESSION_LEVEL).unwrap(); assert!( @@ -659,8 +714,11 @@ mod tests { pack.extend_from_slice(&(compressed.len() as u32).to_le_bytes()); pack.extend_from_slice(&compressed); - let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); - assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + let err = read_object_at_bounded(&pack, 0, &id, cap).unwrap_err(); + assert!( + format!("{err}").contains("decompressed object exceeds"), + "got {err:?}" + ); } #[test] @@ -668,7 +726,7 @@ mod tests { // ADR-0039's explicit scope: "the cap is transfer-path only, not a // local-commit limit... read_object_at's local already-verified // on-disk pack is untouched" — a locally-committed blob legitimately - // larger than MAX_OBJECT_SIZE ("committable but not transferable") + // larger than the object cap ("committable but not transferable") // must still round-trip through `sc gc`'s repack, since `write_pack` // performs no size validation and a cap here would surface as // silent, permanent unreadability on the next read. This is the @@ -676,7 +734,7 @@ mod tests { // briefly did in an earlier, incorrect fix for the zstd-bomb // finding), this test catches it directly, without relying only on // the doc-comment contract. - let big = Object::blob(vec![7u8; crate::MAX_OBJECT_SIZE + 4096]); + let big = Object::blob(vec![7u8; crate::DEFAULT_MAX_OBJECT_SIZE + 4096]); let (id, bytes) = enc(&big); let (pack, idx) = build_pack(&[(id, bytes.clone())]).unwrap(); let e = parse_index(&idx).unwrap().pop().unwrap(); diff --git a/crates/core/tests/max_object_size.rs b/crates/core/tests/max_object_size.rs new file mode 100644 index 00000000..398216a1 --- /dev/null +++ b/crates/core/tests/max_object_size.rs @@ -0,0 +1,23 @@ +//! The process-wide object cap is set once. Own test binaries (one global +//! each) so the unit tests keep the default. + +use scl_core::{ + max_object_size, set_max_object_size, DEFAULT_MAX_OBJECT_SIZE, MAX_MAX_OBJECT_SIZE, + MIN_MAX_OBJECT_SIZE, +}; + +#[test] +fn cap_is_set_once_validated_and_then_fixed() { + // Out-of-range values are refused and leave nothing fixed. + assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); + assert!(set_max_object_size(MAX_MAX_OBJECT_SIZE + 1).is_err()); + + set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); + assert_ne!(2 * MIN_MAX_OBJECT_SIZE, DEFAULT_MAX_OBJECT_SIZE); + assert_eq!(max_object_size(), 2 * MIN_MAX_OBJECT_SIZE); + // Same value again is a no-op; a different one is refused. + set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); + let err = set_max_object_size(MAX_MAX_OBJECT_SIZE).unwrap_err(); + assert!(err.contains("already fixed"), "got {err}"); + assert_eq!(max_object_size(), 2 * MIN_MAX_OBJECT_SIZE); +} diff --git a/crates/core/tests/max_object_size_read_first.rs b/crates/core/tests/max_object_size_read_first.rs new file mode 100644 index 00000000..f5211463 --- /dev/null +++ b/crates/core/tests/max_object_size_read_first.rs @@ -0,0 +1,16 @@ +//! Reading the cap fixes it: a guard that already ran with the default must +//! not be contradicted by a later `set`. + +use scl_core::{ + max_object_size, set_max_object_size, DEFAULT_MAX_OBJECT_SIZE, MIN_MAX_OBJECT_SIZE, +}; + +#[test] +fn a_read_before_set_fixes_the_default() { + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + let err = set_max_object_size(MIN_MAX_OBJECT_SIZE).unwrap_err(); + assert!(err.contains("already fixed"), "got {err}"); + // Setting the value already in force is fine. + set_max_object_size(DEFAULT_MAX_OBJECT_SIZE).unwrap(); + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); +} diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index 306928fd..3c7b3591 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -69,20 +69,29 @@ pub struct BucketTransport { } /// Untrusted-length guard (P28 parity): refuse any WAL metadata value — -/// manifest, log entry, idx — larger than `MAX_OBJECT_SIZE` before decoding. -/// Pack bodies are exempt from this particular guard (they may legitimately -/// exceed it), but are not unguarded: `object_bytes` reads a pack body via +/// manifest, log entry, idx — larger than the default 256 MiB before +/// decoding. Fixed at the default, not the operator's object cap: this is +/// metadata, not an object, and a lowered cap would make a bucket with a +/// large pack index unreadable (idx entries are 48 bytes each). The check runs +/// after the bucket GET has buffered the value, so unlike the frame and pack +/// guards it bounds decode work, not the download. Pack bodies are exempt +/// from this particular guard (they may legitimately exceed it), but are not +/// unguarded: `object_bytes` reads a pack body via /// `scl_core::pack::read_object_at_bounded` (not the unbounded /// `read_object_at` — that path is reserved for `Store`'s own /// already-verified on-disk packs per ADR-0039's explicit trust split), /// which caps both the compressed record length and the decompressed output -/// at `MAX_OBJECT_SIZE`, mirroring `parse_pack_reader`'s bounded decode. A +/// at the live object cap, mirroring `parse_pack_reader`'s bounded decode. A /// hostile bucket pack therefore cannot mount a decompression-bomb DoS /// against `get_object`/`get_pack`. fn capped(what: &str, bytes: Vec) -> Result> { - if bytes.len() > scl_core::MAX_OBJECT_SIZE { + capped_to(what, bytes, scl_core::DEFAULT_MAX_OBJECT_SIZE) +} + +fn capped_to(what: &str, bytes: Vec, max: usize) -> Result> { + if bytes.len() > max { return Err(Error::Wal(format!( - "{what} exceeds MAX_OBJECT_SIZE (256 MiB)" + "{what} exceeds the {max}-byte metadata limit" ))); } Ok(bytes) @@ -322,7 +331,7 @@ impl BucketTransport { *cache = Some((hash.clone(), bytes)); } let (_, pack) = cache.as_ref().unwrap(); - Ok(read_object_at_bounded(pack, offset, id, scl_core::MAX_OBJECT_SIZE)?.encode()) + Ok(read_object_at_bounded(pack, offset, id, scl_core::max_object_size())?.encode()) } /// Upload one pack (+ its idx) content-addressed by BLAKE3 of the pack @@ -663,6 +672,25 @@ impl Transport for BucketTransport { #[cfg(test)] mod tests { + #[test] + fn capped_metadata_guard_is_the_fixed_default_not_the_live_cap() { + // calloc'd zeros: no resident memory for the oversized value. + let at = vec![0u8; scl_core::DEFAULT_MAX_OBJECT_SIZE]; + assert!(capped("idx", at).is_ok()); + let over = vec![0u8; scl_core::DEFAULT_MAX_OBJECT_SIZE + 1]; + let err = capped("idx", over).unwrap_err(); + assert!(format!("{err}").contains("metadata limit"), "got {err}"); + } + + #[test] + fn capped_to_rejects_above_a_lowered_cap_and_accepts_at_a_raised_one() { + let v = vec![0u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; + let err = capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE).unwrap_err(); + assert!(format!("{err}").contains("metadata limit"), "got {err}"); + assert!(capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE + 1).is_ok()); + assert!(capped_to("manifest", v, scl_core::MAX_MAX_OBJECT_SIZE).is_ok()); + } + use super::*; use crate::transport::Transport; use crate::walfmt::{ diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index d9afbd84..318bcd8e 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -1018,7 +1018,7 @@ pub fn serve_http( tls: TlsMode, store: Option<&str>, ) -> Result<()> { - crate::wire::validate_max_pack_size(limits.max_pack_size)?; + crate::wire::validate_max_pack_size(limits.max_pack_size, scl_core::max_object_size())?; let tls_config = resolve_tls(root, &tls)?; if !bind_is_allowed(addr, root, read_only, allow_public, tls_config.is_some())? { return Err(Error::InvalidArgument(format!( diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index aac8aeec..9cf965f9 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -28,6 +28,10 @@ pub struct TransportSource<'a> { impl ObjectSource for TransportSource<'_> { fn get(&mut self, id: &ObjectId) -> Result { let bytes = self.transport.get_object(id)?; + // The receiver's cap, before decoding. (A local-path source has + // already loaded and encoded the object by now; wire sources refuse + // earlier, in `WireClient::get_object`.) + crate::wire::check_object_len(bytes.len(), scl_core::max_object_size())?; let obj = Object::decode(&bytes).map_err(Error::from)?; if obj.id() != *id { return Err(Error::CorruptObject(*id)); diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 23e08cc0..55e1951a 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -13,7 +13,19 @@ use crate::refs; use crate::worktree::{self, Diff}; const DEFAULT_BRANCH: &str = "main"; -pub(crate) const DEFAULT_BUDGET: usize = 512 * 1024 * 1024; +const DEFAULT_BUDGET: usize = 512 * 1024 * 1024; + +/// Resident-blob budget for a repo or transport store: 512 MiB, or twice the +/// object cap if that is larger, so one maximum-size object always fits (a +/// blob bigger than the whole budget would fail with `BudgetExceeded`). Equal +/// to the old fixed 512 MiB at the default 256 MiB cap. +pub(crate) fn default_budget() -> usize { + budget_for(scl_core::max_object_size()) +} + +fn budget_for(object_cap: usize) -> usize { + DEFAULT_BUDGET.max(2 * object_cap) +} /// A working-tree file headed for encryption at commit time: /// `(path, plaintext bytes, file mode, recipient pubkeys)`. @@ -44,17 +56,17 @@ impl Repo { std::fs::create_dir_all(layout.objects_dir())?; std::fs::create_dir_all(layout.refs_heads_dir())?; refs::write_head(&layout, DEFAULT_BRANCH)?; - Self::open_layout(layout, DEFAULT_BUDGET) + Self::open_layout(layout, default_budget()) } /// Open an existing repo by discovering `.sc/` at or above `start`. pub fn open(start: impl AsRef) -> Result { let layout = Layout::discover(start)?; - Self::open_layout(layout, DEFAULT_BUDGET) + Self::open_layout(layout, default_budget()) } /// Open an existing repo with an explicit memory budget for the shared - /// object cache (bytes). `open` uses `DEFAULT_BUDGET`; workspace sessions + /// object cache (bytes). `open` uses `default_budget()`; workspace sessions /// (`sc work --budget-mb`) size the cache to the fleet they fork. pub fn open_with_budget(start: impl AsRef, budget_bytes: usize) -> Result { let layout = Layout::discover(start)?; @@ -2705,6 +2717,20 @@ fn stop_write_blocker( #[cfg(test)] mod tests { + #[test] + fn store_budget_always_fits_one_maximum_size_object() { + // Unchanged at the default cap; grows with a raised cap. + assert_eq!(budget_for(scl_core::DEFAULT_MAX_OBJECT_SIZE), 512 << 20); + assert_eq!(budget_for(scl_core::MIN_MAX_OBJECT_SIZE), 512 << 20); + for cap in [ + scl_core::DEFAULT_MAX_OBJECT_SIZE, + scl_core::MAX_MAX_OBJECT_SIZE, + ] { + assert!(budget_for(cap) >= cap); + } + assert_eq!(budget_for(scl_core::MAX_MAX_OBJECT_SIZE), 2 << 30); + } + use super::*; fn tmp_root(tag: &str) -> std::path::PathBuf { diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 06a4c11a..89a54870 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -102,7 +102,10 @@ impl WireClient { // is already part-way through the stream, so report it as a protocol // violation that callers must not answer by splitting the batch. wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP).map_err(|e| match e { - Error::PackTooLarge(m) => Error::Protocol(format!("by-id reply too large: {m}")), + Error::PackTooLarge(_) => Error::Protocol(format!( + "by-id reply exceeded this client's {}-byte spool limit", + GET_OBJECTS_RESPONSE_CAP + )), other => other, })?; Ok(not_sent) @@ -120,7 +123,9 @@ impl Transport for WireClient { wire::decode_bool_body(&self.call(Request::HasObject(*id))?) } fn get_object(&self, id: &ObjectId) -> Result> { - self.call(Request::GetObject(*id)) + let body = self.call(Request::GetObject(*id))?; + wire::check_object_len(body.len(), scl_core::max_object_size())?; + Ok(body) } fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { self.call(Request::PutObject { @@ -195,8 +200,12 @@ impl Transport for WireClient { } } -/// Most bytes a client accepts in one `GetObjects` reply stream (1 GiB). -const GET_OBJECTS_RESPONSE_CAP: u64 = 1 << 30; +/// Most bytes a client accepts in one `GetObjects` reply stream: twice the +/// server's by-id reply ceiling, so pack framing never trips it on a +/// legitimate reply. A constant, not derived from the local object cap: it +/// bounds a disk spool, not RAM, and a cap that differed between the two ends +/// must not make a by-id batch with no large object fail. +const GET_OBJECTS_RESPONSE_CAP: u64 = 2 * crate::transport::BY_ID_REPLY_CEILING; /// A [`Transport`] whose far end is a child process speaking the wire protocol /// on its stdio — `ssh sc serve --stdio ` for real remotes. diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index c33a913e..cd5e3833 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -82,20 +82,24 @@ pub trait Transport { } } -/// The most canonical object bytes one by-id reply carries (ADR-0048). It sits -/// well under the client's 1 GiB read cap so pack record headers and zstd -/// framing cannot push a legitimate reply past it. -const BY_ID_REPLY_CAP: u64 = 512 << 20; +/// The most canonical object bytes one by-id reply carries (ADR-0048): twice +/// the largest object cap an operator can set, so a reply has room for a +/// maximum-size object plus framing at any setting. A constant, not derived +/// from the local object cap: it bounds a disk spool, not RAM, and ends with +/// different caps must not disagree about whether a by-id batch fits. A single +/// object over the largest configurable cap is still refused individually; the +/// serving process's own cap is not applied (it is receiver-only). +pub(crate) const BY_ID_REPLY_CEILING: u64 = 2 * scl_core::MAX_MAX_OBJECT_SIZE as u64; /// The by-id reply limit for a server whose `--max-pack-size` is /// `max_pack_size` (0 means unlimited): the smaller of that and -/// [`BY_ID_REPLY_CAP`]. Shared by the wire server and [`LocalTransport`], so a -/// lazy clone of a local path cannot spool an unbounded reply either. +/// [`BY_ID_REPLY_CEILING`]. Shared by the wire server and [`LocalTransport`], +/// so a lazy clone of a local path cannot spool an unbounded reply either. pub(crate) fn by_id_reply_cap(max_pack_size: u64) -> u64 { if max_pack_size == 0 { - BY_ID_REPLY_CAP + BY_ID_REPLY_CEILING } else { - max_pack_size.min(BY_ID_REPLY_CAP) + max_pack_size.min(BY_ID_REPLY_CEILING) } } @@ -110,12 +114,16 @@ fn encoded_len(obj: &Object) -> u64 { } } -/// Refuse an object a receiver would reject anyway: `pack::parse_pack_reader` -/// fails any record whose canonical encoding exceeds `MAX_OBJECT_SIZE`. -fn check_servable_size(id: &ObjectId, len: u64) -> Result<()> { - if len > scl_core::MAX_OBJECT_SIZE as u64 { +/// Refuse an object no receiver could accept: `pack::parse_pack_reader` fails +/// any record whose canonical encoding exceeds the receiver's cap, and no cap +/// exceeds `max` (callers pass [`scl_core::MAX_MAX_OBJECT_SIZE`]). The +/// serving process's own cap is deliberately not applied: the cap is +/// receiver-only, as it is for `GetPack`, so a server with a lower cap still +/// serves a locally committed larger object to a client whose cap admits it. +fn check_servable_size(id: &ObjectId, len: u64, max: usize) -> Result<()> { + if len > max as u64 { return Err(Error::PackTooLarge(format!( - "object {id} exceeds MAX_OBJECT_SIZE and cannot be transferred" + "object {id} exceeds the {max}-byte object size limit and cannot be transferred" ))); } Ok(()) @@ -148,7 +156,7 @@ impl LocalTransport { } // Match the repo's store budget so a single large object never fails to resolve // (a blob bigger than the whole budget would BudgetExceed). - let store = Store::open_persistent(layout.objects_dir(), crate::repo::DEFAULT_BUDGET)?; + let store = Store::open_persistent(layout.objects_dir(), crate::repo::default_budget())?; Ok(LocalTransport { layout, store: RefCell::new(store), @@ -160,8 +168,8 @@ impl LocalTransport { /// shared core of [`Transport::get_objects`] and `wire::serve`'s /// `GetObjects` arm. Duplicate ids are served once. `max_bytes` (0 = /// unlimited) bounds the summed canonical encoded size, and any single - /// object over `MAX_OBJECT_SIZE` is refused (every receiver would reject - /// it). Both fail with `PackTooLarge` before any pack byte is written. + /// object over the largest configurable cap is refused (no receiver + /// could accept it; the server's own cap is not applied). Both fail with `PackTooLarge` before any pack byte is written. /// Blobs and sealed objects are measured without re-encoding them. pub(crate) fn build_objects_tempfile( &self, @@ -182,7 +190,7 @@ impl LocalTransport { Ok(obj) if refused_by_id(&obj) => not_sent.push(*id), Ok(obj) => { let size = encoded_len(&obj); - check_servable_size(id, size)?; + check_servable_size(id, size, scl_core::MAX_MAX_OBJECT_SIZE)?; total = total.saturating_add(size); if max_bytes > 0 && total > max_bytes { return Err(Error::PackTooLarge(format!( @@ -408,7 +416,7 @@ pub(crate) fn write_ids_to_temp_pack( /// Two defenses now compose here (updated for P28 Task 2). First, /// `parse_pack_reader` itself hard-rejects any record whose compressed /// length prefix, or whose decompressed zstd output, exceeds -/// `MAX_OBJECT_SIZE` (256 MiB) — an absolute per-record cap enforced +/// the object cap (`scl_core::max_object_size`, default 256 MiB) — an absolute per-record cap enforced /// regardless of source. Second, and independently, both passes here read /// the *already-fully-spilled* temp file rather than the live connection /// (whether it was spilled by `Transport::put_pack`'s own bounded copy from @@ -416,7 +424,7 @@ pub(crate) fn write_ids_to_temp_pack( /// total bytes any record's length prefix can possibly claim is also /// bounded by the file's own size on disk, which was itself bounded by /// however many bytes the sender actually sent. So a hostile length prefix -/// is now rejected outright by the `MAX_OBJECT_SIZE` cap, and even absent +/// is now rejected outright by the object cap, and even absent /// that cap could never claim more than the attacker already transferred — /// both defenses hold. Every call site in this codebase (both here and /// `wire::serve`'s own dispatch) reads a spilled file, never the live @@ -766,14 +774,19 @@ mod tests { assert_eq!(encoded_len(o), o.encode().len() as u64, "{}", o.kind_name()); } let id = blob.id(); - // The receiver rejects a canonical encoding above MAX_OBJECT_SIZE, and - // the encoding of a blob is its payload plus one tag byte. - let max = scl_core::MAX_OBJECT_SIZE as u64; - assert!(check_servable_size(&id, max).is_ok()); - assert!(matches!( - check_servable_size(&id, max + 1), - Err(Error::PackTooLarge(_)) - )); + // `max` is the boundary passed in; production passes the largest + // configurable cap (the serving process's own cap is receiver-only). + for max in [ + scl_core::MIN_MAX_OBJECT_SIZE, + scl_core::DEFAULT_MAX_OBJECT_SIZE, + scl_core::MAX_MAX_OBJECT_SIZE, + ] { + assert!(check_servable_size(&id, max as u64, max).is_ok()); + assert!(matches!( + check_servable_size(&id, max as u64 + 1, max), + Err(Error::PackTooLarge(_)) + )); + } assert_eq!(encoded_len(&Object::blob(vec![0u8; 10])), 11); } @@ -920,9 +933,11 @@ mod tests { let id = blob.id(); let expected = blob.encode(); { - let mut s = - scl_core::Store::open_persistent(layout.objects_dir(), crate::repo::DEFAULT_BUDGET) - .unwrap(); + let mut s = scl_core::Store::open_persistent( + layout.objects_dir(), + crate::repo::default_budget(), + ) + .unwrap(); s.put(Object::blob(vec![0xAB; (1 << 20) + 4096])).unwrap(); } let t = LocalTransport::open(&layout.root).unwrap(); @@ -1145,13 +1160,13 @@ mod tests { } #[test] - fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_512_mib() { + fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_a_fixed_ceiling() { const MIB: u64 = 1 << 20; // 0 means "unlimited" for --max-pack-size, but a by-id reply is still - // bounded so the client's read cap always covers it. - assert_eq!(by_id_reply_cap(0), 512 * MIB); + // bounded, and the bound does not depend on the local object cap. + assert_eq!(by_id_reply_cap(0), 2048 * MIB); assert_eq!(by_id_reply_cap(64 * MIB), 64 * MIB); - assert_eq!(by_id_reply_cap(16 << 30), 512 * MIB); + assert_eq!(by_id_reply_cap(16 << 30), 2048 * MIB); } /// A repo with a stopped merge, pick or rebase is completed against its diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index 53a744fb..52071366 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -81,17 +81,19 @@ pub fn pack_chunk_size() -> usize { /// The `SC_PACK_CHUNK` decision, pure so it can be tested without touching the /// process environment: an unset, unparsable or zero value is the default, and -/// anything larger than a frame can carry is clamped. A chunk frame is the -/// `ST_PACK_CHUNK` tag byte plus the data, and the receiver refuses a frame -/// above `MAX_OBJECT_SIZE`, so the data is capped one byte below it; an -/// oversized setting would otherwise fail every large transfer with a -/// confusing error. +/// anything larger than the smallest valid receiver accepts is clamped. Caps +/// are receiver-local and unnegotiated, and the smallest a receiver can be set +/// to is [`scl_core::MIN_MAX_OBJECT_SIZE`], so the data is capped one byte +/// below that (the `ST_PACK_CHUNK` tag byte makes the frame exactly that size). +/// Every chunk frame therefore fits any valid receiver, and receivers built +/// before the frame limit gained [`FRAME_OVERHEAD`]; an oversized setting would +/// otherwise fail every large transfer with a confusing error. fn chunk_from(value: Option<&str>) -> usize { value .and_then(|v| v.parse::().ok()) .filter(|&n| n > 0) .unwrap_or(CHUNK_SIZE) - .min(scl_core::MAX_OBJECT_SIZE - 1) + .min(scl_core::MIN_MAX_OBJECT_SIZE - 1) } // Wire error codes. Typed errors that sync logic relies on get their own code; @@ -370,6 +372,23 @@ impl Request { c.done()?; Ok(req) } + + /// [`Request::decode`] for a frame the caller owns. A `PutObject` object + /// body is moved out of the frame (the header is shifted off in place) + /// instead of copied, so a near-cap request holds one copy of the object, + /// not two. Anything malformed falls through to `decode` for its error. + pub fn decode_owned(mut frame: Vec) -> Result { + const HEADER: usize = 1 + 32 + 4; + if frame.len() >= HEADER && frame[0] == OP_PUT_OBJECT { + let n = u32::from_be_bytes(frame[33..HEADER].try_into().unwrap()) as usize; + if frame.len() - HEADER == n { + let id = ObjectId::from_bytes(frame[1..33].try_into().unwrap()); + frame.drain(..HEADER); + return Ok(Request::PutObject { id, bytes: frame }); + } + } + Request::decode(&frame) + } } // --- frame IO --- @@ -398,6 +417,32 @@ pub fn read_frame_opt(r: &mut impl Read) -> Result>> { } fn read_frame_inner(r: &mut impl Read) -> Result>> { + read_frame_max(r, scl_core::max_object_size()) +} + +/// Room a frame has beyond the object it carries: the longest header is a +/// `PutObject` request (opcode + 32-byte id + 4-byte length = 37 bytes); a +/// response adds a status byte and a pack chunk a tag byte. Frames may be +/// `cap + FRAME_OVERHEAD` long so an object of exactly `cap` bytes still +/// fits; the object itself is checked against `cap` by [`check_object_len`]. +pub const FRAME_OVERHEAD: usize = 64; + +/// Reject an object body above the cap `max`. Frames carry [`FRAME_OVERHEAD`] +/// extra bytes, so a single-object payload gets this separate exact check. +pub(crate) fn check_object_len(len: usize, max: usize) -> Result<()> { + if len > max { + return Err(Error::Protocol(format!( + "object of {len} bytes exceeds the {max}-byte object size limit ({})", + scl_core::OBJECT_SIZE_HINT + ))); + } + Ok(()) +} + +/// [`read_frame_opt`] with an explicit object cap `max` (the live +/// [`scl_core::max_object_size`] in production); the frame limit is +/// `max + FRAME_OVERHEAD`. +fn read_frame_max(r: &mut impl Read, max: usize) -> Result>> { let mut len_bytes = [0u8; 4]; let mut filled = 0; while filled < 4 { @@ -411,13 +456,15 @@ fn read_frame_inner(r: &mut impl Read) -> Result>> { filled += n; } let len = u32::from_be_bytes(len_bytes) as usize; - if len > scl_core::MAX_OBJECT_SIZE { + if len > max + FRAME_OVERHEAD { return Err(Error::Protocol(format!( - "frame length {len} exceeds MAX_OBJECT_SIZE" + "frame length {len} exceeds the {max}-byte object size limit plus framing ({})", + scl_core::OBJECT_SIZE_HINT ))); } - let mut buf = vec![0u8; len]; - r.read_exact(&mut buf).map_err(|e| match e.kind() { + // The length is the peer's claim: grow with the bytes that actually + // arrive instead of allocating `len` up front. + let buf = scl_core::read_exact_vec(r, len).map_err(|e| match e.kind() { // A read timeout is the peer going quiet, not a malformed frame. std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut => Error::Io(e), _ => Error::Protocol(format!("read inside {len}-byte frame body failed: {e}")), @@ -530,14 +577,18 @@ pub fn write_err(w: &mut impl Write, code: u8, msg: &str) -> Result<()> { } /// Split a response frame into its OK body, or reconstruct the typed error. -pub fn parse_response(frame: Vec) -> Result> { - let (&status, rest) = frame - .split_first() +pub fn parse_response(mut frame: Vec) -> Result> { + let status = *frame + .first() .ok_or_else(|| Error::Protocol("empty response frame".into()))?; match status { - ST_OK => Ok(rest.to_vec()), + ST_OK => { + // Strip the status byte in place: no second copy of a large body. + frame.remove(0); + Ok(frame) + } ST_ERR => { - let mut c = Cur::new(rest); + let mut c = Cur::new(&frame[1..]); let code = c.u8()?; let msg = c.str()?; c.done()?; @@ -729,14 +780,14 @@ impl Default for WirePolicy { } /// Validate an operator-supplied `--max-pack-size` value. `0` (unlimited) is -/// always fine; a nonzero cap below [`scl_core::MAX_OBJECT_SIZE`] can never -/// fit even one object and is rejected as a misconfiguration rather than -/// silently failing every push at runtime. -pub fn validate_max_pack_size(max: u64) -> Result<()> { - if max != 0 && max < scl_core::MAX_OBJECT_SIZE as u64 { +/// always fine; a nonzero cap below `object_cap` (the live +/// [`scl_core::max_object_size`]) can never fit even one object and is +/// rejected as a misconfiguration rather than silently failing every push at +/// runtime. +pub fn validate_max_pack_size(max: u64, object_cap: usize) -> Result<()> { + if max != 0 && max < object_cap as u64 { return Err(Error::InvalidArgument(format!( - "--max-pack-size {max} is below MAX_OBJECT_SIZE ({}); a cap that cannot fit one object is a misconfiguration", - scl_core::MAX_OBJECT_SIZE + "--max-pack-size {max} is below the max object size ({object_cap}); a cap that cannot fit one object is a misconfiguration" ))); } Ok(()) @@ -987,7 +1038,7 @@ fn serve_session( Some(f) => f, None => return Ok(()), // peer hung up between requests }; - let req = match Request::decode(&frame) { + let req = match Request::decode_owned(frame) { Ok(req) => req, Err(e) => { let (code, msg) = err_to_wire(&e); @@ -1129,21 +1180,18 @@ fn serve_session( // `guard` drops here — temp pack file removed, // whether ingestion succeeded or failed. } - Err(e @ Error::PackTooLarge(_)) => { - // Over the cap: the stream is desynced (we stopped - // reading mid-pack, so any bytes still on the wire - // are neither read nor a subsequent request), so a - // best-effort reply is the most we can do — then - // close the connection rather than try to serve - // further requests off a desynced stream. + Err(e) => { + // Any spill failure (over the cap, a malformed or + // over-cap frame, I/O): we stopped reading mid-pack, + // so any bytes still on the wire are neither read + // nor a subsequent request. A best-effort reply is + // the most we can do; then close the connection + // rather than serve further requests off a + // desynced stream. let (code, msg) = err_to_wire(&e); let _ = write_err(w, code, &msg); return Ok(()); } - Err(e) => { - let (code, msg) = err_to_wire(&e); - write_err(w, code, &msg)?; - } } } other => { @@ -1162,10 +1210,14 @@ fn serve_session( transport.as_transport().has_object(&id).map(bool_body) } Request::GetObject(id) => transport.as_transport().get_object(&id), - Request::PutObject { id, bytes } => transport - .as_transport() - .put_object(&id, &bytes) - .map(|()| Vec::new()), + Request::PutObject { id, bytes } => { + check_object_len(bytes.len(), scl_core::max_object_size()).and_then(|()| { + transport + .as_transport() + .put_object(&id, &bytes) + .map(|()| Vec::new()) + }) + } Request::UpdateRef { branch, id, @@ -1387,16 +1439,113 @@ mod tests { #[test] fn frame_over_cap_rejected() { - // A frame header alone (no body) claiming a length past - // MAX_OBJECT_SIZE must be rejected before any allocation of the - // (fictitious) body — feeding only the 4-byte header proves this: - // if the implementation tried to `vec![0u8; len]` and then - // `read_exact`, it would hang/OOM here rather than erroring. - let over = (scl_core::MAX_OBJECT_SIZE + 1) as u32; + // A frame header alone (no body) claiming a length past the cap plus + // framing must be rejected before the (fictitious) body is read or + // allocated: feeding only the 4-byte header proves this, since + // reading first would hit EOF and report a different error. + let cap = scl_core::MIN_MAX_OBJECT_SIZE; + let over = (cap + FRAME_OVERHEAD + 1) as u32; let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); - let err = read_frame_opt(&mut r).unwrap_err(); + let err = read_frame_max(&mut r, cap).unwrap_err(); + assert!( + matches!(&err, crate::error::Error::Protocol(m) if m.contains("exceeds")), + "got {err:?}" + ); + // The production entry point is the same guard at the live cap. + let over = (scl_core::max_object_size() + FRAME_OVERHEAD + 1) as u32; + let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); + assert!(matches!( + read_frame_opt(&mut r), + Err(crate::error::Error::Protocol(m)) if m.contains("exceeds") + )); + } + + #[test] + fn frame_cap_lowered_rejects_and_raised_accepts_the_same_frame() { + let low = scl_core::MIN_MAX_OBJECT_SIZE; + let body = vec![7u8; low + FRAME_OVERHEAD + 1]; + let mut framed = Vec::new(); + write_frame(&mut framed, &body).unwrap(); + assert!(read_frame_max(&mut &framed[..], low).is_err()); + assert_eq!( + read_frame_max(&mut &framed[..], low + 1).unwrap().unwrap(), + body + ); + assert_eq!( + read_frame_max(&mut &framed[..], low * 4).unwrap().unwrap(), + body + ); + } + + /// An object of exactly the cap travels in `PutObject` (37 header bytes) + /// and `GetObject` (1 status byte) frames; one byte more is refused by + /// the exact object check even though the frame itself still fits. + #[test] + fn object_boundary_exactly_cap_accepted_cap_plus_one_rejected() { + let cap = scl_core::MIN_MAX_OBJECT_SIZE; + let id = some_id(7); + for (len, ok) in [(cap, true), (cap + 1, false)] { + let bytes = vec![5u8; len]; + // PutObject request. + let mut framed = Vec::new(); + write_frame( + &mut framed, + &Request::PutObject { + id, + bytes: bytes.clone(), + } + .encode(), + ) + .unwrap(); + let frame = read_frame_max(&mut &framed[..], cap).unwrap().unwrap(); + let Request::PutObject { bytes: got, .. } = Request::decode_owned(frame).unwrap() + else { + panic!("not a PutObject"); + }; + assert_eq!(got, bytes); + assert_eq!(check_object_len(got.len(), cap).is_ok(), ok, "put {len}"); + // GetObject response. + let mut framed = Vec::new(); + write_ok(&mut framed, &bytes).unwrap(); + let frame = read_frame_max(&mut &framed[..], cap).unwrap().unwrap(); + let body = parse_response(frame).unwrap(); + assert_eq!(body, bytes); + assert_eq!(check_object_len(body.len(), cap).is_ok(), ok, "get {len}"); + } + } + + #[test] + fn decode_owned_matches_decode_and_handles_malformed_put_object() { + let id = some_id(3); + let enc = Request::PutObject { + id, + bytes: b"hello".to_vec(), + } + .encode(); + assert_eq!( + Request::decode_owned(enc.clone()).unwrap(), + Request::decode(&enc).unwrap() + ); + // Declared length disagrees with the frame: same error as decode. + let mut bad = enc.clone(); + bad.push(0); + assert!(Request::decode_owned(bad).is_err()); + let mut short = enc; + short.truncate(10); + assert!(Request::decode_owned(short).is_err()); + } + + #[test] + fn frame_claiming_the_ceiling_without_a_body_is_a_short_read_error() { + // At the ceiling, a 4-byte header can claim 1 GiB and passes the + // length guard; with no body behind it the read fails as a short + // frame. (That no cap-sized buffer is allocated is pinned by + // `read_limited_never_over_allocates_by_more_than_one_step`.) + let cap = scl_core::MAX_MAX_OBJECT_SIZE; + let mut r = std::io::Cursor::new((cap as u32).to_be_bytes().to_vec()); + let err = read_frame_max(&mut r, cap).unwrap_err(); assert!( - matches!(err, crate::error::Error::Protocol(_)), + matches!(&err, crate::error::Error::Protocol(m) if m.contains("frame body")), "got {err:?}" ); } @@ -1875,32 +2024,39 @@ mod tests { } #[test] - fn chunk_from_defaults_and_clamps_to_the_object_cap() { + fn chunk_from_defaults_and_clamps_to_the_smallest_valid_receiver_cap() { + let min = scl_core::MIN_MAX_OBJECT_SIZE; assert_eq!(chunk_from(None), CHUNK_SIZE); assert_eq!(chunk_from(Some("")), CHUNK_SIZE); assert_eq!(chunk_from(Some("0")), CHUNK_SIZE); assert_eq!(chunk_from(Some("junk")), CHUNK_SIZE); assert_eq!(chunk_from(Some("4096")), 4096); - // A chunk frame carries a tag byte too: data at the cap would make a - // frame one byte over what the receiver accepts. - let cap = scl_core::MAX_OBJECT_SIZE - 1; - assert_eq!( - chunk_from(Some(&(scl_core::MAX_OBJECT_SIZE * 8).to_string())), - cap, - "an oversized setting is clamped, not passed through" - ); - assert_eq!( - chunk_from(Some(&scl_core::MAX_OBJECT_SIZE.to_string())), - cap - ); - assert_eq!(chunk_from(Some(&cap.to_string())), cap); + // Independent of this process's own cap: a sender raised to 1 GiB + // must still produce frames the smallest valid receiver accepts. + for v in [min, min * 8, scl_core::MAX_MAX_OBJECT_SIZE, usize::MAX] { + assert_eq!(chunk_from(Some(&v.to_string())), min - 1, "{v}"); + } + // The resulting frame (tag byte + data) is exactly the minimum cap and + // is accepted by a receiver at that cap, even with no overhead slack. + let mut frame = vec![ST_PACK_CHUNK]; + frame.extend(vec![0u8; min - 1]); + let mut framed = Vec::new(); + write_frame(&mut framed, &frame).unwrap(); + assert!(read_frame_max(&mut &framed[..], min).unwrap().is_some()); + assert!(CHUNK_SIZE < min, "the default chunk fits every receiver"); } #[test] fn validate_max_pack_size_floor() { - assert!(validate_max_pack_size(0).is_ok()); - assert!(validate_max_pack_size(scl_core::MAX_OBJECT_SIZE as u64).is_ok()); - assert!(validate_max_pack_size(1024).is_err()); + let max = scl_core::DEFAULT_MAX_OBJECT_SIZE; + assert!(validate_max_pack_size(0, max).is_ok()); + assert!(validate_max_pack_size(max as u64, max).is_ok()); + assert!(validate_max_pack_size(1024, max).is_err()); + // The floor tracks the object cap: lowered admits what a default + // cap refuses, raised refuses what a default cap admits. + let low = scl_core::MIN_MAX_OBJECT_SIZE; + assert!(validate_max_pack_size(low as u64, low).is_ok()); + assert!(validate_max_pack_size(max as u64, scl_core::MAX_MAX_OBJECT_SIZE).is_err()); } /// An oversized push on a read-only connection drains at most ro_drain_cap @@ -1946,6 +2102,39 @@ mod tests { let _ = std::fs::remove_dir_all(&root); } + /// A malformed frame inside a pack stream desyncs the connection: the + /// server replies once and closes instead of answering what follows. + #[test] + fn put_pack_protocol_error_closes_the_connection() { + let root = tmp_repo("putpack-proto"); + let mut input = Vec::new(); + for req in [ + Request::Hello { + version: PROTOCOL_VERSION, + }, + Request::PutPack, + ] { + write_frame(&mut input, &req.encode()).unwrap(); + } + write_frame(&mut input, &[0x7f]).unwrap(); // not a pack chunk or end + write_frame(&mut input, &Request::HeadBranch.encode()).unwrap(); // must never be answered + + let mut output = Vec::new(); + serve(&root, &mut std::io::Cursor::new(input), &mut output).unwrap(); + + let mut frames = Vec::new(); + let mut r = std::io::Cursor::new(output); + while let Some(f) = read_frame_opt(&mut r).unwrap() { + frames.push(parse_response(f)); + } + assert_eq!(frames.len(), 2, "hello ok, one error, then closed"); + assert!(matches!( + frames[1].as_ref().unwrap_err(), + Error::Protocol(_) + )); + let _ = std::fs::remove_dir_all(&root); + } + /// `GetObjects` is a read verb: a read-only session serves it, and the /// `max_pack_size` spool cap applies to its reply. #[test] diff --git a/crates/repo/tests/max_object_size_wire.rs b/crates/repo/tests/max_object_size_wire.rs new file mode 100644 index 00000000..d6eb4603 --- /dev/null +++ b/crates/repo/tests/max_object_size_wire.rs @@ -0,0 +1,82 @@ +//! The object cap is wired into the live wire paths (#232): a `PutObject` +//! through `serve` and a `GetObject` through `WireClient`, with the cap fixed +//! at 2 MiB before the first read. Own test binary: the cap is process-global. + +use scl_core::{Object, MIN_MAX_OBJECT_SIZE}; +use scl_repo::stdio_transport::WireClient; +use scl_repo::transport::Transport; +use scl_repo::wire::{self, Request}; + +fn set_cap() { + scl_core::set_max_object_size(MIN_MAX_OBJECT_SIZE).unwrap(); +} + +fn setup() -> std::path::PathBuf { + set_cap(); + let root = std::env::temp_dir().join(format!("scl-wire-cap-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + scl_repo::Repo::init(&root).unwrap(); + root +} + +/// Canonical encoding of a blob is its payload plus one tag byte. +fn blob_of_canonical_len(n: usize) -> Object { + Object::blob(vec![0xAB; n - 1]) +} + +fn put_via_serve(root: &std::path::Path, obj: &Object) -> Vec, scl_repo::Error>> { + let mut input = Vec::new(); + for req in [ + Request::Hello { + version: wire::PROTOCOL_VERSION, + }, + Request::PutObject { + id: obj.id(), + bytes: obj.encode(), + }, + ] { + wire::write_frame(&mut input, &req.encode()).unwrap(); + } + let mut output = Vec::new(); + wire::serve(root, &mut std::io::Cursor::new(input), &mut output).unwrap(); + let mut r = std::io::Cursor::new(output); + let mut out = Vec::new(); + while let Some(f) = wire::read_frame_opt(&mut r).unwrap() { + out.push(wire::parse_response(f)); + } + out +} + +#[test] +fn serve_put_object_exactly_cap_lands_and_cap_plus_one_is_refused() { + let root = setup(); + let ok = put_via_serve(&root, &blob_of_canonical_len(MIN_MAX_OBJECT_SIZE)); + assert!(ok[1].is_ok(), "exactly the cap lands: {:?}", ok[1]); + + let over = put_via_serve(&root, &blob_of_canonical_len(MIN_MAX_OBJECT_SIZE + 1)); + let err = over[1].as_ref().unwrap_err().to_string(); + assert!(err.contains("object size limit"), "{err}"); + let _ = std::fs::remove_dir_all(&root); +} + +#[test] +fn client_get_object_enforces_the_cap_on_the_response() { + set_cap(); + for (len, ok) in [ + (MIN_MAX_OBJECT_SIZE, true), + (MIN_MAX_OBJECT_SIZE + 1, false), + ] { + let obj = blob_of_canonical_len(len); + // Canned server: HELLO ok, then the GetObject response. + let mut canned = Vec::new(); + wire::write_ok(&mut canned, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + wire::write_ok(&mut canned, &obj.encode()).unwrap(); + let client = WireClient::handshake(std::io::Cursor::new(canned), std::io::sink()).unwrap(); + let got = client.get_object(&obj.id()); + assert_eq!(got.is_ok(), ok, "len {len}: {got:?}"); + if let Err(e) = got { + assert!(e.to_string().contains("object size limit"), "{e}"); + } + } +} diff --git a/demo/run_limits_demo.sh b/demo/run_limits_demo.sh index bde0259e..a48ec9cd 100755 --- a/demo/run_limits_demo.sh +++ b/demo/run_limits_demo.sh @@ -2,7 +2,7 @@ # P31 listener-limits demo: `sc serve --http`'s three new operator knobs — # `--max-connections` (503-style busy shed at the limit), `--timeout` # (session idle read+write timeout), and `--max-pack-size` (floor: -# MAX_OBJECT_SIZE, 256 MiB) — layered on P26/P29's `sc serve --http`. +# the max object size, 256 MiB) — layered on P26/P29's `sc serve --http`. # # Honesty note on scope: a connection that never sends its opening is # reaped by the FIXED 30s opening timeout, not by `--timeout` (which @@ -20,6 +20,9 @@ set -euo pipefail cd "$(dirname "$0")/.." +# The --max-pack-size values below assume the default 256 MiB object cap; a +# caller's exported SC_MAX_OBJECT_SIZE must not change what the demo proves. +unset SC_MAX_OBJECT_SIZE cargo build --quiet --bin sc SC="$PWD/target/debug/sc" @@ -164,7 +167,7 @@ echo "clone through a floor-capped server: succeeds ✔" # The cap gates the PutPack spool (the server's receive path) — a clone's # GetPack never consults it. Exercise the actually-gated path: a small # commit pushed THROUGH the capped server must land (its pack is far under -# the 256 MiB floor cap). If the cap's enforcement broke open-endedly or +# the 256 MiB --max-pack-size floor, which is the default max object size). If the cap's enforcement broke open-endedly or # regressed to rejecting everything, this push is the step that notices. cd "$W/c-floor" printf 'v2 via capped push\n' > pushed.txt @@ -183,7 +186,7 @@ SERVER_PID="" ! nc -z 127.0.0.1 "$PORT2" 2>/dev/null || fail "second server still accepting connections after kill" if "$SC" serve --http "127.0.0.1:$PORT2" --max-pack-size 1024 "$A" 2>"$W/err-cap.txt"; then - fail "server should have refused to start with --max-pack-size below MAX_OBJECT_SIZE" + fail "server should have refused to start with --max-pack-size below the max object size" fi grep -qi "max-pack-size" "$W/err-cap.txt" \ || fail "expected a --max-pack-size floor error, got: $(cat "$W/err-cap.txt")" diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index dd77a844..d4e51bd2 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -328,7 +328,7 @@ too, not just the src-control-side metadata. it. The incoming-pack spool used no aggregate size bound — only ADR-0039's per-frame/per-record caps applied — so a single oversized or endless pack could exhaust server disk on **either** transport. `--max-pack-size` - (default 16 GiB, 0 = unlimited, floor 256 MiB = `MAX_OBJECT_SIZE`) now + (default 16 GiB, 0 = unlimited, floor = the max object size, 256 MiB by default) now counts the running total and aborts mid-stream past the cap. This includes the **read-only drain path**: the pre-`EC_READONLY` drain that keeps a connection in sync when a read-only-scoped client streams a pack the server @@ -372,7 +372,7 @@ too, not just the src-control-side metadata. (`walfmt.rs`): unknown versions, bad magic, and any length that would overrun the buffer are refused outright rather than best-effort parsed. Untrusted-length caps apply throughout: manifest/log/idx bytes are capped - at `MAX_OBJECT_SIZE` before decode, and pack bodies are read through + at a fixed 256 MiB (not the operator's cap) after download, before decode, and pack bodies are read through `scl_core::pack::read_object_at_bounded` (not the unbounded `read_object_at` the trusted local `Store` uses for its own already- verified on-disk packs, per ADR-0039's explicit trust split) — capping @@ -429,23 +429,90 @@ too, not just the src-control-side metadata. nothing both sent and refused, nothing omitted) before any object is written. A hostile remote can at worst answer "not sent"; it cannot plant an object under another id or smuggle in extras. The reply stream a client will - read is capped at 1 GiB. + read is capped at 4 GiB, a constant independent of the object cap (it + bounds a disk spool, not RAM). - **Server cost is bounded per request:** 1024 ids (checked while decoding, before any id is read), the server's `--max-pack-size` over the summed - encoded size, and `MAX_OBJECT_SIZE` per object. + encoded size, and 1 GiB (the largest configurable object cap) per object. The + server's own object cap is not applied to replies, since the cap is + receiver-only: a lower-cap server serves a larger local object to a client + whose cap admits it, as `GetPack` does. The reply ceiling is a constant + 2 GiB. - **Laziness is a local opt-in.** `.sc/promisor-lazy` is never committed or transferred; without it a partial clone never dials the remote after clone, and `SC_OFFLINE=1` forces that per process. ## Untrusted-input hardening (DoS) — ADR-0039 -- A single `MAX_OBJECT_SIZE` (256 MiB) caps every untrusted length: wire frames, +- A single max object size caps every untrusted length: wire frames, pack-record compressed length, the zstd *decompressed* output (decompression-bomb - guard), and object-decode collection counts. **Accepted boundary:** the cap is a - *transfer-path* guard — a locally-committed blob larger than 256 MiB commits fine - but cannot then be transferred; and the wire frame-length header can still - allocate up to the cap before a chunk boundary is enforced (a deferred - hostile-peer hardening item). + guard), bucket pack bodies, by-id reply objects, the `SC_PACK_CHUNK` clamp (fixed at the + 2 MiB minimum cap minus one, since caps are receiver-local and unnegotiated), + and object-decode collection counts. The default is 256 MiB. An operator can + lower or raise it with `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` + (#213), within a compile-time range of 2 MiB to 1 GiB; a value outside that + range aborts startup rather than being clamped; an empty `SC_MAX_OBJECT_SIZE` + counts as unset. A guard checks the claimed length before allocating, and + frame bodies, pack records and the zstd output are then read in exact-size + growth steps as bytes arrive, so over-allocation is bounded (at most the + larger of 8 MiB and 12.5% of the bytes received) and a hostile header or + compression bomb cannot buy a cap-sized allocation without sending the bytes. + **Accepted boundaries:** + - The cap is a *transfer-path* guard. A locally-committed blob above it + commits fine but cannot be transferred. + - **The receiver's cap must admit the object.** There is no negotiation and + no protocol bump, and only the receiving side's cap applies to incoming + frames and records (a default server serves a 3 MiB object to a client + capped at 4 MiB). An object above the receiver's cap is refused there, + loudly, with an error naming `SC_MAX_OBJECT_SIZE`. A server raised + to 1 GiB accepts a peer's gigabyte claim only if that peer sends the bytes, + but then holds that object in memory. Raising the cap raises + per-connection memory in two parts. Transient pack-decode memory: a record + holds its compressed payload plus the decompressed output (each up to the + cap) while decompressing, then the decompressed output plus the decoded + object while decoding (the compressed copy is released first), so the peak + is about 2 x cap. A single-object `PutObject` is also about 2 x cap: the + request frame is moved, not copied, into the object body, and storing it + decodes one more copy; a `GetObject` reply on the server holds the body and + its response frame. A read-only listener rejects a `PutObject` only after + reading the frame, so it can hold one cap-sized frame per connection to + refuse it. Plus the connection's own store budget, + `max(512 MiB, 2 x cap)`, because each served connection opens its own + store. That is about 0.5 GiB + 0.5 GiB per connection at the default and + about 2 GiB + 2 GiB at the 1 GiB ceiling, and it multiplies by + `--max-connections` (default 32); size `--max-connections` down when + raising the cap. Lowering the cap below a repository's largest object + makes that repository unpushable until it is restored. `SC_MAX_OBJECT_SIZE` + also sets the default store budget of every local command that opens a + repo or serves one (`Repo::open`, `LocalTransport`), not only transfers. + An explicit budget overrides it: `sc work --budget-mb` and + `Repo::open_with_budget` use exactly the size given, and one smaller than + an object fails with `BudgetExceeded`. + - **Frames carry 64 bytes of framing beyond the cap.** A frame may be + `cap + 64` bytes (`PutObject` header 37 bytes, response status 1, chunk + tag 1), so an object of exactly the cap fits; the object body itself is then + checked against the cap exactly. + - **A local-path source loads before the receiver refuses.** A clone or + fetch from a local-path remote reads each object through + `LocalTransport::get_object` (load plus encode) before the receiver's cap + rejects it, bounded by the local store budget. Wire sources refuse in + `WireClient::get_object`, before decoding. + - **Structural objects have no per-connection aggregate budget (pre-existing, + amplified by a raised cap).** `Object::blob_size` is 0 for trees, + snapshots, secrets and transcripts, so repeated `PutObject` of distinct + structural objects on a writable listener is bounded per object (the cap) + but not in aggregate per connection: only by `--max-connections` times + connection lifetime (`--timeout`). Prefer read-only or token-gated + listeners, keep `--max-object-size` low, and size `--max-connections` + down. An aggregate per-connection byte budget is tracked in #233. + - **One length is checked after allocation:** bucket WAL metadata (manifest, + log entry, idx) is capped at a fixed 256 MiB after the bucket GET has + buffered it, so it bounds decode work, not the download. It is not tied to + the operator's cap, so lowering the cap cannot make a bucket unreadable. + - By-id reply spools (server ceiling 2 GiB, client read cap 4 GiB) are + constants independent of the cap: they bound disk, not RAM. + - The ceiling (1 GiB) is compile-time, so no configuration can make a length + claim unbounded. ## Reporting diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index ba27e6aa..88d38d26 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -3,6 +3,7 @@ - **Status:** Accepted - **Date:** 2026-07-09 - **Phase:** 28 +- **Amended:** #213 (addendum at the end) made the object cap configurable; the `MAX_OBJECT_SIZE` constant and 256 MiB figures in the sections before the addendum are the original design. - **Builds on:** ADR-0021 (durability/concurrency hardening — the P21 sweep precedent), ADR-0022 (ssh wire protocol), ADR-0015 (packfiles), ADR-0014 (protected paths), ADR-0008/0009 (secrets) ## Context @@ -128,3 +129,107 @@ same-user processes, crash dumps, and shell wrappers, and no amount of zeroizing the parent's buffer changes that. Accepted boundary: the child-env copy is fundamental and un-zeroizable; fd/stdin injection stays deferred. 1 pinned regression test. + +## Addendum (#213): the object cap is operator-configurable + +`MAX_OBJECT_SIZE` was a compile-time constant. It is now a process-wide +setting read through one accessor, `scl_core::max_object_size()`, with a +default of 256 MiB. Decisions: + +1. **Lower or raise, inside a compile-time range.** Floor 2 MiB + (`MIN_MAX_OBJECT_SIZE`; a default peer's pack chunk frame is 1 MiB plus a + tag byte, which a 1 MiB cap would refuse), ceiling 1 GiB (`MAX_MAX_OBJECT_SIZE`). A value + outside the range, or one that does not parse, aborts startup with a clear + error. It is never clamped. The ceiling bounds what a hostile peer can + make a receiver commit to for any one object whatever the operator sets. +2. **Server flag and client env, no negotiation.** `sc serve + --max-object-size ` and, for every `sc` process, + `SC_MAX_OBJECT_SIZE` (same parse, same bounds). The flag wins over the + env. An empty `SC_MAX_OBJECT_SIZE` counts as unset. The setting is fixed + by whichever comes first, the startup `set_max_object_size` or the first + `max_object_size()` read; a later, different value is an error. There is no handshake field and no + `PROTOCOL_VERSION` bump: the operator must make each receiver's cap + admit the objects it will receive, because an object larger than the + receiver's cap fails at the receiver (only the receiving side's cap applies + to incoming frames and records). The failure is loud and typed, never silent truncation, and + the frame and pack-record errors name `SC_MAX_OBJECT_SIZE` / + `--max-object-size` as the thing to raise on the refusing side. +3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`, + limit `cap + FRAME_OVERHEAD`, 64 bytes, so an object of exactly the cap fits + inside its `PutObject`/`GetObject` envelope; `wire::check_object_len` then + checks the object body against the cap exactly); + pack record compressed length and decompressed output + (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket pack + bodies (`bucket_transport::object_bytes`); server-side servable-size check + (`check_servable_size`, against the largest configurable cap, not the + server's own: the cap is receiver-only, so a server with a lower cap still + serves a larger local object by id, as `GetPack` does); the `SC_PACK_CHUNK` clamp (fixed at `MIN_MAX_OBJECT_SIZE - 1`, not the local + cap: caps are receiver-local and unnegotiated, so every chunk frame must fit + the smallest valid receiver; a chunk + frame carries a tag byte); the `--max-pack-size` floor + (`validate_max_pack_size`). Two exceptions are deliberately fixed, not derived from the cap. Bucket + WAL metadata (manifest, log entry, idx) stays capped at the default + 256 MiB: it is metadata, not an object, and a lowered cap would make a + bucket with a large pack index (48 bytes per entry) unreadable. That check + runs after the bucket GET has buffered the value, so it bounds decode work + rather than the download, the one place a length is checked after + allocation. The by-id reply ceiling (server, `2 x` the maximum configurable + cap = 2 GiB, still `min(--max-pack-size, ceiling)`) and the client's by-id + read cap (twice that) are constants: they bound a disk spool, not RAM, and + if they followed the local cap, peers with different caps would disagree + about whether a by-id batch containing no large object fits. Each guard takes the cap + as a parameter below the public entry point so tests can drive a lowered + and a raised cap without touching the process-global. +4. **Allocation follows bytes received.** The length check still runs before + any allocation, but the frame body and pack record payload are now read + through `scl_core::read_limited` / `read_exact_vec`, which grow the buffer by + `reserve_exact` steps as bytes arrive instead of pre-allocating the + claimed length. Steps are geometric up to 8 MiB, then `max(8 MiB, filled/8)`, + so capacity never exceeds the bytes received by more than the larger of + 8 MiB and 12.5%, and total copying stays linear; the zstd decompressed + output uses the same reader (still bounded at `cap + 1` to detect overflow). + With the ceiling at 1 GiB, a four-byte header or a decompression bomb no + longer buys a gigabyte-sized allocation. This closes the "frame header can + allocate up to the cap" item in THREAT-MODEL.md. + +The resident-blob budget of a repo or transport store was a fixed 512 MiB; a +raised cap would have admitted frames the store then refused with +`BudgetExceeded`. The default is now `max(512 MiB, 2 x cap)` +(`repo::default_budget`), unchanged at the default cap. It applies to +`Repo::open`/`init` and to `LocalTransport` (so also to a served connection). +The env var therefore changes the default budget of every local command that +opens a repo, not just transfers. An explicit budget (`sc work --budget-mb`, +`Repo::open_with_budget`) overrides the default and can be smaller than an +object. + +Raising the cap raises the memory a peer can make a receiver hold per +connection, in two parts: transient pack decode memory (compressed + +decompressed while decompressing, then decompressed + decoded while decoding, +the compressed copy being released in between: peak about 2 x cap; a +single-object `PutObject` or `GetObject` is also about 2 x cap, the request +frame being moved into the object body rather than copied) plus that +connection's own store budget (each served connection opens its own store, +`max(512 MiB, 2 x cap)`). At the default that is about 0.5 GiB + 0.5 GiB; at +the 1 GiB ceiling about 2 GiB + 2 GiB, per connection, times +`--max-connections` (THREAT-MODEL.md). + +Known gaps, not addressed here. `Object::blob_size` is 0 for structural +objects (trees, snapshots, secrets, transcripts), so a writable listener has no +per-connection aggregate byte bound on repeated `PutObject` of distinct +structural objects; each is capped, the sum is bounded only by +`--max-connections` x connection lifetime. This predates #213 and a raised cap +amplifies it: prefer read-only or token-gated listeners, keep the cap low. +Tracked in #233. A local-path remote loads and encodes an object before the +receiver's cap refuses it (`TransportSource::get`), bounded by the store +budget. + +For `ssh://` remotes the client builds the remote `sc serve --stdio` command +itself and ssh does not forward the environment, so the server end of an ssh +remote gets a non-default cap only from the server's own environment (a +wrapper script, `sshd` `SetEnv`, or a forced command passing +`--max-object-size`). + +Unchanged: the cap is still a transfer-path guard, not a local-commit limit, +so lowering it can leave a repository holding committed objects it cannot +push; `read_object_at` (the trusted on-disk path) stays uncapped. The +desktop app does not read the env var; it does no transfers. diff --git a/docs/adr/0041-listener-resource-limits.md b/docs/adr/0041-listener-resource-limits.md index 1e83297d..8a4d2666 100644 --- a/docs/adr/0041-listener-resource-limits.md +++ b/docs/adr/0041-listener-resource-limits.md @@ -6,7 +6,7 @@ - **Builds on:** ADR-0036 (P26 sc-native HTTP transport — names three of the four gaps closed here), ADR-0040 (P29 access control — the seam where a busy/error status is already written before the wire handshake), ADR-0039 - (P28 DoS hardening — `MAX_OBJECT_SIZE`, composed with, not replaced by, + (P28 DoS hardening — the max object size, composed with, not replaced by, this phase's spool cap) - **Research:** `docs/research/bounded-server-patterns.md` (ticket #27, std-only bounded-server survey pinned to file:line). Decided at the @@ -56,11 +56,11 @@ scoped to the listener (`--stdio` is unaffected except where noted): stall trips it, not a slow-but-moving transfer. The opening keeps its existing, unchanged 30s. -3. **`--max-pack-size ` (default 16 GiB, 0 = unlimited, floor 256 MiB - = `MAX_OBJECT_SIZE`).** Applies to **both** `--http` and `--stdio` — it +3. **`--max-pack-size ` (default 16 GiB, 0 = unlimited, floor = the max + object size, 256 MiB by default).** Applies to **both** `--http` and `--stdio` — it lives in `WirePolicy` (`crates/repo/src/wire.rs`), the shared layer both transports call into, not in `ServeLimits` (the `--http`-only listener - config). A configured non-zero value below `MAX_OBJECT_SIZE` is rejected + config). A configured non-zero value below the max object size (`scl_core::max_object_size()`, configurable since #213, ADR-0039 addendum) is rejected as invalid config at startup. Enforcement is a counted mid-stream abort in the pack-stream readers (mirroring git's `receive.maxInputSize`): once the running total crosses the cap, the reader stops, a best-effort diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index f1bffe14..85046219 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -68,11 +68,13 @@ immutable packs plus one compare-and-swapped manifest**: format has no per-prefix negotiation yet; a filtered clone needs a served remote (ADR-0037) until that lands. - **Untrusted-length guard on reads.** Manifest, log entries, and idx - metadata are capped at `MAX_OBJECT_SIZE` (256 MiB, ADR-0039) before + metadata are capped at a fixed 256 MiB (the default object cap; since #213 + deliberately not the operator's cap, see the ADR-0039 addendum) before decoding — a hostile or corrupted bucket cannot force an unbounded allocation. Pack bodies are read through `scl_core::pack::read_object_at_bounded`, which caps both the compressed - record length and the decompressed output at `MAX_OBJECT_SIZE`, mirroring + record length and the decompressed output at the live object cap (default + 256 MiB, configurable from 2 MiB to 1 GiB since #213), mirroring `parse_pack_reader`'s bounded decode — this is the same zstd-bomb guard P28 put on every other untrusted transfer path. **This is deliberately a different code path from `Store`'s own on-disk reads:** the trusted local @@ -144,7 +146,7 @@ confidentiality perimeter for public content, not the sc protocol. into `open_transport` for `sc+wal://` and `sc+s3://`, validated eagerly at `sc remote add`. 12 tests in `bucket_transport.rs` covering the base round-trip, racing/fleet pushes, crash-mid-push recovery, and the -`MAX_OBJECT_SIZE` cap. CLI plumbing (remote-add validation, push, clone, +object-size cap. CLI plumbing (remote-add validation, push, clone, fetch through the real binary) proven in `crates/cli/tests/bucket_remote.rs`. diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md index ea8a3034..5880bac4 100644 --- a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -109,8 +109,10 @@ not have. The rules for `GetObjects`: confidentiality rests on encryption, not on withholding. - Caps: 1024 ids per request (`MAX_GET_OBJECTS_IDS`, checked while decoding), the server's `--max-pack-size` over the summed canonical encoded size of a reply and - `MAX_OBJECT_SIZE` per object (both `PackTooLarge`, before any pack byte is - written), and a 1 GiB cap on the stream a client will read back. A client + 1 GiB (the largest configurable object cap, ADR-0039 addendum) per object (both `PackTooLarge`, before any pack byte is + written), and a 4 GiB cap on the stream a client will read back (originally + 1 GiB; a constant independent of the object cap since #213, see ADR-0048 and + the ADR-0039 addendum). A client also stops reading a pack as soon as it holds more records than it requested, so repeating one large object cannot multiply the receiver's decompress-and-hash work. @@ -149,9 +151,11 @@ not have. The rules for `GetObjects`: 3. **Merge on a partial clone** needs base, ours and theirs trees for only the conflicting paths. Whether tree ids alone suffice (identical ids need no fetch) or a server-assisted diff is needed is a 37b spike. -4. **Reply size caps and batching.** The client reads at most 1 GiB per reply +4. **Reply size caps and batching.** (Superseded by ADR-0048; the figures below + were 1 GiB then and are a 4 GiB client cap and 2 GiB server ceiling since + #213.) The client reads a capped number of bytes per reply while the server's default `--max-pack-size` is larger, so a batch that - crosses 1 GiB fails mid-stream and leaves that connection out of sync. 37a + crosses the client cap fails mid-stream and leaves that connection out of sync. 37a sends one id per call so it cannot happen; 37b must split batches by size or close the transport on that error. 5. **`GetObject` and unfiltered private transfers.** See the authorization diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 433bd664..59fd6414 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -273,22 +273,23 @@ protocol error, because the stream is no longer in sync. ### 5. Reply size limits -37a left a mismatch: the client reads at most 1 GiB per by-id reply while the -server default `--max-pack-size` is larger, and the server sums canonical +37a left a mismatch: the client read at most 1 GiB per by-id reply (4 GiB since +#213) while the server default `--max-pack-size` is larger, and the server sums canonical object sizes while the client counts pack bytes (36-byte record headers plus zstd framing). Decisions: - The server caps a `GetObjects` reply, measured in canonical size, at - `min(--max-pack-size, 512 MiB)` where a `--max-pack-size` of 0 means - unlimited. Headroom to the client's 1 GiB read cap covers framing and - incompressible data. + `min(--max-pack-size, 2 GiB)` where a `--max-pack-size` of 0 means + unlimited. Headroom to the client's 4 GiB read cap covers framing and + incompressible data. (Originally 512 MiB and 1 GiB; #213 made both + constants independent of the object cap, see the ADR-0039 addendum.) - The cap applies to `LocalTransport::get_objects` as well, which today builds with no limit (`build_objects_tempfile(ids, 0)`). A lazy clone whose promisor is a local path would otherwise spool up to 1024 large blobs in one request. - The over-cap refusal happens before any pack byte is written, so the connection stays in sync. The client splits the batch in half and retries. A batch of one that is still refused is final and is returned as an error; - splitting never loops. (An object above `MAX_OBJECT_SIZE`, or above a small + splitting never loops. (An object above 1 GiB, the largest configurable object cap, or above a small `--max-pack-size`, can exist on an origin.) - The 37a handling for a stream abandoned mid-reply (kill the ssh child, drop the connection) stays as the backstop. diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index 84df85ce..3c265529 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -66,7 +66,7 @@ Recipient-list changes alone do not require re-wrapping a carried ciphertext: th A per-id rule ("prune what you enumerated") is unsound: a blob id seen as deleted at one path may survive under a carried subtree elsewhere, since ids alias across paths. So the rule is coarse: **if the merge carried any subtree, the merged map is the complete union of the input wraps** (ours, theirs, base, with the existing ours/theirs/base precedence and per-recipient dedup), plus fresh wraps. If it carried nothing, the existing prune applies. Union resurrecting a wrap for a recipient revoked since is the intended P16 behavior (tombstones gate future seals, not historical wraps; ADR-0026). -Over-retention costs size: wraps are inline in the snapshot, and an object above `MAX_OBJECT_SIZE` fails transfer. Nothing prunes dead entries today (`sc rewrap` refuses partial clones and only removes ids it upgrades, `rewrap.rs:164`, `:308`). So the growth is real and bounded by the repo's protected blob count; it is measured before any prune is designed. +Over-retention costs size: wraps are inline in the snapshot, and an object above the max object size (256 MiB by default, ADR-0039 addendum) fails transfer. Nothing prunes dead entries today (`sc rewrap` refuses partial clones and only removes ids it upgrades, `rewrap.rs:164`, `:308`). So the growth is real and bounded by the repo's protected blob count; it is measured before any prune is designed. ### 5. The engine is pull-based; filling happens outside the store lock