From 189f94c4591e61685223f0f2bc7ba2961368bb15 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 08:43:46 +0300 Subject: [PATCH 01/11] feat(core,repo,cli): operator-configurable max object size (#213) Replace the MAX_OBJECT_SIZE constant with a process-wide cap read through scl_core::max_object_size() (default 256 MiB, range 1 MiB to 1 GiB, out of range aborts startup). Set by sc serve --max-object-size or SC_MAX_OBJECT_SIZE. No wire negotiation; both ends must agree. Every untrusted-length guard reads the accessor, the by-id reply ceiling and client read cap scale with it, and frame/pack-record bodies are read incrementally so a header cannot force a cap-sized allocation. ADR-0039 addendum, THREAT-MODEL and docs updated. --- .claude/skills/sc-cli-reference/SKILL.md | 9 +- ARCHITECTURE.md | 8 +- CLAUDE.md | 9 +- ROADMAP.md | 9 +- crates/cli/src/main.rs | 34 ++++- crates/cli/tests/max_object_size.rs | 139 ++++++++++++++++++++ crates/core/src/lib.rs | 152 +++++++++++++++++++++- crates/core/src/pack.rs | 119 ++++++++++++----- crates/core/tests/max_object_size.rs | 24 ++++ crates/repo/src/bucket_transport.rs | 23 +++- crates/repo/src/http_transport.rs | 2 +- crates/repo/src/stdio_transport.rs | 20 ++- crates/repo/src/transport.rs | 82 +++++++----- crates/repo/src/wire.rs | 152 ++++++++++++++++------ demo/run_limits_demo.sh | 4 +- docs/THREAT-MODEL.md | 32 +++-- docs/adr/0039-security-hardening-sweep.md | 51 ++++++++ 17 files changed, 721 insertions(+), 148 deletions(-) create mode 100644 crates/cli/tests/max_object_size.rs create mode 100644 crates/core/tests/max_object_size.rs diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index f0bc86d8..4a49fcd7 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -270,8 +270,15 @@ cargo run --bin sc -- serve --http [--read-only] [--allow-public] # trip; opening keeps its own 30s (P31) # --max-pack-size : incoming-pack aggregate # cap, both --http and --stdio, default 16 GiB, - # 0=unlimited, floor 256 MiB (MAX_OBJECT_SIZE); + # 0=unlimited, floor = the max object size; # counted mid-stream abort -> EC_TOO_LARGE (P31) + # --max-object-size : largest + # single object accepted from a peer, default + # 256MiB, range 1MiB..1GiB, out of range aborts + # startup (never clamped). Clients use env + # SC_MAX_OBJECT_SIZE (same parse, any command); + # flag beats env. No negotiation: both ends must + # agree, a larger object fails at the receiver (#213) cargo run --bin sc -- serve token add --label --scope ro|rw # mint an # sct- bearer token in .sc/serve-tokens.toml; # raw value prints once on stdout (P29) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 22b4e23f..ab591297 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -139,7 +139,7 @@ a retired pre-P16 snapshot encoding, refused with a clear error): Objects are serialized canonically (length-prefixed, sorted tree entries) so the hash is stable across machines. Every decode length prefix is bounded by a -`MAX_OBJECT_SIZE` / remaining-bytes guard (P28) so a hostile object can't drive +max-object-size / remaining-bytes guard (P28) so a hostile object can't drive an unbounded allocation. ## In-memory VFS layer @@ -888,7 +888,7 @@ coarsening rather than per-case gap tolerance. See ADR-0037. ## Phase 28 — security hardening sweep (built) A dedicated security pass closing exposure introduced by opening the store -to untrusted network peers (P26/P27). One `MAX_OBJECT_SIZE` cap bounds any +to untrusted network peers (P26/P27). One max-object-size cap (`scl_core::max_object_size()`, ADR-0039 addendum) bounds any single object's decompressed/decoded size at every read/write boundary that takes attacker-controlled bytes, closing a decompression-bomb / unbounded- allocation class that P25's `read_frame_inner` had already named as a known @@ -943,8 +943,8 @@ limit a new connection gets an immediate busy status at the pre-handshake opening seam and closes — no queuing), `--timeout` (default 300s; read+write timeouts now persist for the whole session instead of being cleared after the opening, connection-fatal on trip since a mid-stream abort desyncs the frame -stream), and `--max-pack-size` (default 16 GiB, floor 256 MiB = -`MAX_OBJECT_SIZE`; lives in the shared `WirePolicy`, so it applies to both +stream), and `--max-pack-size` (default 16 GiB, floor = the max object +size, 256 MiB by default; lives in the shared `WirePolicy`, so it applies to both `--http` and `--stdio`; a counted mid-stream abort replies `EC_TOO_LARGE`). The read-only pre-drain is capped at 8 MiB (`RO_DRAIN_CAP`), and the accept diff --git a/CLAUDE.md b/CLAUDE.md index 7c22a353..427e6093 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -142,7 +142,7 @@ the code, those win. | P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) | | P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) | | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | -| P28 | Security hardening: `MAX_OBJECT_SIZE` (256 MiB) caps on every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | +| P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 1 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token`), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | | P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign`, always-encrypted body, gc-coupled index | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff | [0041](docs/adr/0041-listener-resource-limits.md) | @@ -163,9 +163,12 @@ Cross-cutting current facts that bite. Security boundaries are consolidated in transport-adjacent. The rest, imperatively: - **Wire protocol is version 5** (bumped from 4 in P37 for the additive - `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). `MAX_OBJECT_SIZE` (256 MiB) guards the + `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB; + `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 1 MiB to + 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every - subsequent push/fetch/clone at the receiver. + subsequent push/fetch/clone at the receiver. There is no negotiation: both + ends must agree. Never inline a size in a new length guard; read the accessor. - **The P35 desktop app is a keyless, read-only top-level adapter.** Its renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or diff --git a/ROADMAP.md b/ROADMAP.md index 8ee5352b..8bbc3571 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -803,11 +803,12 @@ scale-&-reach horizon): secrets to the command on a descriptor (NUL-terminated `NAME=VALUE` records) instead of its environment. Not built: the same mode for `sc ws run` and `sc work --with-secrets`, and a stdin variant. -- **`--max-object-size` operator config knob (P28).** `MAX_OBJECT_SIZE` - (256 MiB) is a compile-time constant anchoring every untrusted-length - guard; letting an operator raise or lower it per-deployment is deferred. +- **`--max-object-size` operator knob — shipped (#213, ADR-0039 addendum).** + `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap + (default 256 MiB, range 1 MiB to 1 GiB, out of range aborts startup). No wire + negotiation: both ends must agree. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run - `validate_branch_name`; `SC_PACK_CHUNK` is clamped to `MAX_OBJECT_SIZE - 1` (a chunk frame carries a tag byte too; an + `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the object cap minus one (a chunk frame carries a tag byte too; an oversized value used to make every transfer fail with a confusing receiver error); `Repo::worktree_paths` is a paths-only walk (`list_worktree_paths`, the same ignore and tracked-path rules as `read_worktree`, no file read); and diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 7efe3e88..034604da 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -356,10 +356,17 @@ enum Cmd { #[arg(long)] timeout: Option, /// Maximum incoming pack size in bytes (both --http and --stdio); - /// 0 = unlimited. Default 16 GiB. Must be ≥ 256 MiB (MAX_OBJECT_SIZE) - /// when non-zero. (P31) + /// 0 = unlimited. Default 16 GiB. Must be at least the max object + /// size (default 256 MiB) when non-zero. (P31) #[arg(long)] max_pack_size: Option, + /// Largest single object accepted from a peer: bytes, or a number + /// with a KiB/MiB/GiB suffix. Default 256MiB, range 1MiB to 1GiB; + /// out-of-range values are refused at startup, never clamped. Clients + /// set the same limit with `SC_MAX_OBJECT_SIZE`; both ends must + /// agree, since a larger object fails at the receiver. (#213) + #[arg(long, value_name = "SIZE")] + max_object_size: Option, /// Serve TLS (`sc+https://`): auto-mints a self-signed identity into /// `.sc/serve-tls/` unless --tls-cert/--tls-key supply a PEM pair. /// `--http` only. (P32) @@ -927,8 +934,28 @@ struct DemoArgs { checkout: bool, } +/// Fix the process-wide object cap before any command runs: `sc serve +/// --max-object-size` wins over `SC_MAX_OBJECT_SIZE`, which wins over the +/// default. An out-of-range or unparsable value aborts startup (#213). +fn init_max_object_size(flag: Option<&str>) -> Result<()> { + let (src, raw) = match (flag, std::env::var("SC_MAX_OBJECT_SIZE")) { + (Some(f), _) => ("--max-object-size", f.to_string()), + (None, Ok(e)) => ("SC_MAX_OBJECT_SIZE", e), + (None, Err(std::env::VarError::NotPresent)) => return Ok(()), + (None, Err(e)) => anyhow::bail!("SC_MAX_OBJECT_SIZE: {e}"), + }; + let n = scl_core::parse_object_size(&raw).map_err(|e| anyhow::anyhow!("{src}: {e}"))?; + scl_core::set_max_object_size(n).map_err(|e| anyhow::anyhow!("{src}: {e}")) +} + fn main() -> Result<()> { let cli = Cli::parse(); + init_max_object_size(match &cli.cmd { + Cmd::Serve { + max_object_size, .. + } => max_object_size.as_deref(), + _ => None, + })?; match cli.cmd { Cmd::Demo(args) => run_demo(args), Cmd::Import { repo } => run_import(repo), @@ -1023,6 +1050,7 @@ fn main() -> Result<()> { max_connections, timeout, max_pack_size, + max_object_size: _, tls, tls_cert, tls_key, @@ -3760,7 +3788,7 @@ fn run_serve( ); } let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE); - scl_repo::wire::validate_max_pack_size(max_pack)?; + scl_repo::wire::validate_max_pack_size(max_pack, scl_core::max_object_size())?; let policy = scl_repo::wire::WirePolicy { read_only: false, max_pack_size: max_pack, diff --git a/crates/cli/tests/max_object_size.rs b/crates/cli/tests/max_object_size.rs new file mode 100644 index 00000000..90c76057 --- /dev/null +++ b/crates/cli/tests/max_object_size.rs @@ -0,0 +1,139 @@ +//! `SC_MAX_OBJECT_SIZE` / `sc serve --max-object-size` (#213, ADR-0039 +//! addendum): bounds are enforced at startup, and the cap is honoured by a +//! real object transfer. + +use std::path::Path; +use std::process::{Command, Output}; + +fn sc(dir: &Path, envs: &[(&str, &str)], args: &[&str]) -> Output { + let mut c = Command::new(env!("CARGO_BIN_EXE_sc")); + c.args(args) + .current_dir(dir) + .env_remove("SC_MAX_OBJECT_SIZE"); + for (k, v) in envs { + c.env(k, v); + } + c.output().expect("sc runs") +} + +fn tmp(tag: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-maxobj-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +#[test] +fn out_of_range_env_is_rejected_at_startup_not_clamped() { + let root = tmp("env-bounds"); + for bad in [ + "0", + "1023KiB", + "1025MiB", + "2GiB", + "junk", + "99999999999999GiB", + ] { + let o = sc(&root, &[("SC_MAX_OBJECT_SIZE", bad)], &["init"]); + assert!(!o.status.success(), "{bad} must be rejected"); + assert!( + stderr(&o).contains("SC_MAX_OBJECT_SIZE"), + "{bad}: {}", + stderr(&o) + ); + assert!( + !root.join(".sc").exists(), + "{bad}: rejected before any work" + ); + } + for good in ["1048576", "1MiB", "256mib", "1GiB"] { + let o = sc(&root, &[("SC_MAX_OBJECT_SIZE", good)], &["init"]); + assert!(o.status.success(), "{good}: {}", stderr(&o)); + std::fs::remove_dir_all(root.join(".sc")).unwrap(); + } + std::fs::remove_dir_all(&root).unwrap(); +} + +#[test] +fn serve_flag_is_bounds_checked_and_overrides_env() { + let root = tmp("serve-flag"); + assert!(sc(&root, &[], &["init"]).status.success()); + for bad in ["512", "2GiB", "x"] { + let o = sc( + &root, + &[], + &["serve", "--stdio", "--max-object-size", bad, "."], + ); + assert!(!o.status.success(), "{bad} must be rejected"); + assert!( + stderr(&o).contains("--max-object-size"), + "{bad}: {}", + stderr(&o) + ); + } + // The flag wins over a bad env value (the env is never consulted). + let o = sc( + &root, + &[("SC_MAX_OBJECT_SIZE", "junk")], + &[ + "serve", + "--stdio", + "--max-object-size", + "1MiB", + "--max-pack-size", + "1024", + ".", + ], + ); + assert!(!o.status.success()); + assert!( + stderr(&o).contains("below the max object size (1048576)"), + "the pack floor uses the flag's cap: {}", + stderr(&o) + ); + std::fs::remove_dir_all(&root).unwrap(); +} + +#[test] +fn a_lowered_cap_refuses_a_transfer_a_raised_one_allows() { + let src = tmp("xfer-src"); + assert!(sc(&src, &[], &["init"]).status.success()); + // 2 MiB of incompressible bytes: one object over a 1 MiB cap. + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let big: Vec = (0..2 * 1024 * 1024) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + std::fs::write(src.join("big.bin"), &big).unwrap(); + let o = sc(&src, &[("SC_AUTHOR", "t")], &["commit", "-m", "big"]); + assert!(o.status.success(), "commit is not capped: {}", stderr(&o)); + + let parent = tmp("xfer-dst"); + let low = parent.join("low"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "1MiB")], + &["clone", src.to_str().unwrap(), low.to_str().unwrap()], + ); + assert!(!o.status.success(), "1 MiB cap must refuse a 2 MiB blob"); + + let high = parent.join("high"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["clone", src.to_str().unwrap(), high.to_str().unwrap()], + ); + assert!(o.status.success(), "4 MiB cap accepts it: {}", stderr(&o)); + assert_eq!(std::fs::read(high.join("big.bin")).unwrap(), big); + + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 719daef0..61f28bbb 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -11,16 +11,100 @@ pub mod object; pub mod pack; pub mod store; -/// The largest single object (decoded/decompressed canonical bytes) this -/// system will accept from an untrusted source — a pack record's compressed -/// payload, its decompressed output, or a wire frame body. This is the -/// single anchor for every untrusted-length DoS guard added in P28: caps are -/// expressed relative to this constant rather than re-picked ad hoc at each -/// call site, so one number bounds "how big can one object get" everywhere. +/// Default for the largest single object (decoded/decompressed canonical +/// bytes) this system accepts from an untrusted source. The live value is +/// [`max_object_size`]; this is what it is when nothing configured it. /// 256 MiB comfortably exceeds any legitimate single object in this system /// (blobs, trees, snapshots, secrets) while still bounding a malicious /// peer's ability to force multi-GB allocations from a few header bytes. -pub const MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; // 256 MiB +pub const DEFAULT_MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; + +/// Hard floor for the operator-set object cap (1 MiB): below this the +/// 1 MiB default pack chunk and ordinary snapshots stop fitting a frame. +pub const MIN_MAX_OBJECT_SIZE: usize = 1024 * 1024; + +/// Hard ceiling for the operator-set object cap (1 GiB). Compile-time, so no +/// flag or env value can make a hostile peer's allocation claim unbounded. +pub const MAX_MAX_OBJECT_SIZE: usize = 1024 * 1024 * 1024; + +static MAX_OBJECT_SIZE: std::sync::OnceLock = std::sync::OnceLock::new(); + +/// The process-wide cap on one untrusted object (ADR-0039 addendum, #213): +/// a pack record's compressed payload and decompressed output, a wire frame +/// body, a by-id reply object, and the `SC_PACK_CHUNK` clamp. The single +/// source for every untrusted-length guard; never inline a number. Defaults +/// to [`DEFAULT_MAX_OBJECT_SIZE`] until [`set_max_object_size`] runs. +pub fn max_object_size() -> usize { + MAX_OBJECT_SIZE + .get() + .copied() + .unwrap_or(DEFAULT_MAX_OBJECT_SIZE) +} + +/// Set the process-wide object cap, once, at startup. Rejects (never clamps) +/// a value outside [`MIN_MAX_OBJECT_SIZE`]..=[`MAX_MAX_OBJECT_SIZE`]. Setting +/// the same value again is a no-op; a different value after one was fixed +/// is an error, since guards already running would disagree. +pub fn set_max_object_size(n: usize) -> std::result::Result<(), String> { + check_object_size_bounds(n)?; + let cur = *MAX_OBJECT_SIZE.get_or_init(|| n); + if cur == n { + Ok(()) + } else { + Err(format!("max object size is already fixed at {cur} bytes")) + } +} + +fn check_object_size_bounds(n: usize) -> std::result::Result<(), String> { + if (MIN_MAX_OBJECT_SIZE..=MAX_MAX_OBJECT_SIZE).contains(&n) { + Ok(()) + } else { + Err(format!( + "max object size {n} bytes is outside the allowed range {MIN_MAX_OBJECT_SIZE} (1 MiB) to {MAX_MAX_OBJECT_SIZE} (1 GiB)" + )) + } +} + +/// Parse an object-size setting: plain bytes, or a number with a `KiB`, +/// `MiB` or `GiB` suffix (case-insensitive). Applies the same hard bounds as +/// [`set_max_object_size`]; an out-of-range value is an error, not clamped. +pub fn parse_object_size(s: &str) -> std::result::Result { + let t = s.trim(); + let lower = t.to_ascii_lowercase(); + let (digits, mult) = if let Some(d) = lower.strip_suffix("kib") { + (d, 1usize << 10) + } else if let Some(d) = lower.strip_suffix("mib") { + (d, 1 << 20) + } else if let Some(d) = lower.strip_suffix("gib") { + (d, 1 << 30) + } else { + (lower.as_str(), 1) + }; + let n = digits + .trim() + .parse::() + .ok() + .and_then(|n| n.checked_mul(mult)) + .ok_or_else(|| { + format!("invalid object size {s:?}: expected bytes or a KiB/MiB/GiB suffix, e.g. 64MiB") + })?; + check_object_size_bounds(n)?; + Ok(n) +} + +/// Read exactly `len` bytes, growing the buffer as bytes actually arrive +/// rather than pre-allocating `len`: `len` is a peer's claim, and a hostile +/// peer must not get a `len`-sized allocation for a few header bytes. The +/// caller still checks `len` against [`max_object_size`] first. +pub fn read_exact_vec(r: &mut impl std::io::Read, len: usize) -> std::io::Result> { + use std::io::Read; + let mut buf = Vec::with_capacity(len.min(1 << 20)); + let got = r.by_ref().take(len as u64).read_to_end(&mut buf)?; + if got < len { + return Err(std::io::ErrorKind::UnexpectedEof.into()); + } + Ok(buf) +} pub use error::{Error, Result}; pub use id::ObjectId; @@ -30,3 +114,57 @@ pub use object::{ WrappedKey, PROTECTED, RANDOMIZED, }; pub use store::{Backend, SpillPolicy, Store, StoreConfig, StoreStats}; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_object_size_accepts_bytes_and_suffixes() { + assert_eq!(parse_object_size("1048576").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("1MiB").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("1024kib").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!( + parse_object_size(" 256 MiB ").unwrap(), + DEFAULT_MAX_OBJECT_SIZE + ); + assert_eq!(parse_object_size("1GiB").unwrap(), MAX_MAX_OBJECT_SIZE); + } + + #[test] + fn parse_object_size_rejects_out_of_range_and_garbage_without_clamping() { + for bad in [ + "0", + "1048575", + "1023KiB", + "1025MiB", + "2GiB", + "18446744073709551615GiB", // multiplication overflow + "", + "MiB", + "-1", + "12XB", + "1.5GiB", + ] { + assert!(parse_object_size(bad).is_err(), "{bad:?} must be rejected"); + } + assert!(parse_object_size("2GiB") + .unwrap_err() + .contains("allowed range")); + } + + #[test] + fn set_max_object_size_rejects_out_of_range_values() { + assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); + assert!(set_max_object_size(MAX_MAX_OBJECT_SIZE + 1).is_err()); + // Rejection never fixes the setting, so the default still applies. + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + } + + #[test] + fn read_exact_vec_reads_len_and_errors_on_short_input() { + assert_eq!(read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(), b"abcd"); + let err = read_exact_vec(&mut &b"abc"[..], 4).unwrap_err(); + assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof); + } +} diff --git a/crates/core/src/pack.rs b/crates/core/src/pack.rs index 87119bf3..e43b5ea2 100644 --- a/crates/core/src/pack.rs +++ b/crates/core/src/pack.rs @@ -204,7 +204,7 @@ pub fn parse_index(idx: &[u8]) -> Result> { /// untouched"). `Store::read_pack_object` is this function's one caller /// (`crates/core/src/store.rs`), reading packs `sc gc`'s own repack wrote to /// this process's `.sc/objects/`: `write_pack` performs no size validation, -/// so a locally-committed blob legitimately larger than `MAX_OBJECT_SIZE` +/// so a locally-committed blob legitimately larger than the object cap /// ("committable but not transferable", per ADR-0039's final-review /// addendum) must still round-trip through a repack, or `sc gc` would /// silently make that blob permanently unreadable. Do **not** add a length @@ -251,7 +251,7 @@ fn record_bounds(pack: &[u8], offset: u64, max: Option) -> Result<(usize, if let Some(max) = max { if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit" + "record compressed length {len} exceeds the {max}-byte object size limit" ))); } } @@ -287,9 +287,9 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R .read_to_end(&mut canonical) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; if canonical.len() > max { - return Err(Error::PackCorrupt( - "decompressed object exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit".into(), - )); + return Err(Error::PackCorrupt(format!( + "decompressed object exceeds the {max}-byte object size limit" + ))); } canonical } @@ -306,7 +306,7 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R /// every record. Test-convenience whole-pack helper only — its /// `zstd::decode_all` is UNBOUNDED, and every caller in this codebase is /// `#[cfg(test)]`. Untrusted or transport-received input MUST go through -/// [`parse_pack_reader`] instead, which enforces the `MAX_OBJECT_SIZE` cap +/// [`parse_pack_reader`] instead, which enforces the [`crate::max_object_size`] cap /// on both compressed length and decompressed output. pub fn parse_pack(pack: &[u8]) -> Result> { if pack.len() < 8 || &pack[..4] != PACK_MAGIC { @@ -376,7 +376,17 @@ fn read_up_to(r: &mut R, buf: &mut [u8]) -> std::io::Result { /// pack.len()`, generalized to a reader that has no total length to check /// against. pub fn parse_pack_reader( + r: R, + f: impl FnMut(ObjectId, Object) -> Result<()>, +) -> Result<()> { + parse_pack_reader_max(r, crate::max_object_size(), f) +} + +/// [`parse_pack_reader`] with an explicit per-object cap `max` (the live +/// [`crate::max_object_size`] in production). +pub fn parse_pack_reader_max( mut r: R, + max: usize, mut f: impl FnMut(ObjectId, Object) -> Result<()>, ) -> Result<()> { let mut header = [0u8; 8]; @@ -405,32 +415,31 @@ pub fn parse_pack_reader( r.read_exact(&mut len_bytes) .map_err(|e| Error::PackCorrupt(format!("truncated record length: {e}")))?; let len = u32::from_le_bytes(len_bytes) as usize; - if len > crate::MAX_OBJECT_SIZE { + if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit" + "record compressed length {len} exceeds the {max}-byte object size limit" ))); } - let mut payload = vec![0u8; len]; - r.read_exact(&mut payload) + let payload = crate::read_exact_vec(&mut r, len) .map_err(|e| Error::PackCorrupt(format!("truncated record payload: {e}")))?; // Bound the decompressed output too: a small compressed payload can // still decompress to an enormous plaintext (a "zstd bomb"). Read at - // most MAX_OBJECT_SIZE + 1 bytes so we can detect and reject an + // most max + 1 bytes so we can detect and reject an // over-cap output without ever materializing it in full. let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; let mut canonical = Vec::new(); decoder .by_ref() - .take(crate::MAX_OBJECT_SIZE as u64 + 1) + .take(max as u64 + 1) .read_to_end(&mut canonical) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - if canonical.len() > crate::MAX_OBJECT_SIZE { - return Err(Error::PackCorrupt( - "decompressed object exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit".into(), - )); + if canonical.len() > max { + return Err(Error::PackCorrupt(format!( + "decompressed object exceeds the {max}-byte object size limit" + ))); } let actual_id = ObjectId::of(&canonical); if actual_id != expected_id { @@ -577,30 +586,64 @@ mod tests { #[test] fn pack_record_over_cap_rejected() { // A well-formed header, then one record whose id is arbitrary and - // whose compressed_len claims MAX_OBJECT_SIZE + 1 — parse_pack_reader - // must reject this before `vec![0u8; len]` allocates a 256+ MiB - // buffer for a length that was never actually sent. + // whose compressed_len claims cap + 1 — parse_pack_reader must + // reject this before allocating a buffer for a length that was never + // actually sent. let mut buf = Vec::new(); buf.extend_from_slice(PACK_MAGIC); buf.extend_from_slice(&FORMAT_VERSION.to_le_bytes()); buf.extend_from_slice(&[0u8; 32]); // arbitrary id - let over = (crate::MAX_OBJECT_SIZE + 1) as u32; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let over = (cap + 1) as u32; buf.extend_from_slice(&over.to_le_bytes()); // No payload bytes follow — if the implementation allocated first it // would then fail on read_exact anyway, but we want the cap check to // fire first (see zstd_bomb_rejected for the case where a payload // IS present but must still be rejected on the length check alone). - let err = parse_pack_reader(&buf[..], |_id, _obj| Ok(())).unwrap_err(); + let err = parse_pack_reader_max(&buf[..], cap, |_id, _obj| Ok(())).unwrap_err(); assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + // The same claim is not a length-cap error at a raised cap: it only + // fails later, on the (absent) payload bytes. + let err = parse_pack_reader_max(&buf[..], cap * 2, |_id, _obj| Ok(())).unwrap_err(); + assert!( + format!("{err}").contains("truncated record payload"), + "got {err:?}" + ); + } + + #[test] + fn a_legitimate_object_is_rejected_below_its_size_and_accepted_at_a_raised_cap() { + // Incompressible so the compressed record is also > 1 MiB. + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let payload: Vec = (0..crate::MIN_MAX_OBJECT_SIZE + 4096) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + let obj = Object::blob(payload); + let (pack, _idx) = build_pack(&[enc(&obj)]).unwrap(); + let low = crate::MIN_MAX_OBJECT_SIZE; + assert!(parse_pack_reader_max(&pack[..], low, |_, _| Ok(())).is_err()); + let mut n = 0; + parse_pack_reader_max(&pack[..], low * 4, |_, _| { + n += 1; + Ok(()) + }) + .unwrap(); + assert_eq!(n, 1); } #[test] fn zstd_bomb_rejected() { // A small compressed payload that decompresses to well beyond - // MAX_OBJECT_SIZE. parse_pack_reader must bound the zstd output + // the cap. parse_pack_reader must bound the zstd output // (decode-with-limit), not decode_all-then-check, or this test // would itself OOM the test process. - let bomb_plain = vec![0u8; crate::MAX_OBJECT_SIZE + 1024]; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let bomb_plain = vec![0u8; cap + 1024]; let compressed = zstd::encode_all(std::io::Cursor::new(&bomb_plain[..]), COMPRESSION_LEVEL).unwrap(); assert!( @@ -615,8 +658,15 @@ mod tests { buf.extend_from_slice(&(compressed.len() as u32).to_le_bytes()); buf.extend_from_slice(&compressed); - let err = parse_pack_reader(&buf[..], |_id, _obj| Ok(())).unwrap_err(); - assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + let err = parse_pack_reader_max(&buf[..], cap, |_id, _obj| Ok(())).unwrap_err(); + assert!( + format!("{err}").contains("decompressed object exceeds"), + "got {err:?}" + ); + // A raised cap lets the same record through the size guard (the + // all-zero payload then fails id verification, not the cap). + let err = parse_pack_reader_max(&buf[..], cap * 2, |_id, _obj| Ok(())).unwrap_err(); + assert!(format!("{err}").contains("hash mismatch"), "got {err:?}"); } #[test] @@ -628,10 +678,11 @@ mod tests { // untrusted input, unlike `read_object_at`'s one caller (`Store`'s // own self-written packs, ADR-0039's explicit unbounded carve-out). let id = ObjectId::of(b"whatever"); - let over = (crate::MAX_OBJECT_SIZE + 1) as u32; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let over = (cap + 1) as u32; let mut pack = Vec::new(); pack.extend_from_slice(&over.to_le_bytes()); // the length prefix read at offset 0 - let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); + let err = read_object_at_bounded(&pack, 0, &id, cap).unwrap_err(); assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); } @@ -644,7 +695,8 @@ mod tests { // trusted, unbounded path) deliberately does NOT get this guard — // see `read_object_at_reads_a_legitimately_oversized_local_record` // below for the regression that pins the other side of that split. - let bomb_plain = vec![0u8; crate::MAX_OBJECT_SIZE + 1024]; + let cap = crate::MIN_MAX_OBJECT_SIZE; + let bomb_plain = vec![0u8; cap + 1024]; let compressed = zstd::encode_all(std::io::Cursor::new(&bomb_plain[..]), COMPRESSION_LEVEL).unwrap(); assert!( @@ -659,8 +711,11 @@ mod tests { pack.extend_from_slice(&(compressed.len() as u32).to_le_bytes()); pack.extend_from_slice(&compressed); - let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); - assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + let err = read_object_at_bounded(&pack, 0, &id, cap).unwrap_err(); + assert!( + format!("{err}").contains("decompressed object exceeds"), + "got {err:?}" + ); } #[test] @@ -668,7 +723,7 @@ mod tests { // ADR-0039's explicit scope: "the cap is transfer-path only, not a // local-commit limit... read_object_at's local already-verified // on-disk pack is untouched" — a locally-committed blob legitimately - // larger than MAX_OBJECT_SIZE ("committable but not transferable") + // larger than the object cap ("committable but not transferable") // must still round-trip through `sc gc`'s repack, since `write_pack` // performs no size validation and a cap here would surface as // silent, permanent unreadability on the next read. This is the @@ -676,7 +731,7 @@ mod tests { // briefly did in an earlier, incorrect fix for the zstd-bomb // finding), this test catches it directly, without relying only on // the doc-comment contract. - let big = Object::blob(vec![7u8; crate::MAX_OBJECT_SIZE + 4096]); + let big = Object::blob(vec![7u8; crate::DEFAULT_MAX_OBJECT_SIZE + 4096]); let (id, bytes) = enc(&big); let (pack, idx) = build_pack(&[(id, bytes.clone())]).unwrap(); let e = parse_index(&idx).unwrap().pop().unwrap(); diff --git a/crates/core/tests/max_object_size.rs b/crates/core/tests/max_object_size.rs new file mode 100644 index 00000000..bbd670c3 --- /dev/null +++ b/crates/core/tests/max_object_size.rs @@ -0,0 +1,24 @@ +//! The process-wide object cap is set once. Own test binary so the global is +//! not shared with the unit tests (which rely on the default). + +use scl_core::{ + max_object_size, set_max_object_size, DEFAULT_MAX_OBJECT_SIZE, MAX_MAX_OBJECT_SIZE, + MIN_MAX_OBJECT_SIZE, +}; + +#[test] +fn cap_is_set_once_validated_and_then_fixed() { + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + // Out-of-range values are refused and leave the default in place. + assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); + assert!(set_max_object_size(MAX_MAX_OBJECT_SIZE + 1).is_err()); + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + + set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); + assert_eq!(max_object_size(), 2 * MIN_MAX_OBJECT_SIZE); + // Same value again is a no-op; a different one is refused. + set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); + let err = set_max_object_size(MAX_MAX_OBJECT_SIZE).unwrap_err(); + assert!(err.contains("already fixed"), "got {err}"); + assert_eq!(max_object_size(), 2 * MIN_MAX_OBJECT_SIZE); +} diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index 306928fd..ede72333 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -69,20 +69,24 @@ pub struct BucketTransport { } /// Untrusted-length guard (P28 parity): refuse any WAL metadata value — -/// manifest, log entry, idx — larger than `MAX_OBJECT_SIZE` before decoding. +/// manifest, log entry, idx — larger than the object cap before decoding. /// Pack bodies are exempt from this particular guard (they may legitimately /// exceed it), but are not unguarded: `object_bytes` reads a pack body via /// `scl_core::pack::read_object_at_bounded` (not the unbounded /// `read_object_at` — that path is reserved for `Store`'s own /// already-verified on-disk packs per ADR-0039's explicit trust split), /// which caps both the compressed record length and the decompressed output -/// at `MAX_OBJECT_SIZE`, mirroring `parse_pack_reader`'s bounded decode. A +/// at the object cap, mirroring `parse_pack_reader`'s bounded decode. A /// hostile bucket pack therefore cannot mount a decompression-bomb DoS /// against `get_object`/`get_pack`. fn capped(what: &str, bytes: Vec) -> Result> { - if bytes.len() > scl_core::MAX_OBJECT_SIZE { + capped_to(what, bytes, scl_core::max_object_size()) +} + +fn capped_to(what: &str, bytes: Vec, max: usize) -> Result> { + if bytes.len() > max { return Err(Error::Wal(format!( - "{what} exceeds MAX_OBJECT_SIZE (256 MiB)" + "{what} exceeds the {max}-byte object size limit" ))); } Ok(bytes) @@ -322,7 +326,7 @@ impl BucketTransport { *cache = Some((hash.clone(), bytes)); } let (_, pack) = cache.as_ref().unwrap(); - Ok(read_object_at_bounded(pack, offset, id, scl_core::MAX_OBJECT_SIZE)?.encode()) + Ok(read_object_at_bounded(pack, offset, id, scl_core::max_object_size())?.encode()) } /// Upload one pack (+ its idx) content-addressed by BLAKE3 of the pack @@ -663,6 +667,15 @@ impl Transport for BucketTransport { #[cfg(test)] mod tests { + #[test] + fn capped_to_rejects_above_a_lowered_cap_and_accepts_at_a_raised_one() { + let v = vec![0u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; + let err = capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE).unwrap_err(); + assert!(format!("{err}").contains("object size limit"), "got {err}"); + assert!(capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE + 1).is_ok()); + assert!(capped_to("manifest", v, scl_core::MAX_MAX_OBJECT_SIZE).is_ok()); + } + use super::*; use crate::transport::Transport; use crate::walfmt::{ diff --git a/crates/repo/src/http_transport.rs b/crates/repo/src/http_transport.rs index a57cf7d4..66311290 100644 --- a/crates/repo/src/http_transport.rs +++ b/crates/repo/src/http_transport.rs @@ -985,7 +985,7 @@ pub fn serve_http( tls: TlsMode, store: Option<&str>, ) -> Result<()> { - crate::wire::validate_max_pack_size(limits.max_pack_size)?; + crate::wire::validate_max_pack_size(limits.max_pack_size, scl_core::max_object_size())?; let tls_config = resolve_tls(root, &tls)?; if !bind_is_allowed(addr, root, read_only, allow_public, tls_config.is_some())? { return Err(Error::InvalidArgument(format!( diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 06a4c11a..d8675b03 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -101,10 +101,12 @@ impl WireClient { // The client's own read cap tripping is NOT a server refusal: the reply // is already part-way through the stream, so report it as a protocol // violation that callers must not answer by splitting the batch. - wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP).map_err(|e| match e { - Error::PackTooLarge(m) => Error::Protocol(format!("by-id reply too large: {m}")), - other => other, - })?; + wire::read_pack_stream(&mut rw.0, out, get_objects_response_cap()).map_err( + |e| match e { + Error::PackTooLarge(m) => Error::Protocol(format!("by-id reply too large: {m}")), + other => other, + }, + )?; Ok(not_sent) } } @@ -188,15 +190,19 @@ impl Transport for WireClient { /// One round trip (P37, ADR-0047): the OK body lists the ids the server /// will not send, and a chunk stream with the rest follows. The reply - /// stream is capped (`GET_OBJECTS_RESPONSE_CAP`) so a hostile server + /// stream is capped (`get_objects_response_cap`) so a hostile server /// cannot make a by-id fetch spool without bound. fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { self.get_objects_flagged(ids, out, &mut false) } } -/// Most bytes a client accepts in one `GetObjects` reply stream (1 GiB). -const GET_OBJECTS_RESPONSE_CAP: u64 = 1 << 30; +/// Most bytes a client accepts in one `GetObjects` reply stream: four times +/// the per-object cap (1 GiB at the default), twice the server's own by-id +/// reply ceiling, so pack framing never trips it on a legitimate reply. +fn get_objects_response_cap() -> u64 { + scl_core::max_object_size() as u64 * 4 +} /// A [`Transport`] whose far end is a child process speaking the wire protocol /// on its stdio — `ssh sc serve --stdio ` for real remotes. diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index c2616bc9..b0b54441 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -82,20 +82,25 @@ pub trait Transport { } } -/// The most canonical object bytes one by-id reply carries (ADR-0048). It sits -/// well under the client's 1 GiB read cap so pack record headers and zstd -/// framing cannot push a legitimate reply past it. -const BY_ID_REPLY_CAP: u64 = 512 << 20; +/// The most canonical object bytes one by-id reply carries (ADR-0048): twice +/// the per-object cap (`object_cap`, the live [`scl_core::max_object_size`]; +/// 512 MiB at the default), so a reply always has room for one maximum-size +/// object plus pack record headers and zstd framing, and scales with an +/// operator-set cap. +fn by_id_reply_ceiling(object_cap: usize) -> u64 { + (object_cap as u64) * 2 +} /// The by-id reply limit for a server whose `--max-pack-size` is /// `max_pack_size` (0 means unlimited): the smaller of that and -/// [`BY_ID_REPLY_CAP`]. Shared by the wire server and [`LocalTransport`], so a -/// lazy clone of a local path cannot spool an unbounded reply either. -pub(crate) fn by_id_reply_cap(max_pack_size: u64) -> u64 { +/// [`by_id_reply_ceiling`]. Shared by the wire server and [`LocalTransport`], +/// so a lazy clone of a local path cannot spool an unbounded reply either. +pub(crate) fn by_id_reply_cap(max_pack_size: u64, object_cap: usize) -> u64 { + let ceiling = by_id_reply_ceiling(object_cap); if max_pack_size == 0 { - BY_ID_REPLY_CAP + ceiling } else { - max_pack_size.min(BY_ID_REPLY_CAP) + max_pack_size.min(ceiling) } } @@ -111,11 +116,12 @@ fn encoded_len(obj: &Object) -> u64 { } /// Refuse an object a receiver would reject anyway: `pack::parse_pack_reader` -/// fails any record whose canonical encoding exceeds `MAX_OBJECT_SIZE`. -fn check_servable_size(id: &ObjectId, len: u64) -> Result<()> { - if len > scl_core::MAX_OBJECT_SIZE as u64 { +/// fails any record whose canonical encoding exceeds the object cap `max` +/// (the live [`scl_core::max_object_size`]). +fn check_servable_size(id: &ObjectId, len: u64, max: usize) -> Result<()> { + if len > max as u64 { return Err(Error::PackTooLarge(format!( - "object {id} exceeds MAX_OBJECT_SIZE and cannot be transferred" + "object {id} exceeds the {max}-byte object size limit and cannot be transferred" ))); } Ok(()) @@ -160,7 +166,7 @@ impl LocalTransport { /// shared core of [`Transport::get_objects`] and `wire::serve`'s /// `GetObjects` arm. Duplicate ids are served once. `max_bytes` (0 = /// unlimited) bounds the summed canonical encoded size, and any single - /// object over `MAX_OBJECT_SIZE` is refused (every receiver would reject + /// object over the object cap is refused (every receiver would reject /// it). Both fail with `PackTooLarge` before any pack byte is written. /// Blobs and sealed objects are measured without re-encoding them. pub(crate) fn build_objects_tempfile( @@ -182,7 +188,7 @@ impl LocalTransport { Ok(obj) if refused_by_id(&obj) => not_sent.push(*id), Ok(obj) => { let size = encoded_len(&obj); - check_servable_size(id, size)?; + check_servable_size(id, size, scl_core::max_object_size())?; total = total.saturating_add(size); if max_bytes > 0 && total > max_bytes { return Err(Error::PackTooLarge(format!( @@ -408,7 +414,7 @@ pub(crate) fn write_ids_to_temp_pack( /// Two defenses now compose here (updated for P28 Task 2). First, /// `parse_pack_reader` itself hard-rejects any record whose compressed /// length prefix, or whose decompressed zstd output, exceeds -/// `MAX_OBJECT_SIZE` (256 MiB) — an absolute per-record cap enforced +/// the object cap (`scl_core::max_object_size`, default 256 MiB) — an absolute per-record cap enforced /// regardless of source. Second, and independently, both passes here read /// the *already-fully-spilled* temp file rather than the live connection /// (whether it was spilled by `Transport::put_pack`'s own bounded copy from @@ -416,7 +422,7 @@ pub(crate) fn write_ids_to_temp_pack( /// total bytes any record's length prefix can possibly claim is also /// bounded by the file's own size on disk, which was itself bounded by /// however many bytes the sender actually sent. So a hostile length prefix -/// is now rejected outright by the `MAX_OBJECT_SIZE` cap, and even absent +/// is now rejected outright by the object cap, and even absent /// that cap could never claim more than the attacker already transferred — /// both defenses hold. Every call site in this codebase (both here and /// `wire::serve`'s own dispatch) reads a spilled file, never the live @@ -537,7 +543,8 @@ impl Transport for LocalTransport { crate::wire::MAX_GET_OBJECTS_IDS ))); } - let (not_sent, guard) = self.build_objects_tempfile(ids, by_id_reply_cap(0))?; + let (not_sent, guard) = + self.build_objects_tempfile(ids, by_id_reply_cap(0, scl_core::max_object_size()))?; let mut f = std::fs::File::open(guard.path())?; std::io::copy(&mut f, out)?; Ok(not_sent) @@ -730,14 +737,20 @@ mod tests { assert_eq!(encoded_len(o), o.encode().len() as u64, "{}", o.kind_name()); } let id = blob.id(); - // The receiver rejects a canonical encoding above MAX_OBJECT_SIZE, and - // the encoding of a blob is its payload plus one tag byte. - let max = scl_core::MAX_OBJECT_SIZE as u64; - assert!(check_servable_size(&id, max).is_ok()); - assert!(matches!( - check_servable_size(&id, max + 1), - Err(Error::PackTooLarge(_)) - )); + // The receiver rejects a canonical encoding above the object cap, and + // the encoding of a blob is its payload plus one tag byte. Lowered and + // raised caps move the boundary. + for max in [ + scl_core::MIN_MAX_OBJECT_SIZE, + scl_core::DEFAULT_MAX_OBJECT_SIZE, + scl_core::MAX_MAX_OBJECT_SIZE, + ] { + assert!(check_servable_size(&id, max as u64, max).is_ok()); + assert!(matches!( + check_servable_size(&id, max as u64 + 1, max), + Err(Error::PackTooLarge(_)) + )); + } assert_eq!(encoded_len(&Object::blob(vec![0u8; 10])), 11); } @@ -1109,13 +1122,20 @@ mod tests { } #[test] - fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_512_mib() { + fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_twice_the_object_cap() { const MIB: u64 = 1 << 20; + let default = scl_core::DEFAULT_MAX_OBJECT_SIZE; // 0 means "unlimited" for --max-pack-size, but a by-id reply is still - // bounded so the client's read cap always covers it. - assert_eq!(by_id_reply_cap(0), 512 * MIB); - assert_eq!(by_id_reply_cap(64 * MIB), 64 * MIB); - assert_eq!(by_id_reply_cap(16 << 30), 512 * MIB); + // bounded: 2 x the object cap, 512 MiB at the default. + assert_eq!(by_id_reply_cap(0, default), 512 * MIB); + assert_eq!(by_id_reply_cap(64 * MIB, default), 64 * MIB); + assert_eq!(by_id_reply_cap(16 << 30, default), 512 * MIB); + // The ceiling follows a lowered or raised object cap. + assert_eq!(by_id_reply_cap(0, scl_core::MIN_MAX_OBJECT_SIZE), 2 * MIB); + assert_eq!( + by_id_reply_cap(16 << 30, scl_core::MAX_MAX_OBJECT_SIZE), + 2048 * MIB + ); } /// A repo with a stopped merge, pick or rebase is completed against its diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index bae08a5c..0c32f618 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -76,22 +76,25 @@ pub const CHUNK_SIZE: usize = 1 << 20; /// sender, the client's `PutPack` sender) call this, so setting the var /// before either side starts affects the whole round trip. pub fn pack_chunk_size() -> usize { - chunk_from(std::env::var("SC_PACK_CHUNK").ok().as_deref()) + chunk_from( + std::env::var("SC_PACK_CHUNK").ok().as_deref(), + scl_core::max_object_size(), + ) } /// The `SC_PACK_CHUNK` decision, pure so it can be tested without touching the /// process environment: an unset, unparsable or zero value is the default, and /// anything larger than a frame can carry is clamped. A chunk frame is the /// `ST_PACK_CHUNK` tag byte plus the data, and the receiver refuses a frame -/// above `MAX_OBJECT_SIZE`, so the data is capped one byte below it; an +/// above the object cap, so the data is capped one byte below it; an /// oversized setting would otherwise fail every large transfer with a /// confusing error. -fn chunk_from(value: Option<&str>) -> usize { +fn chunk_from(value: Option<&str>, object_cap: usize) -> usize { value .and_then(|v| v.parse::().ok()) .filter(|&n| n > 0) .unwrap_or(CHUNK_SIZE) - .min(scl_core::MAX_OBJECT_SIZE - 1) + .min(object_cap - 1) } // Wire error codes. Typed errors that sync logic relies on get their own code; @@ -398,6 +401,12 @@ pub fn read_frame_opt(r: &mut impl Read) -> Result>> { } fn read_frame_inner(r: &mut impl Read) -> Result>> { + read_frame_max(r, scl_core::max_object_size()) +} + +/// [`read_frame_opt`] with an explicit frame-length cap `max` (the live +/// [`scl_core::max_object_size`] in production). +fn read_frame_max(r: &mut impl Read, max: usize) -> Result>> { let mut len_bytes = [0u8; 4]; let mut filled = 0; while filled < 4 { @@ -411,13 +420,14 @@ fn read_frame_inner(r: &mut impl Read) -> Result>> { filled += n; } let len = u32::from_be_bytes(len_bytes) as usize; - if len > scl_core::MAX_OBJECT_SIZE { + if len > max { return Err(Error::Protocol(format!( - "frame length {len} exceeds MAX_OBJECT_SIZE" + "frame length {len} exceeds the {max}-byte object size limit" ))); } - let mut buf = vec![0u8; len]; - r.read_exact(&mut buf).map_err(|e| match e.kind() { + // The length is the peer's claim: grow with the bytes that actually + // arrive instead of allocating `len` up front. + let buf = scl_core::read_exact_vec(r, len).map_err(|e| match e.kind() { // A read timeout is the peer going quiet, not a malformed frame. std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut => Error::Io(e), _ => Error::Protocol(format!("read inside {len}-byte frame body failed: {e}")), @@ -729,14 +739,14 @@ impl Default for WirePolicy { } /// Validate an operator-supplied `--max-pack-size` value. `0` (unlimited) is -/// always fine; a nonzero cap below [`scl_core::MAX_OBJECT_SIZE`] can never -/// fit even one object and is rejected as a misconfiguration rather than -/// silently failing every push at runtime. -pub fn validate_max_pack_size(max: u64) -> Result<()> { - if max != 0 && max < scl_core::MAX_OBJECT_SIZE as u64 { +/// always fine; a nonzero cap below `object_cap` (the live +/// [`scl_core::max_object_size`]) can never fit even one object and is +/// rejected as a misconfiguration rather than silently failing every push at +/// runtime. +pub fn validate_max_pack_size(max: u64, object_cap: usize) -> Result<()> { + if max != 0 && max < object_cap as u64 { return Err(Error::InvalidArgument(format!( - "--max-pack-size {max} is below MAX_OBJECT_SIZE ({}); a cap that cannot fit one object is a misconfiguration", - scl_core::MAX_OBJECT_SIZE + "--max-pack-size {max} is below the max object size ({object_cap}); a cap that cannot fit one object is a misconfiguration" ))); } Ok(()) @@ -1091,7 +1101,10 @@ fn serve_session( let result = match &transport { ServeTransport::Local(t) => t.build_objects_tempfile( &ids, - crate::transport::by_id_reply_cap(policy.max_pack_size), + crate::transport::by_id_reply_cap( + policy.max_pack_size, + scl_core::max_object_size(), + ), ), ServeTransport::Bucket { .. } => Err(Error::InvalidArgument( "bucket remotes do not support by-id object fetch".into(), @@ -1386,16 +1399,54 @@ mod tests { #[test] fn frame_over_cap_rejected() { - // A frame header alone (no body) claiming a length past - // MAX_OBJECT_SIZE must be rejected before any allocation of the - // (fictitious) body — feeding only the 4-byte header proves this: - // if the implementation tried to `vec![0u8; len]` and then - // `read_exact`, it would hang/OOM here rather than erroring. - let over = (scl_core::MAX_OBJECT_SIZE + 1) as u32; + // A frame header alone (no body) claiming a length past the cap must + // be rejected before the (fictitious) body is read or allocated: + // feeding only the 4-byte header proves this, since reading first + // would hit EOF and report a different error. + let cap = scl_core::MIN_MAX_OBJECT_SIZE; + let over = (cap + 1) as u32; let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); - let err = read_frame_opt(&mut r).unwrap_err(); + let err = read_frame_max(&mut r, cap).unwrap_err(); assert!( - matches!(err, crate::error::Error::Protocol(_)), + matches!(&err, crate::error::Error::Protocol(m) if m.contains("exceeds")), + "got {err:?}" + ); + // The production entry point is the same guard at the live cap. + let over = (scl_core::max_object_size() + 1) as u32; + let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); + assert!(matches!( + read_frame_opt(&mut r), + Err(crate::error::Error::Protocol(m)) if m.contains("exceeds") + )); + } + + #[test] + fn frame_cap_lowered_rejects_and_raised_accepts_the_same_frame() { + let body = vec![7u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; + let mut framed = Vec::new(); + write_frame(&mut framed, &body).unwrap(); + let low = scl_core::MIN_MAX_OBJECT_SIZE; + assert!(read_frame_max(&mut &framed[..], low).is_err()); + assert_eq!( + read_frame_max(&mut &framed[..], low + 1).unwrap().unwrap(), + body + ); + assert_eq!( + read_frame_max(&mut &framed[..], low * 4).unwrap().unwrap(), + body + ); + } + + #[test] + fn frame_claiming_the_cap_without_a_body_fails_without_a_cap_sized_allocation() { + // At the ceiling, a 4-byte header can claim 1 GiB. The body is read + // incrementally, so a peer that sends nothing gets a short-read + // error, not a pre-allocated buffer. + let cap = scl_core::MAX_MAX_OBJECT_SIZE; + let mut r = std::io::Cursor::new((cap as u32).to_be_bytes().to_vec()); + let err = read_frame_max(&mut r, cap).unwrap_err(); + assert!( + matches!(&err, crate::error::Error::Protocol(m) if m.contains("frame body")), "got {err:?}" ); } @@ -1875,31 +1926,54 @@ mod tests { #[test] fn chunk_from_defaults_and_clamps_to_the_object_cap() { - assert_eq!(chunk_from(None), CHUNK_SIZE); - assert_eq!(chunk_from(Some("")), CHUNK_SIZE); - assert_eq!(chunk_from(Some("0")), CHUNK_SIZE); - assert_eq!(chunk_from(Some("junk")), CHUNK_SIZE); - assert_eq!(chunk_from(Some("4096")), 4096); + let max = scl_core::DEFAULT_MAX_OBJECT_SIZE; + assert_eq!(chunk_from(None, max), CHUNK_SIZE); + assert_eq!(chunk_from(Some(""), max), CHUNK_SIZE); + assert_eq!(chunk_from(Some("0"), max), CHUNK_SIZE); + assert_eq!(chunk_from(Some("junk"), max), CHUNK_SIZE); + assert_eq!(chunk_from(Some("4096"), max), 4096); // A chunk frame carries a tag byte too: data at the cap would make a // frame one byte over what the receiver accepts. - let cap = scl_core::MAX_OBJECT_SIZE - 1; + let cap = max - 1; assert_eq!( - chunk_from(Some(&(scl_core::MAX_OBJECT_SIZE * 8).to_string())), + chunk_from(Some(&(max * 8).to_string()), max), cap, "an oversized setting is clamped, not passed through" ); - assert_eq!( - chunk_from(Some(&scl_core::MAX_OBJECT_SIZE.to_string())), - cap - ); - assert_eq!(chunk_from(Some(&cap.to_string())), cap); + assert_eq!(chunk_from(Some(&max.to_string()), max), cap); + assert_eq!(chunk_from(Some(&cap.to_string()), max), cap); + } + + #[test] + fn chunk_clamp_follows_a_lowered_and_a_raised_cap() { + // Lowered: the 1 MiB default chunk (data) no longer fits with its tag + // byte, so it clamps to cap - 1 and the frame is exactly the cap. + let low = scl_core::MIN_MAX_OBJECT_SIZE; + let chunk = chunk_from(None, low); + assert_eq!(chunk, low - 1); + let mut frame = vec![ST_PACK_CHUNK]; + frame.extend(vec![0u8; chunk]); + let mut framed = Vec::new(); + write_frame(&mut framed, &frame).unwrap(); + assert!(read_frame_max(&mut &framed[..], low).unwrap().is_some()); + assert!(read_frame_max(&mut &framed[..], low - 1).is_err()); + // Raised: a big explicit chunk is honoured up to the new cap - 1. + let high = scl_core::MAX_MAX_OBJECT_SIZE; + assert_eq!(chunk_from(Some("536870912"), high), 536_870_912); + assert_eq!(chunk_from(Some(&high.to_string()), high), high - 1); } #[test] fn validate_max_pack_size_floor() { - assert!(validate_max_pack_size(0).is_ok()); - assert!(validate_max_pack_size(scl_core::MAX_OBJECT_SIZE as u64).is_ok()); - assert!(validate_max_pack_size(1024).is_err()); + let max = scl_core::DEFAULT_MAX_OBJECT_SIZE; + assert!(validate_max_pack_size(0, max).is_ok()); + assert!(validate_max_pack_size(max as u64, max).is_ok()); + assert!(validate_max_pack_size(1024, max).is_err()); + // The floor tracks the object cap: lowered admits what a default + // cap refuses, raised refuses what a default cap admits. + let low = scl_core::MIN_MAX_OBJECT_SIZE; + assert!(validate_max_pack_size(low as u64, low).is_ok()); + assert!(validate_max_pack_size(max as u64, scl_core::MAX_MAX_OBJECT_SIZE).is_err()); } /// An oversized push on a read-only connection drains at most ro_drain_cap diff --git a/demo/run_limits_demo.sh b/demo/run_limits_demo.sh index bde0259e..6b568281 100755 --- a/demo/run_limits_demo.sh +++ b/demo/run_limits_demo.sh @@ -2,7 +2,7 @@ # P31 listener-limits demo: `sc serve --http`'s three new operator knobs — # `--max-connections` (503-style busy shed at the limit), `--timeout` # (session idle read+write timeout), and `--max-pack-size` (floor: -# MAX_OBJECT_SIZE, 256 MiB) — layered on P26/P29's `sc serve --http`. +# the max object size, 256 MiB) — layered on P26/P29's `sc serve --http`. # # Honesty note on scope: a connection that never sends its opening is # reaped by the FIXED 30s opening timeout, not by `--timeout` (which @@ -183,7 +183,7 @@ SERVER_PID="" ! nc -z 127.0.0.1 "$PORT2" 2>/dev/null || fail "second server still accepting connections after kill" if "$SC" serve --http "127.0.0.1:$PORT2" --max-pack-size 1024 "$A" 2>"$W/err-cap.txt"; then - fail "server should have refused to start with --max-pack-size below MAX_OBJECT_SIZE" + fail "server should have refused to start with --max-pack-size below the max object size" fi grep -qi "max-pack-size" "$W/err-cap.txt" \ || fail "expected a --max-pack-size floor error, got: $(cat "$W/err-cap.txt")" diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index ba8d9ed0..81bec846 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -305,7 +305,7 @@ too, not just the src-control-side metadata. it. The incoming-pack spool used no aggregate size bound — only ADR-0039's per-frame/per-record caps applied — so a single oversized or endless pack could exhaust server disk on **either** transport. `--max-pack-size` - (default 16 GiB, 0 = unlimited, floor 256 MiB = `MAX_OBJECT_SIZE`) now + (default 16 GiB, 0 = unlimited, floor = the max object size, 256 MiB by default) now counts the running total and aborts mid-stream past the cap. This includes the **read-only drain path**: the pre-`EC_READONLY` drain that keeps a connection in sync when a read-only-scoped client streams a pack the server @@ -349,7 +349,7 @@ too, not just the src-control-side metadata. (`walfmt.rs`): unknown versions, bad magic, and any length that would overrun the buffer are refused outright rather than best-effort parsed. Untrusted-length caps apply throughout: manifest/log/idx bytes are capped - at `MAX_OBJECT_SIZE` before decode, and pack bodies are read through + at the max object size before decode, and pack bodies are read through `scl_core::pack::read_object_at_bounded` (not the unbounded `read_object_at` the trusted local `Store` uses for its own already- verified on-disk packs, per ADR-0039's explicit trust split) — capping @@ -409,20 +409,34 @@ too, not just the src-control-side metadata. read is capped at 1 GiB. - **Server cost is bounded per request:** 1024 ids (checked while decoding, before any id is read), the server's `--max-pack-size` over the summed - encoded size, and `MAX_OBJECT_SIZE` per object. + encoded size, and the max object size per object. - **Laziness is a local opt-in.** `.sc/promisor-lazy` is never committed or transferred; without it a partial clone never dials the remote after clone, and `SC_OFFLINE=1` forces that per process. ## Untrusted-input hardening (DoS) — ADR-0039 -- A single `MAX_OBJECT_SIZE` (256 MiB) caps every untrusted length: wire frames, +- A single max object size caps every untrusted length: wire frames, pack-record compressed length, the zstd *decompressed* output (decompression-bomb - guard), and object-decode collection counts. **Accepted boundary:** the cap is a - *transfer-path* guard — a locally-committed blob larger than 256 MiB commits fine - but cannot then be transferred; and the wire frame-length header can still - allocate up to the cap before a chunk boundary is enforced (a deferred - hostile-peer hardening item). + guard), bucket WAL metadata, by-id reply objects, the `SC_PACK_CHUNK` clamp, + and object-decode collection counts. The default is 256 MiB. An operator can + lower or raise it with `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` + (#213), within a compile-time range of 1 MiB to 1 GiB; a value outside that + range aborts startup rather than being clamped. A guard checks the claimed + length before allocating, and frame bodies and pack records are then read + incrementally, so a hostile header cannot buy a cap-sized allocation without + sending the bytes. **Accepted boundaries:** + - The cap is a *transfer-path* guard. A locally-committed blob above it + commits fine but cannot be transferred. + - **Both ends must agree.** There is no negotiation and no protocol bump. An + object above the receiver's cap is refused there, loudly. A server raised + to 1 GiB accepts a peer's gigabyte claim only if that peer sends the bytes, + but then holds that object in memory: raising the cap raises per-connection + memory by up to the new cap (times `--max-connections`). Lowering it below + a repository's largest object makes that repository unpushable until the + cap is restored. + - The ceiling (1 GiB) is compile-time, so no configuration can make a length + claim unbounded. ## Reporting diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index ba27e6aa..e61b05ac 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -128,3 +128,54 @@ same-user processes, crash dumps, and shell wrappers, and no amount of zeroizing the parent's buffer changes that. Accepted boundary: the child-env copy is fundamental and un-zeroizable; fd/stdin injection stays deferred. 1 pinned regression test. + +## Addendum (#213): the object cap is operator-configurable + +`MAX_OBJECT_SIZE` was a compile-time constant. It is now a process-wide +setting read through one accessor, `scl_core::max_object_size()`, with a +default of 256 MiB. Decisions: + +1. **Lower or raise, inside a compile-time range.** Floor 1 MiB + (`MIN_MAX_OBJECT_SIZE`), ceiling 1 GiB (`MAX_MAX_OBJECT_SIZE`). A value + outside the range, or one that does not parse, aborts startup with a clear + error. It is never clamped. The ceiling bounds what a hostile peer can + make a receiver commit to for any one object whatever the operator sets. +2. **Server flag and client env, no negotiation.** `sc serve + --max-object-size ` and, for every `sc` process, + `SC_MAX_OBJECT_SIZE` (same parse, same bounds). The flag wins over the + env. The setting is fixed once at startup (`set_max_object_size`; a second, + different value is an error). There is no handshake field and no + `PROTOCOL_VERSION` bump: matching the two ends is an operator + responsibility, because an object larger than the receiver's cap fails at + the receiver. The failure is loud and typed, never silent truncation. +3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`); + pack record compressed length and decompressed output + (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket WAL + metadata and pack bodies (`bucket_transport::capped_to`, + `object_bytes`); server-side servable-size check + (`check_servable_size`); the `SC_PACK_CHUNK` clamp (`cap - 1`, a chunk + frame carries a tag byte); the `--max-pack-size` floor + (`validate_max_pack_size`). Two limits that were independent constants are + now derived: the by-id reply ceiling is `2 x cap` (512 MiB at the default; + still `min(--max-pack-size, ceiling)`) and the client's by-id reply read + cap is `4 x cap` (1 GiB at the default), so both keep room for one + maximum-size object plus framing at any setting. Each guard takes the cap + as a parameter below the public entry point so tests can drive a lowered + and a raised cap without touching the process-global. +4. **Allocation follows bytes received.** The length check still runs before + any allocation, but the frame body and pack record payload are now read + through `scl_core::read_exact_vec`, which grows the buffer as bytes arrive + instead of pre-allocating the claimed length. With the ceiling at 1 GiB, a + four-byte header no longer buys a gigabyte-sized allocation. This closes the + "frame header can allocate up to the cap" item in THREAT-MODEL.md. + +For `ssh://` remotes the client builds the remote `sc serve --stdio` command +itself and ssh does not forward the environment, so the server end of an ssh +remote gets a non-default cap only from the server's own environment (a +wrapper script, `sshd` `SetEnv`, or a forced command passing +`--max-object-size`). + +Unchanged: the cap is still a transfer-path guard, not a local-commit limit, +so lowering it can leave a repository holding committed objects it cannot +push; `read_object_at` (the trusted on-disk path) stays uncapped. The +desktop app does not read the env var; it does no transfers. From 2ac8838fc670c0471f9ff46ababa48e591501427 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 09:02:27 +0300 Subject: [PATCH 02/11] fix(repo): scale the store budget with the object cap, document memory bounds (#213) Review findings on #227: a raised cap admitted frames the fixed 512 MiB store budget then refused; THREAT-MODEL still said the by-id read cap is 1 GiB and understated per-connection memory at a raised cap. --- crates/repo/src/repo.rs | 34 ++++++++++++++++++++--- crates/repo/src/transport.rs | 10 ++++--- docs/THREAT-MODEL.md | 12 ++++++-- docs/adr/0039-security-hardening-sweep.md | 8 ++++++ 4 files changed, 53 insertions(+), 11 deletions(-) diff --git a/crates/repo/src/repo.rs b/crates/repo/src/repo.rs index 5e960e96..b8937651 100644 --- a/crates/repo/src/repo.rs +++ b/crates/repo/src/repo.rs @@ -13,7 +13,19 @@ use crate::refs; use crate::worktree::{self, Diff}; const DEFAULT_BRANCH: &str = "main"; -pub(crate) const DEFAULT_BUDGET: usize = 512 * 1024 * 1024; +const DEFAULT_BUDGET: usize = 512 * 1024 * 1024; + +/// Resident-blob budget for a repo or transport store: 512 MiB, or twice the +/// object cap if that is larger, so one maximum-size object always fits (a +/// blob bigger than the whole budget would fail with `BudgetExceeded`). Equal +/// to the old fixed 512 MiB at the default 256 MiB cap. +pub(crate) fn default_budget() -> usize { + budget_for(scl_core::max_object_size()) +} + +fn budget_for(object_cap: usize) -> usize { + DEFAULT_BUDGET.max(2 * object_cap) +} /// A working-tree file headed for encryption at commit time: /// `(path, plaintext bytes, file mode, recipient pubkeys)`. @@ -44,17 +56,17 @@ impl Repo { std::fs::create_dir_all(layout.objects_dir())?; std::fs::create_dir_all(layout.refs_heads_dir())?; refs::write_head(&layout, DEFAULT_BRANCH)?; - Self::open_layout(layout, DEFAULT_BUDGET) + Self::open_layout(layout, default_budget()) } /// Open an existing repo by discovering `.sc/` at or above `start`. pub fn open(start: impl AsRef) -> Result { let layout = Layout::discover(start)?; - Self::open_layout(layout, DEFAULT_BUDGET) + Self::open_layout(layout, default_budget()) } /// Open an existing repo with an explicit memory budget for the shared - /// object cache (bytes). `open` uses `DEFAULT_BUDGET`; workspace sessions + /// object cache (bytes). `open` uses `default_budget()`; workspace sessions /// (`sc work --budget-mb`) size the cache to the fleet they fork. pub fn open_with_budget(start: impl AsRef, budget_bytes: usize) -> Result { let layout = Layout::discover(start)?; @@ -2678,6 +2690,20 @@ fn stop_write_blocker( #[cfg(test)] mod tests { + #[test] + fn store_budget_always_fits_one_maximum_size_object() { + // Unchanged at the default cap; grows with a raised cap. + assert_eq!(budget_for(scl_core::DEFAULT_MAX_OBJECT_SIZE), 512 << 20); + assert_eq!(budget_for(scl_core::MIN_MAX_OBJECT_SIZE), 512 << 20); + for cap in [ + scl_core::DEFAULT_MAX_OBJECT_SIZE, + scl_core::MAX_MAX_OBJECT_SIZE, + ] { + assert!(budget_for(cap) >= cap); + } + assert_eq!(budget_for(scl_core::MAX_MAX_OBJECT_SIZE), 2 << 30); + } + use super::*; fn tmp_root(tag: &str) -> std::path::PathBuf { diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index b0b54441..88b30a49 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -154,7 +154,7 @@ impl LocalTransport { } // Match the repo's store budget so a single large object never fails to resolve // (a blob bigger than the whole budget would BudgetExceed). - let store = Store::open_persistent(layout.objects_dir(), crate::repo::DEFAULT_BUDGET)?; + let store = Store::open_persistent(layout.objects_dir(), crate::repo::default_budget())?; Ok(LocalTransport { layout, store: RefCell::new(store), @@ -897,9 +897,11 @@ mod tests { let id = blob.id(); let expected = blob.encode(); { - let mut s = - scl_core::Store::open_persistent(layout.objects_dir(), crate::repo::DEFAULT_BUDGET) - .unwrap(); + let mut s = scl_core::Store::open_persistent( + layout.objects_dir(), + crate::repo::default_budget(), + ) + .unwrap(); s.put(Object::blob(vec![0xAB; (1 << 20) + 4096])).unwrap(); } let t = LocalTransport::open(&layout.root).unwrap(); diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 81bec846..61f07329 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -406,7 +406,8 @@ too, not just the src-control-side metadata. nothing both sent and refused, nothing omitted) before any object is written. A hostile remote can at worst answer "not sent"; it cannot plant an object under another id or smuggle in extras. The reply stream a client will - read is capped at 1 GiB. + read is capped at four times the max object size (1 GiB at the default, + up to 4 GiB with the cap raised to its 1 GiB ceiling). - **Server cost is bounded per request:** 1024 ids (checked while decoding, before any id is read), the server's `--max-pack-size` over the summed encoded size, and the max object size per object. @@ -431,8 +432,13 @@ too, not just the src-control-side metadata. - **Both ends must agree.** There is no negotiation and no protocol bump. An object above the receiver's cap is refused there, loudly. A server raised to 1 GiB accepts a peer's gigabyte claim only if that peer sends the bytes, - but then holds that object in memory: raising the cap raises per-connection - memory by up to the new cap (times `--max-connections`). Lowering it below + but then holds that object in memory. Raising the cap raises per-connection + memory: a pack record is buffered compressed and then decompressed, and the + decoded object is a further copy, so budget a small multiple of the cap + (roughly 3x) per connection, times `--max-connections` (default 32). Size + `--max-connections` down when raising the cap. The repository's resident + blob budget rises with the cap (to twice the cap, never below 512 MiB) so one + maximum-size object can still be admitted. Lowering it below a repository's largest object makes that repository unpushable until the cap is restored. - The ceiling (1 GiB) is compile-time, so no configuration can make a length diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index e61b05ac..d6b05148 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -169,6 +169,14 @@ default of 256 MiB. Decisions: four-byte header no longer buys a gigabyte-sized allocation. This closes the "frame header can allocate up to the cap" item in THREAT-MODEL.md. +The resident-blob budget of a repo or transport store was a fixed 512 MiB; a +raised cap would have admitted frames the store then refused with +`BudgetExceeded`. It is now `max(512 MiB, 2 x cap)` (`repo::default_budget`), +unchanged at the default. Raising the cap also raises the memory a peer can +make a receiver hold per connection: a pack record is held compressed, then +decompressed, then decoded, so plan for roughly 3x the cap times +`--max-connections` (THREAT-MODEL.md). + For `ssh://` remotes the client builds the remote `sc serve --stdio` command itself and ssh does not forward the environment, so the server end of an ssh remote gets a non-default cap only from the server's own environment (a From eeb8d616fa066126c860874689c4c37a289fb266 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 09:30:05 +0300 Subject: [PATCH 03/11] fix(core,repo,cli): stepped reads, 2 MiB floor, peer-mismatch hint, fixed spool caps (#213) - read_limited/read_exact_vec grow by reserve_exact steps (<= 8 MiB), no doubling; zstd output uses the same reader. - Raise the floor to 2 MiB (a default chunk frame is 1 MiB + tag byte); frame and pack-record errors name SC_MAX_OBJECT_SIZE / --max-object-size. - By-id reply ceiling and client read cap are constants, not derived from the local cap; fix the misleading message. - max_object_size() fixes the value on first read. - PutPack arm replies and closes on a frame-level Protocol error. - Bucket WAL metadata cap stays at the fixed 256 MiB default. - Empty SC_MAX_OBJECT_SIZE counts as unset. - Correct the operator sizing numbers in ADR-0039 and THREAT-MODEL.md. --- .claude/skills/sc-cli-reference/SKILL.md | 4 +- CLAUDE.md | 4 +- ROADMAP.md | 2 +- crates/cli/src/main.rs | 7 +- crates/cli/tests/max_object_size.rs | 142 ++++++++++++++---- crates/core/src/lib.rs | 121 ++++++++++++--- crates/core/src/pack.rs | 24 ++- crates/core/tests/max_object_size.rs | 9 +- .../core/tests/max_object_size_read_first.rs | 16 ++ crates/repo/src/bucket_transport.rs | 19 ++- crates/repo/src/stdio_transport.rs | 27 ++-- crates/repo/src/transport.rs | 42 ++---- crates/repo/src/wire.rs | 30 ++-- demo/run_limits_demo.sh | 2 +- docs/THREAT-MODEL.md | 40 +++-- docs/adr/0039-security-hardening-sweep.md | 59 +++++--- 16 files changed, 377 insertions(+), 171 deletions(-) create mode 100644 crates/core/tests/max_object_size_read_first.rs diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 4a49fcd7..9f71131f 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -274,9 +274,9 @@ cargo run --bin sc -- serve --http [--read-only] [--allow-public] # counted mid-stream abort -> EC_TOO_LARGE (P31) # --max-object-size : largest # single object accepted from a peer, default - # 256MiB, range 1MiB..1GiB, out of range aborts + # 256MiB, range 2MiB..1GiB, out of range aborts # startup (never clamped). Clients use env - # SC_MAX_OBJECT_SIZE (same parse, any command); + # SC_MAX_OBJECT_SIZE (same parse, any command; empty = unset); # flag beats env. No negotiation: both ends must # agree, a larger object fails at the receiver (#213) cargo run --bin sc -- serve token add --label --scope ro|rw # mint an diff --git a/CLAUDE.md b/CLAUDE.md index 427e6093..dc512370 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -142,7 +142,7 @@ the code, those win. | P25 | Streaming pack transfer: bounded RAM on both sides, `ST_PACK_CHUNK` frames (`SC_PACK_CHUNK` tunes), two-pass atomic-after-verify ingest | [0035](docs/adr/0035-streaming-transfer.md) | | P26 | `sc+http://` transport: `sc serve --http`, thread-per-connection, status-line-before-handshake opening | [0036](docs/adr/0036-http-transport.md) | | P27 | Partial clone: `sc clone --filter `, `.sc/promisor`, `sc backfill [--all]`; wire `PROTOCOL_VERSION` 3 | [0037](docs/adr/0037-partial-clone.md) | -| P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 1 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | +| P28 | Security hardening: a max object size (default 256 MiB; `--max-object-size` / `SC_MAX_OBJECT_SIZE`, 2 MiB to 1 GiB) caps every untrusted transfer length; strict ref-name validation | [0039](docs/adr/0039-security-hardening-sweep.md) | | P29 | `sc serve --http` access control: bearer tokens (`sc serve token`), `--read-only`, fail-closed non-loopback bind gate | [0040](docs/adr/0040-sc-http-access-control.md) | | P30 | Sealed agent-session transcripts: `sc transcript attach/show/list/sign`, always-encrypted body, gc-coupled index | [0038](docs/adr/0038-agent-session-transcripts.md) | | P31 | Listener resource limits: `--max-connections`, `--timeout`, `--max-pack-size` (both transports), capped read-only drain, accept backoff | [0041](docs/adr/0041-listener-resource-limits.md) | @@ -164,7 +164,7 @@ transport-adjacent. The rest, imperatively: - **Wire protocol is version 5** (bumped from 4 in P37 for the additive `GetObjects` verb; 3 to 4 in P34 for the sealed-branch object kinds). The max object size (`scl_core::max_object_size()`, default 256 MiB; - `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 1 MiB to + `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every subsequent push/fetch/clone at the receiver. There is no negotiation: both diff --git a/ROADMAP.md b/ROADMAP.md index 8bbc3571..9d2fb8ab 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -805,7 +805,7 @@ scale-&-reach horizon): `sc work --with-secrets`, and a stdin variant. - **`--max-object-size` operator knob — shipped (#213, ADR-0039 addendum).** `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap - (default 256 MiB, range 1 MiB to 1 GiB, out of range aborts startup). No wire + (default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire negotiation: both ends must agree. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the object cap minus one (a chunk frame carries a tag byte too; an diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 034604da..47a0a9a0 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -361,7 +361,7 @@ enum Cmd { #[arg(long)] max_pack_size: Option, /// Largest single object accepted from a peer: bytes, or a number - /// with a KiB/MiB/GiB suffix. Default 256MiB, range 1MiB to 1GiB; + /// with a KiB/MiB/GiB suffix. Default 256MiB, range 2MiB to 1GiB; /// out-of-range values are refused at startup, never clamped. Clients /// set the same limit with `SC_MAX_OBJECT_SIZE`; both ends must /// agree, since a larger object fails at the receiver. (#213) @@ -935,11 +935,14 @@ struct DemoArgs { } /// Fix the process-wide object cap before any command runs: `sc serve -/// --max-object-size` wins over `SC_MAX_OBJECT_SIZE`, which wins over the +/// --max-object-size` wins over `SC_MAX_OBJECT_SIZE` (empty = unset), which wins over the /// default. An out-of-range or unparsable value aborts startup (#213). fn init_max_object_size(flag: Option<&str>) -> Result<()> { let (src, raw) = match (flag, std::env::var("SC_MAX_OBJECT_SIZE")) { (Some(f), _) => ("--max-object-size", f.to_string()), + // An empty variable (`SC_MAX_OBJECT_SIZE=`) means unset, so exporting + // it blank in a profile does not abort every command. + (None, Ok(e)) if e.trim().is_empty() => return Ok(()), (None, Ok(e)) => ("SC_MAX_OBJECT_SIZE", e), (None, Err(std::env::VarError::NotPresent)) => return Ok(()), (None, Err(e)) => anyhow::bail!("SC_MAX_OBJECT_SIZE: {e}"), diff --git a/crates/cli/tests/max_object_size.rs b/crates/cli/tests/max_object_size.rs index 90c76057..54ed7fc3 100644 --- a/crates/cli/tests/max_object_size.rs +++ b/crates/cli/tests/max_object_size.rs @@ -2,8 +2,9 @@ //! addendum): bounds are enforced at startup, and the cap is honoured by a //! real object transfer. -use std::path::Path; -use std::process::{Command, Output}; +use std::io::BufRead; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Output, Stdio}; fn sc(dir: &Path, envs: &[(&str, &str)], args: &[&str]) -> Output { let mut c = Command::new(env!("CARGO_BIN_EXE_sc")); @@ -16,7 +17,7 @@ fn sc(dir: &Path, envs: &[(&str, &str)], args: &[&str]) -> Output { c.output().expect("sc runs") } -fn tmp(tag: &str) -> std::path::PathBuf { +fn tmp(tag: &str) -> PathBuf { let d = std::env::temp_dir().join(format!("scl-cli-maxobj-{tag}-{}", std::process::id())); let _ = std::fs::remove_dir_all(&d); std::fs::create_dir_all(&d).unwrap(); @@ -27,12 +28,32 @@ fn stderr(o: &Output) -> String { String::from_utf8_lossy(&o.stderr).into_owned() } +/// A repo at `dir` with one committed `big.bin` of `len` incompressible bytes. +fn repo_with_big_file(dir: &Path, len: usize) -> Vec { + assert!(sc(dir, &[], &["init"]).status.success()); + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let big: Vec = (0..len) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + std::fs::write(dir.join("big.bin"), &big).unwrap(); + let o = sc(dir, &[("SC_AUTHOR", "t")], &["commit", "-m", "big"]); + assert!(o.status.success(), "commit is not capped: {}", stderr(&o)); + big +} + #[test] fn out_of_range_env_is_rejected_at_startup_not_clamped() { let root = tmp("env-bounds"); for bad in [ "0", - "1023KiB", + "2097151", + "1MiB", + "2047KiB", "1025MiB", "2GiB", "junk", @@ -50,9 +71,9 @@ fn out_of_range_env_is_rejected_at_startup_not_clamped() { "{bad}: rejected before any work" ); } - for good in ["1048576", "1MiB", "256mib", "1GiB"] { + for good in ["2097152", "2MiB", "256mib", "1GiB", "", " "] { let o = sc(&root, &[("SC_MAX_OBJECT_SIZE", good)], &["init"]); - assert!(o.status.success(), "{good}: {}", stderr(&o)); + assert!(o.status.success(), "{good:?}: {}", stderr(&o)); std::fs::remove_dir_all(root.join(".sc")).unwrap(); } std::fs::remove_dir_all(&root).unwrap(); @@ -62,7 +83,7 @@ fn out_of_range_env_is_rejected_at_startup_not_clamped() { fn serve_flag_is_bounds_checked_and_overrides_env() { let root = tmp("serve-flag"); assert!(sc(&root, &[], &["init"]).status.success()); - for bad in ["512", "2GiB", "x"] { + for bad in ["512", "1MiB", "2GiB", "x"] { let o = sc( &root, &[], @@ -75,7 +96,8 @@ fn serve_flag_is_bounds_checked_and_overrides_env() { stderr(&o) ); } - // The flag wins over a bad env value (the env is never consulted). + // The flag wins over a bad env value (the env is never consulted); the + // pack-size floor then reports the flag's cap. let o = sc( &root, &[("SC_MAX_OBJECT_SIZE", "junk")], @@ -83,7 +105,7 @@ fn serve_flag_is_bounds_checked_and_overrides_env() { "serve", "--stdio", "--max-object-size", - "1MiB", + "2MiB", "--max-pack-size", "1024", ".", @@ -91,39 +113,50 @@ fn serve_flag_is_bounds_checked_and_overrides_env() { ); assert!(!o.status.success()); assert!( - stderr(&o).contains("below the max object size (1048576)"), + stderr(&o).contains("below the max object size (2097152)"), "the pack floor uses the flag's cap: {}", stderr(&o) ); std::fs::remove_dir_all(&root).unwrap(); } +#[test] +fn serve_honours_the_env_when_the_flag_is_absent() { + let root = tmp("serve-env"); + assert!(sc(&root, &[], &["init"]).status.success()); + let o = sc( + &root, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["serve", "--stdio", "--max-pack-size", "1024", "."], + ); + assert!(!o.status.success()); + assert!( + stderr(&o).contains("below the max object size (4194304)"), + "the pack floor uses the env's cap: {}", + stderr(&o) + ); + std::fs::remove_dir_all(&root).unwrap(); +} + #[test] fn a_lowered_cap_refuses_a_transfer_a_raised_one_allows() { let src = tmp("xfer-src"); - assert!(sc(&src, &[], &["init"]).status.success()); - // 2 MiB of incompressible bytes: one object over a 1 MiB cap. - let mut x = 0x9E37_79B9_7F4A_7C15u64; - let big: Vec = (0..2 * 1024 * 1024) - .map(|_| { - x ^= x << 13; - x ^= x >> 7; - x ^= x << 17; - x as u8 - }) - .collect(); - std::fs::write(src.join("big.bin"), &big).unwrap(); - let o = sc(&src, &[("SC_AUTHOR", "t")], &["commit", "-m", "big"]); - assert!(o.status.success(), "commit is not capped: {}", stderr(&o)); + // 3 MiB: one object over a 2 MiB cap. + let big = repo_with_big_file(&src, 3 * 1024 * 1024); let parent = tmp("xfer-dst"); let low = parent.join("low"); let o = sc( &parent, - &[("SC_MAX_OBJECT_SIZE", "1MiB")], + &[("SC_MAX_OBJECT_SIZE", "2MiB")], &["clone", src.to_str().unwrap(), low.to_str().unwrap()], ); - assert!(!o.status.success(), "1 MiB cap must refuse a 2 MiB blob"); + assert!(!o.status.success(), "a 2 MiB cap must refuse a 3 MiB blob"); + assert!( + stderr(&o).contains("2097152-byte object size limit"), + "refused by the object cap, not something else: {}", + stderr(&o) + ); let high = parent.join("high"); let o = sc( @@ -131,9 +164,64 @@ fn a_lowered_cap_refuses_a_transfer_a_raised_one_allows() { &[("SC_MAX_OBJECT_SIZE", "4MiB")], &["clone", src.to_str().unwrap(), high.to_str().unwrap()], ); - assert!(o.status.success(), "4 MiB cap accepts it: {}", stderr(&o)); + assert!(o.status.success(), "a 4 MiB cap accepts it: {}", stderr(&o)); assert_eq!(std::fs::read(high.join("big.bin")).unwrap(), big); std::fs::remove_dir_all(&src).unwrap(); std::fs::remove_dir_all(&parent).unwrap(); } + +fn spawn_http_server(root: &Path) -> (Child, String) { + let mut child = Command::new(env!("CARGO_BIN_EXE_sc")) + .args(["serve", "--http", "127.0.0.1:0", root.to_str().unwrap()]) + .env_remove("SC_MAX_OBJECT_SIZE") + .stdout(Stdio::piped()) + .spawn() + .expect("spawn sc serve --http"); + let mut reader = std::io::BufReader::new(child.stdout.take().unwrap()); + let mut line = String::new(); + assert!(reader.read_line(&mut line).unwrap() > 0, "server announced"); + let addr = line + .trim() + .strip_prefix("listening on ") + .unwrap_or_else(|| panic!("unexpected startup line: {line:?}")) + .to_string(); + // Keep draining so the server never blocks on a full stdout pipe. + std::thread::spawn(move || for _ in reader.lines() {}); + (child, addr) +} + +#[test] +fn a_client_with_a_lower_cap_than_the_server_fails_with_the_mismatch_hint() { + let src = tmp("mismatch-src"); + let big = repo_with_big_file(&src, 3 * 1024 * 1024); + // The server runs at the default cap and serves the 3 MiB blob happily. + let (mut server, addr) = spawn_http_server(&src); + let url = format!("sc+http://{addr}"); + + let parent = tmp("mismatch-dst"); + let low = parent.join("low"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "2MiB")], + &["clone", &url, low.to_str().unwrap()], + ); + assert!(!o.status.success(), "the lowered client must refuse"); + let err = stderr(&o); + assert!(err.contains("2097152-byte object size limit"), "{err}"); + assert!(err.contains("SC_MAX_OBJECT_SIZE"), "names the knob: {err}"); + + let ok = parent.join("ok"); + let o = sc( + &parent, + &[("SC_MAX_OBJECT_SIZE", "4MiB")], + &["clone", &url, ok.to_str().unwrap()], + ); + assert!(o.status.success(), "an adequate cap passes: {}", stderr(&o)); + assert_eq!(std::fs::read(ok.join("big.bin")).unwrap(), big); + + let _ = server.kill(); + let _ = server.wait(); + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 61f28bbb..ee508b3a 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -19,9 +19,14 @@ pub mod store; /// peer's ability to force multi-GB allocations from a few header bytes. pub const DEFAULT_MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; -/// Hard floor for the operator-set object cap (1 MiB): below this the -/// 1 MiB default pack chunk and ordinary snapshots stop fitting a frame. -pub const MIN_MAX_OBJECT_SIZE: usize = 1024 * 1024; +/// Hard floor for the operator-set object cap (2 MiB). A default peer sends +/// pack chunk frames of `CHUNK_SIZE` (1 MiB) plus a tag byte, which a +/// 1 MiB cap would refuse, so anything lower cannot talk to a default peer. +pub const MIN_MAX_OBJECT_SIZE: usize = 2 * 1024 * 1024; + +/// What to tell an operator whose peer refused (or was refused by) the cap: +/// there is no negotiation, so both ends must use the same value. +pub const OBJECT_SIZE_HINT: &str = "peer mismatch? both ends must use the same max object size: set SC_MAX_OBJECT_SIZE (or `sc serve --max-object-size`) to the same value"; /// Hard ceiling for the operator-set object cap (1 GiB). Compile-time, so no /// flag or env value can make a hostile peer's allocation claim unbounded. @@ -33,12 +38,12 @@ static MAX_OBJECT_SIZE: std::sync::OnceLock = std::sync::OnceLock::new(); /// a pack record's compressed payload and decompressed output, a wire frame /// body, a by-id reply object, and the `SC_PACK_CHUNK` clamp. The single /// source for every untrusted-length guard; never inline a number. Defaults -/// to [`DEFAULT_MAX_OBJECT_SIZE`] until [`set_max_object_size`] runs. +/// to [`DEFAULT_MAX_OBJECT_SIZE`]; the first call here or to +/// [`set_max_object_size`] fixes the value for the process. pub fn max_object_size() -> usize { - MAX_OBJECT_SIZE - .get() - .copied() - .unwrap_or(DEFAULT_MAX_OBJECT_SIZE) + // The first read fixes the value, so a later `set_max_object_size` cannot + // disagree with guards that already ran. + *MAX_OBJECT_SIZE.get_or_init(|| DEFAULT_MAX_OBJECT_SIZE) } /// Set the process-wide object cap, once, at startup. Rejects (never clamps) @@ -92,15 +97,52 @@ pub fn parse_object_size(s: &str) -> std::result::Result { Ok(n) } -/// Read exactly `len` bytes, growing the buffer as bytes actually arrive -/// rather than pre-allocating `len`: `len` is a peer's claim, and a hostile -/// peer must not get a `len`-sized allocation for a few header bytes. The -/// caller still checks `len` against [`max_object_size`] first. +/// Largest single buffer growth in the incremental readers below. Peak +/// over-allocation past the bytes actually received is at most this. +const READ_STEP: usize = 8 * 1024 * 1024; + +/// Read from `r` until EOF or `limit` bytes, whichever is first, growing the +/// buffer by `reserve_exact` steps as bytes arrive. `first_step` sizes the +/// first allocation (the caller's best guess); later steps grow with the data +/// seen so far but never past [`READ_STEP`], so nothing doubles and the +/// capacity never exceeds the bytes read plus one step. A hostile length or +/// decompression bomb therefore cannot buy a `limit`-sized allocation without +/// supplying the bytes. +pub fn read_limited( + r: &mut impl std::io::Read, + limit: usize, + first_step: usize, +) -> std::io::Result> { + let mut buf: Vec = Vec::new(); + let mut filled = 0; + while filled < limit { + if filled == buf.len() { + let want = if filled == 0 { + first_step.clamp(1, READ_STEP) + } else { + filled.clamp(64 * 1024, READ_STEP) + }; + let step = want.min(limit - filled); + buf.reserve_exact(step); + buf.resize(filled + step, 0); + } + match r.read(&mut buf[filled..]) { + Ok(0) => break, + Ok(n) => filled += n, + Err(e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return Err(e), + } + } + buf.truncate(filled); + Ok(buf) +} + +/// Read exactly `len` bytes via [`read_limited`]: `len` is a peer's claim, so +/// the buffer follows the bytes that arrive. The caller still checks `len` +/// against [`max_object_size`] first. A short read is `UnexpectedEof`. pub fn read_exact_vec(r: &mut impl std::io::Read, len: usize) -> std::io::Result> { - use std::io::Read; - let mut buf = Vec::with_capacity(len.min(1 << 20)); - let got = r.by_ref().take(len as u64).read_to_end(&mut buf)?; - if got < len { + let buf = read_limited(r, len, len)?; + if buf.len() < len { return Err(std::io::ErrorKind::UnexpectedEof.into()); } Ok(buf) @@ -121,9 +163,9 @@ mod tests { #[test] fn parse_object_size_accepts_bytes_and_suffixes() { - assert_eq!(parse_object_size("1048576").unwrap(), MIN_MAX_OBJECT_SIZE); - assert_eq!(parse_object_size("1MiB").unwrap(), MIN_MAX_OBJECT_SIZE); - assert_eq!(parse_object_size("1024kib").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("2097152").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("2MiB").unwrap(), MIN_MAX_OBJECT_SIZE); + assert_eq!(parse_object_size("2048kib").unwrap(), MIN_MAX_OBJECT_SIZE); assert_eq!( parse_object_size(" 256 MiB ").unwrap(), DEFAULT_MAX_OBJECT_SIZE @@ -135,8 +177,9 @@ mod tests { fn parse_object_size_rejects_out_of_range_and_garbage_without_clamping() { for bad in [ "0", - "1048575", - "1023KiB", + "2097151", + "2047KiB", + "1MiB", "1025MiB", "2GiB", "18446744073709551615GiB", // multiplication overflow @@ -161,6 +204,42 @@ mod tests { assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); } + #[test] + fn read_limited_never_over_allocates_by_more_than_one_step() { + // An endless reader (a decompression bomb's shape): stops at the + // limit, and the capacity is within one step of the bytes read. + let limit = 3 * READ_STEP + 12_345; + let v = read_limited(&mut std::io::repeat(0), limit, 4096).unwrap(); + assert_eq!(v.len(), limit); + assert!(v.capacity() <= limit + READ_STEP, "cap {}", v.capacity()); + // A claimed length with all bytes present: also within one step + // (in fact exact, since the last step is the remainder). + let src = vec![9u8; 2 * READ_STEP + 7]; + let v = read_exact_vec(&mut &src[..], src.len()).unwrap(); + assert_eq!(v, src); + assert!(v.capacity() <= src.len() + READ_STEP); + // A small claim allocates about the claim, not a step. + let v = read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(); + assert!(v.capacity() < 1024); + } + + #[test] + fn read_limited_handles_short_reads_and_eof_before_the_limit() { + struct Dribble<'a>(&'a [u8]); + impl std::io::Read for Dribble<'_> { + fn read(&mut self, out: &mut [u8]) -> std::io::Result { + let n = self.0.len().min(out.len()).min(1000); + out[..n].copy_from_slice(&self.0[..n]); + self.0 = &self.0[n..]; + Ok(n) + } + } + let src: Vec = (0..(READ_STEP + 5000)).map(|i| i as u8).collect(); + let got = read_limited(&mut Dribble(&src), usize::MAX / 2, 10).unwrap(); + assert_eq!(got, src); + assert!(read_exact_vec(&mut Dribble(&src), src.len() + 1).is_err()); + } + #[test] fn read_exact_vec_reads_len_and_errors_on_short_input() { assert_eq!(read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(), b"abcd"); diff --git a/crates/core/src/pack.rs b/crates/core/src/pack.rs index e43b5ea2..336d634d 100644 --- a/crates/core/src/pack.rs +++ b/crates/core/src/pack.rs @@ -251,7 +251,8 @@ fn record_bounds(pack: &[u8], offset: u64, max: Option) -> Result<(usize, if let Some(max) = max { if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds the {max}-byte object size limit" + "record compressed length {len} exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } } @@ -280,15 +281,12 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R Some(max) => { let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(payload)) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let mut canonical = Vec::new(); - decoder - .by_ref() - .take(max as u64 + 1) - .read_to_end(&mut canonical) + let canonical = crate::read_limited(&mut decoder, max + 1, payload.len() * 4) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; if canonical.len() > max { return Err(Error::PackCorrupt(format!( - "decompressed object exceeds the {max}-byte object size limit" + "decompressed object exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } canonical @@ -417,7 +415,8 @@ pub fn parse_pack_reader_max( let len = u32::from_le_bytes(len_bytes) as usize; if len > max { return Err(Error::PackCorrupt(format!( - "record compressed length {len} exceeds the {max}-byte object size limit" + "record compressed length {len} exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } @@ -430,15 +429,12 @@ pub fn parse_pack_reader_max( // over-cap output without ever materializing it in full. let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let mut canonical = Vec::new(); - decoder - .by_ref() - .take(max as u64 + 1) - .read_to_end(&mut canonical) + let canonical = crate::read_limited(&mut decoder, max + 1, payload.len() * 4) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; if canonical.len() > max { return Err(Error::PackCorrupt(format!( - "decompressed object exceeds the {max}-byte object size limit" + "decompressed object exceeds the {max}-byte object size limit ({})", + crate::OBJECT_SIZE_HINT ))); } let actual_id = ObjectId::of(&canonical); diff --git a/crates/core/tests/max_object_size.rs b/crates/core/tests/max_object_size.rs index bbd670c3..398216a1 100644 --- a/crates/core/tests/max_object_size.rs +++ b/crates/core/tests/max_object_size.rs @@ -1,5 +1,5 @@ -//! The process-wide object cap is set once. Own test binary so the global is -//! not shared with the unit tests (which rely on the default). +//! The process-wide object cap is set once. Own test binaries (one global +//! each) so the unit tests keep the default. use scl_core::{ max_object_size, set_max_object_size, DEFAULT_MAX_OBJECT_SIZE, MAX_MAX_OBJECT_SIZE, @@ -8,13 +8,12 @@ use scl_core::{ #[test] fn cap_is_set_once_validated_and_then_fixed() { - assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); - // Out-of-range values are refused and leave the default in place. + // Out-of-range values are refused and leave nothing fixed. assert!(set_max_object_size(MIN_MAX_OBJECT_SIZE - 1).is_err()); assert!(set_max_object_size(MAX_MAX_OBJECT_SIZE + 1).is_err()); - assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); + assert_ne!(2 * MIN_MAX_OBJECT_SIZE, DEFAULT_MAX_OBJECT_SIZE); assert_eq!(max_object_size(), 2 * MIN_MAX_OBJECT_SIZE); // Same value again is a no-op; a different one is refused. set_max_object_size(2 * MIN_MAX_OBJECT_SIZE).unwrap(); diff --git a/crates/core/tests/max_object_size_read_first.rs b/crates/core/tests/max_object_size_read_first.rs new file mode 100644 index 00000000..f5211463 --- /dev/null +++ b/crates/core/tests/max_object_size_read_first.rs @@ -0,0 +1,16 @@ +//! Reading the cap fixes it: a guard that already ran with the default must +//! not be contradicted by a later `set`. + +use scl_core::{ + max_object_size, set_max_object_size, DEFAULT_MAX_OBJECT_SIZE, MIN_MAX_OBJECT_SIZE, +}; + +#[test] +fn a_read_before_set_fixes_the_default() { + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); + let err = set_max_object_size(MIN_MAX_OBJECT_SIZE).unwrap_err(); + assert!(err.contains("already fixed"), "got {err}"); + // Setting the value already in force is fine. + set_max_object_size(DEFAULT_MAX_OBJECT_SIZE).unwrap(); + assert_eq!(max_object_size(), DEFAULT_MAX_OBJECT_SIZE); +} diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index ede72333..a622b76c 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -69,24 +69,29 @@ pub struct BucketTransport { } /// Untrusted-length guard (P28 parity): refuse any WAL metadata value — -/// manifest, log entry, idx — larger than the object cap before decoding. -/// Pack bodies are exempt from this particular guard (they may legitimately -/// exceed it), but are not unguarded: `object_bytes` reads a pack body via +/// manifest, log entry, idx — larger than the default 256 MiB before +/// decoding. Fixed at the default, not the operator's object cap: this is +/// metadata, not an object, and a lowered cap would make a bucket with a +/// large pack index unreadable (idx entries are 48 bytes each). The check runs +/// after the bucket GET has buffered the value, so unlike the frame and pack +/// guards it bounds decode work, not the download. Pack bodies are exempt +/// from this particular guard (they may legitimately exceed it), but are not +/// unguarded: `object_bytes` reads a pack body via /// `scl_core::pack::read_object_at_bounded` (not the unbounded /// `read_object_at` — that path is reserved for `Store`'s own /// already-verified on-disk packs per ADR-0039's explicit trust split), /// which caps both the compressed record length and the decompressed output -/// at the object cap, mirroring `parse_pack_reader`'s bounded decode. A +/// at the live object cap, mirroring `parse_pack_reader`'s bounded decode. A /// hostile bucket pack therefore cannot mount a decompression-bomb DoS /// against `get_object`/`get_pack`. fn capped(what: &str, bytes: Vec) -> Result> { - capped_to(what, bytes, scl_core::max_object_size()) + capped_to(what, bytes, scl_core::DEFAULT_MAX_OBJECT_SIZE) } fn capped_to(what: &str, bytes: Vec, max: usize) -> Result> { if bytes.len() > max { return Err(Error::Wal(format!( - "{what} exceeds the {max}-byte object size limit" + "{what} exceeds the {max}-byte metadata limit" ))); } Ok(bytes) @@ -671,7 +676,7 @@ mod tests { fn capped_to_rejects_above_a_lowered_cap_and_accepts_at_a_raised_one() { let v = vec![0u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; let err = capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE).unwrap_err(); - assert!(format!("{err}").contains("object size limit"), "got {err}"); + assert!(format!("{err}").contains("metadata limit"), "got {err}"); assert!(capped_to("manifest", v.clone(), scl_core::MIN_MAX_OBJECT_SIZE + 1).is_ok()); assert!(capped_to("manifest", v, scl_core::MAX_MAX_OBJECT_SIZE).is_ok()); } diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index d8675b03..0dc9fce3 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -101,12 +101,13 @@ impl WireClient { // The client's own read cap tripping is NOT a server refusal: the reply // is already part-way through the stream, so report it as a protocol // violation that callers must not answer by splitting the batch. - wire::read_pack_stream(&mut rw.0, out, get_objects_response_cap()).map_err( - |e| match e { - Error::PackTooLarge(m) => Error::Protocol(format!("by-id reply too large: {m}")), - other => other, - }, - )?; + wire::read_pack_stream(&mut rw.0, out, GET_OBJECTS_RESPONSE_CAP).map_err(|e| match e { + Error::PackTooLarge(_) => Error::Protocol(format!( + "by-id reply exceeded this client's {}-byte spool limit", + GET_OBJECTS_RESPONSE_CAP + )), + other => other, + })?; Ok(not_sent) } } @@ -190,19 +191,19 @@ impl Transport for WireClient { /// One round trip (P37, ADR-0047): the OK body lists the ids the server /// will not send, and a chunk stream with the rest follows. The reply - /// stream is capped (`get_objects_response_cap`) so a hostile server + /// stream is capped (`GET_OBJECTS_RESPONSE_CAP`) so a hostile server /// cannot make a by-id fetch spool without bound. fn get_objects(&self, ids: &[ObjectId], out: &mut dyn Write) -> Result> { self.get_objects_flagged(ids, out, &mut false) } } -/// Most bytes a client accepts in one `GetObjects` reply stream: four times -/// the per-object cap (1 GiB at the default), twice the server's own by-id -/// reply ceiling, so pack framing never trips it on a legitimate reply. -fn get_objects_response_cap() -> u64 { - scl_core::max_object_size() as u64 * 4 -} +/// Most bytes a client accepts in one `GetObjects` reply stream: twice the +/// server's by-id reply ceiling, so pack framing never trips it on a +/// legitimate reply. A constant, not derived from the local object cap: it +/// bounds a disk spool, not RAM, and a cap that differed between the two ends +/// must not make a by-id batch with no large object fail. +const GET_OBJECTS_RESPONSE_CAP: u64 = 2 * crate::transport::BY_ID_REPLY_CEILING; /// A [`Transport`] whose far end is a child process speaking the wire protocol /// on its stdio — `ssh sc serve --stdio ` for real remotes. diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 88b30a49..1f5fa4b5 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -83,24 +83,22 @@ pub trait Transport { } /// The most canonical object bytes one by-id reply carries (ADR-0048): twice -/// the per-object cap (`object_cap`, the live [`scl_core::max_object_size`]; -/// 512 MiB at the default), so a reply always has room for one maximum-size -/// object plus pack record headers and zstd framing, and scales with an -/// operator-set cap. -fn by_id_reply_ceiling(object_cap: usize) -> u64 { - (object_cap as u64) * 2 -} +/// the largest object cap an operator can set, so a reply has room for a +/// maximum-size object plus framing at any setting. A constant, not derived +/// from the local object cap: it bounds a disk spool, not RAM, and ends with +/// different caps must not disagree about whether a by-id batch fits. An +/// object over a server's own cap is still refused individually. +pub(crate) const BY_ID_REPLY_CEILING: u64 = 2 * scl_core::MAX_MAX_OBJECT_SIZE as u64; /// The by-id reply limit for a server whose `--max-pack-size` is /// `max_pack_size` (0 means unlimited): the smaller of that and -/// [`by_id_reply_ceiling`]. Shared by the wire server and [`LocalTransport`], +/// [`BY_ID_REPLY_CEILING`]. Shared by the wire server and [`LocalTransport`], /// so a lazy clone of a local path cannot spool an unbounded reply either. -pub(crate) fn by_id_reply_cap(max_pack_size: u64, object_cap: usize) -> u64 { - let ceiling = by_id_reply_ceiling(object_cap); +pub(crate) fn by_id_reply_cap(max_pack_size: u64) -> u64 { if max_pack_size == 0 { - ceiling + BY_ID_REPLY_CEILING } else { - max_pack_size.min(ceiling) + max_pack_size.min(BY_ID_REPLY_CEILING) } } @@ -543,8 +541,7 @@ impl Transport for LocalTransport { crate::wire::MAX_GET_OBJECTS_IDS ))); } - let (not_sent, guard) = - self.build_objects_tempfile(ids, by_id_reply_cap(0, scl_core::max_object_size()))?; + let (not_sent, guard) = self.build_objects_tempfile(ids, by_id_reply_cap(0))?; let mut f = std::fs::File::open(guard.path())?; std::io::copy(&mut f, out)?; Ok(not_sent) @@ -1124,20 +1121,13 @@ mod tests { } #[test] - fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_twice_the_object_cap() { + fn by_id_reply_cap_is_the_smaller_of_max_pack_size_and_a_fixed_ceiling() { const MIB: u64 = 1 << 20; - let default = scl_core::DEFAULT_MAX_OBJECT_SIZE; // 0 means "unlimited" for --max-pack-size, but a by-id reply is still - // bounded: 2 x the object cap, 512 MiB at the default. - assert_eq!(by_id_reply_cap(0, default), 512 * MIB); - assert_eq!(by_id_reply_cap(64 * MIB, default), 64 * MIB); - assert_eq!(by_id_reply_cap(16 << 30, default), 512 * MIB); - // The ceiling follows a lowered or raised object cap. - assert_eq!(by_id_reply_cap(0, scl_core::MIN_MAX_OBJECT_SIZE), 2 * MIB); - assert_eq!( - by_id_reply_cap(16 << 30, scl_core::MAX_MAX_OBJECT_SIZE), - 2048 * MIB - ); + // bounded, and the bound does not depend on the local object cap. + assert_eq!(by_id_reply_cap(0), 2048 * MIB); + assert_eq!(by_id_reply_cap(64 * MIB), 64 * MIB); + assert_eq!(by_id_reply_cap(16 << 30), 2048 * MIB); } /// A repo with a stopped merge, pick or rebase is completed against its diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index 0c32f618..c65f5819 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -422,7 +422,8 @@ fn read_frame_max(r: &mut impl Read, max: usize) -> Result>> { let len = u32::from_be_bytes(len_bytes) as usize; if len > max { return Err(Error::Protocol(format!( - "frame length {len} exceeds the {max}-byte object size limit" + "frame length {len} exceeds the {max}-byte object size limit ({})", + scl_core::OBJECT_SIZE_HINT ))); } // The length is the peer's claim: grow with the bytes that actually @@ -1101,10 +1102,7 @@ fn serve_session( let result = match &transport { ServeTransport::Local(t) => t.build_objects_tempfile( &ids, - crate::transport::by_id_reply_cap( - policy.max_pack_size, - scl_core::max_object_size(), - ), + crate::transport::by_id_reply_cap(policy.max_pack_size), ), ServeTransport::Bucket { .. } => Err(Error::InvalidArgument( "bucket remotes do not support by-id object fetch".into(), @@ -1141,8 +1139,8 @@ fn serve_session( // `guard` drops here — temp pack file removed, // whether ingestion succeeded or failed. } - Err(e @ Error::PackTooLarge(_)) => { - // Over the cap: the stream is desynced (we stopped + Err(e @ (Error::PackTooLarge(_) | Error::Protocol(_))) => { + // Over the cap, or a malformed/over-cap frame: the stream is desynced (we stopped // reading mid-pack, so any bytes still on the wire // are neither read nor a subsequent request), so a // best-effort reply is the most we can do — then @@ -1438,10 +1436,11 @@ mod tests { } #[test] - fn frame_claiming_the_cap_without_a_body_fails_without_a_cap_sized_allocation() { - // At the ceiling, a 4-byte header can claim 1 GiB. The body is read - // incrementally, so a peer that sends nothing gets a short-read - // error, not a pre-allocated buffer. + fn frame_claiming_the_ceiling_without_a_body_is_a_short_read_error() { + // At the ceiling, a 4-byte header can claim 1 GiB and passes the + // length guard; with no body behind it the read fails as a short + // frame. (That no cap-sized buffer is allocated is pinned by + // `read_limited_never_over_allocates_by_more_than_one_step`.) let cap = scl_core::MAX_MAX_OBJECT_SIZE; let mut r = std::io::Cursor::new((cap as u32).to_be_bytes().to_vec()); let err = read_frame_max(&mut r, cap).unwrap_err(); @@ -1946,10 +1945,13 @@ mod tests { #[test] fn chunk_clamp_follows_a_lowered_and_a_raised_cap() { - // Lowered: the 1 MiB default chunk (data) no longer fits with its tag - // byte, so it clamps to cap - 1 and the frame is exactly the cap. + // Lowered: an explicit chunk larger than the cap clamps to cap - 1, + // so the frame (tag byte + data) is exactly the cap. The default + // 1 MiB chunk still fits at the 2 MiB floor. let low = scl_core::MIN_MAX_OBJECT_SIZE; - let chunk = chunk_from(None, low); + assert_eq!(chunk_from(None, low), CHUNK_SIZE); + assert!(CHUNK_SIZE < low, "a default peer's frame fits the floor"); + let chunk = chunk_from(Some(&(low * 2).to_string()), low); assert_eq!(chunk, low - 1); let mut frame = vec![ST_PACK_CHUNK]; frame.extend(vec![0u8; chunk]); diff --git a/demo/run_limits_demo.sh b/demo/run_limits_demo.sh index 6b568281..b21b73d5 100755 --- a/demo/run_limits_demo.sh +++ b/demo/run_limits_demo.sh @@ -164,7 +164,7 @@ echo "clone through a floor-capped server: succeeds ✔" # The cap gates the PutPack spool (the server's receive path) — a clone's # GetPack never consults it. Exercise the actually-gated path: a small # commit pushed THROUGH the capped server must land (its pack is far under -# the 256 MiB floor cap). If the cap's enforcement broke open-endedly or +# the 256 MiB --max-pack-size floor, which is the default max object size). If the cap's enforcement broke open-endedly or # regressed to rejecting everything, this push is the step that notices. cd "$W/c-floor" printf 'v2 via capped push\n' > pushed.txt diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 61f07329..636bef24 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -419,28 +419,38 @@ too, not just the src-control-side metadata. - A single max object size caps every untrusted length: wire frames, pack-record compressed length, the zstd *decompressed* output (decompression-bomb - guard), bucket WAL metadata, by-id reply objects, the `SC_PACK_CHUNK` clamp, + guard), bucket pack bodies, by-id reply objects, the `SC_PACK_CHUNK` clamp, and object-decode collection counts. The default is 256 MiB. An operator can lower or raise it with `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` - (#213), within a compile-time range of 1 MiB to 1 GiB; a value outside that - range aborts startup rather than being clamped. A guard checks the claimed - length before allocating, and frame bodies and pack records are then read - incrementally, so a hostile header cannot buy a cap-sized allocation without - sending the bytes. **Accepted boundaries:** + (#213), within a compile-time range of 2 MiB to 1 GiB; a value outside that + range aborts startup rather than being clamped; an empty `SC_MAX_OBJECT_SIZE` + counts as unset. A guard checks the claimed length before allocating, and + frame bodies, pack records and the zstd output are then read in steps of at + most 8 MiB as bytes arrive (no doubling), so a hostile header or compression + bomb cannot buy a cap-sized allocation without sending the bytes. + **Accepted boundaries:** - The cap is a *transfer-path* guard. A locally-committed blob above it commits fine but cannot be transferred. - **Both ends must agree.** There is no negotiation and no protocol bump. An object above the receiver's cap is refused there, loudly. A server raised to 1 GiB accepts a peer's gigabyte claim only if that peer sends the bytes, - but then holds that object in memory. Raising the cap raises per-connection - memory: a pack record is buffered compressed and then decompressed, and the - decoded object is a further copy, so budget a small multiple of the cap - (roughly 3x) per connection, times `--max-connections` (default 32). Size - `--max-connections` down when raising the cap. The repository's resident - blob budget rises with the cap (to twice the cap, never below 512 MiB) so one - maximum-size object can still be admitted. Lowering it below - a repository's largest object makes that repository unpushable until the - cap is restored. + but then holds that object in memory. Raising the cap raises + per-connection memory in two parts. Transient pack-decode memory: a + record is held compressed, then decompressed, then decoded, up to about + 3 x cap. Plus the connection's own store budget, `max(512 MiB, 2 x cap)`, + because each served connection opens its own store. That is about + 0.75 GiB + 0.5 GiB per connection at the default and about 3 GiB + 2 GiB + at the 1 GiB ceiling, times `--max-connections` (default 32); size + `--max-connections` down when raising the cap. Lowering the cap below a + repository's largest object makes that repository unpushable until it is + restored. `SC_MAX_OBJECT_SIZE` also sets the store budget of every local + command, not only transfers. + - **One length is checked after allocation:** bucket WAL metadata (manifest, + log entry, idx) is capped at a fixed 256 MiB after the bucket GET has + buffered it, so it bounds decode work, not the download. It is not tied to + the operator's cap, so lowering the cap cannot make a bucket unreadable. + - By-id reply spools (server ceiling 2 GiB, client read cap 4 GiB) are + constants independent of the cap: they bound disk, not RAM. - The ceiling (1 GiB) is compile-time, so no configuration can make a length claim unbounded. diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index d6b05148..38f2f883 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -135,47 +135,64 @@ regression test. setting read through one accessor, `scl_core::max_object_size()`, with a default of 256 MiB. Decisions: -1. **Lower or raise, inside a compile-time range.** Floor 1 MiB - (`MIN_MAX_OBJECT_SIZE`), ceiling 1 GiB (`MAX_MAX_OBJECT_SIZE`). A value +1. **Lower or raise, inside a compile-time range.** Floor 2 MiB + (`MIN_MAX_OBJECT_SIZE`; a default peer's pack chunk frame is 1 MiB plus a + tag byte, which a 1 MiB cap would refuse), ceiling 1 GiB (`MAX_MAX_OBJECT_SIZE`). A value outside the range, or one that does not parse, aborts startup with a clear error. It is never clamped. The ceiling bounds what a hostile peer can make a receiver commit to for any one object whatever the operator sets. 2. **Server flag and client env, no negotiation.** `sc serve --max-object-size ` and, for every `sc` process, `SC_MAX_OBJECT_SIZE` (same parse, same bounds). The flag wins over the - env. The setting is fixed once at startup (`set_max_object_size`; a second, - different value is an error). There is no handshake field and no + env. An empty `SC_MAX_OBJECT_SIZE` counts as unset. The setting is fixed + by whichever comes first, the startup `set_max_object_size` or the first + `max_object_size()` read; a later, different value is an error. There is no handshake field and no `PROTOCOL_VERSION` bump: matching the two ends is an operator responsibility, because an object larger than the receiver's cap fails at - the receiver. The failure is loud and typed, never silent truncation. + the receiver. The failure is loud and typed, never silent truncation, and + the frame and pack-record errors name `SC_MAX_OBJECT_SIZE` / + `--max-object-size` as the thing to align. 3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`); pack record compressed length and decompressed output - (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket WAL - metadata and pack bodies (`bucket_transport::capped_to`, - `object_bytes`); server-side servable-size check + (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket pack + bodies (`bucket_transport::object_bytes`); server-side servable-size check (`check_servable_size`); the `SC_PACK_CHUNK` clamp (`cap - 1`, a chunk frame carries a tag byte); the `--max-pack-size` floor - (`validate_max_pack_size`). Two limits that were independent constants are - now derived: the by-id reply ceiling is `2 x cap` (512 MiB at the default; - still `min(--max-pack-size, ceiling)`) and the client's by-id reply read - cap is `4 x cap` (1 GiB at the default), so both keep room for one - maximum-size object plus framing at any setting. Each guard takes the cap + (`validate_max_pack_size`). Two exceptions are deliberately fixed, not derived from the cap. Bucket + WAL metadata (manifest, log entry, idx) stays capped at the default + 256 MiB: it is metadata, not an object, and a lowered cap would make a + bucket with a large pack index (48 bytes per entry) unreadable. That check + runs after the bucket GET has buffered the value, so it bounds decode work + rather than the download, the one place a length is checked after + allocation. The by-id reply ceiling (server, `2 x` the maximum configurable + cap = 2 GiB, still `min(--max-pack-size, ceiling)`) and the client's by-id + read cap (twice that) are constants: they bound a disk spool, not RAM, and + if they followed the local cap, peers with different caps would disagree + about whether a by-id batch containing no large object fits. Each guard takes the cap as a parameter below the public entry point so tests can drive a lowered and a raised cap without touching the process-global. 4. **Allocation follows bytes received.** The length check still runs before any allocation, but the frame body and pack record payload are now read - through `scl_core::read_exact_vec`, which grows the buffer as bytes arrive - instead of pre-allocating the claimed length. With the ceiling at 1 GiB, a - four-byte header no longer buys a gigabyte-sized allocation. This closes the - "frame header can allocate up to the cap" item in THREAT-MODEL.md. + through `scl_core::read_limited` / `read_exact_vec`, which grow the buffer by + `reserve_exact` steps of at most 8 MiB as bytes arrive instead of + pre-allocating the claimed length. Nothing doubles, so capacity never + exceeds the bytes received by more than one step; the zstd decompressed + output uses the same reader (still bounded at `cap + 1` to detect overflow). + With the ceiling at 1 GiB, a four-byte header or a decompression bomb no + longer buys a gigabyte-sized allocation. This closes the "frame header can + allocate up to the cap" item in THREAT-MODEL.md. The resident-blob budget of a repo or transport store was a fixed 512 MiB; a raised cap would have admitted frames the store then refused with `BudgetExceeded`. It is now `max(512 MiB, 2 x cap)` (`repo::default_budget`), -unchanged at the default. Raising the cap also raises the memory a peer can -make a receiver hold per connection: a pack record is held compressed, then -decompressed, then decoded, so plan for roughly 3x the cap times -`--max-connections` (THREAT-MODEL.md). +unchanged at the default. The env var therefore also changes that budget for +every local command, not just transfers. Raising the cap raises the memory a +peer can make a receiver hold per connection, in two parts: transient pack +decode memory (the record is held compressed, then decompressed, then decoded +into an object, up to about 3 x cap) plus that connection's own store budget +(each served connection opens its own store, `max(512 MiB, 2 x cap)`). At +the default that is about 0.75 GiB + 0.5 GiB; at the 1 GiB ceiling about +3 GiB + 2 GiB, per connection, times `--max-connections` (THREAT-MODEL.md). For `ssh://` remotes the client builds the remote `sc serve --stdio` command itself and ssh does not forward the environment, so the server end of an ssh From 76adb06f09af90e9902eae03494d98b320acc946 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 09:44:04 +0300 Subject: [PATCH 04/11] docs(core): fix floor label and stale by-id/bucket cap statements (#213) --- CLAUDE.md | 2 +- crates/core/src/lib.rs | 2 +- docs/THREAT-MODEL.md | 9 +++++---- docs/adr/0048-presence-based-partial-clones.md | 3 ++- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index dc512370..cc21eb89 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -152,7 +152,7 @@ the code, those win. | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | | P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | -| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 512 MiB)` and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | +| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 2 GiB)` (a constant since #213, independent of the object cap) and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | | P37d | Agent sessions on a partial clone: `sc ws fork`/`run`/`harvest`/`abandon` and `sc work` work; a workspace or agent sees the host's sparse view only (`sc work` no longer gets a complete tree there), the view closure is filled (lazy, online) or refused with the gap error before anything is written, a workspace commit grafts the out-of-view subtrees from the base without reading them, and harvest lands through the ADR-0049 merge (any conflict falls back to a `work-` branch as on a full clone) | [0050](docs/adr/0050-sessions-on-partial-clones.md) | diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index ee508b3a..3636d634 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -65,7 +65,7 @@ fn check_object_size_bounds(n: usize) -> std::result::Result<(), String> { Ok(()) } else { Err(format!( - "max object size {n} bytes is outside the allowed range {MIN_MAX_OBJECT_SIZE} (1 MiB) to {MAX_MAX_OBJECT_SIZE} (1 GiB)" + "max object size {n} bytes is outside the allowed range {MIN_MAX_OBJECT_SIZE} (2 MiB) to {MAX_MAX_OBJECT_SIZE} (1 GiB)" )) } } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 636bef24..35bd421f 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -349,7 +349,7 @@ too, not just the src-control-side metadata. (`walfmt.rs`): unknown versions, bad magic, and any length that would overrun the buffer are refused outright rather than best-effort parsed. Untrusted-length caps apply throughout: manifest/log/idx bytes are capped - at the max object size before decode, and pack bodies are read through + at a fixed 256 MiB (not the operator's cap) after download, before decode, and pack bodies are read through `scl_core::pack::read_object_at_bounded` (not the unbounded `read_object_at` the trusted local `Store` uses for its own already- verified on-disk packs, per ADR-0039's explicit trust split) — capping @@ -406,11 +406,12 @@ too, not just the src-control-side metadata. nothing both sent and refused, nothing omitted) before any object is written. A hostile remote can at worst answer "not sent"; it cannot plant an object under another id or smuggle in extras. The reply stream a client will - read is capped at four times the max object size (1 GiB at the default, - up to 4 GiB with the cap raised to its 1 GiB ceiling). + read is capped at 4 GiB, a constant independent of the object cap (it + bounds a disk spool, not RAM). - **Server cost is bounded per request:** 1024 ids (checked while decoding, before any id is read), the server's `--max-pack-size` over the summed - encoded size, and the max object size per object. + encoded size, and the max object size per object. The server's reply ceiling is a constant + 2 GiB, independent of the object cap. - **Laziness is a local opt-in.** `.sc/promisor-lazy` is never committed or transferred; without it a partial clone never dials the remote after clone, and `SC_OFFLINE=1` forces that per process. diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 433bd664..025f44ad 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -281,7 +281,8 @@ zstd framing). Decisions: - The server caps a `GetObjects` reply, measured in canonical size, at `min(--max-pack-size, 512 MiB)` where a `--max-pack-size` of 0 means unlimited. Headroom to the client's 1 GiB read cap covers framing and - incompressible data. + incompressible data. (#213 later made both constants independent of the + object cap: reply ceiling 2 GiB, client read cap 4 GiB. See ADR-0039.) - The cap applies to `LocalTransport::get_objects` as well, which today builds with no limit (`build_objects_tempfile(ids, 0)`). A lazy clone whose promisor is a local path would otherwise spool up to 1024 large blobs in one request. From e7e015d2327b9d6e87bf9f220882ca7d4808327b Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 09:55:30 +0300 Subject: [PATCH 05/11] chore(core,repo): proportional read growth, close PutPack on any spill error, tests (#213) --- crates/core/src/lib.rs | 45 +++++++++++----- crates/repo/src/bucket_transport.rs | 10 ++++ crates/repo/src/wire.rs | 52 +++++++++++++++---- docs/THREAT-MODEL.md | 7 +-- docs/adr/0039-security-hardening-sweep.md | 7 +-- .../adr/0047-want-by-id-fetch-and-gap-fill.md | 8 ++- 6 files changed, 98 insertions(+), 31 deletions(-) diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 3636d634..75f15fe4 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -97,17 +97,29 @@ pub fn parse_object_size(s: &str) -> std::result::Result { Ok(n) } -/// Largest single buffer growth in the incremental readers below. Peak -/// over-allocation past the bytes actually received is at most this. +/// Floor of a buffer growth step in the incremental readers below. Peak +/// over-allocation past the bytes actually received is at most +/// `max(READ_STEP, received / 8)`. const READ_STEP: usize = 8 * 1024 * 1024; +/// Size of the next growth step once `filled` bytes are in hand: geometric up +/// to [`READ_STEP`] (cheap for small objects), then `max(READ_STEP, filled/8)`, +/// so total copying stays linear even on an allocator that always copies on +/// growth, while over-allocation stays within 12.5% (or one `READ_STEP`). +fn next_step(filled: usize) -> usize { + if filled < READ_STEP { + filled.clamp(64 * 1024, READ_STEP) + } else { + READ_STEP.max(filled / 8) + } +} + /// Read from `r` until EOF or `limit` bytes, whichever is first, growing the -/// buffer by `reserve_exact` steps as bytes arrive. `first_step` sizes the -/// first allocation (the caller's best guess); later steps grow with the data -/// seen so far but never past [`READ_STEP`], so nothing doubles and the -/// capacity never exceeds the bytes read plus one step. A hostile length or -/// decompression bomb therefore cannot buy a `limit`-sized allocation without -/// supplying the bytes. +/// buffer by `reserve_exact` steps (see [`next_step`]) as bytes arrive. +/// `first_step` sizes the first allocation (the caller's best guess). The +/// capacity never exceeds the bytes read by more than `max(8 MiB, 12.5%)`, so +/// a hostile length or decompression bomb cannot buy a `limit`-sized +/// allocation without supplying the bytes. pub fn read_limited( r: &mut impl std::io::Read, limit: usize, @@ -120,7 +132,7 @@ pub fn read_limited( let want = if filled == 0 { first_step.clamp(1, READ_STEP) } else { - filled.clamp(64 * 1024, READ_STEP) + next_step(filled) }; let step = want.min(limit - filled); buf.reserve_exact(step); @@ -208,16 +220,25 @@ mod tests { fn read_limited_never_over_allocates_by_more_than_one_step() { // An endless reader (a decompression bomb's shape): stops at the // limit, and the capacity is within one step of the bytes read. - let limit = 3 * READ_STEP + 12_345; + let slack = |n: usize| READ_STEP.max(n / 8); + // Past the 12.5% regime (limit / 8 > READ_STEP). + let limit = 12 * READ_STEP + 12_345; let v = read_limited(&mut std::io::repeat(0), limit, 4096).unwrap(); assert_eq!(v.len(), limit); - assert!(v.capacity() <= limit + READ_STEP, "cap {}", v.capacity()); + assert!( + v.capacity() <= limit + slack(limit), + "cap {} for {limit}", + v.capacity() + ); + let limit = 3 * READ_STEP + 12_345; + let v = read_limited(&mut std::io::repeat(0), limit, 4096).unwrap(); + assert!(v.capacity() <= limit + slack(limit), "cap {}", v.capacity()); // A claimed length with all bytes present: also within one step // (in fact exact, since the last step is the remainder). let src = vec![9u8; 2 * READ_STEP + 7]; let v = read_exact_vec(&mut &src[..], src.len()).unwrap(); assert_eq!(v, src); - assert!(v.capacity() <= src.len() + READ_STEP); + assert!(v.capacity() <= src.len() + slack(src.len())); // A small claim allocates about the claim, not a step. let v = read_exact_vec(&mut &b"abcdef"[..], 4).unwrap(); assert!(v.capacity() < 1024); diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index a622b76c..3c7b3591 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -672,6 +672,16 @@ impl Transport for BucketTransport { #[cfg(test)] mod tests { + #[test] + fn capped_metadata_guard_is_the_fixed_default_not_the_live_cap() { + // calloc'd zeros: no resident memory for the oversized value. + let at = vec![0u8; scl_core::DEFAULT_MAX_OBJECT_SIZE]; + assert!(capped("idx", at).is_ok()); + let over = vec![0u8; scl_core::DEFAULT_MAX_OBJECT_SIZE + 1]; + let err = capped("idx", over).unwrap_err(); + assert!(format!("{err}").contains("metadata limit"), "got {err}"); + } + #[test] fn capped_to_rejects_above_a_lowered_cap_and_accepts_at_a_raised_one() { let v = vec![0u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index c65f5819..1838e1bb 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -1139,21 +1139,18 @@ fn serve_session( // `guard` drops here — temp pack file removed, // whether ingestion succeeded or failed. } - Err(e @ (Error::PackTooLarge(_) | Error::Protocol(_))) => { - // Over the cap, or a malformed/over-cap frame: the stream is desynced (we stopped - // reading mid-pack, so any bytes still on the wire - // are neither read nor a subsequent request), so a - // best-effort reply is the most we can do — then - // close the connection rather than try to serve - // further requests off a desynced stream. + Err(e) => { + // Any spill failure (over the cap, a malformed or + // over-cap frame, I/O): we stopped reading mid-pack, + // so any bytes still on the wire are neither read + // nor a subsequent request. A best-effort reply is + // the most we can do; then close the connection + // rather than serve further requests off a + // desynced stream. let (code, msg) = err_to_wire(&e); let _ = write_err(w, code, &msg); return Ok(()); } - Err(e) => { - let (code, msg) = err_to_wire(&e); - write_err(w, code, &msg)?; - } } } other => { @@ -2021,6 +2018,39 @@ mod tests { let _ = std::fs::remove_dir_all(&root); } + /// A malformed frame inside a pack stream desyncs the connection: the + /// server replies once and closes instead of answering what follows. + #[test] + fn put_pack_protocol_error_closes_the_connection() { + let root = tmp_repo("putpack-proto"); + let mut input = Vec::new(); + for req in [ + Request::Hello { + version: PROTOCOL_VERSION, + }, + Request::PutPack, + ] { + write_frame(&mut input, &req.encode()).unwrap(); + } + write_frame(&mut input, &[0x7f]).unwrap(); // not a pack chunk or end + write_frame(&mut input, &Request::HeadBranch.encode()).unwrap(); // must never be answered + + let mut output = Vec::new(); + serve(&root, &mut std::io::Cursor::new(input), &mut output).unwrap(); + + let mut frames = Vec::new(); + let mut r = std::io::Cursor::new(output); + while let Some(f) = read_frame_opt(&mut r).unwrap() { + frames.push(parse_response(f)); + } + assert_eq!(frames.len(), 2, "hello ok, one error, then closed"); + assert!(matches!( + frames[1].as_ref().unwrap_err(), + Error::Protocol(_) + )); + let _ = std::fs::remove_dir_all(&root); + } + /// `GetObjects` is a read verb: a read-only session serves it, and the /// `max_pack_size` spool cap applies to its reply. #[test] diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 35bd421f..ce465bec 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -426,9 +426,10 @@ too, not just the src-control-side metadata. (#213), within a compile-time range of 2 MiB to 1 GiB; a value outside that range aborts startup rather than being clamped; an empty `SC_MAX_OBJECT_SIZE` counts as unset. A guard checks the claimed length before allocating, and - frame bodies, pack records and the zstd output are then read in steps of at - most 8 MiB as bytes arrive (no doubling), so a hostile header or compression - bomb cannot buy a cap-sized allocation without sending the bytes. + frame bodies, pack records and the zstd output are then read in exact-size + growth steps as bytes arrive, so over-allocation is bounded (at most the + larger of 8 MiB and 12.5% of the bytes received) and a hostile header or + compression bomb cannot buy a cap-sized allocation without sending the bytes. **Accepted boundaries:** - The cap is a *transfer-path* guard. A locally-committed blob above it commits fine but cannot be transferred. diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index 38f2f883..2d64488b 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -174,9 +174,10 @@ default of 256 MiB. Decisions: 4. **Allocation follows bytes received.** The length check still runs before any allocation, but the frame body and pack record payload are now read through `scl_core::read_limited` / `read_exact_vec`, which grow the buffer by - `reserve_exact` steps of at most 8 MiB as bytes arrive instead of - pre-allocating the claimed length. Nothing doubles, so capacity never - exceeds the bytes received by more than one step; the zstd decompressed + `reserve_exact` steps as bytes arrive instead of pre-allocating the + claimed length. Steps are geometric up to 8 MiB, then `max(8 MiB, filled/8)`, + so capacity never exceeds the bytes received by more than the larger of + 8 MiB and 12.5%, and total copying stays linear; the zstd decompressed output uses the same reader (still bounded at `cap + 1` to detect overflow). With the ceiling at 1 GiB, a four-byte header or a decompression bomb no longer buys a gigabyte-sized allocation. This closes the "frame header can diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md index ea8a3034..efa6cbb9 100644 --- a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -110,7 +110,9 @@ not have. The rules for `GetObjects`: - Caps: 1024 ids per request (`MAX_GET_OBJECTS_IDS`, checked while decoding), the server's `--max-pack-size` over the summed canonical encoded size of a reply and `MAX_OBJECT_SIZE` per object (both `PackTooLarge`, before any pack byte is - written), and a 1 GiB cap on the stream a client will read back. A client + written), and a 1 GiB cap on the stream a client will read back (since #213 a 4 GiB + constant independent of the object cap; see ADR-0048 and the ADR-0039 + addendum). A client also stops reading a pack as soon as it holds more records than it requested, so repeating one large object cannot multiply the receiver's decompress-and-hash work. @@ -149,7 +151,9 @@ not have. The rules for `GetObjects`: 3. **Merge on a partial clone** needs base, ours and theirs trees for only the conflicting paths. Whether tree ids alone suffice (identical ids need no fetch) or a server-assisted diff is needed is a 37b spike. -4. **Reply size caps and batching.** The client reads at most 1 GiB per reply +4. **Reply size caps and batching.** (Superseded: since #213 the client read cap + is a 4 GiB constant and the server reply ceiling 2 GiB; ADR-0048, ADR-0039.) + The client reads at most 1 GiB per reply while the server's default `--max-pack-size` is larger, so a batch that crosses 1 GiB fails mid-stream and leaves that connection out of sync. 37a sends one id per call so it cannot happen; 37b must split batches by size From fcc715af6885ed00740d003cd2d5240832e7eff6 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 10:15:51 +0300 Subject: [PATCH 06/11] fix(core,repo): review follow-ups: overflow-safe hint, 2x pack decode peak, receiver-cap wording, stale ADR figures (#213) --- .claude/skills/sc-cli-reference/SKILL.md | 5 +-- CLAUDE.md | 4 +-- ROADMAP.md | 2 +- crates/cli/src/main.rs | 5 +-- crates/core/src/lib.rs | 6 ++-- crates/core/src/pack.rs | 19 +++++++---- docs/THREAT-MODEL.md | 33 +++++++++++------- docs/adr/0039-security-hardening-sweep.md | 34 ++++++++++++------- docs/adr/0041-listener-resource-limits.md | 6 ++-- .../adr/0047-want-by-id-fetch-and-gap-fill.md | 16 ++++----- .../adr/0048-presence-based-partial-clones.md | 14 ++++---- 11 files changed, 85 insertions(+), 59 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index d3c4aefa..25c2acaa 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -277,8 +277,9 @@ cargo run --bin sc -- serve --http [--read-only] [--allow-public] # 256MiB, range 2MiB..1GiB, out of range aborts # startup (never clamped). Clients use env # SC_MAX_OBJECT_SIZE (same parse, any command; empty = unset); - # flag beats env. No negotiation: both ends must - # agree, a larger object fails at the receiver (#213) + # flag beats env. No negotiation: each receiver + # applies its own cap; a larger object fails at + # whichever side receives it (#213) cargo run --bin sc -- serve token add --label --scope ro|rw [--expires <30m|12h|7d|2w|YYYY-MM-DD>] # mint an sct- bearer token in # .sc/serve-tokens.toml; raw value prints once on diff --git a/CLAUDE.md b/CLAUDE.md index 69c47b66..7af39d36 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -167,8 +167,8 @@ transport-adjacent. The rest, imperatively: `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set it, range 2 MiB to 1 GiB, out of range aborts startup, ADR-0039 addendum) guards the transfer path only — a larger blob commits locally but fails every - subsequent push/fetch/clone at the receiver. There is no negotiation: both - ends must agree. Never inline a size in a new length guard; read the accessor. + subsequent push/fetch/clone at the receiver. There is no negotiation: each + receiver applies its own cap, so it must admit the objects it receives. Never inline a size in a new length guard; read the accessor. - **The P35 desktop app is a keyless, read-only top-level adapter.** Its renderer has five typed repository commands, no general filesystem/shell capability, and no DTO field capable of carrying protected ciphertext or diff --git a/ROADMAP.md b/ROADMAP.md index 239df413..31635e27 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -796,7 +796,7 @@ scale-&-reach horizon): - **`--max-object-size` operator knob — shipped (#213, ADR-0039 addendum).** `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap (default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire - negotiation: both ends must agree. + negotiation: each receiver applies its own cap. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the object cap minus one (a chunk frame carries a tag byte too; an oversized value used to make every transfer fail with a confusing receiver diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index a7cac9a0..7aee6278 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -363,8 +363,9 @@ enum Cmd { /// Largest single object accepted from a peer: bytes, or a number /// with a KiB/MiB/GiB suffix. Default 256MiB, range 2MiB to 1GiB; /// out-of-range values are refused at startup, never clamped. Clients - /// set the same limit with `SC_MAX_OBJECT_SIZE`; both ends must - /// agree, since a larger object fails at the receiver. (#213) + /// set it with `SC_MAX_OBJECT_SIZE`. Each receiver applies its own + /// cap, so a larger object fails at whichever side is receiving it. + /// (#213) #[arg(long, value_name = "SIZE")] max_object_size: Option, /// Serve TLS (`sc+https://`): auto-mints a self-signed identity into diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 75f15fe4..ed28b3e8 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -24,9 +24,9 @@ pub const DEFAULT_MAX_OBJECT_SIZE: usize = 256 * 1024 * 1024; /// 1 MiB cap would refuse, so anything lower cannot talk to a default peer. pub const MIN_MAX_OBJECT_SIZE: usize = 2 * 1024 * 1024; -/// What to tell an operator whose peer refused (or was refused by) the cap: -/// there is no negotiation, so both ends must use the same value. -pub const OBJECT_SIZE_HINT: &str = "peer mismatch? both ends must use the same max object size: set SC_MAX_OBJECT_SIZE (or `sc serve --max-object-size`) to the same value"; +/// What to tell an operator whose receiver refused an object: there is no +/// negotiation, and each receiver applies its own cap to what it accepts. +pub const OBJECT_SIZE_HINT: &str = "only the receiving side's cap applies: raise SC_MAX_OBJECT_SIZE (or `sc serve --max-object-size` on a server) on the side that is refusing"; /// Hard ceiling for the operator-set object cap (1 GiB). Compile-time, so no /// flag or env value can make a hostile peer's allocation claim unbounded. diff --git a/crates/core/src/pack.rs b/crates/core/src/pack.rs index 336d634d..386a2c04 100644 --- a/crates/core/src/pack.rs +++ b/crates/core/src/pack.rs @@ -281,8 +281,9 @@ fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> R Some(max) => { let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(payload)) .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let canonical = crate::read_limited(&mut decoder, max + 1, payload.len() * 4) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + let canonical = + crate::read_limited(&mut decoder, max + 1, payload.len().saturating_mul(4)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; if canonical.len() > max { return Err(Error::PackCorrupt(format!( "decompressed object exceeds the {max}-byte object size limit ({})", @@ -427,10 +428,15 @@ pub fn parse_pack_reader_max( // still decompress to an enormous plaintext (a "zstd bomb"). Read at // most max + 1 bytes so we can detect and reject an // over-cap output without ever materializing it in full. - let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; - let canonical = crate::read_limited(&mut decoder, max + 1, payload.len() * 4) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + let canonical = { + let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(&payload)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + crate::read_limited(&mut decoder, max + 1, payload.len().saturating_mul(4)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))? + }; + // Release the compressed copy before decoding, so peak per record is + // decompressed + decoded (about 2 x cap), not three copies. + drop(payload); if canonical.len() > max { return Err(Error::PackCorrupt(format!( "decompressed object exceeds the {max}-byte object size limit ({})", @@ -444,6 +450,7 @@ pub fn parse_pack_reader_max( ))); } let obj = Object::decode(&canonical)?; + drop(canonical); f(actual_id, obj)?; } } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index ce465bec..11ad6ae3 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -433,20 +433,29 @@ too, not just the src-control-side metadata. **Accepted boundaries:** - The cap is a *transfer-path* guard. A locally-committed blob above it commits fine but cannot be transferred. - - **Both ends must agree.** There is no negotiation and no protocol bump. An - object above the receiver's cap is refused there, loudly. A server raised + - **The receiver's cap must admit the object.** There is no negotiation and + no protocol bump, and only the receiving side's cap applies to incoming + frames and records (a default server serves a 3 MiB object to a client + capped at 4 MiB). An object above the receiver's cap is refused there, + loudly, with an error naming `SC_MAX_OBJECT_SIZE`. A server raised to 1 GiB accepts a peer's gigabyte claim only if that peer sends the bytes, but then holds that object in memory. Raising the cap raises - per-connection memory in two parts. Transient pack-decode memory: a - record is held compressed, then decompressed, then decoded, up to about - 3 x cap. Plus the connection's own store budget, `max(512 MiB, 2 x cap)`, - because each served connection opens its own store. That is about - 0.75 GiB + 0.5 GiB per connection at the default and about 3 GiB + 2 GiB - at the 1 GiB ceiling, times `--max-connections` (default 32); size - `--max-connections` down when raising the cap. Lowering the cap below a - repository's largest object makes that repository unpushable until it is - restored. `SC_MAX_OBJECT_SIZE` also sets the store budget of every local - command, not only transfers. + per-connection memory in two parts. Transient pack-decode memory: a record + holds its compressed payload plus the decompressed output (each up to the + cap) while decompressing, then the decompressed output plus the decoded + object while decoding (the compressed copy is released first), so the peak + is about 2 x cap. Plus the connection's own store budget, + `max(512 MiB, 2 x cap)`, because each served connection opens its own + store. That is about 0.5 GiB + 0.5 GiB per connection at the default and + about 2 GiB + 2 GiB at the 1 GiB ceiling, and it multiplies by + `--max-connections` (default 32); size `--max-connections` down when + raising the cap. Lowering the cap below a repository's largest object + makes that repository unpushable until it is restored. `SC_MAX_OBJECT_SIZE` + also sets the default store budget of every local command that opens a + repo or serves one (`Repo::open`, `LocalTransport`), not only transfers. + An explicit budget overrides it: `sc work --budget-mb` and + `Repo::open_with_budget` use exactly the size given, and one smaller than + an object fails with `BudgetExceeded`. - **One length is checked after allocation:** bucket WAL metadata (manifest, log entry, idx) is capped at a fixed 256 MiB after the bucket GET has buffered it, so it bounds decode work, not the download. It is not tied to diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index 2d64488b..eb7c4e27 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -147,11 +147,12 @@ default of 256 MiB. Decisions: env. An empty `SC_MAX_OBJECT_SIZE` counts as unset. The setting is fixed by whichever comes first, the startup `set_max_object_size` or the first `max_object_size()` read; a later, different value is an error. There is no handshake field and no - `PROTOCOL_VERSION` bump: matching the two ends is an operator - responsibility, because an object larger than the receiver's cap fails at - the receiver. The failure is loud and typed, never silent truncation, and + `PROTOCOL_VERSION` bump: the operator must make each receiver's cap + admit the objects it will receive, because an object larger than the + receiver's cap fails at the receiver (only the receiving side's cap applies + to incoming frames and records). The failure is loud and typed, never silent truncation, and the frame and pack-record errors name `SC_MAX_OBJECT_SIZE` / - `--max-object-size` as the thing to align. + `--max-object-size` as the thing to raise on the refusing side. 3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`); pack record compressed length and decompressed output (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket pack @@ -185,15 +186,22 @@ default of 256 MiB. Decisions: The resident-blob budget of a repo or transport store was a fixed 512 MiB; a raised cap would have admitted frames the store then refused with -`BudgetExceeded`. It is now `max(512 MiB, 2 x cap)` (`repo::default_budget`), -unchanged at the default. The env var therefore also changes that budget for -every local command, not just transfers. Raising the cap raises the memory a -peer can make a receiver hold per connection, in two parts: transient pack -decode memory (the record is held compressed, then decompressed, then decoded -into an object, up to about 3 x cap) plus that connection's own store budget -(each served connection opens its own store, `max(512 MiB, 2 x cap)`). At -the default that is about 0.75 GiB + 0.5 GiB; at the 1 GiB ceiling about -3 GiB + 2 GiB, per connection, times `--max-connections` (THREAT-MODEL.md). +`BudgetExceeded`. The default is now `max(512 MiB, 2 x cap)` +(`repo::default_budget`), unchanged at the default cap. It applies to +`Repo::open`/`init` and to `LocalTransport` (so also to a served connection). +The env var therefore changes the default budget of every local command that +opens a repo, not just transfers. An explicit budget (`sc work --budget-mb`, +`Repo::open_with_budget`) overrides the default and can be smaller than an +object. + +Raising the cap raises the memory a peer can make a receiver hold per +connection, in two parts: transient pack decode memory (compressed + +decompressed while decompressing, then decompressed + decoded while decoding, +the compressed copy being released in between: peak about 2 x cap) plus that +connection's own store budget (each served connection opens its own store, +`max(512 MiB, 2 x cap)`). At the default that is about 0.5 GiB + 0.5 GiB; at +the 1 GiB ceiling about 2 GiB + 2 GiB, per connection, times +`--max-connections` (THREAT-MODEL.md). For `ssh://` remotes the client builds the remote `sc serve --stdio` command itself and ssh does not forward the environment, so the server end of an ssh diff --git a/docs/adr/0041-listener-resource-limits.md b/docs/adr/0041-listener-resource-limits.md index 1e83297d..444cfe86 100644 --- a/docs/adr/0041-listener-resource-limits.md +++ b/docs/adr/0041-listener-resource-limits.md @@ -56,11 +56,11 @@ scoped to the listener (`--stdio` is unaffected except where noted): stall trips it, not a slow-but-moving transfer. The opening keeps its existing, unchanged 30s. -3. **`--max-pack-size ` (default 16 GiB, 0 = unlimited, floor 256 MiB - = `MAX_OBJECT_SIZE`).** Applies to **both** `--http` and `--stdio` — it +3. **`--max-pack-size ` (default 16 GiB, 0 = unlimited, floor = the max + object size, 256 MiB by default).** Applies to **both** `--http` and `--stdio` — it lives in `WirePolicy` (`crates/repo/src/wire.rs`), the shared layer both transports call into, not in `ServeLimits` (the `--http`-only listener - config). A configured non-zero value below `MAX_OBJECT_SIZE` is rejected + config). A configured non-zero value below the max object size (`scl_core::max_object_size()`, configurable since #213, ADR-0039 addendum) is rejected as invalid config at startup. Enforcement is a counted mid-stream abort in the pack-stream readers (mirroring git's `receive.maxInputSize`): once the running total crosses the cap, the reader stops, a best-effort diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md index efa6cbb9..e151552d 100644 --- a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -109,10 +109,10 @@ not have. The rules for `GetObjects`: confidentiality rests on encryption, not on withholding. - Caps: 1024 ids per request (`MAX_GET_OBJECTS_IDS`, checked while decoding), the server's `--max-pack-size` over the summed canonical encoded size of a reply and - `MAX_OBJECT_SIZE` per object (both `PackTooLarge`, before any pack byte is - written), and a 1 GiB cap on the stream a client will read back (since #213 a 4 GiB - constant independent of the object cap; see ADR-0048 and the ADR-0039 - addendum). A client + the max object size per object (both `PackTooLarge`, before any pack byte is + written), and a 4 GiB cap on the stream a client will read back (originally + 1 GiB; a constant independent of the object cap since #213, see ADR-0048 and + the ADR-0039 addendum). A client also stops reading a pack as soon as it holds more records than it requested, so repeating one large object cannot multiply the receiver's decompress-and-hash work. @@ -151,11 +151,11 @@ not have. The rules for `GetObjects`: 3. **Merge on a partial clone** needs base, ours and theirs trees for only the conflicting paths. Whether tree ids alone suffice (identical ids need no fetch) or a server-assisted diff is needed is a 37b spike. -4. **Reply size caps and batching.** (Superseded: since #213 the client read cap - is a 4 GiB constant and the server reply ceiling 2 GiB; ADR-0048, ADR-0039.) - The client reads at most 1 GiB per reply +4. **Reply size caps and batching.** (Superseded by ADR-0048; the figures below + were 1 GiB then and are a 4 GiB client cap and 2 GiB server ceiling since + #213.) The client reads a capped number of bytes per reply while the server's default `--max-pack-size` is larger, so a batch that - crosses 1 GiB fails mid-stream and leaves that connection out of sync. 37a + crosses the client cap fails mid-stream and leaves that connection out of sync. 37a sends one id per call so it cannot happen; 37b must split batches by size or close the transport on that error. 5. **`GetObject` and unfiltered private transfers.** See the authorization diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 025f44ad..2e71e8ad 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -273,23 +273,23 @@ protocol error, because the stream is no longer in sync. ### 5. Reply size limits -37a left a mismatch: the client reads at most 1 GiB per by-id reply while the -server default `--max-pack-size` is larger, and the server sums canonical +37a left a mismatch: the client read at most 1 GiB per by-id reply (4 GiB since +#213) while the server default `--max-pack-size` is larger, and the server sums canonical object sizes while the client counts pack bytes (36-byte record headers plus zstd framing). Decisions: - The server caps a `GetObjects` reply, measured in canonical size, at - `min(--max-pack-size, 512 MiB)` where a `--max-pack-size` of 0 means - unlimited. Headroom to the client's 1 GiB read cap covers framing and - incompressible data. (#213 later made both constants independent of the - object cap: reply ceiling 2 GiB, client read cap 4 GiB. See ADR-0039.) + `min(--max-pack-size, 2 GiB)` where a `--max-pack-size` of 0 means + unlimited. Headroom to the client's 4 GiB read cap covers framing and + incompressible data. (Originally 512 MiB and 1 GiB; #213 made both + constants independent of the object cap, see the ADR-0039 addendum.) - The cap applies to `LocalTransport::get_objects` as well, which today builds with no limit (`build_objects_tempfile(ids, 0)`). A lazy clone whose promisor is a local path would otherwise spool up to 1024 large blobs in one request. - The over-cap refusal happens before any pack byte is written, so the connection stays in sync. The client splits the batch in half and retries. A batch of one that is still refused is final and is returned as an error; - splitting never loops. (An object above `MAX_OBJECT_SIZE`, or above a small + splitting never loops. (An object above the max object size, or above a small `--max-pack-size`, can exist on an origin.) - The 37a handling for a stream abandoned mid-reply (kill the ssh child, drop the connection) stays as the backstop. From 3d2935240b6b2aaa345e7fbb2d56995d33b343db Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 10:34:52 +0300 Subject: [PATCH 07/11] fix(repo): do not apply the server's own object cap to by-id replies (#213) The cap is receiver-only. check_servable_size now refuses only objects above the largest configurable cap, so a lower-cap server serves a larger local object by id to a client whose cap admits it, consistent with GetPack. --- crates/cli/tests/max_object_size.rs | 72 +++++++++++++++++-- crates/repo/src/transport.rs | 25 ++++--- docs/THREAT-MODEL.md | 7 +- docs/adr/0039-security-hardening-sweep.md | 4 +- .../adr/0047-want-by-id-fetch-and-gap-fill.md | 2 +- .../adr/0048-presence-based-partial-clones.md | 2 +- 6 files changed, 90 insertions(+), 22 deletions(-) diff --git a/crates/cli/tests/max_object_size.rs b/crates/cli/tests/max_object_size.rs index 54ed7fc3..c72a9a23 100644 --- a/crates/cli/tests/max_object_size.rs +++ b/crates/cli/tests/max_object_size.rs @@ -171,9 +171,22 @@ fn a_lowered_cap_refuses_a_transfer_a_raised_one_allows() { std::fs::remove_dir_all(&parent).unwrap(); } -fn spawn_http_server(root: &Path) -> (Child, String) { +/// Kills the server when dropped, so a failing assertion cannot leave it +/// holding the test's stderr open. +struct Server(Child); +impl Drop for Server { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn spawn_http_server(root: &Path, extra: &[&str]) -> (Server, String) { + let mut args = vec!["serve", "--http", "127.0.0.1:0"]; + args.extend_from_slice(extra); + args.push(root.to_str().unwrap()); let mut child = Command::new(env!("CARGO_BIN_EXE_sc")) - .args(["serve", "--http", "127.0.0.1:0", root.to_str().unwrap()]) + .args(&args) .env_remove("SC_MAX_OBJECT_SIZE") .stdout(Stdio::piped()) .spawn() @@ -188,7 +201,7 @@ fn spawn_http_server(root: &Path) -> (Child, String) { .to_string(); // Keep draining so the server never blocks on a full stdout pipe. std::thread::spawn(move || for _ in reader.lines() {}); - (child, addr) + (Server(child), addr) } #[test] @@ -196,7 +209,7 @@ fn a_client_with_a_lower_cap_than_the_server_fails_with_the_mismatch_hint() { let src = tmp("mismatch-src"); let big = repo_with_big_file(&src, 3 * 1024 * 1024); // The server runs at the default cap and serves the 3 MiB blob happily. - let (mut server, addr) = spawn_http_server(&src); + let (_server, addr) = spawn_http_server(&src, &[]); let url = format!("sc+http://{addr}"); let parent = tmp("mismatch-dst"); @@ -220,8 +233,55 @@ fn a_client_with_a_lower_cap_than_the_server_fails_with_the_mismatch_hint() { assert!(o.status.success(), "an adequate cap passes: {}", stderr(&o)); assert_eq!(std::fs::read(ok.join("big.bin")).unwrap(), big); - let _ = server.kill(); - let _ = server.wait(); + std::fs::remove_dir_all(&src).unwrap(); + std::fs::remove_dir_all(&parent).unwrap(); +} + +#[test] +fn a_lower_cap_server_still_serves_a_by_id_fetch_to_a_higher_cap_client() { + // The cap is receiver-only: a server capped at 2 MiB that holds a + // locally committed 3 MiB blob serves it by id to a 4 MiB lazy client, + // as GetPack would. + let src = tmp("byid-src"); + assert!(sc(&src, &[], &["init"]).status.success()); + std::fs::create_dir_all(src.join("src")).unwrap(); + std::fs::create_dir_all(src.join("docs")).unwrap(); + std::fs::write(src.join("src/a.txt"), "small\n").unwrap(); + let mut x = 0x9E37_79B9_7F4A_7C15u64; + let big: Vec = (0..3 * 1024 * 1024) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + std::fs::write(src.join("docs/big.bin"), &big).unwrap(); + let o = sc(&src, &[("SC_AUTHOR", "t")], &["commit", "-m", "c"]); + assert!(o.status.success(), "{}", stderr(&o)); + + let (_server, addr) = spawn_http_server(&src, &["--max-object-size", "2MiB"]); + let url = format!("sc+http://{addr}"); + let parent = tmp("byid-dst"); + let dst = parent.join("lazy"); + let envs = [("SC_MAX_OBJECT_SIZE", "4MiB")]; + let o = sc( + &parent, + &envs, + &[ + "clone", + &url, + dst.to_str().unwrap(), + "--filter", + "src/", + "--lazy", + ], + ); + assert!(o.status.success(), "filtered clone: {}", stderr(&o)); + let o = sc(&dst, &envs, &["cat", "main:docs/big.bin"]); + assert!(o.status.success(), "by-id fill: {}", stderr(&o)); + assert_eq!(o.stdout, big); + std::fs::remove_dir_all(&src).unwrap(); std::fs::remove_dir_all(&parent).unwrap(); } diff --git a/crates/repo/src/transport.rs b/crates/repo/src/transport.rs index 7d0d9241..cba941bd 100644 --- a/crates/repo/src/transport.rs +++ b/crates/repo/src/transport.rs @@ -86,8 +86,9 @@ pub trait Transport { /// the largest object cap an operator can set, so a reply has room for a /// maximum-size object plus framing at any setting. A constant, not derived /// from the local object cap: it bounds a disk spool, not RAM, and ends with -/// different caps must not disagree about whether a by-id batch fits. An -/// object over a server's own cap is still refused individually. +/// different caps must not disagree about whether a by-id batch fits. A single +/// object over the largest configurable cap is still refused individually; the +/// serving process's own cap is not applied (it is receiver-only). pub(crate) const BY_ID_REPLY_CEILING: u64 = 2 * scl_core::MAX_MAX_OBJECT_SIZE as u64; /// The by-id reply limit for a server whose `--max-pack-size` is @@ -113,9 +114,12 @@ fn encoded_len(obj: &Object) -> u64 { } } -/// Refuse an object a receiver would reject anyway: `pack::parse_pack_reader` -/// fails any record whose canonical encoding exceeds the object cap `max` -/// (the live [`scl_core::max_object_size`]). +/// Refuse an object no receiver could accept: `pack::parse_pack_reader` fails +/// any record whose canonical encoding exceeds the receiver's cap, and no cap +/// exceeds `max` (callers pass [`scl_core::MAX_MAX_OBJECT_SIZE`]). The +/// serving process's own cap is deliberately not applied: the cap is +/// receiver-only, as it is for `GetPack`, so a server with a lower cap still +/// serves a locally committed larger object to a client whose cap admits it. fn check_servable_size(id: &ObjectId, len: u64, max: usize) -> Result<()> { if len > max as u64 { return Err(Error::PackTooLarge(format!( @@ -164,8 +168,8 @@ impl LocalTransport { /// shared core of [`Transport::get_objects`] and `wire::serve`'s /// `GetObjects` arm. Duplicate ids are served once. `max_bytes` (0 = /// unlimited) bounds the summed canonical encoded size, and any single - /// object over the object cap is refused (every receiver would reject - /// it). Both fail with `PackTooLarge` before any pack byte is written. + /// object over the largest configurable cap is refused (no receiver + /// could accept it; the server's own cap is not applied). Both fail with `PackTooLarge` before any pack byte is written. /// Blobs and sealed objects are measured without re-encoding them. pub(crate) fn build_objects_tempfile( &self, @@ -186,7 +190,7 @@ impl LocalTransport { Ok(obj) if refused_by_id(&obj) => not_sent.push(*id), Ok(obj) => { let size = encoded_len(&obj); - check_servable_size(id, size, scl_core::max_object_size())?; + check_servable_size(id, size, scl_core::MAX_MAX_OBJECT_SIZE)?; total = total.saturating_add(size); if max_bytes > 0 && total > max_bytes { return Err(Error::PackTooLarge(format!( @@ -734,9 +738,8 @@ mod tests { assert_eq!(encoded_len(o), o.encode().len() as u64, "{}", o.kind_name()); } let id = blob.id(); - // The receiver rejects a canonical encoding above the object cap, and - // the encoding of a blob is its payload plus one tag byte. Lowered and - // raised caps move the boundary. + // `max` is the boundary passed in; production passes the largest + // configurable cap (the serving process's own cap is receiver-only). for max in [ scl_core::MIN_MAX_OBJECT_SIZE, scl_core::DEFAULT_MAX_OBJECT_SIZE, diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 11ad6ae3..24d43e51 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -410,8 +410,11 @@ too, not just the src-control-side metadata. bounds a disk spool, not RAM). - **Server cost is bounded per request:** 1024 ids (checked while decoding, before any id is read), the server's `--max-pack-size` over the summed - encoded size, and the max object size per object. The server's reply ceiling is a constant - 2 GiB, independent of the object cap. + encoded size, and 1 GiB (the largest configurable object cap) per object. The + server's own object cap is not applied to replies, since the cap is + receiver-only: a lower-cap server serves a larger local object to a client + whose cap admits it, as `GetPack` does. The reply ceiling is a constant + 2 GiB. - **Laziness is a local opt-in.** `.sc/promisor-lazy` is never committed or transferred; without it a partial clone never dials the remote after clone, and `SC_OFFLINE=1` forces that per process. diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index eb7c4e27..2231d350 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -157,7 +157,9 @@ default of 256 MiB. Decisions: pack record compressed length and decompressed output (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket pack bodies (`bucket_transport::object_bytes`); server-side servable-size check - (`check_servable_size`); the `SC_PACK_CHUNK` clamp (`cap - 1`, a chunk + (`check_servable_size`, against the largest configurable cap, not the + server's own: the cap is receiver-only, so a server with a lower cap still + serves a larger local object by id, as `GetPack` does); the `SC_PACK_CHUNK` clamp (`cap - 1`, a chunk frame carries a tag byte); the `--max-pack-size` floor (`validate_max_pack_size`). Two exceptions are deliberately fixed, not derived from the cap. Bucket WAL metadata (manifest, log entry, idx) stays capped at the default diff --git a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md index e151552d..5880bac4 100644 --- a/docs/adr/0047-want-by-id-fetch-and-gap-fill.md +++ b/docs/adr/0047-want-by-id-fetch-and-gap-fill.md @@ -109,7 +109,7 @@ not have. The rules for `GetObjects`: confidentiality rests on encryption, not on withholding. - Caps: 1024 ids per request (`MAX_GET_OBJECTS_IDS`, checked while decoding), the server's `--max-pack-size` over the summed canonical encoded size of a reply and - the max object size per object (both `PackTooLarge`, before any pack byte is + 1 GiB (the largest configurable object cap, ADR-0039 addendum) per object (both `PackTooLarge`, before any pack byte is written), and a 4 GiB cap on the stream a client will read back (originally 1 GiB; a constant independent of the object cap since #213, see ADR-0048 and the ADR-0039 addendum). A client diff --git a/docs/adr/0048-presence-based-partial-clones.md b/docs/adr/0048-presence-based-partial-clones.md index 2e71e8ad..59fd6414 100644 --- a/docs/adr/0048-presence-based-partial-clones.md +++ b/docs/adr/0048-presence-based-partial-clones.md @@ -289,7 +289,7 @@ zstd framing). Decisions: - The over-cap refusal happens before any pack byte is written, so the connection stays in sync. The client splits the batch in half and retries. A batch of one that is still refused is final and is returned as an error; - splitting never loops. (An object above the max object size, or above a small + splitting never loops. (An object above 1 GiB, the largest configurable object cap, or above a small `--max-pack-size`, can exist on an origin.) - The 37a handling for a stream abandoned mid-reply (kill the ssh child, drop the connection) stays as the backstop. From bd65d525c24dc99d8f0ab4b670cfc03d0b3822ea Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 10:53:36 +0300 Subject: [PATCH 08/11] fix(repo): frame limit includes framing overhead, move PutObject body out of its frame (#213) A frame may be cap + 64 bytes so an object of exactly the cap fits its PutObject/GetObject envelope; the object body is checked against the cap exactly (check_object_len). Request::decode_owned and an in-place parse_response avoid a second copy of a cap-sized body. --- crates/repo/src/stdio_transport.rs | 4 +- crates/repo/src/wire.rs | 149 ++++++++++++++++++++++++----- docs/THREAT-MODEL.md | 11 ++- 3 files changed, 139 insertions(+), 25 deletions(-) diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 0dc9fce3..89a54870 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -123,7 +123,9 @@ impl Transport for WireClient { wire::decode_bool_body(&self.call(Request::HasObject(*id))?) } fn get_object(&self, id: &ObjectId) -> Result> { - self.call(Request::GetObject(*id)) + let body = self.call(Request::GetObject(*id))?; + wire::check_object_len(body.len(), scl_core::max_object_size())?; + Ok(body) } fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { self.call(Request::PutObject { diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index b647be2f..943e4c5e 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -373,6 +373,23 @@ impl Request { c.done()?; Ok(req) } + + /// [`Request::decode`] for a frame the caller owns. A `PutObject` object + /// body is moved out of the frame (the header is shifted off in place) + /// instead of copied, so a near-cap request holds one copy of the object, + /// not two. Anything malformed falls through to `decode` for its error. + pub fn decode_owned(mut frame: Vec) -> Result { + const HEADER: usize = 1 + 32 + 4; + if frame.len() >= HEADER && frame[0] == OP_PUT_OBJECT { + let n = u32::from_be_bytes(frame[33..HEADER].try_into().unwrap()) as usize; + if frame.len() - HEADER == n { + let id = ObjectId::from_bytes(frame[1..33].try_into().unwrap()); + frame.drain(..HEADER); + return Ok(Request::PutObject { id, bytes: frame }); + } + } + Request::decode(&frame) + } } // --- frame IO --- @@ -404,8 +421,28 @@ fn read_frame_inner(r: &mut impl Read) -> Result>> { read_frame_max(r, scl_core::max_object_size()) } -/// [`read_frame_opt`] with an explicit frame-length cap `max` (the live -/// [`scl_core::max_object_size`] in production). +/// Room a frame has beyond the object it carries: the longest header is a +/// `PutObject` request (opcode + 32-byte id + 4-byte length = 37 bytes); a +/// response adds a status byte and a pack chunk a tag byte. Frames may be +/// `cap + FRAME_OVERHEAD` long so an object of exactly `cap` bytes still +/// fits; the object itself is checked against `cap` by [`check_object_len`]. +pub const FRAME_OVERHEAD: usize = 64; + +/// Reject an object body above the cap `max`. Frames carry [`FRAME_OVERHEAD`] +/// extra bytes, so a single-object payload gets this separate exact check. +pub(crate) fn check_object_len(len: usize, max: usize) -> Result<()> { + if len > max { + return Err(Error::Protocol(format!( + "object of {len} bytes exceeds the {max}-byte object size limit ({})", + scl_core::OBJECT_SIZE_HINT + ))); + } + Ok(()) +} + +/// [`read_frame_opt`] with an explicit object cap `max` (the live +/// [`scl_core::max_object_size`] in production); the frame limit is +/// `max + FRAME_OVERHEAD`. fn read_frame_max(r: &mut impl Read, max: usize) -> Result>> { let mut len_bytes = [0u8; 4]; let mut filled = 0; @@ -420,9 +457,9 @@ fn read_frame_max(r: &mut impl Read, max: usize) -> Result>> { filled += n; } let len = u32::from_be_bytes(len_bytes) as usize; - if len > max { + if len > max + FRAME_OVERHEAD { return Err(Error::Protocol(format!( - "frame length {len} exceeds the {max}-byte object size limit ({})", + "frame length {len} exceeds the {max}-byte object size limit plus framing ({})", scl_core::OBJECT_SIZE_HINT ))); } @@ -541,14 +578,18 @@ pub fn write_err(w: &mut impl Write, code: u8, msg: &str) -> Result<()> { } /// Split a response frame into its OK body, or reconstruct the typed error. -pub fn parse_response(frame: Vec) -> Result> { - let (&status, rest) = frame - .split_first() +pub fn parse_response(mut frame: Vec) -> Result> { + let status = *frame + .first() .ok_or_else(|| Error::Protocol("empty response frame".into()))?; match status { - ST_OK => Ok(rest.to_vec()), + ST_OK => { + // Strip the status byte in place: no second copy of a large body. + frame.remove(0); + Ok(frame) + } ST_ERR => { - let mut c = Cur::new(rest); + let mut c = Cur::new(&frame[1..]); let code = c.u8()?; let msg = c.str()?; c.done()?; @@ -998,7 +1039,7 @@ fn serve_session( Some(f) => f, None => return Ok(()), // peer hung up between requests }; - let req = match Request::decode(&frame) { + let req = match Request::decode_owned(frame) { Ok(req) => req, Err(e) => { let (code, msg) = err_to_wire(&e); @@ -1170,10 +1211,14 @@ fn serve_session( transport.as_transport().has_object(&id).map(bool_body) } Request::GetObject(id) => transport.as_transport().get_object(&id), - Request::PutObject { id, bytes } => transport - .as_transport() - .put_object(&id, &bytes) - .map(|()| Vec::new()), + Request::PutObject { id, bytes } => { + check_object_len(bytes.len(), scl_core::max_object_size()).and_then(|()| { + transport + .as_transport() + .put_object(&id, &bytes) + .map(|()| Vec::new()) + }) + } Request::UpdateRef { branch, id, @@ -1395,12 +1440,12 @@ mod tests { #[test] fn frame_over_cap_rejected() { - // A frame header alone (no body) claiming a length past the cap must - // be rejected before the (fictitious) body is read or allocated: - // feeding only the 4-byte header proves this, since reading first - // would hit EOF and report a different error. + // A frame header alone (no body) claiming a length past the cap plus + // framing must be rejected before the (fictitious) body is read or + // allocated: feeding only the 4-byte header proves this, since + // reading first would hit EOF and report a different error. let cap = scl_core::MIN_MAX_OBJECT_SIZE; - let over = (cap + 1) as u32; + let over = (cap + FRAME_OVERHEAD + 1) as u32; let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); let err = read_frame_max(&mut r, cap).unwrap_err(); assert!( @@ -1408,7 +1453,7 @@ mod tests { "got {err:?}" ); // The production entry point is the same guard at the live cap. - let over = (scl_core::max_object_size() + 1) as u32; + let over = (scl_core::max_object_size() + FRAME_OVERHEAD + 1) as u32; let mut r = std::io::Cursor::new(over.to_be_bytes().to_vec()); assert!(matches!( read_frame_opt(&mut r), @@ -1418,10 +1463,10 @@ mod tests { #[test] fn frame_cap_lowered_rejects_and_raised_accepts_the_same_frame() { - let body = vec![7u8; scl_core::MIN_MAX_OBJECT_SIZE + 1]; + let low = scl_core::MIN_MAX_OBJECT_SIZE; + let body = vec![7u8; low + FRAME_OVERHEAD + 1]; let mut framed = Vec::new(); write_frame(&mut framed, &body).unwrap(); - let low = scl_core::MIN_MAX_OBJECT_SIZE; assert!(read_frame_max(&mut &framed[..], low).is_err()); assert_eq!( read_frame_max(&mut &framed[..], low + 1).unwrap().unwrap(), @@ -1433,6 +1478,64 @@ mod tests { ); } + /// An object of exactly the cap travels in `PutObject` (37 header bytes) + /// and `GetObject` (1 status byte) frames; one byte more is refused by + /// the exact object check even though the frame itself still fits. + #[test] + fn object_boundary_exactly_cap_accepted_cap_plus_one_rejected() { + let cap = scl_core::MIN_MAX_OBJECT_SIZE; + let id = some_id(7); + for (len, ok) in [(cap, true), (cap + 1, false)] { + let bytes = vec![5u8; len]; + // PutObject request. + let mut framed = Vec::new(); + write_frame( + &mut framed, + &Request::PutObject { + id, + bytes: bytes.clone(), + } + .encode(), + ) + .unwrap(); + let frame = read_frame_max(&mut &framed[..], cap).unwrap().unwrap(); + let Request::PutObject { bytes: got, .. } = Request::decode_owned(frame).unwrap() + else { + panic!("not a PutObject"); + }; + assert_eq!(got, bytes); + assert_eq!(check_object_len(got.len(), cap).is_ok(), ok, "put {len}"); + // GetObject response. + let mut framed = Vec::new(); + write_ok(&mut framed, &bytes).unwrap(); + let frame = read_frame_max(&mut &framed[..], cap).unwrap().unwrap(); + let body = parse_response(frame).unwrap(); + assert_eq!(body, bytes); + assert_eq!(check_object_len(body.len(), cap).is_ok(), ok, "get {len}"); + } + } + + #[test] + fn decode_owned_matches_decode_and_handles_malformed_put_object() { + let id = some_id(3); + let enc = Request::PutObject { + id, + bytes: b"hello".to_vec(), + } + .encode(); + assert_eq!( + Request::decode_owned(enc.clone()).unwrap(), + Request::decode(&enc).unwrap() + ); + // Declared length disagrees with the frame: same error as decode. + let mut bad = enc.clone(); + bad.push(0); + assert!(Request::decode_owned(bad).is_err()); + let mut short = enc; + short.truncate(10); + assert!(Request::decode_owned(short).is_err()); + } + #[test] fn frame_claiming_the_ceiling_without_a_body_is_a_short_read_error() { // At the ceiling, a 4-byte header can claim 1 GiB and passes the @@ -1956,7 +2059,7 @@ mod tests { let mut framed = Vec::new(); write_frame(&mut framed, &frame).unwrap(); assert!(read_frame_max(&mut &framed[..], low).unwrap().is_some()); - assert!(read_frame_max(&mut &framed[..], low - 1).is_err()); + assert!(read_frame_max(&mut &framed[..], low - FRAME_OVERHEAD - 1).is_err()); // Raised: a big explicit chunk is honoured up to the new cap - 1. let high = scl_core::MAX_MAX_OBJECT_SIZE; assert_eq!(chunk_from(Some("536870912"), high), 536_870_912); diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index da19a58c..42ade20c 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -460,7 +460,12 @@ too, not just the src-control-side metadata. holds its compressed payload plus the decompressed output (each up to the cap) while decompressing, then the decompressed output plus the decoded object while decoding (the compressed copy is released first), so the peak - is about 2 x cap. Plus the connection's own store budget, + is about 2 x cap. A single-object `PutObject` is also about 2 x cap: the + request frame is moved, not copied, into the object body, and storing it + decodes one more copy; a `GetObject` reply on the server holds the body and + its response frame. A read-only listener rejects a `PutObject` only after + reading the frame, so it can hold one cap-sized frame per connection to + refuse it. Plus the connection's own store budget, `max(512 MiB, 2 x cap)`, because each served connection opens its own store. That is about 0.5 GiB + 0.5 GiB per connection at the default and about 2 GiB + 2 GiB at the 1 GiB ceiling, and it multiplies by @@ -472,6 +477,10 @@ too, not just the src-control-side metadata. An explicit budget overrides it: `sc work --budget-mb` and `Repo::open_with_budget` use exactly the size given, and one smaller than an object fails with `BudgetExceeded`. + - **Frames carry 64 bytes of framing beyond the cap.** A frame may be + `cap + 64` bytes (`PutObject` header 37 bytes, response status 1, chunk + tag 1), so an object of exactly the cap fits; the object body itself is then + checked against the cap exactly. - **One length is checked after allocation:** bucket WAL metadata (manifest, log entry, idx) is capped at a fixed 256 MiB after the bucket GET has buffered it, so it bounds decode work, not the download. It is not tied to From 19260943b886aabcc72e12590c4a528a2d0e2916 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 10:53:45 +0300 Subject: [PATCH 09/11] docs: ADR-0039 framing overhead and single-object peak (#213) --- docs/adr/0039-security-hardening-sweep.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index 2231d350..e8826e5d 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -153,7 +153,10 @@ default of 256 MiB. Decisions: to incoming frames and records). The failure is loud and typed, never silent truncation, and the frame and pack-record errors name `SC_MAX_OBJECT_SIZE` / `--max-object-size` as the thing to raise on the refusing side. -3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`); +3. **Every guard reads the accessor.** Frame length (`wire::read_frame_max`, + limit `cap + FRAME_OVERHEAD`, 64 bytes, so an object of exactly the cap fits + inside its `PutObject`/`GetObject` envelope; `wire::check_object_len` then + checks the object body against the cap exactly); pack record compressed length and decompressed output (`pack::parse_pack_reader_max`, `read_object_at_bounded`); bucket pack bodies (`bucket_transport::object_bytes`); server-side servable-size check @@ -199,7 +202,9 @@ object. Raising the cap raises the memory a peer can make a receiver hold per connection, in two parts: transient pack decode memory (compressed + decompressed while decompressing, then decompressed + decoded while decoding, -the compressed copy being released in between: peak about 2 x cap) plus that +the compressed copy being released in between: peak about 2 x cap; a +single-object `PutObject` or `GetObject` is also about 2 x cap, the request +frame being moved into the object body rather than copied) plus that connection's own store budget (each served connection opens its own store, `max(512 MiB, 2 x cap)`). At the default that is about 0.5 GiB + 0.5 GiB; at the 1 GiB ceiling about 2 GiB + 2 GiB, per connection, times From 301a8d6766f55c86b64c3728264be638894f3962 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 11:13:26 +0300 Subject: [PATCH 10/11] docs,demo: stale object-cap claims in ADRs, isolate the limits demo from SC_MAX_OBJECT_SIZE (#213) --- demo/run_limits_demo.sh | 3 +++ docs/adr/0039-security-hardening-sweep.md | 1 + docs/adr/0041-listener-resource-limits.md | 2 +- docs/adr/0046-wal-bucket-remotes.md | 8 +++++--- docs/adr/0049-merge-on-partial-clones.md | 2 +- 5 files changed, 11 insertions(+), 5 deletions(-) diff --git a/demo/run_limits_demo.sh b/demo/run_limits_demo.sh index b21b73d5..a48ec9cd 100755 --- a/demo/run_limits_demo.sh +++ b/demo/run_limits_demo.sh @@ -20,6 +20,9 @@ set -euo pipefail cd "$(dirname "$0")/.." +# The --max-pack-size values below assume the default 256 MiB object cap; a +# caller's exported SC_MAX_OBJECT_SIZE must not change what the demo proves. +unset SC_MAX_OBJECT_SIZE cargo build --quiet --bin sc SC="$PWD/target/debug/sc" diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index e8826e5d..ca31ab84 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -3,6 +3,7 @@ - **Status:** Accepted - **Date:** 2026-07-09 - **Phase:** 28 +- **Amended:** #213 (addendum at the end) made the object cap configurable; the `MAX_OBJECT_SIZE` constant and 256 MiB figures in the sections before the addendum are the original design. - **Builds on:** ADR-0021 (durability/concurrency hardening — the P21 sweep precedent), ADR-0022 (ssh wire protocol), ADR-0015 (packfiles), ADR-0014 (protected paths), ADR-0008/0009 (secrets) ## Context diff --git a/docs/adr/0041-listener-resource-limits.md b/docs/adr/0041-listener-resource-limits.md index 444cfe86..8a4d2666 100644 --- a/docs/adr/0041-listener-resource-limits.md +++ b/docs/adr/0041-listener-resource-limits.md @@ -6,7 +6,7 @@ - **Builds on:** ADR-0036 (P26 sc-native HTTP transport — names three of the four gaps closed here), ADR-0040 (P29 access control — the seam where a busy/error status is already written before the wire handshake), ADR-0039 - (P28 DoS hardening — `MAX_OBJECT_SIZE`, composed with, not replaced by, + (P28 DoS hardening — the max object size, composed with, not replaced by, this phase's spool cap) - **Research:** `docs/research/bounded-server-patterns.md` (ticket #27, std-only bounded-server survey pinned to file:line). Decided at the diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index f1bffe14..85046219 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -68,11 +68,13 @@ immutable packs plus one compare-and-swapped manifest**: format has no per-prefix negotiation yet; a filtered clone needs a served remote (ADR-0037) until that lands. - **Untrusted-length guard on reads.** Manifest, log entries, and idx - metadata are capped at `MAX_OBJECT_SIZE` (256 MiB, ADR-0039) before + metadata are capped at a fixed 256 MiB (the default object cap; since #213 + deliberately not the operator's cap, see the ADR-0039 addendum) before decoding — a hostile or corrupted bucket cannot force an unbounded allocation. Pack bodies are read through `scl_core::pack::read_object_at_bounded`, which caps both the compressed - record length and the decompressed output at `MAX_OBJECT_SIZE`, mirroring + record length and the decompressed output at the live object cap (default + 256 MiB, configurable from 2 MiB to 1 GiB since #213), mirroring `parse_pack_reader`'s bounded decode — this is the same zstd-bomb guard P28 put on every other untrusted transfer path. **This is deliberately a different code path from `Store`'s own on-disk reads:** the trusted local @@ -144,7 +146,7 @@ confidentiality perimeter for public content, not the sc protocol. into `open_transport` for `sc+wal://` and `sc+s3://`, validated eagerly at `sc remote add`. 12 tests in `bucket_transport.rs` covering the base round-trip, racing/fleet pushes, crash-mid-push recovery, and the -`MAX_OBJECT_SIZE` cap. CLI plumbing (remote-add validation, push, clone, +object-size cap. CLI plumbing (remote-add validation, push, clone, fetch through the real binary) proven in `crates/cli/tests/bucket_remote.rs`. diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index 84df85ce..3c265529 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -66,7 +66,7 @@ Recipient-list changes alone do not require re-wrapping a carried ciphertext: th A per-id rule ("prune what you enumerated") is unsound: a blob id seen as deleted at one path may survive under a carried subtree elsewhere, since ids alias across paths. So the rule is coarse: **if the merge carried any subtree, the merged map is the complete union of the input wraps** (ours, theirs, base, with the existing ours/theirs/base precedence and per-recipient dedup), plus fresh wraps. If it carried nothing, the existing prune applies. Union resurrecting a wrap for a recipient revoked since is the intended P16 behavior (tombstones gate future seals, not historical wraps; ADR-0026). -Over-retention costs size: wraps are inline in the snapshot, and an object above `MAX_OBJECT_SIZE` fails transfer. Nothing prunes dead entries today (`sc rewrap` refuses partial clones and only removes ids it upgrades, `rewrap.rs:164`, `:308`). So the growth is real and bounded by the repo's protected blob count; it is measured before any prune is designed. +Over-retention costs size: wraps are inline in the snapshot, and an object above the max object size (256 MiB by default, ADR-0039 addendum) fails transfer. Nothing prunes dead entries today (`sc rewrap` refuses partial clones and only removes ids it upgrades, `rewrap.rs:164`, `:308`). So the growth is real and bounded by the repo's protected blob count; it is measured before any prune is designed. ### 5. The engine is pull-based; filling happens outside the store lock From b7b1cc3951470615ea014e21277c6fefe902c340 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 11:35:43 +0300 Subject: [PATCH 11/11] fix(repo): chunk size fits the smallest valid receiver, early cap check on reachability reads, wiring tests; document structural-object gap (#213, closes #232, refs #233) --- ROADMAP.md | 2 +- crates/repo/src/reachable.rs | 4 ++ crates/repo/src/wire.rs | 74 ++++++++------------ crates/repo/tests/max_object_size_wire.rs | 82 +++++++++++++++++++++++ docs/THREAT-MODEL.md | 16 ++++- docs/adr/0039-security-hardening-sweep.md | 14 +++- 6 files changed, 142 insertions(+), 50 deletions(-) create mode 100644 crates/repo/tests/max_object_size_wire.rs diff --git a/ROADMAP.md b/ROADMAP.md index a0854afa..a6acd996 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -805,7 +805,7 @@ scale-&-reach horizon): (default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire negotiation: each receiver applies its own cap. - **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run - `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the object cap minus one (a chunk frame carries a tag byte too; an + `validate_branch_name`; `SC_PACK_CHUNK` is clamped to the minimum configurable object cap minus one (2 MiB - 1, so every chunk frame fits any valid receiver) (a chunk frame carries a tag byte too; an oversized value used to make every transfer fail with a confusing receiver error); `Repo::worktree_paths` is a paths-only walk (`list_worktree_paths`, the same ignore and tracked-path rules as `read_worktree`, no file read); and diff --git a/crates/repo/src/reachable.rs b/crates/repo/src/reachable.rs index aac8aeec..9cf965f9 100644 --- a/crates/repo/src/reachable.rs +++ b/crates/repo/src/reachable.rs @@ -28,6 +28,10 @@ pub struct TransportSource<'a> { impl ObjectSource for TransportSource<'_> { fn get(&mut self, id: &ObjectId) -> Result { let bytes = self.transport.get_object(id)?; + // The receiver's cap, before decoding. (A local-path source has + // already loaded and encoded the object by now; wire sources refuse + // earlier, in `WireClient::get_object`.) + crate::wire::check_object_len(bytes.len(), scl_core::max_object_size())?; let obj = Object::decode(&bytes).map_err(Error::from)?; if obj.id() != *id { return Err(Error::CorruptObject(*id)); diff --git a/crates/repo/src/wire.rs b/crates/repo/src/wire.rs index 943e4c5e..52071366 100644 --- a/crates/repo/src/wire.rs +++ b/crates/repo/src/wire.rs @@ -76,25 +76,24 @@ pub const CHUNK_SIZE: usize = 1 << 20; /// sender, the client's `PutPack` sender) call this, so setting the var /// before either side starts affects the whole round trip. pub fn pack_chunk_size() -> usize { - chunk_from( - std::env::var("SC_PACK_CHUNK").ok().as_deref(), - scl_core::max_object_size(), - ) + chunk_from(std::env::var("SC_PACK_CHUNK").ok().as_deref()) } /// The `SC_PACK_CHUNK` decision, pure so it can be tested without touching the /// process environment: an unset, unparsable or zero value is the default, and -/// anything larger than a frame can carry is clamped. A chunk frame is the -/// `ST_PACK_CHUNK` tag byte plus the data, and the receiver refuses a frame -/// above the object cap, so the data is capped one byte below it; an -/// oversized setting would otherwise fail every large transfer with a -/// confusing error. -fn chunk_from(value: Option<&str>, object_cap: usize) -> usize { +/// anything larger than the smallest valid receiver accepts is clamped. Caps +/// are receiver-local and unnegotiated, and the smallest a receiver can be set +/// to is [`scl_core::MIN_MAX_OBJECT_SIZE`], so the data is capped one byte +/// below that (the `ST_PACK_CHUNK` tag byte makes the frame exactly that size). +/// Every chunk frame therefore fits any valid receiver, and receivers built +/// before the frame limit gained [`FRAME_OVERHEAD`]; an oversized setting would +/// otherwise fail every large transfer with a confusing error. +fn chunk_from(value: Option<&str>) -> usize { value .and_then(|v| v.parse::().ok()) .filter(|&n| n > 0) .unwrap_or(CHUNK_SIZE) - .min(object_cap - 1) + .min(scl_core::MIN_MAX_OBJECT_SIZE - 1) } // Wire error codes. Typed errors that sync logic relies on get their own code; @@ -2025,45 +2024,26 @@ mod tests { } #[test] - fn chunk_from_defaults_and_clamps_to_the_object_cap() { - let max = scl_core::DEFAULT_MAX_OBJECT_SIZE; - assert_eq!(chunk_from(None, max), CHUNK_SIZE); - assert_eq!(chunk_from(Some(""), max), CHUNK_SIZE); - assert_eq!(chunk_from(Some("0"), max), CHUNK_SIZE); - assert_eq!(chunk_from(Some("junk"), max), CHUNK_SIZE); - assert_eq!(chunk_from(Some("4096"), max), 4096); - // A chunk frame carries a tag byte too: data at the cap would make a - // frame one byte over what the receiver accepts. - let cap = max - 1; - assert_eq!( - chunk_from(Some(&(max * 8).to_string()), max), - cap, - "an oversized setting is clamped, not passed through" - ); - assert_eq!(chunk_from(Some(&max.to_string()), max), cap); - assert_eq!(chunk_from(Some(&cap.to_string()), max), cap); - } - - #[test] - fn chunk_clamp_follows_a_lowered_and_a_raised_cap() { - // Lowered: an explicit chunk larger than the cap clamps to cap - 1, - // so the frame (tag byte + data) is exactly the cap. The default - // 1 MiB chunk still fits at the 2 MiB floor. - let low = scl_core::MIN_MAX_OBJECT_SIZE; - assert_eq!(chunk_from(None, low), CHUNK_SIZE); - assert!(CHUNK_SIZE < low, "a default peer's frame fits the floor"); - let chunk = chunk_from(Some(&(low * 2).to_string()), low); - assert_eq!(chunk, low - 1); + fn chunk_from_defaults_and_clamps_to_the_smallest_valid_receiver_cap() { + let min = scl_core::MIN_MAX_OBJECT_SIZE; + assert_eq!(chunk_from(None), CHUNK_SIZE); + assert_eq!(chunk_from(Some("")), CHUNK_SIZE); + assert_eq!(chunk_from(Some("0")), CHUNK_SIZE); + assert_eq!(chunk_from(Some("junk")), CHUNK_SIZE); + assert_eq!(chunk_from(Some("4096")), 4096); + // Independent of this process's own cap: a sender raised to 1 GiB + // must still produce frames the smallest valid receiver accepts. + for v in [min, min * 8, scl_core::MAX_MAX_OBJECT_SIZE, usize::MAX] { + assert_eq!(chunk_from(Some(&v.to_string())), min - 1, "{v}"); + } + // The resulting frame (tag byte + data) is exactly the minimum cap and + // is accepted by a receiver at that cap, even with no overhead slack. let mut frame = vec![ST_PACK_CHUNK]; - frame.extend(vec![0u8; chunk]); + frame.extend(vec![0u8; min - 1]); let mut framed = Vec::new(); write_frame(&mut framed, &frame).unwrap(); - assert!(read_frame_max(&mut &framed[..], low).unwrap().is_some()); - assert!(read_frame_max(&mut &framed[..], low - FRAME_OVERHEAD - 1).is_err()); - // Raised: a big explicit chunk is honoured up to the new cap - 1. - let high = scl_core::MAX_MAX_OBJECT_SIZE; - assert_eq!(chunk_from(Some("536870912"), high), 536_870_912); - assert_eq!(chunk_from(Some(&high.to_string()), high), high - 1); + assert!(read_frame_max(&mut &framed[..], min).unwrap().is_some()); + assert!(CHUNK_SIZE < min, "the default chunk fits every receiver"); } #[test] diff --git a/crates/repo/tests/max_object_size_wire.rs b/crates/repo/tests/max_object_size_wire.rs new file mode 100644 index 00000000..d6eb4603 --- /dev/null +++ b/crates/repo/tests/max_object_size_wire.rs @@ -0,0 +1,82 @@ +//! The object cap is wired into the live wire paths (#232): a `PutObject` +//! through `serve` and a `GetObject` through `WireClient`, with the cap fixed +//! at 2 MiB before the first read. Own test binary: the cap is process-global. + +use scl_core::{Object, MIN_MAX_OBJECT_SIZE}; +use scl_repo::stdio_transport::WireClient; +use scl_repo::transport::Transport; +use scl_repo::wire::{self, Request}; + +fn set_cap() { + scl_core::set_max_object_size(MIN_MAX_OBJECT_SIZE).unwrap(); +} + +fn setup() -> std::path::PathBuf { + set_cap(); + let root = std::env::temp_dir().join(format!("scl-wire-cap-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + scl_repo::Repo::init(&root).unwrap(); + root +} + +/// Canonical encoding of a blob is its payload plus one tag byte. +fn blob_of_canonical_len(n: usize) -> Object { + Object::blob(vec![0xAB; n - 1]) +} + +fn put_via_serve(root: &std::path::Path, obj: &Object) -> Vec, scl_repo::Error>> { + let mut input = Vec::new(); + for req in [ + Request::Hello { + version: wire::PROTOCOL_VERSION, + }, + Request::PutObject { + id: obj.id(), + bytes: obj.encode(), + }, + ] { + wire::write_frame(&mut input, &req.encode()).unwrap(); + } + let mut output = Vec::new(); + wire::serve(root, &mut std::io::Cursor::new(input), &mut output).unwrap(); + let mut r = std::io::Cursor::new(output); + let mut out = Vec::new(); + while let Some(f) = wire::read_frame_opt(&mut r).unwrap() { + out.push(wire::parse_response(f)); + } + out +} + +#[test] +fn serve_put_object_exactly_cap_lands_and_cap_plus_one_is_refused() { + let root = setup(); + let ok = put_via_serve(&root, &blob_of_canonical_len(MIN_MAX_OBJECT_SIZE)); + assert!(ok[1].is_ok(), "exactly the cap lands: {:?}", ok[1]); + + let over = put_via_serve(&root, &blob_of_canonical_len(MIN_MAX_OBJECT_SIZE + 1)); + let err = over[1].as_ref().unwrap_err().to_string(); + assert!(err.contains("object size limit"), "{err}"); + let _ = std::fs::remove_dir_all(&root); +} + +#[test] +fn client_get_object_enforces_the_cap_on_the_response() { + set_cap(); + for (len, ok) in [ + (MIN_MAX_OBJECT_SIZE, true), + (MIN_MAX_OBJECT_SIZE + 1, false), + ] { + let obj = blob_of_canonical_len(len); + // Canned server: HELLO ok, then the GetObject response. + let mut canned = Vec::new(); + wire::write_ok(&mut canned, &wire::u32_body(wire::PROTOCOL_VERSION)).unwrap(); + wire::write_ok(&mut canned, &obj.encode()).unwrap(); + let client = WireClient::handshake(std::io::Cursor::new(canned), std::io::sink()).unwrap(); + let got = client.get_object(&obj.id()); + assert_eq!(got.is_ok(), ok, "len {len}: {got:?}"); + if let Err(e) = got { + assert!(e.to_string().contains("object size limit"), "{e}"); + } + } +} diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 3e0c3dca..d4e51bd2 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -446,7 +446,8 @@ too, not just the src-control-side metadata. - A single max object size caps every untrusted length: wire frames, pack-record compressed length, the zstd *decompressed* output (decompression-bomb - guard), bucket pack bodies, by-id reply objects, the `SC_PACK_CHUNK` clamp, + guard), bucket pack bodies, by-id reply objects, the `SC_PACK_CHUNK` clamp (fixed at the + 2 MiB minimum cap minus one, since caps are receiver-local and unnegotiated), and object-decode collection counts. The default is 256 MiB. An operator can lower or raise it with `sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` (#213), within a compile-time range of 2 MiB to 1 GiB; a value outside that @@ -491,6 +492,19 @@ too, not just the src-control-side metadata. `cap + 64` bytes (`PutObject` header 37 bytes, response status 1, chunk tag 1), so an object of exactly the cap fits; the object body itself is then checked against the cap exactly. + - **A local-path source loads before the receiver refuses.** A clone or + fetch from a local-path remote reads each object through + `LocalTransport::get_object` (load plus encode) before the receiver's cap + rejects it, bounded by the local store budget. Wire sources refuse in + `WireClient::get_object`, before decoding. + - **Structural objects have no per-connection aggregate budget (pre-existing, + amplified by a raised cap).** `Object::blob_size` is 0 for trees, + snapshots, secrets and transcripts, so repeated `PutObject` of distinct + structural objects on a writable listener is bounded per object (the cap) + but not in aggregate per connection: only by `--max-connections` times + connection lifetime (`--timeout`). Prefer read-only or token-gated + listeners, keep `--max-object-size` low, and size `--max-connections` + down. An aggregate per-connection byte budget is tracked in #233. - **One length is checked after allocation:** bucket WAL metadata (manifest, log entry, idx) is capped at a fixed 256 MiB after the bucket GET has buffered it, so it bounds decode work, not the download. It is not tied to diff --git a/docs/adr/0039-security-hardening-sweep.md b/docs/adr/0039-security-hardening-sweep.md index ca31ab84..88d38d26 100644 --- a/docs/adr/0039-security-hardening-sweep.md +++ b/docs/adr/0039-security-hardening-sweep.md @@ -163,7 +163,9 @@ default of 256 MiB. Decisions: bodies (`bucket_transport::object_bytes`); server-side servable-size check (`check_servable_size`, against the largest configurable cap, not the server's own: the cap is receiver-only, so a server with a lower cap still - serves a larger local object by id, as `GetPack` does); the `SC_PACK_CHUNK` clamp (`cap - 1`, a chunk + serves a larger local object by id, as `GetPack` does); the `SC_PACK_CHUNK` clamp (fixed at `MIN_MAX_OBJECT_SIZE - 1`, not the local + cap: caps are receiver-local and unnegotiated, so every chunk frame must fit + the smallest valid receiver; a chunk frame carries a tag byte); the `--max-pack-size` floor (`validate_max_pack_size`). Two exceptions are deliberately fixed, not derived from the cap. Bucket WAL metadata (manifest, log entry, idx) stays capped at the default @@ -211,6 +213,16 @@ connection's own store budget (each served connection opens its own store, the 1 GiB ceiling about 2 GiB + 2 GiB, per connection, times `--max-connections` (THREAT-MODEL.md). +Known gaps, not addressed here. `Object::blob_size` is 0 for structural +objects (trees, snapshots, secrets, transcripts), so a writable listener has no +per-connection aggregate byte bound on repeated `PutObject` of distinct +structural objects; each is capped, the sum is bounded only by +`--max-connections` x connection lifetime. This predates #213 and a raised cap +amplifies it: prefer read-only or token-gated listeners, keep the cap low. +Tracked in #233. A local-path remote loads and encodes an object before the +receiver's cap refuses it (`TransportSource::get`), bounded by the store +budget. + For `ssh://` remotes the client builds the remote `sc serve --stdio` command itself and ssh does not forward the environment, so the server end of an ssh remote gets a non-default cap only from the server's own environment (a