Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .claude/skills/sc-cli-reference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,20 @@ cargo run --bin sc -- serve --http <addr> <path> [--read-only] [--allow-public]
# cap, both --http and --stdio, default 16 GiB,
# 0=unlimited, floor = the max object size;
# counted mid-stream abort -> EC_TOO_LARGE (P31)
# --max-conn-bytes <bytes|KiB|MiB|GiB>: per-connection
# write budget, both --http and --stdio: every
# PutObject's bytes + per PutPack the larger of
# its spool bytes and the decoded bytes held
# (structural records; all records on --store);
# over it -> typed "connection write budget
# exceeded" error, then the connection closes.
# Default = --max-pack-size (16 GiB if 0) + the
# max object size; must be > 0, >= the max object
# size and >= a nonzero --max-pack-size, else
# startup aborts. Reads unbudgeted. The default is
# ~16.25 GiB and does NOT bound RAM: lower it with
# --max-pack-size; --http warns when a writable
# non-loopback listener uses the default (#233)
# --max-object-size <bytes|KiB|MiB|GiB>: largest
# single object accepted from a peer, default
# 256MiB, range 2MiB..1GiB, out of range aborts
Expand Down
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,14 @@ transport-adjacent. The rest, imperatively:
transfer path only — a larger blob commits locally but fails every
subsequent push/fetch/clone at the receiver. There is no negotiation: each
receiver applies its own cap, so it must admit the objects it receives. Never inline a size in a new length guard; read the accessor.
- **Writes are budgeted per connection (#233).** `sc serve --max-conn-bytes`
(default `--max-pack-size` plus the max object size; must be > 0 and at least
both) counts `PutObject` bytes (structural objects included, `blob_size` 0 or
not) and, per `PutPack`, the larger of its spool bytes and the decoded bytes
held (structural records; every record on a `--store` bucket); the default
(~16.25 GiB) does not bound RAM, lower it with `--max-pack-size`; exceeding it replies `ConnectionBudgetExceeded` (`EC_BUDGET`, no
version bump) and closes the connection. Reads are not budgeted. A new
object-write verb must charge `ConnBudget` in `serve_session`.
- **The P35 desktop app is a keyless, read-only top-level adapter.** Its
renderer has five typed repository commands, no general filesystem/shell
capability, and no DTO field capable of carrying protected ciphertext or
Expand Down
6 changes: 6 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,12 @@ scale-&-reach horizon):
`sc serve --max-object-size` and `SC_MAX_OBJECT_SIZE` set the per-object cap
(default 256 MiB, range 2 MiB to 1 GiB, out of range aborts startup). No wire
negotiation: each receiver applies its own cap.
- **Per-connection write budget — shipped (#233, ADR-0039 second addendum).**
`sc serve --max-conn-bytes` bounds the sum of `PutObject` bytes and `PutPack`
bytes (per pack, the larger of spool bytes and decoded bytes held) one connection can
make the server retain; over it the server replies
`ConnectionBudgetExceeded` and closes. No `PROTOCOL_VERSION` bump. Not built:
a request-count or CPU budget, and a per-peer (cross-connection) budget.
- **P28 review follow-ons — shipped.** `refs::write_head`/`delete_branch` run
`validate_branch_name`; `SC_PACK_CHUNK` is clamped to the minimum configurable object cap minus one (2 MiB - 1, so every chunk frame fits any valid receiver) (a chunk frame carries a tag byte too; an
oversized value used to make every transfer fail with a confusing receiver
Expand Down
55 changes: 53 additions & 2 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,17 @@ enum Cmd {
/// size (default 256 MiB) when non-zero. (P31)
#[arg(long)]
max_pack_size: Option<u64>,
/// Per-connection budget over everything a peer makes the server
/// write (every PutObject, structural objects included, plus per
/// PutPack the larger of its spool bytes and the decoded bytes held):
/// bytes, or a number with a KiB/MiB/GiB suffix (both --http and
/// --stdio). Exceeding it replies an error and closes the
/// connection; reads are not budgeted. Default: --max-pack-size (16
/// GiB if unlimited) plus the max object size. Must be greater than
/// zero, at least the max object size, and at least --max-pack-size
/// (a nonzero one). (#233)
#[arg(long, value_name = "SIZE")]
max_conn_bytes: Option<String>,
/// Largest single object accepted from a peer: bytes, or a number
/// with a KiB/MiB/GiB suffix. Default 256MiB, range 2MiB to 1GiB;
/// out-of-range values are refused at startup, never clamped. Clients
Expand Down Expand Up @@ -1099,6 +1110,7 @@ fn main() -> Result<()> {
max_connections,
timeout,
max_pack_size,
max_conn_bytes,
max_object_size: _,
tls,
tls_cert,
Expand Down Expand Up @@ -1129,6 +1141,7 @@ fn main() -> Result<()> {
max_connections,
timeout,
max_pack_size,
max_conn_bytes,
tls,
tls_cert,
tls_key,
Expand Down Expand Up @@ -3801,14 +3814,22 @@ fn run_clone_git(url: &str, dst: &std::path::Path) -> Result<()> {
Ok(())
}

/// Does the listener run on the default connection budget with no explicit
/// bound on either knob? A nonzero `--max-pack-size` already lowers the default
/// budget with it; `0` (unlimited) still falls back to the 16 GiB default.
fn runs_default_budget(conn_bytes_given: bool, max_pack_size: Option<u64>) -> bool {
!conn_bytes_given && matches!(max_pack_size, None | Some(0))
}

/// Serve a repo over the wire protocol: either on stdin/stdout (invoked as
/// the remote-side command by `ssh` for `ssh://` remotes), or over TCP for
/// `sc+http://` clients (P26, `--http <addr>`; blocks until the listener is
/// dropped, e.g. by process termination). Exactly one mode is required.
/// `--read-only`/`--allow-public` are `--http`-only (P29): `--stdio`
/// delegates auth/access entirely to ssh, so combining them is refused
/// rather than silently ignored. `--max-connections`/`--timeout` are also
/// `--http`-only (P31). `--max-pack-size` applies to both (P31). `--store`
/// `--http`-only (P31). `--max-pack-size` and `--max-conn-bytes` apply
/// to both (P31, #233). `--store`
/// (P36c) redirects served content to a bucket WAL remote; `path` remains
/// the serve home (tokens/TLS identity/scratch) either way. A malformed
/// `--store` URL is validated up front, before any bind — same fail-fast
Expand All @@ -3821,12 +3842,22 @@ fn run_serve(
max_connections: Option<u32>,
timeout: Option<u64>,
max_pack_size: Option<u64>,
max_conn_bytes: Option<String>,
tls: bool,
tls_cert: Option<PathBuf>,
tls_key: Option<PathBuf>,
store: Option<String>,
path: PathBuf,
) -> Result<()> {
let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE);
let default_budget = runs_default_budget(max_conn_bytes.is_some(), max_pack_size);
let conn_bytes = match max_conn_bytes {
Some(raw) => {
scl_core::parse_byte_size(&raw).map_err(|e| anyhow::anyhow!("--max-conn-bytes: {e}"))?
}
None => scl_repo::wire::default_max_conn_bytes(max_pack),
};
scl_repo::wire::validate_max_conn_bytes(conn_bytes, max_pack, scl_core::max_object_size())?;
if let Some(url) = &store {
scl_repo::BucketUrl::parse(url)?; // fail fast on malformed URLs, before any bind
}
Expand Down Expand Up @@ -3863,12 +3894,12 @@ fn run_serve(
path.display()
);
}
let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE);
scl_repo::wire::validate_max_pack_size(max_pack, scl_core::max_object_size())?;
let policy = scl_repo::wire::WirePolicy {
read_only: false,
max_pack_size: max_pack,
ro_drain_cap: scl_repo::wire::RO_DRAIN_CAP,
max_conn_bytes: conn_bytes,
};
let mut stdin = std::io::stdin().lock();
let mut stdout = std::io::stdout().lock();
Expand All @@ -3895,6 +3926,15 @@ fn run_serve(
if let Some(m) = max_pack_size {
limits.max_pack_size = m;
}
limits.max_conn_bytes = conn_bytes;
if default_budget && !read_only && !scl_repo::http_transport::addr_is_loopback(&addr) {
eprintln!(
"warning: writable listener on {addr} runs with the default --max-conn-bytes \
({conn_bytes} bytes per connection, times --max-connections); this does not \
bound server memory or disk tightly. Lower --max-conn-bytes and \
--max-pack-size together."
);
}
let tls_mode = match (tls, tls_cert, tls_key) {
(false, None, None) => scl_repo::http_transport::TlsMode::Off,
(false, _, _) => anyhow::bail!("--tls-cert/--tls-key require --tls"),
Expand Down Expand Up @@ -4615,6 +4655,17 @@ mod tests {
use super::*;
use std::time::Duration;

#[test]
fn default_budget_warning_logic() {
assert!(runs_default_budget(false, None));
assert!(
runs_default_budget(false, Some(0)),
"0 = unlimited pack cap"
);
assert!(!runs_default_budget(false, Some(1 << 30)));
assert!(!runs_default_budget(true, None));
}

#[test]
fn parses_suffixed_durations() {
assert_eq!(
Expand Down
163 changes: 163 additions & 0 deletions crates/cli/tests/max_conn_bytes.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
//! `sc serve --max-conn-bytes` (#233, ADR-0039/0041 addenda): flag
//! parsing and bounds are checked at startup, and a real push over
//! `sc+http://` that exceeds the connection's write budget is refused with the
//! typed error while a push under it works.

use std::io::BufRead;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Output, Stdio};

fn sc(dir: &Path, args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_sc"))
.args(args)
.current_dir(dir)
.env_remove("SC_MAX_OBJECT_SIZE")
.env("SC_AUTHOR", "t")
.output()
.expect("sc runs")
}

fn tmp(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("scl-cli-connbytes-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}

fn stderr(o: &Output) -> String {
String::from_utf8_lossy(&o.stderr).into_owned()
}

struct Server(Child);
impl Drop for Server {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}

fn spawn_http_server(root: &Path, extra: &[&str]) -> (Server, String) {
let mut args = vec!["serve", "--http", "127.0.0.1:0"];
args.extend_from_slice(extra);
args.push(root.to_str().unwrap());
let mut child = Command::new(env!("CARGO_BIN_EXE_sc"))
.args(&args)
.env_remove("SC_MAX_OBJECT_SIZE")
.stdout(Stdio::piped())
.spawn()
.expect("spawn sc serve --http");
let mut reader = std::io::BufReader::new(child.stdout.take().unwrap());
let mut line = String::new();
assert!(reader.read_line(&mut line).unwrap() > 0, "server announced");
let addr = line
.trim()
.strip_prefix("listening on ")
.unwrap_or_else(|| panic!("unexpected startup line: {line:?}"))
.to_string();
std::thread::spawn(move || for _ in reader.lines() {});
(Server(child), addr)
}

#[test]
fn flag_is_parsed_and_bounds_checked_at_startup() {
let root = tmp("flag");
assert!(sc(&root, &["init"]).status.success());
// Unparsable, below the object cap (default 256 MiB), and below an
// explicit pack cap are all refused before serving.
for (args, needle) in [
(vec!["--max-conn-bytes", "x"], "--max-conn-bytes"),
(vec!["--max-conn-bytes", "1.5MiB"], "--max-conn-bytes"),
(vec!["--max-conn-bytes", "0"], "greater than zero"),
(
vec!["--max-conn-bytes", "1MiB"],
"below the max object size (268435456)",
),
(
vec![
"--max-object-size",
"2MiB",
"--max-pack-size",
"8388608",
"--max-conn-bytes",
"4MiB",
],
"below --max-pack-size",
),
] {
let mut a = vec!["serve", "--stdio"];
a.extend(&args);
a.push(".");
let o = sc(&root, &a);
assert!(!o.status.success(), "{args:?} must be rejected");
assert!(stderr(&o).contains(needle), "{args:?}: {}", stderr(&o));
}
std::fs::remove_dir_all(&root).unwrap();
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
assert!(!root.exists());
}

#[test]
fn a_push_over_the_budget_is_refused_and_one_under_it_works() {
// Unlimited pack cap so the connection budget is the binding limit.
let server_dir = tmp("srv");
assert!(sc(&server_dir, &["init"]).status.success());
let (server, addr) = spawn_http_server(
&server_dir,
&[
"--max-object-size",
"2MiB",
"--max-pack-size",
"0",
"--max-conn-bytes",
"2MiB",
],
);
let url = format!("sc+http://{addr}");

let client = tmp("cli");
assert!(sc(&client, &["init"]).status.success());
assert!(sc(&client, &["remote", "add", "origin", &url])
.status
.success());

// Incompressible content so the spooled pack is about as big as the file.
let noise = |len: usize| -> Vec<u8> {
let mut x = 0x9E37_79B9_7F4A_7C15u64;
(0..len)
.map(|_| {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
x as u8
})
.collect()
};
std::fs::write(client.join("big.bin"), noise(3 * 1024 * 1024)).unwrap();
assert!(sc(&client, &["commit", "-m", "big"]).status.success());
let o = sc(&client, &["push", "origin"]);
assert!(!o.status.success(), "3 MiB push exceeds a 2 MiB budget");
let err = stderr(&o);
assert!(
err.contains("connection write budget exceeded") && err.contains("--max-conn-bytes"),
"typed refusal names the knob: {err}"
);

let small = tmp("small");
assert!(sc(&small, &["init"]).status.success());
assert!(sc(&small, &["remote", "add", "origin", &url])
.status
.success());
std::fs::write(small.join("a.txt"), "hello\n").unwrap();
assert!(sc(&small, &["commit", "-m", "small"]).status.success());
let o = sc(&small, &["push", "origin"]);
assert!(
o.status.success(),
"a push under the budget works: {}",
stderr(&o)
);

drop(server); // stop the listener before deleting its root
for d in [server_dir, client, small] {
std::fs::remove_dir_all(&d).unwrap();
assert!(!d.exists());
}
}
Loading
Loading