From 39c45ce3370a3a47ef8fcce118b6f5ff000f8320 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:56:40 +0300 Subject: [PATCH 01/12] docs(adr): design bucket remote compaction and gc (#216) --- docs/adr/0055-bucket-compaction.md | 222 +++++++++++++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 docs/adr/0055-bucket-compaction.md diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md new file mode 100644 index 00000000..27c706d9 --- /dev/null +++ b/docs/adr/0055-bucket-compaction.md @@ -0,0 +1,222 @@ +# ADR-0055: Bucket remote compaction and gc + +- **Status:** Accepted +- **Date:** 2026-10-20 +- **Phase:** 36 follow-on (#216) + +## Context + +ADR-0046's bucket remote only ever adds objects. `log/` entries, +`checkpoints/` objects and `packs/.{pack,idx}` pairs accumulate +without bound. Two costs follow: + +- Storage and listing cost grow with every push. A cold start already stops at + the latest checkpoint (P36b), but it still fetches one `.idx` per on-chain + pack, so the pack count is the cold-start cost. +- `walfmt::MAX_LIST` (65536) caps a checkpoint's `refs` and `packs`. A fold + that would exceed it is skipped (`maybe_fold_checkpoint`), so a remote with + more than 65536 packs loses folding for good and every reader walks an + ever-longer tail. + +The hard part is the cutoff: nothing may be removed that a reader or writer +can still need, with disposable `sc serve --store` instances, concurrent +writers, and no lease primitive (ROADMAP "Leases", unbuilt). + +## Decision + +### No format change + +`Manifest` decodes strictly: `header()` refuses any version other than 1 and +`done()` refuses trailing bytes. A new `compacted_through` field would +therefore make every deployed reader fail closed, which is a format break. It +is also unnecessary: `Manifest.checkpoint_seq` already says where a cold start +stops. Compaction adds no field, no object kind and no `PROTOCOL_VERSION` +bump. Everything it writes is an ordinary record an old client decodes: a log +entry with `updates = []`, and a checkpoint whose `packs` list is a set that +covers the chain instead of a literal accumulation. + +### What compaction does + +`BucketTransport::compact(&CompactOptions)` runs two phases. Each is +independently safe and either may be a no-op. + +**Phase 1, consolidate (additive, fenced by the manifest CAS).** The live pack +set is `view.packs`, deduplicated. Packs smaller than `small_pack_bytes` +(default 16 MiB, sizes from `list_meta`) are grouped greedily, smallest first, +into groups of at most `group_bytes` (default 64 MiB); a group needs at least +two members. Each group is read, its objects are deduplicated by id, and one +new content-addressed pack plus idx is uploaded. This bounds peak RAM to a few +times `group_bytes`, and packs above the threshold are never rewritten. + +The result is committed with one CAS, which is both the fence and the fold: + +1. `refresh`, then recompute the plan if the members are no longer all live. +2. Claim `log/` (`s > head_seq`, same claim loop as `update_ref`) with + `LogEntry { parent_seq: head_seq, packs: [new packs], updates: [] }`. +3. `put_new` `checkpoints/` with `refs = view.refs` and + `packs = dedup(view.packs) - replaced + new`. If this exceeds `MAX_LIST` + the run stops before writing it. +4. CAS the manifest from the tag of the view in step 1 to + `{head_seq: s, checkpoint_seq: s}`. + +The CAS validates that the manifest has not moved since step 1, so the +checkpoint is exactly the state through `s`. A lost CAS leaves an off-chain +entry and checkpoint (ordinary debris) and the run re-plans, bounded retries. +Because the manifest tag changes, every writer holding an older tag loses its +CAS and must refresh. + +Old packs stay in place. After the CAS the superseded packs are named by no +chain a new reader walks, but older checkpoints and log entries still name +them. They are removed by phase 2, a grace period later. + +**Phase 2, sweep (deletes, behind a cutoff).** All deletions are of objects +that no current or future manifest chain references; the manifest itself is +never deleted. + +- **Cutoff K.** Let `K` be the highest `checkpoints/` object with + `seq <= manifest.checkpoint_seq` whose last-modified time is at least + `grace` ago. No `K` means nothing is eligible by seq (orphan packs may still + be, below). +- **Removed:** `log/` for `seq <= K` (including off-chain debris), + `checkpoints/` for `seq < K`, and pack objects named by nothing + retained and at least `grace` old. +- **Retained set** for packs: the current chain (`view.packs`), every kept + checkpoint (`seq >= K`) and every kept log entry (`seq > K`, on-chain or + not). A pack named by an old checkpoint or a straggler's in-flight entry is + therefore kept. If a retained object cannot be decoded the sweep stops + without deleting packs (fail closed). +- **Order:** log entries, then older checkpoints, then idx, then pack. Every + prefix of this order leaves a bucket a cold start reads, because none of + these are reachable from the manifest being deleted-against. The manifest + is re-read before each batch; if it moved the sweep stops and reports it. +- `K` rather than "the current checkpoint" matters on a busy remote: folds + happen every 64 pushes, so the newest checkpoint is rarely `grace` old, + while an older one is. + +`K = 0` is a valid answer on a young remote. A run on a bucket whose newest +checkpoint is younger than `grace` consolidates (if worthwhile) and deletes +nothing; the next run after `grace` deletes. This is deliberate: it is the +two-phase "supersede, then delete after T" scheme without a tombstone object, +using the checkpoint's own creation time as the supersession clock. + +### Reader protocol + +Readers never take a lock and never need compaction to be quiet. The rule: + +> When a key the current view names is absent (checkpoint, log entry, idx, or +> pack body), re-read the manifest unconditionally. If its tag differs from +> the view's, discard the view and restart from the new manifest (bounded +> retries, 8). If it is unchanged, the absence is real corruption and the +> operation fails closed exactly as before. + +This is sound because deletion only follows a manifest CAS that moved the +tag: the sweep deletes only objects unreferenced by the manifest it read, and +manifest tags never repeat (`head_seq` is strictly increasing and a fold +raises `checkpoint_seq`, so the content hash cannot return to an earlier +value; S3 ETags are content hashes). A reader that lost an object therefore +always finds a newer manifest on re-read. Object ids stay valid across the +restart because consolidation preserves every object, only moving it to a new +pack, so `get_pack`'s id lists computed before a restart remain valid. +Restart is at the object-fetch level (`object_bytes`, the signature/transcript +scan, `refresh`), never in the middle of the bytes `get_pack` has already +written. + +`grace` is therefore not what makes readers correct; it only makes restarts +rare. A reader older than `grace` is the only one that can be restarted. + +### Writer protocol + +A push uploads packs, claims a log slot, then CASes the manifest. An +uploaded-but-unreferenced pack is indistinguishable from debris, and an object +store offers no atomic "delete if the manifest is unchanged". Two defenses: + +1. **Age.** The sweep removes an unreferenced pack only if its last-modified + time is at least `grace` old (default 24 h). A writer's gap between upload + and CAS is milliseconds to minutes. +2. **Verify before commit.** `update_ref` checks (`list_meta("packs/.")`) + that each pending pack and idx still exist before every log-slot claim, and + fails with a clear retry-the-push error if not, instead of committing an + entry that names a deleted pack. + +**Residual limit (documented, not closed).** A writer that stalls longer than +`grace` between uploading a pack and its CAS, or re-pushes byte-identical +content to an old orphan in the same instant a sweep selects it, can still +commit an entry naming a deleted pack. Closing that needs a lease or a +conditional delete, which the `Bucket` trait lacks (ROADMAP "Leases"). Run +compaction with a `grace` well above any plausible writer stall; the 24 h +default matches `sc gc --prune-expire`. + +### MAX_LIST + +Folding is skipped only when the cumulative `packs` (or `refs`) of a fold +exceed 65536. Phase 1 shrinks the pack axis: after consolidation the +checkpoint holds `non-replaced + new` packs, so later folds fit again and the +tail stops growing. The ref axis cannot be relieved: refs are live state, not +history. A remote with more than 65536 live branches stays unfoldable; raising +the cap is a format question and out of scope. Compaction refuses (no writes +beyond debris) if even the consolidated list would exceed the cap. + +Dropping on-chain packs whose objects are unreachable from every ref is +**out of scope**. It changes what `has_object` reports, so a pusher that +skipped re-uploading an object could reference a dropped pack; it also needs a +reachability walk of the whole bucket. A pack is "live" here if the chain names +it. + +### Command surface + +`sc remote compact [--dry-run] [--prune-expire ]` (default +`24h`, same flag and default as `sc gc`). It is the smallest surface: it needs +only the configured remote's URL, adds no wire verb, and is not reachable +through `sc serve`, so a served bucket's clients can never trigger a delete. +`--dry-run` plans phase 1 and phase 2 against the live bucket and prints what +would be consolidated and deleted (counts, bytes, and how many objects are +held back by the grace window), writing nothing. + +It refuses a non-bucket remote. Rejected alternatives: `sc gc --remote` +(couples two unrelated reachability models), and a `--compact` flag on +`sc serve --store` (servers are disposable and many; a delete should be an +operator action, not a side effect of an instance starting). + +### objio + +The `Bucket` trait gains two methods, implemented by `DirBucket` and +`S3Bucket`: + +- `list_meta(prefix)`: keys with size and last-modified time, raw string + prefix, sorted. S3 uses `ListObjectsV2`. +- `delete(key)`: idempotent; an absent key is success. + +## Consequences + +- Storage and cold-start cost are bounded by the live data plus one grace + window. A bucket that was past `MAX_LIST` regains folding after one run. +- Compaction needs bucket `list` and `delete` rights, which an ordinary + pusher or reader does not. See the THREAT-MODEL bucket section. +- Only one compaction is useful at a time. Concurrent runs are safe (each + recomputes from the manifest it read, the loser's CAS fails) but waste work. +- Consolidation rewrites small packs once. Large packs are never touched, so + there is no write amplification on big pushes. +- Old clients keep working against a compacted bucket. A very old reader may + see one "retry" error if it straddles the sweep, and a re-run succeeds. + +## Alternatives considered + +- **`compacted_through` in the manifest.** Rejected: version-gated field breaks + every deployed reader (strict decode) and is redundant with + `checkpoint_seq`. +- **Tombstone objects (`gc/` listing keys to delete after T).** Needs a + new object kind and a second pass; using the checkpoint's creation time as + the supersession clock gives the same ordering with nothing new in the + bucket. The checkpoint clock can be wrong (a folder stalled between + `put_new` and CAS), but that only costs a reader a restart, not safety. +- **Delete immediately after consolidation.** Simple, correct for readers + (they restart), but makes the writer-stall window as wide as one object's + age and restarts every in-flight clone. The grace window costs one extra run. +- **Repack everything into one pack.** Unbounded RAM and write amplification. + +## As built + +`crates/objio` (`ObjectMeta`, `Bucket::list_meta`/`delete`, contract suite); +`crates/repo/src/bucket_transport/compact.rs` (`compact`, `CompactOptions`, +`CompactReport`); reader restart and the pre-commit pack check in +`bucket_transport.rs`; `sc remote compact` in `crates/cli`. From f197af82a6e8c5693039783f468b811a8028dbd2 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 19:56:41 +0300 Subject: [PATCH 02/12] feat(objio): list_meta and delete on the Bucket trait for compaction (#216) --- crates/objio/src/dir.rs | 38 +++++++++++++---- crates/objio/src/lib.rs | 44 ++++++++++++++++++- crates/objio/src/s3.rs | 66 ++++++++++++++++++++++++++++- crates/repo/src/bucket_transport.rs | 6 +++ 4 files changed, 144 insertions(+), 10 deletions(-) diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs index 6ea3f288..22827417 100644 --- a/crates/objio/src/dir.rs +++ b/crates/objio/src/dir.rs @@ -1,6 +1,6 @@ //! Local-directory `Bucket` — the test/demo backend behind `sc+wal://`. -use crate::{validate_key, Bucket, Error, Fetched, Result}; +use crate::{validate_key, Bucket, Error, Fetched, ObjectMeta, Result}; use std::path::{Path, PathBuf}; /// A local filesystem bucket implementation using a directory as the backing @@ -116,7 +116,11 @@ impl Bucket for DirBucket { } fn list(&self, prefix: &str) -> Result> { - fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { + Ok(self.list_meta(prefix)?.into_iter().map(|m| m.key).collect()) + } + + fn list_meta(&self, prefix: &str) -> Result> { + fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { let rd = match std::fs::read_dir(dir) { Ok(rd) => rd, Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), @@ -131,15 +135,25 @@ impl Bucket for DirBucket { }) { continue; } - if p.is_dir() { + // A concurrent delete/rename can remove the entry between the + // readdir and the stat; a key that is gone is just not listed. + let md = match std::fs::metadata(&p) { + Ok(md) => md, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, + Err(e) => return Err(e), + }; + if md.is_dir() { walk(&p, root, out)?; } else { - out.push( - p.strip_prefix(root) + out.push(ObjectMeta { + key: p + .strip_prefix(root) .unwrap() .to_string_lossy() .replace('\\', "/"), - ); + size: md.len(), + modified: md.modified()?, + }); } } Ok(()) @@ -147,8 +161,16 @@ impl Bucket for DirBucket { validate_key(prefix.trim_end_matches('/'))?; let mut out = Vec::new(); walk(&self.root, &self.root, &mut out)?; - out.retain(|k| k.starts_with(prefix)); - out.sort(); + out.retain(|m| m.key.starts_with(prefix)); + out.sort_by(|a, b| a.key.cmp(&b.key)); Ok(out) } + + fn delete(&self, key: &str) -> Result<()> { + match std::fs::remove_file(self.key_path(key)?) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), + } + } } diff --git a/crates/objio/src/lib.rs b/crates/objio/src/lib.rs index 9900051b..5cae152a 100644 --- a/crates/objio/src/lib.rs +++ b/crates/objio/src/lib.rs @@ -34,8 +34,18 @@ pub enum Fetched { New { bytes: Vec, tag: String }, } +/// A listed key with the metadata compaction needs (ADR-0055). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ObjectMeta { + pub key: String, + pub size: u64, + /// Last-modified time as the bucket reports it (S3 `LastModified`, file mtime). + pub modified: std::time::SystemTime, +} + /// A flat key/value object store with the three primitives the WAL needs: -/// conditional read, create-if-absent, and compare-and-swap overwrite. +/// conditional read, create-if-absent, and compare-and-swap overwrite; plus +/// the metadata listing and delete that compaction (ADR-0055) adds. pub trait Bucket: Send { /// Conditional read. `cached_tag` matching the current value returns /// [`Fetched::Unchanged`] without transferring bytes. @@ -54,6 +64,12 @@ pub trait Bucket: Send { ) -> Result>; /// Keys under `prefix`, sorted. fn list(&self, prefix: &str) -> Result>; + /// Keys whose name starts with the raw string `prefix` (not a directory + /// scope, so `packs/ab12.` matches `packs/ab12.pack`), with size and + /// last-modified time, sorted by key. + fn list_meta(&self, prefix: &str) -> Result>; + /// Remove `key`. Idempotent: an absent key is success. + fn delete(&self, key: &str) -> Result<()>; } /// Reject traversal and absolute keys before any backend touches them. @@ -117,6 +133,32 @@ pub fn contract_suite(b: &dyn Bucket) { vec!["log/1".to_string(), "log/2".to_string()] ); assert_eq!(b.list("packs/").unwrap(), Vec::::new()); + // list_meta: raw string prefix, sorted, with size and a sane mtime + assert!(b.put_new("packs/ab12.pack", b"12345").unwrap()); + assert!(b.put_new("packs/ab12.idx", b"1").unwrap()); + assert!(b.put_new("packs/ab34.pack", b"x").unwrap()); + let metas = b.list_meta("packs/ab12.").unwrap(); + assert_eq!( + metas.iter().map(|m| m.key.as_str()).collect::>(), + vec!["packs/ab12.idx", "packs/ab12.pack"] + ); + assert_eq!(metas[1].size, 5); + let age = std::time::SystemTime::now() + .duration_since(metas[1].modified) + .unwrap_or_default(); + assert!(age < std::time::Duration::from_secs(3600)); + assert_eq!(b.list_meta("packs/").unwrap().len(), 3); + assert!(b.list_meta("nothing/").unwrap().is_empty()); + // delete: removes, is idempotent, touches only its key + b.delete("packs/ab12.pack").unwrap(); + b.delete("packs/ab12.pack").unwrap(); + assert!(matches!( + b.get("packs/ab12.pack", None).unwrap(), + Fetched::Absent + )); + assert_eq!(b.list_meta("packs/").unwrap().len(), 2); + // a deleted key can be created again + assert!(b.put_new("packs/ab12.pack", b"again").unwrap()); } #[cfg(test)] diff --git a/crates/objio/src/s3.rs b/crates/objio/src/s3.rs index 2eb9d36c..67ddc665 100644 --- a/crates/objio/src/s3.rs +++ b/crates/objio/src/s3.rs @@ -1,6 +1,6 @@ //! S3-compatible `Bucket` backend (AWS, MinIO, R2 via AWS_ENDPOINT_URL_S3). -use crate::{validate_key, Bucket, Error, Fetched, Result}; +use crate::{validate_key, Bucket, Error, Fetched, ObjectMeta, Result}; /// A `Bucket` backed by an S3-compatible object store. Credentials, region, /// and endpoint come entirely from the SDK's standard chain (`AWS_*` env, @@ -173,6 +173,70 @@ impl Bucket for S3Bucket { out.sort(); Ok(out) } + + fn list_meta(&self, prefix: &str) -> Result> { + // Raw string prefix (no directory scoping), unlike `list`. + validate_key(prefix.trim_end_matches('/'))?; + let full = if self.prefix.is_empty() { + prefix.to_string() + } else { + format!("{}/{prefix}", self.prefix) + }; + let strip = if self.prefix.is_empty() { + String::new() + } else { + format!("{}/", self.prefix) + }; + let mut out = Vec::new(); + let mut cont: Option = None; + loop { + let mut req = self + .client + .list_objects_v2() + .bucket(&self.bucket) + .prefix(&full); + if let Some(c) = &cont { + req = req.continuation_token(c); + } + let resp = self + .rt + .block_on(req.send()) + .map_err(|e| Error::Backend(format!("s3 list: {e}")))?; + for obj in resp.contents() { + let Some(k) = obj.key() else { continue }; + let modified = obj + .last_modified() + .and_then(|t| std::time::SystemTime::try_from(*t).ok()) + .ok_or_else(|| Error::Backend(format!("s3 list: no LastModified for {k}")))?; + out.push(ObjectMeta { + key: k.strip_prefix(&strip).unwrap_or(k).to_string(), + size: obj.size().unwrap_or(0).max(0) as u64, + modified, + }); + } + match resp.next_continuation_token() { + Some(c) => cont = Some(c.to_string()), + None => break, + } + } + out.sort_by(|a, b| a.key.cmp(&b.key)); + Ok(out) + } + + fn delete(&self, key: &str) -> Result<()> { + let k = self.full_key(key)?; + // S3 DeleteObject succeeds on an absent key, which is the contract. + self.rt + .block_on( + self.client + .delete_object() + .bucket(&self.bucket) + .key(&k) + .send(), + ) + .map(|_| ()) + .map_err(|e| Error::Backend(format!("s3 delete {k}: {e}"))) + } } #[cfg(test)] diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index b16c9043..508757ec 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -1737,6 +1737,12 @@ mod tests { fn list(&self, prefix: &str) -> scl_objio::Result> { self.0.list(prefix) } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + self.0.list_meta(prefix) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.0.delete(key) + } } #[test] From bafea54712c25d498acd78d72966036046be12ff Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:21:24 +0300 Subject: [PATCH 03/12] feat(repo): bucket remote compaction, reader restart and pre-commit pack check (#216) --- crates/cli/src/main.rs | 71 ++ crates/cli/tests/bucket_remote.rs | 72 ++ crates/objio/src/dir.rs | 8 +- crates/repo/src/bucket_transport.rs | 129 ++- crates/repo/src/bucket_transport/compact.rs | 446 ++++++++++ .../src/bucket_transport/compact/tests.rs | 798 ++++++++++++++++++ crates/repo/src/lib.rs | 4 +- 7 files changed, 1514 insertions(+), 14 deletions(-) create mode 100644 crates/repo/src/bucket_transport/compact.rs create mode 100644 crates/repo/src/bucket_transport/compact/tests.rs diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 26269204..61c6bbc4 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -803,6 +803,19 @@ enum RemoteOp { }, /// List configured remotes. List, + /// Compact a bucket remote (`sc+wal://`, `sc+s3://`): merge small packs, + /// then delete log entries, checkpoints and packs no reader or writer can + /// still need (ADR-0055). Needs bucket list and delete rights. + Compact { + name: String, + /// Plan and report; write and delete nothing. + #[arg(long)] + dry_run: bool, + /// Delete only what has been unreferenced for at least this long + /// (e.g. 24h, 7d). Keep it well above any push's duration. + #[arg(long, default_value = "24h")] + prune_expire: String, + }, } /// Top-level `sc serve` subcommand namespace, distinct from the plain @@ -4109,6 +4122,29 @@ fn run_remote(op: RemoteOp) -> Result<()> { println!("added remote {name} -> {url}"); } } + RemoteOp::Compact { + name, + dry_run, + prune_expire, + } => { + let grace = parse_duration(&prune_expire)?; + let url = repo + .remotes()? + .into_iter() + .find(|(n, _)| *n == name) + .map(|(_, u)| u) + .ok_or_else(|| anyhow::anyhow!("no such remote: {name}"))?; + if !(url.starts_with("sc+wal://") || url.starts_with("sc+s3://")) { + anyhow::bail!("remote '{name}' is not a bucket remote ({url}); nothing to compact"); + } + let transport = scl_repo::BucketTransport::open(&url)?; + let rep = transport.compact(&scl_repo::CompactOptions { + prune_expire: grace, + dry_run, + ..Default::default() + })?; + print_compact_report(&name, &url, &rep); + } RemoteOp::List => { let cfg = scl_repo::RemoteConfig::load(repo.layout())?; for (name, url) in repo.remotes()? { @@ -4123,6 +4159,41 @@ fn run_remote(op: RemoteOp) -> Result<()> { Ok(()) } +/// Human-readable summary of a bucket compaction (`dry_run` reads "would"). +fn print_compact_report(name: &str, url: &str, rep: &scl_repo::CompactReport) { + let (merge, delete) = if rep.dry_run { + ("would merge", "would delete") + } else { + ("merged", "deleted") + }; + println!( + "compact {name} ({url}){}", + if rep.dry_run { " [dry run]" } else { "" } + ); + println!( + " {merge} {} pack(s) into {}", + rep.packs_merged, rep.packs_created + ); + println!( + " {delete} {} log entr{}, {} checkpoint(s), {} pack(s) ({} bytes), cutoff checkpoint {}", + rep.logs_deleted, + if rep.logs_deleted == 1 { "y" } else { "ies" }, + rep.checkpoints_deleted, + rep.packs_deleted, + rep.bytes_deleted, + rep.cutoff + ); + if rep.packs_held_back > 0 { + println!( + " kept {} unreferenced pack(s) younger than the grace window", + rep.packs_held_back + ); + } + for note in &rep.notes { + println!(" note: {note}"); + } +} + /// Fetch from `remote`, dispatching on its configured kind: a `git`-kind /// remote gets a full-history import into a remote-tracking ref (see /// `run_fetch_git`); an `sc`-kind remote (or unconfigured, defaulting to `sc`) diff --git a/crates/cli/tests/bucket_remote.rs b/crates/cli/tests/bucket_remote.rs index 4a984bab..68b902b5 100644 --- a/crates/cli/tests/bucket_remote.rs +++ b/crates/cli/tests/bucket_remote.rs @@ -86,6 +86,78 @@ fn bucket_clone_push_fetch_round_trip_and_url_validation() { } } +/// #216: `sc remote compact` merges and sweeps a bucket remote, `--dry-run` +/// writes nothing, a default grace window deletes nothing young, and a +/// non-bucket remote is refused. A clone and a fetch still work afterwards. +#[test] +fn remote_compact_merges_and_sweeps_a_bucket_and_clones_still_work() { + let a = tmp("compact-a"); + let bucket = tmp("compact-bucket"); + let local = tmp("compact-local"); + let url = format!("sc+wal://{}", bucket.display()); + assert!(sc(&a, &["init"]).status.success()); + assert!(sc(&a, &["remote", "add", "origin", &url]).status.success()); + assert!(sc(&a, &["remote", "add", "plain", local.to_str().unwrap()]) + .status + .success()); + for i in 0..3 { + std::fs::write(a.join(format!("f{i}.txt")), format!("content {i}")).unwrap(); + assert!(sc(&a, &["commit", "-m", &format!("c{i}")]).status.success()); + assert!(sc(&a, &["push", "origin"]).status.success()); + } + let count = |sub: &str| std::fs::read_dir(bucket.join(sub)).unwrap().count(); + let (logs, packs) = (count("log"), count("packs")); + assert_eq!((logs, packs), (3, 6), "3 pushes: 3 entries, 3 packs + idx"); + + // dry run: reports the plan, changes nothing + let out = sc(&a, &["remote", "compact", "origin", "--dry-run"]); + assert!(out.status.success(), "{out:?}"); + let text = String::from_utf8_lossy(&out.stdout); + assert!( + text.contains("[dry run]") && text.contains("would merge 3 pack(s) into 1"), + "{text}" + ); + assert_eq!((count("log"), count("packs")), (logs, packs)); + + // default 24h grace: merges, but deletes nothing young + let out = sc(&a, &["remote", "compact", "origin"]); + assert!(out.status.success(), "{out:?}"); + let text = String::from_utf8_lossy(&out.stdout); + assert!( + text.contains("merged 3 pack(s) into 1") && text.contains("kept 3 unreferenced"), + "{text}" + ); + assert_eq!( + count("packs"), + 8, + "3 old packs kept for the grace window + the merged one" + ); + + // zero grace: sweeps the old packs and the log + let out = sc(&a, &["remote", "compact", "origin", "--prune-expire", "0s"]); + assert!(out.status.success(), "{out:?}"); + assert_eq!((count("log"), count("packs")), (0, 2)); + + // refuses a non-bucket remote and an unknown one + assert!(!sc(&a, &["remote", "compact", "plain"]).status.success()); + assert!(!sc(&a, &["remote", "compact", "nope"]).status.success()); + + // the compacted bucket still clones and fetches + let parent = tmp("compact-parent"); + let b = parent.join("b"); + let out = sc(&parent, &["clone", &url, b.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + assert_eq!(std::fs::read(b.join("f2.txt")).unwrap(), b"content 2"); + std::fs::write(b.join("g.txt"), b"after compact").unwrap(); + assert!(sc(&b, &["commit", "-m", "c3"]).status.success()); + assert!(sc(&b, &["push", "origin"]).status.success()); + assert!(sc(&a, &["fetch", "origin"]).status.success()); + + for d in [&a, &bucket, &local, &parent] { + let _ = std::fs::remove_dir_all(d); + } +} + /// #229: a signed snapshot pushed to a bucket keeps its signature on a fresh /// clone, so `sc verify --require` passes there once the signer is trusted /// (trust is local config, written into the clone). diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs index 22827417..7029a849 100644 --- a/crates/objio/src/dir.rs +++ b/crates/objio/src/dir.rs @@ -160,7 +160,13 @@ impl Bucket for DirBucket { } validate_key(prefix.trim_end_matches('/'))?; let mut out = Vec::new(); - walk(&self.root, &self.root, &mut out)?; + // Keys live under their directory, so start the walk at the prefix's + // directory part instead of scanning the whole bucket. + let start = match prefix.rsplit_once('/') { + Some((dir, _)) => self.root.join(dir), + None => self.root.clone(), + }; + walk(&start, &self.root, &mut out)?; out.retain(|m| m.key.starts_with(prefix)); out.sort_by(|a, b| a.key.cmp(&b.key)); Ok(out) diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index 508757ec..fbd772e5 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -10,10 +10,18 @@ use crate::walfmt::{ use scl_core::pack::{parse_index, read_object_at_bounded, IndexEntry, PackWriter}; use scl_core::{Object, ObjectId}; use scl_objio::{Bucket, Fetched}; -use std::cell::RefCell; +use std::cell::{Cell, RefCell}; use std::collections::BTreeMap; use std::io::Write; +mod compact; +pub use compact::{CompactOptions, CompactReport}; + +/// How many times a read restarts from a fresh manifest after an object the +/// view names has vanished (ADR-0055). Each restart needs the manifest to have +/// moved, so a handful is plenty: one per compaction that lands mid-read. +const MAX_RESTARTS: u32 = 8; + /// Reconstructed state of the WAL at one manifest tag: the manifest itself, /// the flattened branch tips after replaying the parent chain oldest-first, /// and a lookup from object id to the pack that holds it. @@ -71,6 +79,14 @@ pub struct BucketTransport { /// it spares a long-lived `serve --store` session from re-reading every /// pack body on each `get_pack`. leaf_cache: RefCell>>, + /// Set whenever a read finds an object the manifest chain names absent + /// (ADR-0055): the signal for `refresh`/`restarting` to re-read the + /// manifest and tell a compaction that moved on (restart) from a corrupt + /// bucket (fail closed). + saw_absent: Cell, + /// Tag of the manifest the in-progress `refresh_once` is building from, so + /// a failed attempt can say which manifest it failed on. + attempt_tag: RefCell>, } /// A leaf object a pack holds that no tree or snapshot references, so the @@ -218,6 +234,8 @@ impl BucketTransport { staged: RefCell::new(Vec::new()), pending_packs: RefCell::new(Vec::new()), leaf_cache: RefCell::new(BTreeMap::new()), + saw_absent: Cell::new(false), + attempt_tag: RefCell::new(None), }; t.refresh()?; Ok(t) @@ -228,7 +246,60 @@ impl BucketTransport { /// (P36b) — seeded from that checkpoint's own `refs`/`packs` rather than /// replayed from `0` — then folding the tail entries on top (seq numbers /// are claims; the chain is the truth — off-chain entries are garbage). + /// + /// A compaction (ADR-0055) can delete log entries, checkpoints or packs + /// between reading the manifest and reading what it names. When that + /// happens the manifest has necessarily moved on, so the read restarts from + /// the new one; if the same manifest fails twice with a key absent, the + /// bucket is corrupt and the error stands. fn refresh(&self) -> Result<()> { + let mut failed_on: Option = None; + for _ in 0..MAX_RESTARTS { + self.saw_absent.set(false); + *self.attempt_tag.borrow_mut() = None; + match self.refresh_once() { + Err(e) if self.saw_absent.get() => { + let tag = self.attempt_tag.borrow_mut().take(); + if tag.is_none() || tag == failed_on { + return Err(e); + } + failed_on = tag; + } + r => return r, + } + } + self.refresh_once() + } + + /// Record that a key the chain names is absent and build the error for it. + fn gone(&self, msg: String) -> Error { + self.saw_absent.set(true); + Error::Wal(msg) + } + + /// Run a read against the current view; if it finds a named object gone + /// (a compaction deleted it), drop the view, re-read the manifest and run + /// it again. An unchanged manifest means the absence is real: fail closed. + fn restarting(&self, mut f: impl FnMut() -> Result) -> Result { + for _ in 0..MAX_RESTARTS { + self.saw_absent.set(false); + let before = self.view.borrow().as_ref().map(|v| v.tag.clone()); + match f() { + Err(e) if self.saw_absent.get() => { + *self.view.borrow_mut() = None; + self.refresh()?; + let after = self.view.borrow().as_ref().map(|v| v.tag.clone()); + if before == after { + return Err(e); + } + } + r => return r, + } + } + f() + } + + fn refresh_once(&self) -> Result<()> { let cached_tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); match self.bucket.get("manifest", cached_tag.as_deref())? { Fetched::Unchanged => Ok(()), @@ -237,6 +308,7 @@ impl BucketTransport { Ok(()) } Fetched::New { bytes, tag } => { + *self.attempt_tag.borrow_mut() = Some(tag.clone()); let bytes = capped("manifest", bytes)?; let manifest = Manifest::decode(&bytes)?; let stop = manifest.checkpoint_seq; @@ -247,7 +319,7 @@ impl BucketTransport { let ck_bytes = match self.bucket.get(&checkpoint_key(stop), None)? { Fetched::New { bytes, .. } => capped("checkpoint", bytes)?, _ => { - return Err(Error::Wal(format!( + return Err(self.gone(format!( "checkpoint {stop} referenced by manifest but absent" ))) } @@ -277,9 +349,9 @@ impl BucketTransport { ))); } let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { - return Err(Error::Wal(format!( - "log entry {seq} referenced by chain but absent" - ))); + return Err( + self.gone(format!("log entry {seq} referenced by chain but absent")) + ); }; let e = LogEntry::decode(&capped("log entry", bytes)?)?; if e.seq != seq { @@ -312,7 +384,7 @@ impl BucketTransport { let mut index = BTreeMap::new(); for hash in &packs { let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but idx absent"))); + return Err(self.gone(format!("pack {hash} on chain but idx absent"))); }; let bytes = capped("pack idx", bytes)?; for IndexEntry { id, offset, length } in parse_index(&bytes)? { @@ -334,6 +406,10 @@ impl BucketTransport { /// Canonical `encode()` bytes of one object, fetched via its pack /// (downloaded once per distinct pack hash and cached for walk locality). fn object_bytes(&self, id: &ObjectId) -> Result> { + self.restarting(|| self.object_bytes_once(id)) + } + + fn object_bytes_once(&self, id: &ObjectId) -> Result> { let (hash, offset, _len) = { let view = self.view.borrow(); let view = view @@ -347,7 +423,7 @@ impl BucketTransport { let mut cache = self.pack_cache.borrow_mut(); if cache.as_ref().map(|(h, _)| h.as_str()) != Some(hash.as_str()) { let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(&hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + return Err(self.gone(format!("pack {hash} on chain but body absent"))); }; *cache = Some((hash.clone(), bytes)); } @@ -366,7 +442,7 @@ impl BucketTransport { return Ok(hit.clone()); } let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + return Err(self.gone(format!("pack {hash} on chain but body absent"))); }; let mut leaves = Vec::new(); scl_core::pack::parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { @@ -400,6 +476,13 @@ impl BucketTransport { /// An unsigned tombstone is withheld, as on `LocalTransport`. Whether a /// tombstone is honored is the receiver's call (`index_incoming`). fn leaf_ids_for(&self, snaps: &std::collections::BTreeSet) -> Result> { + self.restarting(|| self.leaf_ids_for_once(snaps)) + } + + fn leaf_ids_for_once( + &self, + snaps: &std::collections::BTreeSet, + ) -> Result> { let packs = match self.view.borrow().as_ref() { Some(v) => v.packs.clone(), None => return Ok(Vec::new()), @@ -474,6 +557,26 @@ impl BucketTransport { } } + /// Fail if a pack this push is about to commit has vanished from the bucket + /// (a compaction swept it as an unreferenced orphan after an unusually long + /// gap, ADR-0055). Committing a log entry that names a deleted pack would + /// corrupt the remote; failing lets the caller retry the push, which + /// re-uploads. One prefix listing per pending pack, matching both objects. + /// Called right before the manifest CAS. + fn ensure_packs_present(&self, hashes: &[String]) -> Result<()> { + for hash in hashes { + let have = self.bucket.list_meta(&format!("packs/{hash}."))?; + let has = |key: String| have.iter().any(|m| m.key == key); + if !(has(pack_key(hash)) && has(idx_key(hash))) { + return Err(Error::Remote(format!( + "pack {hash} was removed from the bucket before this push committed \ + (bucket compaction swept it); retry the push" + ))); + } + } + Ok(()) + } + /// Opportunistic, coordinator-free checkpoint fold. Called after a /// successful commit; every failure path is deliberately non-fatal — /// the checkpoint is derived data any reader can rebuild from the log, @@ -524,9 +627,8 @@ impl BucketTransport { // best-effort by design (see the doc above) — and let every // reader keep doing the slower but correct full-tail walk back // to the last checkpoint that DID fit. Durable relief is - // compaction (ROADMAP "Bucket compaction/gc"), which can retire - // refs/packs instead of letting them accumulate into ever-larger - // checkpoints forever. + // `compact` (ADR-0055), which merges small packs so the pack + // axis fits again. The ref axis is live state and is not relieved. return Ok(()); } let ck = Checkpoint { @@ -697,7 +799,7 @@ impl Transport for BucketTransport { // this function — success, failure, or the loop below — simply lets // `pending` go out of scope; the packs themselves stay uploaded in // the bucket regardless (content-addressed, inert orphan garbage if - // never referenced — correct per spec, compaction is a later phase). + // never referenced — correct per spec; `compact` sweeps it later, ADR-0055). let pending: Vec = std::mem::take(&mut *self.pending_packs.borrow_mut()); const MAX_CAS_RETRIES: u32 = 16; for _ in 0..MAX_CAS_RETRIES { @@ -756,6 +858,9 @@ impl Transport for BucketTransport { checkpoint_seq, head_branch, }; + // Last look before the commit point, to keep the window in which a + // concurrent sweep could still remove a pack as small as possible. + self.ensure_packs_present(&pending)?; if self .bucket .put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs new file mode 100644 index 00000000..25cf4b79 --- /dev/null +++ b/crates/repo/src/bucket_transport/compact.rs @@ -0,0 +1,446 @@ +//! Bucket remote compaction and gc (#216, ADR-0055). +//! +//! Two phases, each safe on its own. `consolidate` merges small live packs and +//! commits the result with one manifest CAS (a no-op log entry plus a +//! checkpoint whose pack list is the trimmed set). `sweep` then deletes what no +//! manifest chain can reach: log entries and checkpoints behind a cutoff old +//! enough that no reader still needs them, and packs nothing names. Nothing is +//! ever deleted that the current manifest references, and the manifest itself +//! is never deleted, so a failure between any two operations leaves a bucket a +//! cold start reads. + +use super::{capped, BucketTransport}; +use crate::error::Result; +use crate::walfmt::{checkpoint_key, log_key, pack_key, Checkpoint, LogEntry, Manifest, MAX_LIST}; +use scl_core::pack::parse_pack_reader; +use scl_core::ObjectId; +use scl_objio::{Fetched, ObjectMeta}; +use std::collections::{BTreeMap, BTreeSet}; +use std::time::{Duration, SystemTime}; + +/// Replans after a lost manifest CAS or a concurrent compactor before giving up. +const MAX_ATTEMPTS: u32 = 4; + +/// Knobs for [`BucketTransport::compact`]. The defaults are what +/// `sc remote compact` uses. +#[derive(Debug, Clone)] +pub struct CompactOptions { + /// Nothing unreferenced is deleted until it is at least this old (and the + /// checkpoint that superseded it is too). Mirrors `sc gc --prune-expire`. + pub prune_expire: Duration, + /// Plan and report; write and delete nothing. + pub dry_run: bool, + /// Live packs smaller than this are merge candidates. + pub small_pack_bytes: u64, + /// Upper bound on the summed size of one merge group (bounds peak RAM). + pub group_bytes: u64, + /// Clock override for tests; `None` is the wall clock at the start of the run. + pub now: Option, +} + +impl Default for CompactOptions { + fn default() -> Self { + CompactOptions { + prune_expire: Duration::from_secs(24 * 3600), + dry_run: false, + small_pack_bytes: 16 << 20, + group_bytes: 64 << 20, + now: None, + } + } +} + +/// What a compaction did, or with `dry_run`, what it would do. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct CompactReport { + pub dry_run: bool, + /// Live packs folded into new ones, and how many new packs replaced them. + pub packs_merged: usize, + pub packs_created: usize, + /// The manifest's `checkpoint_seq` after phase 1. + pub checkpoint_seq: u64, + /// The sweep cutoff `K` (0 when no checkpoint is old enough yet). + pub cutoff: u64, + pub logs_deleted: usize, + pub checkpoints_deleted: usize, + /// Pack hashes removed (each is a `.pack` and a `.idx`). + pub packs_deleted: usize, + pub bytes_deleted: u64, + /// Unreferenced packs kept because they are younger than `prune_expire`. + pub packs_held_back: usize, + /// Why something was skipped or stopped early. + pub notes: Vec, +} + +/// `checkpoints/00…065` or `log/00…065` -> 65. +fn seq_of(key: &str) -> Option { + key.rsplit('/').next()?.parse().ok() +} + +/// `packs/.pack` / `packs/.idx` -> ``. +fn hash_of(key: &str) -> Option<&str> { + let name = key.strip_prefix("packs/")?; + name.strip_suffix(".pack") + .or_else(|| name.strip_suffix(".idx")) +} + +/// Greedy size-tiered grouping: smallest candidates first, a group closes when +/// the next pack would push it past `group_bytes`. Groups of one are dropped +/// (there is nothing to merge). +fn plan_groups( + live: &[String], + sizes: &BTreeMap, + opts: &CompactOptions, +) -> Vec> { + let mut small: Vec<(u64, &String)> = live + .iter() + .filter_map(|h| sizes.get(h).map(|s| (*s, h))) + .filter(|(s, _)| *s < opts.small_pack_bytes) + .collect(); + small.sort(); + let mut groups: Vec> = Vec::new(); + let (mut cur, mut sum): (Vec, u64) = (Vec::new(), 0); + for (size, hash) in small { + if !cur.is_empty() && sum + size > opts.group_bytes { + groups.push(std::mem::take(&mut cur)); + sum = 0; + } + cur.push(hash.clone()); + sum += size; + } + groups.push(cur); + groups.retain(|g| g.len() >= 2); + groups +} + +impl BucketTransport { + /// Compact this bucket remote (ADR-0055): merge small packs, then delete + /// log entries, checkpoints and packs no reader or writer can still need. + /// Safe to run while others push and fetch; run one at a time (concurrent + /// runs stay correct but waste work). Needs bucket list and delete rights. + pub fn compact(&self, opts: &CompactOptions) -> Result { + let mut rep = CompactReport { + dry_run: opts.dry_run, + ..CompactReport::default() + }; + self.refresh()?; + if self.view.borrow().is_none() { + rep.notes + .push("bucket has no manifest: nothing to compact".into()); + return Ok(rep); + } + self.consolidate(opts, &mut rep)?; + // Read the clock after phase 1 so a zero grace sees its own checkpoint. + let now = opts.now.unwrap_or_else(SystemTime::now); + self.sweep(opts, now, &mut rep)?; + Ok(rep) + } + + /// Phase 1: merge small live packs and commit with one manifest CAS. + fn consolidate(&self, opts: &CompactOptions, rep: &mut CompactReport) -> Result<()> { + 'attempt: for _ in 0..MAX_ATTEMPTS { + self.refresh()?; + let (live, nrefs) = { + let view = self.view.borrow(); + let v = view.as_ref().expect("checked non-empty"); + let mut seen = BTreeSet::new(); + let live: Vec = v + .packs + .iter() + .filter(|h| seen.insert((*h).clone())) + .cloned() + .collect(); + (live, v.refs.len()) + }; + let sizes: BTreeMap = self + .bucket + .list_meta("packs/")? + .into_iter() + .filter(|m| m.key.ends_with(".pack")) + .filter_map(|m| hash_of(&m.key).map(|h| (h.to_string(), m.size))) + .collect(); + let groups = plan_groups(&live, &sizes, opts); + if groups.is_empty() { + if live.len() > MAX_LIST { + rep.notes.push(format!( + "{} live packs exceed the checkpoint cap of {MAX_LIST} and none is small \ + enough to merge; folding stays off", + live.len() + )); + } + return Ok(()); + } + let replaced: BTreeSet = groups.iter().flatten().cloned().collect(); + if live.len() - replaced.len() + groups.len() > MAX_LIST || nrefs > MAX_LIST { + rep.notes.push(format!( + "merging would leave more than {MAX_LIST} packs (or {nrefs} refs exceed the \ + cap): a checkpoint could not hold it, nothing merged" + )); + return Ok(()); + } + rep.packs_merged = replaced.len(); + rep.packs_created = groups.len(); + if opts.dry_run { + return Ok(()); + } + let mut created = Vec::new(); + for group in &groups { + let mut objects: BTreeMap> = BTreeMap::new(); + for hash in group { + let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { + // Another compactor swept it: the plan is stale. + continue 'attempt; + }; + parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { + objects.insert(id, obj.encode()); + Ok(()) + })?; + } + let objects: Vec<(ObjectId, Vec)> = objects.into_iter().collect(); + created.push(self.upload_pack(&objects)?.0); + } + // Fence + fold in one CAS: the manifest tag we read is the state + // the checkpoint describes, so a lost CAS just replans. + self.refresh()?; + let (tag, head_seq, head_branch, refs, packs) = { + let view = self.view.borrow(); + let v = view.as_ref().expect("checked non-empty"); + ( + v.tag.clone(), + v.manifest.head_seq, + v.manifest.head_branch.clone(), + v.refs + .iter() + .map(|(b, id)| (b.clone(), *id)) + .collect::>(), + v.packs.clone(), + ) + }; + let mut seen = BTreeSet::new(); + let mut kept: Vec = packs + .into_iter() + .filter(|h| !replaced.contains(h) && seen.insert(h.clone())) + .collect(); + for h in &created { + if seen.insert(h.clone()) { + kept.push(h.clone()); + } + } + if kept.len() > MAX_LIST { + continue 'attempt; // new pushes pushed the list over the cap + } + let mut try_seq = head_seq + 1; + let entry = LogEntry { + seq: 0, + parent_seq: head_seq, + packs: created.clone(), + updates: vec![], + }; + let seq = loop { + let mut candidate = entry.clone(); + candidate.seq = try_seq; + if self + .bucket + .put_new(&log_key(try_seq), &candidate.encode())? + { + break try_seq; + } + try_seq += 1; + }; + let ck = Checkpoint { + seq, + refs, + packs: kept, + }; + if !self.bucket.put_new(&checkpoint_key(seq), &ck.encode())? { + continue 'attempt; // slot debris from a crashed run: claim a later one + } + let manifest = Manifest { + head_seq: seq, + checkpoint_seq: seq, + head_branch, + }; + if self + .bucket + .put_if_tag("manifest", &manifest.encode(), Some(&tag))? + .is_some() + { + self.refresh()?; + rep.checkpoint_seq = seq; + return Ok(()); + } + } + rep.notes.push(format!( + "manifest kept moving or a concurrent compaction raced ({MAX_ATTEMPTS} attempts): \ + nothing merged, rerun" + )); + Ok(()) + } + + /// Phase 2: delete what is behind the cutoff and what nothing names. + fn sweep(&self, opts: &CompactOptions, now: SystemTime, rep: &mut CompactReport) -> Result<()> { + // `modified <= old_before` means "at least prune_expire old". + let old_before = now + .checked_sub(opts.prune_expire) + .unwrap_or(SystemTime::UNIX_EPOCH); + let old = |m: &ObjectMeta| m.modified <= old_before; + for _ in 0..MAX_ATTEMPTS { + self.refresh()?; + let (tag, ck_seq, live) = { + let view = self.view.borrow(); + let v = view.as_ref().expect("checked non-empty"); + ( + v.tag.clone(), + v.manifest.checkpoint_seq, + v.packs.iter().cloned().collect::>(), + ) + }; + let ckpts: Vec<(u64, ObjectMeta)> = self + .bucket + .list_meta("checkpoints/")? + .into_iter() + .filter_map(|m| seq_of(&m.key).map(|s| (s, m))) + .collect(); + let logs: Vec<(u64, ObjectMeta)> = self + .bucket + .list_meta("log/")? + .into_iter() + .filter_map(|m| seq_of(&m.key).map(|s| (s, m))) + .collect(); + let cutoff = ckpts + .iter() + .filter(|(s, m)| *s <= ck_seq && old(m)) + .map(|(s, _)| *s) + .max() + .unwrap_or(0); + if cutoff == 0 && ck_seq > 0 { + rep.notes.push( + "no checkpoint is older than the grace window yet: log entries and older \ + checkpoints are kept" + .into(), + ); + } + let del_logs: Vec<&ObjectMeta> = logs + .iter() + .filter(|(s, _)| cutoff > 0 && *s <= cutoff) + .map(|(_, m)| m) + .collect(); + let del_ckpts: Vec<&ObjectMeta> = ckpts + .iter() + .filter(|(s, _)| cutoff > 0 && *s < cutoff) + .map(|(_, m)| m) + .collect(); + + // Pack objects by hash; a hash is a candidate if the live chain + // does not name it and every one of its objects is old enough. + let mut by_hash: BTreeMap> = BTreeMap::new(); + for m in self.bucket.list_meta("packs/")? { + if let Some(h) = hash_of(&m.key) { + by_hash.entry(h.to_string()).or_default().push(m); + } + } + let unnamed: Vec<(&String, &Vec)> = + by_hash.iter().filter(|(h, _)| !live.contains(*h)).collect(); + let (aged, young): (Vec<_>, Vec<_>) = + unnamed.into_iter().partition(|(_, ms)| ms.iter().all(old)); + rep.packs_held_back = young.len(); + + let mut del_packs: Vec<(&String, &Vec)> = Vec::new(); + if !aged.is_empty() { + // Retained structure may still name them: every kept + // checkpoint and every kept log entry (on-chain or not). + match self.retained_packs(&ckpts, &logs, cutoff) { + Ok(named) => { + del_packs = aged + .into_iter() + .filter(|(h, _)| !named.contains(*h)) + .collect() + } + Err(e) => rep + .notes + .push(format!("packs kept, a retained record is unreadable: {e}")), + } + } + + rep.cutoff = cutoff; + rep.logs_deleted = del_logs.len(); + rep.checkpoints_deleted = del_ckpts.len(); + rep.packs_deleted = del_packs.len(); + rep.bytes_deleted = del_logs + .iter() + .chain(del_ckpts.iter()) + .map(|m| m.size) + .chain( + del_packs + .iter() + .flat_map(|(_, ms)| ms.iter().map(|m| m.size)), + ) + .sum(); + if opts.dry_run { + if rep.packs_created > 0 { + rep.notes.push( + "dry run: packs the planned merge replaces become deletable once it \ + lands and the grace window passes" + .into(), + ); + } + return Ok(()); + } + // The manifest moving means a push or fold landed during planning: + // replan rather than delete against a stale picture. Log entries + // and checkpoints stay safe regardless (the cutoff only grows), + // but the pack candidates may just have been committed. + if !matches!(self.bucket.get("manifest", Some(&tag))?, Fetched::Unchanged) { + continue; + } + for m in &del_logs { + self.bucket.delete(&m.key)?; + } + for m in &del_ckpts { + self.bucket.delete(&m.key)?; + } + for (hash, _) in &del_packs { + // idx first: a half-deleted pack is then plain debris. + self.bucket.delete(&format!("packs/{hash}.idx"))?; + self.bucket.delete(&pack_key(hash))?; + } + return Ok(()); + } + rep.notes + .push("manifest kept moving during the sweep: rerun".into()); + Ok(()) + } + + /// Pack hashes named by structure the sweep keeps: checkpoints with + /// `seq >= cutoff` and log entries with `seq > cutoff` (everything when + /// `cutoff` is 0). Any record that fails to decode aborts: deleting packs + /// on a partial picture is never right. + fn retained_packs( + &self, + ckpts: &[(u64, ObjectMeta)], + logs: &[(u64, ObjectMeta)], + cutoff: u64, + ) -> Result> { + let mut named = BTreeSet::new(); + for (seq, m) in ckpts { + if cutoff == 0 || *seq >= cutoff { + let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { + continue; // deleted since the listing: no longer retained + }; + named.extend(Checkpoint::decode(&capped("checkpoint", bytes)?)?.packs); + } + } + for (seq, m) in logs { + if cutoff == 0 || *seq > cutoff { + let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { + continue; + }; + named.extend(LogEntry::decode(&capped("log entry", bytes)?)?.packs); + } + } + Ok(named) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/repo/src/bucket_transport/compact/tests.rs b/crates/repo/src/bucket_transport/compact/tests.rs new file mode 100644 index 00000000..c323b39f --- /dev/null +++ b/crates/repo/src/bucket_transport/compact/tests.rs @@ -0,0 +1,798 @@ +//! Compaction tests (#216, ADR-0055). Most run on an in-memory bucket (fast, +//! and lets a test fork a bucket or fake object ages); the full-run and the +//! concurrency tests use `DirBucket`, the real backend behind `sc+wal://`. + +use super::super::CHECKPOINT_INTERVAL; +use super::*; +use crate::error::Error; +use crate::transport::Transport; +use crate::walfmt::{idx_key, RefUpdate}; +use scl_core::Object; +use scl_objio::{Bucket, DirBucket}; +use std::path::PathBuf; +use std::sync::atomic::{AtomicBool, AtomicI64, Ordering}; +use std::sync::{Arc, Mutex}; + +/// In-memory `Bucket` with the same semantics as `DirBucket`. +#[derive(Default)] +struct MemBucket { + map: Mutex, SystemTime)>>, +} + +fn tag_of(bytes: &[u8]) -> String { + hex::encode(blake3::hash(bytes).as_bytes()) +} + +impl MemBucket { + fn fork(&self) -> Arc { + Arc::new(MemBucket { + map: Mutex::new(self.map.lock().unwrap().clone()), + }) + } + /// Pretend `key` was written at `at`. + fn touch(&self, key: &str, at: SystemTime) { + self.map.lock().unwrap().get_mut(key).unwrap().1 = at; + } +} + +struct Shared(Arc); +impl Bucket for Shared { + fn get(&self, key: &str, cached: Option<&str>) -> scl_objio::Result { + Ok(match self.0.map.lock().unwrap().get(key) { + None => Fetched::Absent, + Some((b, _)) if cached == Some(tag_of(b).as_str()) => Fetched::Unchanged, + Some((b, _)) => Fetched::New { + bytes: b.clone(), + tag: tag_of(b), + }, + }) + } + fn put_new(&self, key: &str, bytes: &[u8]) -> scl_objio::Result { + let mut m = self.0.map.lock().unwrap(); + if m.contains_key(key) { + return Ok(false); + } + m.insert(key.into(), (bytes.to_vec(), SystemTime::now())); + Ok(true) + } + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected: Option<&str>, + ) -> scl_objio::Result> { + let mut m = self.0.map.lock().unwrap(); + if m.get(key).map(|(b, _)| tag_of(b)).as_deref() != expected { + return Ok(None); + } + m.insert(key.into(), (bytes.to_vec(), SystemTime::now())); + Ok(Some(tag_of(bytes))) + } + fn list(&self, prefix: &str) -> scl_objio::Result> { + Ok(self.list_meta(prefix)?.into_iter().map(|m| m.key).collect()) + } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + Ok(self + .0 + .map + .lock() + .unwrap() + .iter() + .filter(|(k, _)| k.starts_with(prefix)) + .map(|(k, (b, t))| ObjectMeta { + key: k.clone(), + size: b.len() as u64, + modified: *t, + }) + .collect()) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.0.map.lock().unwrap().remove(key); + Ok(()) + } +} + +/// Where a test's bucket lives, so helpers work on either backend. +trait Store { + fn bucket(&self) -> Box; +} +impl Store for PathBuf { + fn bucket(&self) -> Box { + Box::new(DirBucket::open(self).unwrap()) + } +} +impl Store for Arc { + fn bucket(&self) -> Box { + Box::new(Shared(self.clone())) + } +} + +/// Backdate every file under `dir` by an hour, as if the data had sat there. +fn age_files(dir: &std::path::Path) { + let then = SystemTime::now() - Duration::from_secs(3600); + for e in std::fs::read_dir(dir).unwrap().flatten() { + if e.path().is_dir() { + age_files(&e.path()); + } else { + std::fs::File::options() + .write(true) + .open(e.path()) + .unwrap() + .set_modified(then) + .unwrap(); + } + } +} + +fn scratch(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-bt-compact-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + d +} + +fn open(s: &impl Store) -> BucketTransport { + BucketTransport::from_bucket(s.bucket()).unwrap() +} + +fn keys(s: &impl Store, prefix: &str) -> Vec { + s.bucket().list(prefix).unwrap() +} + +fn manifest(s: &impl Store) -> Manifest { + let Fetched::New { bytes, .. } = s.bucket().get("manifest", None).unwrap() else { + panic!("no manifest") + }; + Manifest::decode(&bytes).unwrap() +} + +/// Options that merge everything and treat every object as old enough. +fn eager() -> CompactOptions { + CompactOptions { + prune_expire: Duration::ZERO, + ..CompactOptions::default() + } +} + +/// A one-commit history: an empty tree shared by every commit plus a +/// snapshot distinguished by `label`. Returns (tip, objects). +fn commit(label: &str) -> (ObjectId, Vec<(ObjectId, Vec)>) { + let tree = Object::Tree(scl_core::Tree::new(vec![])); + let snap = Object::Snapshot(scl_core::Snapshot { + root: tree.id(), + parents: vec![], + author: "t".into(), + timestamp: 1, + message: label.into(), + secrets: Default::default(), + protection: Default::default(), + }); + ( + snap.id(), + vec![(tree.id(), tree.encode()), (snap.id(), snap.encode())], + ) +} + +/// Push `n` single-commit branches `-` through the real write path +/// (folds a checkpoint once the tail passes the interval). +fn push_commits(t: &BucketTransport, prefix: &str, n: u64) -> Vec<(String, ObjectId)> { + (0..n) + .map(|i| { + let (tip, objects) = commit(&format!("{prefix}-{i}")); + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + let branch = format!("{prefix}-{i}"); + t.update_ref(&branch, &tip, None).unwrap(); + (branch, tip) + }) + .collect() +} + +/// Hand-build a WAL of `n` one-commit pushes with a checkpoint folded at `ck` +/// (0 = none): the shape real pushes leave, without their cost. +fn hand_built(s: &impl Store, n: u64, ck: u64) -> Vec<(String, ObjectId)> { + let b = s.bucket(); + let (mut refs, mut packs) = (Vec::new(), Vec::new()); + for i in 1..=n { + let (tip, objects) = commit(&format!("hb-{i}")); + let (pack, idx) = scl_core::pack::build_pack(&objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + b.put_new(&pack_key(&hash), &pack).unwrap(); + b.put_new(&idx_key(&hash), &idx).unwrap(); + let entry = LogEntry { + seq: i, + parent_seq: i - 1, + packs: vec![hash.clone()], + updates: vec![RefUpdate { + branch: format!("b-{i}"), + old: None, + new: tip, + }], + }; + b.put_new(&log_key(i), &entry.encode()).unwrap(); + refs.push((format!("b-{i}"), tip)); + packs.push(hash); + if i == ck { + let c = Checkpoint { + seq: ck, + refs: refs.clone(), + packs: packs.clone(), + }; + b.put_new(&checkpoint_key(ck), &c.encode()).unwrap(); + } + } + let m = Manifest { + head_seq: n, + checkpoint_seq: ck, + head_branch: "b-1".into(), + }; + b.put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + refs.sort(); + refs +} + +/// A cold-start reader sees exactly `expect` and can read every object both +/// singly and as one pack. +fn assert_readable(s: &impl Store, expect: &[(String, ObjectId)]) { + let mut expect = expect.to_vec(); + expect.sort(); + let t = open(s); + assert_eq!(t.list_refs().unwrap(), expect); + for (_, id) in &expect { + assert_eq!(ObjectId::of(&t.get_object(id).unwrap()), *id); + } + let tips: Vec = expect.iter().map(|(_, id)| *id).collect(); + let mut out = Vec::new(); + t.get_pack(&tips, &[], None, &mut out).unwrap(); + // every tip's snapshot plus the one tree they share + assert_eq!( + scl_core::pack::parse_pack(&out).unwrap().len(), + tips.len() + 1 + ); +} + +#[test] +fn the_in_memory_double_honors_the_bucket_contract() { + scl_objio::contract_suite(&Shared(Arc::new(MemBucket::default()))); +} + +#[test] +fn merges_small_packs_and_sweeps_everything_behind_the_checkpoint() { + let root = scratch("full"); + let t = open(&root); + let refs = push_commits(&t, "w", 70); + // 70 pushes folded a checkpoint at 65; 70 log entries, 70 packs. + assert_eq!(manifest(&root).checkpoint_seq, 65); + assert_eq!(keys(&root, "packs/").len(), 140); + + let rep = t.compact(&eager()).unwrap(); + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1), "{rep:?}"); + assert_eq!(rep.checkpoint_seq, 71); + assert_eq!(rep.cutoff, 71); + assert_eq!((rep.logs_deleted, rep.checkpoints_deleted), (71, 1)); + assert_eq!(rep.packs_deleted, 70); + + let m = manifest(&root); + assert_eq!((m.head_seq, m.checkpoint_seq), (71, 71)); + assert!(keys(&root, "log/").is_empty()); + assert_eq!(keys(&root, "checkpoints/").len(), 1); + assert_eq!(keys(&root, "packs/").len(), 2, "one merged pack + idx"); + assert_readable(&root, &refs); + + // Idempotent: nothing left to do. + let again = t.compact(&eager()).unwrap(); + assert_eq!( + ( + again.packs_merged, + again.logs_deleted, + again.checkpoints_deleted, + again.packs_deleted + ), + (0, 0, 0, 0), + "{again:?}" + ); + // And the remote still takes pushes. + let mut all = refs; + all.extend(push_commits(&t, "after", 3)); + assert_readable(&root, &all); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn dry_run_reports_the_plan_and_writes_nothing() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + push_commits(&t, "w", 70); + let before = mem.map.lock().unwrap().clone(); + let rep = t + .compact(&CompactOptions { + dry_run: true, + ..eager() + }) + .unwrap(); + assert!(rep.dry_run); + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1)); + assert!(rep.logs_deleted > 0 && rep.bytes_deleted > 0, "{rep:?}"); + assert!(!rep.notes.is_empty()); + assert!( + *mem.map.lock().unwrap() == before, + "dry run must not change the bucket" + ); +} + +#[test] +fn nothing_is_deleted_inside_the_grace_window_and_everything_after_it() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let refs = push_commits(&t, "w", 70); + let rep = t.compact(&CompactOptions::default()).unwrap(); + // Phase 1 is additive and runs; the sweep keeps everything young. + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1)); + assert_eq!( + (rep.logs_deleted, rep.checkpoints_deleted, rep.packs_deleted), + (0, 0, 0) + ); + assert_eq!(rep.packs_held_back, 70, "{rep:?}"); + assert_eq!(keys(&mem, "log/").len(), 71); + assert_readable(&mem, &refs); + + // Two days later every object is past the window. + let later = SystemTime::now() + Duration::from_secs(2 * 24 * 3600); + let rep = t + .compact(&CompactOptions { + now: Some(later), + ..CompactOptions::default() + }) + .unwrap(); + assert_eq!( + (rep.logs_deleted, rep.checkpoints_deleted, rep.packs_deleted), + (71, 1, 70) + ); + assert_readable(&mem, &refs); +} + +#[test] +fn the_cutoff_is_the_newest_checkpoint_that_has_aged_not_the_newest_checkpoint() { + // A busy remote folds every 64 pushes, so its newest checkpoint is rarely + // old; an older one still sets the cutoff. + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let b = mem.bucket(); + let Fetched::New { bytes, tag } = b.get("manifest", None).unwrap() else { + panic!() + }; + let mut m = Manifest::decode(&bytes).unwrap(); + let mut packs = Vec::new(); + for seq in 1..=12 { + let Fetched::New { bytes, .. } = b.get(&log_key(seq), None).unwrap() else { + panic!() + }; + packs.extend(LogEntry::decode(&bytes).unwrap().packs); + } + let ck12 = Checkpoint { + seq: 12, + refs: refs.clone(), + packs, + }; + b.put_new(&checkpoint_key(12), &ck12.encode()).unwrap(); + m.checkpoint_seq = 12; + b.put_if_tag("manifest", &m.encode(), Some(&tag)) + .unwrap() + .unwrap(); + // Checkpoint 10 is two hours old, checkpoint 12 just landed; window 1h. + let now = SystemTime::now(); + mem.touch(&checkpoint_key(10), now - Duration::from_secs(7200)); + + let rep = open(&mem) + .compact(&CompactOptions { + small_pack_bytes: 0, // no merging: isolate the sweep + prune_expire: Duration::from_secs(3600), + now: Some(now), + ..CompactOptions::default() + }) + .unwrap(); + assert_eq!(rep.packs_merged, 0); + assert_eq!(rep.cutoff, 10, "{rep:?}"); + assert_eq!((rep.logs_deleted, rep.checkpoints_deleted), (10, 0)); + assert_eq!(keys(&mem, "log/").len(), 2, "entries past the cutoff stay"); + assert_eq!(keys(&mem, "checkpoints/").len(), 2); + assert_readable(&mem, &refs); +} + +/// A bucket that fails every operation once a shared budget runs out, like a +/// process dying mid-compaction. +struct FailAfter { + inner: Shared, + left: Arc, +} +impl FailAfter { + fn tick(&self) -> scl_objio::Result<()> { + if self.left.fetch_sub(1, Ordering::SeqCst) <= 0 { + Err(scl_objio::Error::Backend("injected failure".into())) + } else { + Ok(()) + } + } +} +impl Bucket for FailAfter { + fn get(&self, key: &str, tag: Option<&str>) -> scl_objio::Result { + self.tick()?; + self.inner.get(key, tag) + } + fn put_new(&self, key: &str, bytes: &[u8]) -> scl_objio::Result { + self.tick()?; + self.inner.put_new(key, bytes) + } + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + tag: Option<&str>, + ) -> scl_objio::Result> { + self.tick()?; + self.inner.put_if_tag(key, bytes, tag) + } + fn list(&self, prefix: &str) -> scl_objio::Result> { + self.tick()?; + self.inner.list(prefix) + } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + self.tick()?; + self.inner.list_meta(prefix) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.tick()?; + self.inner.delete(key) + } +} + +#[test] +fn a_failure_at_every_operation_boundary_leaves_a_readable_bucket() { + let template = Arc::new(MemBucket::default()); + let refs = hand_built(&template, 12, 10); + let mut finished_at = None; + for budget in 0..5000i64 { + let work = template.fork(); + let left = Arc::new(AtomicI64::new(budget)); + let outcome = BucketTransport::from_bucket(Box::new(FailAfter { + inner: Shared(work.clone()), + left: left.clone(), + })) + .and_then(|t| t.compact(&eager())); + + // Whatever happened, a fresh reader still sees everything. + assert_readable(&work, &refs); + if outcome.is_ok() && left.load(Ordering::SeqCst) > 0 { + finished_at = Some(budget); + break; + } + // And a healthy rerun finishes the job. + open(&work).compact(&eager()).unwrap(); + assert_readable(&work, &refs); + assert_eq!(keys(&work, "packs/").len(), 2, "budget {budget}"); + } + let n = finished_at.expect("compaction never completed within the budget"); + assert!(n > 20, "the loop should cross many boundaries, got {n}"); +} + +#[test] +fn readers_and_writers_stay_consistent_while_compaction_runs() { + let root = scratch("concurrent"); + let seeded = hand_built(&root, 12, 10); + age_files(&root); // the seed is old enough to sweep from the first run + const WRITERS: usize = 3; + const PER_WRITER: u64 = 20; + const GRACE: Duration = Duration::from_millis(1500); + let done = AtomicBool::new(false); + let opts = CompactOptions { + // Far longer than a writer's upload-to-commit gap even on a loaded + // machine, short enough that sweeps really delete during the run. + prune_expire: GRACE, + ..CompactOptions::default() + }; + let landed = Mutex::new(Vec::new()); + let deleted = std::sync::atomic::AtomicUsize::new(0); + std::thread::scope(|s| { + let writers: Vec<_> = (0..WRITERS) + .map(|w| { + let (root, landed) = (&root, &landed); + s.spawn(move || { + let t = open(root); + for i in 0..PER_WRITER { + let (tip, objects) = commit(&format!("writer-{w}-{i}")); + let branch = format!("writer-{w}-{i}"); + loop { + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + match t.update_ref(&branch, &tip, None) { + Ok(()) => break, + // The documented guard: a pack swept before + // commit fails the push; the retry re-uploads. + Err(Error::Remote(m)) if m.contains("removed from the bucket") => {} + Err(e) => panic!("writer {w}: {e}"), + } + } + landed.lock().unwrap().push((branch, tip)); + std::thread::sleep(Duration::from_millis(80)); + } + }) + }) + .collect(); + let readers: Vec<_> = (0..3) + .map(|r| { + let (root, done, seeded) = (&root, &done, &seeded); + s.spawn(move || { + let long_lived = open(root); + let mut seen = seeded.len(); + let mut rounds = 0u32; + while !done.load(Ordering::SeqCst) || rounds < 3 { + rounds += 1; + // alternate a long-lived (stale-view) reader and a cold start + let cold; + let t = if rounds.is_multiple_of(3) { + cold = open(root); + &cold + } else { + &long_lived + }; + let refs = t.list_refs().unwrap_or_else(|e| panic!("reader {r}: {e}")); + assert!(refs.len() >= seen, "reader {r}: refs went backwards"); + seen = refs.len(); + for (_, id) in &refs { + let bytes = t + .get_object(id) + .unwrap_or_else(|e| panic!("reader {r}: get_object: {e}")); + assert_eq!(ObjectId::of(&bytes), *id); + } + let tips: Vec = refs.iter().map(|(_, id)| *id).collect(); + t.get_pack(&tips, &[], None, &mut Vec::new()) + .unwrap_or_else(|e| panic!("reader {r}: get_pack: {e}")); + } + }) + }) + .collect(); + let compactor = s.spawn(|| { + let t = open(&root); + let mut runs = 0u32; + while !done.load(Ordering::SeqCst) { + let rep = t + .compact(&opts) + .unwrap_or_else(|e| panic!("compactor: {e}")); + deleted.fetch_add( + rep.logs_deleted + rep.checkpoints_deleted + rep.packs_deleted, + Ordering::SeqCst, + ); + runs += 1; + std::thread::sleep(Duration::from_millis(40)); + } + runs + }); + for w in writers { + w.join().unwrap(); + } + // Keep readers and the compactor running until everything the writers + // produced has also aged past the window and been swept. + std::thread::sleep(GRACE * 2); + done.store(true, Ordering::SeqCst); + for r in readers { + r.join().unwrap(); + } + assert!(compactor.join().unwrap() > 2, "compactor barely ran"); + }); + assert!( + deleted.load(Ordering::SeqCst) > 0, + "the sweep never deleted anything during the run, so this proved nothing" + ); + let mut expect = seeded.clone(); + expect.extend(landed.into_inner().unwrap()); + // After the dust settles a zero-grace pass leaves a tight bucket. + open(&root).compact(&eager()).unwrap(); + assert_readable(&root, &expect); + assert_eq!(keys(&root, "packs/").len(), 2, "all packs merged into one"); + assert!(keys(&root, "log/").is_empty()); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn a_reader_whose_view_was_swept_restarts_and_a_corrupt_bucket_still_fails() { + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let reader = open(&mem); + let (_, tip) = refs[0]; + assert!(reader.has_object(&tip).unwrap()); + // Compaction lands behind the reader's back and sweeps the old packs. + open(&mem).compact(&eager()).unwrap(); + assert_eq!(keys(&mem, "packs/").len(), 2); + // The reader's cached index points into a pack that no longer exists; the + // fetch notices, re-reads the manifest and restarts on it. + let bytes = reader.object_bytes(&tip).unwrap(); + assert_eq!(ObjectId::of(&bytes), tip); + + // With the manifest unchanged, a missing pack is corruption and fails. + let pack = keys(&mem, "packs/") + .into_iter() + .find(|k| k.ends_with(".pack")) + .unwrap(); + mem.bucket().delete(&pack).unwrap(); + assert!(open(&mem).object_bytes(&tip).is_err()); +} + +#[test] +fn a_cold_start_straddling_a_sweep_restarts_instead_of_failing() { + // The reader read the manifest, then the checkpoint and log it names were + // swept. Model it with a bucket that runs a compaction right after the + // reader's first manifest read. + struct SweepAfterManifest { + inner: Shared, + fired: Mutex, + } + impl Bucket for SweepAfterManifest { + fn get(&self, key: &str, tag: Option<&str>) -> scl_objio::Result { + let got = self.inner.get(key, tag)?; + let mut fired = self.fired.lock().unwrap(); + if key == "manifest" && !*fired { + *fired = true; + drop(fired); + BucketTransport::from_bucket(Box::new(Shared(self.inner.0.clone()))) + .unwrap() + .compact(&eager()) + .unwrap(); + } + Ok(got) + } + fn put_new(&self, k: &str, b: &[u8]) -> scl_objio::Result { + self.inner.put_new(k, b) + } + fn put_if_tag( + &self, + k: &str, + b: &[u8], + t: Option<&str>, + ) -> scl_objio::Result> { + self.inner.put_if_tag(k, b, t) + } + fn list(&self, p: &str) -> scl_objio::Result> { + self.inner.list(p) + } + fn list_meta(&self, p: &str) -> scl_objio::Result> { + self.inner.list_meta(p) + } + fn delete(&self, k: &str) -> scl_objio::Result<()> { + self.inner.delete(k) + } + } + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let t = BucketTransport::from_bucket(Box::new(SweepAfterManifest { + inner: Shared(mem.clone()), + fired: Mutex::new(false), + })) + .expect("the cold start restarts on the new manifest"); + let mut expect = refs.clone(); + expect.sort(); + assert_eq!(t.list_refs().unwrap(), expect); + assert_eq!(manifest(&mem).checkpoint_seq, 13, "the sweep really ran"); +} + +#[test] +fn a_push_whose_pack_was_swept_before_commit_fails_instead_of_corrupting() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let (tip, objects) = commit("orphaned"); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + // The sweep removes the not-yet-committed pack. + for k in keys(&mem, "packs/") { + mem.bucket().delete(&k).unwrap(); + } + let err = t.update_ref("main", &tip, None).unwrap_err(); + assert!( + format!("{err}").contains("removed from the bucket"), + "{err}" + ); + assert!(matches!( + mem.bucket().get("manifest", None).unwrap(), + Fetched::Absent + )); +} + +#[test] +fn a_remote_past_max_list_regains_folding_after_compaction() { + let mem = Arc::new(MemBucket::default()); + // 66 commits in one real pack, plus MAX_LIST placeholder packs (valid, + // empty), so the cumulative pack list is MAX_LIST + 1: one more than a + // checkpoint can hold, which is what makes folding skip. + const ENTRIES: u64 = 66; + let commits: Vec<_> = (1..=ENTRIES).map(|i| commit(&format!("big-{i}"))).collect(); + let mut real: Vec<_> = commits.iter().flat_map(|(_, o)| o.clone()).collect(); + real.sort(); + real.dedup(); + let (rpack, ridx) = scl_core::pack::build_pack(&real).unwrap(); + let rhash = hex::encode(blake3::hash(&rpack).as_bytes()); + let (epack, eidx) = scl_core::pack::build_pack(&[]).unwrap(); + let fakes: Vec = (0..MAX_LIST).map(|i| format!("fake-{i:06}")).collect(); + let b = mem.bucket(); + for h in &fakes { + b.put_new(&pack_key(h), &epack).unwrap(); + b.put_new(&idx_key(h), &eidx).unwrap(); + } + b.put_new(&pack_key(&rhash), &rpack).unwrap(); + b.put_new(&idx_key(&rhash), &ridx).unwrap(); + let chunk = fakes.len().div_ceil(ENTRIES as usize); + let mut refs = Vec::new(); + for (i, (tip, _)) in commits.iter().enumerate() { + let seq = i as u64 + 1; + let mut packs: Vec = fakes.chunks(chunk).nth(i).unwrap_or(&[]).to_vec(); + if i == 0 { + packs.push(rhash.clone()); + } + let entry = LogEntry { + seq, + parent_seq: seq - 1, + packs, + updates: vec![RefUpdate { + branch: format!("b-{seq}"), + old: None, + new: *tip, + }], + }; + b.put_new(&log_key(seq), &entry.encode()).unwrap(); + refs.push((format!("b-{seq}"), *tip)); + } + let m = Manifest { + head_seq: ENTRIES, + checkpoint_seq: 0, + head_branch: "b-1".into(), + }; + b.put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + + // Before: a push lands but its fold is skipped (over the cap). + let t = open(&mem); + refs.extend(push_commits(&t, "extra", 1)); + assert_eq!( + manifest(&mem).checkpoint_seq, + 0, + "fold should be skipped over the cap" + ); + + let rep = t.compact(&eager()).unwrap(); + // the fakes, the real pack and the extra push's pack + assert_eq!( + (rep.packs_merged, rep.packs_created), + (MAX_LIST + 2, 1), + "{rep:?}" + ); + assert!(manifest(&mem).checkpoint_seq > 0); + assert_eq!(rep.packs_deleted, MAX_LIST + 2); + assert_readable(&mem, &refs); + + // After: ordinary pushes fold again. + let ck_after = manifest(&mem).checkpoint_seq; + refs.extend(push_commits(&t, "later", CHECKPOINT_INTERVAL + 2)); + assert!( + manifest(&mem).checkpoint_seq > ck_after, + "folding must resume once the pack list fits" + ); + assert_readable(&mem, &refs); +} + +#[test] +fn empty_bucket_and_single_pack_are_noops() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let rep = t.compact(&eager()).unwrap(); + assert_eq!(rep.packs_merged, 0); + assert!(!rep.notes.is_empty()); + // One pack is not a merge. + let refs = push_commits(&t, "solo", 1); + let rep = t.compact(&eager()).unwrap(); + assert_eq!((rep.packs_merged, rep.logs_deleted), (0, 0)); + assert_readable(&mem, &refs); +} diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 18c4202d..f8f27411 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -64,7 +64,9 @@ pub mod workspace; pub mod worktree; pub(crate) mod ws; -pub use bucket_transport::{BucketScheme, BucketTransport, BucketUrl}; +pub use bucket_transport::{ + BucketScheme, BucketTransport, BucketUrl, CompactOptions, CompactReport, +}; pub use conflicts::{ActiveOp, ConflictKind, ConflictVersions, ResolveSide, Side}; pub use error::{Error, Result}; pub use gc::GcStats; From e3f523437b7f38b0ddecb04a696bf34e122f2fc8 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:23:51 +0300 Subject: [PATCH 04/12] fix(repo): gate only pack deletion on an unmoved manifest; restart reads without dropping the view (#216) --- crates/repo/src/bucket_transport.rs | 5 +- crates/repo/src/bucket_transport/compact.rs | 52 ++++++++++++--------- 2 files changed, 31 insertions(+), 26 deletions(-) diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index fbd772e5..f72eb962 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -278,15 +278,14 @@ impl BucketTransport { } /// Run a read against the current view; if it finds a named object gone - /// (a compaction deleted it), drop the view, re-read the manifest and run - /// it again. An unchanged manifest means the absence is real: fail closed. + /// (a compaction deleted it), re-read the manifest and run it again. An unchanged manifest means the absence is real: fail closed. fn restarting(&self, mut f: impl FnMut() -> Result) -> Result { for _ in 0..MAX_RESTARTS { self.saw_absent.set(false); let before = self.view.borrow().as_ref().map(|v| v.tag.clone()); match f() { Err(e) if self.saw_absent.get() => { - *self.view.borrow_mut() = None; + // A conditional read: `Unchanged` keeps the view and its tag. self.refresh()?; let after = self.view.borrow().as_ref().map(|v| v.tag.clone()); if before == after { diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index 25cf4b79..f86e08e0 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -362,20 +362,27 @@ impl BucketTransport { } } - rep.cutoff = cutoff; - rep.logs_deleted = del_logs.len(); - rep.checkpoints_deleted = del_ckpts.len(); - rep.packs_deleted = del_packs.len(); - rep.bytes_deleted = del_logs + // Log entries and checkpoints behind the cutoff are safe whatever + // the manifest does meanwhile: the cutoff only ever grows, so they + // stay unreachable. Counters add up across replans. + rep.cutoff = rep.cutoff.max(cutoff); + rep.logs_deleted += del_logs.len(); + rep.checkpoints_deleted += del_ckpts.len(); + rep.bytes_deleted += del_logs .iter() .chain(del_ckpts.iter()) .map(|m| m.size) - .chain( - del_packs - .iter() - .flat_map(|(_, ms)| ms.iter().map(|m| m.size)), - ) - .sum(); + .sum::(); + if !opts.dry_run { + for m in del_logs.iter().chain(del_ckpts.iter()) { + self.bucket.delete(&m.key)?; + } + } + rep.packs_deleted += del_packs.len(); + rep.bytes_deleted += del_packs + .iter() + .flat_map(|(_, ms)| ms.iter().map(|m| m.size)) + .sum::(); if opts.dry_run { if rep.packs_created > 0 { rep.notes.push( @@ -386,19 +393,17 @@ impl BucketTransport { } return Ok(()); } - // The manifest moving means a push or fold landed during planning: - // replan rather than delete against a stale picture. Log entries - // and checkpoints stay safe regardless (the cutoff only grows), - // but the pack candidates may just have been committed. + // A pack nothing names can still be named by a push that lands + // while we plan: if the manifest moved, replan the pack set + // (nothing is deleted against a stale picture). if !matches!(self.bucket.get("manifest", Some(&tag))?, Fetched::Unchanged) { + rep.packs_deleted -= del_packs.len(); + rep.bytes_deleted -= del_packs + .iter() + .flat_map(|(_, ms)| ms.iter().map(|m| m.size)) + .sum::(); continue; } - for m in &del_logs { - self.bucket.delete(&m.key)?; - } - for m in &del_ckpts { - self.bucket.delete(&m.key)?; - } for (hash, _) in &del_packs { // idx first: a half-deleted pack is then plain debris. self.bucket.delete(&format!("packs/{hash}.idx"))?; @@ -406,8 +411,9 @@ impl BucketTransport { } return Ok(()); } - rep.notes - .push("manifest kept moving during the sweep: rerun".into()); + rep.notes.push( + "the manifest kept moving during the sweep: unreferenced packs were kept, rerun".into(), + ); Ok(()) } From ae8ca0a342b6a037012b1bac6d17bb15be89dda1 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:23:52 +0300 Subject: [PATCH 05/12] docs: ADR-0046 addendum, threat model, roadmap, capability map for bucket compaction (#216) --- .claude/skills/sc-cli-reference/SKILL.md | 10 +++++++++ ARCHITECTURE.md | 6 ++++-- CLAUDE.md | 10 +++++++-- README.md | 4 ++-- ROADMAP.md | 22 ++++++------------- docs/THREAT-MODEL.md | 23 +++++++++++++++++--- docs/adr/0046-wal-bucket-remotes.md | 27 +++++++++++++++++++++++- docs/adr/0055-bucket-compaction.md | 19 +++++++++++------ docs/adr/README.md | 3 ++- 9 files changed, 91 insertions(+), 33 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 3e93947c..3606aa7d 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -357,6 +357,16 @@ cargo run --bin sc -- remote add sc+s3:/// # same, on a # content stays ciphertext); `clone --filter` is # refused; signatures, transcripts and drop # tombstones travel like on any native remote (#229) +cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] + # bucket remotes only (#216, ADR-0055): merge small + # packs behind one manifest CAS, then delete log + # entries, checkpoints and packs nothing names that + # are older than --prune-expire (same flag and default + # as `sc gc`; keep it well above any push's duration). + # Needs bucket list + delete rights. --dry-run prints + # the plan and writes nothing. Safe beside live + # pushes/fetches; run one at a time. Relieves the + # 65536-pack checkpoint cap (not the ref cap) cargo run --bin sc -- serve --http --store sc+wal://|sc+s3:/// # serve a bucket remote over sc+http(s):// (P36c): # all content lives in the bucket (instance is diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 8ea729d1..8aadfd38 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1173,8 +1173,10 @@ the checkpoint and walks only the tail. P36c adds `sc serve --store `: an identity and scratch space while all served content lives in the bucket, so instances are disposable. Public content sits in the bucket as plaintext at rest (the bucket ACL is the perimeter; sealed content stays ciphertext), and -`sc clone --filter` is refused against a bucket remote. Bucket compaction and gc -are not built (open issue #216). A bucket keeps no signature or transcript +`sc clone --filter` is refused against a bucket remote. `sc remote compact` (ADR-0055, #216) merges small packs behind one manifest +CAS, then deletes log entries, checkpoints and unreferenced packs behind a cutoff +that has aged past `--prune-expire`; readers that find a named object gone +re-read the manifest and restart, and there is no format change. A bucket keeps no signature or transcript index, so its `get_pack` rebuilds the over-send from the packs the manifest chain names; signatures, transcripts and signed drop tombstones therefore travel through a bucket like any other remote, but each fresh fetch or clone diff --git a/CLAUDE.md b/CLAUDE.md index bb2bde70..14bc0936 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,7 +151,7 @@ the code, those win. | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) | | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | -| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change) | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change); `sc remote compact [--dry-run] [--prune-expire 24h]` (#216) merges small packs behind one manifest CAS, then deletes log entries, checkpoints and unreferenced packs behind an aged-checkpoint cutoff, readers restart on a moved manifest, no format or wire change | [0046](docs/adr/0046-wal-bucket-remotes.md), [0055](docs/adr/0055-bucket-compaction.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 2 GiB)` (a constant since #213, independent of the object cap) and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | @@ -194,6 +194,12 @@ transport-adjacent. The rest, imperatively: - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. +- **`sc remote compact` deletes bucket objects** (ADR-0055): it needs bucket + list and delete rights, which pushers and `sc serve --store` instances should + not hold. It never touches the manifest and never deletes what the current + chain names. Keep `--prune-expire` well above any push's duration (default + 24h): a writer stalled longer than that between uploading a pack and its + commit can still lose the pack (the leases follow-on closes it). - **`sc serve --store` still requires a local serve home with `.sc/`** — tokens, TLS identity, and pack spills live there; the bucket holds all served content. @@ -305,7 +311,7 @@ transport-adjacent. The rest, imperatively: Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of truth — do not re-list items here); entries that name an issue mirror the open GitHub issues. Standing entries: named/concurrent `sc ws` sessions (#215), -blob-size clone filters (#217), bucket compaction and gc (#216), the non-unix +blob-size clone filters (#217), the non-unix `.sc/local-key` ACL (#214), transcript-drop delivery gaps (#230), and the open sparse, private-branch and abort edge cases (#241, #246, #247, #248). diff --git a/README.md b/README.md index edb031f2..812c769d 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ back to Git commits, and reaches hosted Git (e.g. GitHub) through a spawned system-git mirror bridge. See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design, -[docs/adr/](docs/adr/) for the architecture decision records (0001-0054) behind +[docs/adr/](docs/adr/) for the architecture decision records (0001-0055) behind every phase, and [CLAUDE.md](CLAUDE.md) for project conventions and the per-phase capability map. [CONTRIBUTING.md](CONTRIBUTING.md) covers the development workflow. @@ -170,7 +170,7 @@ and Git (including hosted GitHub) network transports, plus private branches sealed to a recipient set until publish and a read-only Tauri desktop browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds 27 independent end-to-end proofs, one per major capability. Remaining -follow-ons (transcript drop delivery gaps, bucket compaction and gc, named +follow-ons (transcript drop delivery gaps, named concurrent `sc ws` sessions, size/count clone filters, and similar hardening items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open GitHub issues rather than being open MVP scope. diff --git a/ROADMAP.md b/ROADMAP.md index c21a54c4..0dc7bafb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -564,7 +564,9 @@ across every phase. Public content is plaintext at rest (the bucket ACL is the perimeter); `clone --filter` is refused. `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format - change). (ADR-0046.) + change). `sc remote compact` (#216) merges small packs and deletes the log + entries, checkpoints and unreferenced packs behind an aged cutoff; readers + restart on a moved manifest; no format change. (ADR-0046, ADR-0055.) - **Phase 37a — Want-by-id fetch.** A batched `GetObjects` verb (wire v5), an opt-in `.sc/promisor-lazy` marker, and `sc cat :`, which fills only the absent trees and the blob on its path. `Store::get` stays network-free. @@ -649,7 +651,7 @@ under Deferred below. | **P33 — Randomized protected encryption** | Close equality confirmation for new protected content | fresh DEK + nonce per seal, dual-read/randomized-write, keyed local stat cache, rewrap upgrade | [0043](docs/adr/0043-randomized-protected-encryption.md) | | **P34 — Private branches** | Keep an entire branch opaque before publication | sealed object closure behind a manifest; O(1) grant, KEK-rotating revoke, scanner-gated publish | [0044](docs/adr/0044-per-branch-access-control.md) | | **P35 — Native desktop browser** | Prove native `.sc` objects can power a polished visual tool | Tauri app opens a repository and browses refs, snapshot DAG/provenance, public trees, and first-parent diffs while locked/private states remain safe | [0045](docs/adr/0045-native-desktop-read-model.md) | -| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add sc+wal://` or `sc+s3:///`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store ` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add sc+wal://` or `sc+s3:///`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store ` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); `sc remote compact` prunes and merges (#216); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md), [0055](docs/adr/0055-bucket-compaction.md) | | **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) | | **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) | @@ -873,22 +875,12 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in fetch-time freshness attestation so a hostile remote can't re-serve a stale pre-revoke manifest — the same gittuf-shaped signed-ref effort the P22 provenance boundary already defers. -- **Bucket compaction/gc (#216, P36a follow-on).** The WAL log (`log/` keys, - each a hand-rolled versioned binary `walfmt` entry, not protobuf) grows - unboundedly with no compaction — a cold-start reader now stops at the - latest checkpoint (P36b) instead of walking to `0`, but no log entry, - checkpoint, or superseded pack is ever actually removed from the bucket. - Deferred until a safe pruning cutoff (e.g. "no reader can still need - anything before checkpoint N") is designed. Compaction is also what - relieves `walfmt::MAX_LIST` (65536): a checkpoint fold with more refs or - packs than that cap is skipped outright (`maybe_fold_checkpoint`'s guard, - P36b review) rather than writing an object `Checkpoint::decode` would then - refuse to read back, so a remote whose live ref/pack count grows past the - cap loses folding entirely until compaction can retire entries below it. - **Leases (P36a follow-on).** The only cross-writer coordination today is the manifest's compare-and-swap; there is no lease/TTL primitive for operations that need to hold exclusive intent across more than one bucket - round-trip (e.g. a long-running compaction). Deferred. + round-trip (e.g. a long-running compaction). Deferred. It would also close + the one residual hazard ADR-0055 documents: a writer stalled longer than + `sc remote compact --prune-expire` between uploading a pack and committing it. - **Derived-data layer (Scion-inspired, after P37).** Blame, path history, commit-graph skip lists and copy-detection computed by one generic recursive mechanism over snapshots, stored outside the CAS and rebuildable diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 5d73de0d..c75fed20 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -402,9 +402,26 @@ too, not just the src-control-side metadata. from the key it was fetched at, and a log chain that steps past the checkpoint without landing on it. A writer with bucket access can fold a checkpoint, but it is derived data any reader can refold from the log, and a - lost manifest CAS drops a fold silently. Nothing is ever removed from a - bucket (no compaction or gc yet, #216), so a hostile writer can grow it - without bound; bucket quotas are the operator's bound. + lost manifest CAS drops a fold silently. A hostile writer can grow a bucket + without bound; bucket quotas are the operator's bound, and `sc remote + compact` reclaims what is unreferenced (below). +- **Compaction deletes data, so it is a privileged operation (#216, + ADR-0055).** `sc remote compact` needs bucket **list and delete** rights in + addition to the get/put a pusher holds. Grant them to a dedicated operator or + scheduled-job identity, not to ordinary pushers, read-only clients or the + `sc serve --store` instances (they never need them: compaction is not a wire + verb, so a served bucket's clients cannot trigger a delete). It cannot + create new authority: it never touches the manifest's refs, only removes + objects no current manifest chain names, and a writer who can already + write to the bucket can do worse with raw access. What it does change for a + hostile or mistaken writer: (a) a stalled push whose uploaded pack is older + than `--prune-expire` can lose it and fail on retry, or in a race narrower + than one bucket round trip commit an entry naming a deleted pack, which + makes the remote unreadable until fixed (hence the 24h default, and + leases as the real fix); (b) a hostile write that fills `log/` or + `checkpoints/` with junk is reclaimed once it falls behind the cutoff. + Compaction decodes the records it keeps strictly and stops deleting packs if + one is unreadable (fail closed). Use `--dry-run` first on a new bucket. - **Sealed content stays end-to-end sealed, unchanged.** Protected-path ciphertext (ADR-0014/0043) and committed secrets travel through a bucket remote exactly as through any other transport: a bucket reader without the diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index e3f58f3f..f780ca37 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -106,7 +106,8 @@ confidentiality perimeter for public content, not the sc protocol. suite). - The log grows unboundedly with no compaction yet — every open reader walks the full parent chain back from `head_seq`. Checkpoint folding (P36b) and - bucket GC/compaction are deferred (see `ROADMAP.md`). + bucket GC/compaction are deferred (see `ROADMAP.md`). *(Both are built: + P36b below, compaction and gc in ADR-0055.)* - `sc serve` still cannot host a bucket as its backing store (P36c) — a bucket remote today is written to directly by every `sc push`/`sc fetch` client, not brokered through a server process. @@ -218,3 +219,27 @@ hash). No manifest, log, checkpoint or idx change and no `PROTOCOL_VERSION` bump. The receiver's trust gate for tombstones and the `--no-transcripts` skip are unchanged. Rationale and cost in ADR-0038's #229 addendum. +## As built (#216, 2026-10): compaction and gc + +`sc remote compact [--dry-run] [--prune-expire 24h]` merges small live +packs behind one manifest CAS, then deletes log entries, checkpoints and +unreferenced packs that no reader or writer can still need. The design, the +pruning cutoff, the reader and writer protocols and the residual limit live in +ADR-0055. Effects on this ADR's contracts: + +- **No format change.** The manifest, log entry and checkpoint encodings and + `PROTOCOL_VERSION` are untouched. Compaction writes a log entry with + `updates = []` and a checkpoint whose `packs` list is a covering set rather + than a literal accumulation; both decode in every released reader. +- **Readers restart instead of failing when an object they just read about + is gone.** The "absent checkpoint / log entry / idx / pack" refusals in + P36b stay fail-closed, but only after a re-read of the manifest shows it did + not move. A moved manifest restarts the read on the new one. +- **A pending pack is verified present right before the manifest CAS**, so a + push whose pack was swept fails with a retry message rather than committing + an entry that names a deleted pack. +- `objio`'s `Bucket` gained `list_meta` and `delete`. The "nothing already in + the bucket is ever overwritten" property holds; compaction only adds new + objects and removes unreferenced ones. +- The ADR-0046 `MAX_LIST` fold guard is unchanged. Compaction is what makes a + skipped fold recoverable, on the pack axis. diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 27c706d9..4806fe0d 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -85,10 +85,14 @@ never deleted. not). A pack named by an old checkpoint or a straggler's in-flight entry is therefore kept. If a retained object cannot be decoded the sweep stops without deleting packs (fail closed). -- **Order:** log entries, then older checkpoints, then idx, then pack. Every - prefix of this order leaves a bucket a cold start reads, because none of - these are reachable from the manifest being deleted-against. The manifest - is re-read before each batch; if it moved the sweep stops and reports it. +- **Order:** log entries, then older checkpoints, then (for each pack) idx, + then pack. Every prefix of this order leaves a bucket a cold start reads, + because none of these are reachable from the manifest being deleted against. + Log entries and checkpoints behind the cutoff are safe whatever the manifest + does meanwhile (the cutoff only grows). Pack deletion alone is gated on the + manifest being unchanged since the pack set was computed; if it moved, the + pack set is replanned (bounded), and otherwise the run keeps the packs and + says so. - `K` rather than "the current checkpoint" matters on a busy remote: folds happen every 64 pushes, so the newest checkpoint is rarely `grace` old, while an older one is. @@ -134,9 +138,10 @@ store offers no atomic "delete if the manifest is unchanged". Two defenses: time is at least `grace` old (default 24 h). A writer's gap between upload and CAS is milliseconds to minutes. 2. **Verify before commit.** `update_ref` checks (`list_meta("packs/.")`) - that each pending pack and idx still exist before every log-slot claim, and - fails with a clear retry-the-push error if not, instead of committing an - entry that names a deleted pack. + that each pending pack and idx still exist immediately before every + manifest CAS (after the log slot is claimed), and fails with a clear + retry-the-push error if not, instead of committing an entry that names a + deleted pack. The window left is one bucket round trip. **Residual limit (documented, not closed).** A writer that stalls longer than `grace` between uploading a pack and its CAS, or re-pushes byte-identical diff --git a/docs/adr/README.md b/docs/adr/README.md index f3ba57be..13e7d587 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0043](0043-randomized-protected-encryption.md) | Randomized protected-path encryption (dual-read, randomized-write) | Accepted; rotate-for-paths built as 0051 | 33 | | [0044](0044-per-branch-access-control.md) | Per-branch access control with opaque private branches | Accepted; sparse merge-in section (#240) | 34 | | [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 | -| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | +| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted; compaction 0055 | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (built) | 37 | | [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (built) | 37b | | [0049](0049-merge-on-partial-clones.md) | Merge on partial clones | Accepted (built) | 37c | @@ -65,6 +65,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0052](0052-secret-injection-by-fd.md) | Secret injection by file descriptor (`sc run --secrets-fd`) | Accepted (built) | — | | [0053](0053-tls-pin-management.md) | TLS pin management (`sc tls`) | Accepted (built) | — | | [0054](0054-tls-ca-validation.md) | CA-path validation for `sc+https://` | Accepted (built) | — | +| [0055](0055-bucket-compaction.md) | Bucket remote compaction and gc (`sc remote compact`) | Accepted (built) | 36 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, From dfa8ea09d796aabb60f3a58b4975144957298e33 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:29:39 +0300 Subject: [PATCH 06/12] fix(repo): recheck the ref cap on the state compaction publishes (#216) --- crates/repo/src/bucket_transport/compact.rs | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index f86e08e0..04a21248 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -226,8 +226,15 @@ impl BucketTransport { kept.push(h.clone()); } } - if kept.len() > MAX_LIST { - continue 'attempt; // new pushes pushed the list over the cap + // Recheck both axes on the state we are about to publish: pushes + // that landed while we merged can have pushed either over the cap, + // and an undecodable checkpoint would brick the remote. + if super::fold_would_overflow(refs.len(), kept.len()) { + rep.notes.push( + "new pushes took the checkpoint over the cap while merging: nothing merged" + .into(), + ); + return Ok(()); } let mut try_seq = head_seq + 1; let entry = LogEntry { From b178992c8617ae87d11d1b70c70767a17ed2e776 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:29:51 +0300 Subject: [PATCH 07/12] docs(adr): note the stale-writer slot reuse limit (#216) --- docs/adr/0055-bucket-compaction.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 4806fe0d..8d68736f 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -147,7 +147,13 @@ store offers no atomic "delete if the manifest is unchanged". Two defenses: `grace` between uploading a pack and its CAS, or re-pushes byte-identical content to an old orphan in the same instant a sweep selects it, can still commit an entry naming a deleted pack. Closing that needs a lease or a -conditional delete, which the `Bucket` trait lacks (ROADMAP "Leases"). Run +conditional delete, which the `Bucket` trait lacks (ROADMAP "Leases"). The +same staleness has a second, milder form: a writer paused longer than `grace` +between reading the manifest and claiming a log slot can claim a slot the +sweep freed behind the cutoff. Its manifest CAS fails, but a reader equally +stale (older than `grace`, still walking the chain it read) could read that +phantom entry instead of seeing the slot absent. Both need a participant stalled +past the window; neither affects anyone who read the manifest within it. Run compaction with a `grace` well above any plausible writer stall; the 24 h default matches `sc gc --prune-expire`. From a6b5a18f31ce29d9fa503772c6cce5479f712160 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 20:59:35 +0300 Subject: [PATCH 08/12] fix(repo,objio,cli): review fixes - HEAD-based pack check, grace floor, replan on rival compaction (#216) --- .claude/skills/sc-cli-reference/SKILL.md | 4 +- crates/cli/src/main.rs | 11 +++ crates/cli/tests/bucket_remote.rs | 17 ++++- crates/objio/src/dir.rs | 30 ++++++--- crates/objio/src/lib.rs | 4 ++ crates/objio/src/s3.rs | 15 +++++ crates/repo/src/bucket_transport.rs | 15 +++-- crates/repo/src/bucket_transport/compact.rs | 30 +++++++-- .../src/bucket_transport/compact/tests.rs | 67 +++++++++++++------ docs/THREAT-MODEL.md | 3 +- docs/adr/0055-bucket-compaction.md | 23 ++++--- 11 files changed, 173 insertions(+), 46 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 3606aa7d..8c4d5c7d 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -365,7 +365,9 @@ cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] # as `sc gc`; keep it well above any push's duration). # Needs bucket list + delete rights. --dry-run prints # the plan and writes nothing. Safe beside live - # pushes/fetches; run one at a time. Relieves the + # pushes/fetches; run one at a time. A --prune-expire under 1h is refused + # unless --allow-short-grace (only with no pushes + # or fetches running). Relieves the # 65536-pack checkpoint cap (not the ref cap) cargo run --bin sc -- serve --http --store sc+wal://|sc+s3:/// # serve a bucket remote over sc+http(s):// (P36c): diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 61c6bbc4..cf37e056 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -815,6 +815,10 @@ enum RemoteOp { /// (e.g. 24h, 7d). Keep it well above any push's duration. #[arg(long, default_value = "24h")] prune_expire: String, + /// Allow a --prune-expire under 1h. Only safe while nothing pushes to + /// or fetches from the bucket: a push in flight can lose its pack. + #[arg(long)] + allow_short_grace: bool, }, } @@ -4126,8 +4130,15 @@ fn run_remote(op: RemoteOp) -> Result<()> { name, dry_run, prune_expire, + allow_short_grace, } => { let grace = parse_duration(&prune_expire)?; + if grace < std::time::Duration::from_secs(3600) && !allow_short_grace { + anyhow::bail!( + "--prune-expire under 1h can delete a pack an in-flight push has uploaded but \ + not committed; pass --allow-short-grace only if nothing is pushing or fetching" + ); + } let url = repo .remotes()? .into_iter() diff --git a/crates/cli/tests/bucket_remote.rs b/crates/cli/tests/bucket_remote.rs index 68b902b5..8ed4bac1 100644 --- a/crates/cli/tests/bucket_remote.rs +++ b/crates/cli/tests/bucket_remote.rs @@ -134,7 +134,22 @@ fn remote_compact_merges_and_sweeps_a_bucket_and_clones_still_work() { ); // zero grace: sweeps the old packs and the log - let out = sc(&a, &["remote", "compact", "origin", "--prune-expire", "0s"]); + let refused = sc(&a, &["remote", "compact", "origin", "--prune-expire", "0s"]); + assert!( + !refused.status.success(), + "a zero grace needs an explicit opt-in" + ); + let out = sc( + &a, + &[ + "remote", + "compact", + "origin", + "--prune-expire", + "0s", + "--allow-short-grace", + ], + ); assert!(out.status.success(), "{out:?}"); assert_eq!((count("log"), count("packs")), (0, 2)); diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs index 7029a849..5dfba640 100644 --- a/crates/objio/src/dir.rs +++ b/crates/objio/src/dir.rs @@ -120,7 +120,12 @@ impl Bucket for DirBucket { } fn list_meta(&self, prefix: &str) -> Result> { - fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { + fn walk( + dir: &Path, + root: &Path, + prefix: &str, + out: &mut Vec, + ) -> std::io::Result<()> { let rd = match std::fs::read_dir(dir) { Ok(rd) => rd, Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), @@ -135,6 +140,15 @@ impl Bucket for DirBucket { }) { continue; } + let rel = p + .strip_prefix(root) + .unwrap() + .to_string_lossy() + .replace('\\', "/"); + // Skip files outside the prefix before paying for a stat. + if entry.file_type()?.is_file() && !rel.starts_with(prefix) { + continue; + } // A concurrent delete/rename can remove the entry between the // readdir and the stat; a key that is gone is just not listed. let md = match std::fs::metadata(&p) { @@ -143,14 +157,10 @@ impl Bucket for DirBucket { Err(e) => return Err(e), }; if md.is_dir() { - walk(&p, root, out)?; + walk(&p, root, prefix, out)?; } else { out.push(ObjectMeta { - key: p - .strip_prefix(root) - .unwrap() - .to_string_lossy() - .replace('\\', "/"), + key: rel, size: md.len(), modified: md.modified()?, }); @@ -166,12 +176,16 @@ impl Bucket for DirBucket { Some((dir, _)) => self.root.join(dir), None => self.root.clone(), }; - walk(&start, &self.root, &mut out)?; + walk(&start, &self.root, prefix, &mut out)?; out.retain(|m| m.key.starts_with(prefix)); out.sort_by(|a, b| a.key.cmp(&b.key)); Ok(out) } + fn exists(&self, key: &str) -> Result { + Ok(self.key_path(key)?.is_file()) + } + fn delete(&self, key: &str) -> Result<()> { match std::fs::remove_file(self.key_path(key)?) { Ok(()) => Ok(()), diff --git a/crates/objio/src/lib.rs b/crates/objio/src/lib.rs index 5cae152a..84c95aeb 100644 --- a/crates/objio/src/lib.rs +++ b/crates/objio/src/lib.rs @@ -68,6 +68,8 @@ pub trait Bucket: Send { /// scope, so `packs/ab12.` matches `packs/ab12.pack`), with size and /// last-modified time, sorted by key. fn list_meta(&self, prefix: &str) -> Result>; + /// Whether `key` exists (a HEAD: needs read rights only, no listing). + fn exists(&self, key: &str) -> Result; /// Remove `key`. Idempotent: an absent key is success. fn delete(&self, key: &str) -> Result<()>; } @@ -149,6 +151,8 @@ pub fn contract_suite(b: &dyn Bucket) { assert!(age < std::time::Duration::from_secs(3600)); assert_eq!(b.list_meta("packs/").unwrap().len(), 3); assert!(b.list_meta("nothing/").unwrap().is_empty()); + assert!(b.exists("packs/ab12.pack").unwrap()); + assert!(!b.exists("packs/zz.pack").unwrap()); // delete: removes, is idempotent, touches only its key b.delete("packs/ab12.pack").unwrap(); b.delete("packs/ab12.pack").unwrap(); diff --git a/crates/objio/src/s3.rs b/crates/objio/src/s3.rs index 67ddc665..68f8683e 100644 --- a/crates/objio/src/s3.rs +++ b/crates/objio/src/s3.rs @@ -223,6 +223,21 @@ impl Bucket for S3Bucket { Ok(out) } + fn exists(&self, key: &str) -> Result { + let k = self.full_key(key)?; + match self.rt.block_on( + self.client + .head_object() + .bucket(&self.bucket) + .key(&k) + .send(), + ) { + Ok(_) => Ok(true), + Err(e) if http_status(&e) == Some(404) => Ok(false), + Err(e) => Err(Error::Backend(format!("s3 head {k}: {e}"))), + } + } + fn delete(&self, key: &str) -> Result<()> { let k = self.full_key(key)?; // S3 DeleteObject succeeds on an absent key, which is the contract. diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index f72eb962..c6f5d90d 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -390,6 +390,11 @@ impl BucketTransport { index.insert(id, (hash.clone(), offset, length)); } } + // Packs a compaction retired will never be asked about again. + { + let live: std::collections::HashSet<&String> = packs.iter().collect(); + self.leaf_cache.borrow_mut().retain(|h, _| live.contains(h)); + } *self.view.borrow_mut() = Some(WalView { tag, manifest, @@ -560,13 +565,12 @@ impl BucketTransport { /// (a compaction swept it as an unreferenced orphan after an unusually long /// gap, ADR-0055). Committing a log entry that names a deleted pack would /// corrupt the remote; failing lets the caller retry the push, which - /// re-uploads. One prefix listing per pending pack, matching both objects. + /// re-uploads. Two HEADs per pending pack (read rights only: pushers need + /// no list or delete rights). /// Called right before the manifest CAS. fn ensure_packs_present(&self, hashes: &[String]) -> Result<()> { for hash in hashes { - let have = self.bucket.list_meta(&format!("packs/{hash}."))?; - let has = |key: String| have.iter().any(|m| m.key == key); - if !(has(pack_key(hash)) && has(idx_key(hash))) { + if !(self.bucket.exists(&pack_key(hash))? && self.bucket.exists(&idx_key(hash))?) { return Err(Error::Remote(format!( "pack {hash} was removed from the bucket before this push committed \ (bucket compaction swept it); retry the push" @@ -1844,6 +1848,9 @@ mod tests { fn list_meta(&self, prefix: &str) -> scl_objio::Result> { self.0.list_meta(prefix) } + fn exists(&self, key: &str) -> scl_objio::Result { + self.0.exists(key) + } fn delete(&self, key: &str) -> scl_objio::Result<()> { self.0.delete(key) } diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index 04a21248..8dc1f74b 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -10,7 +10,7 @@ //! cold start reads. use super::{capped, BucketTransport}; -use crate::error::Result; +use crate::error::{Error, Result}; use crate::walfmt::{checkpoint_key, log_key, pack_key, Checkpoint, LogEntry, Manifest, MAX_LIST}; use scl_core::pack::parse_pack_reader; use scl_core::ObjectId; @@ -138,6 +138,8 @@ impl BucketTransport { /// Phase 1: merge small live packs and commit with one manifest CAS. fn consolidate(&self, opts: &CompactOptions, rep: &mut CompactReport) -> Result<()> { + // Manifest tag of an attempt that found a planned member pack missing. + let mut missing_on: Option = None; 'attempt: for _ in 0..MAX_ATTEMPTS { self.refresh()?; let (live, nrefs) = { @@ -178,9 +180,9 @@ impl BucketTransport { )); return Ok(()); } - rep.packs_merged = replaced.len(); - rep.packs_created = groups.len(); if opts.dry_run { + rep.packs_merged = replaced.len(); + rep.packs_created = groups.len(); return Ok(()); } let mut created = Vec::new(); @@ -188,7 +190,16 @@ impl BucketTransport { let mut objects: BTreeMap> = BTreeMap::new(); for hash in group { let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { - // Another compactor swept it: the plan is stale. + // Either another compactor retired it (the manifest has + // moved: replan) or the bucket is damaged (same manifest + // twice: say so instead of retrying). + let tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); + if tag == missing_on { + return Err(Error::Wal(format!( + "pack {hash} on chain but body absent" + ))); + } + missing_on = tag; continue 'attempt; }; parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { @@ -216,6 +227,7 @@ impl BucketTransport { v.packs.clone(), ) }; + let packs_now: BTreeSet = packs.iter().cloned().collect(); let mut seen = BTreeSet::new(); let mut kept: Vec = packs .into_iter() @@ -236,6 +248,11 @@ impl BucketTransport { ); return Ok(()); } + // Another compactor may have retired some of the packs we merged + // already: replan rather than publish a second copy of the same data. + if !replaced.is_subset(&packs_now) { + continue 'attempt; + } let mut try_seq = head_seq + 1; let entry = LogEntry { seq: 0, @@ -267,12 +284,17 @@ impl BucketTransport { checkpoint_seq: seq, head_branch, }; + // A sweep by another compactor may have removed a merged pack that + // was a content-identical leftover; never commit a name that is gone. + self.ensure_packs_present(&created)?; if self .bucket .put_if_tag("manifest", &manifest.encode(), Some(&tag))? .is_some() { self.refresh()?; + rep.packs_merged = replaced.len(); + rep.packs_created = created.len(); rep.checkpoint_seq = seq; return Ok(()); } diff --git a/crates/repo/src/bucket_transport/compact/tests.rs b/crates/repo/src/bucket_transport/compact/tests.rs index c323b39f..9f406262 100644 --- a/crates/repo/src/bucket_transport/compact/tests.rs +++ b/crates/repo/src/bucket_transport/compact/tests.rs @@ -86,6 +86,9 @@ impl Bucket for Shared { }) .collect()) } + fn exists(&self, key: &str) -> scl_objio::Result { + Ok(self.0.map.lock().unwrap().contains_key(key)) + } fn delete(&self, key: &str) -> scl_objio::Result<()> { self.0.map.lock().unwrap().remove(key); Ok(()) @@ -443,6 +446,10 @@ impl Bucket for FailAfter { self.tick()?; self.inner.list_meta(prefix) } + fn exists(&self, key: &str) -> scl_objio::Result { + self.tick()?; + self.inner.exists(key) + } fn delete(&self, key: &str) -> scl_objio::Result<()> { self.tick()?; self.inner.delete(key) @@ -555,22 +562,32 @@ fn readers_and_writers_stay_consistent_while_compaction_runs() { }) }) .collect(); - let compactor = s.spawn(|| { - let t = open(&root); - let mut runs = 0u32; - while !done.load(Ordering::SeqCst) { - let rep = t - .compact(&opts) - .unwrap_or_else(|e| panic!("compactor: {e}")); - deleted.fetch_add( - rep.logs_deleted + rep.checkpoints_deleted + rep.packs_deleted, - Ordering::SeqCst, - ); - runs += 1; - std::thread::sleep(Duration::from_millis(40)); - } - runs - }); + // Two compactors at once: the ADR promises that is safe (just wasteful). + let compactors: Vec<_> = (0..2) + .map(|_| { + s.spawn(|| { + let t = open(&root); + let mut runs = 0u32; + while !done.load(Ordering::SeqCst) { + match t.compact(&opts) { + Ok(rep) => { + deleted.fetch_add( + rep.logs_deleted + rep.checkpoints_deleted + rep.packs_deleted, + Ordering::SeqCst, + ); + } + // A rival's sweep took a pack this run was about to + // name; the guard fails the run instead of committing. + Err(Error::Remote(m)) if m.contains("removed from the bucket") => {} + Err(e) => panic!("compactor: {e}"), + } + runs += 1; + std::thread::sleep(Duration::from_millis(40)); + } + runs + }) + }) + .collect(); for w in writers { w.join().unwrap(); } @@ -581,7 +598,9 @@ fn readers_and_writers_stay_consistent_while_compaction_runs() { for r in readers { r.join().unwrap(); } - assert!(compactor.join().unwrap() > 2, "compactor barely ran"); + for c in compactors { + assert!(c.join().unwrap() > 2, "compactor barely ran"); + } }); assert!( deleted.load(Ordering::SeqCst) > 0, @@ -593,7 +612,13 @@ fn readers_and_writers_stay_consistent_while_compaction_runs() { open(&root).compact(&eager()).unwrap(); assert_readable(&root, &expect); assert_eq!(keys(&root, "packs/").len(), 2, "all packs merged into one"); - assert!(keys(&root, "log/").is_empty()); + // Nothing at or below the head survives. A slot above it can: a rival + // compactor's lost CAS leaves its claimed entry as debris, swept once a + // checkpoint passes it. + let head = manifest(&root).head_seq; + assert!(keys(&root, "log/") + .iter() + .all(|k| seq_of(k).unwrap() > head)); std::fs::remove_dir_all(&root).unwrap(); assert!(!root.exists()); } @@ -619,7 +644,8 @@ fn a_reader_whose_view_was_swept_restarts_and_a_corrupt_bucket_still_fails() { .find(|k| k.ends_with(".pack")) .unwrap(); mem.bucket().delete(&pack).unwrap(); - assert!(open(&mem).object_bytes(&tip).is_err()); + let err = open(&mem).object_bytes(&tip).unwrap_err(); + assert!(format!("{err}").contains("body absent"), "{err}"); } #[test] @@ -662,6 +688,9 @@ fn a_cold_start_straddling_a_sweep_restarts_instead_of_failing() { fn list_meta(&self, p: &str) -> scl_objio::Result> { self.inner.list_meta(p) } + fn exists(&self, k: &str) -> scl_objio::Result { + self.inner.exists(k) + } fn delete(&self, k: &str) -> scl_objio::Result<()> { self.inner.delete(k) } diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index c75fed20..40b3c6b8 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -407,7 +407,8 @@ too, not just the src-control-side metadata. compact` reclaims what is unreferenced (below). - **Compaction deletes data, so it is a privileged operation (#216, ADR-0055).** `sc remote compact` needs bucket **list and delete** rights in - addition to the get/put a pusher holds. Grant them to a dedicated operator or + addition to the get/put/head a pusher holds (pushes themselves need no list + or delete rights). Grant them to a dedicated operator or scheduled-job identity, not to ordinary pushers, read-only clients or the `sc serve --store` instances (they never need them: compaction is not a wire verb, so a served bucket's clients cannot trigger a delete). It cannot diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 8d68736f..10a9aa19 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -115,9 +115,10 @@ Readers never take a lock and never need compaction to be quiet. The rule: This is sound because deletion only follows a manifest CAS that moved the tag: the sweep deletes only objects unreferenced by the manifest it read, and -manifest tags never repeat (`head_seq` is strictly increasing and a fold -raises `checkpoint_seq`, so the content hash cannot return to an earlier -value; S3 ETags are content hashes). A reader that lost an object therefore +manifest tags never repeat (every successful CAS strictly increases +`(head_seq, checkpoint_seq)`, so the content cannot return to an earlier value, +and an S3 ETag changes with the content, whether it is an MD5 or an opaque +SSE value). A reader that lost an object therefore always finds a newer manifest on re-read. Object ids stay valid across the restart because consolidation preserves every object, only moving it to a new pack, so `get_pack`'s id lists computed before a restart remain valid. @@ -137,9 +138,10 @@ store offers no atomic "delete if the manifest is unchanged". Two defenses: 1. **Age.** The sweep removes an unreferenced pack only if its last-modified time is at least `grace` old (default 24 h). A writer's gap between upload and CAS is milliseconds to minutes. -2. **Verify before commit.** `update_ref` checks (`list_meta("packs/.")`) - that each pending pack and idx still exist immediately before every - manifest CAS (after the log slot is claimed), and fails with a clear +2. **Verify before commit.** `update_ref` checks (`exists`, a HEAD) that each + pending pack and idx still exist immediately before every manifest CAS + (after the log slot is claimed), so pushers need no list rights; + consolidation does the same for the packs it created, and fails with a clear retry-the-push error if not, instead of committing an entry that names a deleted pack. The window left is one bucket round trip. @@ -176,7 +178,9 @@ it. ### Command surface `sc remote compact [--dry-run] [--prune-expire ]` (default -`24h`, same flag and default as `sc gc`). It is the smallest surface: it needs +`24h`, same flag and default as `sc gc`; a value under 1h is refused unless +`--allow-short-grace` is given, because a short grace turns the residual limit +below into an ordinary race). It is the smallest surface: it needs only the configured remote's URL, adds no wire verb, and is not reachable through `sc serve`, so a served bucket's clients can never trigger a delete. `--dry-run` plans phase 1 and phase 2 against the live bucket and prints what @@ -195,7 +199,10 @@ The `Bucket` trait gains two methods, implemented by `DirBucket` and - `list_meta(prefix)`: keys with size and last-modified time, raw string prefix, sorted. S3 uses `ListObjectsV2`. -- `delete(key)`: idempotent; an absent key is success. +- `exists(key)`: a HEAD, read rights only. Used by the writer-side check. +- `delete(key)`: idempotent; an absent key is success. On a versioned S3 bucket + a delete only adds a delete marker, so space comes back only with a + lifecycle rule that expires noncurrent versions. ## Consequences From 09d1fee70ae43db35854e354c1a7295e2bb05ffe Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 21:18:36 +0300 Subject: [PATCH 09/12] fix(repo): sweep orphans without a manifest, bound decoded merge bytes, tighten docs (#216) --- .claude/skills/sc-cli-reference/SKILL.md | 4 +- CLAUDE.md | 4 +- crates/repo/src/bucket_transport/compact.rs | 53 +++++++++++++------ .../src/bucket_transport/compact/tests.rs | 21 ++++++++ docs/adr/0055-bucket-compaction.md | 8 +-- 5 files changed, 69 insertions(+), 21 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index 8c4d5c7d..38b20bbb 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -367,7 +367,9 @@ cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] # the plan and writes nothing. Safe beside live # pushes/fetches; run one at a time. A --prune-expire under 1h is refused # unless --allow-short-grace (only with no pushes - # or fetches running). Relieves the + # or fetches running). On a versioned S3 + # bucket deletes only add delete markers (add a + # noncurrent-version lifecycle rule). Relieves the # 65536-pack checkpoint cap (not the ref cap) cargo run --bin sc -- serve --http --store sc+wal://|sc+s3:/// # serve a bucket remote over sc+http(s):// (P36c): diff --git a/CLAUDE.md b/CLAUDE.md index 14bc0936..819c4c10 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -196,8 +196,8 @@ transport-adjacent. The rest, imperatively: `filter` against bucket remotes is refused. - **`sc remote compact` deletes bucket objects** (ADR-0055): it needs bucket list and delete rights, which pushers and `sc serve --store` instances should - not hold. It never touches the manifest and never deletes what the current - chain names. Keep `--prune-expire` well above any push's duration (default + not hold. It advances the manifest only through the normal CAS (one no-op + entry plus a checkpoint) and never deletes what the current chain names. Keep `--prune-expire` well above any push's duration (default 24h): a writer stalled longer than that between uploading a pack and its commit can still lose the pack (the leases follow-on closes it). - **`sc serve --store` still requires a local serve home with `.sc/`** — diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index 8dc1f74b..b2d72449 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -21,6 +21,10 @@ use std::time::{Duration, SystemTime}; /// Replans after a lost manifest CAS or a concurrent compactor before giving up. const MAX_ATTEMPTS: u32 = 4; +/// Decoded bytes one merge pack may hold, as a multiple of `group_bytes` +/// (which counts compressed bytes). +const DECODED_FACTOR: u64 = 4; + /// Knobs for [`BucketTransport::compact`]. The defaults are what /// `sc remote compact` uses. #[derive(Debug, Clone)] @@ -125,11 +129,13 @@ impl BucketTransport { }; self.refresh()?; if self.view.borrow().is_none() { + // No manifest yet: nothing to merge, but a first push that died after + // uploading its pack left debris the sweep can still age out. rep.notes - .push("bucket has no manifest: nothing to compact".into()); - return Ok(rep); + .push("bucket has no manifest: only aged orphan packs are swept".into()); + } else { + self.consolidate(opts, &mut rep)?; } - self.consolidate(opts, &mut rep)?; // Read the clock after phase 1 so a zero grace sees its own checkpoint. let now = opts.now.unwrap_or_else(SystemTime::now); self.sweep(opts, now, &mut rep)?; @@ -188,6 +194,7 @@ impl BucketTransport { let mut created = Vec::new(); for group in &groups { let mut objects: BTreeMap> = BTreeMap::new(); + let mut decoded = 0u64; for hash in group { let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { // Either another compactor retired it (the manifest has @@ -203,12 +210,24 @@ impl BucketTransport { continue 'attempt; }; parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { - objects.insert(id, obj.encode()); + let enc = obj.encode(); + decoded += enc.len() as u64; + objects.insert(id, enc); Ok(()) })?; + // Compressed sizes bound the group, not the decoded bytes held + // here; spill to a pack of its own before RAM balloons. + if decoded > opts.group_bytes.saturating_mul(DECODED_FACTOR) { + let part: Vec<(ObjectId, Vec)> = + std::mem::take(&mut objects).into_iter().collect(); + created.push(self.upload_pack(&part)?.0); + decoded = 0; + } + } + if !objects.is_empty() { + let objects: Vec<(ObjectId, Vec)> = objects.into_iter().collect(); + created.push(self.upload_pack(&objects)?.0); } - let objects: Vec<(ObjectId, Vec)> = objects.into_iter().collect(); - created.push(self.upload_pack(&objects)?.0); } // Fence + fold in one CAS: the manifest tag we read is the state // the checkpoint describes, so a lost CAS just replans. @@ -315,14 +334,13 @@ impl BucketTransport { let old = |m: &ObjectMeta| m.modified <= old_before; for _ in 0..MAX_ATTEMPTS { self.refresh()?; - let (tag, ck_seq, live) = { - let view = self.view.borrow(); - let v = view.as_ref().expect("checked non-empty"); - ( - v.tag.clone(), + let (tag, ck_seq, live) = match self.view.borrow().as_ref() { + Some(v) => ( + Some(v.tag.clone()), v.manifest.checkpoint_seq, v.packs.iter().cloned().collect::>(), - ) + ), + None => (None, 0, BTreeSet::new()), }; let ckpts: Vec<(u64, ObjectMeta)> = self .bucket @@ -415,8 +433,9 @@ impl BucketTransport { if opts.dry_run { if rep.packs_created > 0 { rep.notes.push( - "dry run: packs the planned merge replaces become deletable once it \ - lands and the grace window passes" + "dry run: the sweep figures describe the bucket as it is now; the merge \ + and what it retires (its replaced packs, the log up to its new \ + checkpoint) are not simulated" .into(), ); } @@ -425,7 +444,11 @@ impl BucketTransport { // A pack nothing names can still be named by a push that lands // while we plan: if the manifest moved, replan the pack set // (nothing is deleted against a stale picture). - if !matches!(self.bucket.get("manifest", Some(&tag))?, Fetched::Unchanged) { + let unmoved = matches!( + (&tag, self.bucket.get("manifest", tag.as_deref())?), + (Some(_), Fetched::Unchanged) | (None, Fetched::Absent) + ); + if !unmoved { rep.packs_deleted -= del_packs.len(); rep.bytes_deleted -= del_packs .iter() diff --git a/crates/repo/src/bucket_transport/compact/tests.rs b/crates/repo/src/bucket_transport/compact/tests.rs index 9f406262..e686af7e 100644 --- a/crates/repo/src/bucket_transport/compact/tests.rs +++ b/crates/repo/src/bucket_transport/compact/tests.rs @@ -825,3 +825,24 @@ fn empty_bucket_and_single_pack_are_noops() { assert_eq!((rep.packs_merged, rep.logs_deleted), (0, 0)); assert_readable(&mem, &refs); } + +#[test] +fn a_first_push_that_died_before_the_manifest_leaves_orphans_the_sweep_ages_out() { + let mem = Arc::new(MemBucket::default()); + let (_, objects) = commit("never committed"); + let (pack, idx) = scl_core::pack::build_pack(&objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + let b = mem.bucket(); + b.put_new(&pack_key(&hash), &pack).unwrap(); + b.put_new(&idx_key(&hash), &idx).unwrap(); + let t = open(&mem); + // young: kept + let rep = t.compact(&CompactOptions::default()).unwrap(); + assert_eq!((rep.packs_deleted, rep.packs_held_back), (0, 1)); + assert_eq!(keys(&mem, "packs/").len(), 2); + // aged: swept, and the bucket is still a valid empty remote + let rep = t.compact(&eager()).unwrap(); + assert_eq!(rep.packs_deleted, 1, "{rep:?}"); + assert!(keys(&mem, "packs/").is_empty()); + assert!(open(&mem).list_refs().unwrap().is_empty()); +} diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 10a9aa19..69fd40f0 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -70,8 +70,8 @@ chain a new reader walks, but older checkpoints and log entries still name them. They are removed by phase 2, a grace period later. **Phase 2, sweep (deletes, behind a cutoff).** All deletions are of objects -that no current or future manifest chain references; the manifest itself is -never deleted. +that the current manifest chain does not reference (and, outside the residual +limit below, no later chain will); the manifest itself is never deleted. - **Cutoff K.** Let `K` be the highest `checkpoints/` object with `seq <= manifest.checkpoint_seq` whose last-modified time is at least @@ -207,7 +207,9 @@ The `Bucket` trait gains two methods, implemented by `DirBucket` and ## Consequences - Storage and cold-start cost are bounded by the live data plus one grace - window. A bucket that was past `MAX_LIST` regains folding after one run. + window. A bucket past `MAX_LIST` on the pack axis regains folding after one run if + its small packs merge to a list under the cap; packs above the merge + threshold are never rewritten, and the ref axis is not relieved. - Compaction needs bucket `list` and `delete` rights, which an ordinary pusher or reader does not. See the THREAT-MODEL bucket section. - Only one compaction is useful at a time. Concurrent runs are safe (each From b3cd4cbf8c2f58a974a8325b782e85c3b92a6406 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 22:06:38 +0300 Subject: [PATCH 10/12] test(repo): a clone after compaction still gets signatures, transcripts and tombstones; review fixes (#216) --- crates/objio/src/dir.rs | 6 +- crates/repo/src/bucket_transport.rs | 62 +++++++++++++++++++++ crates/repo/src/bucket_transport/compact.rs | 5 ++ docs/THREAT-MODEL.md | 2 +- docs/adr/0055-bucket-compaction.md | 11 ++++ 5 files changed, 84 insertions(+), 2 deletions(-) diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs index 5dfba640..8957c64c 100644 --- a/crates/objio/src/dir.rs +++ b/crates/objio/src/dir.rs @@ -183,7 +183,11 @@ impl Bucket for DirBucket { } fn exists(&self, key: &str) -> Result { - Ok(self.key_path(key)?.is_file()) + match std::fs::metadata(self.key_path(key)?) { + Ok(md) => Ok(md.is_file()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(e) => Err(e.into()), + } } fn delete(&self, key: &str) -> Result<()> { diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index c6f5d90d..a8aa444b 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -2035,6 +2035,68 @@ mod tests { assert!(!base.exists()); } + /// #216 x #229: compaction merges the packs the sender seam scans for + /// signatures, transcripts and tombstones, so a clone after it must still + /// be served all three. + #[test] + fn a_clone_after_compaction_still_gets_signatures_transcripts_and_tombstones() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("compact"); + for (i, f) in ["f2", "f3"].iter().enumerate() { + std::fs::write(base.join("a").join(f), format!("more {i}")).unwrap(); + a.commit("t", &format!("c{}", i + 2)).unwrap(); + a.push("origin").unwrap(); + } + let compact = || { + BucketTransport::open(&url) + .unwrap() + .compact(&crate::bucket_transport::CompactOptions { + prune_expire: std::time::Duration::ZERO, + ..Default::default() + }) + .unwrap() + }; + let rep = compact(); + assert!(rep.packs_merged >= 3 && rep.packs_deleted >= 3, "{rep:?}"); + let bucket = base.join("bucket"); + let count = |sub: &str| std::fs::read_dir(bucket.join(sub)).unwrap().count(); + assert_eq!(count("packs"), 2, "merged into one pack + idx"); + + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + assert_eq!( + b.sig_status(&snap, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()) + ); + assert!(b + .transcripts_for(&snap) + .unwrap() + .iter() + .any(|(t, _)| *t == tid)); + assert_eq!(b.signatures_for(&tid).unwrap().len(), 1); + + // A signed drop pushed on top of the compacted bucket, then compacted again. + crate::transcripts::tests::trust_signer(&a, &identity); + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(base.join("a/f4"), b"four").unwrap(); + a.commit("t", "c5").unwrap(); + a.push("origin").unwrap(); + compact(); + let tomb = ObjectId::of( + &Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)).encode(), + ); + let c = crate::repo::Repo::clone_url(&url, base.join("c")).unwrap(); + assert!( + c.store_arc().lock().unwrap().contains(&tomb), + "the tombstone survives compaction and is served" + ); + crate::transcripts::tests::trust_signer(&b, &identity); + b.fetch("origin").unwrap(); + assert!(b.transcripts_for(&snap).unwrap().is_empty()); + + drop((a, b, c)); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + #[test] fn fetch_from_a_bucket_delivers_a_later_transcript_and_signature() { let (a, url, snap, tid, identity, base) = signed_bucket_origin("fetch"); diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index b2d72449..afc4e1cc 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -134,6 +134,11 @@ impl BucketTransport { rep.notes .push("bucket has no manifest: only aged orphan packs are swept".into()); } else { + rep.checkpoint_seq = self + .view + .borrow() + .as_ref() + .map_or(0, |v| v.manifest.checkpoint_seq); self.consolidate(opts, &mut rep)?; } // Read the clock after phase 1 so a zero grace sees its own checkpoint. diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 61c0344d..a30ee779 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -423,7 +423,7 @@ too, not just the src-control-side metadata. compact` reclaims what is unreferenced (below). - **Compaction deletes data, so it is a privileged operation (#216, ADR-0055).** `sc remote compact` needs bucket **list and delete** rights in - addition to the get/put/head a pusher holds (pushes themselves need no list + addition to the get/put/head permissions a pusher holds (pushes themselves need no list or delete rights). Grant them to a dedicated operator or scheduled-job identity, not to ordinary pushers, read-only clients or the `sc serve --store` instances (they never need them: compaction is not a wire diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 69fd40f0..e8a7eafd 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -175,6 +175,17 @@ skipped re-uploading an object could reference a dropped pack; it also needs a reachability walk of the whole bucket. A pack is "live" here if the chain names it. +### Signatures, transcripts and tombstones (#229, #254) + +The bucket's sender seam finds signatures, transcripts and signed drop +tombstones by decoding every on-chain pack (`pack_leaves`). Compaction keeps +that sound three ways: an on-chain pack is never dropped, only merged; +consolidation decodes and re-encodes every object of a merged pack, whatever its +kind, so the merged pack holds the same leaves; and `pack_leaves` runs against +the post-compaction pack list (the leaf cache is keyed by pack hash and trimmed +to live packs on refresh). `a_clone_after_compaction_still_gets_signatures_transcripts_and_tombstones` +covers a clone and a fetch across two compactions with a signed drop in between. + ### Command surface `sc remote compact [--dry-run] [--prune-expire ]` (default From 91597ece404e974f28301fc7e660ba81b7bb4778 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 22:18:07 +0300 Subject: [PATCH 11/12] docs: list --allow-short-grace and the stalled-writer limit; qualify the storage bound (#216) --- .claude/skills/sc-cli-reference/SKILL.md | 6 ++++-- docs/adr/0055-bucket-compaction.md | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index ab42f469..3fd043df 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -368,7 +368,7 @@ cargo run --bin sc -- remote add sc+s3:/// # same, on a # content stays ciphertext); `clone --filter` is # refused; signatures, transcripts and drop # tombstones travel like on any native remote (#229) -cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] +cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] [--allow-short-grace] # bucket remotes only (#216, ADR-0055): merge small # packs behind one manifest CAS, then delete log # entries, checkpoints and packs nothing names that @@ -376,7 +376,9 @@ cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] # as `sc gc`; keep it well above any push's duration). # Needs bucket list + delete rights. --dry-run prints # the plan and writes nothing. Safe beside live - # pushes/fetches; run one at a time. A --prune-expire under 1h is refused + # pushes/fetches within --prune-expire (a writer + # stalled longer than that between uploading a pack + # and committing it can lose the pack); run one at a time. A --prune-expire under 1h is refused # unless --allow-short-grace (only with no pushes # or fetches running). On a versioned S3 # bucket deletes only add delete markers (add a diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index e8a7eafd..92462596 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -217,8 +217,9 @@ The `Bucket` trait gains two methods, implemented by `DirBucket` and ## Consequences -- Storage and cold-start cost are bounded by the live data plus one grace - window. A bucket past `MAX_LIST` on the pack axis regains folding after one run if +- On a non-versioned bucket, storage and cold-start cost are bounded by the + live data plus one grace window (a versioned bucket keeps delete-marked + versions until a lifecycle rule expires them). A bucket past `MAX_LIST` on the pack axis regains folding after one run if its small packs merge to a list under the cap; packs above the merge threshold are never rewritten, and the ref axis is not relieved. - Compaction needs bucket `list` and `delete` rights, which an ordinary From 359d083d0072af6cf2fdefe3cda10b3b519cbb81 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Tue, 20 Oct 2026 22:28:15 +0300 Subject: [PATCH 12/12] fix(repo): look at the manifest between pack-delete batches; name the unreadable key; sharpen limits (#216) --- crates/repo/src/bucket_transport/compact.rs | 42 ++++++++++-- .../src/bucket_transport/compact/tests.rs | 68 +++++++++++++++++++ docs/THREAT-MODEL.md | 11 +-- docs/adr/0055-bucket-compaction.md | 22 ++++-- 4 files changed, 128 insertions(+), 15 deletions(-) diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs index afc4e1cc..ad4fdce3 100644 --- a/crates/repo/src/bucket_transport/compact.rs +++ b/crates/repo/src/bucket_transport/compact.rs @@ -21,6 +21,9 @@ use std::time::{Duration, SystemTime}; /// Replans after a lost manifest CAS or a concurrent compactor before giving up. const MAX_ATTEMPTS: u32 = 4; +/// Unreferenced packs deleted between two looks at the manifest. +const PACK_DELETE_BATCH: usize = 16; + /// Decoded bytes one merge pack may hold, as a multiple of `group_bytes` /// (which counts compressed bytes). const DECODED_FACTOR: u64 = 4; @@ -461,10 +464,29 @@ impl BucketTransport { .sum::(); continue; } - for (hash, _) in &del_packs { - // idx first: a half-deleted pack is then plain debris. - self.bucket.delete(&format!("packs/{hash}.idx"))?; - self.bucket.delete(&pack_key(hash))?; + // The delete loop can run long (one request per object), so look at + // the manifest again between batches: a push that lands meanwhile + // may have re-used one of these old packs byte for byte. + for (i, batch) in del_packs.chunks(PACK_DELETE_BATCH).enumerate() { + if i > 0 + && !matches!( + (&tag, self.bucket.get("manifest", tag.as_deref())?), + (Some(_), Fetched::Unchanged) | (None, Fetched::Absent) + ) + { + let left = del_packs.len() - i * PACK_DELETE_BATCH; + rep.packs_deleted -= left; + rep.notes.push(format!( + "the manifest moved during pack deletion: {left} unreferenced pack(s) \ + kept, rerun" + )); + return Ok(()); + } + for (hash, _) in batch { + // idx first: a half-deleted pack is then plain debris. + self.bucket.delete(&format!("packs/{hash}.idx"))?; + self.bucket.delete(&pack_key(hash))?; + } } return Ok(()); } @@ -490,7 +512,11 @@ impl BucketTransport { let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { continue; // deleted since the listing: no longer retained }; - named.extend(Checkpoint::decode(&capped("checkpoint", bytes)?)?.packs); + named.extend( + Checkpoint::decode(&capped("checkpoint", bytes)?) + .map_err(|e| Error::Wal(format!("{}: {e}", m.key)))? + .packs, + ); } } for (seq, m) in logs { @@ -498,7 +524,11 @@ impl BucketTransport { let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { continue; }; - named.extend(LogEntry::decode(&capped("log entry", bytes)?)?.packs); + named.extend( + LogEntry::decode(&capped("log entry", bytes)?) + .map_err(|e| Error::Wal(format!("{}: {e}", m.key)))? + .packs, + ); } } Ok(named) diff --git a/crates/repo/src/bucket_transport/compact/tests.rs b/crates/repo/src/bucket_transport/compact/tests.rs index e686af7e..5bf76520 100644 --- a/crates/repo/src/bucket_transport/compact/tests.rs +++ b/crates/repo/src/bucket_transport/compact/tests.rs @@ -846,3 +846,71 @@ fn a_first_push_that_died_before_the_manifest_leaves_orphans_the_sweep_ages_out( assert!(keys(&mem, "packs/").is_empty()); assert!(open(&mem).list_refs().unwrap().is_empty()); } + +#[test] +fn a_manifest_that_moves_during_pack_deletion_stops_the_deletes() { + // Wraps a bucket so the first pack delete is followed by a push by someone + // else: the sweep must notice at its next look and keep the rest. + struct PushOnFirstPackDelete { + inner: Shared, + fired: Mutex, + } + impl Bucket for PushOnFirstPackDelete { + fn get(&self, k: &str, t: Option<&str>) -> scl_objio::Result { + self.inner.get(k, t) + } + fn put_new(&self, k: &str, b: &[u8]) -> scl_objio::Result { + self.inner.put_new(k, b) + } + fn put_if_tag( + &self, + k: &str, + b: &[u8], + t: Option<&str>, + ) -> scl_objio::Result> { + self.inner.put_if_tag(k, b, t) + } + fn list(&self, p: &str) -> scl_objio::Result> { + self.inner.list(p) + } + fn list_meta(&self, p: &str) -> scl_objio::Result> { + self.inner.list_meta(p) + } + fn exists(&self, k: &str) -> scl_objio::Result { + self.inner.exists(k) + } + fn delete(&self, k: &str) -> scl_objio::Result<()> { + self.inner.delete(k)?; + let mut fired = self.fired.lock().unwrap(); + if k.starts_with("packs/") && !*fired { + *fired = true; + drop(fired); + push_commits(&open(&self.inner.0), "late", 1); + } + Ok(()) + } + } + let mem = Arc::new(MemBucket::default()); + let refs = push_commits(&open(&mem), "w", 70); + let t = BucketTransport::from_bucket(Box::new(PushOnFirstPackDelete { + inner: Shared(mem.clone()), + fired: Mutex::new(false), + })) + .unwrap(); + let rep = t.compact(&eager()).unwrap(); + assert!( + rep.notes + .iter() + .any(|n| n.contains("moved during pack deletion")), + "{rep:?}" + ); + assert!(rep.packs_deleted < 70, "{rep:?}"); + // some old packs were kept, the bucket is readable, and a rerun finishes + assert!(keys(&mem, "packs/").len() > 4); + let mut all = refs; + all.push(("late-0".to_string(), commit("late-0").0)); + assert_readable(&mem, &all); + open(&mem).compact(&eager()).unwrap(); + assert_eq!(keys(&mem, "packs/").len(), 2); + assert_readable(&mem, &all); +} diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index a30ee779..9c30e9e8 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -432,13 +432,14 @@ too, not just the src-control-side metadata. objects no current manifest chain names, and a writer who can already write to the bucket can do worse with raw access. What it does change for a hostile or mistaken writer: (a) a stalled push whose uploaded pack is older - than `--prune-expire` can lose it and fail on retry, or in a race narrower - than one bucket round trip commit an entry naming a deleted pack, which + than `--prune-expire` can lose it and fail on retry, or, if its pack is byte-identical to an old + orphan that a sweep is deleting at that moment, commit an entry naming a + deleted pack, which makes the remote unreadable until fixed (hence the 24h default, and leases as the real fix); (b) a hostile write that fills `log/` or - `checkpoints/` with junk is reclaimed once it falls behind the cutoff. - Compaction decodes the records it keeps strictly and stops deleting packs if - one is unreadable (fail closed). Use `--dry-run` first on a new bucket. + `checkpoints/` with junk is reclaimed once it falls behind the cutoff; junk + at a far-future seq never falls behind it, and an undecodable one stops pack + deletion (fail closed, the report names the key to remove by hand). Use `--dry-run` first on a new bucket. - **Sealed content stays end-to-end sealed, unchanged.** Protected-path ciphertext (ADR-0014/0043) and committed secrets travel through a bucket remote exactly as through any other transport: a bucket reader without the diff --git a/docs/adr/0055-bucket-compaction.md b/docs/adr/0055-bucket-compaction.md index 92462596..f2b7566c 100644 --- a/docs/adr/0055-bucket-compaction.md +++ b/docs/adr/0055-bucket-compaction.md @@ -67,7 +67,10 @@ CAS and must refresh. Old packs stay in place. After the CAS the superseded packs are named by no chain a new reader walks, but older checkpoints and log entries still name -them. They are removed by phase 2, a grace period later. +them. They are removed by phase 2 once the checkpoint that dropped them has +aged past `grace`, normally a later run. (A leftover checkpoint from a lost +CAS can set that cutoff earlier than the real one, which only costs a reader +that read the manifest moments before a restart; see the reader protocol.) **Phase 2, sweep (deletes, behind a cutoff).** All deletions are of objects that the current manifest chain does not reference (and, outside the residual @@ -127,7 +130,7 @@ scan, `refresh`), never in the middle of the bytes `get_pack` has already written. `grace` is therefore not what makes readers correct; it only makes restarts -rare. A reader older than `grace` is the only one that can be restarted. +rare; a reader straddling a sweep can be restarted. ### Writer protocol @@ -147,7 +150,8 @@ store offers no atomic "delete if the manifest is unchanged". Two defenses: **Residual limit (documented, not closed).** A writer that stalls longer than `grace` between uploading a pack and its CAS, or re-pushes byte-identical -content to an old orphan in the same instant a sweep selects it, can still +content to an old orphan while a sweep that already selected it is still +deleting (the manifest is looked at again every 16 packs), can still commit an entry naming a deleted pack. Closing that needs a lease or a conditional delete, which the `Bucket` trait lacks (ROADMAP "Leases"). The same staleness has a second, milder form: a writer paused longer than `grace` @@ -186,6 +190,16 @@ the post-compaction pack list (the leaf cache is keyed by pack hash and trimmed to live packs on refresh). `a_clone_after_compaction_still_gets_signatures_transcripts_and_tombstones` covers a clone and a fetch across two compactions with a signed drop in between. +### Rollout and permissions + +Upgrade pushers and `sc serve --store` instances before the first `sc remote +compact`: a client without the pre-commit pack check can still lose a pack to +the residual limit above. Pushers and readers need only get, put and head on +objects, but the "absent object" cases (restart on a swept object, the pre-commit +check) rely on a missing key answering 404. S3 answers 403 instead when the +caller lacks `s3:ListBucket`, so grant pushers and readers list as well if you +want the retry messages; without it they fail closed, with an opaque error. + ### Command surface `sc remote compact [--dry-run] [--prune-expire ]` (default @@ -205,7 +219,7 @@ operator action, not a side effect of an instance starting). ### objio -The `Bucket` trait gains two methods, implemented by `DirBucket` and +The `Bucket` trait gains three methods, implemented by `DirBucket` and `S3Bucket`: - `list_meta(prefix)`: keys with size and last-modified time, raw string