diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index c57a0672..b5a55dab 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -368,6 +368,22 @@ cargo run --bin sc -- remote add sc+s3:/// # same, on a # content stays ciphertext); `clone --filter` is # refused; signatures, transcripts and drop # tombstones travel like on any native remote (#229) +cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] [--allow-short-grace] + # bucket remotes only (#216, ADR-0057): merge small + # packs behind one manifest CAS, then delete log + # entries, checkpoints and packs nothing names that + # are older than --prune-expire (same flag and default + # as `sc gc`; keep it well above any push's duration). + # Needs bucket list + delete rights. --dry-run prints + # the plan and writes nothing. Safe beside live + # pushes/fetches within --prune-expire (a writer + # stalled longer than that between uploading a pack + # and committing it can lose the pack); run one at a time. A --prune-expire under 1h is refused + # unless --allow-short-grace (only with no pushes + # or fetches running). On a versioned S3 + # bucket deletes only add delete markers (add a + # noncurrent-version lifecycle rule). Relieves the + # 65536-pack checkpoint cap (not the ref cap) cargo run --bin sc -- serve --http --store sc+wal://|sc+s3:/// # serve a bucket remote over sc+http(s):// (P36c): # all content lives in the bucket (instance is diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 35efef20..8f66a77c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1192,8 +1192,10 @@ the checkpoint and walks only the tail. P36c adds `sc serve --store `: an identity and scratch space while all served content lives in the bucket, so instances are disposable. Public content sits in the bucket as plaintext at rest (the bucket ACL is the perimeter; sealed content stays ciphertext), and -`sc clone --filter` is refused against a bucket remote. Bucket compaction and gc -are not built (open issue #216). A bucket keeps no signature or transcript +`sc clone --filter` is refused against a bucket remote. `sc remote compact` (ADR-0057, #216) merges small packs behind one manifest +CAS, then deletes log entries, checkpoints and unreferenced packs behind a cutoff +that has aged past `--prune-expire`; readers that find a named object gone +re-read the manifest and restart, and there is no format change. A bucket keeps no signature or transcript index, so its `get_pack` rebuilds the over-send from the packs the manifest chain names; signatures, transcripts and signed drop tombstones therefore travel through a bucket like any other remote, but each fresh fetch or clone diff --git a/CLAUDE.md b/CLAUDE.md index ae7ae088..aa803119 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,7 +151,7 @@ the code, those win. | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) | | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | -| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change) | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change); `sc remote compact [--dry-run] [--prune-expire 24h]` (#216) merges small packs behind one manifest CAS, then deletes log entries, checkpoints and unreferenced packs behind an aged-checkpoint cutoff, readers restart on a moved manifest, no format or wire change | [0046](docs/adr/0046-wal-bucket-remotes.md), [0057](docs/adr/0057-bucket-compaction.md) | | P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat :` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 2 GiB)` (a constant since #213, independent of the object cap) and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) | | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | @@ -206,6 +206,12 @@ transport-adjacent. The rest, imperatively: - **Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused. +- **`sc remote compact` deletes bucket objects** (ADR-0057): it needs bucket + list and delete rights, which pushers and `sc serve --store` instances should + not hold. It advances the manifest only through the normal CAS (one no-op + entry plus a checkpoint) and never deletes what the current chain names. Keep `--prune-expire` well above any push's duration (default + 24h): a writer stalled longer than that between uploading a pack and its + commit can still lose the pack (the leases follow-on closes it). - **`sc serve --store` still requires a local serve home with `.sc/`** — tokens, TLS identity, and pack spills live there; the bucket holds all served content. @@ -323,7 +329,7 @@ transport-adjacent. The rest, imperatively: Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of truth — do not re-list items here); entries that name an issue mirror the open GitHub issues. Standing entries: `sc ws` workspace refresh (#215), -a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), bucket compaction and gc (#216), the non-unix +a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), the non-unix `.sc/local-key` ACL (#214), and the open sparse, private-branch and abort edge cases (#241, #246, #247, #248). diff --git a/README.md b/README.md index 01ed372b..b38fc8ac 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ back to Git commits, and reaches hosted Git (e.g. GitHub) through a spawned system-git mirror bridge. See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design, -[docs/adr/](docs/adr/) for the architecture decision records (0001-0054) behind +[docs/adr/](docs/adr/) for the architecture decision records (0001-0057) behind every phase, and [CLAUDE.md](CLAUDE.md) for project conventions and the per-phase capability map. [CONTRIBUTING.md](CONTRIBUTING.md) covers the development workflow. @@ -170,7 +170,7 @@ and Git (including hosted GitHub) network transports, plus private branches sealed to a recipient set until publish and a read-only Tauri desktop browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds 28 independent end-to-end proofs, one per major capability. Remaining -follow-ons (transcript drop delivery gaps, bucket compaction and gc, `sc ws` +follow-ons (transcript drop delivery gaps, `sc ws` workspace refresh, a limit-aware fetch and an object-count clone limit, and similar hardening items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open GitHub issues rather than being open MVP scope. diff --git a/ROADMAP.md b/ROADMAP.md index af478562..00d1c370 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -577,7 +577,9 @@ across every phase. Public content is plaintext at rest (the bucket ACL is the perimeter); `clone --filter` is refused. `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format - change). (ADR-0046.) + change). `sc remote compact` (#216) merges small packs and deletes the log + entries, checkpoints and unreferenced packs behind an aged cutoff; readers + restart on a moved manifest; no format change. (ADR-0046, ADR-0057.) - **Phase 37a — Want-by-id fetch.** A batched `GetObjects` verb (wire v5), an opt-in `.sc/promisor-lazy` marker, and `sc cat :`, which fills only the absent trees and the blob on its path. `Store::get` stays network-free. @@ -672,7 +674,7 @@ under Deferred below. | **P33 — Randomized protected encryption** | Close equality confirmation for new protected content | fresh DEK + nonce per seal, dual-read/randomized-write, keyed local stat cache, rewrap upgrade | [0043](docs/adr/0043-randomized-protected-encryption.md) | | **P34 — Private branches** | Keep an entire branch opaque before publication | sealed object closure behind a manifest; O(1) grant, KEK-rotating revoke, scanner-gated publish | [0044](docs/adr/0044-per-branch-access-control.md) | | **P35 — Native desktop browser** | Prove native `.sc` objects can power a polished visual tool | Tauri app opens a repository and browses refs, snapshot DAG/provenance, public trees, and first-parent diffs while locked/private states remain safe | [0045](docs/adr/0045-native-desktop-read-model.md) | -| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add sc+wal://` or `sc+s3:///`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store ` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md) | +| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add sc+wal://` or `sc+s3:///`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store ` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); `sc remote compact` prunes and merges (#216); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md), [0057](docs/adr/0057-bucket-compaction.md) | | **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) | | **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) | | **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) | @@ -897,22 +899,12 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in fetch-time freshness attestation so a hostile remote can't re-serve a stale pre-revoke manifest — the same gittuf-shaped signed-ref effort the P22 provenance boundary already defers. -- **Bucket compaction/gc (#216, P36a follow-on).** The WAL log (`log/` keys, - each a hand-rolled versioned binary `walfmt` entry, not protobuf) grows - unboundedly with no compaction — a cold-start reader now stops at the - latest checkpoint (P36b) instead of walking to `0`, but no log entry, - checkpoint, or superseded pack is ever actually removed from the bucket. - Deferred until a safe pruning cutoff (e.g. "no reader can still need - anything before checkpoint N") is designed. Compaction is also what - relieves `walfmt::MAX_LIST` (65536): a checkpoint fold with more refs or - packs than that cap is skipped outright (`maybe_fold_checkpoint`'s guard, - P36b review) rather than writing an object `Checkpoint::decode` would then - refuse to read back, so a remote whose live ref/pack count grows past the - cap loses folding entirely until compaction can retire entries below it. - **Leases (P36a follow-on).** The only cross-writer coordination today is the manifest's compare-and-swap; there is no lease/TTL primitive for operations that need to hold exclusive intent across more than one bucket - round-trip (e.g. a long-running compaction). Deferred. + round-trip (e.g. a long-running compaction). Deferred. It would also close + the one residual hazard ADR-0057 documents: a writer stalled longer than + `sc remote compact --prune-expire` between uploading a pack and committing it. - **Derived-data layer (Scion-inspired, after P37).** Blame, path history, commit-graph skip lists and copy-detection computed by one generic recursive mechanism over snapshots, stored outside the CAS and rebuildable diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index faf9e224..dad2688c 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -833,6 +833,23 @@ enum RemoteOp { }, /// List configured remotes. List, + /// Compact a bucket remote (`sc+wal://`, `sc+s3://`): merge small packs, + /// then delete log entries, checkpoints and packs no reader or writer can + /// still need (ADR-0057). Needs bucket list and delete rights. + Compact { + name: String, + /// Plan and report; write and delete nothing. + #[arg(long)] + dry_run: bool, + /// Delete only what has been unreferenced for at least this long + /// (e.g. 24h, 7d). Keep it well above any push's duration. + #[arg(long, default_value = "24h")] + prune_expire: String, + /// Allow a --prune-expire under 1h. Only safe while nothing pushes to + /// or fetches from the bucket: a push in flight can lose its pack. + #[arg(long)] + allow_short_grace: bool, + }, } /// Top-level `sc serve` subcommand namespace, distinct from the plain @@ -4159,6 +4176,36 @@ fn run_remote(op: RemoteOp) -> Result<()> { println!("added remote {name} -> {url}"); } } + RemoteOp::Compact { + name, + dry_run, + prune_expire, + allow_short_grace, + } => { + let grace = parse_duration(&prune_expire)?; + if grace < std::time::Duration::from_secs(3600) && !allow_short_grace { + anyhow::bail!( + "--prune-expire under 1h can delete a pack an in-flight push has uploaded but \ + not committed; pass --allow-short-grace only if nothing is pushing or fetching" + ); + } + let url = repo + .remotes()? + .into_iter() + .find(|(n, _)| *n == name) + .map(|(_, u)| u) + .ok_or_else(|| anyhow::anyhow!("no such remote: {name}"))?; + if !(url.starts_with("sc+wal://") || url.starts_with("sc+s3://")) { + anyhow::bail!("remote '{name}' is not a bucket remote ({url}); nothing to compact"); + } + let transport = scl_repo::BucketTransport::open(&url)?; + let rep = transport.compact(&scl_repo::CompactOptions { + prune_expire: grace, + dry_run, + ..Default::default() + })?; + print_compact_report(&name, &url, &rep); + } RemoteOp::List => { let cfg = scl_repo::RemoteConfig::load(repo.layout())?; for (name, url) in repo.remotes()? { @@ -4173,6 +4220,41 @@ fn run_remote(op: RemoteOp) -> Result<()> { Ok(()) } +/// Human-readable summary of a bucket compaction (`dry_run` reads "would"). +fn print_compact_report(name: &str, url: &str, rep: &scl_repo::CompactReport) { + let (merge, delete) = if rep.dry_run { + ("would merge", "would delete") + } else { + ("merged", "deleted") + }; + println!( + "compact {name} ({url}){}", + if rep.dry_run { " [dry run]" } else { "" } + ); + println!( + " {merge} {} pack(s) into {}", + rep.packs_merged, rep.packs_created + ); + println!( + " {delete} {} log entr{}, {} checkpoint(s), {} pack(s) ({} bytes), cutoff checkpoint {}", + rep.logs_deleted, + if rep.logs_deleted == 1 { "y" } else { "ies" }, + rep.checkpoints_deleted, + rep.packs_deleted, + rep.bytes_deleted, + rep.cutoff + ); + if rep.packs_held_back > 0 { + println!( + " kept {} unreferenced pack(s) younger than the grace window", + rep.packs_held_back + ); + } + for note in &rep.notes { + println!(" note: {note}"); + } +} + /// Fetch from `remote`, dispatching on its configured kind: a `git`-kind /// remote gets a full-history import into a remote-tracking ref (see /// `run_fetch_git`); an `sc`-kind remote (or unconfigured, defaulting to `sc`) diff --git a/crates/cli/tests/bucket_remote.rs b/crates/cli/tests/bucket_remote.rs index 4a984bab..8ed4bac1 100644 --- a/crates/cli/tests/bucket_remote.rs +++ b/crates/cli/tests/bucket_remote.rs @@ -86,6 +86,93 @@ fn bucket_clone_push_fetch_round_trip_and_url_validation() { } } +/// #216: `sc remote compact` merges and sweeps a bucket remote, `--dry-run` +/// writes nothing, a default grace window deletes nothing young, and a +/// non-bucket remote is refused. A clone and a fetch still work afterwards. +#[test] +fn remote_compact_merges_and_sweeps_a_bucket_and_clones_still_work() { + let a = tmp("compact-a"); + let bucket = tmp("compact-bucket"); + let local = tmp("compact-local"); + let url = format!("sc+wal://{}", bucket.display()); + assert!(sc(&a, &["init"]).status.success()); + assert!(sc(&a, &["remote", "add", "origin", &url]).status.success()); + assert!(sc(&a, &["remote", "add", "plain", local.to_str().unwrap()]) + .status + .success()); + for i in 0..3 { + std::fs::write(a.join(format!("f{i}.txt")), format!("content {i}")).unwrap(); + assert!(sc(&a, &["commit", "-m", &format!("c{i}")]).status.success()); + assert!(sc(&a, &["push", "origin"]).status.success()); + } + let count = |sub: &str| std::fs::read_dir(bucket.join(sub)).unwrap().count(); + let (logs, packs) = (count("log"), count("packs")); + assert_eq!((logs, packs), (3, 6), "3 pushes: 3 entries, 3 packs + idx"); + + // dry run: reports the plan, changes nothing + let out = sc(&a, &["remote", "compact", "origin", "--dry-run"]); + assert!(out.status.success(), "{out:?}"); + let text = String::from_utf8_lossy(&out.stdout); + assert!( + text.contains("[dry run]") && text.contains("would merge 3 pack(s) into 1"), + "{text}" + ); + assert_eq!((count("log"), count("packs")), (logs, packs)); + + // default 24h grace: merges, but deletes nothing young + let out = sc(&a, &["remote", "compact", "origin"]); + assert!(out.status.success(), "{out:?}"); + let text = String::from_utf8_lossy(&out.stdout); + assert!( + text.contains("merged 3 pack(s) into 1") && text.contains("kept 3 unreferenced"), + "{text}" + ); + assert_eq!( + count("packs"), + 8, + "3 old packs kept for the grace window + the merged one" + ); + + // zero grace: sweeps the old packs and the log + let refused = sc(&a, &["remote", "compact", "origin", "--prune-expire", "0s"]); + assert!( + !refused.status.success(), + "a zero grace needs an explicit opt-in" + ); + let out = sc( + &a, + &[ + "remote", + "compact", + "origin", + "--prune-expire", + "0s", + "--allow-short-grace", + ], + ); + assert!(out.status.success(), "{out:?}"); + assert_eq!((count("log"), count("packs")), (0, 2)); + + // refuses a non-bucket remote and an unknown one + assert!(!sc(&a, &["remote", "compact", "plain"]).status.success()); + assert!(!sc(&a, &["remote", "compact", "nope"]).status.success()); + + // the compacted bucket still clones and fetches + let parent = tmp("compact-parent"); + let b = parent.join("b"); + let out = sc(&parent, &["clone", &url, b.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + assert_eq!(std::fs::read(b.join("f2.txt")).unwrap(), b"content 2"); + std::fs::write(b.join("g.txt"), b"after compact").unwrap(); + assert!(sc(&b, &["commit", "-m", "c3"]).status.success()); + assert!(sc(&b, &["push", "origin"]).status.success()); + assert!(sc(&a, &["fetch", "origin"]).status.success()); + + for d in [&a, &bucket, &local, &parent] { + let _ = std::fs::remove_dir_all(d); + } +} + /// #229: a signed snapshot pushed to a bucket keeps its signature on a fresh /// clone, so `sc verify --require` passes there once the signer is trusted /// (trust is local config, written into the clone). diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs index 6ea3f288..8957c64c 100644 --- a/crates/objio/src/dir.rs +++ b/crates/objio/src/dir.rs @@ -1,6 +1,6 @@ //! Local-directory `Bucket` — the test/demo backend behind `sc+wal://`. -use crate::{validate_key, Bucket, Error, Fetched, Result}; +use crate::{validate_key, Bucket, Error, Fetched, ObjectMeta, Result}; use std::path::{Path, PathBuf}; /// A local filesystem bucket implementation using a directory as the backing @@ -116,7 +116,16 @@ impl Bucket for DirBucket { } fn list(&self, prefix: &str) -> Result> { - fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { + Ok(self.list_meta(prefix)?.into_iter().map(|m| m.key).collect()) + } + + fn list_meta(&self, prefix: &str) -> Result> { + fn walk( + dir: &Path, + root: &Path, + prefix: &str, + out: &mut Vec, + ) -> std::io::Result<()> { let rd = match std::fs::read_dir(dir) { Ok(rd) => rd, Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), @@ -131,24 +140,61 @@ impl Bucket for DirBucket { }) { continue; } - if p.is_dir() { - walk(&p, root, out)?; + let rel = p + .strip_prefix(root) + .unwrap() + .to_string_lossy() + .replace('\\', "/"); + // Skip files outside the prefix before paying for a stat. + if entry.file_type()?.is_file() && !rel.starts_with(prefix) { + continue; + } + // A concurrent delete/rename can remove the entry between the + // readdir and the stat; a key that is gone is just not listed. + let md = match std::fs::metadata(&p) { + Ok(md) => md, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, + Err(e) => return Err(e), + }; + if md.is_dir() { + walk(&p, root, prefix, out)?; } else { - out.push( - p.strip_prefix(root) - .unwrap() - .to_string_lossy() - .replace('\\', "/"), - ); + out.push(ObjectMeta { + key: rel, + size: md.len(), + modified: md.modified()?, + }); } } Ok(()) } validate_key(prefix.trim_end_matches('/'))?; let mut out = Vec::new(); - walk(&self.root, &self.root, &mut out)?; - out.retain(|k| k.starts_with(prefix)); - out.sort(); + // Keys live under their directory, so start the walk at the prefix's + // directory part instead of scanning the whole bucket. + let start = match prefix.rsplit_once('/') { + Some((dir, _)) => self.root.join(dir), + None => self.root.clone(), + }; + walk(&start, &self.root, prefix, &mut out)?; + out.retain(|m| m.key.starts_with(prefix)); + out.sort_by(|a, b| a.key.cmp(&b.key)); Ok(out) } + + fn exists(&self, key: &str) -> Result { + match std::fs::metadata(self.key_path(key)?) { + Ok(md) => Ok(md.is_file()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(e) => Err(e.into()), + } + } + + fn delete(&self, key: &str) -> Result<()> { + match std::fs::remove_file(self.key_path(key)?) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), + } + } } diff --git a/crates/objio/src/lib.rs b/crates/objio/src/lib.rs index 9900051b..e9dfefcf 100644 --- a/crates/objio/src/lib.rs +++ b/crates/objio/src/lib.rs @@ -34,8 +34,18 @@ pub enum Fetched { New { bytes: Vec, tag: String }, } +/// A listed key with the metadata compaction needs (ADR-0057). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ObjectMeta { + pub key: String, + pub size: u64, + /// Last-modified time as the bucket reports it (S3 `LastModified`, file mtime). + pub modified: std::time::SystemTime, +} + /// A flat key/value object store with the three primitives the WAL needs: -/// conditional read, create-if-absent, and compare-and-swap overwrite. +/// conditional read, create-if-absent, and compare-and-swap overwrite; plus +/// the metadata listing and delete that compaction (ADR-0057) adds. pub trait Bucket: Send { /// Conditional read. `cached_tag` matching the current value returns /// [`Fetched::Unchanged`] without transferring bytes. @@ -54,6 +64,14 @@ pub trait Bucket: Send { ) -> Result>; /// Keys under `prefix`, sorted. fn list(&self, prefix: &str) -> Result>; + /// Keys whose name starts with the raw string `prefix` (not a directory + /// scope, so `packs/ab12.` matches `packs/ab12.pack`), with size and + /// last-modified time, sorted by key. + fn list_meta(&self, prefix: &str) -> Result>; + /// Whether `key` exists (a HEAD: needs read rights only, no listing). + fn exists(&self, key: &str) -> Result; + /// Remove `key`. Idempotent: an absent key is success. + fn delete(&self, key: &str) -> Result<()>; } /// Reject traversal and absolute keys before any backend touches them. @@ -117,6 +135,34 @@ pub fn contract_suite(b: &dyn Bucket) { vec!["log/1".to_string(), "log/2".to_string()] ); assert_eq!(b.list("packs/").unwrap(), Vec::::new()); + // list_meta: raw string prefix, sorted, with size and a sane mtime + assert!(b.put_new("packs/ab12.pack", b"12345").unwrap()); + assert!(b.put_new("packs/ab12.idx", b"1").unwrap()); + assert!(b.put_new("packs/ab34.pack", b"x").unwrap()); + let metas = b.list_meta("packs/ab12.").unwrap(); + assert_eq!( + metas.iter().map(|m| m.key.as_str()).collect::>(), + vec!["packs/ab12.idx", "packs/ab12.pack"] + ); + assert_eq!(metas[1].size, 5); + let age = std::time::SystemTime::now() + .duration_since(metas[1].modified) + .unwrap_or_default(); + assert!(age < std::time::Duration::from_secs(3600)); + assert_eq!(b.list_meta("packs/").unwrap().len(), 3); + assert!(b.list_meta("nothing/").unwrap().is_empty()); + assert!(b.exists("packs/ab12.pack").unwrap()); + assert!(!b.exists("packs/zz.pack").unwrap()); + // delete: removes, is idempotent, touches only its key + b.delete("packs/ab12.pack").unwrap(); + b.delete("packs/ab12.pack").unwrap(); + assert!(matches!( + b.get("packs/ab12.pack", None).unwrap(), + Fetched::Absent + )); + assert_eq!(b.list_meta("packs/").unwrap().len(), 2); + // a deleted key can be created again + assert!(b.put_new("packs/ab12.pack", b"again").unwrap()); } #[cfg(test)] diff --git a/crates/objio/src/s3.rs b/crates/objio/src/s3.rs index 2eb9d36c..68f8683e 100644 --- a/crates/objio/src/s3.rs +++ b/crates/objio/src/s3.rs @@ -1,6 +1,6 @@ //! S3-compatible `Bucket` backend (AWS, MinIO, R2 via AWS_ENDPOINT_URL_S3). -use crate::{validate_key, Bucket, Error, Fetched, Result}; +use crate::{validate_key, Bucket, Error, Fetched, ObjectMeta, Result}; /// A `Bucket` backed by an S3-compatible object store. Credentials, region, /// and endpoint come entirely from the SDK's standard chain (`AWS_*` env, @@ -173,6 +173,85 @@ impl Bucket for S3Bucket { out.sort(); Ok(out) } + + fn list_meta(&self, prefix: &str) -> Result> { + // Raw string prefix (no directory scoping), unlike `list`. + validate_key(prefix.trim_end_matches('/'))?; + let full = if self.prefix.is_empty() { + prefix.to_string() + } else { + format!("{}/{prefix}", self.prefix) + }; + let strip = if self.prefix.is_empty() { + String::new() + } else { + format!("{}/", self.prefix) + }; + let mut out = Vec::new(); + let mut cont: Option = None; + loop { + let mut req = self + .client + .list_objects_v2() + .bucket(&self.bucket) + .prefix(&full); + if let Some(c) = &cont { + req = req.continuation_token(c); + } + let resp = self + .rt + .block_on(req.send()) + .map_err(|e| Error::Backend(format!("s3 list: {e}")))?; + for obj in resp.contents() { + let Some(k) = obj.key() else { continue }; + let modified = obj + .last_modified() + .and_then(|t| std::time::SystemTime::try_from(*t).ok()) + .ok_or_else(|| Error::Backend(format!("s3 list: no LastModified for {k}")))?; + out.push(ObjectMeta { + key: k.strip_prefix(&strip).unwrap_or(k).to_string(), + size: obj.size().unwrap_or(0).max(0) as u64, + modified, + }); + } + match resp.next_continuation_token() { + Some(c) => cont = Some(c.to_string()), + None => break, + } + } + out.sort_by(|a, b| a.key.cmp(&b.key)); + Ok(out) + } + + fn exists(&self, key: &str) -> Result { + let k = self.full_key(key)?; + match self.rt.block_on( + self.client + .head_object() + .bucket(&self.bucket) + .key(&k) + .send(), + ) { + Ok(_) => Ok(true), + Err(e) if http_status(&e) == Some(404) => Ok(false), + Err(e) => Err(Error::Backend(format!("s3 head {k}: {e}"))), + } + } + + fn delete(&self, key: &str) -> Result<()> { + let k = self.full_key(key)?; + // S3 DeleteObject succeeds on an absent key, which is the contract. + self.rt + .block_on( + self.client + .delete_object() + .bucket(&self.bucket) + .key(&k) + .send(), + ) + .map(|_| ()) + .map_err(|e| Error::Backend(format!("s3 delete {k}: {e}"))) + } } #[cfg(test)] diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index a913311b..431e0d8e 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -10,10 +10,18 @@ use crate::walfmt::{ use scl_core::pack::{parse_index, read_object_at_bounded, IndexEntry, PackWriter}; use scl_core::{Object, ObjectId}; use scl_objio::{Bucket, Fetched}; -use std::cell::RefCell; +use std::cell::{Cell, RefCell}; use std::collections::BTreeMap; use std::io::Write; +mod compact; +pub use compact::{CompactOptions, CompactReport}; + +/// How many times a read restarts from a fresh manifest after an object the +/// view names has vanished (ADR-0057). Each restart needs the manifest to have +/// moved, so a handful is plenty: one per compaction that lands mid-read. +const MAX_RESTARTS: u32 = 8; + /// Reconstructed state of the WAL at one manifest tag: the manifest itself, /// the flattened branch tips after replaying the parent chain oldest-first, /// and a lookup from object id to the pack that holds it. @@ -71,6 +79,14 @@ pub struct BucketTransport { /// it spares a long-lived `serve --store` session from re-reading every /// pack body on each `get_pack`. leaf_cache: RefCell>>, + /// Set whenever a read finds an object the manifest chain names absent + /// (ADR-0057): the signal for `refresh`/`restarting` to re-read the + /// manifest and tell a compaction that moved on (restart) from a corrupt + /// bucket (fail closed). + saw_absent: Cell, + /// Tag of the manifest the in-progress `refresh_once` is building from, so + /// a failed attempt can say which manifest it failed on. + attempt_tag: RefCell>, } /// A leaf object a pack holds that no tree or snapshot references, so the @@ -218,6 +234,8 @@ impl BucketTransport { staged: RefCell::new(Vec::new()), pending_packs: RefCell::new(Vec::new()), leaf_cache: RefCell::new(BTreeMap::new()), + saw_absent: Cell::new(false), + attempt_tag: RefCell::new(None), }; t.refresh()?; Ok(t) @@ -228,7 +246,59 @@ impl BucketTransport { /// (P36b) — seeded from that checkpoint's own `refs`/`packs` rather than /// replayed from `0` — then folding the tail entries on top (seq numbers /// are claims; the chain is the truth — off-chain entries are garbage). + /// + /// A compaction (ADR-0057) can delete log entries, checkpoints or packs + /// between reading the manifest and reading what it names. When that + /// happens the manifest has necessarily moved on, so the read restarts from + /// the new one; if the same manifest fails twice with a key absent, the + /// bucket is corrupt and the error stands. fn refresh(&self) -> Result<()> { + let mut failed_on: Option = None; + for _ in 0..MAX_RESTARTS { + self.saw_absent.set(false); + *self.attempt_tag.borrow_mut() = None; + match self.refresh_once() { + Err(e) if self.saw_absent.get() => { + let tag = self.attempt_tag.borrow_mut().take(); + if tag.is_none() || tag == failed_on { + return Err(e); + } + failed_on = tag; + } + r => return r, + } + } + self.refresh_once() + } + + /// Record that a key the chain names is absent and build the error for it. + fn gone(&self, msg: String) -> Error { + self.saw_absent.set(true); + Error::Wal(msg) + } + + /// Run a read against the current view; if it finds a named object gone + /// (a compaction deleted it), re-read the manifest and run it again. An unchanged manifest means the absence is real: fail closed. + fn restarting(&self, mut f: impl FnMut() -> Result) -> Result { + for _ in 0..MAX_RESTARTS { + self.saw_absent.set(false); + let before = self.view.borrow().as_ref().map(|v| v.tag.clone()); + match f() { + Err(e) if self.saw_absent.get() => { + // A conditional read: `Unchanged` keeps the view and its tag. + self.refresh()?; + let after = self.view.borrow().as_ref().map(|v| v.tag.clone()); + if before == after { + return Err(e); + } + } + r => return r, + } + } + f() + } + + fn refresh_once(&self) -> Result<()> { let cached_tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); match self.bucket.get("manifest", cached_tag.as_deref())? { Fetched::Unchanged => Ok(()), @@ -237,6 +307,7 @@ impl BucketTransport { Ok(()) } Fetched::New { bytes, tag } => { + *self.attempt_tag.borrow_mut() = Some(tag.clone()); let bytes = capped("manifest", bytes)?; let manifest = Manifest::decode(&bytes)?; let stop = manifest.checkpoint_seq; @@ -247,7 +318,7 @@ impl BucketTransport { let ck_bytes = match self.bucket.get(&checkpoint_key(stop), None)? { Fetched::New { bytes, .. } => capped("checkpoint", bytes)?, _ => { - return Err(Error::Wal(format!( + return Err(self.gone(format!( "checkpoint {stop} referenced by manifest but absent" ))) } @@ -277,9 +348,9 @@ impl BucketTransport { ))); } let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { - return Err(Error::Wal(format!( - "log entry {seq} referenced by chain but absent" - ))); + return Err( + self.gone(format!("log entry {seq} referenced by chain but absent")) + ); }; let e = LogEntry::decode(&capped("log entry", bytes)?)?; if e.seq != seq { @@ -312,13 +383,18 @@ impl BucketTransport { let mut index = BTreeMap::new(); for hash in &packs { let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but idx absent"))); + return Err(self.gone(format!("pack {hash} on chain but idx absent"))); }; let bytes = capped("pack idx", bytes)?; for IndexEntry { id, offset, length } in parse_index(&bytes)? { index.insert(id, (hash.clone(), offset, length)); } } + // Packs a compaction retired will never be asked about again. + { + let live: std::collections::HashSet<&String> = packs.iter().collect(); + self.leaf_cache.borrow_mut().retain(|h, _| live.contains(h)); + } *self.view.borrow_mut() = Some(WalView { tag, manifest, @@ -334,6 +410,10 @@ impl BucketTransport { /// Canonical `encode()` bytes of one object, fetched via its pack /// (downloaded once per distinct pack hash and cached for walk locality). fn object_bytes(&self, id: &ObjectId) -> Result> { + self.restarting(|| self.object_bytes_once(id)) + } + + fn object_bytes_once(&self, id: &ObjectId) -> Result> { let (hash, offset, _len) = { let view = self.view.borrow(); let view = view @@ -347,7 +427,7 @@ impl BucketTransport { let mut cache = self.pack_cache.borrow_mut(); if cache.as_ref().map(|(h, _)| h.as_str()) != Some(hash.as_str()) { let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(&hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + return Err(self.gone(format!("pack {hash} on chain but body absent"))); }; *cache = Some((hash.clone(), bytes)); } @@ -366,7 +446,7 @@ impl BucketTransport { return Ok(hit.clone()); } let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { - return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + return Err(self.gone(format!("pack {hash} on chain but body absent"))); }; let mut leaves = Vec::new(); scl_core::pack::parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { @@ -400,6 +480,13 @@ impl BucketTransport { /// An unsigned tombstone is withheld, as on `LocalTransport`. Whether a /// tombstone is honored is the receiver's call (`index_incoming`). fn leaf_ids_for(&self, snaps: &std::collections::BTreeSet) -> Result> { + self.restarting(|| self.leaf_ids_for_once(snaps)) + } + + fn leaf_ids_for_once( + &self, + snaps: &std::collections::BTreeSet, + ) -> Result> { let packs = match self.view.borrow().as_ref() { Some(v) => v.packs.clone(), None => return Ok(Vec::new()), @@ -474,6 +561,25 @@ impl BucketTransport { } } + /// Fail if a pack this push is about to commit has vanished from the bucket + /// (a compaction swept it as an unreferenced orphan after an unusually long + /// gap, ADR-0057). Committing a log entry that names a deleted pack would + /// corrupt the remote; failing lets the caller retry the push, which + /// re-uploads. Two HEADs per pending pack (read rights only: pushers need + /// no list or delete rights). + /// Called right before the manifest CAS. + fn ensure_packs_present(&self, hashes: &[String]) -> Result<()> { + for hash in hashes { + if !(self.bucket.exists(&pack_key(hash))? && self.bucket.exists(&idx_key(hash))?) { + return Err(Error::Remote(format!( + "pack {hash} was removed from the bucket before this push committed \ + (bucket compaction swept it); retry the push" + ))); + } + } + Ok(()) + } + /// Opportunistic, coordinator-free checkpoint fold. Called after a /// successful commit; every failure path is deliberately non-fatal — /// the checkpoint is derived data any reader can rebuild from the log, @@ -524,9 +630,8 @@ impl BucketTransport { // best-effort by design (see the doc above) — and let every // reader keep doing the slower but correct full-tail walk back // to the last checkpoint that DID fit. Durable relief is - // compaction (ROADMAP "Bucket compaction/gc"), which can retire - // refs/packs instead of letting them accumulate into ever-larger - // checkpoints forever. + // `compact` (ADR-0057), which merges small packs so the pack + // axis fits again. The ref axis is live state and is not relieved. return Ok(()); } let ck = Checkpoint { @@ -697,7 +802,7 @@ impl Transport for BucketTransport { // this function — success, failure, or the loop below — simply lets // `pending` go out of scope; the packs themselves stay uploaded in // the bucket regardless (content-addressed, inert orphan garbage if - // never referenced — correct per spec, compaction is a later phase). + // never referenced — correct per spec; `compact` sweeps it later, ADR-0057). let pending: Vec = std::mem::take(&mut *self.pending_packs.borrow_mut()); const MAX_CAS_RETRIES: u32 = 16; for _ in 0..MAX_CAS_RETRIES { @@ -756,6 +861,9 @@ impl Transport for BucketTransport { checkpoint_seq, head_branch, }; + // Last look before the commit point, to keep the window in which a + // concurrent sweep could still remove a pack as small as possible. + self.ensure_packs_present(&pending)?; if self .bucket .put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? @@ -1752,6 +1860,15 @@ mod tests { fn list(&self, prefix: &str) -> scl_objio::Result> { self.0.list(prefix) } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + self.0.list_meta(prefix) + } + fn exists(&self, key: &str) -> scl_objio::Result { + self.0.exists(key) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.0.delete(key) + } } #[test] @@ -1933,6 +2050,68 @@ mod tests { assert!(!base.exists()); } + /// #216 x #229: compaction merges the packs the sender seam scans for + /// signatures, transcripts and tombstones, so a clone after it must still + /// be served all three. + #[test] + fn a_clone_after_compaction_still_gets_signatures_transcripts_and_tombstones() { + let (a, url, snap, tid, identity, base) = signed_bucket_origin("compact"); + for (i, f) in ["f2", "f3"].iter().enumerate() { + std::fs::write(base.join("a").join(f), format!("more {i}")).unwrap(); + a.commit("t", &format!("c{}", i + 2)).unwrap(); + a.push("origin").unwrap(); + } + let compact = || { + BucketTransport::open(&url) + .unwrap() + .compact(&crate::bucket_transport::CompactOptions { + prune_expire: std::time::Duration::ZERO, + ..Default::default() + }) + .unwrap() + }; + let rep = compact(); + assert!(rep.packs_merged >= 3 && rep.packs_deleted >= 3, "{rep:?}"); + let bucket = base.join("bucket"); + let count = |sub: &str| std::fs::read_dir(bucket.join(sub)).unwrap().count(); + assert_eq!(count("packs"), 2, "merged into one pack + idx"); + + let b = crate::repo::Repo::clone_url(&url, base.join("b")).unwrap(); + assert_eq!( + b.sig_status(&snap, &trust_map(&identity)).unwrap(), + crate::signatures::SigStatus::Trusted("alice".to_string()) + ); + assert!(b + .transcripts_for(&snap) + .unwrap() + .iter() + .any(|(t, _)| *t == tid)); + assert_eq!(b.signatures_for(&tid).unwrap().len(), 1); + + // A signed drop pushed on top of the compacted bucket, then compacted again. + crate::transcripts::tests::trust_signer(&a, &identity); + a.drop_transcript(&tid.to_hex(), Some(&identity)).unwrap(); + std::fs::write(base.join("a/f4"), b"four").unwrap(); + a.commit("t", "c5").unwrap(); + a.push("origin").unwrap(); + compact(); + let tomb = ObjectId::of( + &Object::Transcript(scl_core::Transcript::drop_tombstone(snap, &tid)).encode(), + ); + let c = crate::repo::Repo::clone_url(&url, base.join("c")).unwrap(); + assert!( + c.store_arc().lock().unwrap().contains(&tomb), + "the tombstone survives compaction and is served" + ); + crate::transcripts::tests::trust_signer(&b, &identity); + b.fetch("origin").unwrap(); + assert!(b.transcripts_for(&snap).unwrap().is_empty()); + + drop((a, b, c)); + std::fs::remove_dir_all(&base).unwrap(); + assert!(!base.exists()); + } + #[test] fn fetch_from_a_bucket_delivers_a_later_transcript_and_signature() { let (a, url, snap, tid, identity, base) = signed_bucket_origin("fetch"); diff --git a/crates/repo/src/bucket_transport/compact.rs b/crates/repo/src/bucket_transport/compact.rs new file mode 100644 index 00000000..4771d31a --- /dev/null +++ b/crates/repo/src/bucket_transport/compact.rs @@ -0,0 +1,539 @@ +//! Bucket remote compaction and gc (#216, ADR-0057). +//! +//! Two phases, each safe on its own. `consolidate` merges small live packs and +//! commits the result with one manifest CAS (a no-op log entry plus a +//! checkpoint whose pack list is the trimmed set). `sweep` then deletes what no +//! manifest chain can reach: log entries and checkpoints behind a cutoff old +//! enough that no reader still needs them, and packs nothing names. Nothing is +//! ever deleted that the current manifest references, and the manifest itself +//! is never deleted, so a failure between any two operations leaves a bucket a +//! cold start reads. + +use super::{capped, BucketTransport}; +use crate::error::{Error, Result}; +use crate::walfmt::{checkpoint_key, log_key, pack_key, Checkpoint, LogEntry, Manifest, MAX_LIST}; +use scl_core::pack::parse_pack_reader; +use scl_core::ObjectId; +use scl_objio::{Fetched, ObjectMeta}; +use std::collections::{BTreeMap, BTreeSet}; +use std::time::{Duration, SystemTime}; + +/// Replans after a lost manifest CAS or a concurrent compactor before giving up. +const MAX_ATTEMPTS: u32 = 4; + +/// Unreferenced packs deleted between two looks at the manifest. +const PACK_DELETE_BATCH: usize = 16; + +/// Decoded bytes one merge pack may hold, as a multiple of `group_bytes` +/// (which counts compressed bytes). +const DECODED_FACTOR: u64 = 4; + +/// Knobs for [`BucketTransport::compact`]. The defaults are what +/// `sc remote compact` uses. +#[derive(Debug, Clone)] +pub struct CompactOptions { + /// Nothing unreferenced is deleted until it is at least this old (and the + /// checkpoint that superseded it is too). Mirrors `sc gc --prune-expire`. + pub prune_expire: Duration, + /// Plan and report; write and delete nothing. + pub dry_run: bool, + /// Live packs smaller than this are merge candidates. + pub small_pack_bytes: u64, + /// Upper bound on the summed size of one merge group (bounds peak RAM). + pub group_bytes: u64, + /// Clock override for tests; `None` is the wall clock at the start of the run. + pub now: Option, +} + +impl Default for CompactOptions { + fn default() -> Self { + CompactOptions { + prune_expire: Duration::from_secs(24 * 3600), + dry_run: false, + small_pack_bytes: 16 << 20, + group_bytes: 64 << 20, + now: None, + } + } +} + +/// What a compaction did, or with `dry_run`, what it would do. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct CompactReport { + pub dry_run: bool, + /// Live packs folded into new ones, and how many new packs replaced them. + pub packs_merged: usize, + pub packs_created: usize, + /// The manifest's `checkpoint_seq` after phase 1. + pub checkpoint_seq: u64, + /// The sweep cutoff `K` (0 when no checkpoint is old enough yet). + pub cutoff: u64, + pub logs_deleted: usize, + pub checkpoints_deleted: usize, + /// Pack hashes removed (each is a `.pack` and a `.idx`). + pub packs_deleted: usize, + pub bytes_deleted: u64, + /// Unreferenced packs kept because they are younger than `prune_expire`. + pub packs_held_back: usize, + /// Why something was skipped or stopped early. + pub notes: Vec, +} + +/// `checkpoints/00…065` or `log/00…065` -> 65. +fn seq_of(key: &str) -> Option { + key.rsplit('/').next()?.parse().ok() +} + +/// `packs/.pack` / `packs/.idx` -> ``. +fn hash_of(key: &str) -> Option<&str> { + let name = key.strip_prefix("packs/")?; + name.strip_suffix(".pack") + .or_else(|| name.strip_suffix(".idx")) +} + +/// Greedy size-tiered grouping: smallest candidates first, a group closes when +/// the next pack would push it past `group_bytes`. Groups of one are dropped +/// (there is nothing to merge). +fn plan_groups( + live: &[String], + sizes: &BTreeMap, + opts: &CompactOptions, +) -> Vec> { + let mut small: Vec<(u64, &String)> = live + .iter() + .filter_map(|h| sizes.get(h).map(|s| (*s, h))) + .filter(|(s, _)| *s < opts.small_pack_bytes) + .collect(); + small.sort(); + let mut groups: Vec> = Vec::new(); + let (mut cur, mut sum): (Vec, u64) = (Vec::new(), 0); + for (size, hash) in small { + if !cur.is_empty() && sum + size > opts.group_bytes { + groups.push(std::mem::take(&mut cur)); + sum = 0; + } + cur.push(hash.clone()); + sum += size; + } + groups.push(cur); + groups.retain(|g| g.len() >= 2); + groups +} + +impl BucketTransport { + /// Compact this bucket remote (ADR-0057): merge small packs, then delete + /// log entries, checkpoints and packs no reader or writer can still need. + /// Safe to run while others push and fetch; run one at a time (concurrent + /// runs stay correct but waste work). Needs bucket list and delete rights. + pub fn compact(&self, opts: &CompactOptions) -> Result { + let mut rep = CompactReport { + dry_run: opts.dry_run, + ..CompactReport::default() + }; + self.refresh()?; + if self.view.borrow().is_none() { + // No manifest yet: nothing to merge, but a first push that died after + // uploading its pack left debris the sweep can still age out. + rep.notes + .push("bucket has no manifest: only aged orphan packs are swept".into()); + } else { + rep.checkpoint_seq = self + .view + .borrow() + .as_ref() + .map_or(0, |v| v.manifest.checkpoint_seq); + self.consolidate(opts, &mut rep)?; + } + // Read the clock after phase 1 so a zero grace sees its own checkpoint. + let now = opts.now.unwrap_or_else(SystemTime::now); + self.sweep(opts, now, &mut rep)?; + Ok(rep) + } + + /// Phase 1: merge small live packs and commit with one manifest CAS. + fn consolidate(&self, opts: &CompactOptions, rep: &mut CompactReport) -> Result<()> { + // Manifest tag of an attempt that found a planned member pack missing. + let mut missing_on: Option = None; + 'attempt: for _ in 0..MAX_ATTEMPTS { + self.refresh()?; + let (live, nrefs) = { + let view = self.view.borrow(); + let v = view.as_ref().expect("checked non-empty"); + let mut seen = BTreeSet::new(); + let live: Vec = v + .packs + .iter() + .filter(|h| seen.insert((*h).clone())) + .cloned() + .collect(); + (live, v.refs.len()) + }; + let sizes: BTreeMap = self + .bucket + .list_meta("packs/")? + .into_iter() + .filter(|m| m.key.ends_with(".pack")) + .filter_map(|m| hash_of(&m.key).map(|h| (h.to_string(), m.size))) + .collect(); + let groups = plan_groups(&live, &sizes, opts); + if groups.is_empty() { + if live.len() > MAX_LIST { + rep.notes.push(format!( + "{} live packs exceed the checkpoint cap of {MAX_LIST} and none is small \ + enough to merge; folding stays off", + live.len() + )); + } + return Ok(()); + } + let replaced: BTreeSet = groups.iter().flatten().cloned().collect(); + if live.len() - replaced.len() + groups.len() > MAX_LIST || nrefs > MAX_LIST { + rep.notes.push(format!( + "merging would leave more than {MAX_LIST} packs (or {nrefs} refs exceed the \ + cap): a checkpoint could not hold it, nothing merged" + )); + return Ok(()); + } + if opts.dry_run { + rep.packs_merged = replaced.len(); + rep.packs_created = groups.len(); + return Ok(()); + } + let mut created = Vec::new(); + for group in &groups { + let mut objects: BTreeMap> = BTreeMap::new(); + let mut decoded = 0u64; + for hash in group { + let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(hash), None)? else { + // Either another compactor retired it (the manifest has + // moved: replan) or the bucket is damaged (same manifest + // twice: say so instead of retrying). + let tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); + if tag == missing_on { + return Err(Error::Wal(format!( + "pack {hash} on chain but body absent" + ))); + } + missing_on = tag; + continue 'attempt; + }; + parse_pack_reader(std::io::Cursor::new(&bytes[..]), |id, obj| { + let enc = obj.encode(); + decoded += enc.len() as u64; + objects.insert(id, enc); + Ok(()) + })?; + // Compressed sizes bound the group, not the decoded bytes held + // here; spill to a pack of its own before RAM balloons. + if decoded > opts.group_bytes.saturating_mul(DECODED_FACTOR) { + let part: Vec<(ObjectId, Vec)> = + std::mem::take(&mut objects).into_iter().collect(); + created.push(self.upload_pack(&part)?.0); + decoded = 0; + } + } + if !objects.is_empty() { + let objects: Vec<(ObjectId, Vec)> = objects.into_iter().collect(); + created.push(self.upload_pack(&objects)?.0); + } + } + // Fence + fold in one CAS: the manifest tag we read is the state + // the checkpoint describes, so a lost CAS just replans. + self.refresh()?; + let (tag, head_seq, head_branch, refs, packs) = { + let view = self.view.borrow(); + let v = view.as_ref().expect("checked non-empty"); + ( + v.tag.clone(), + v.manifest.head_seq, + v.manifest.head_branch.clone(), + v.refs + .iter() + .map(|(b, id)| (b.clone(), *id)) + .collect::>(), + v.packs.clone(), + ) + }; + let packs_now: BTreeSet = packs.iter().cloned().collect(); + let mut seen = BTreeSet::new(); + let mut kept: Vec = packs + .into_iter() + .filter(|h| !replaced.contains(h) && seen.insert(h.clone())) + .collect(); + for h in &created { + if seen.insert(h.clone()) { + kept.push(h.clone()); + } + } + // Recheck both axes on the state we are about to publish: pushes + // that landed while we merged can have pushed either over the cap, + // and an undecodable checkpoint would brick the remote. + if super::fold_would_overflow(refs.len(), kept.len()) { + rep.notes.push( + "new pushes took the checkpoint over the cap while merging: nothing merged" + .into(), + ); + return Ok(()); + } + // Another compactor may have retired some of the packs we merged + // already: replan rather than publish a second copy of the same data. + if !replaced.is_subset(&packs_now) { + continue 'attempt; + } + let mut try_seq = head_seq + 1; + let entry = LogEntry { + seq: 0, + parent_seq: head_seq, + packs: created.clone(), + updates: vec![], + }; + let seq = loop { + let mut candidate = entry.clone(); + candidate.seq = try_seq; + if self + .bucket + .put_new(&log_key(try_seq), &candidate.encode())? + { + break try_seq; + } + try_seq += 1; + }; + let ck = Checkpoint { + seq, + refs, + packs: kept, + }; + if !self.bucket.put_new(&checkpoint_key(seq), &ck.encode())? { + continue 'attempt; // slot debris from a crashed run: claim a later one + } + let manifest = Manifest { + head_seq: seq, + checkpoint_seq: seq, + head_branch, + }; + // A sweep by another compactor may have removed a merged pack that + // was a content-identical leftover; never commit a name that is gone. + self.ensure_packs_present(&created)?; + if self + .bucket + .put_if_tag("manifest", &manifest.encode(), Some(&tag))? + .is_some() + { + self.refresh()?; + rep.packs_merged = replaced.len(); + rep.packs_created = created.len(); + rep.checkpoint_seq = seq; + return Ok(()); + } + } + rep.notes.push(format!( + "manifest kept moving or a concurrent compaction raced ({MAX_ATTEMPTS} attempts): \ + nothing merged, rerun" + )); + Ok(()) + } + + /// Phase 2: delete what is behind the cutoff and what nothing names. + fn sweep(&self, opts: &CompactOptions, now: SystemTime, rep: &mut CompactReport) -> Result<()> { + // `modified <= old_before` means "at least prune_expire old". + let old_before = now + .checked_sub(opts.prune_expire) + .unwrap_or(SystemTime::UNIX_EPOCH); + let old = |m: &ObjectMeta| m.modified <= old_before; + for _ in 0..MAX_ATTEMPTS { + self.refresh()?; + let (tag, ck_seq, live) = match self.view.borrow().as_ref() { + Some(v) => ( + Some(v.tag.clone()), + v.manifest.checkpoint_seq, + v.packs.iter().cloned().collect::>(), + ), + None => (None, 0, BTreeSet::new()), + }; + let ckpts: Vec<(u64, ObjectMeta)> = self + .bucket + .list_meta("checkpoints/")? + .into_iter() + .filter_map(|m| seq_of(&m.key).map(|s| (s, m))) + .collect(); + let logs: Vec<(u64, ObjectMeta)> = self + .bucket + .list_meta("log/")? + .into_iter() + .filter_map(|m| seq_of(&m.key).map(|s| (s, m))) + .collect(); + let cutoff = ckpts + .iter() + .filter(|(s, m)| *s <= ck_seq && old(m)) + .map(|(s, _)| *s) + .max() + .unwrap_or(0); + if cutoff == 0 && ck_seq > 0 { + rep.notes.push( + "no checkpoint is older than the grace window yet: log entries and older \ + checkpoints are kept" + .into(), + ); + } + let del_logs: Vec<&ObjectMeta> = logs + .iter() + .filter(|(s, _)| cutoff > 0 && *s <= cutoff) + .map(|(_, m)| m) + .collect(); + let del_ckpts: Vec<&ObjectMeta> = ckpts + .iter() + .filter(|(s, _)| cutoff > 0 && *s < cutoff) + .map(|(_, m)| m) + .collect(); + + // Pack objects by hash; a hash is a candidate if the live chain + // does not name it and every one of its objects is old enough. + let mut by_hash: BTreeMap> = BTreeMap::new(); + for m in self.bucket.list_meta("packs/")? { + if let Some(h) = hash_of(&m.key) { + by_hash.entry(h.to_string()).or_default().push(m); + } + } + let unnamed: Vec<(&String, &Vec)> = + by_hash.iter().filter(|(h, _)| !live.contains(*h)).collect(); + let (aged, young): (Vec<_>, Vec<_>) = + unnamed.into_iter().partition(|(_, ms)| ms.iter().all(old)); + rep.packs_held_back = young.len(); + + let mut del_packs: Vec<(&String, &Vec)> = Vec::new(); + if !aged.is_empty() { + // Retained structure may still name them: every kept + // checkpoint and every kept log entry (on-chain or not). + match self.retained_packs(&ckpts, &logs, cutoff) { + Ok(named) => { + del_packs = aged + .into_iter() + .filter(|(h, _)| !named.contains(*h)) + .collect() + } + Err(e) => rep + .notes + .push(format!("packs kept, a retained record is unreadable: {e}")), + } + } + + // Log entries and checkpoints behind the cutoff are safe whatever + // the manifest does meanwhile: the cutoff only ever grows, so they + // stay unreachable. Counters add up across replans. + rep.cutoff = rep.cutoff.max(cutoff); + rep.logs_deleted += del_logs.len(); + rep.checkpoints_deleted += del_ckpts.len(); + rep.bytes_deleted += del_logs + .iter() + .chain(del_ckpts.iter()) + .map(|m| m.size) + .sum::(); + if !opts.dry_run { + for m in del_logs.iter().chain(del_ckpts.iter()) { + self.bucket.delete(&m.key)?; + } + } + rep.packs_deleted += del_packs.len(); + rep.bytes_deleted += del_packs + .iter() + .flat_map(|(_, ms)| ms.iter().map(|m| m.size)) + .sum::(); + if opts.dry_run { + if rep.packs_created > 0 { + rep.notes.push( + "dry run: the sweep figures describe the bucket as it is now; the merge \ + and what it retires (its replaced packs, the log up to its new \ + checkpoint) are not simulated" + .into(), + ); + } + return Ok(()); + } + // A pack nothing names can still be named by a push that lands + // while we plan: if the manifest moved, replan the pack set + // (nothing is deleted against a stale picture). + let unmoved = matches!( + (&tag, self.bucket.get("manifest", tag.as_deref())?), + (Some(_), Fetched::Unchanged) | (None, Fetched::Absent) + ); + if !unmoved { + rep.packs_deleted -= del_packs.len(); + rep.bytes_deleted -= del_packs + .iter() + .flat_map(|(_, ms)| ms.iter().map(|m| m.size)) + .sum::(); + continue; + } + // The delete loop can run long (one request per object), so look at + // the manifest again between batches: a push that lands meanwhile + // may have re-used one of these old packs byte for byte. + for (i, batch) in del_packs.chunks(PACK_DELETE_BATCH).enumerate() { + if i > 0 + && !matches!( + (&tag, self.bucket.get("manifest", tag.as_deref())?), + (Some(_), Fetched::Unchanged) | (None, Fetched::Absent) + ) + { + let left = del_packs.len() - i * PACK_DELETE_BATCH; + rep.packs_deleted -= left; + rep.notes.push(format!( + "the manifest moved during pack deletion: {left} unreferenced pack(s) \ + kept, rerun" + )); + return Ok(()); + } + for (hash, _) in batch { + // idx first: a half-deleted pack is then plain debris. + self.bucket.delete(&format!("packs/{hash}.idx"))?; + self.bucket.delete(&pack_key(hash))?; + } + } + return Ok(()); + } + rep.notes.push( + "the manifest kept moving during the sweep: unreferenced packs were kept, rerun".into(), + ); + Ok(()) + } + + /// Pack hashes named by structure the sweep keeps: checkpoints with + /// `seq >= cutoff` and log entries with `seq > cutoff` (everything when + /// `cutoff` is 0). Any record that fails to decode aborts: deleting packs + /// on a partial picture is never right. + fn retained_packs( + &self, + ckpts: &[(u64, ObjectMeta)], + logs: &[(u64, ObjectMeta)], + cutoff: u64, + ) -> Result> { + let mut named = BTreeSet::new(); + for (seq, m) in ckpts { + if cutoff == 0 || *seq >= cutoff { + let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { + continue; // deleted since the listing: no longer retained + }; + named.extend( + Checkpoint::decode(&capped("checkpoint", bytes)?) + .map_err(|e| Error::Wal(format!("{}: {e}", m.key)))? + .packs, + ); + } + } + for (seq, m) in logs { + if cutoff == 0 || *seq > cutoff { + let Fetched::New { bytes, .. } = self.bucket.get(&m.key, None)? else { + continue; + }; + named.extend( + LogEntry::decode(&capped("log entry", bytes)?) + .map_err(|e| Error::Wal(format!("{}: {e}", m.key)))? + .packs, + ); + } + } + Ok(named) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/repo/src/bucket_transport/compact/tests.rs b/crates/repo/src/bucket_transport/compact/tests.rs new file mode 100644 index 00000000..621301a7 --- /dev/null +++ b/crates/repo/src/bucket_transport/compact/tests.rs @@ -0,0 +1,916 @@ +//! Compaction tests (#216, ADR-0057). Most run on an in-memory bucket (fast, +//! and lets a test fork a bucket or fake object ages); the full-run and the +//! concurrency tests use `DirBucket`, the real backend behind `sc+wal://`. + +use super::super::CHECKPOINT_INTERVAL; +use super::*; +use crate::error::Error; +use crate::transport::Transport; +use crate::walfmt::{idx_key, RefUpdate}; +use scl_core::Object; +use scl_objio::{Bucket, DirBucket}; +use std::path::PathBuf; +use std::sync::atomic::{AtomicBool, AtomicI64, Ordering}; +use std::sync::{Arc, Mutex}; + +/// In-memory `Bucket` with the same semantics as `DirBucket`. +#[derive(Default)] +struct MemBucket { + map: Mutex, SystemTime)>>, +} + +fn tag_of(bytes: &[u8]) -> String { + hex::encode(blake3::hash(bytes).as_bytes()) +} + +impl MemBucket { + fn fork(&self) -> Arc { + Arc::new(MemBucket { + map: Mutex::new(self.map.lock().unwrap().clone()), + }) + } + /// Pretend `key` was written at `at`. + fn touch(&self, key: &str, at: SystemTime) { + self.map.lock().unwrap().get_mut(key).unwrap().1 = at; + } +} + +struct Shared(Arc); +impl Bucket for Shared { + fn get(&self, key: &str, cached: Option<&str>) -> scl_objio::Result { + Ok(match self.0.map.lock().unwrap().get(key) { + None => Fetched::Absent, + Some((b, _)) if cached == Some(tag_of(b).as_str()) => Fetched::Unchanged, + Some((b, _)) => Fetched::New { + bytes: b.clone(), + tag: tag_of(b), + }, + }) + } + fn put_new(&self, key: &str, bytes: &[u8]) -> scl_objio::Result { + let mut m = self.0.map.lock().unwrap(); + if m.contains_key(key) { + return Ok(false); + } + m.insert(key.into(), (bytes.to_vec(), SystemTime::now())); + Ok(true) + } + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected: Option<&str>, + ) -> scl_objio::Result> { + let mut m = self.0.map.lock().unwrap(); + if m.get(key).map(|(b, _)| tag_of(b)).as_deref() != expected { + return Ok(None); + } + m.insert(key.into(), (bytes.to_vec(), SystemTime::now())); + Ok(Some(tag_of(bytes))) + } + fn list(&self, prefix: &str) -> scl_objio::Result> { + Ok(self.list_meta(prefix)?.into_iter().map(|m| m.key).collect()) + } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + Ok(self + .0 + .map + .lock() + .unwrap() + .iter() + .filter(|(k, _)| k.starts_with(prefix)) + .map(|(k, (b, t))| ObjectMeta { + key: k.clone(), + size: b.len() as u64, + modified: *t, + }) + .collect()) + } + fn exists(&self, key: &str) -> scl_objio::Result { + Ok(self.0.map.lock().unwrap().contains_key(key)) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.0.map.lock().unwrap().remove(key); + Ok(()) + } +} + +/// Where a test's bucket lives, so helpers work on either backend. +trait Store { + fn bucket(&self) -> Box; +} +impl Store for PathBuf { + fn bucket(&self) -> Box { + Box::new(DirBucket::open(self).unwrap()) + } +} +impl Store for Arc { + fn bucket(&self) -> Box { + Box::new(Shared(self.clone())) + } +} + +/// Backdate every file under `dir` by an hour, as if the data had sat there. +fn age_files(dir: &std::path::Path) { + let then = SystemTime::now() - Duration::from_secs(3600); + for e in std::fs::read_dir(dir).unwrap().flatten() { + if e.path().is_dir() { + age_files(&e.path()); + } else { + std::fs::File::options() + .write(true) + .open(e.path()) + .unwrap() + .set_modified(then) + .unwrap(); + } + } +} + +fn scratch(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-bt-compact-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + d +} + +fn open(s: &impl Store) -> BucketTransport { + BucketTransport::from_bucket(s.bucket()).unwrap() +} + +fn keys(s: &impl Store, prefix: &str) -> Vec { + s.bucket().list(prefix).unwrap() +} + +fn manifest(s: &impl Store) -> Manifest { + let Fetched::New { bytes, .. } = s.bucket().get("manifest", None).unwrap() else { + panic!("no manifest") + }; + Manifest::decode(&bytes).unwrap() +} + +/// Options that merge everything and treat every object as old enough. +fn eager() -> CompactOptions { + CompactOptions { + prune_expire: Duration::ZERO, + ..CompactOptions::default() + } +} + +/// A one-commit history: an empty tree shared by every commit plus a +/// snapshot distinguished by `label`. Returns (tip, objects). +fn commit(label: &str) -> (ObjectId, Vec<(ObjectId, Vec)>) { + let tree = Object::Tree(scl_core::Tree::new(vec![])); + let snap = Object::Snapshot(scl_core::Snapshot { + root: tree.id(), + parents: vec![], + author: "t".into(), + timestamp: 1, + message: label.into(), + secrets: Default::default(), + protection: Default::default(), + }); + ( + snap.id(), + vec![(tree.id(), tree.encode()), (snap.id(), snap.encode())], + ) +} + +/// Push `n` single-commit branches `-` through the real write path +/// (folds a checkpoint once the tail passes the interval). +fn push_commits(t: &BucketTransport, prefix: &str, n: u64) -> Vec<(String, ObjectId)> { + (0..n) + .map(|i| { + let (tip, objects) = commit(&format!("{prefix}-{i}")); + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + let branch = format!("{prefix}-{i}"); + t.update_ref(&branch, &tip, None).unwrap(); + (branch, tip) + }) + .collect() +} + +/// Hand-build a WAL of `n` one-commit pushes with a checkpoint folded at `ck` +/// (0 = none): the shape real pushes leave, without their cost. +fn hand_built(s: &impl Store, n: u64, ck: u64) -> Vec<(String, ObjectId)> { + let b = s.bucket(); + let (mut refs, mut packs) = (Vec::new(), Vec::new()); + for i in 1..=n { + let (tip, objects) = commit(&format!("hb-{i}")); + let (pack, idx) = scl_core::pack::build_pack(&objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + b.put_new(&pack_key(&hash), &pack).unwrap(); + b.put_new(&idx_key(&hash), &idx).unwrap(); + let entry = LogEntry { + seq: i, + parent_seq: i - 1, + packs: vec![hash.clone()], + updates: vec![RefUpdate { + branch: format!("b-{i}"), + old: None, + new: tip, + }], + }; + b.put_new(&log_key(i), &entry.encode()).unwrap(); + refs.push((format!("b-{i}"), tip)); + packs.push(hash); + if i == ck { + let c = Checkpoint { + seq: ck, + refs: refs.clone(), + packs: packs.clone(), + }; + b.put_new(&checkpoint_key(ck), &c.encode()).unwrap(); + } + } + let m = Manifest { + head_seq: n, + checkpoint_seq: ck, + head_branch: "b-1".into(), + }; + b.put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + refs.sort(); + refs +} + +/// A cold-start reader sees exactly `expect` and can read every object both +/// singly and as one pack. +fn assert_readable(s: &impl Store, expect: &[(String, ObjectId)]) { + let mut expect = expect.to_vec(); + expect.sort(); + let t = open(s); + assert_eq!(t.list_refs().unwrap(), expect); + for (_, id) in &expect { + assert_eq!(ObjectId::of(&t.get_object(id).unwrap()), *id); + } + let tips: Vec = expect.iter().map(|(_, id)| *id).collect(); + let mut out = Vec::new(); + t.get_pack(&tips, &[], None, &mut out).unwrap(); + // every tip's snapshot plus the one tree they share + assert_eq!( + scl_core::pack::parse_pack(&out).unwrap().len(), + tips.len() + 1 + ); +} + +#[test] +fn the_in_memory_double_honors_the_bucket_contract() { + scl_objio::contract_suite(&Shared(Arc::new(MemBucket::default()))); +} + +#[test] +fn merges_small_packs_and_sweeps_everything_behind_the_checkpoint() { + let root = scratch("full"); + let t = open(&root); + let refs = push_commits(&t, "w", 70); + // 70 pushes folded a checkpoint at 65; 70 log entries, 70 packs. + assert_eq!(manifest(&root).checkpoint_seq, 65); + assert_eq!(keys(&root, "packs/").len(), 140); + + let rep = t.compact(&eager()).unwrap(); + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1), "{rep:?}"); + assert_eq!(rep.checkpoint_seq, 71); + assert_eq!(rep.cutoff, 71); + assert_eq!((rep.logs_deleted, rep.checkpoints_deleted), (71, 1)); + assert_eq!(rep.packs_deleted, 70); + + let m = manifest(&root); + assert_eq!((m.head_seq, m.checkpoint_seq), (71, 71)); + assert!(keys(&root, "log/").is_empty()); + assert_eq!(keys(&root, "checkpoints/").len(), 1); + assert_eq!(keys(&root, "packs/").len(), 2, "one merged pack + idx"); + assert_readable(&root, &refs); + + // Idempotent: nothing left to do. + let again = t.compact(&eager()).unwrap(); + assert_eq!( + ( + again.packs_merged, + again.logs_deleted, + again.checkpoints_deleted, + again.packs_deleted + ), + (0, 0, 0, 0), + "{again:?}" + ); + // And the remote still takes pushes. + let mut all = refs; + all.extend(push_commits(&t, "after", 3)); + assert_readable(&root, &all); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn dry_run_reports_the_plan_and_writes_nothing() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + push_commits(&t, "w", 70); + let before = mem.map.lock().unwrap().clone(); + let rep = t + .compact(&CompactOptions { + dry_run: true, + ..eager() + }) + .unwrap(); + assert!(rep.dry_run); + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1)); + assert!(rep.logs_deleted > 0 && rep.bytes_deleted > 0, "{rep:?}"); + assert!(!rep.notes.is_empty()); + assert!( + *mem.map.lock().unwrap() == before, + "dry run must not change the bucket" + ); +} + +#[test] +fn nothing_is_deleted_inside_the_grace_window_and_everything_after_it() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let refs = push_commits(&t, "w", 70); + let rep = t.compact(&CompactOptions::default()).unwrap(); + // Phase 1 is additive and runs; the sweep keeps everything young. + assert_eq!((rep.packs_merged, rep.packs_created), (70, 1)); + assert_eq!( + (rep.logs_deleted, rep.checkpoints_deleted, rep.packs_deleted), + (0, 0, 0) + ); + assert_eq!(rep.packs_held_back, 70, "{rep:?}"); + assert_eq!(keys(&mem, "log/").len(), 71); + assert_readable(&mem, &refs); + + // Two days later every object is past the window. + let later = SystemTime::now() + Duration::from_secs(2 * 24 * 3600); + let rep = t + .compact(&CompactOptions { + now: Some(later), + ..CompactOptions::default() + }) + .unwrap(); + assert_eq!( + (rep.logs_deleted, rep.checkpoints_deleted, rep.packs_deleted), + (71, 1, 70) + ); + assert_readable(&mem, &refs); +} + +#[test] +fn the_cutoff_is_the_newest_checkpoint_that_has_aged_not_the_newest_checkpoint() { + // A busy remote folds every 64 pushes, so its newest checkpoint is rarely + // old; an older one still sets the cutoff. + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let b = mem.bucket(); + let Fetched::New { bytes, tag } = b.get("manifest", None).unwrap() else { + panic!() + }; + let mut m = Manifest::decode(&bytes).unwrap(); + let mut packs = Vec::new(); + for seq in 1..=12 { + let Fetched::New { bytes, .. } = b.get(&log_key(seq), None).unwrap() else { + panic!() + }; + packs.extend(LogEntry::decode(&bytes).unwrap().packs); + } + let ck12 = Checkpoint { + seq: 12, + refs: refs.clone(), + packs, + }; + b.put_new(&checkpoint_key(12), &ck12.encode()).unwrap(); + m.checkpoint_seq = 12; + b.put_if_tag("manifest", &m.encode(), Some(&tag)) + .unwrap() + .unwrap(); + // Checkpoint 10 is two hours old, checkpoint 12 just landed; window 1h. + let now = SystemTime::now(); + mem.touch(&checkpoint_key(10), now - Duration::from_secs(7200)); + + let rep = open(&mem) + .compact(&CompactOptions { + small_pack_bytes: 0, // no merging: isolate the sweep + prune_expire: Duration::from_secs(3600), + now: Some(now), + ..CompactOptions::default() + }) + .unwrap(); + assert_eq!(rep.packs_merged, 0); + assert_eq!(rep.cutoff, 10, "{rep:?}"); + assert_eq!((rep.logs_deleted, rep.checkpoints_deleted), (10, 0)); + assert_eq!(keys(&mem, "log/").len(), 2, "entries past the cutoff stay"); + assert_eq!(keys(&mem, "checkpoints/").len(), 2); + assert_readable(&mem, &refs); +} + +/// A bucket that fails every operation once a shared budget runs out, like a +/// process dying mid-compaction. +struct FailAfter { + inner: Shared, + left: Arc, +} +impl FailAfter { + fn tick(&self) -> scl_objio::Result<()> { + if self.left.fetch_sub(1, Ordering::SeqCst) <= 0 { + Err(scl_objio::Error::Backend("injected failure".into())) + } else { + Ok(()) + } + } +} +impl Bucket for FailAfter { + fn get(&self, key: &str, tag: Option<&str>) -> scl_objio::Result { + self.tick()?; + self.inner.get(key, tag) + } + fn put_new(&self, key: &str, bytes: &[u8]) -> scl_objio::Result { + self.tick()?; + self.inner.put_new(key, bytes) + } + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + tag: Option<&str>, + ) -> scl_objio::Result> { + self.tick()?; + self.inner.put_if_tag(key, bytes, tag) + } + fn list(&self, prefix: &str) -> scl_objio::Result> { + self.tick()?; + self.inner.list(prefix) + } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + self.tick()?; + self.inner.list_meta(prefix) + } + fn exists(&self, key: &str) -> scl_objio::Result { + self.tick()?; + self.inner.exists(key) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.tick()?; + self.inner.delete(key) + } +} + +#[test] +fn a_failure_at_every_operation_boundary_leaves_a_readable_bucket() { + let template = Arc::new(MemBucket::default()); + let refs = hand_built(&template, 12, 10); + let mut finished_at = None; + for budget in 0..5000i64 { + let work = template.fork(); + let left = Arc::new(AtomicI64::new(budget)); + let outcome = BucketTransport::from_bucket(Box::new(FailAfter { + inner: Shared(work.clone()), + left: left.clone(), + })) + .and_then(|t| t.compact(&eager())); + + // Whatever happened, a fresh reader still sees everything. + assert_readable(&work, &refs); + if outcome.is_ok() && left.load(Ordering::SeqCst) > 0 { + finished_at = Some(budget); + break; + } + // And a healthy rerun finishes the job. + open(&work).compact(&eager()).unwrap(); + assert_readable(&work, &refs); + assert_eq!(keys(&work, "packs/").len(), 2, "budget {budget}"); + } + let n = finished_at.expect("compaction never completed within the budget"); + assert!(n > 20, "the loop should cross many boundaries, got {n}"); +} + +#[test] +fn readers_and_writers_stay_consistent_while_compaction_runs() { + let root = scratch("concurrent"); + let seeded = hand_built(&root, 12, 10); + age_files(&root); // the seed is old enough to sweep from the first run + const WRITERS: usize = 3; + const PER_WRITER: u64 = 20; + const GRACE: Duration = Duration::from_millis(1500); + let done = AtomicBool::new(false); + let opts = CompactOptions { + // Far longer than a writer's upload-to-commit gap even on a loaded + // machine, short enough that sweeps really delete during the run. + prune_expire: GRACE, + ..CompactOptions::default() + }; + let landed = Mutex::new(Vec::new()); + let deleted = std::sync::atomic::AtomicUsize::new(0); + std::thread::scope(|s| { + let writers: Vec<_> = (0..WRITERS) + .map(|w| { + let (root, landed) = (&root, &landed); + s.spawn(move || { + let t = open(root); + for i in 0..PER_WRITER { + let (tip, objects) = commit(&format!("writer-{w}-{i}")); + let branch = format!("writer-{w}-{i}"); + loop { + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + match t.update_ref(&branch, &tip, None) { + Ok(()) => break, + // The documented guard: a pack swept before + // commit fails the push; the retry re-uploads. + Err(Error::Remote(m)) if m.contains("removed from the bucket") => {} + Err(e) => panic!("writer {w}: {e}"), + } + } + landed.lock().unwrap().push((branch, tip)); + std::thread::sleep(Duration::from_millis(80)); + } + }) + }) + .collect(); + let readers: Vec<_> = (0..3) + .map(|r| { + let (root, done, seeded) = (&root, &done, &seeded); + s.spawn(move || { + let long_lived = open(root); + let mut seen = seeded.len(); + let mut rounds = 0u32; + while !done.load(Ordering::SeqCst) || rounds < 3 { + rounds += 1; + // alternate a long-lived (stale-view) reader and a cold start + let cold; + let t = if rounds.is_multiple_of(3) { + cold = open(root); + &cold + } else { + &long_lived + }; + let refs = t.list_refs().unwrap_or_else(|e| panic!("reader {r}: {e}")); + assert!(refs.len() >= seen, "reader {r}: refs went backwards"); + seen = refs.len(); + for (_, id) in &refs { + let bytes = t + .get_object(id) + .unwrap_or_else(|e| panic!("reader {r}: get_object: {e}")); + assert_eq!(ObjectId::of(&bytes), *id); + } + let tips: Vec = refs.iter().map(|(_, id)| *id).collect(); + t.get_pack(&tips, &[], None, &mut Vec::new()) + .unwrap_or_else(|e| panic!("reader {r}: get_pack: {e}")); + } + }) + }) + .collect(); + // Two compactors at once: the ADR promises that is safe (just wasteful). + let compactors: Vec<_> = (0..2) + .map(|_| { + s.spawn(|| { + let t = open(&root); + let mut runs = 0u32; + while !done.load(Ordering::SeqCst) { + match t.compact(&opts) { + Ok(rep) => { + deleted.fetch_add( + rep.logs_deleted + rep.checkpoints_deleted + rep.packs_deleted, + Ordering::SeqCst, + ); + } + // A rival's sweep took a pack this run was about to + // name; the guard fails the run instead of committing. + Err(Error::Remote(m)) if m.contains("removed from the bucket") => {} + Err(e) => panic!("compactor: {e}"), + } + runs += 1; + std::thread::sleep(Duration::from_millis(40)); + } + runs + }) + }) + .collect(); + for w in writers { + w.join().unwrap(); + } + // Keep readers and the compactor running until everything the writers + // produced has also aged past the window and been swept. + std::thread::sleep(GRACE * 2); + done.store(true, Ordering::SeqCst); + for r in readers { + r.join().unwrap(); + } + for c in compactors { + assert!(c.join().unwrap() > 2, "compactor barely ran"); + } + }); + assert!( + deleted.load(Ordering::SeqCst) > 0, + "the sweep never deleted anything during the run, so this proved nothing" + ); + let mut expect = seeded.clone(); + expect.extend(landed.into_inner().unwrap()); + // After the dust settles a zero-grace pass leaves a tight bucket. + open(&root).compact(&eager()).unwrap(); + assert_readable(&root, &expect); + assert_eq!(keys(&root, "packs/").len(), 2, "all packs merged into one"); + // Nothing at or below the head survives. A slot above it can: a rival + // compactor's lost CAS leaves its claimed entry as debris, swept once a + // checkpoint passes it. + let head = manifest(&root).head_seq; + assert!(keys(&root, "log/") + .iter() + .all(|k| seq_of(k).unwrap() > head)); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn a_reader_whose_view_was_swept_restarts_and_a_corrupt_bucket_still_fails() { + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let reader = open(&mem); + let (_, tip) = refs[0]; + assert!(reader.has_object(&tip).unwrap()); + // Compaction lands behind the reader's back and sweeps the old packs. + open(&mem).compact(&eager()).unwrap(); + assert_eq!(keys(&mem, "packs/").len(), 2); + // The reader's cached index points into a pack that no longer exists; the + // fetch notices, re-reads the manifest and restarts on it. + let bytes = reader.object_bytes(&tip).unwrap(); + assert_eq!(ObjectId::of(&bytes), tip); + + // With the manifest unchanged, a missing pack is corruption and fails. + let pack = keys(&mem, "packs/") + .into_iter() + .find(|k| k.ends_with(".pack")) + .unwrap(); + mem.bucket().delete(&pack).unwrap(); + let err = open(&mem).object_bytes(&tip).unwrap_err(); + assert!(format!("{err}").contains("body absent"), "{err}"); +} + +#[test] +fn a_cold_start_straddling_a_sweep_restarts_instead_of_failing() { + // The reader read the manifest, then the checkpoint and log it names were + // swept. Model it with a bucket that runs a compaction right after the + // reader's first manifest read. + struct SweepAfterManifest { + inner: Shared, + fired: Mutex, + } + impl Bucket for SweepAfterManifest { + fn get(&self, key: &str, tag: Option<&str>) -> scl_objio::Result { + let got = self.inner.get(key, tag)?; + let mut fired = self.fired.lock().unwrap(); + if key == "manifest" && !*fired { + *fired = true; + drop(fired); + BucketTransport::from_bucket(Box::new(Shared(self.inner.0.clone()))) + .unwrap() + .compact(&eager()) + .unwrap(); + } + Ok(got) + } + fn put_new(&self, k: &str, b: &[u8]) -> scl_objio::Result { + self.inner.put_new(k, b) + } + fn put_if_tag( + &self, + k: &str, + b: &[u8], + t: Option<&str>, + ) -> scl_objio::Result> { + self.inner.put_if_tag(k, b, t) + } + fn list(&self, p: &str) -> scl_objio::Result> { + self.inner.list(p) + } + fn list_meta(&self, p: &str) -> scl_objio::Result> { + self.inner.list_meta(p) + } + fn exists(&self, k: &str) -> scl_objio::Result { + self.inner.exists(k) + } + fn delete(&self, k: &str) -> scl_objio::Result<()> { + self.inner.delete(k) + } + } + let mem = Arc::new(MemBucket::default()); + let refs = hand_built(&mem, 12, 10); + let t = BucketTransport::from_bucket(Box::new(SweepAfterManifest { + inner: Shared(mem.clone()), + fired: Mutex::new(false), + })) + .expect("the cold start restarts on the new manifest"); + let mut expect = refs.clone(); + expect.sort(); + assert_eq!(t.list_refs().unwrap(), expect); + assert_eq!(manifest(&mem).checkpoint_seq, 13, "the sweep really ran"); +} + +#[test] +fn a_push_whose_pack_was_swept_before_commit_fails_instead_of_corrupting() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let (tip, objects) = commit("orphaned"); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + // The sweep removes the not-yet-committed pack. + for k in keys(&mem, "packs/") { + mem.bucket().delete(&k).unwrap(); + } + let err = t.update_ref("main", &tip, None).unwrap_err(); + assert!( + format!("{err}").contains("removed from the bucket"), + "{err}" + ); + assert!(matches!( + mem.bucket().get("manifest", None).unwrap(), + Fetched::Absent + )); +} + +#[test] +fn a_remote_past_max_list_regains_folding_after_compaction() { + let mem = Arc::new(MemBucket::default()); + // 66 commits in one real pack, plus MAX_LIST placeholder packs (valid, + // empty), so the cumulative pack list is MAX_LIST + 1: one more than a + // checkpoint can hold, which is what makes folding skip. + const ENTRIES: u64 = 66; + let commits: Vec<_> = (1..=ENTRIES).map(|i| commit(&format!("big-{i}"))).collect(); + let mut real: Vec<_> = commits.iter().flat_map(|(_, o)| o.clone()).collect(); + real.sort(); + real.dedup(); + let (rpack, ridx) = scl_core::pack::build_pack(&real).unwrap(); + let rhash = hex::encode(blake3::hash(&rpack).as_bytes()); + let (epack, eidx) = scl_core::pack::build_pack(&[]).unwrap(); + let fakes: Vec = (0..MAX_LIST).map(|i| format!("fake-{i:06}")).collect(); + let b = mem.bucket(); + for h in &fakes { + b.put_new(&pack_key(h), &epack).unwrap(); + b.put_new(&idx_key(h), &eidx).unwrap(); + } + b.put_new(&pack_key(&rhash), &rpack).unwrap(); + b.put_new(&idx_key(&rhash), &ridx).unwrap(); + let chunk = fakes.len().div_ceil(ENTRIES as usize); + let mut refs = Vec::new(); + for (i, (tip, _)) in commits.iter().enumerate() { + let seq = i as u64 + 1; + let mut packs: Vec = fakes.chunks(chunk).nth(i).unwrap_or(&[]).to_vec(); + if i == 0 { + packs.push(rhash.clone()); + } + let entry = LogEntry { + seq, + parent_seq: seq - 1, + packs, + updates: vec![RefUpdate { + branch: format!("b-{seq}"), + old: None, + new: *tip, + }], + }; + b.put_new(&log_key(seq), &entry.encode()).unwrap(); + refs.push((format!("b-{seq}"), *tip)); + } + let m = Manifest { + head_seq: ENTRIES, + checkpoint_seq: 0, + head_branch: "b-1".into(), + }; + b.put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + + // Before: a push lands but its fold is skipped (over the cap). + let t = open(&mem); + refs.extend(push_commits(&t, "extra", 1)); + assert_eq!( + manifest(&mem).checkpoint_seq, + 0, + "fold should be skipped over the cap" + ); + + let rep = t.compact(&eager()).unwrap(); + // the fakes, the real pack and the extra push's pack + assert_eq!( + (rep.packs_merged, rep.packs_created), + (MAX_LIST + 2, 1), + "{rep:?}" + ); + assert!(manifest(&mem).checkpoint_seq > 0); + assert_eq!(rep.packs_deleted, MAX_LIST + 2); + assert_readable(&mem, &refs); + + // After: ordinary pushes fold again. + let ck_after = manifest(&mem).checkpoint_seq; + refs.extend(push_commits(&t, "later", CHECKPOINT_INTERVAL + 2)); + assert!( + manifest(&mem).checkpoint_seq > ck_after, + "folding must resume once the pack list fits" + ); + assert_readable(&mem, &refs); +} + +#[test] +fn empty_bucket_and_single_pack_are_noops() { + let mem = Arc::new(MemBucket::default()); + let t = open(&mem); + let rep = t.compact(&eager()).unwrap(); + assert_eq!(rep.packs_merged, 0); + assert!(!rep.notes.is_empty()); + // One pack is not a merge. + let refs = push_commits(&t, "solo", 1); + let rep = t.compact(&eager()).unwrap(); + assert_eq!((rep.packs_merged, rep.logs_deleted), (0, 0)); + assert_readable(&mem, &refs); +} + +#[test] +fn a_first_push_that_died_before_the_manifest_leaves_orphans_the_sweep_ages_out() { + let mem = Arc::new(MemBucket::default()); + let (_, objects) = commit("never committed"); + let (pack, idx) = scl_core::pack::build_pack(&objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + let b = mem.bucket(); + b.put_new(&pack_key(&hash), &pack).unwrap(); + b.put_new(&idx_key(&hash), &idx).unwrap(); + let t = open(&mem); + // young: kept + let rep = t.compact(&CompactOptions::default()).unwrap(); + assert_eq!((rep.packs_deleted, rep.packs_held_back), (0, 1)); + assert_eq!(keys(&mem, "packs/").len(), 2); + // aged: swept, and the bucket is still a valid empty remote + let rep = t.compact(&eager()).unwrap(); + assert_eq!(rep.packs_deleted, 1, "{rep:?}"); + assert!(keys(&mem, "packs/").is_empty()); + assert!(open(&mem).list_refs().unwrap().is_empty()); +} + +#[test] +fn a_manifest_that_moves_during_pack_deletion_stops_the_deletes() { + // Wraps a bucket so the first pack delete is followed by a push by someone + // else: the sweep must notice at its next look and keep the rest. + struct PushOnFirstPackDelete { + inner: Shared, + fired: Mutex, + } + impl Bucket for PushOnFirstPackDelete { + fn get(&self, k: &str, t: Option<&str>) -> scl_objio::Result { + self.inner.get(k, t) + } + fn put_new(&self, k: &str, b: &[u8]) -> scl_objio::Result { + self.inner.put_new(k, b) + } + fn put_if_tag( + &self, + k: &str, + b: &[u8], + t: Option<&str>, + ) -> scl_objio::Result> { + self.inner.put_if_tag(k, b, t) + } + fn list(&self, p: &str) -> scl_objio::Result> { + self.inner.list(p) + } + fn list_meta(&self, p: &str) -> scl_objio::Result> { + self.inner.list_meta(p) + } + fn exists(&self, k: &str) -> scl_objio::Result { + self.inner.exists(k) + } + fn delete(&self, k: &str) -> scl_objio::Result<()> { + self.inner.delete(k)?; + let mut fired = self.fired.lock().unwrap(); + if k.starts_with("packs/") && !*fired { + *fired = true; + drop(fired); + push_commits(&open(&self.inner.0), "late", 1); + } + Ok(()) + } + } + let mem = Arc::new(MemBucket::default()); + let refs = push_commits(&open(&mem), "w", 70); + let t = BucketTransport::from_bucket(Box::new(PushOnFirstPackDelete { + inner: Shared(mem.clone()), + fired: Mutex::new(false), + })) + .unwrap(); + let rep = t.compact(&eager()).unwrap(); + assert!( + rep.notes + .iter() + .any(|n| n.contains("moved during pack deletion")), + "{rep:?}" + ); + assert!(rep.packs_deleted < 70, "{rep:?}"); + // some old packs were kept, the bucket is readable, and a rerun finishes + assert!(keys(&mem, "packs/").len() > 4); + let mut all = refs; + all.push(("late-0".to_string(), commit("late-0").0)); + assert_readable(&mem, &all); + open(&mem).compact(&eager()).unwrap(); + assert_eq!(keys(&mem, "packs/").len(), 2); + assert_readable(&mem, &all); +} diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 9edff190..b2d78d28 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -66,7 +66,9 @@ pub mod workspace; pub mod worktree; pub(crate) mod ws; -pub use bucket_transport::{BucketScheme, BucketTransport, BucketUrl}; +pub use bucket_transport::{ + BucketScheme, BucketTransport, BucketUrl, CompactOptions, CompactReport, +}; pub use conflicts::{ActiveOp, ConflictKind, ConflictVersions, ResolveSide, Side}; pub use error::{Error, Result}; pub use gc::GcStats; diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 5091a62d..bb273ef4 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -418,9 +418,28 @@ too, not just the src-control-side metadata. from the key it was fetched at, and a log chain that steps past the checkpoint without landing on it. A writer with bucket access can fold a checkpoint, but it is derived data any reader can refold from the log, and a - lost manifest CAS drops a fold silently. Nothing is ever removed from a - bucket (no compaction or gc yet, #216), so a hostile writer can grow it - without bound; bucket quotas are the operator's bound. + lost manifest CAS drops a fold silently. A hostile writer can grow a bucket + without bound; bucket quotas are the operator's bound, and `sc remote + compact` reclaims what is unreferenced (below). +- **Compaction deletes data, so it is a privileged operation (#216, + ADR-0057).** `sc remote compact` needs bucket **list and delete** rights in + addition to the get/put/head permissions a pusher holds (pushes themselves need no list + or delete rights). Grant them to a dedicated operator or + scheduled-job identity, not to ordinary pushers, read-only clients or the + `sc serve --store` instances (they never need them: compaction is not a wire + verb, so a served bucket's clients cannot trigger a delete). It cannot + create new authority: it never touches the manifest's refs, only removes + objects no current manifest chain names, and a writer who can already + write to the bucket can do worse with raw access. What it does change for a + hostile or mistaken writer: (a) a stalled push whose uploaded pack is older + than `--prune-expire` can lose it and fail on retry, or, if its pack is byte-identical to an old + orphan that a sweep is deleting at that moment, commit an entry naming a + deleted pack, which + makes the remote unreadable until fixed (hence the 24h default, and + leases as the real fix); (b) a hostile write that fills `log/` or + `checkpoints/` with junk is reclaimed once it falls behind the cutoff; junk + at a far-future seq never falls behind it, and an undecodable one stops pack + deletion (fail closed, the report names the key to remove by hand). Use `--dry-run` first on a new bucket. - **Sealed content stays end-to-end sealed, unchanged.** Protected-path ciphertext (ADR-0014/0043) and committed secrets travel through a bucket remote exactly as through any other transport: a bucket reader without the diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index e3f58f3f..71943842 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -106,7 +106,8 @@ confidentiality perimeter for public content, not the sc protocol. suite). - The log grows unboundedly with no compaction yet — every open reader walks the full parent chain back from `head_seq`. Checkpoint folding (P36b) and - bucket GC/compaction are deferred (see `ROADMAP.md`). + bucket GC/compaction are deferred (see `ROADMAP.md`). *(Both are built: + P36b below, compaction and gc in ADR-0057.)* - `sc serve` still cannot host a bucket as its backing store (P36c) — a bucket remote today is written to directly by every `sc push`/`sc fetch` client, not brokered through a server process. @@ -218,3 +219,27 @@ hash). No manifest, log, checkpoint or idx change and no `PROTOCOL_VERSION` bump. The receiver's trust gate for tombstones and the `--no-transcripts` skip are unchanged. Rationale and cost in ADR-0038's #229 addendum. +## As built (#216, 2026-10): compaction and gc + +`sc remote compact [--dry-run] [--prune-expire 24h]` merges small live +packs behind one manifest CAS, then deletes log entries, checkpoints and +unreferenced packs that no reader or writer can still need. The design, the +pruning cutoff, the reader and writer protocols and the residual limit live in +ADR-0057. Effects on this ADR's contracts: + +- **No format change.** The manifest, log entry and checkpoint encodings and + `PROTOCOL_VERSION` are untouched. Compaction writes a log entry with + `updates = []` and a checkpoint whose `packs` list is a covering set rather + than a literal accumulation; both decode in every released reader. +- **Readers restart instead of failing when an object they just read about + is gone.** The "absent checkpoint / log entry / idx / pack" refusals in + P36b stay fail-closed, but only after a re-read of the manifest shows it did + not move. A moved manifest restarts the read on the new one. +- **A pending pack is verified present right before the manifest CAS**, so a + push whose pack was swept fails with a retry message rather than committing + an entry that names a deleted pack. +- `objio`'s `Bucket` gained `list_meta` and `delete`. The "nothing already in + the bucket is ever overwritten" property holds; compaction only adds new + objects and removes unreferenced ones. +- The ADR-0046 `MAX_LIST` fold guard is unchanged. Compaction is what makes a + skipped fold recoverable, on the pack axis. diff --git a/docs/adr/0057-bucket-compaction.md b/docs/adr/0057-bucket-compaction.md new file mode 100644 index 00000000..9c4b3daa --- /dev/null +++ b/docs/adr/0057-bucket-compaction.md @@ -0,0 +1,268 @@ +# ADR-0057: Bucket remote compaction and gc + +- **Status:** Accepted +- **Date:** 2026-10-20 +- **Phase:** 36 follow-on (#216) + +## Context + +ADR-0046's bucket remote only ever adds objects. `log/` entries, +`checkpoints/` objects and `packs/.{pack,idx}` pairs accumulate +without bound. Two costs follow: + +- Storage and listing cost grow with every push. A cold start already stops at + the latest checkpoint (P36b), but it still fetches one `.idx` per on-chain + pack, so the pack count is the cold-start cost. +- `walfmt::MAX_LIST` (65536) caps a checkpoint's `refs` and `packs`. A fold + that would exceed it is skipped (`maybe_fold_checkpoint`), so a remote with + more than 65536 packs loses folding for good and every reader walks an + ever-longer tail. + +The hard part is the cutoff: nothing may be removed that a reader or writer +can still need, with disposable `sc serve --store` instances, concurrent +writers, and no lease primitive (ROADMAP "Leases", unbuilt). + +## Decision + +### No format change + +`Manifest` decodes strictly: `header()` refuses any version other than 1 and +`done()` refuses trailing bytes. A new `compacted_through` field would +therefore make every deployed reader fail closed, which is a format break. It +is also unnecessary: `Manifest.checkpoint_seq` already says where a cold start +stops. Compaction adds no field, no object kind and no `PROTOCOL_VERSION` +bump. Everything it writes is an ordinary record an old client decodes: a log +entry with `updates = []`, and a checkpoint whose `packs` list is a set that +covers the chain instead of a literal accumulation. + +### What compaction does + +`BucketTransport::compact(&CompactOptions)` runs two phases. Each is +independently safe and either may be a no-op. + +**Phase 1, consolidate (additive, fenced by the manifest CAS).** The live pack +set is `view.packs`, deduplicated. Packs smaller than `small_pack_bytes` +(default 16 MiB, sizes from `list_meta`) are grouped greedily, smallest first, +into groups of at most `group_bytes` (default 64 MiB); a group needs at least +two members. Each group is read, its objects are deduplicated by id, and one +new content-addressed pack plus idx is uploaded. This bounds peak RAM to a few +times `group_bytes`, and packs above the threshold are never rewritten. + +The result is committed with one CAS, which is both the fence and the fold: + +1. `refresh`, then recompute the plan if the members are no longer all live. +2. Claim `log/` (`s > head_seq`, same claim loop as `update_ref`) with + `LogEntry { parent_seq: head_seq, packs: [new packs], updates: [] }`. +3. `put_new` `checkpoints/` with `refs = view.refs` and + `packs = dedup(view.packs) - replaced + new`. If this exceeds `MAX_LIST` + the run stops before writing it. +4. CAS the manifest from the tag of the view in step 1 to + `{head_seq: s, checkpoint_seq: s}`. + +The CAS validates that the manifest has not moved since step 1, so the +checkpoint is exactly the state through `s`. A lost CAS leaves an off-chain +entry and checkpoint (ordinary debris) and the run re-plans, bounded retries. +Because the manifest tag changes, every writer holding an older tag loses its +CAS and must refresh. + +Old packs stay in place. After the CAS the superseded packs are named by no +chain a new reader walks, but older checkpoints and log entries still name +them. They are removed by phase 2 once the checkpoint that dropped them has +aged past `grace`, normally a later run. (A leftover checkpoint from a lost +CAS can set that cutoff earlier than the real one, which only costs a reader +that read the manifest moments before a restart; see the reader protocol.) + +**Phase 2, sweep (deletes, behind a cutoff).** All deletions are of objects +that the current manifest chain does not reference (and, outside the residual +limit below, no later chain will); the manifest itself is never deleted. + +- **Cutoff K.** Let `K` be the highest `checkpoints/` object with + `seq <= manifest.checkpoint_seq` whose last-modified time is at least + `grace` ago. No `K` means nothing is eligible by seq (orphan packs may still + be, below). +- **Removed:** `log/` for `seq <= K` (including off-chain debris), + `checkpoints/` for `seq < K`, and pack objects named by nothing + retained and at least `grace` old. +- **Retained set** for packs: the current chain (`view.packs`), every kept + checkpoint (`seq >= K`) and every kept log entry (`seq > K`, on-chain or + not). A pack named by an old checkpoint or a straggler's in-flight entry is + therefore kept. If a retained object cannot be decoded the sweep stops + without deleting packs (fail closed). +- **Order:** log entries, then older checkpoints, then (for each pack) idx, + then pack. Every prefix of this order leaves a bucket a cold start reads, + because none of these are reachable from the manifest being deleted against. + Log entries and checkpoints behind the cutoff are safe whatever the manifest + does meanwhile (the cutoff only grows). Pack deletion alone is gated on the + manifest being unchanged since the pack set was computed; if it moved, the + pack set is replanned (bounded), and otherwise the run keeps the packs and + says so. +- `K` rather than "the current checkpoint" matters on a busy remote: folds + happen every 64 pushes, so the newest checkpoint is rarely `grace` old, + while an older one is. + +`K = 0` is a valid answer on a young remote. A run on a bucket whose newest +checkpoint is younger than `grace` consolidates (if worthwhile) and deletes +nothing; the next run after `grace` deletes. This is deliberate: it is the +two-phase "supersede, then delete after T" scheme without a tombstone object, +using the checkpoint's own creation time as the supersession clock. + +### Reader protocol + +Readers never take a lock and never need compaction to be quiet. The rule: + +> When a key the current view names is absent (checkpoint, log entry, idx, or +> pack body), re-read the manifest unconditionally. If its tag differs from +> the view's, discard the view and restart from the new manifest (bounded +> retries, 8). If it is unchanged, the absence is real corruption and the +> operation fails closed exactly as before. + +This is sound because deletion only follows a manifest CAS that moved the +tag: the sweep deletes only objects unreferenced by the manifest it read, and +manifest tags never repeat (every successful CAS strictly increases +`(head_seq, checkpoint_seq)`, so the content cannot return to an earlier value, +and an S3 ETag changes with the content, whether it is an MD5 or an opaque +SSE value). A reader that lost an object therefore +always finds a newer manifest on re-read. Object ids stay valid across the +restart because consolidation preserves every object, only moving it to a new +pack, so `get_pack`'s id lists computed before a restart remain valid. +Restart is at the object-fetch level (`object_bytes`, the signature/transcript +scan, `refresh`), never in the middle of the bytes `get_pack` has already +written. + +`grace` is therefore not what makes readers correct; it only makes restarts +rare; a reader straddling a sweep can be restarted. + +### Writer protocol + +A push uploads packs, claims a log slot, then CASes the manifest. An +uploaded-but-unreferenced pack is indistinguishable from debris, and an object +store offers no atomic "delete if the manifest is unchanged". Two defenses: + +1. **Age.** The sweep removes an unreferenced pack only if its last-modified + time is at least `grace` old (default 24 h). A writer's gap between upload + and CAS is milliseconds to minutes. +2. **Verify before commit.** `update_ref` checks (`exists`, a HEAD) that each + pending pack and idx still exist immediately before every manifest CAS + (after the log slot is claimed), so pushers need no list rights; + consolidation does the same for the packs it created, and fails with a clear + retry-the-push error if not, instead of committing an entry that names a + deleted pack. The window left is one bucket round trip. + +**Residual limit (documented, not closed).** A writer that stalls longer than +`grace` between uploading a pack and its CAS, or re-pushes byte-identical +content to an old orphan while a sweep that already selected it is still +deleting (the manifest is looked at again every 16 packs), can still +commit an entry naming a deleted pack. Closing that needs a lease or a +conditional delete, which the `Bucket` trait lacks (ROADMAP "Leases"). The +same staleness has a second, milder form: a writer paused longer than `grace` +between reading the manifest and claiming a log slot can claim a slot the +sweep freed behind the cutoff. Its manifest CAS fails, but a reader equally +stale (older than `grace`, still walking the chain it read) could read that +phantom entry instead of seeing the slot absent. Both need a participant stalled +past the window; neither affects anyone who read the manifest within it. Run +compaction with a `grace` well above any plausible writer stall; the 24 h +default matches `sc gc --prune-expire`. + +### MAX_LIST + +Folding is skipped only when the cumulative `packs` (or `refs`) of a fold +exceed 65536. Phase 1 shrinks the pack axis: after consolidation the +checkpoint holds `non-replaced + new` packs, so later folds fit again and the +tail stops growing. The ref axis cannot be relieved: refs are live state, not +history. A remote with more than 65536 live branches stays unfoldable; raising +the cap is a format question and out of scope. Compaction refuses (no writes +beyond debris) if even the consolidated list would exceed the cap. + +Dropping on-chain packs whose objects are unreachable from every ref is +**out of scope**. It changes what `has_object` reports, so a pusher that +skipped re-uploading an object could reference a dropped pack; it also needs a +reachability walk of the whole bucket. A pack is "live" here if the chain names +it. + +### Signatures, transcripts and tombstones (#229, #254) + +The bucket's sender seam finds signatures, transcripts and signed drop +tombstones by decoding every on-chain pack (`pack_leaves`). Compaction keeps +that sound three ways: an on-chain pack is never dropped, only merged; +consolidation decodes and re-encodes every object of a merged pack, whatever its +kind, so the merged pack holds the same leaves; and `pack_leaves` runs against +the post-compaction pack list (the leaf cache is keyed by pack hash and trimmed +to live packs on refresh). `a_clone_after_compaction_still_gets_signatures_transcripts_and_tombstones` +covers a clone and a fetch across two compactions with a signed drop in between. + +### Rollout and permissions + +Upgrade pushers and `sc serve --store` instances before the first `sc remote +compact`: a client without the pre-commit pack check can still lose a pack to +the residual limit above. Pushers and readers need only get, put and head on +objects, but the "absent object" cases (restart on a swept object, the pre-commit +check) rely on a missing key answering 404. S3 answers 403 instead when the +caller lacks `s3:ListBucket`, so grant pushers and readers list as well if you +want the retry messages; without it they fail closed, with an opaque error. + +### Command surface + +`sc remote compact [--dry-run] [--prune-expire ]` (default +`24h`, same flag and default as `sc gc`; a value under 1h is refused unless +`--allow-short-grace` is given, because a short grace turns the residual limit +below into an ordinary race). It is the smallest surface: it needs +only the configured remote's URL, adds no wire verb, and is not reachable +through `sc serve`, so a served bucket's clients can never trigger a delete. +`--dry-run` plans phase 1 and phase 2 against the live bucket and prints what +would be consolidated and deleted (counts, bytes, and how many objects are +held back by the grace window), writing nothing. + +It refuses a non-bucket remote. Rejected alternatives: `sc gc --remote` +(couples two unrelated reachability models), and a `--compact` flag on +`sc serve --store` (servers are disposable and many; a delete should be an +operator action, not a side effect of an instance starting). + +### objio + +The `Bucket` trait gains three methods, implemented by `DirBucket` and +`S3Bucket`: + +- `list_meta(prefix)`: keys with size and last-modified time, raw string + prefix, sorted. S3 uses `ListObjectsV2`. +- `exists(key)`: a HEAD, read rights only. Used by the writer-side check. +- `delete(key)`: idempotent; an absent key is success. On a versioned S3 bucket + a delete only adds a delete marker, so space comes back only with a + lifecycle rule that expires noncurrent versions. + +## Consequences + +- On a non-versioned bucket, storage and cold-start cost are bounded by the + live data plus one grace window (a versioned bucket keeps delete-marked + versions until a lifecycle rule expires them). A bucket past `MAX_LIST` on the pack axis regains folding after one run if + its small packs merge to a list under the cap; packs above the merge + threshold are never rewritten, and the ref axis is not relieved. +- Compaction needs bucket `list` and `delete` rights, which an ordinary + pusher or reader does not. See the THREAT-MODEL bucket section. +- Only one compaction is useful at a time. Concurrent runs are safe (each + recomputes from the manifest it read, the loser's CAS fails) but waste work. +- Consolidation rewrites small packs once. Large packs are never touched, so + there is no write amplification on big pushes. +- Old clients keep working against a compacted bucket. A very old reader may + see one "retry" error if it straddles the sweep, and a re-run succeeds. + +## Alternatives considered + +- **`compacted_through` in the manifest.** Rejected: version-gated field breaks + every deployed reader (strict decode) and is redundant with + `checkpoint_seq`. +- **Tombstone objects (`gc/` listing keys to delete after T).** Needs a + new object kind and a second pass; using the checkpoint's creation time as + the supersession clock gives the same ordering with nothing new in the + bucket. The checkpoint clock can be wrong (a folder stalled between + `put_new` and CAS), but that only costs a reader a restart, not safety. +- **Delete immediately after consolidation.** Simple, correct for readers + (they restart), but makes the writer-stall window as wide as one object's + age and restarts every in-flight clone. The grace window costs one extra run. +- **Repack everything into one pack.** Unbounded RAM and write amplification. + +## As built + +`crates/objio` (`ObjectMeta`, `Bucket::list_meta`/`delete`, contract suite); +`crates/repo/src/bucket_transport/compact.rs` (`compact`, `CompactOptions`, +`CompactReport`); reader restart and the pre-commit pack check in +`bucket_transport.rs`; `sc remote compact` in `crates/cli`. diff --git a/docs/adr/README.md b/docs/adr/README.md index 6a5eb47d..1e1da46f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0043](0043-randomized-protected-encryption.md) | Randomized protected-path encryption (dual-read, randomized-write) | Accepted; rotate-for-paths built as 0051 | 33 | | [0044](0044-per-branch-access-control.md) | Per-branch access control with opaque private branches | Accepted; sparse merge-in section (#240) | 34 | | [0045](0045-native-desktop-read-model.md) | Native desktop adapter and keyless read model | Accepted | 35 | -| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | +| [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted; compaction 0057 | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (built) | 37 | | [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (built) | 37b | | [0049](0049-merge-on-partial-clones.md) | Merge on partial clones | Accepted (built) | 37c | @@ -67,6 +67,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0054](0054-tls-ca-validation.md) | CA-path validation for `sc+https://` | Accepted (built) | — | | [0055](0055-blob-size-clone-filter.md) | Blob-size clone filter (`--filter blob:limit=`) | Accepted (built) | 27 | | [0056](0056-named-concurrent-ws-sessions.md) | Named and concurrent `sc ws` sessions | Accepted (built; refresh designed, deferred) | 20 | +| [0057](0057-bucket-compaction.md) | Bucket remote compaction and gc (`sc remote compact`) | Accepted (built) | 36 | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5,