Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
39c45ce
docs(adr): design bucket remote compaction and gc (#216)
tonibergholm-codento Oct 20, 2026
f197af8
feat(objio): list_meta and delete on the Bucket trait for compaction …
tonibergholm-codento Oct 20, 2026
bafea54
feat(repo): bucket remote compaction, reader restart and pre-commit p…
tonibergholm-codento Oct 20, 2026
e3f5234
fix(repo): gate only pack deletion on an unmoved manifest; restart re…
tonibergholm-codento Oct 20, 2026
ae8ca0a
docs: ADR-0046 addendum, threat model, roadmap, capability map for bu…
tonibergholm-codento Oct 20, 2026
dfa8ea0
fix(repo): recheck the ref cap on the state compaction publishes (#216)
tonibergholm-codento Oct 20, 2026
b178992
docs(adr): note the stale-writer slot reuse limit (#216)
tonibergholm-codento Oct 20, 2026
a6b5a18
fix(repo,objio,cli): review fixes - HEAD-based pack check, grace floo…
tonibergholm-codento Oct 20, 2026
09d1fee
fix(repo): sweep orphans without a manifest, bound decoded merge byte…
tonibergholm-codento Oct 20, 2026
ee4442b
merge origin/main into feat/216-bucket-compaction
tonibergholm-codento Oct 20, 2026
5ddfa83
merge origin/main into feat/216-bucket-compaction
tonibergholm-codento Oct 20, 2026
b3cd4cb
test(repo): a clone after compaction still gets signatures, transcrip…
tonibergholm-codento Oct 20, 2026
91597ec
docs: list --allow-short-grace and the stalled-writer limit; qualify …
tonibergholm-codento Oct 20, 2026
359d083
fix(repo): look at the manifest between pack-delete batches; name the…
tonibergholm-codento Oct 20, 2026
1ec2393
merge origin/main; renumber the compaction ADR to 0057 (0055 and 0056…
tonibergholm-codento Oct 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .claude/skills/sc-cli-reference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,22 @@ cargo run --bin sc -- remote add <name> sc+s3://<bucket>/<prefix> # same, on a
# content stays ciphertext); `clone --filter` is
# refused; signatures, transcripts and drop
# tombstones travel like on any native remote (#229)
cargo run --bin sc -- remote compact <name> [--dry-run] [--prune-expire 24h] [--allow-short-grace]
# bucket remotes only (#216, ADR-0057): merge small
# packs behind one manifest CAS, then delete log
# entries, checkpoints and packs nothing names that
# are older than --prune-expire (same flag and default
# as `sc gc`; keep it well above any push's duration).
# Needs bucket list + delete rights. --dry-run prints
# the plan and writes nothing. Safe beside live
# pushes/fetches within --prune-expire (a writer
# stalled longer than that between uploading a pack
# and committing it can lose the pack); run one at a time. A --prune-expire under 1h is refused
# unless --allow-short-grace (only with no pushes
# or fetches running). On a versioned S3
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
# bucket deletes only add delete markers (add a
# noncurrent-version lifecycle rule). Relieves the
# 65536-pack checkpoint cap (not the ref cap)
cargo run --bin sc -- serve --http <addr> <home> --store sc+wal://<dir>|sc+s3://<bucket>/<prefix>
# serve a bucket remote over sc+http(s):// (P36c):
# all content lives in the bucket (instance is
Expand Down
6 changes: 4 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1192,8 +1192,10 @@ the checkpoint and walks only the tail. P36c adds `sc serve --store <url>`: an
identity and scratch space while all served content lives in the bucket, so
instances are disposable. Public content sits in the bucket as plaintext at
rest (the bucket ACL is the perimeter; sealed content stays ciphertext), and
`sc clone --filter` is refused against a bucket remote. Bucket compaction and gc
are not built (open issue #216). A bucket keeps no signature or transcript
`sc clone --filter` is refused against a bucket remote. `sc remote compact` (ADR-0057, #216) merges small packs behind one manifest
CAS, then deletes log entries, checkpoints and unreferenced packs behind a cutoff
that has aged past `--prune-expire`; readers that find a named object gone
re-read the manifest and restart, and there is no format change. A bucket keeps no signature or transcript
index, so its `get_pack` rebuilds the over-send from the packs the manifest
chain names; signatures, transcripts and signed drop tombstones therefore
travel through a bucket like any other remote, but each fresh fetch or clone
Expand Down
10 changes: 8 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ the code, those win.
| P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) |
| P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) |
| P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) |
| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change) | [0046](docs/adr/0046-wal-bucket-remotes.md) |
| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances; `get_pack` serves signatures, transcripts and signed drop tombstones by decoding the on-chain packs (#229, no format change); `sc remote compact <name> [--dry-run] [--prune-expire 24h]` (#216) merges small packs behind one manifest CAS, then deletes log entries, checkpoints and unreferenced packs behind an aged-checkpoint cutoff, readers restart on a moved manifest, no format or wire change | [0046](docs/adr/0046-wal-bucket-remotes.md), [0057](docs/adr/0057-bucket-compaction.md) |
| P37a | Want-by-id fetch: batched `GetObjects` verb (wire v5), opt-in `.sc/promisor-lazy`, `sc cat <ref>:<path>` fills only the absent trees and blob on its path; `Store::get` stays network-free; sealed objects refused by kind; sparse widen and the lifted P27 refusals are P37b | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |
| P37b-i | Presence-based partial clones: push, verify and gc walk what the store holds (`reachable_objects_partial`), not the fetch filter; push checks every gap id against the destination before sending; a view gap fails with `GapInView` before any write (preflight at the top of `materialize`, plus `undo` and `sparse set`); a partial clone fetches from its origin only. 37b-ii: on a lazy, online clone `ensure_view` fills the view closure by id before switch, fast-forward, undo and `sparse set`, `sparse set` may widen past the filter, a by-id reply is capped at `min(--max-pack-size, 2 GiB)` (a constant since #213, independent of the object cap) and a refused batch is split. | [0048](docs/adr/0048-presence-based-partial-clones.md) |
| P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill <path>`); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) |
Expand Down Expand Up @@ -206,6 +206,12 @@ transport-adjacent. The rest, imperatively:
- **Bucket remotes hold public content plaintext at rest** — bucket ACL is
the perimeter (sealed content stays ciphertext, unchanged); partial-clone
`filter` against bucket remotes is refused.
- **`sc remote compact` deletes bucket objects** (ADR-0057): it needs bucket
list and delete rights, which pushers and `sc serve --store` instances should
not hold. It advances the manifest only through the normal CAS (one no-op
entry plus a checkpoint) and never deletes what the current chain names. Keep `--prune-expire` well above any push's duration (default
24h): a writer stalled longer than that between uploading a pack and its
commit can still lose the pack (the leases follow-on closes it).
- **`sc serve --store` still requires a local serve home with `.sc/`** —
tokens, TLS identity, and pack spills live there; the bucket holds all
served content.
Expand Down Expand Up @@ -323,7 +329,7 @@ transport-adjacent. The rest, imperatively:
Deferred work is tracked in **`ROADMAP.md` → Deferred** (single source of
truth — do not re-list items here); entries that name an issue mirror the open
GitHub issues. Standing entries: `sc ws` workspace refresh (#215),
a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), bucket compaction and gc (#216), the non-unix
a limit-aware `sc fetch` and an object-count clone limit (#217 remainder), the non-unix
`.sc/local-key` ACL (#214), and the open
sparse, private-branch and abort edge cases (#241, #246, #247, #248).

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ back to Git commits, and reaches hosted Git (e.g. GitHub) through a spawned
system-git mirror bridge.

See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design,
[docs/adr/](docs/adr/) for the architecture decision records (0001-0054) behind
[docs/adr/](docs/adr/) for the architecture decision records (0001-0057) behind
every phase, and [CLAUDE.md](CLAUDE.md) for project conventions and the
per-phase capability map. [CONTRIBUTING.md](CONTRIBUTING.md) covers the
development workflow.
Expand Down Expand Up @@ -170,7 +170,7 @@ and Git (including hosted GitHub) network transports, plus private branches
sealed to a recipient set until publish and a read-only Tauri desktop
browser. All 36 phases and P37a-37d are implemented and tested; `demo/` holds
28 independent end-to-end proofs, one per major capability. Remaining
follow-ons (transcript drop delivery gaps, bucket compaction and gc, `sc ws`
follow-ons (transcript drop delivery gaps, `sc ws`
workspace refresh, a limit-aware fetch and an object-count clone limit, and similar hardening
items) are tracked in [ROADMAP.md](ROADMAP.md)'s Deferred section and as open
GitHub issues rather than being open MVP scope.
22 changes: 7 additions & 15 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -577,7 +577,9 @@ across every phase.
Public content is plaintext at rest (the bucket ACL is the perimeter);
`clone --filter` is refused. `get_pack` serves signatures, transcripts and
signed drop tombstones by decoding the on-chain packs (#229, no format
change). (ADR-0046.)
change). `sc remote compact` (#216) merges small packs and deletes the log
entries, checkpoints and unreferenced packs behind an aged cutoff; readers
restart on a moved manifest; no format change. (ADR-0046, ADR-0057.)
- **Phase 37a — Want-by-id fetch.** A batched `GetObjects` verb (wire v5), an
opt-in `.sc/promisor-lazy` marker, and `sc cat <ref>:<path>`, which fills only
the absent trees and the blob on its path. `Store::get` stays network-free.
Expand Down Expand Up @@ -672,7 +674,7 @@ under Deferred below.
| **P33 — Randomized protected encryption** | Close equality confirmation for new protected content | fresh DEK + nonce per seal, dual-read/randomized-write, keyed local stat cache, rewrap upgrade | [0043](docs/adr/0043-randomized-protected-encryption.md) |
| **P34 — Private branches** | Keep an entire branch opaque before publication | sealed object closure behind a manifest; O(1) grant, KEK-rotating revoke, scanner-gated publish | [0044](docs/adr/0044-per-branch-access-control.md) |
| **P35 — Native desktop browser** | Prove native `.sc` objects can power a polished visual tool | Tauri app opens a repository and browses refs, snapshot DAG/provenance, public trees, and first-parent diffs while locked/private states remain safe | [0045](docs/adr/0045-native-desktop-read-model.md) |
| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add <name> sc+wal://<dir>` or `sc+s3://<bucket>/<prefix>`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store <url>` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md) |
| **P36 — Bucket WAL remotes** | Sync through an object store with no long-lived server | `sc remote add <name> sc+wal://<dir>` or `sc+s3://<bucket>/<prefix>`; immutable packs plus one CAS'd manifest, checkpoints for cold start (P36b), `sc serve --store <url>` for a disposable bucket-backed server (P36c); signatures, transcripts and drop tombstones travel (#229); `sc remote compact` prunes and merges (#216); covered by unit and integration tests, no demo script | [0046](docs/adr/0046-wal-bucket-remotes.md), [0057](docs/adr/0057-bucket-compaction.md) |
| **P37a — Want-by-id fetch (`sc cat`)** | Read outside a partial clone's filter without backfilling everything | `sc clone --filter src/ --lazy`; `sc cat main:docs/api/guide.bin` fetches exactly the absent trees and the blob by object id (`GetObjects`, wire v5), sibling untouched; `SC_OFFLINE=1` and non-lazy clones keep the gap error; proven by `demo/run_lazy_fetch_demo.sh` | [0047](docs/adr/0047-want-by-id-fetch-and-gap-fill.md) |
| **P37b — Presence-based partial clones** | Make push, verify, gc and the view agree on what a partial clone holds | `reachable_objects_partial` walks the store, not the filter; `GapInView` before any write; on a lazy, online clone the view closure is filled by id before switch, fast-forward, undo and `sparse set` (which may widen past the filter) | [0048](docs/adr/0048-presence-based-partial-clones.md) |
| **P37c — Merge on partial clones** | `sc merge`, `sc cherry-pick` and `sc rebase` without a full clone | tree-recursive engine carries subtrees by id; conflicts in the view are marked and resolved with `sc conflicts`/`sc resolve`; conflicts outside the view are refused before any write | [0049](docs/adr/0049-merge-on-partial-clones.md) |
Expand Down Expand Up @@ -897,22 +899,12 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in
fetch-time freshness attestation so a hostile remote can't re-serve a
stale pre-revoke manifest — the same gittuf-shaped signed-ref effort the
P22 provenance boundary already defers.
- **Bucket compaction/gc (#216, P36a follow-on).** The WAL log (`log/<seq>` keys,
each a hand-rolled versioned binary `walfmt` entry, not protobuf) grows
unboundedly with no compaction — a cold-start reader now stops at the
latest checkpoint (P36b) instead of walking to `0`, but no log entry,
checkpoint, or superseded pack is ever actually removed from the bucket.
Deferred until a safe pruning cutoff (e.g. "no reader can still need
anything before checkpoint N") is designed. Compaction is also what
relieves `walfmt::MAX_LIST` (65536): a checkpoint fold with more refs or
packs than that cap is skipped outright (`maybe_fold_checkpoint`'s guard,
P36b review) rather than writing an object `Checkpoint::decode` would then
refuse to read back, so a remote whose live ref/pack count grows past the
cap loses folding entirely until compaction can retire entries below it.
- **Leases (P36a follow-on).** The only cross-writer coordination today is
the manifest's compare-and-swap; there is no lease/TTL primitive for
operations that need to hold exclusive intent across more than one bucket
round-trip (e.g. a long-running compaction). Deferred.
round-trip (e.g. a long-running compaction). Deferred. It would also close
the one residual hazard ADR-0057 documents: a writer stalled longer than
`sc remote compact --prune-expire` between uploading a pack and committing it.
- **Derived-data layer (Scion-inspired, after P37).** Blame, path history,
commit-graph skip lists and copy-detection computed by one generic
recursive mechanism over snapshots, stored outside the CAS and rebuildable
Expand Down
82 changes: 82 additions & 0 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -833,6 +833,23 @@ enum RemoteOp {
},
/// List configured remotes.
List,
/// Compact a bucket remote (`sc+wal://`, `sc+s3://`): merge small packs,
/// then delete log entries, checkpoints and packs no reader or writer can
/// still need (ADR-0057). Needs bucket list and delete rights.
Compact {
name: String,
/// Plan and report; write and delete nothing.
#[arg(long)]
dry_run: bool,
/// Delete only what has been unreferenced for at least this long
/// (e.g. 24h, 7d). Keep it well above any push's duration.
#[arg(long, default_value = "24h")]
prune_expire: String,
/// Allow a --prune-expire under 1h. Only safe while nothing pushes to
/// or fetches from the bucket: a push in flight can lose its pack.
#[arg(long)]
allow_short_grace: bool,
},
}

/// Top-level `sc serve` subcommand namespace, distinct from the plain
Expand Down Expand Up @@ -4159,6 +4176,36 @@ fn run_remote(op: RemoteOp) -> Result<()> {
println!("added remote {name} -> {url}");
}
}
RemoteOp::Compact {
name,
dry_run,
prune_expire,
allow_short_grace,
} => {
let grace = parse_duration(&prune_expire)?;
if grace < std::time::Duration::from_secs(3600) && !allow_short_grace {
anyhow::bail!(
"--prune-expire under 1h can delete a pack an in-flight push has uploaded but \
not committed; pass --allow-short-grace only if nothing is pushing or fetching"
);
}
let url = repo
.remotes()?
.into_iter()
.find(|(n, _)| *n == name)
.map(|(_, u)| u)
.ok_or_else(|| anyhow::anyhow!("no such remote: {name}"))?;
if !(url.starts_with("sc+wal://") || url.starts_with("sc+s3://")) {
anyhow::bail!("remote '{name}' is not a bucket remote ({url}); nothing to compact");
}
let transport = scl_repo::BucketTransport::open(&url)?;
let rep = transport.compact(&scl_repo::CompactOptions {
prune_expire: grace,
dry_run,
..Default::default()
})?;
print_compact_report(&name, &url, &rep);
}
RemoteOp::List => {
let cfg = scl_repo::RemoteConfig::load(repo.layout())?;
for (name, url) in repo.remotes()? {
Expand All @@ -4173,6 +4220,41 @@ fn run_remote(op: RemoteOp) -> Result<()> {
Ok(())
}

/// Human-readable summary of a bucket compaction (`dry_run` reads "would").
fn print_compact_report(name: &str, url: &str, rep: &scl_repo::CompactReport) {
let (merge, delete) = if rep.dry_run {
("would merge", "would delete")
} else {
("merged", "deleted")
};
println!(
"compact {name} ({url}){}",
if rep.dry_run { " [dry run]" } else { "" }
);
println!(
" {merge} {} pack(s) into {}",
rep.packs_merged, rep.packs_created
);
println!(
" {delete} {} log entr{}, {} checkpoint(s), {} pack(s) ({} bytes), cutoff checkpoint {}",
rep.logs_deleted,
if rep.logs_deleted == 1 { "y" } else { "ies" },
rep.checkpoints_deleted,
rep.packs_deleted,
rep.bytes_deleted,
rep.cutoff
);
if rep.packs_held_back > 0 {
println!(
" kept {} unreferenced pack(s) younger than the grace window",
rep.packs_held_back
);
}
for note in &rep.notes {
println!(" note: {note}");
}
}

/// Fetch from `remote`, dispatching on its configured kind: a `git`-kind
/// remote gets a full-history import into a remote-tracking ref (see
/// `run_fetch_git`); an `sc`-kind remote (or unconfigured, defaulting to `sc`)
Expand Down
Loading
Loading