[02/03] service-backup: publish immutable BackupSets manifest-last and retain complete sets #17

Activity

1. chrischeng-c4

   added

   priority:p1Do nextDo next

   phase:created

   agent::claude-code

   lib:service-backup

   type:change

   epic:3029

   depends-on:3034

   on Jul 30, 2026
2. chrischeng-c4

   added this to the service-backup: versioned BackupSet lifecycle - streaming, atomic publication, restart-safe restore milestone on Aug 30, 2026
3. chrischeng-c4

   transferred this issue fromfaberline/axiomon Sep 27, 2026
4. chrischeng-c4 commented on Oct 23, 2026

   

   chrischeng-c4

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Scope of this spec: the first PR of #17 only — R1 (immutable, collision-safe set keys), plus the pure decision logic of R4 and R5 (which sets retention keeps, which partial sets are orphans). The publish order against real sinks (R2, R3), the file/GCS/S3 adapters (R6), and the readback I/O need later specs on this issue, after #16's streaming traits exist. This PR is pure domain code: it adds types and functions, performs no I/O, and changes no existing public item. It does not depend on #16's BackupSetV1: its inputs are plain ids, times and flags.

   Goal

   service-backup gets one place that decides where a backup set lives and what retention may delete, so every backend applies the same rules. Add SetKeyLayout (maps a set id to the immutable keys of its artifacts, its manifest, and the latest pointer), and a pure plan_retention function that, given the sets found in a sink, returns which complete sets to keep, which to delete, and which partial sets are orphans old enough to delete. The rules that matter: the manifest key is the last thing written, so a set with no manifest is partial; retention never deletes the newest keep_latest complete sets, and never deletes a partial set younger than the grace period or one that a complete set references. Later PRs of #17 call these from the publish and prune paths.

   Acceptance

   • New domain files under crates/service-backup/src/domain/ (one main type per file): set_key_layout.rs (SetKeyLayout, SetKeyError), retention_plan.rs (SetInventory, SetState, RetentionPlan, plan_retention). Declare them in src/domain.rs and re-export from src/lib.rs next to the current exports.
   • SetKeyLayout::new(prefix: &str) -> Result<Self, SetKeyError>: prefix is empty or /-separated segments of [A-Za-z0-9._-], none empty, . or .., no leading or trailing /. Keys: artifact_key(set_id, name) = <prefix>/sets/<set_id>/artifacts/<name>, manifest_key(set_id) = <prefix>/sets/<set_id>/manifest.json, latest_key() = <prefix>/latest.json, set_prefix(set_id) = <prefix>/sets/<set_id>/. With an empty prefix the leading <prefix>/ is omitted. set_id uses [A-Za-z0-9._-] without /, non-empty, at most 128 bytes. name uses the artifact-name rule from [01/03] service-backup: define streaming BackupSetV1 manifest and single-object compatibility #16's spec (relative / segments of [A-Za-z0-9._-], no ./.., at most 256 bytes) and may not be manifest.json-shadowing, which the artifacts/ directory already prevents. Invalid input returns SetKeyError (thiserror, one variant per rule), never a panic.
   • SetKeyLayout::parse_set_id(key: &str) -> Option<&str> returns the set id of any key that lies under <prefix>/sets/<id>/ and None for anything else, including latest.json and keys of another prefix. A test shows parse_set_id(artifact_key(id, n)) == Some(id) for ids and names that contain ., -, _ and a nested name a/b.bin.
   • Keys are collision-safe: two different (set_id, name) pairs never give the same key, and no artifact key equals a manifest key or the latest key. A test checks this over a small cross product including names manifest.json and latest.json.
   • SetInventory { set_id: String, state: SetState, newest_object_at: SystemTime } and SetState::{Complete { created_at: SystemTime }, Partial}. Complete means the set's manifest object exists; Partial means objects exist under the set prefix but no manifest.
   • plan_retention(inventory: &[SetInventory], keep_latest: usize, now: SystemTime, orphan_grace: Duration) -> RetentionPlan is pure and deterministic (input order does not change the output; ties on created_at break by set_id). RetentionPlan { keep: Vec<String>, delete_complete: Vec<String>, delete_orphans: Vec<String> }, each sorted by set_id.
   • Rules of plan_retention: the keep_latest newest Complete sets are kept, with keep_latest raised to at least 1 so the plan never deletes the last complete set; older Complete sets go to delete_complete; a Partial set goes to delete_orphans only when now - newest_object_at >= orphan_grace, otherwise it is neither kept nor deleted (left out of all three lists); an empty inventory gives an empty plan; with zero Complete sets nothing in delete_complete is produced.
   • New test module crates/service-backup/tests/it/set_layout.rs, registered in tests/it/main.rs without a feature gate, with one case each for: the key shapes with and without a prefix; each invalid prefix, set id and name; parse_set_id round trip and rejection of latest.json and a foreign prefix; the collision cross product; retention keeps the newest N; keep_latest = 0 still keeps one; a young partial set is untouched; an aged partial set is an orphan; input order does not change the plan; an inventory with only partial sets deletes no complete set.

   Never

   • Do not change BackupSink, SnapshotSource, run_backup_once, LocalFsSink, GcsSink, the S3 sink, fetch_backup_object, or RetentionPolicy. The existing age-based prune stays as it is until a later PR replaces it.
   • Do not do I/O, read clocks or use randomness in the new domain files (std::fs, SystemTime::now, Utc::now); ddd.toml [policy.domain] forbids them. The caller passes now.
   • Do not derive schemars::JsonSchema on the new types, and do not change tests/it/schema_golden.rs or tests/it/wire_golden.rs.
   • Do not add a dependency to crates/service-backup/Cargo.toml.
   • Do not edit README.md, llms.txt, src/api/llm.rs, or docs/ in this PR.
   • Do not touch other crates, ddd.toml, or CI.

   Gate

   cargo test --locked -p service-backup
   cargo test --locked -p service-backup --all-features
   cargo clippy --locked -p service-backup --all-features --all-targets -- -D warnings
   cargo fmt -p service-backup -- --check
5. chrischeng-c4

   added

   todoReady for the work step of the issue loopReady for the work step of the issue loop

   doingThe work step of the issue loop is on itThe work step of the issue loop is on it

   and removed

   priority:p1Do nextDo next

   todoReady for the work step of the issue loopReady for the work step of the issue loop

   on Oct 24, 2026
6. chrischeng-c4 commented on Oct 24, 2026

   

   chrischeng-c4

   on Oct 24, 2026

   ContributorAuthor

   More actions

   Work submitted in #63; waiting for required CI on the current head.
7. chrischeng-c4

   modified the milestones: service-backup: versioned BackupSet lifecycle - streaming, atomic publication, restart-safe restore, core@0.7.0 on Oct 24, 2026