You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Remove data files that no snapshot references. Failed task attempts, crashes before a commit, and expired snapshots all leave Parquet files on disk (ADR 0004). iceberg-rust has ExpireSnapshotsAction but no orphan-file action.
Crate:crates/storage
Done when
Lists files under a table's location that no live snapshot references and deletes those older than a minimum age (default 3 days), so in-flight jobs are never touched
Dry-run mode lists what it would delete
Test: a failed attempt's staged file is removed; committed files and recent files are kept
Depends on:#13 (file naming by job, task and attempt) Blocks: nothing in the quick start
The design review of 2026-11-01 touches this issue. It widens the sweep from orphan Parquet files to every file under the table location, old metadata.json files included, makes it keep the files of jobs still running whatever their age, and ties what it may delete to the snapshot retention rule in #75.
Suggested additions to the Done-when:
A file under the table location is live, and kept, when it is the current metadata.json or one that its metadata_log() names, or when any snapshot still in the current metadata, not only the current snapshot, reaches it through its manifest list and manifests; every other file is a candidate for deletion, whatever its kind. Besides Parquet files, a commit that loses the race can leave a manifest, a manifest list and a staged metadata.json, which keep iceberg-rust's own names, so reachability and age are what identify them. Test: with one unreferenced file of each kind (Parquet, manifest, manifest list, metadata.json) older than the minimum age (the grace period), a sweep removes exactly those four, and the table still reads back the same rows.
Besides the minimum age, the sweep is given the ids of the jobs still running and keeps any file whose name carries one of them, whatever its age. File names carry the job id once In-process runner: stage, run tasks, commit once #13 (the runner) and connectors: stage under a job prefix and run checks before the writer #82 (the connector path) land; today write_files names each write's files after a fresh UUID (crates/storage/src/lib.rs:354-359 at a0b5e6c), where ADR 0006 expects names that identify the job, task and attempt so that this cleanup can find what failed attempts and crashes leave (docs/adr/0006-ordered-job-commits.md:33-36, :77-78). Test: a sweep with a grace period of zero, run while a job has staged files it has not committed, keeps them.
Old metadata files are deleted too. Every commit writes a complete new metadata.json and so adds one entry to the metadata log, as rows_and_properties_land_in_one_commit asserts (crates/storage/src/lib.rs:816), and each file lists every snapshot with its summary, so the files grow with the table's history. The pinned iceberg-rust (Cargo.toml:29) drops log entries past write.metadata.previous-versions-max (default 100) without deleting their files, and nothing in this repository deletes them. The sweep deletes each metadata.json that is neither current nor named in metadata_log(), under the same grace period, because a commit in flight writes its new metadata.json before the catalog swap. Test: with that property set to 2, ten commits and then a sweep with a grace period of zero leave only the current metadata.json and the two its log names, and every row still reads back.
The sweep never expires snapshots itself; it is the step that makes an expiry final. Snapshot expiry in the pinned iceberg-rust only rewrites the metadata and leaves every file in place, so what the sweep may delete after an expiry is bounded by the docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 rule that expiry never goes past the oldest consumer checkpoint, with the storage: set a snapshot expiry floor at create_table #91 floor behind it, and every snapshot they retain holds on to its files through the liveness rule above. Test: with S1 to S3 committed at distinct timestamps and a consumer checkpoint at S2, an expiry with retain_last(1) and its cutoff at S2's timestamp, then a sweep with a grace period of zero, removes only S1's manifest list, and the delta read from S2 to S3 (Warehouse::scan_range, crates/storage/src/lib.rs:239) still returns exactly S3's rows.
Suggested re-score: no change, since section 8.5 of the design review of 2026-11-01 scores this sweep with its grace period and metadata-file deletion at 5, as #19 does, within milestone M4 at 10 (the sweep 5, the expiry floor at create_table 2 in #91, the HeadGuard conflict mapping and doc fix 2 in #92, three deny-list tests and a catalog.db existence check 1 in #93); #19 decides.
Source: design review of 2026-11-01, findings C05, C10 and C13, sections 4.2 to 4.4, and section 8.5 (D4 and M4).
ADR 0009 (PR #103) adds one item to the scope here: the old metadata.json files that nothing deletes today. Every commit writes a new one and the directory keeps them all, so they belong with the orphan files of abandoned jobs that this ticket removes.
Scope addition from the review of RFC 0002 (#102): a job record under .infostash/jobs/ is now kept with its Committed outcome after the last commit instead of being deleted. The sweep removes such records together with the staging manifests older than the minimum age. Records without a Committed outcome are left alone; they belong to the rerun path.
Activity
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It widens the sweep from orphan Parquet files to every file under the table location, old
metadata.jsonfiles included, makes it keep the files of jobs still running whatever their age, and ties what it may delete to the snapshot retention rule in #75.Suggested additions to the Done-when:
metadata.jsonor one that itsmetadata_log()names, or when any snapshot still in the current metadata, not only the current snapshot, reaches it through its manifest list and manifests; every other file is a candidate for deletion, whatever its kind. Besides Parquet files, a commit that loses the race can leave a manifest, a manifest list and a stagedmetadata.json, which keep iceberg-rust's own names, so reachability and age are what identify them. Test: with one unreferenced file of each kind (Parquet, manifest, manifest list,metadata.json) older than the minimum age (the grace period), a sweep removes exactly those four, and the table still reads back the same rows.write_filesnames each write's files after a fresh UUID (crates/storage/src/lib.rs:354-359at a0b5e6c), where ADR 0006 expects names that identify the job, task and attempt so that this cleanup can find what failed attempts and crashes leave (docs/adr/0006-ordered-job-commits.md:33-36,:77-78). Test: a sweep with a grace period of zero, run while a job has staged files it has not committed, keeps them.metadata.jsonand so adds one entry to the metadata log, asrows_and_properties_land_in_one_commitasserts (crates/storage/src/lib.rs:816), and each file lists every snapshot with its summary, so the files grow with the table's history. The pinned iceberg-rust (Cargo.toml:29) drops log entries pastwrite.metadata.previous-versions-max(default 100) without deleting their files, and nothing in this repository deletes them. The sweep deletes eachmetadata.jsonthat is neither current nor named inmetadata_log(), under the same grace period, because a commit in flight writes its newmetadata.jsonbefore the catalog swap. Test: with that property set to 2, ten commits and then a sweep with a grace period of zero leave only the currentmetadata.jsonand the two its log names, and every row still reads back.retain_last(1)and its cutoff at S2's timestamp, then a sweep with a grace period of zero, removes only S1's manifest list, and the delta read from S2 to S3 (Warehouse::scan_range,crates/storage/src/lib.rs:239) still returns exactly S3's rows.Work that moved to new tickets:
create_tableas a floor, where today only the Parquet codec is set (crates/storage/src/lib.rs:122-130).Suggested re-score: no change, since section 8.5 of the design review of 2026-11-01 scores this sweep with its grace period and metadata-file deletion at 5, as #19 does, within milestone M4 at 10 (the sweep 5, the expiry floor at
create_table2 in #91, theHeadGuardconflict mapping and doc fix 2 in #92, three deny-list tests and acatalog.dbexistence check 1 in #93); #19 decides.Source: design review of 2026-11-01, findings C05, C10 and C13, sections 4.2 to 4.4, and section 8.5 (D4 and M4).
codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) adds one item to the scope here: the old
metadata.jsonfiles that nothing deletes today. Every commit writes a new one and the directory keeps them all, so they belong with the orphan files of abandoned jobs that this ticket removes.codecapo commentedon Nov 10, 2026
Scope addition from the review of RFC 0002 (#102): a job record under
.infostash/jobs/is now kept with itsCommittedoutcome after the last commit instead of being deleted. The sweep removes such records together with the staging manifests older than the minimum age. Records without aCommittedoutcome are left alone; they belong to the rerun path.docs: record the replace commit route (ADR 0007, ADR 0008) (#100)
feat(storage): replace live data files in one overwrite snapshot (#106)