Repository navigation
Description
At a0b5e6c, Warehouse::create_table (crates/storage/src/lib.rs:113-137) puts only the Parquet codec into a new table's properties (:122-130), so every table carries Iceberg's snapshot expiry defaults. Nothing in this repository expires snapshots yet, but any engine that does so under the table's own settings removes every snapshot older than five days except those a branch or tag points at. That can remove two kinds of snapshot that ADR 0006 relies on: the input snapshot that a model's checkpoint names as the from of its next delta read (docs/adr/0006-ordered-job-commits.md:49-51), and the snapshots whose summaries carry the infostash.job-id that, under the ADR, a rerun must find so that it does not commit the same job twice (docs/adr/0006-ordered-job-commits.md:39-40, :47-48). After such an expiry, a model that has not run for six days while its input kept committing finds its checkpoint gone, and appends_between fails with the same Error::NotFound as a snapshot that was never an ancestor (crates/storage/src/lib.rs:564-566, :581-583). This ticket writes history.expire.max-snapshot-age-ms and history.expire.min-snapshots-to-keep at create time, with values above those defaults, as a floor for any expiry that relies on the table's settings. It is a mitigation, not the guarantee. The guarantee is the #75 rule that expiry never goes past the oldest consumer checkpoint, implemented with the Phase 1 snapshot expiry (ROADMAP.md:28). The distinct error for an expired from and its plan-time fallback belong to #89. The orphan sweep in #40 deletes only files that no live snapshot references, so the snapshots this floor keeps also keep their files.
Crate: crates/storage (src/lib.rs and its unit tests)
Done when
-
Warehouse::create_tablewritesTableProperties::PROPERTY_MAX_SNAPSHOT_AGE_MSandTableProperties::PROPERTY_MIN_SNAPSHOTS_TO_KEEPnext to the codec (crates/storage/src/lib.rs:123-126) from two named constants above Iceberg's defaults of five days and one snapshot, so the tables the CSV connector creates on a first run (crates/connectors/src/lib.rs:117) get them as well. - The doc comments on
create_tableand on the two constants say that the values are a floor, not the guarantee: an explicitexpire_older_than_msreplaces the age, an explicitretain_lastreplaces the count, and a branch's ownmax_snapshot_age_msandmin_snapshots_to_keeptake precedence over both the table values and the explicit ones. They name the ADR from docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 as the place where the guarantee is defined. - The change does not set
gc.enabledto false and creates no tags. Withgc.enabled = falseevery expiry commit fails, so nothing could ever be expired, and a tag keeps only the snapshot it points at, not the ancestors a delta read walks back through. - Test: beside
new_tables_write_zstd(crates/storage/src/lib.rs:764-774), a newly created table'sWarehouse::stateholds both keys, and the gettersmax_snapshot_age_ms()andmin_snapshots_to_keep()on the loaded table'smetadata().table_properties()return the two constants. - Test: after N commits through
append_if_head, where N is the count constant, anexpire_snapshots()transaction withexpire_older_than_ms(i64::MAX)and noretain_lastkeeps all N snapshots, which fails once the floor is removed (CONTRIBUTING.md:160-161). The transaction is committed the waythe_table_codec_is_honouredcommits its property change (crates/storage/src/lib.rs:780-790). A second expiry with the same cutoff andretain_last(1)then keeps only the head, which shows that an explicit count replaces the floor. - Test: with the second of three commits taken as a consumer checkpoint, an
expire_snapshots()transaction withexpire_older_than_msset to that snapshot's timestamp andretain_last(1)keeps that snapshot and the head. Withretain_last(1)the floor's count is not what keeps them, so the test shows that the pinned iceberg-rust keeps a snapshot at exactly the cutoff and every later ancestor, as the docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 rule relies on.
Points: 2 (M4 expiry floor at create_table 2)
Depends on: #75 (the ADR that defines the retention rule and calls these properties a floor)
Blocks: nothing
Source: design review of 2026-11-01, finding C05 and section 4.1 (points from section 8.5, M4).
Activity
codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) places the expiry properties written here as a floor under any engine that expires on its own, not the guarantee. The guarantee is the retention rule in the ADR: expiry never goes past the oldest consumer checkpoint or a snapshot committed by a job whose record still exists, implemented with the Phase 1 snapshot expiry.