Skip to content

storage: set a snapshot expiry floor at create_table #91

Open

Description

@codecapo

At a0b5e6c, Warehouse::create_table (crates/storage/src/lib.rs:113-137) puts only the Parquet codec into a new table's properties (:122-130), so every table carries Iceberg's snapshot expiry defaults. Nothing in this repository expires snapshots yet, but any engine that does so under the table's own settings removes every snapshot older than five days except those a branch or tag points at. That can remove two kinds of snapshot that ADR 0006 relies on: the input snapshot that a model's checkpoint names as the from of its next delta read (docs/adr/0006-ordered-job-commits.md:49-51), and the snapshots whose summaries carry the infostash.job-id that, under the ADR, a rerun must find so that it does not commit the same job twice (docs/adr/0006-ordered-job-commits.md:39-40, :47-48). After such an expiry, a model that has not run for six days while its input kept committing finds its checkpoint gone, and appends_between fails with the same Error::NotFound as a snapshot that was never an ancestor (crates/storage/src/lib.rs:564-566, :581-583). This ticket writes history.expire.max-snapshot-age-ms and history.expire.min-snapshots-to-keep at create time, with values above those defaults, as a floor for any expiry that relies on the table's settings. It is a mitigation, not the guarantee. The guarantee is the #75 rule that expiry never goes past the oldest consumer checkpoint, implemented with the Phase 1 snapshot expiry (ROADMAP.md:28). The distinct error for an expired from and its plan-time fallback belong to #89. The orphan sweep in #40 deletes only files that no live snapshot references, so the snapshots this floor keeps also keep their files.

Crate: crates/storage (src/lib.rs and its unit tests)

Done when

  • Warehouse::create_table writes TableProperties::PROPERTY_MAX_SNAPSHOT_AGE_MS and TableProperties::PROPERTY_MIN_SNAPSHOTS_TO_KEEP next to the codec (crates/storage/src/lib.rs:123-126) from two named constants above Iceberg's defaults of five days and one snapshot, so the tables the CSV connector creates on a first run (crates/connectors/src/lib.rs:117) get them as well.
  • The doc comments on create_table and on the two constants say that the values are a floor, not the guarantee: an explicit expire_older_than_ms replaces the age, an explicit retain_last replaces the count, and a branch's own max_snapshot_age_ms and min_snapshots_to_keep take precedence over both the table values and the explicit ones. They name the ADR from docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 as the place where the guarantee is defined.
  • The change does not set gc.enabled to false and creates no tags. With gc.enabled = false every expiry commit fails, so nothing could ever be expired, and a tag keeps only the snapshot it points at, not the ancestors a delta read walks back through.
  • Test: beside new_tables_write_zstd (crates/storage/src/lib.rs:764-774), a newly created table's Warehouse::state holds both keys, and the getters max_snapshot_age_ms() and min_snapshots_to_keep() on the loaded table's metadata().table_properties() return the two constants.
  • Test: after N commits through append_if_head, where N is the count constant, an expire_snapshots() transaction with expire_older_than_ms(i64::MAX) and no retain_last keeps all N snapshots, which fails once the floor is removed (CONTRIBUTING.md:160-161). The transaction is committed the way the_table_codec_is_honoured commits its property change (crates/storage/src/lib.rs:780-790). A second expiry with the same cutoff and retain_last(1) then keeps only the head, which shows that an explicit count replaces the floor.
  • Test: with the second of three commits taken as a consumer checkpoint, an expire_snapshots() transaction with expire_older_than_ms set to that snapshot's timestamp and retain_last(1) keeps that snapshot and the head. With retain_last(1) the floor's count is not what keeps them, so the test shows that the pinned iceberg-rust keeps a snapshot at exactly the cutoff and every later ancestor, as the docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 rule relies on.

Points: 2 (M4 expiry floor at create_table 2)

Depends on: #75 (the ADR that defines the retention rule and calls these properties a floor)
Blocks: nothing

Source: design review of 2026-11-01, finding C05 and section 4.1 (points from section 8.5, M4).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) places the expiry properties written here as a floor under any engine that expires on its own, not the guarantee. The guarantee is the retention rule in the ADR: expiry never goes past the oldest consumer checkpoint or a snapshot committed by a job whose record still exists, implemented with the Phase 1 snapshot expiry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storagecrates/storageblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions