Skip to content

storage: map HeadGuard conflicts and fix its doc #92

Open

Description

@codecapo

Warehouse::append_if_head (crates/storage/src/lib.rs:171-221 at a0b5e6c) commits new rows only if the table's head snapshot is still the one the caller read, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:52-54). It loads the table and checks the head (crates/storage/src/lib.rs:198-202), writes the Parquet files (:203), and commits through commit_if_head (:219, :523-549). commit_if_head wraps the SQL catalog in HeadGuard (:404-449), a catalog whose load_table sets a tripped flag and fails whenever the table's current snapshot id is not the expected head (:431-445). The runner in #13 will reuse the same guard so that a stale run fails instead of committing.

The guard is needed because of how iceberg-rust, at the revision pinned in Cargo.toml:29, commits a transaction. Transaction::commit loads the table through the catalog it is given, rebases the transaction onto that table if the table changed, runs the transaction's actions (an append writes its manifest and manifest list here), and calls the catalog's update_table. SqlCatalog::update_table then loads the table a second time, checks the commit's requirements against it, writes the new metadata.json, and switches the table to it with one SQL UPDATE that matches only the old metadata location, which makes it a compare-and-swap. A failed requirement and a failed compare-and-swap both raise a retryable ErrorKind::CatalogCommitConflicts. Transaction::commit retries such errors up to commit.retry.num-retries times, a count it reads from the table the transaction started from, and every retry begins with a new load through the guard.

A foreign commit can therefore land in one of three windows. In the first, before the guard's load, the head check stops the writer if the foreign commit landed before append_if_head loaded the table, and the guard stops it otherwise. In the second, between the guard's load and the SQL catalog's own load in update_table, the requirement that fast_append adds on the main branch (TableRequirement::RefSnapshotIdMatch) fails. In the third, after that load and before the UPDATE, the compare-and-swap fails. In the second and third windows the retry reloads the table through the guard, the guard trips, and commit_if_head reports Error::Conflict (crates/storage/src/lib.rs:539-546).

This ticket closes four gaps. First, when there is no retry, because commit.retry.num-retries is 0 or the retries are used up, the conflict error reaches commit_if_head with tripped still false and becomes Error::Storage (crates/storage/src/lib.rs:547), so the caller cannot tell a lost race from a broken catalog. Nothing pins the retry count either, because create_table writes only the Parquet codec into a new table's properties (:122-126). Second, the guard compares only the snapshot id (:433-434), so a schema change that lands after the writer's load goes unnoticed: iceberg-rust rebases the commit onto the changed table, and Parquet written with the old schema (:342-343) can end up in a snapshot that records the new schema id. Third, the guard's doc comment (:408-410) credits the compare-and-swap for every commit that lands between the guard's load and the catalog update, although in the second window the branch requirement is what stops it. Fourth, the guard test (:845-859) commits the foreign snapshot before commit_if_head runs, so no test commits inside the second or third window.

The code and its unit tests are all in crates/storage/src/lib.rs, with the tests in mod tests (:654-655). Run them with cargo test -p infostash-storage, and run the checks in CONTRIBUTING.md:29-34 before opening a pull request.

Crate: crates/storage (src/lib.rs only)

Done when

  • The HeadGuard doc comment (crates/storage/src/lib.rs:404-410) names the three windows and what stops a foreign commit in each: the head check at :198-202 or the guard itself before the guard's load, the main branch requirement between the guard's load and the SQL catalog's own load in update_table, and the compare-and-swap on the metadata location after that load. It also says that the last two raise a retryable ErrorKind::CatalogCommitConflicts, and that the retry reloads the table through the guard, which trips, so commit_if_head reports Error::Conflict.
  • The same doc comment says that a transaction which only sets properties carries no branch requirement and is last-writer-wins, so the infostash.* properties (docs/adr/0006-ordered-job-commits.md:49-51) are protected only because append_if_head commits them in one transaction with a fast_append (crates/storage/src/lib.rs:205-218).
  • commit_if_head (crates/storage/src/lib.rs:537-548) also returns Error::Conflict when tripped is false but the error's kind is ErrorKind::CatalogCommitConflicts, reloading the head for the message as the tripped branch does (:539-546). Every other error still becomes Error::Storage (:547), and the doc comment on commit_if_head (:523-524) describes both cases.
  • create_table (crates/storage/src/lib.rs:122-131) also writes commit.retry.num-retries (TableProperties::PROPERTY_COMMIT_NUM_RETRIES) next to the Parquet codec, set to iceberg-rust's default of 4 (TableProperties::PROPERTY_COMMIT_NUM_RETRIES_DEFAULT). A test beside new_tables_write_zstd (:764-774) checks it on a new table the way :768-770 checks the codec. storage: set a snapshot expiry floor at create_table #91 adds two expiry properties to the same map (:123-126), so whichever change lands second keeps the other's properties.
  • HeadGuard also holds the schema id of the table the writer loaded at crates/storage/src/lib.rs:198, whose schema write_files uses (:342-343). It sets tripped and refuses any load whose current schema id differs, as it does for a moved head (:434-443), and its doc comment says so. commit_if_head and the code at :219, :855 and :865-870 are updated to supply the schema id. When the head did not move, the Error::Conflict message says that the schema changed, instead of the two equal head ids that moved (:326-333) would print. A concurrent schema change thus becomes Error::Conflict, which is right under one writer per table (docs/adr/0006-ordered-job-commits.md:73-74).
  • Test: HeadGuard and commit_if_head take the inner catalog as &dyn Catalog instead of &SqlCatalog (crates/storage/src/lib.rs:412, :526), so that a test can put its own catalog under the guard. That test-only catalog, defined in mod tests, delegates every method to the SQL catalog the way the guard does (:447-520), except that its update_table first commits another snapshot directly through the SQL catalog, for example with Warehouse::append as at :850, and then delegates. The test asserts that commit_if_head returns Error::Conflict and that the foreign snapshot is the table's only one. It also counts the files under the table's directory by kind against a count taken right after the table is created, and asserts two new Parquet files, two new manifests, two new manifest lists (the .avro files whose names start with snap-) and one new metadata.json. That is one of each from the foreign commit, plus the losing writer's Parquet file, manifest and manifest list, which are the orphans storage: remove orphan data files #40 will remove.
  • Test: the same race on a table whose commit.retry.num-retries was set to 0 before the writer loaded it, set the way crates/storage/src/lib.rs:780-790 sets the codec (iceberg-rust reads the retry count from the table a transaction starts from), returns Error::Conflict where the code today returns Error::Storage.
  • Test: modeled on crates/storage/src/lib.rs:845-859, a column added after the writer loaded the table, through a transaction such as update_schema().add_column(AddColumn::optional(..)) committed with &wh.catalog, makes commit_if_head return Error::Conflict and leaves the table with no snapshot, where today the commit succeeds.

Points: 2 (M4 HeadGuard conflict mapping and doc fix 2)

Depends on: nothing
Blocks: nothing

Source: design review of 2026-11-01, finding C20.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions