Skip to content

connectors: stage under a job prefix and run checks before the writer #82

Open

Description

@codecapo

Name the Parquet files that a commit stages after the job that writes them, and check every batch against the table before the first file exists. At a0b5e6c, write_files names each write's files after a fresh random UUID (crates/storage/src/lib.rs:354-359). A file that a failed commit leaves behind, such as the one from the lost commit in the guard test, which the test never inspects (:852-858), therefore cannot be tied to a job. ADR 0006 requires staged file names that include the job, task and attempt, so that #40 can find what a failed attempt left (docs/adr/0006-ordered-job-commits.md:33-36, :77-78). The same function also compares each batch's column names with the table's (crates/storage/src/lib.rs:373) only after it has built the writer (:361) and written every earlier batch (:383). The iceberg-rust revision pinned in Cargo.toml:29 creates a Parquet file on the first write, so a mismatch in any batch after the first leaves a partial file that no snapshot references.

Every name keeps a random UUID suffix. The name generator counts up from 0 on every write (crates/storage/src/lib.rs:354) and iceberg-rust's local file writer truncates an existing file, so without the suffix a second write under the same job id would overwrite a committed file. The commit UUID stays random too, because a UUID derived from the job or the attempt would let iceberg-rust's commit retry rewrite a manifest that an earlier try may have committed. Manifests, manifest lists and metadata.json keep iceberg-rust's own names, and #40 finds them by reachability and age. #13 still owns the runner that supplies the task and attempt, and the staging manifest. Files left by a write that fails part-way on I/O are for #40 to remove. #79 checks CSV headers before any row is read, and the storage check here backs it up for every caller.

Crates: crates/storage (write_files, Warehouse::append_if_head and their tests), crates/connectors (Prepared::commit)

Done when

  • write_files (crates/storage/src/lib.rs:337-341) takes an optional job id and, given one, puts it in the name generator's prefix and a fresh random UUID in its suffix (now None at :357), so its files are named {job}-{n:05}-{uuid}.parquet, and its doc comment (:335-336) states that format and that manifests, manifest lists and metadata.json keep iceberg-rust's own names.
  • Prepared::commit supplies the job id it already commits as infostash.job-id (crates/connectors/src/lib.rs:125-132) to append_if_head as a new argument, append_if_head hands it to write_files (crates/storage/src/lib.rs:203), and Warehouse::append, which calls write_files at :142 without a job id, keeps today's bare UUID until storage: make append crate-private #83 takes it out of the public API.
  • write_files runs the column-name check (crates/storage/src/lib.rs:373) and the RecordBatch::try_new conversion (:380-381) on every batch before it builds the writer (:361), so a batch that does not match the table fails before any file exists.
  • Once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted, write_files also takes the run id, task and attempt that RFC defines and puts them after the job id in the prefix, in the order it sets (proposed: {job}-{run}-{stage}-{partition}-{attempt}). The runner of In-process runner: stage, run tasks, commit once #13 supplies these values; this ticket only takes and formats them. The doc comment follows the new format, and a test shows two attempts of one task writing under different prefixes.
  • Test: next to the single-batch case at crates/storage/src/lib.rs:941-957, an append_if_head call whose third batch has other column names fails with Error::Storage, commits nothing and adds no file to the table's data directory.
  • Test: the guard test the_guard_stops_a_commit_that_lands_after_the_writer_loaded (crates/storage/src/lib.rs:845-859) hands write_files a job id and also asserts that the file its lost commit leaves in the data directory carries that job id in its name.
  • Test, at the storage level and apart from the rerun skip of connectors: skip tables that already carry the job id and use distinct test ids #81: two append_if_head calls on one table under the same job id, as a reused job or attempt key would make, write files with different names, and the first commit's rows still read back after the second.

Points: 3 (C13 3)

Depends on: #74 (the attempt in staged names only)
Blocks: nothing

Source: design review of 2026-11-01, finding C13.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    RFC 0002 (#102) sets the staged file name as {job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet, the order proposed above, and bundles the five leading fields as AttemptKey { job, run, stage, partition, attempt } in crates/model. append_if_head and the new Warehouse::stage take an AttemptKey rather than an optional job id; a connector load builds its key with AttemptKey::single(job, run) (stage 0, partition 0, attempt 1; attempts count from 1 as in run_in_process), and connectors::prepare takes the RunId next to the JobId. stage also writes {job}-{run}-{stage}-{partition}-{attempt}.staged.json next to the data files, which #40 can use to tell a complete attempt from a half-written one. #40 should also remove manifests older than its minimum age whether or not their files are referenced, since a crash between the commit and the delete leaves one behind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions