Repository navigation
Description
Name the Parquet files that a commit stages after the job that writes them, and check every batch against the table before the first file exists. At a0b5e6c, write_files names each write's files after a fresh random UUID (crates/storage/src/lib.rs:354-359). A file that a failed commit leaves behind, such as the one from the lost commit in the guard test, which the test never inspects (:852-858), therefore cannot be tied to a job. ADR 0006 requires staged file names that include the job, task and attempt, so that #40 can find what a failed attempt left (docs/adr/0006-ordered-job-commits.md:33-36, :77-78). The same function also compares each batch's column names with the table's (crates/storage/src/lib.rs:373) only after it has built the writer (:361) and written every earlier batch (:383). The iceberg-rust revision pinned in Cargo.toml:29 creates a Parquet file on the first write, so a mismatch in any batch after the first leaves a partial file that no snapshot references.
Every name keeps a random UUID suffix. The name generator counts up from 0 on every write (crates/storage/src/lib.rs:354) and iceberg-rust's local file writer truncates an existing file, so without the suffix a second write under the same job id would overwrite a committed file. The commit UUID stays random too, because a UUID derived from the job or the attempt would let iceberg-rust's commit retry rewrite a manifest that an earlier try may have committed. Manifests, manifest lists and metadata.json keep iceberg-rust's own names, and #40 finds them by reachability and age. #13 still owns the runner that supplies the task and attempt, and the staging manifest. Files left by a write that fails part-way on I/O are for #40 to remove. #79 checks CSV headers before any row is read, and the storage check here backs it up for every caller.
Crates: crates/storage (write_files, Warehouse::append_if_head and their tests), crates/connectors (Prepared::commit)
Done when
-
write_files(crates/storage/src/lib.rs:337-341) takes an optional job id and, given one, puts it in the name generator's prefix and a fresh random UUID in its suffix (nowNoneat:357), so its files are named{job}-{n:05}-{uuid}.parquet, and its doc comment (:335-336) states that format and that manifests, manifest lists andmetadata.jsonkeep iceberg-rust's own names. -
Prepared::commitsupplies the job id it already commits asinfostash.job-id(crates/connectors/src/lib.rs:125-132) toappend_if_headas a new argument,append_if_headhands it towrite_files(crates/storage/src/lib.rs:203), andWarehouse::append, which callswrite_filesat:142without a job id, keeps today's bare UUID until storage: make append crate-private #83 takes it out of the public API. -
write_filesruns the column-name check (crates/storage/src/lib.rs:373) and theRecordBatch::try_newconversion (:380-381) on every batch before it builds the writer (:361), so a batch that does not match the table fails before any file exists. - Once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted,
write_filesalso takes the run id, task and attempt that RFC defines and puts them after the job id in the prefix, in the order it sets (proposed:{job}-{run}-{stage}-{partition}-{attempt}). The runner of In-process runner: stage, run tasks, commit once #13 supplies these values; this ticket only takes and formats them. The doc comment follows the new format, and a test shows two attempts of one task writing under different prefixes. - Test: next to the single-batch case at
crates/storage/src/lib.rs:941-957, anappend_if_headcall whose third batch has other column names fails withError::Storage, commits nothing and adds no file to the table's data directory. - Test: the guard test
the_guard_stops_a_commit_that_lands_after_the_writer_loaded(crates/storage/src/lib.rs:845-859) handswrite_filesa job id and also asserts that the file its lost commit leaves in the data directory carries that job id in its name. - Test, at the storage level and apart from the rerun skip of connectors: skip tables that already carry the job id and use distinct test ids #81: two
append_if_headcalls on one table under the same job id, as a reused job or attempt key would make, write files with different names, and the first commit's rows still read back after the second.
Points: 3 (C13 3)
Depends on: #74 (the attempt in staged names only)
Blocks: nothing
Source: design review of 2026-11-01, finding C13.
Activity
codecapo commentedon Nov 10, 2026
RFC 0002 (#102) sets the staged file name as
{job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet, the order proposed above, and bundles the five leading fields asAttemptKey { job, run, stage, partition, attempt }incrates/model.append_if_headand the newWarehouse::stagetake anAttemptKeyrather than an optional job id; a connector load builds its key withAttemptKey::single(job, run)(stage 0, partition 0, attempt 1; attempts count from 1 as inrun_in_process), andconnectors::preparetakes theRunIdnext to theJobId.stagealso writes{job}-{run}-{stage}-{partition}-{attempt}.staged.jsonnext to the data files, which #40 can use to tell a complete attempt from a half-written one. #40 should also remove manifests older than its minimum age whether or not their files are referenced, since a crash between the commit and the delete leaves one behind.docs: add RFC 0002 for the job and runner contracts (#102)