Repository navigation
Description
At a0b5e6c, Warehouse::append (crates/storage/src/lib.rs:140-151) is public and commits with a plain fast_append. It skips what ADR 0006 asks of every job commit: the job id as infostash.job-id in the snapshot summary (docs/adr/0006-ordered-job-commits.md:39-40), a connector's read position or a model's last processed snapshot written in the same commit as the rows (docs/adr/0006-ordered-job-commits.md:49-51), and a check that the table head has not moved (docs/adr/0006-ordered-job-commits.md:52-54). A caller that commits through it leaves the stored position where it was, so the next run reads the same input and loads those rows a second time. Without the job id, a rerun cannot see that its job already committed, so it commits again (docs/adr/0006-ordered-job-commits.md:47-48). Without the head check, a run that lost a race is rebased onto the new head and duplicates rows instead of failing (docs/adr/0006-ordered-job-commits.md:18-21). Every production write already goes through the guarded append_if_head (crates/storage/src/lib.rs:181-221): Prepared::commit, the commit path every connector shares, calls it with the new position and the job id (crates/connectors/src/lib.rs:125-130), and only tests still call append. The ADR itself notes that append predates the decision (docs/adr/0006-ordered-job-commits.md:56-58). This issue takes append out of the public API before the runner in #13 adds more code that commits. The matching rule for datafusion-iceberg's insert path, that model runs never commit through its IcebergCommitExec, belongs to the execution RFC in #74, not to this issue.
Crate: crates/storage (src/lib.rs and tests/iceberg_roundtrip.rs)
Done when
-
Warehouse::appendis no longer reachable outsidecrates/storage. Either it becomes a private#[cfg(test)]method, which the unit tests inmod tests(crates/storage/src/lib.rs:654-655) can still call because a child module can use its parent's private items, or it is deleted and the unit tests atcrates/storage/src/lib.rs:680,:772,:792,:850and:952callappend_if_headinstead. A plainpub(crate)is not enough: only tests would call it, socargo clippy --workspace --all-targets -- -D warnings(CONTRIBUTING.md:33) fails on it as dead code, andCONTRIBUTING.md:162-164discourages silencing a check with a new#[allow]. - The
appendcalls at lines 60, 67, 142, 151, 177, 201, 204, 206, 228 and 229 ofcrates/storage/tests/iceberg_roundtrip.rsbecomeappend_if_headcalls, since that file builds as its own crate and can no longer seeappend. Each call supplies the expected head, which isNoneon a table with no commits and otherwise the previous commit's snapshot (such asSome(first.snapshot)at line 67), and an emptyHashMapof properties, which needsuse std::collections::HashMap;beside the imports at lines 8-14. - The missing-table assertion at
crates/storage/tests/iceberg_roundtrip.rs:130callsappend_if_headwithNoneand a one-row batch such asbatch(&[1], &["a"]). A call with no rows and no properties is refused atcrates/storage/src/lib.rs:193-197before the table is loaded, so with an empty batchis_err()would hold without the missing table ever being looked up. - The doc comment of
append_if_head(crates/storage/src/lib.rs:171-173) describes the method on its own, without the[`Warehouse::append`]link. That link would point at a private or missing item and failRUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps(CONTRIBUTING.md:45). The note atdocs/adr/0006-ordered-job-commits.md:56-58stays as written, since accepted ADRs are not edited (docs/adr/README.md:19-21). - Test: a
compile_faildoctest in the doc comment ofWarehouse(crates/storage/src/lib.rs:82) containinglet _ = infostash_storage::Warehouse::append;shows that no other crate can reach the method. A doctest builds as its own crate and runs undercargo test --workspace(CONTRIBUTING.md:34). The doctest fails when a publicappendis put back locally, asCONTRIBUTING.md:160-161asks of every test.
Points: 1 (C26 1)
Depends on: nothing
Blocks: nothing
Source: design review of 2026-11-01, finding C26.
Activity
codecapo commentedon Nov 9, 2026
PR #46 adds two test files outside
crates/storagethat seed their warehouses throughWarehouse::append:crates/engine/tests/query.rs:33andcrates/cli/tests/query.rs:36(at b3017eb on its branch). Both build as their own crates, so onceappendis crate-private they need the change this issue already prescribes foriceberg_roundtrip.rs:append_if_headwithNoneas the expected head and an empty property map. Worth adding both files to the list under "Done when", and starting this after #46 has merged so the two changes do not cross.