Skip to content

storage: make append crate-private #83

Open

Description

@codecapo

At a0b5e6c, Warehouse::append (crates/storage/src/lib.rs:140-151) is public and commits with a plain fast_append. It skips what ADR 0006 asks of every job commit: the job id as infostash.job-id in the snapshot summary (docs/adr/0006-ordered-job-commits.md:39-40), a connector's read position or a model's last processed snapshot written in the same commit as the rows (docs/adr/0006-ordered-job-commits.md:49-51), and a check that the table head has not moved (docs/adr/0006-ordered-job-commits.md:52-54). A caller that commits through it leaves the stored position where it was, so the next run reads the same input and loads those rows a second time. Without the job id, a rerun cannot see that its job already committed, so it commits again (docs/adr/0006-ordered-job-commits.md:47-48). Without the head check, a run that lost a race is rebased onto the new head and duplicates rows instead of failing (docs/adr/0006-ordered-job-commits.md:18-21). Every production write already goes through the guarded append_if_head (crates/storage/src/lib.rs:181-221): Prepared::commit, the commit path every connector shares, calls it with the new position and the job id (crates/connectors/src/lib.rs:125-130), and only tests still call append. The ADR itself notes that append predates the decision (docs/adr/0006-ordered-job-commits.md:56-58). This issue takes append out of the public API before the runner in #13 adds more code that commits. The matching rule for datafusion-iceberg's insert path, that model runs never commit through its IcebergCommitExec, belongs to the execution RFC in #74, not to this issue.

Crate: crates/storage (src/lib.rs and tests/iceberg_roundtrip.rs)

Done when

  • Warehouse::append is no longer reachable outside crates/storage. Either it becomes a private #[cfg(test)] method, which the unit tests in mod tests (crates/storage/src/lib.rs:654-655) can still call because a child module can use its parent's private items, or it is deleted and the unit tests at crates/storage/src/lib.rs:680, :772, :792, :850 and :952 call append_if_head instead. A plain pub(crate) is not enough: only tests would call it, so cargo clippy --workspace --all-targets -- -D warnings (CONTRIBUTING.md:33) fails on it as dead code, and CONTRIBUTING.md:162-164 discourages silencing a check with a new #[allow].
  • The append calls at lines 60, 67, 142, 151, 177, 201, 204, 206, 228 and 229 of crates/storage/tests/iceberg_roundtrip.rs become append_if_head calls, since that file builds as its own crate and can no longer see append. Each call supplies the expected head, which is None on a table with no commits and otherwise the previous commit's snapshot (such as Some(first.snapshot) at line 67), and an empty HashMap of properties, which needs use std::collections::HashMap; beside the imports at lines 8-14.
  • The missing-table assertion at crates/storage/tests/iceberg_roundtrip.rs:130 calls append_if_head with None and a one-row batch such as batch(&[1], &["a"]). A call with no rows and no properties is refused at crates/storage/src/lib.rs:193-197 before the table is loaded, so with an empty batch is_err() would hold without the missing table ever being looked up.
  • The doc comment of append_if_head (crates/storage/src/lib.rs:171-173) describes the method on its own, without the [`Warehouse::append`] link. That link would point at a private or missing item and fail RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps (CONTRIBUTING.md:45). The note at docs/adr/0006-ordered-job-commits.md:56-58 stays as written, since accepted ADRs are not edited (docs/adr/README.md:19-21).
  • Test: a compile_fail doctest in the doc comment of Warehouse (crates/storage/src/lib.rs:82) containing let _ = infostash_storage::Warehouse::append; shows that no other crate can reach the method. A doctest builds as its own crate and runs under cargo test --workspace (CONTRIBUTING.md:34). The doctest fails when a public append is put back locally, as CONTRIBUTING.md:160-161 asks of every test.

Points: 1 (C26 1)

Depends on: nothing
Blocks: nothing

Source: design review of 2026-11-01, finding C26.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 9, 2026

    @codecapo
    OwnerAuthor

    PR #46 adds two test files outside crates/storage that seed their warehouses through Warehouse::append: crates/engine/tests/query.rs:33 and crates/cli/tests/query.rs:36 (at b3017eb on its branch). Both build as their own crates, so once append is crate-private they need the change this issue already prescribes for iceberg_roundtrip.rs: append_if_head with None as the expected head and an empty property map. Worth adding both files to the list under "Done when", and starting this after #46 has merged so the two changes do not cross.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions