Repository navigation
Description
At a0b5e6c, Prepared::commit puts infostash.position and infostash.job-id in one map (crates/connectors/src/lib.rs:125-130), and append_if_head writes that map both into the new snapshot's summary (crates/storage/src/lib.rs:209) and into the table properties (:212-218), as ADR 0006 asks (docs/adr/0006-ordered-job-commits.md:49-51). For CSV the position is the list of every file loaded so far (crates/connectors/src/csv.rs:7-8). Each commit writes a new metadata.json that keeps every earlier snapshot with its summary, so each new file grows with the square of the number of runs. Old metadata files are never deleted, so the metadata directory grows with the cube. With 65-character paths, the estimate after 1000 hourly one-file runs is about 35 MB per file and 12 GB on disk, and every run after the first parses the current file four times.
Production code reads only the table property (crates/connectors/src/lib.rs:72). This ticket keeps the position and the job id as table properties, set in the same commit as the rows, and takes the position out of the snapshot summary, which keeps the job id that ADR 0006:39-40 requires on every commit. If the amending ADR from #75 also asks for a fixed-size digest of the position, the digest goes in the summary beside the job id. Dropping the summary copy changes ADR 0006:49-51, which is why this waits for #75. Compressing metadata.json is no substitute, because the reader still inflates the whole file, and deleting the old metadata files belongs to #40.
Crates: crates/storage (Warehouse::append_if_head and its tests), crates/connectors (Prepared::commit, the POSITION doc, tests/csv_ingest.rs)
Done when
-
append_if_headtakes two maps, one for the snapshot summary and one for the table properties. It checks both againstPROPERTY_PREFIXas it does now (crates/storage/src/lib.rs:188-192), writes both in the one transaction, and its doc (:171-180) says which map goes where, so the runner in In-process runner: stage, run tasks, commit once #13 can put a model's per-input checkpoints in either or both. -
Prepared::commit(crates/connectors/src/lib.rs:125-130) setsinfostash.positionandinfostash.job-idas table properties, and putsinfostash.job-idbut notinfostash.positionin the snapshot summary, with a digest of the position beside it only if the ADR from docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 asks for one. - The marker of ADR 0006:43-46 still works: with no rows, a summary map holding only
infostash.job-idmakes a snapshot. A call with no rows and an empty summary map is refused before anything is written, as the check atcrates/storage/src/lib.rs:193-197does now, whatever the table-properties map holds, because afast_appendwith no data files and no snapshot properties fails in iceberg-rust (ADR 0006:75-76). The test atcrates/storage/src/lib.rs:907moves to the two maps and covers both cases. - The test at
crates/storage/src/lib.rs:797-824asserts that one new metadata file holds the rows, both table properties and the job id in the summary, and that the summary has noinfostash.position, replacing the assertion at:820. - Test: after a CSV run, the current snapshot's summary in the file named by
Commit::metadata_location(crates/storage/src/lib.rs:47), parsed withserde_json, holdsinfostash.job-idand noinfostash.position, next to the table-property checks ofcrates/connectors/tests/csv_ingest.rs:168-185. - Test in
crates/connectors/tests/csv_ingest.rs: 200 CSV runs into one table, each loading one new file under its own job id rather than the fixedJobId(7)ofSetup::run(:51-58), and the size of themetadata.jsonnamed by eachCommit::metadata_locationgrows linearly with the number of commits (for example, commits 101 to 200 add at most 1.5 times what commits 1 to 100 added). - The
POSITIONdoc atcrates/connectors/src/lib.rs:19-20names the table property as the one authoritative copy of the position and no longer says it is mirrored in the snapshot summary.
Points: 3 (C10 3)
Depends on: #75
Blocks: nothing
Source: design review of 2026-11-01, finding C10.
Activity
codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) decides what this ticket waits on. The snapshot summary keeps only
infostash.job-id, with no digest of the position, so the second Done-when item here is a plain removal: the position goes to the table properties and nowhere else. A model's per-input checkpoints stay mirrored in the summary. The ADR also asks for the test that the size of eachmetadata.jsongrows linearly over 200 one-file commits.