Skip to content

connectors: split the snapshot summary from the position #80

Open

Description

@codecapo

At a0b5e6c, Prepared::commit puts infostash.position and infostash.job-id in one map (crates/connectors/src/lib.rs:125-130), and append_if_head writes that map both into the new snapshot's summary (crates/storage/src/lib.rs:209) and into the table properties (:212-218), as ADR 0006 asks (docs/adr/0006-ordered-job-commits.md:49-51). For CSV the position is the list of every file loaded so far (crates/connectors/src/csv.rs:7-8). Each commit writes a new metadata.json that keeps every earlier snapshot with its summary, so each new file grows with the square of the number of runs. Old metadata files are never deleted, so the metadata directory grows with the cube. With 65-character paths, the estimate after 1000 hourly one-file runs is about 35 MB per file and 12 GB on disk, and every run after the first parses the current file four times.

Production code reads only the table property (crates/connectors/src/lib.rs:72). This ticket keeps the position and the job id as table properties, set in the same commit as the rows, and takes the position out of the snapshot summary, which keeps the job id that ADR 0006:39-40 requires on every commit. If the amending ADR from #75 also asks for a fixed-size digest of the position, the digest goes in the summary beside the job id. Dropping the summary copy changes ADR 0006:49-51, which is why this waits for #75. Compressing metadata.json is no substitute, because the reader still inflates the whole file, and deleting the old metadata files belongs to #40.

Crates: crates/storage (Warehouse::append_if_head and its tests), crates/connectors (Prepared::commit, the POSITION doc, tests/csv_ingest.rs)

Done when

  • append_if_head takes two maps, one for the snapshot summary and one for the table properties. It checks both against PROPERTY_PREFIX as it does now (crates/storage/src/lib.rs:188-192), writes both in the one transaction, and its doc (:171-180) says which map goes where, so the runner in In-process runner: stage, run tasks, commit once #13 can put a model's per-input checkpoints in either or both.
  • Prepared::commit (crates/connectors/src/lib.rs:125-130) sets infostash.position and infostash.job-id as table properties, and puts infostash.job-id but not infostash.position in the snapshot summary, with a digest of the position beside it only if the ADR from docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 asks for one.
  • The marker of ADR 0006:43-46 still works: with no rows, a summary map holding only infostash.job-id makes a snapshot. A call with no rows and an empty summary map is refused before anything is written, as the check at crates/storage/src/lib.rs:193-197 does now, whatever the table-properties map holds, because a fast_append with no data files and no snapshot properties fails in iceberg-rust (ADR 0006:75-76). The test at crates/storage/src/lib.rs:907 moves to the two maps and covers both cases.
  • The test at crates/storage/src/lib.rs:797-824 asserts that one new metadata file holds the rows, both table properties and the job id in the summary, and that the summary has no infostash.position, replacing the assertion at :820.
  • Test: after a CSV run, the current snapshot's summary in the file named by Commit::metadata_location (crates/storage/src/lib.rs:47), parsed with serde_json, holds infostash.job-id and no infostash.position, next to the table-property checks of crates/connectors/tests/csv_ingest.rs:168-185.
  • Test in crates/connectors/tests/csv_ingest.rs: 200 CSV runs into one table, each loading one new file under its own job id rather than the fixed JobId(7) of Setup::run (:51-58), and the size of the metadata.json named by each Commit::metadata_location grows linearly with the number of commits (for example, commits 101 to 200 add at most 1.5 times what commits 1 to 100 added).
  • The POSITION doc at crates/connectors/src/lib.rs:19-20 names the table property as the one authoritative copy of the position and no longer says it is mirrored in the snapshot summary.

Points: 3 (C10 3)

Depends on: #75
Blocks: nothing

Source: design review of 2026-11-01, finding C10.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) decides what this ticket waits on. The snapshot summary keeps only infostash.job-id, with no digest of the position, so the second Done-when item here is a plain removal: the position goes to the table properties and nowhere else. A model's per-input checkpoints stay mirrored in the summary. The ADR also asks for the test that the size of each metadata.json grows linearly over 200 one-file commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/connectorscrates/connectorsarea/storagecrates/storageblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions