Skip to content

connectors: add _source_row to Raw tables #86

Open

Description

@codecapo

The CSV connector at a0b5e6c adds two load columns to every Raw row, _source_file and _loaded_at (crates/connectors/src/csv.rs:219-229), and neither can order rows. _loaded_at is one wall-clock reading per read batch (:207, :228), files are read in lexical path order (:94), and a loaded path is never read again (:8), so a corrected row has to arrive in a new file. Raw is append only (crates/model/src/lib.rs:15), so a correction is a second row with the same key, and nothing on either row reliably says which came later. Latest-wins under @unique_key (docs/rfcs/0001-project-files.md:118) therefore has no deterministic rule, open question 4 (:175-176) proposes "the last one read", and a rerun cannot promise the same split between target and quarantine that ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72).

This ticket adds _source_row, the 1-based position of a row within its file, taken from the row count the connector already keeps to resume a file (crates/connectors/src/csv.rs:37-39, :158). Each load is one commit carrying the job id (docs/adr/0006-ordered-job-commits.md:39-40), so the Raw snapshot sequence number, then _source_file, then _source_row orders every Raw row, and a quarantined row can point back to its row in the source file. The load order itself is not stored on rows: it comes from the snapshot that committed each data file, and a stored column for it waits for compaction (ROADMAP.md:28), the first step that merges files from several loads.

The column has to be in the Raw schema before 0.1.0. The positional name check at crates/storage/src/lib.rs:373 fails every later load into a Raw table created without it, and rows already loaded can never get one. It waits for #73, which writes the column and the ordering rule it serves into RFC 0001. A Raw table created before this lands has to be loaded again from scratch, which is acceptable before 0.1.0. Deduplicating by this order on @unique_key belongs to #35 and carrying the column into quarantine rows belongs to #12; models read it like any other column through the table registration in #10.

Crate: crates/connectors (src/csv.rs, src/lib.rs, tests/csv_ingest.rs)

Done when

  • with_load_columns (crates/connectors/src/csv.rs:204-205) appends a non-null Int64 column _source_row after _loaded_at, holding each row's 1-based position among the data rows of its file (the header is not counted), and uses Int64 because the pinned iceberg-rust refuses UInt64 when create_table converts the schema (crates/storage/src/lib.rs:129).
  • The numbering continues across batches and across runs, because within a run the count advances in Open::rows (crates/connectors/src/csv.rs:158) and a resume restores it from the position's reading entry (:37-39) into Open::rows (:120), skipping the rows already read with with_bounds (:109-113), so each row of a file gets one number whichever batch or run reads it.
  • A public SOURCE_ROW constant is defined beside SOURCE_FILE and LOADED_AT (crates/connectors/src/csv.rs:27-30) and re-exported with them (crates/connectors/src/lib.rs:17), and the name joins the reserved load-metadata names (crates/connectors/src/csv.rs:189-191), so a file whose header has a _source_row column fails with Error::Parse, tested by a new case beside reserved.csv in bad_headers_are_refused (:280).
  • The module doc, which says two columns are added to every row (crates/connectors/src/csv.rs:3-5), names all three, says what _source_row holds and points to the RFC 0001 ordering rule from docs: revise RFC 0001 with the incremental rules (decision D2) #73 for how Raw rows are ordered; the doc comment of with_load_columns (:204) names the new column too.
  • Test: in reads_resume_in_the_middle_of_a_file (crates/connectors/src/csv.rs:257-273), the first batch carries _source_row 1 and 2 and the batch a fresh connector reads from the saved position carries 3.
  • Test: first_run_loads_every_file_with_load_metadata (crates/connectors/tests/csv_ingest.rs:88-120), whose runs read in batches of two (:54), checks that _source_row is a non-null Int64 and that the sorted (_source_file, _source_row) pairs are a.csv 1, 2 and 3 and b.csv 1.
  • Test: after the crash in a_crash_before_the_commit_loads_each_row_once_on_rerun (crates/connectors/tests/csv_ingest.rs:146-165), the rerun numbers b.csv from the stored position, not from what the dropped attempt read, so the b.csv rows it commits carry _source_row 1, 2 and 3.
  • Test: a new case in crates/connectors/tests/csv_ingest.rs loads a file holding the same id in two rows; both rows land in Raw, and the later row carries the larger _source_row.

Points: 2 (M2 Raw _source_row 2)

Depends on: #73 (the ordering rule)
Blocks: nothing

Source: design review of 2026-11-01, findings C15 and C17 (points from section 8.5, M2).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/connectorscrates/connectorsblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions