Repository navigation
Description
The CSV connector at a0b5e6c adds two load columns to every Raw row, _source_file and _loaded_at (crates/connectors/src/csv.rs:219-229), and neither can order rows. _loaded_at is one wall-clock reading per read batch (:207, :228), files are read in lexical path order (:94), and a loaded path is never read again (:8), so a corrected row has to arrive in a new file. Raw is append only (crates/model/src/lib.rs:15), so a correction is a second row with the same key, and nothing on either row reliably says which came later. Latest-wins under @unique_key (docs/rfcs/0001-project-files.md:118) therefore has no deterministic rule, open question 4 (:175-176) proposes "the last one read", and a rerun cannot promise the same split between target and quarantine that ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72).
This ticket adds _source_row, the 1-based position of a row within its file, taken from the row count the connector already keeps to resume a file (crates/connectors/src/csv.rs:37-39, :158). Each load is one commit carrying the job id (docs/adr/0006-ordered-job-commits.md:39-40), so the Raw snapshot sequence number, then _source_file, then _source_row orders every Raw row, and a quarantined row can point back to its row in the source file. The load order itself is not stored on rows: it comes from the snapshot that committed each data file, and a stored column for it waits for compaction (ROADMAP.md:28), the first step that merges files from several loads.
The column has to be in the Raw schema before 0.1.0. The positional name check at crates/storage/src/lib.rs:373 fails every later load into a Raw table created without it, and rows already loaded can never get one. It waits for #73, which writes the column and the ordering rule it serves into RFC 0001. A Raw table created before this lands has to be loaded again from scratch, which is acceptable before 0.1.0. Deduplicating by this order on @unique_key belongs to #35 and carrying the column into quarantine rows belongs to #12; models read it like any other column through the table registration in #10.
Crate: crates/connectors (src/csv.rs, src/lib.rs, tests/csv_ingest.rs)
Done when
-
with_load_columns(crates/connectors/src/csv.rs:204-205) appends a non-nullInt64column_source_rowafter_loaded_at, holding each row's 1-based position among the data rows of its file (the header is not counted), and usesInt64because the pinned iceberg-rust refusesUInt64whencreate_tableconverts the schema (crates/storage/src/lib.rs:129). - The numbering continues across batches and across runs, because within a run the count advances in
Open::rows(crates/connectors/src/csv.rs:158) and a resume restores it from the position'sreadingentry (:37-39) intoOpen::rows(:120), skipping the rows already read withwith_bounds(:109-113), so each row of a file gets one number whichever batch or run reads it. - A public
SOURCE_ROWconstant is defined besideSOURCE_FILEandLOADED_AT(crates/connectors/src/csv.rs:27-30) and re-exported with them (crates/connectors/src/lib.rs:17), and the name joins the reserved load-metadata names (crates/connectors/src/csv.rs:189-191), so a file whose header has a_source_rowcolumn fails withError::Parse, tested by a new case besidereserved.csvinbad_headers_are_refused(:280). - The module doc, which says two columns are added to every row (
crates/connectors/src/csv.rs:3-5), names all three, says what_source_rowholds and points to the RFC 0001 ordering rule from docs: revise RFC 0001 with the incremental rules (decision D2) #73 for how Raw rows are ordered; the doc comment ofwith_load_columns(:204) names the new column too. - Test: in
reads_resume_in_the_middle_of_a_file(crates/connectors/src/csv.rs:257-273), the first batch carries_source_row1 and 2 and the batch a fresh connector reads from the saved position carries 3. - Test:
first_run_loads_every_file_with_load_metadata(crates/connectors/tests/csv_ingest.rs:88-120), whose runs read in batches of two (:54), checks that_source_rowis a non-nullInt64and that the sorted (_source_file,_source_row) pairs area.csv1, 2 and 3 andb.csv1. - Test: after the crash in
a_crash_before_the_commit_loads_each_row_once_on_rerun(crates/connectors/tests/csv_ingest.rs:146-165), the rerun numbersb.csvfrom the stored position, not from what the dropped attempt read, so theb.csvrows it commits carry_source_row1, 2 and 3. - Test: a new case in
crates/connectors/tests/csv_ingest.rsloads a file holding the sameidin two rows; both rows land in Raw, and the later row carries the larger_source_row.
Points: 2 (M2 Raw _source_row 2)
Depends on: #73 (the ordering rule)
Blocks: nothing
Source: design review of 2026-11-01, findings C15 and C17 (points from section 8.5, M2).
Activity
docs: revise RFC 0001 with the incremental rules (#101)