Skip to content

Fixture: Add reference table for V3 row lineage - #26

Open
slachiewicz wants to merge 1 commit into
apache:mainfrom
slachiewicz:v3-row-lineage-table
Open

slachiewicz wants to merge 1 commit into
apache:mainfrom
slachiewicz:v3-row-lineage-table

Conversation

@slachiewicz

Copy link
Copy Markdown
Member

Closes #13. Builds on #24, whose commit this branch carries until it merges; only the second commit is this change.

v3/row-lineage-parquet-data has three snapshots. The first appends a (row IDs 0 to 2). The second appends b and c in one manifest (3 to 4, then 5), so a reader has to add the preceding file's record_count. The third is an overwrite that replaces a with a2, which stores row IDs 0 to 2 and sequence number 1 for the copied rows, a null sequence number for the updated row, and nulls for an added row that inherits _row_id 9 (a2's first_row_id 6 plus position 3). A reader that ignores stored values returns 6 to 8 for the copied rows.

expected/scan.json adds _row_id and _last_updated_sequence_number keyed by their reserved field IDs, and the README asks consumers to project them where rows carries those keys. The generator gains an overwrite commit whose rows may carry the two columns; the existing tables generate unchanged.

The table is 52 KB over three commits; the overwrite writes a second manifest for the deleted entry.

Checked at every snapshot: an Iceberg Java 1.11.0 generic scan projecting both columns returns expected/scan.json, and so does a read with no Iceberg library (plain Avro for manifest lists and manifests, DuckDB for Parquet). iceberg-go (1827440) resolves the same per-file first_row_id and data sequence numbers but cannot project _row_id yet. PyIceberg was not run.

@slachiewicz slachiewicz changed the title Add reference table for V3 row lineage Fixture: Add reference table for V3 row lineage Nov 19, 2026
@github-actions github-actions Bot added the fixtures Surfaces, cases, reference tables, and fixture corrections label Nov 19, 2026
@slachiewicz
slachiewicz marked this pull request as ready for review November 19, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fixtures Surfaces, cases, reference tables, and fixture corrections

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add reference tables for V3 row-lineage enabled table

1 participant