Skip to content

CSV connector with exactly-once loading into Raw #9

Closed

Description

@codecapo

A CSV connector that loads files matching a glob into a Raw table, reading only files it has not loaded before.

Crate: crates/connectors

Done when

  • Implements the Connector trait; the Position records which files were loaded
  • Adds load metadata columns (source file, load time)
  • The position is saved in the same Iceberg commit as the rows (ADR 0004)
  • Second run with no new files commits nothing; with one new file, loads only that file
  • Test for a crash between read and commit: rerunning loads each row exactly once
  • The position is a table property, mirrored in the snapshot summary
  • Two runs from the same position don't duplicate rows: the second fails because the table head moved (iceberg-rust rebases stale commits silently)

Depends on: #6
Blocks: #14

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 30, 2026

    @codecapo
    OwnerAuthor

    Spike findings (commit semantics)

    A spike against the pinned iceberg-rust proved the commit side of this issue.

    • Position: one Transaction holds the fast_append and an update_table_properties. The position is stored as the table property infostash.position and mirrored in the snapshot summary. Both land in one metadata file. Read it back from the table property.
    • Crash between read and commit: a rerun loads each row exactly once. The spike's crash test can be reused as is.
    • Watch out: Transaction::do_commit reloads the table and silently rebases a stale transaction, so two runs from the same position duplicate rows instead of conflicting. We need a guard that fails the commit if the table head moved after the position was read (about 40 lines, wrapping the catalog's compare-and-swap).
    • An empty fast_append is an error, so a run with nothing new must skip the commit.
    • Don't use summary keys that start with added- or total-; iceberg-rust overwrites them.

    Points: 5 (unchanged).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/connectorscrates/connectorsconnectorA source or destination connectorenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions