Skip to content

Tracking: Phase 0 (0.1.0) #19

Open
10 of 42 issues completed
Open

Description

@codecapo

Phase 0 from the roadmap: one binary that turns CSV files into Raw, Clean and Ready Iceberg tables on a laptop. Ships as 0.1.0.

Done when a new user can run the quick start in 5 minutes: infoctl init, infoctl run twice (the second run processes only new rows), see a bad row land in quarantine, and infoctl query the result.

Work, in dependency order

# Item Crate Depends on Complexity
#5 RFC 0001: project file format docs 1
#6 Persist the Iceberg catalog (done in #37) storage 3
#7 Compile a project into a dependency graph compiler #5 5
#8 infoctl init cli #5 2
#9 CSV connector, exactly-once connectors #6 5
#10 Run a model with DataFusion (append) engine #6, #7 5
#35 Merge rows on @unique_key engine, storage #10 8
#36 Partition tables with @partition_by engine, storage #10 5
#38 Parallel Iceberg table provider engine #10 5
#39 Compress the Parquet files we write storage, engine 2
#11 Incremental scan between snapshots storage #6 3
#12 Row checks and quarantine compiler, engine #5, #10 8
#13 In-process runner, one commit per job jobs, dist #10, #11 8
#40 Remove orphan data files storage #13 5
#14 infoctl run cli #8, #9, #12, #13 3
#15 infoctl query cli #6 3
#16 Quick start, tested in CI docs #14, #15 3
#17 Docs site docs #16 3
#18 Release workflow ci 8

Complexity is in Fibonacci story points (1, 2, 3, 5, 8, 13). Anything above 13 gets split before work starts: #10 was about 21, so merging (#35) and partitioning (#36) were split out.

Points were re-scored after R&D spikes (2026-10-30) that ran each risky item against the pinned versions; the findings are in comments on each issue. In short:

The critical path is #6, #10, #13, #14, #16. #35 and #36 can run alongside it; the quick start only needs appends.

Start anywhere with no open dependencies: #9, #11, #15, #18 and #39 today. #5 is in its comment period until 2026-11-05. Comment on an item before starting it so work isn't duplicated.

Design background: architecture and the decision records in docs/adr/.

New work from the design review (2026-11-02). A design review of a0b5e6c against RFC 0001 and the ADRs found the append-only slice sound, but Phase 0 still needs a replace commit that the pinned iceberg-rust does not expose, ready/daily_revenue must be a full refresh, and incremental eligibility needs an allow-list over the logical plan. Five decisions come first (D1 to D5), and the build tickets that wait on one carry the blocked label until it is made. Each new issue states its points and the finding it comes from; the rows below use the columns of the table above and add 87 points.

# Item Crate Depends on Complexity
#72 Spike the replace commit route (decision D1) storage, docs 5
#73 Revise RFC 0001 with the incremental rules (decision D2) docs #5, #72 3
#74 RFC for the Job, JobOutcome and runner contracts (decision D3) docs, model, jobs, dist, storage 3
#75 ADR amending 0006 on position, reruns and expiry (decision D4) docs, storage 2
#76 Measure ingest memory and metadata growth at scale (decision D5) connectors, storage 5
#77 Cap each job at a budget and load a larger backlog in chunks connectors #76 5
#78 Glob once per job, skip dotfiles, symlinks, empty and fresh files connectors #73 5
#79 Check the header before reading rows and handle header-only files connectors, storage 5
#80 Split the snapshot summary from the position storage, connectors #75 3
#81 Skip tables that already carry the job id and use distinct test ids connectors, storage #75 3
#82 Stage under a job prefix and run checks before the writer storage, connectors #74 3
#83 Make append crate-private storage 1
#84 Incremental eligibility classifier with a function allow-list compiler #73, #7 5
#85 StreamingTable delta provider over scan_range engine, storage #10 8
#86 Add _source_row to Raw tables connectors #73 2
#87 Model version hash, refuse a run on mismatch compiler, jobs #73, #7, #13 3
#88 The replace_if_head commit storage #72 8
#89 Fall back to a full refresh at plan time and fix files_added_by jobs, engine, storage, model #72, #84, #88, #13 5
#90 Implement the @full_refresh directive compiler, engine, jobs #72, #73, #84, #88, #7, #13 2
#91 Snapshot expiry floor at create_table storage #75 2
#92 Map HeadGuard conflicts and fix its doc storage 2
#93 Deny-list tests and a catalog.db existence check for infoctl query engine, cli #15 1
#94 Fix README, architecture and ROADMAP statements the code does not back docs, storage, dist 1
#95 Give infoctl query a QueryContext, a single-pass load and a memory limit engine, cli, docs #15 5
#96 Let infoctl query take SQL that starts with a comment cli #15 1
#97 Let infoctl query list tables with show tables engine, cli #15 1

Re-scores of existing rows are suggested in comments and left to this issue: #7 to 13, #13 to 13, #35 down to 5 once #88 carries the compare-and-swap commit, and #36 may leave 0.1.0. With no open dependency, #72, #74, #75, #76, #79, #83, #92 and #94 can start today; #78, #81 and #82 can start apart from the one item each waits on.

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. added
    trackingTracks a release or epic through sub-issues
    on Oct 29, 2026
  3. 17 remaining items

  4. codecapo commented on Nov 11, 2026

    @codecapo
    OwnerAuthor

    #76 measured ingest memory and metadata growth at the Phase 0 target of a 10 GB CSV export on a 16 GB laptop. Full numbers, versions and the harness are in #76 (comment).

    • C06 confirmed. Narrow rows hold about 24 bytes of memory per input byte in prepare. The largest narrow input that commits under 16 GiB is under 1 GiB. Wide rows commit the full 10 GiB at 14.4 GiB peak, most of it in the Parquet writer.
    • C10 confirmed. After 365 one-file commits, metadata.json is 5.3 MB and the metadata directory 684 MB, within 12% of the quadratic and cubic derivations.
    • Delta reads stream. 10 GiB of appended input reads in 9 to 40 seconds with a peak of 81 MB to 3.5 GB.

    For #77: cut chunks on held Arrow memory at 4 GiB, not on input bytes or rows. A 10 GB backlog then takes about 55 commits for narrow rows and 3 for wide rows. #80 has to land before chunked commits, or metadata growth reaches a year's worth within a week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    trackingTracks a release or epic through sub-issues

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions