Repository navigation
Description
Phase 0 from the roadmap: one binary that turns CSV files into Raw, Clean and Ready Iceberg tables on a laptop. Ships as 0.1.0.
Done when a new user can run the quick start in 5 minutes: infoctl init, infoctl run twice (the second run processes only new rows), see a bad row land in quarantine, and infoctl query the result.
Work, in dependency order
| # | Item | Crate | Depends on | Complexity |
|---|---|---|---|---|
| #5 | RFC 0001: project file format | docs | 1 | |
| storage | 3 | |||
| #7 | Compile a project into a dependency graph | compiler | #5 | 5 |
| #8 | infoctl init |
cli | #5 | 2 |
| #9 | CSV connector, exactly-once | connectors | #6 | 5 |
| #10 | Run a model with DataFusion (append) | engine | #6, #7 | 5 |
| #35 | Merge rows on @unique_key |
engine, storage | #10 | 8 |
| #36 | Partition tables with @partition_by |
engine, storage | #10 | 5 |
| #38 | Parallel Iceberg table provider | engine | #10 | 5 |
| #39 | Compress the Parquet files we write | storage, engine | 2 | |
| #11 | Incremental scan between snapshots | storage | #6 | 3 |
| #12 | Row checks and quarantine | compiler, engine | #5, #10 | 8 |
| #13 | In-process runner, one commit per job | jobs, dist | #10, #11 | 8 |
| #40 | Remove orphan data files | storage | #13 | 5 |
| #14 | infoctl run |
cli | #8, #9, #12, #13 | 3 |
| #15 | infoctl query |
cli | #6 | 3 |
| #16 | Quick start, tested in CI | docs | #14, #15 | 3 |
| #17 | Docs site | docs | #16 | 3 |
| #18 | Release workflow | ci | 8 |
Complexity is in Fibonacci story points (1, 2, 3, 5, 8, 13). Anything above 13 gets split before work starts: #10 was about 21, so merging (#35) and partitioning (#36) were split out.
Points were re-scored after R&D spikes (2026-10-30) that ran each risky item against the pinned versions; the findings are in comments on each issue. In short:
- Incremental scan: read rows added between two snapshots #11 drops to 3 and Run a model with DataFusion and append to its table #10 to 5: the incremental scan and the DataFusion write path both work, and a 1B-row raw to clean to ready run finished in 17.3 min with 5.7 GB peak memory and exact results.
- Merge rows on @unique_key #35 drops to 8: copy-on-write merges work, but the commit needs our own compare-and-swap on the catalog pointer.
- In-process runner: stage, run tasks, commit once #13 drops to 8: staged files with one commit per job work.
- Row checks and quarantine tables #12 rises to 8: iceberg-rust has no multi-table commit, so target and quarantine commit in order under one job id.
- New: engine: parallel Iceberg table provider with pushdown #38 fixes the single-core Iceberg scan (the main speed gap against DataFusion and DuckDB), storage: compress the Parquet files we write #39 turns on compression, storage: remove orphan data files #40 cleans up orphan files.
The critical path is #6, #10, #13, #14, #16. #35 and #36 can run alongside it; the quick start only needs appends.
Start anywhere with no open dependencies: #9, #11, #15, #18 and #39 today. #5 is in its comment period until 2026-11-05. Comment on an item before starting it so work isn't duplicated.
Design background: architecture and the decision records in docs/adr/.
New work from the design review (2026-11-02). A design review of a0b5e6c against RFC 0001 and the ADRs found the append-only slice sound, but Phase 0 still needs a replace commit that the pinned iceberg-rust does not expose, ready/daily_revenue must be a full refresh, and incremental eligibility needs an allow-list over the logical plan. Five decisions come first (D1 to D5), and the build tickets that wait on one carry the blocked label until it is made. Each new issue states its points and the finding it comes from; the rows below use the columns of the table above and add 87 points.
| # | Item | Crate | Depends on | Complexity |
|---|---|---|---|---|
| #72 | Spike the replace commit route (decision D1) | storage, docs | 5 | |
| #73 | Revise RFC 0001 with the incremental rules (decision D2) | docs | #5, #72 | 3 |
| #74 | RFC for the Job, JobOutcome and runner contracts (decision D3) | docs, model, jobs, dist, storage | 3 | |
| #75 | ADR amending 0006 on position, reruns and expiry (decision D4) | docs, storage | 2 | |
| #76 | Measure ingest memory and metadata growth at scale (decision D5) | connectors, storage | 5 | |
| #77 | Cap each job at a budget and load a larger backlog in chunks | connectors | #76 | 5 |
| #78 | Glob once per job, skip dotfiles, symlinks, empty and fresh files | connectors | #73 | 5 |
| #79 | Check the header before reading rows and handle header-only files | connectors, storage | 5 | |
| #80 | Split the snapshot summary from the position | storage, connectors | #75 | 3 |
| #81 | Skip tables that already carry the job id and use distinct test ids | connectors, storage | #75 | 3 |
| #82 | Stage under a job prefix and run checks before the writer | storage, connectors | #74 | 3 |
| #83 | Make append crate-private |
storage | 1 | |
| #84 | Incremental eligibility classifier with a function allow-list | compiler | #73, #7 | 5 |
| #85 | StreamingTable delta provider over scan_range |
engine, storage | #10 | 8 |
| #86 | Add _source_row to Raw tables |
connectors | #73 | 2 |
| #87 | Model version hash, refuse a run on mismatch | compiler, jobs | #73, #7, #13 | 3 |
| #88 | The replace_if_head commit |
storage | #72 | 8 |
| #89 | Fall back to a full refresh at plan time and fix files_added_by |
jobs, engine, storage, model | #72, #84, #88, #13 | 5 |
| #90 | Implement the @full_refresh directive |
compiler, engine, jobs | #72, #73, #84, #88, #7, #13 | 2 |
| #91 | Snapshot expiry floor at create_table |
storage | #75 | 2 |
| #92 | Map HeadGuard conflicts and fix its doc |
storage | 2 | |
| #93 | Deny-list tests and a catalog.db existence check for infoctl query |
engine, cli | #15 | 1 |
| #94 | Fix README, architecture and ROADMAP statements the code does not back | docs, storage, dist | 1 | |
| #95 | Give infoctl query a QueryContext, a single-pass load and a memory limit |
engine, cli, docs | #15 | 5 |
| #96 | Let infoctl query take SQL that starts with a comment |
cli | #15 | 1 |
| #97 | Let infoctl query list tables with show tables |
engine, cli | #15 | 1 |
Re-scores of existing rows are suggested in comments and left to this issue: #7 to 13, #13 to 13, #35 down to 5 once #88 carries the compare-and-swap commit, and #36 may leave 0.1.0. With no open dependency, #72, #74, #75, #76, #79, #83, #92 and #94 can start today; #78, #81 and #82 can start apart from the one item each waits on.
Activity
17 remaining items
codecapo commentedon Nov 11, 2026
#76 measured ingest memory and metadata growth at the Phase 0 target of a 10 GB CSV export on a 16 GB laptop. Full numbers, versions and the harness are in #76 (comment).
prepare. The largest narrow input that commits under 16 GiB is under 1 GiB. Wide rows commit the full 10 GiB at 14.4 GiB peak, most of it in the Parquet writer.metadata.jsonis 5.3 MB and the metadata directory 684 MB, within 12% of the quadratic and cubic derivations.For #77: cut chunks on held Arrow memory at 4 GiB, not on input bytes or rows. A 10 GB backlog then takes about 55 commits for narrow rows and 3 for wide rows. #80 has to land before chunked commits, or metadata growth reaches a year's worth within a week.