Repository navigation
Description
Several statements in README.md, ROADMAP.md, docs/architecture.md, docs/architecture.svg and two crate doc comments describe behaviour the code at a0b5e6c does not have, and one contradicts an accepted decision. docs/architecture.md:52-53 still promises one atomic commit per job, while ADR 0006 settles on one commit per output table carrying the job id, quarantine first and the target last (docs/adr/0006-ordered-job-commits.md:39-46), and docs/architecture.md:91-92 and :96 say the framework handles schema changes, while the CSV connector fails the run on any file with rows whose columns differ from the table's (crates/connectors/src/csv.rs:12-13). This issue rewrites each statement to say what is true today or to mark it as planned, and changes no behaviour: the ADR 0006 rules (one commit per table carrying the job id; a connector's position commits with its data, docs/adr/0006-ordered-job-commits.md:49-51) stay as they are. Two statements are out of scope: the HeadGuard doc comment (crates/storage/src/lib.rs:404-410), which #92 rewrites together with the code it describes, and the "deduplicated" claim at README.md:21, which waits on the decision in #72.
Where: README.md, ROADMAP.md, docs/architecture.md, docs/architecture.svg, and the crate doc comments at the top of crates/storage/src/lib.rs and crates/dist/src/lib.rs
Done when
-
README.md:11andREADME.md:25-26stop saying that Infostash runs on AWS, GCP or Azure and keeps its tables on object storage today. They say that 0.1.0 runs on one machine with its Iceberg tables on local disk, and that cloud accounts and object storage on S3, GCS, Azure Blob and MinIO come in Phase 1 (ROADMAP.md:18-22). For example,:11can readIt runs on one machine today and on AWS, GCP, Azure or your own servers from Phase 1, and:25-26can readeverything is stored as Apache Iceberg tables, on local disk in 0.1.0 and on object storage from Phase 1, so Spark, Trino, Snowflake and Databricks can read the same tables. - The crate doc at
crates/storage/src/lib.rs:1-2stops listing S3, GCS, ADLS and MinIO and stops placing the data in the customer's bucket, since onlyLocalFsStorageFactoryis wired (:20,:97). It says instead that the crate reads and writes Iceberg tables on local disk, in open formats, and that object storage comes in Phase 1.:4-7already describe the local directory and stay as they are. -
docs/architecture.md:52-53says that a job commits each output table once, atomically and with its job id, side tables such as quarantine before the target, so readers of a table see its old or its new snapshot. It adds that between those commits a reader can see quarantine rows whose target rows have not landed, so a reader that needs a consistent view keeps only quarantine rows whose job id is in the target's history (docs/adr/0006-ordered-job-commits.md:39-46,:67-70). - The same claim changes where it repeats:
docs/architecture.md:11ends withmakes one atomic Iceberg commit per output table., the arrow label atdocs/architecture.svg:141readsone commit per tableinstead ofone atomic commit, and the description atdocs/architecture.svg:3ends its second sentence withmakes one atomic Iceberg commit per output table, quarantine before the target. -
docs/architecture.md:55-56and the crate doc atcrates/dist/src/lib.rs:4-5keep saying that the same plan runs in one process and across many machines, but replace "Only theRunnerchanges" and "scaling out changes the runner, not the plan" withthe Runner is the only component that chooses where tasks run; the contracts between jobs, tasks and commits are fixed before that choice. Those contracts are what dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 settles. The note atdocs/architecture.svg:124replacesOnly the Runner changes.with a line of about the same length, such asThe Runner picks the machines. -
docs/architecture.md:68-69dropsdeletion vectors for updatesfrom the list and says that updates are copy-on-write first and deletion vectors come later, for exampleUpdates rewrite the files they touch (copy-on-write) at first; deletion vectors come later.Deletion vectors need Iceberg format version 3, but tables are created as version 2 (create_tablesets no format version,crates/storage/src/lib.rs:127-131), and the pinned iceberg-rust has no public commit that adds delete files. -
docs/architecture.md:91-92saysThe framework:instead ofThe framework does the rest:, and the position bullet at:94-95stays, since it is true.:96becomes a bullet that marks schema changes, retries, secrets and sandboxing as planned and says what happens today, for example a bullet readingWill handle schema changes, retries, secrets and sandboxing. Today a CSV file with rows whose header adds, drops or reorders a column fails the run and commits nothing; the position stays, so every later run fails until the file is fixed or removed.That is what the code does for a file with rows (crates/connectors/src/csv.rs:12-13,crates/storage/src/lib.rs:373-378). A header-only file never reaches that check and is recorded as loaded whatever its columns (crates/connectors/src/lib.rs:80-83), which connectors: check the header before reading rows and handle header-only files #79 changes. -
ROADMAP.md:13andREADME.md:23stop promising that every run processes only new or changed rows. They say that sources and row-wise models (no joins or aggregates) over append-only inputs process only new rows, and that joins and aggregates are not incremental in 0.1.0. For example,ROADMAP.md:13can readIncremental runs: the second run processes only new rows for sources and row-wise models over append-only inputs; joins and aggregates are not incremental yet. -
typosandRUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-depsstill succeed (CONTRIBUTING.md:43,:45), since two of the edits are crate doc comments, anddocs/architecture.svg, opened in a browser, shows each changed line inside its box or beside its arrow without overlapping other text.
Points: 1 (M5)
Depends on: nothing
Blocks: nothing
Source: design review of 2026-11-01, section 8.5, M5 (findings C16, C18, C27, C28 and C30, and sections 1.4, 2.4 and 5.4).
Activity
codecapo commentedon Nov 10, 2026
One more for this list, from RFC 0002 (#102): the doc comment on
RetryPolicyincrates/dist/src/lib.rs("a lost shuffle output means the producing stage reruns") and step 4 of the overview indocs/architecture.mddescribe the multi-stage runner of a later phase. In Phase 0 a lost exchange input (Error::InputLost, reserved) isRetry::Job, because a single-stage job has no narrower unit to rerun. Both places should say they describe the target, not Phase 0.docs: add RFC 0002 for the job and runner contracts (#102)