Skip to content

docs: fix README, architecture and ROADMAP statements the code does not back #94

Open

Description

@codecapo

Several statements in README.md, ROADMAP.md, docs/architecture.md, docs/architecture.svg and two crate doc comments describe behaviour the code at a0b5e6c does not have, and one contradicts an accepted decision. docs/architecture.md:52-53 still promises one atomic commit per job, while ADR 0006 settles on one commit per output table carrying the job id, quarantine first and the target last (docs/adr/0006-ordered-job-commits.md:39-46), and docs/architecture.md:91-92 and :96 say the framework handles schema changes, while the CSV connector fails the run on any file with rows whose columns differ from the table's (crates/connectors/src/csv.rs:12-13). This issue rewrites each statement to say what is true today or to mark it as planned, and changes no behaviour: the ADR 0006 rules (one commit per table carrying the job id; a connector's position commits with its data, docs/adr/0006-ordered-job-commits.md:49-51) stay as they are. Two statements are out of scope: the HeadGuard doc comment (crates/storage/src/lib.rs:404-410), which #92 rewrites together with the code it describes, and the "deduplicated" claim at README.md:21, which waits on the decision in #72.

Where: README.md, ROADMAP.md, docs/architecture.md, docs/architecture.svg, and the crate doc comments at the top of crates/storage/src/lib.rs and crates/dist/src/lib.rs

Done when

  • README.md:11 and README.md:25-26 stop saying that Infostash runs on AWS, GCP or Azure and keeps its tables on object storage today. They say that 0.1.0 runs on one machine with its Iceberg tables on local disk, and that cloud accounts and object storage on S3, GCS, Azure Blob and MinIO come in Phase 1 (ROADMAP.md:18-22). For example, :11 can read It runs on one machine today and on AWS, GCP, Azure or your own servers from Phase 1, and :25-26 can read everything is stored as Apache Iceberg tables, on local disk in 0.1.0 and on object storage from Phase 1, so Spark, Trino, Snowflake and Databricks can read the same tables.
  • The crate doc at crates/storage/src/lib.rs:1-2 stops listing S3, GCS, ADLS and MinIO and stops placing the data in the customer's bucket, since only LocalFsStorageFactory is wired (:20, :97). It says instead that the crate reads and writes Iceberg tables on local disk, in open formats, and that object storage comes in Phase 1. :4-7 already describe the local directory and stay as they are.
  • docs/architecture.md:52-53 says that a job commits each output table once, atomically and with its job id, side tables such as quarantine before the target, so readers of a table see its old or its new snapshot. It adds that between those commits a reader can see quarantine rows whose target rows have not landed, so a reader that needs a consistent view keeps only quarantine rows whose job id is in the target's history (docs/adr/0006-ordered-job-commits.md:39-46, :67-70).
  • The same claim changes where it repeats: docs/architecture.md:11 ends with makes one atomic Iceberg commit per output table., the arrow label at docs/architecture.svg:141 reads one commit per table instead of one atomic commit, and the description at docs/architecture.svg:3 ends its second sentence with makes one atomic Iceberg commit per output table, quarantine before the target.
  • docs/architecture.md:55-56 and the crate doc at crates/dist/src/lib.rs:4-5 keep saying that the same plan runs in one process and across many machines, but replace "Only the Runner changes" and "scaling out changes the runner, not the plan" with the Runner is the only component that chooses where tasks run; the contracts between jobs, tasks and commits are fixed before that choice. Those contracts are what dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 settles. The note at docs/architecture.svg:124 replaces Only the Runner changes. with a line of about the same length, such as The Runner picks the machines.
  • docs/architecture.md:68-69 drops deletion vectors for updates from the list and says that updates are copy-on-write first and deletion vectors come later, for example Updates rewrite the files they touch (copy-on-write) at first; deletion vectors come later. Deletion vectors need Iceberg format version 3, but tables are created as version 2 (create_table sets no format version, crates/storage/src/lib.rs:127-131), and the pinned iceberg-rust has no public commit that adds delete files.
  • docs/architecture.md:91-92 says The framework: instead of The framework does the rest:, and the position bullet at :94-95 stays, since it is true. :96 becomes a bullet that marks schema changes, retries, secrets and sandboxing as planned and says what happens today, for example a bullet reading Will handle schema changes, retries, secrets and sandboxing. Today a CSV file with rows whose header adds, drops or reorders a column fails the run and commits nothing; the position stays, so every later run fails until the file is fixed or removed. That is what the code does for a file with rows (crates/connectors/src/csv.rs:12-13, crates/storage/src/lib.rs:373-378). A header-only file never reaches that check and is recorded as loaded whatever its columns (crates/connectors/src/lib.rs:80-83), which connectors: check the header before reading rows and handle header-only files #79 changes.
  • ROADMAP.md:13 and README.md:23 stop promising that every run processes only new or changed rows. They say that sources and row-wise models (no joins or aggregates) over append-only inputs process only new rows, and that joins and aggregates are not incremental in 0.1.0. For example, ROADMAP.md:13 can read Incremental runs: the second run processes only new rows for sources and row-wise models over append-only inputs; joins and aggregates are not incremental yet.
  • typos and RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps still succeed (CONTRIBUTING.md:43, :45), since two of the edits are crate doc comments, and docs/architecture.svg, opened in a browser, shows each changed line inside its box or beside its arrow without overlapping other text.

Points: 1 (M5)

Depends on: nothing
Blocks: nothing

Source: design review of 2026-11-01, section 8.5, M5 (findings C16, C18, C27, C28 and C30, and sections 1.4, 2.4 and 5.4).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    One more for this list, from RFC 0002 (#102): the doc comment on RetryPolicy in crates/dist/src/lib.rs ("a lost shuffle output means the producing stage reruns") and step 4 of the overview in docs/architecture.md describe the multi-stage runner of a later phase. In Phase 0 a lost exchange input (Error::InputLost, reserved) is Retry::Job, because a single-stage job has no narrower unit to rerun. Both places should say they describe the target, not Phase 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/docsDocs, RFCs and ADRsdocumentationImprovements or additions to documentationgood first issueGood for newcomers

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions