Skip to content

RFC 0001: project file format #5

Open

Description

@codecapo

Agree the format of an Infostash project before building the compiler and infoctl init: infostash.toml, sources/*.toml, and SQL models in clean/ and ready/ with -- @key: value directives.

Where: docs/rfcs/0001-project-files.md

Done when

  • The RFC has been open for comment for at least one week
  • Its open questions are answered in the RFC
  • It is approved as set out in GOVERNANCE.md and merged with status Accepted

Blocks: #7, #8, #12

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 29, 2026

    @codecapo
    OwnerAuthor

    Draft RFC is up for comment in #20.

  3. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. It collects the decisions that RFC 0001 has to settle into one revision, #73, which should be folded into the RFC before it is accepted, if the comment period allows; the RFC is still a Draft (docs/rfcs/0001-project-files.md:4 at a0b5e6c). Bare line numbers below refer to that file.

    Suggested additions to the Done-when:

    • The docs: revise RFC 0001 with the incremental rules (decision D2) #73 revision is folded into the RFC before its status changes to Accepted, if the comment period allows (Tracking: Phase 0 (0.1.0) #19 gives its end as 2026-11-05). If the revision is not ready when the period ends, the RFC stays a Draft and the period runs longer, which GOVERNANCE.md:28-29 permits because one week open for comment is the minimum. The RFC is to be agreed before the compiler is written (:18-21), so the design review of 2026-11-01 counts the replace-commit and incremental-rule questions (findings C01 to C03) as blockers for its acceptance, and once it is accepted, changing what a directive means needs a new RFC and a format bump (:156-158).
    • The RFC settles what a relative source path is resolved against. It says relative to the project (:77), but the CSV connector hands the pattern to glob::glob as written (crates/connectors/src/csv.rs:82), so today the files found depend on the directory the process runs in. The position records each file by the path the glob yields (crates/connectors/src/csv.rs:7-10), so a run from another directory skips any file whose relative name is already recorded, even though it is a different file.
    • The RFC says whether one source per Raw table is a rule. It says that sources/ defines Raw tables and that each file is one source (:45-46), and its layout maps orders.toml to bronze.orders (:38), but it does not say that a Raw table has only one source. The position is held in one property of the Raw table, infostash.position (crates/connectors/src/lib.rs:19-21), so two sources feeding one table would overwrite each other's position. connectors: glob once per job, skip dotfiles, symlinks, empty and fresh files #78 also lists one source per Raw table in the producer contract it writes into the CSV connector's doc comment.

    Work that moved to new tickets:

    • docs: revise RFC 0001 with the incremental rules (decision D2) #73: revises RFC 0001 in one change (decision D2, 3 points), with these headline items:
      • The @check row (:119) says that repeated checks form a list; today directives() keeps one value per key (crates/compiler/src/lib.rs:9-16, as :160-163 notes), so the two checks at :90-91 become one (C32).
      • The incremental rule at :124-128 is specified as an allow-list over the unoptimized DataFusion logical plan, and the RFC says whether the compiler or the runner checks it (C03).
      • Joins and all aggregates are named non-incremental in Phase 0, and ready/daily_revenue.sql (:101-107) gains -- @full_refresh: true (C02, C18).
      • The @unique_key (:118) and @full_refresh (:121) rows say the compiler rejects both until a replace commit lands, which storage: spike the replace commit route (decision D1) #72 decides, or both are withdrawn if storage: spike the replace commit route (decision D1) #72 ends append-only (C01, C02).
      • @partition_by (:120) gets a grammar for Partition tables with @partition_by #36, or its row and its use at :103 are removed (C19).
      • The clean/orders.sql example (:93-97) uses try_cast, a lint rejects strict casts over Raw columns in Clean models, since Raw columns are all text (crates/connectors/src/csv.rs:3), and every cast column gets an implicit check (C14).
      • Explicit checks run after the cast check on the typed output columns, so amount >= 0 (:90) tests the decimal from :96 and not the Raw text, and the @check row (:119) either keeps sending a NULL result to quarantine or switches once to accepting NULL, with is not null for required values (C14, S6-08).
      • Open question 4 (:175-176) gets an ordering rule built on the Raw snapshot sequence number and a new _source_row column (the row number within its file), and now() is pinned to one start time per job and reused on rerun (C15, C17).
      • Each model commits a hash of its normalized SQL, directives and input schema ids with its checkpoint, and a run refuses on a mismatch (C16).
      • The CSV position also stores the source's path pattern, and a run fails when the pattern changes (C08).

    Source: design review of 2026-11-01, findings C01, C02, C03, C08, C14, C15, C16, C17, C18, C19 and C32, design point S6-08, decision D2, and the open questions of its sections 3.6 and 6.5.

  4. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    #101 answers open question 3 of RFC 0001: every CSV column is read as nullable text and typing happens in Clean models with try_cast. A declared schema in the source file can come later as a connector key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/docsDocs, RFCs and ADRsrfcNeeds or is an RFC

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions