Skip to content

Compile a project into a dependency graph #7

Open

Description

@codecapo

Load a project from disk and build its dependency graph, as specified in RFC 0001.

Crate: crates/compiler

Done when

  • Reads infostash.toml, sources/*.toml and every .sql file under clean/ and ready/
  • Parses directives from the header only; @check may repeat; unknown keys are errors
  • Finds the tables each model reads with DataFusion's SQL parser (not regex)
  • Errors, each with file and line: unqualified table, unknown table, cycle (naming it), reading a higher layer, two models with one name
  • Returns models in a valid run order
  • Unit tests for every error above

Depends on: #5
Blocks: #10, #12

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. It keeps the directive parser, the dependency graph and the run order here, adds a DataFusion logical plan for each model, and leaves the incremental rules built on them to new tickets.

    Suggested additions to the Done-when:

    Work that moved to new tickets:

    Suggested re-score: 13 instead of the 5 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores two milestone M2 items that stay in this issue, "directive parser" at 3 and "SQL to logical plan, DAG and layer rules" at 8. Together they are 11, more than the 5 that #19 lists, and the next number on the Fibonacci scale is 13. This is a suggestion only; #19 decides.

    Source: design review of 2026-11-01, findings C03, C14, C16, C18, C19 and C32 (points from section 8.5, M2).

  3. codecapo commented on Nov 12, 2026

    @codecapo
    OwnerAuthor

    Starting this in two pull requests, to keep each under the size limit. The first covers the original Done-when list: loading infostash.toml, the sources and the models, the header grammar from RFC 0001 as revised in #73, the tables each model reads from the parsed SQL, the layer, name and cycle errors with file and line, and the run order. The second plans each model into an unoptimised DataFusion logical plan in run order; it needs a decision first on where a Raw table's columns come from at compile time, before any file has been loaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions