Skip to content

feat(compiler): load a project and order its models - #120

Open
codecapo wants to merge 1 commit into
mainfrom
feat/compiler-project-graph
Open

codecapo wants to merge 1 commit into
mainfrom
feat/compiler-project-graph

Conversation

@codecapo

Copy link
Copy Markdown
Owner

What

The compiler now loads a whole project. Project::load reads
infostash.toml, the sources and the models as RFC 0001 describes (the
revision merged in #101), checks every name, directive and table read,
and returns the models in run order.

This covers the Done-when list of #7. Planning each model into a logical
plan follows in a second PR, so #7 stays open.

How

  • directives.rs implements the header grammar: the header is the run of
    blank and -- lines at the top, a directive starts at column one with
    -- @, and the key (or check name) ends at the first :. Unknown keys,
    empty values, names on anything but @check, repeats and duplicate check
    names are errors. @partition_by is now an unknown key.
  • reads.rs parses the SQL with DataFusion and walks it for table names.
    CTE names and table functions are skipped, and unquoted names are lower
    cased as DataFusion does. A name must be <layer>.<table>, using either
    name for the layer.
  • graph.rs checks that each read is defined and is not in a later
    layer, then orders the models so each runs after what it reads. Ties go
    in table order, so the order is the same every run. A cycle is named in
    full, from the file and line where it is first read.
  • config.rs reads the TOML files with unknown keys rejected.
  • lib.rs walks sources/ (top level only) and clean/ and ready/
    (recursively), validates file names and rejects two files with one name
    in a layer, naming both.

New public API: Project (with load, name, warehouse, sources,
models), Source, Model, Directives and Check. It replaces the old
directives() function, which nothing else used.

Things to know

  • New dependencies:
    • datafusion, already in the workspace: the engine runs this SQL, so
      reading it with the same parser means there is no second dialect to
      keep in step.
    • serde and toml: RFC 0001 chose TOML, and deny_unknown_fields
      gives the "unknown keys are an error" rule for free.
    • tempfile (dev only), for project directories in tests.
  • Source::options is a toml::Table. Every key but connector belongs
    to the connector, so the connector checks them in their own types.
  • A missing infostash.toml is Error::NotFound. A file that can't be
    read is Error::Storage. Everything else is Error::Parse, with
    file:line where there is a line.
  • Rejecting @unique_key and @full_refresh until the replace commit is
    in use is left to compiler: add an incremental eligibility classifier with a function allow-list #84. This PR only parses them.
  • Where the Raw schemas come from at compile time decides the shape of
    the planning step, so that comes next.

Tests

  • 33 unit tests in the compiler crate. Each error has its own test, and
    each one writes a project into a temporary directory and checks the
    full message, including the file and line. The RFC's own example loads
    with both checks in file order.
  • Each rule was broken in turn (layer order, undefined reads, CTEs,
    table functions, lower casing, the end of the header, repeats,
    __quarantine, subdirectories, file extensions), and a test failed
    every time.
  • cargo fmt, clippy -D warnings, cargo test --workspace,
    cargo deny, typos, cargo machete and cargo doc -D warnings pass.
    Workspace line coverage is 98.0%.

Project::load reads a project laid out as RFC 0001 (revised in #101)
describes: infostash.toml, the sources in sources/*.toml and the models
in clean/ and ready/, where subdirectories only organise files.

Each model's header is read with the RFC's grammar, and the tables it
reads come from its SQL as DataFusion parses it, so CTE names and table
functions are not mistaken for tables. Every name must carry its layer,
reads must be defined and may not reach a later layer, and a cycle is
named in full. Errors give the file and, where there is one, the line.
Models are returned in run order, ties broken by table name.

The old string-splitting directives() is replaced by the Directives
type. Planning each model is left for a follow-up.

Refs #7

Signed-off-by: Mayur Kale <hello@mayurkale.com>
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • New Features
    • Added project loading for configuration, data sources and SQL models.
    • Added validation of project settings, model metadata and SQL table references.
    • Models are now ordered by their dependencies, with clear errors for invalid references and cycles.
  • Bug Fixes
    • Project and parsing errors now identify the relevant file and, where available, the line number.
  • Tests
    • Added coverage for project loading, configuration validation, SQL parsing and dependency ordering.

The compiler now loads projects from TOML configuration, parses source and model metadata, extracts SQL table reads, validates dependencies, and returns models in dependency order.

Changes: Project loading and model ordering

Layer / File(s) Summary
Project format and model metadata
crates/compiler/Cargo.toml, crates/compiler/src/config.rs, crates/compiler/src/directives.rs, crates/compiler/src/reads.rs
Adds project and source TOML parsing, model-header directive parsing, and SQL table-read extraction. Parsing validates configuration fields, directive values, and table references.
Read validation and dependency ordering
crates/compiler/src/graph.rs
Validates model reads against defined tables and layer order. Orders models by dependency and reports dependency cycles.
Project loading and integration
crates/compiler/src/lib.rs, crates/compiler/src/tests.rs
Adds Project::load, source and model discovery, model validation, and project accessors. Integration tests cover loading, ordering, parsing errors, and storage errors.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Loader as Project::load
  participant Config as Project configuration parser
  participant Sources as Source loader
  participant Models as Model file scan
  participant Reads as SQL read parser
  participant Graph as Dependency graph
  Loader->>Config: Parse project configuration
  Loader->>Sources: Load source TOML files
  Loader->>Models: Discover Clean and Ready SQL files
  Models->>Reads: Parse directives and table reads
  Loader->>Graph: Validate reads and order models
Loading

Possibly related issues

  • #5: The loader implements the project format defined by the RFC.
  • #7: The loader adds project loading and dependency graph construction; planning each model remains deferred.

Possibly related PRs

Merge Risk: Moderate · up to 7539a

The loader currently accepts projects that the project-file RFC says must be rejected, and it can miss invalid table reads in nested queries. Fix these two validation gaps and the warehouse path check before merging; the symlink handling is a minor edge case.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a compiler project loader that loads a project and orders its models. It matches the raw summary, which describes Project::load and run-order graph logic. It is concis…
Description check Passed The description explains the new project loader, the directive grammar, DataFusion-based read parsing, dependency ordering, config parsing, the new public API, dependencies, and tests. It matches the …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a compiler project loader that loads a project and orders its models. It matches the raw summary, which describes Project::load and run-order graph logic. It is concise and specific.

Full details: Description check

Explanation

The description explains the new project loader, the directive grammar, DataFusion-based read parsing, dependency ordering, config parsing, the new public API, dependencies, and tests. It matches the changeset in the raw summary and objectives.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.17864% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.06%. Comparing base (2492912) to head (7539a87).

Files with missing lines Patch % Lines
crates/compiler/src/reads.rs 98.31% 2 Missing ⚠️
crates/compiler/src/config.rs 96.42% 1 Missing ⚠️
crates/compiler/src/directives.rs 99.31% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #120      +/-   ##
==========================================
+ Coverage   99.04%   99.06%   +0.01%     
==========================================
  Files          16       20       +4     
  Lines        2521     2995     +474     
==========================================
+ Hits         2497     2967     +470     
- Misses         24       28       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2492912 and 7539a87.

Files ignored due to path filters: 1

  • Cargo.lock is excluded by !**/*.lock
Files selected for processing: 7

  • crates/compiler/Cargo.toml
  • crates/compiler/src/config.rs
  • crates/compiler/src/directives.rs
  • crates/compiler/src/graph.rs
  • crates/compiler/src/lib.rs
  • crates/compiler/src/reads.rs
  • crates/compiler/src/tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment on lines +78 to +79
"unique_key" => directives.unique_key = Some(columns(value).map_err(err)?),
"full_refresh" if value == "true" => directives.full_refresh = true,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Reject directives that the current compiler cannot support.

If a model declares @unique_key or @full_refresh, Project::load currently succeeds and returns that directive. RFC 0001 requires rejection until the replace commit is in use; it records that the commit is not built yet. Reject these directives at load time rather than presenting the project as valid. (raw.githubusercontent.com)

.collect();
Ok(Self {
name: file.project.name,
warehouse: root.join(file.warehouse.path),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Reject absolute warehouse paths.

root.join returns the configured path unchanged when that path is absolute. An absolute warehouse.path therefore places the warehouse outside the project. RFC 0001 defines this setting as "a local directory, relative to the project". Check that the path is relative before joining it to root, and return an error otherwise.

A .. component can also resolve outside the project root. Consider rejecting it in the same check.

Reference

RFC 0001: Project file format

Comment on lines +211 to +213
let kind = fs::metadata(root.join(&path)).map_err(|e| io_error(&path, &e))?;
if kind.is_dir() && recurse {
collect(root, &path, ext, recurse, files)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

Guard directory recursion against symlink loops.

fs::metadata follows symbolic links. A link that points to an ancestor directory is reported as a directory, so collect enters it again. The path grows with each level, for example clean/loop/loop/....

A single loop stops with an error. The operating system limits how many symbolic links one path lookup can follow, and the resulting ELOOP is propagated by ?. A directory with several links to an ancestor branches at each level. The work can then grow quickly with depth, and loading can run for a long time before it fails.

Skip symbolic links, or track visited directories by canonical path or by device and inode. Choose the option that matches the intended symlink policy.

The cited Rust documentation describes symlink_metadata, which does not follow links. Cite the fs::metadata documentation instead.

Comment on lines +50 to +51
self.ctes
.extend(with.cte_tables.iter().map(|cte| normalise(&cte.alias.name)));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Heavy lift

Keep CTE names within their query scope.

For SELECT * FROM orders WHERE EXISTS (WITH orders AS (SELECT 1) SELECT 1 FROM orders), the inner CTE adds orders to the global set. resolve then discards the outer, unqualified orders read too. Project::load accepts a query that should fail the unqualified-table check. Resolve table references against the CTEs visible at each reference, and test this nested-scope case. The pinned sqlparser visitor traverses nested queries. (raw.githubusercontent.com)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant