You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(compiler): load a project and order its models - #120
The compiler now loads a whole project. Project::load reads infostash.toml, the sources and the models as RFC 0001 describes (the
revision merged in #101), checks every name, directive and table read,
and returns the models in run order.
This covers the Done-when list of #7. Planning each model into a logical
plan follows in a second PR, so #7 stays open.
How
directives.rs implements the header grammar: the header is the run of
blank and -- lines at the top, a directive starts at column one with -- @, and the key (or check name) ends at the first :. Unknown keys,
empty values, names on anything but @check, repeats and duplicate check
names are errors. @partition_by is now an unknown key.
reads.rs parses the SQL with DataFusion and walks it for table names.
CTE names and table functions are skipped, and unquoted names are lower
cased as DataFusion does. A name must be <layer>.<table>, using either
name for the layer.
graph.rs checks that each read is defined and is not in a later
layer, then orders the models so each runs after what it reads. Ties go
in table order, so the order is the same every run. A cycle is named in
full, from the file and line where it is first read.
config.rs reads the TOML files with unknown keys rejected.
lib.rs walks sources/ (top level only) and clean/ and ready/
(recursively), validates file names and rejects two files with one name
in a layer, naming both.
New public API: Project (with load, name, warehouse, sources, models), Source, Model, Directives and Check. It replaces the old directives() function, which nothing else used.
Things to know
New dependencies:
datafusion, already in the workspace: the engine runs this SQL, so
reading it with the same parser means there is no second dialect to
keep in step.
serde and toml: RFC 0001 chose TOML, and deny_unknown_fields
gives the "unknown keys are an error" rule for free.
tempfile (dev only), for project directories in tests.
Source::options is a toml::Table. Every key but connector belongs
to the connector, so the connector checks them in their own types.
A missing infostash.toml is Error::NotFound. A file that can't be
read is Error::Storage. Everything else is Error::Parse, with file:line where there is a line.
Where the Raw schemas come from at compile time decides the shape of
the planning step, so that comes next.
Tests
33 unit tests in the compiler crate. Each error has its own test, and
each one writes a project into a temporary directory and checks the
full message, including the file and line. The RFC's own example loads
with both checks in file order.
Each rule was broken in turn (layer order, undefined reads, CTEs,
table functions, lower casing, the end of the header, repeats, __quarantine, subdirectories, file extensions), and a test failed
every time.
cargo fmt, clippy -D warnings, cargo test --workspace, cargo deny, typos, cargo machete and cargo doc -D warnings pass.
Workspace line coverage is 98.0%.
Project::load reads a project laid out as RFC 0001 (revised in #101)
describes: infostash.toml, the sources in sources/*.toml and the models
in clean/ and ready/, where subdirectories only organise files.
Each model's header is read with the RFC's grammar, and the tables it
reads come from its SQL as DataFusion parses it, so CTE names and table
functions are not mistaken for tables. Every name must carry its layer,
reads must be defined and may not reach a later layer, and a cycle is
named in full. Errors give the file and, where there is one, the line.
Models are returned in run order, ties broken by table name.
The old string-splitting directives() is replaced by the Directives
type. Planning each model is left for a follow-up.
Refs #7
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Added project loading for configuration, data sources and SQL models.
Added validation of project settings, model metadata and SQL table references.
Models are now ordered by their dependencies, with clear errors for invalid references and cycles.
Bug Fixes
Project and parsing errors now identify the relevant file and, where available, the line number.
Tests
Added coverage for project loading, configuration validation, SQL parsing and dependency ordering.
The compiler now loads projects from TOML configuration, parses source and model metadata, extracts SQL table reads, validates dependencies, and returns models in dependency order.
Changes: Project loading and model ordering
Layer / File(s)
Summary
Project format and model metadata crates/compiler/Cargo.toml, crates/compiler/src/config.rs, crates/compiler/src/directives.rs, crates/compiler/src/reads.rs
Adds project and source TOML parsing, model-header directive parsing, and SQL table-read extraction. Parsing validates configuration fields, directive values, and table references.
Read validation and dependency ordering crates/compiler/src/graph.rs
Validates model reads against defined tables and layer order. Orders models by dependency and reports dependency cycles.
Project loading and integration crates/compiler/src/lib.rs, crates/compiler/src/tests.rs
Adds Project::load, source and model discovery, model validation, and project accessors. Integration tests cover loading, ordering, parsing errors, and storage errors.
sequenceDiagram
participant Loader as Project::load
participant Config as Project configuration parser
participant Sources as Source loader
participant Models as Model file scan
participant Reads as SQL read parser
participant Graph as Dependency graph
Loader->>Config: Parse project configuration
Loader->>Sources: Load source TOML files
Loader->>Models: Discover Clean and Ready SQL files
Models->>Reads: Parse directives and table reads
Loader->>Graph: Validate reads and order models
Loading
Possibly related issues
#5: The loader implements the project format defined by the RFC.
#7: The loader adds project loading and dependency graph construction; planning each model remains deferred.
Possibly related PRs
codecapo/infostash#20: This PR defines the project format that the compiler loader parses.
Merge Risk: Moderate · up to 7539a
The loader currently accepts projects that the project-file RFC says must be rejected, and it can miss invalid table reads in nested queries. Fix these two validation gaps and the warehouse path check before merging; the symlink handling is a minor edge case.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: a compiler project loader that loads a project and orders its models. It matches the raw summary, which describes Project::load and run-order graph logic. It is concis…
Description check
The description explains the new project loader, the directive grammar, DataFusion-based read parsing, dependency ordering, config parsing, the new public API, dependencies, and tests. It matches the …
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: a compiler project loader that loads a project and orders its models. It matches the raw summary, which describes Project::load and run-order graph logic. It is concise and specific.
Full details: Description check
Explanation
The description explains the new project loader, the directive grammar, DataFusion-based read parsing, dependency ordering, config parsing, the new public API, dependencies, and tests. It matches the changeset in the raw summary and objectives.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❌ Patch coverage is 99.17864% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.06%. Comparing base (2492912) to head (7539a87).
Reviewing files that changed from the base of the PR and between 2492912 and 7539a87.
Files ignored due to path filters: 1
Cargo.lock is excluded by !**/*.lock
Files selected for processing: 7
crates/compiler/Cargo.toml
crates/compiler/src/config.rs
crates/compiler/src/directives.rs
crates/compiler/src/graph.rs
crates/compiler/src/lib.rs
crates/compiler/src/reads.rs
crates/compiler/src/tests.rs
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Reject directives that the current compiler cannot support.
If a model declares @unique_key or @full_refresh, Project::load currently succeeds and returns that directive. RFC 0001 requires rejection until the replace commit is in use; it records that the commit is not built yet. Reject these directives at load time rather than presenting the project as valid. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
Reject absolute warehouse paths.
root.join returns the configured path unchanged when that path is absolute. An absolute warehouse.path therefore places the warehouse outside the project. RFC 0001 defines this setting as "a local directory, relative to the project". Check that the path is relative before joining it to root, and return an error otherwise.
A .. component can also resolve outside the project root. Consider rejecting it in the same check.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Stability & Availability · Quick win
Guard directory recursion against symlink loops.
fs::metadata follows symbolic links. A link that points to an ancestor directory is reported as a directory, so collect enters it again. The path grows with each level, for example clean/loop/loop/....
A single loop stops with an error. The operating system limits how many symbolic links one path lookup can follow, and the resulting ELOOP is propagated by ?. A directory with several links to an ancestor branches at each level. The work can then grow quickly with depth, and loading can run for a long time before it fails.
Skip symbolic links, or track visited directories by canonical path or by device and inode. Choose the option that matches the intended symlink policy.
The cited Rust documentation describes symlink_metadata, which does not follow links. Cite the fs::metadata documentation instead.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Heavy lift
Keep CTE names within their query scope.
For SELECT * FROM orders WHERE EXISTS (WITH orders AS (SELECT 1) SELECT 1 FROM orders), the inner CTE adds orders to the global set. resolve then discards the outer, unqualified orders read too. Project::load accepts a query that should fail the unqualified-table check. Resolve table references against the CTEs visible at each reference, and test this nested-scope case. The pinned sqlparser visitor traverses nested queries. (raw.githubusercontent.com)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The compiler now loads a whole project.
Project::loadreadsinfostash.toml, the sources and the models as RFC 0001 describes (therevision merged in #101), checks every name, directive and table read,
and returns the models in run order.
This covers the Done-when list of #7. Planning each model into a logical
plan follows in a second PR, so #7 stays open.
How
directives.rsimplements the header grammar: the header is the run ofblank and
--lines at the top, a directive starts at column one with-- @, and the key (or check name) ends at the first:. Unknown keys,empty values, names on anything but
@check, repeats and duplicate checknames are errors.
@partition_byis now an unknown key.reads.rsparses the SQL with DataFusion and walks it for table names.CTE names and table functions are skipped, and unquoted names are lower
cased as DataFusion does. A name must be
<layer>.<table>, using eithername for the layer.
graph.rschecks that each read is defined and is not in a laterlayer, then orders the models so each runs after what it reads. Ties go
in table order, so the order is the same every run. A cycle is named in
full, from the file and line where it is first read.
config.rsreads the TOML files with unknown keys rejected.lib.rswalkssources/(top level only) andclean/andready/(recursively), validates file names and rejects two files with one name
in a layer, naming both.
New public API:
Project(withload,name,warehouse,sources,models),Source,Model,DirectivesandCheck. It replaces the olddirectives()function, which nothing else used.Things to know
datafusion, already in the workspace: the engine runs this SQL, soreading it with the same parser means there is no second dialect to
keep in step.
serdeandtoml: RFC 0001 chose TOML, anddeny_unknown_fieldsgives the "unknown keys are an error" rule for free.
tempfile(dev only), for project directories in tests.Source::optionsis atoml::Table. Every key butconnectorbelongsto the connector, so the connector checks them in their own types.
infostash.tomlisError::NotFound. A file that can't beread is
Error::Storage. Everything else isError::Parse, withfile:linewhere there is a line.@unique_keyand@full_refreshuntil the replace commit isin use is left to compiler: add an incremental eligibility classifier with a function allow-list #84. This PR only parses them.
the planning step, so that comes next.
Tests
each one writes a project into a temporary directory and checks the
full message, including the file and line. The RFC's own example loads
with both checks in file order.
table functions, lower casing, the end of the header, repeats,
__quarantine, subdirectories, file extensions), and a test failedevery time.
cargo fmt,clippy -D warnings,cargo test --workspace,cargo deny,typos,cargo macheteandcargo doc -D warningspass.Workspace line coverage is 98.0%.