Skip to content

compiler: add a model version hash and refuse a run on mismatch #87

Open

Description

@codecapo

Nothing detects that a model's SQL changed between runs. Job carries a ProjectVersion documented as a content hash of the project files (crates/model/src/lib.rs:93-96 at a0b5e6c, the project field at :105), but no code computes one: the only value ever built is the literal "abc" in a test module (crates/jobs/src/lib.rs:30, :52). Once the second run processes only new rows (ROADMAP.md:13), an edited model appends rows from its new SQL beside rows from its old SQL. The write path already refuses an edit that changes an output column's type (crates/storage/src/lib.rs:380-381), but not one that keeps it. For example, take a model that reads amount from an append-only table where it is already decimal(12, 2), the type the RFC 0001 example casts it to (docs/rfcs/0001-project-files.md:96). If after ten runs it selects cast(amount * 100 as decimal(12, 2)) as amount instead, every later run appends amounts 100 times larger beside the old ones, and nothing flags it.

This issue computes a version hash per model and commits it under an infostash. key beside the model's per-input checkpoint, in the same commit as the rows. ADR 0006 already requires this for checkpoints (docs/adr/0006-ordered-job-commits.md:49-51) and append_if_head already supports it (crates/storage/src/lib.rs:63, :188, :209, :213), so no new ADR is needed. In Phase 0 a mismatch fails the model's run with an error that names the model, which infoctl run (#14) handles like any other failed model. Falling back to a full refresh instead (docs/rfcs/0001-project-files.md:121) needs the replace commit that #72 decides and #88 builds, so it is not part of this issue.

Schema drift on the CSV side is out of scope too: a new source column makes every later ingest of its table fail (crates/storage/src/lib.rs:373-378). #79 turns that into an early error that names the file, #94 corrects the "Handles schema changes" line in docs/architecture.md:96, and additive Raw schema evolution stays a later opt-in.

Crates: crates/compiler (the hash), crates/jobs (the check before a model runs and the key in its commit)

Done when

  • Each model's version hash is computed from its normalized SQL, its directives and the current schema id of each input table, as the rule from docs: revise RFC 0001 with the incremental rules (decision D2) #73 defines them, with a fixed digest rather than std::hash::DefaultHasher (whose algorithm may change between Rust releases), so the same files and input schemas always give the same value
  • Every commit to a model's target writes the hash under an infostash.-prefixed key through the append_if_head properties path, in the same commit as the rows and the per-input checkpoint from In-process runner: stage, run tasks, commit once #13, so a model's first run records it and no checkpoint is ever stored without the hash of the SQL that produced it
  • Before a model reads any input, the hash stored on its target is taken from the same Warehouse::state load that supplies the head the commit is checked against (crates/storage/src/lib.rs:153-161) and compared with the computed hash, and a model with no stored hash or a matching hash runs normally
  • On a mismatch the model is refused before anything is committed to its target or its quarantine table (Row checks and quarantine tables #12), with an error that names the model and says that its SQL, directives or input schemas changed since its last run, and the refusal is not retried
  • Test: a model selects amount from an append-only input table where it is decimal(12, 2), such as one the test creates, and runs; new rows land in the input and a second run without edits is not refused; then the model selects cast(amount * 100 as decimal(12, 2)) as amount instead, more rows land, and the third run is refused with the target's head, rows and checkpoint unchanged, so rows from the old and the new SQL never mix
  • Unit tests: the same files and input schema ids always give the same hash, pinned to a literal value, reformatting the query's whitespace leaves it unchanged, and changing an expression in the query, one @check directive or one input's schema id changes it

Points: 3 (M2 version hash with refuse-on-mismatch 3)

Depends on: #73 (the version-hash rule), #7 (the header directives and input tables the hash covers), #13 (the model commit that carries the per-input checkpoint)
Blocks: nothing

Source: design review of 2026-11-01, finding C16 (points from section 8.5, M2).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    #101 adds one case to the mismatch rule: a run of a model that declares @full_refresh: true does not compare the stored hash; it replaces the table and commits the new hash in the same commit. An incremental run still refuses on a mismatch. The hash covers each check's name and expression in file order. See the Model versions paragraph in that PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/compilercrates/compilerblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions