Skip to content

Row checks and quarantine tables #12

Open

Description

@codecapo

Rows failing a model's @check expressions go to <table>__quarantine instead of the target, and the run still succeeds.

Crates: crates/compiler, crates/engine

Done when

  • Each @check is a SQL boolean; false or null sends the row to quarantine
  • Quarantine rows keep the original columns plus which check failed and the run time
  • Target and quarantine rows land under the same job id: quarantine commits first and the target last, and a rerun of the job completes it without duplicates (iceberg-rust has no multi-table commit)
  • Test: a crash between the two commits, then a rerun, leaves each row in exactly one table once
  • JobMetrics::rows_quarantined is filled in
  • Tests for pass, fail and null cases

Depends on: #5, #10
Blocks: #14

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 30, 2026

    @codecapo
    OwnerAuthor

    Spike findings (commit semantics)

    Target and quarantine can't share one atomic commit. iceberg-rust has no public multi-table commit (TableCommit::build is pub(crate), and nothing calls the REST V1_COMMIT_TRANSACTION).

    The spike tested three fallbacks. Recommended, and now in the criteria:

    • Both commits carry infostash.job-id in the snapshot summary.
    • Quarantine commits first and the target last, so the target's job id marks the job as done. If every row fails, the target still gets a marker snapshot.
    • A rerun with the same job id commits only the table that is missing it.
    • Between the two commits a reader can see quarantine rows whose target rows aren't there yet. Only trust quarantine rows whose job id is in the target's history.

    Alternatives (a failed_check column in the target, or swapping both SQLite catalog pointers in one transaction) work but break ADR 0003 naming or tie us to SQLite.

    Points: 5 -> 8. Criteria updated below.

  3. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. It keeps the quarantine split here and adds what the split needs to be correct and repeatable, while the RFC text, the job outcome type and a new Raw column move to new tickets. The issue now waits on two decisions, #73 and #74.

    Suggested additions to the Done-when:

    • A strict cast over a Raw-derived column in a Clean model is a compile error naming the column, found on the analysed plan so that :: casts and the casts that type coercion inserts (where amount > 0 over text) are caught too. Raw columns read from a file are nullable text (crates/connectors/src/csv.rs:199 at a0b5e6c) and DataFusion's CAST is strict, so one bad string under the casts of the RFC example (docs/rfcs/0001-project-files.md:94-97) fails the whole run before any check runs.
    • Every cast column gets the implicit check x is null or try_cast(x) is not null, the only way to tell a NULL in the source from a failed conversion; explicit @checks then run on the typed value, and a failed conversion is reported as cast:<col>, not as a failure of the column's own check. Test: a Clean model that casts amount and checks amount >= 0, over N Raw rows where one amount is 12,50, commits N-1 target rows and one quarantine row whose _checks_failed holds cast:amount and not amount >= 0. A bad value outside a cast (checked arithmetic, a timestamp parse with a format) still fails the run, with the non-retryable Data error dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 defines, naming the model.
    • Every @check of a model applies, not only the last: directives() collects into a BTreeMap (crates/compiler/src/lib.rs:9-16), so the RFC's clean/orders keeps only its second check (docs/rfcs/0001-project-files.md:90-91) and a negative amount reaches Clean unnoticed. With the list parsed in Compile a project into a dependency graph #7, a test with both checks shows both applied, and a row failing both is quarantined once with both checks in _checks_failed.
    • The existing item "false or null sends the row to quarantine" follows the one NULL rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 settles: today's text, which quarantines a NULL result unlike SQL CHECK (docs/rfcs/0001-project-files.md:119), or NULL accepted with required values written as is not null. A blank optional cell arrives as NULL because the reader keeps arrow-csv's default null handling (crates/connectors/src/csv.rs:106-108), so the null tests include @check: discount >= 0 over a blank discount, quarantined under today's text and accepted under the alternative.
    • The existing quarantine-row item becomes concrete: each quarantine row is the input row as read (for a Clean model, the Raw text before any cast, with _source_file, _loaded_at and, once connectors: add _source_row to Raw tables #86 lands, _source_row) plus _checks_failed, a list<utf8> of the checks the row failed (implicit ones included, named as docs: revise RFC 0001 with the incremental rules (decision D2) #73 decides), and a per-row _job_id in place of the run time, all from one plan split on whether that list is empty. The job id has to be a column because readers that need a consistent view trust only quarantine rows whose job id is in the target's history (docs/adr/0006-ordered-job-commits.md:67-70), and the job id in a snapshot summary cannot be joined from SQL.
    • The quarantine and target commits both carry each input's to, the end of the range the job read, beside the job id the existing item already asks for, because under ADR 0006 a model's last processed snapshot per input is written in the same commit as the rows it describes (docs/adr/0006-ordered-job-commits.md:49-51); on the target this is the checkpoint of In-process runner: stage, run tasks, commit once #13, and the quarantine commit records the same value.
    • Checks run on every input row before any deduplication, so the @unique_key merge of Merge rows on @unique_key #35 only sees accepted rows, and in a @unique_key model a NULL key fails an implicit key_not_null:<col> check: a blank order_id in clean/orders (docs/rfcs/0001-project-files.md:89, :94) satisfies its cast check and would otherwise collapse with every other NULL key.
    • The crash-and-rerun test also covers @check: ordered_at <= now() over a row whose ordered_at falls between the first attempt's start and the rerun: the engine plans under the job's start time (Run a model with DataFusion and append to its table #10) and the runner keeps that time with the job id and supplies it again on a rerun (In-process runner: stage, run tasks, commit once #13), so the rerun computes the same split as the first attempt, as docs/adr/0006-ordered-job-commits.md:71-72 requires, and no row lands in both tables or in neither.

    Work that moved to new tickets:

    Suggested re-score: no change. Section 8.5 of the design review of 2026-11-01 scores the quarantine split with try_cast, ordering and _job_id at 8, as #19 does, within milestone M2 at 50 points, and scores Raw _source_row (2, #86) and the directive parser (3, #7) as separate items; #19 decides.

    Source: design review of 2026-11-01, findings C14, C15 and C32 and design points S6-02, S6-04 and S6-08, with its sections 6.4 and 8.5 (D2, D3 and M2).

  4. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    RFC 0002 (#102) fixes how this ticket reports its two commits: JobOutcome::Committed { commits, metrics } carries one CommitRecord per output table in commit order, so the quarantine record comes first and the target last. TaskStats gains rows_quarantined, which the runner sums into JobMetrics::rows_quarantined. Values that fail a cast and still reach the engine map to the new Error::Data, which is never retried. One note for the crash test: a job that loses its target commit to a Conflict, or to a non-retryable error, after its quarantine commit is abandoned for a new id, which quarantines the same rows again; the "exactly once" property holds over the rows the ADR 0006 reader rule shows (job id in the target's history), and the RFC says so.

  5. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) names this ticket, with #13 and #81, as where the rerun rule is implemented: a job that failed after any of its commits is rerun under the same job id, skipping the tables that already carry it, before any new job for the target is created. The quarantine commit must set infostash.job-id as a table property as well as in the summary, since the ADR requires that of every commit and the runner precheck compares the two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions