Slice 3: validation stage (gates promotion)

[wolfieman]

Epic: Validation · Depends on: slices 1 + 2.

Covers: aggregate vocabulary/format/cross-table checks against configured thresholds; cross-field date rules with an injected reference date (never the wall clock); referential check with the per-run reconciliation metric (orphan count, match rate, trend).

Gate: a clean two-source run promotes; a deliberately poisoned fixture run halts with a red report before any load; reports keyed by run id.

[wolfieman]

Requirements mined (Requirements phase, 2026-06-13)

Full detail in docs/requirements-validation.md (drafted). Grounded only in the project's own docs (project-plan.md, data-cleaning-protocol.md, master-data-documentation.md, DATA_DICTIONARY.md). Slice runs offline on the slice-1/2 processed frames — no live acquisition, no D3 terms gate. Report-don't-coerce: validation flags, never mutates; no load (that is slice 4).

V1 Offline input contract — consumes the slice-1 deliverable + slice-2 license master + resolution report; reports keyed by run id.
V2 Referential check — every vendor GC license must exist in the NCLBGC license set (join key); orphans flagged, never dropped.
V3 Reconciliation metric — orphan count, match rate, trend (first run = baseline), from the slice-2 resolution status.
V4 Subcontractor license-format check — (predicate pinned at checkpoint). Scope: NCLBGC is GC-only, so this is a self-contained format/pattern check on the eVP trade-license columns, not a cross-table lookup.
V5 HUB license-appropriateness — HUB vendors hold an appropriate license for each participating trade (predicate pinned at checkpoint).
V6 Aggregate vocab/format checks against thresholds — (threshold values pinned at checkpoint); thresholds gate, never equality (REQ-20); a zero-where-tens-expected is a red flag.
V7 Cross-field date rules with an injected reference date (never the wall clock) — (rule set pinned at checkpoint).
V8 Promotion gate — promote only when tests pass AND thresholds hold AND conservation holds.
V9 Poisoned-fixture red test — seeded orphan / out-of-vocab / bad date halts red before load.

Ready check: deps (slices 1+2) met; gate stated above on this card; no blocking entry decisions (the V4–V7 predicates/thresholds are pinned at the mid-slice owner checkpoint after the Analysis profile, not at entry). Next phase = Analysis (profile the two-source frames), then the blocking owner checkpoint.

[wolfieman]

Requirements REVISED after the owner walkthrough (2026-06-13)

Full detail in docs/requirements-validation.md. Slice 3 is the validation + reconciliation gate between slice 2's matched data and the slice-4 load — not re-matching. Offline; report-don't-coerce; no DB write.

V1 Offline input contract — slice-1/2 processed deliverables + the slice-2 vendor -> discovered-license linkage.
V2 Reconcile — for each matched vendor (by license OR name; a match is a match), write the board-confirmed GC license into the vendor master (fill blanks, overwrite invalids), logged as a correction. License master read-only. Unresolved = the difference. Output = corrected eVP vendor master -> slice 4.
V3 Reconciliation metric — match rate %, bucket counts, fix counts, trend (first run = baseline).
V4 Aggregate vocab/format violation tally vs configured thresholds (values pinned at checkpoint).
V5 Promotion gate (A + zero-floor) — promote on conservation + threshold + green suite; low match rate reports (not halt); ~zero matches / threshold breach / structural failure halts before load.
V6 Poisoned-fixture red test — seeded orphan + out-of-vocab halts red before load.
V7 Validation report — values-free, keyed by run id (metric + fix counts + violations + conservation + verdict).

Dropped (deliberate, not gaps): subcontractor license-format check, HUB check, cross-field date rules.
Opens: a slice-2 follow-up (persist the linkage) — prerequisite for V2.
Doc edit done: design-nclbgc.md §5 no longer asks slice 3 to re-scrutinize name matches.

[wolfieman]

Checkpoint — Analysis complete, thresholds pinned (slice 3)

Mid-slice owner checkpoint (plan §8.2 step 3). All figures are counts / percentages only, re-derived from the frozen slice-1 + slice-2 frames (index-aligned 570/570). Profile artifact: data/processed/profile/slice3-reconciliation-profile-*.json (gitignored).

Reconciliation profile (V2/V3 sizing)

• 570 vendors. Buckets: matched-by-license 374 (65.6%), matched-by-name 32 (5.6%), unresolved 164 (28.8%). Match rate 71.2%.
• eVP GC-license state: 379 well-formed / 157 blank / 34 malformed.
• Reconciliation writes = exactly the matched-by-name set: 16 blanks filled + 16 invalids overwritten = 32 logged corrections. The 374 matched-by-license already hold the confirmed number → no-op.
• Canary: 5 digit-class orphans (well-formed numbers that did not resolve) → left unresolved.
• Conservation: 406 matched + 164 unresolved = 570 ✓.

Violation tallies (V4 input)

• eVP: 74 total — GC 34, Trades 19, Electrical 15, Plumbing 3, Mechanical 2, phone 1.
• NCLBGC: 0 (7 dedup drops).

Decisions pinned

1. V4 thresholds = upper edge of the 95% normal range, per rule (x + 1.645·√x, one-sided — a run above the line is a regression, not chance):
   GC ≤ 44 · Trades ≤ 27 · Electrical ≤ 22 · Plumbing ≤ 6 · Mechanical ≤ 5 · phone ≤ 3 · eVP total ≤ ~90 · NCLBGC ≤ 3 (ε on a zero baseline). Gated per-rule, total as backstop. (n=1 baseline ⇒ model-based Poisson interval; swap to empirical control limits once refresh runs accumulate.)
2. Match rate: report a degraded rate (below the ~68% CI floor) but still promote; halt only on near-zero matches (collapse, operationally <5%).
3. Linkage: offline backfill of the discovered board license from the frozen run (no live re-pull) — slice 3 stays offline.

Next: Design (solo) → test-first → implement → gate → milestone tag v0.5.0.