diff --git a/.gitignore b/.gitignore index 2f1ca2e..7f5fa70 100644 --- a/.gitignore +++ b/.gitignore @@ -43,6 +43,10 @@ temp/ *.tmp .worktrees/ +# DRC-3402 eval fixtures — regenerable artifacts (see evals/agent-blind-spots/fixtures/README.md) +evals/agent-blind-spots/.tmp/ +evals/agent-blind-spots/fixtures/*/artifacts/ + # Coverage / editor logs .coverage .nvimlog diff --git a/evals/agent-blind-spots/README.md b/evals/agent-blind-spots/README.md new file mode 100644 index 0000000..863b375 --- /dev/null +++ b/evals/agent-blind-spots/README.md @@ -0,0 +1,59 @@ +# Eval — Agent Blind Spots / `/recce-verify` v1 + +Qualitative case-study eval for the `/recce-verify` skill v1. + +## What this is + +Six PR fixtures from `DataRecce/jaffle_shop_golden` exercising distinct verification classes (semantic, row-grain, refactor, type, schema-expansion, multi-model). Each fixture produces one structured case study viewed through three lenses (binary catch, primary evidence tier, counterfactual delta against a frozen agent-only baseline) — these are observation axes, not metrics; **do not aggregate them**. Output is an action-prioritized gap report (target ≤5 entries, with an overflow path) that gates v1 backend additions. + +## Why qualitative, not quantitative + +N=6 PRs is too small for statistics. The eval is a **named-case narrative**, not a leaderboard. Presenting it as "X% accuracy" or "Y% improvement" overstates rigor and breaks credibility with Super / 205DataLab. The rubric is built around case studies; resist the temptation to aggregate. + +## Layout + +``` +evals/agent-blind-spots/ +├── README.md ← this file +├── RUBRIC.md ← scoring rules; read before adding or scoring fixtures +├── build_fixtures.sh ← rebuilds the gitignored artifacts/ per fixture (DRC-3402) +├── fixtures/ ← one directory per PR fixture +│ ├── README.md ← fixture-set caveats + build instructions +│ └── / +│ ├── README.md ← what the PR does + expected verdicts +│ ├── tier-0-baseline.md ← frozen agent-only verdict (template in templates/) +│ ├── commits.txt ← base + head SHAs read by build_fixtures.sh +│ ├── diff.patch ← small source-models diff base..head (committed) +│ └── artifacts/ ← gitignored; produced by build_fixtures.sh +├── templates/ +│ ├── tier-0-baseline.md ← per-fixture frozen baseline template +│ └── gap-report.md ← gap-report template (target ≤5 entries) +└── runs/ + └── / + ├── gap-report.md ← filled gap report for the run + └── -scoring.md ← per-fixture scoring per RUBRIC.md +``` + +Before any eval run, build the gitignored artifacts: + +```bash +cd evals/agent-blind-spots && ./build_fixtures.sh +``` + +See [`fixtures/README.md`](./fixtures/README.md) for the full per-fixture caveats (PR #16 merge head, PR #20 intermediate trap, PR #46 stress test, empty-DuckDB catalog stats, PR #14 older base) and pinned versions. + +## How to run + +1. Pick a fixture in `fixtures/`. +2. If `tier-0-baseline.md` is missing, run the agent in Tier-0 mode (no Recce) and capture the verdict per the template. The baseline file is **frozen at commit**: the commander commits it (per the workspace's normal change-control flow) before authorizing the with-Recce run, and the baseline must not be edited afterwards even if later evidence suggests revision. +3. Run the agent with `/recce-verify` available. +4. Score the run in `runs//-scoring.md` using `RUBRIC.md`. +5. Once all six fixtures are scored, fill `runs//gap-report.md` (action-prioritized; target ≤5 entries, exceed-with-rationale allowed). + +## References + +- Project: [Agent-blind spots: /recce-verify v1](https://linear.app/recce/project/agent-blind-spots-recce-verify-v1-d2bb2d77bff8) +- Rubric source: [DRC-3403](https://linear.app/recce/issue/DRC-3403) +- Fixture set: [DRC-3402](https://linear.app/recce/issue/DRC-3402) +- Skill: [DRC-3404](https://linear.app/recce/issue/DRC-3404) +- Notion: [Single-env Capability Breakdown](https://www.notion.so/35a79451d357807ba2befc365cb74217) diff --git a/evals/agent-blind-spots/RUBRIC.md b/evals/agent-blind-spots/RUBRIC.md new file mode 100644 index 0000000..3bb7bc6 --- /dev/null +++ b/evals/agent-blind-spots/RUBRIC.md @@ -0,0 +1,118 @@ +# Scoring Rubric — Agent Blind Spots / `/recce-verify` v1 + +## Framing + +Qualitative case studies, **not** quantitative eval. N=6 PR fixtures is too small for statistics. The rubric below produces structured case studies; reading them as "X% accuracy" or "Y% improvement" is wrong and breaks credibility with Super / 205DataLab. Always present results as named-case narratives, not aggregates. + +## Per-fixture case-study lenses (3, not metrics) + +Each lens produces a structured observation, not a score. The three together form one case study per fixture. **Do not aggregate across fixtures.** + +### 1. Binary catch + +Did the agent reach the correct verdict — catch the intentional bug, or correctly flag a behavior-preserving refactor as safe? + +| Value | Meaning | +|-------|---------| +| `catch` | Agent correctly identified the situation and recommended the right action (block or approve). | +| `miss` | Agent shipped the change as safe when it wasn't, or rejected a safe change. | +| `partial` | Agent flagged a real concern but for the wrong reason, or missed half of a multi-issue PR. | + +### 2. Primary evidence tier (+ capability subset used) + +Which capability tier produced the **decisive** piece of evidence the agent cited in its verdict? Tiers are nested — a higher tier always has access to lower-tier inputs. + +If the agent cited evidence from multiple tiers, record the higher tier as decisive (tier ordering `0 < 1 < 2` applies across tiers) and list secondary citations under Notes. Within Tier 1, the subsets `1a / 1b / 1c` are **not** ordered — they are orthogonal capability surfaces (and `1a` is computed via `1b`, so they're entangled). If the agent cited evidence from multiple Tier-1 subsets, mark the subset whose evidence appeared in the verdict sentence as **primary**, the rest as **supporting**; primary/supporting replaces "highest" within Tier 1. + +| Tier | Agent capability set | +|------|----------------------| +| 0 | dbt manifest + compiled SQL + git diff only — no Recce. | +| 1 | Tier 0 + Recce against a **single dev environment**. v1 target. Record *which subset* the agent actually used: **1a** column-level lineage (CLL), **1b** AST / SQL semantic analysis, **1c** structured queries against the current dev env (row counts, distributions, nulls). "Tier 1" alone is ambiguous — always record the subset(s). | +| 2 | **Beyond v1 — base environment needed.** Not part of `/recce-verify` v1's offering. Record only when a fixture's verdict provably requires a base/prod comparison (data diff, row-grain delta, lineage delta vs prod). A fixture reaching Tier 2 is a **v2 signal** for the gap report, not a v1 capability claim. | + +### 3. Counterfactual delta against frozen baseline + +Binary catch (lens 1) is recorded **for both** the Tier-0 baseline run and the with-Recce run. The case study's headline finding is the **delta** between those two binary-catch values, not the absolute with-Recce value. + +Order the three catch values as `catch > partial > miss` (closer to ground truth → less close). Every baseline → with-Recce pair falls into one of three buckets: + +| Delta bucket | Examples | What it means | +|--------------|----------|---------------| +| **Improvement** | `miss → catch`, `miss → partial`, `partial → catch` | Recce shifted the verdict toward ground truth. Positive signal — describe *what* evidence drove the shift. | +| **Same** | `miss → miss`, `partial → partial`, `catch → catch` | No shift in verdict. Still useful — for `catch → catch`, record what Recce showed (validates redundancy or reveals Recce wasn't needed); for `miss → miss` or `partial → partial`, the agent ignored or didn't surface decisive evidence — feed back into the skill prompt. | +| **Regression** | `catch → partial`, `catch → miss`, `partial → miss` | Recce misled the agent. Rare but important — investigate in Notes; this is a v1-release-blocker signal. | + +The baseline file is **frozen at commit**: once it lands on the branch, the with-Recce run can begin, and the baseline must not be edited even if later evidence suggests it should be revised. Without this control, results conflate model variance with Recce signal. Baseline format → see `templates/tier-0-baseline.md`. + +**Same-model contract.** The with-Recce run MUST use the same agent + model as the frozen Tier-0 baseline (the `Agent` and `Model` fields in `templates/tier-0-baseline.md`). If a model upgrade lands mid-eval, either re-capture the baseline (and re-commit it) or record the mismatch in the per-fixture artifact's Notes section and treat that fixture's delta as confounded. Without this constraint, the lens-3 delta conflates Recce signal with model drift. + +## Tier-0 agent runtime contract + +To make Tier-0 baselines reproducible across runs, the agent receives **the same raw material Recce ingests, minus Recce's structured surfacing**. This isolates "what Recce contributes" from "what the agent could have figured out from artifacts alone." A weaker Tier-0 setup (e.g., diff-only, no artifacts) would understate the agent's solo capability and overstate Recce's signal. + +**Inputs per fixture** (populated by `build_fixtures.sh`): + +- `fixtures//diff.patch` — source-model diff between base and head +- `fixtures//artifacts/manifest-before.json`, `manifest-after.json` — dbt manifests pre/post +- `fixtures//artifacts/compiled-before/`, `compiled-after/` — compiled SQL trees pre/post +- `fixtures//artifacts/catalog-before.json`, `catalog-after.json` — schema-only (row/col stats are zero in this fixture set; documented in the fixtures README) +- Read access to the dbt project source at the head SHA — materialized **per fixture** by `build_fixtures.sh` at `evals/agent-blind-spots/.tmp/sources//`. The shared clone at `.tmp/jaffle_shop_golden/` is build-script scratch; do NOT read from it because the build loop leaves it at the last fixture's SHA. + +**Generic tools allowed at Tier 0:** file read, grep / ripgrep, `jq`, `git log` / `git diff` / `git show` against the per-fixture source tree at `.tmp/sources//`. Anything the agent could plausibly run on a developer's laptop without Recce installed *and without regenerating any frozen Tier-0 input*. + +**Explicitly NOT allowed at Tier 0:** Recce CLI, Recce MCP, any `/recce-*` skill (including `/recce-verify`), warehouse access, `dbt run`, `dbt test`, `dbt parse`, `dbt compile`, `dbt docs generate`, live SQL execution, or any other command that regenerates the manifest/compiled/catalog artifacts. The artifacts under `fixtures//artifacts/` are the **frozen Tier-0 inputs**; regenerating them lets the agent reach beyond the captured snapshot (e.g., picking up later upstream-package changes) and breaks reproducibility across runs. Also not allowed: comparison against a base/prod environment beyond what is already in the artifacts above. + +**Prompt shape** — eval runners write the actual prompt and **MUST record it verbatim** in the Tier-0 baseline's "Prompt given to agent" section, including any agent-specific framing. To keep baselines comparable across runs: + +- The prompt MUST describe the inputs above without paraphrasing what each contains (re-describing the inputs primes the agent in ways that vary between runners). +- The prompt MUST ask for catch / miss / partial and approve / request-changes / abstain in those terms. +- The prompt MUST NOT add steering language toward humility ("be cautious about flagging issues", "only flag when you're confident") OR aggression ("find as many issues as possible", "be thorough"). Use neutral framing; "review this PR" is enough. +- Agent-specific scaffolding (file-access mode, tool whitelisting, system prompt) is allowed but must be recorded in the baseline's Notes section so the delta is interpretable. + +A reference shape that satisfies the constraints: + +> "Review this dbt PR. The inputs listed in the Tier-0 runtime contract are available. Decide catch / miss / partial per the rubric, recommend approve / request-changes / abstain, and write verdict + verbatim reasoning into `tier-0-baseline.md`." + +## Per-fixture artifact + +Each fixture's scoring lives in `runs//-scoring.md` with this structure: + +```markdown +# Fixture — + +- Tier-0 baseline (frozen): /tier-0-baseline.md> +- Run date: +- Agent / model: +- Primary evidence tier + subset: <0 | 1a | 1b | 1c | 2> (list secondary citations under Notes if any) +- Binary catch (this run): +- Binary catch (Tier-0 baseline): +- Delta: + +## Evidence Recce surfaced + +Verbatim from tool output (or paraphrased with a link). One bullet per piece of evidence. + +## Conclusion the agent reached + +What the agent actually said / recommended. Quote. + +## Delta vs Tier-0 baseline + +What changed between Tier-0 baseline and this run. Why. If no delta, say so explicitly. + +## Notes + +Failure modes, hallucinations, suspicious reasoning, anything worth feeding back into the skill prompt. +``` + +## Gap report (output of the eval run) + +The eval output is **prioritized for action**, **targets ≤5 entries**, and lives at `runs//gap-report.md`. Format → see `templates/gap-report.md`. + +The broader capability backlog (e.g., the 33-item Notion list) is reference material; the gap report is the deliberate cut of "what to do next" derived from the six case studies. The ≤5 target is a discipline against the backlog bleeding back in — if a receipts-style review leaves ≥6 genuinely independent blockers, exceed the target and add a one-line note explaining why one couldn't be subsumed or deferred. "Prioritized" here means *priority of action*, not ordinal performance. + +## What this rubric is NOT + +- **Not a leaderboard.** Don't compute "agent A vs agent B" scores. +- **Not a regression suite.** Don't run it on every commit; re-run when the skill prompt or backend changes meaningfully. +- **Not statistical evidence.** N=6 means stories, not averages. If a stakeholder asks for "the number", refer them to this section. diff --git a/evals/agent-blind-spots/build_fixtures.sh b/evals/agent-blind-spots/build_fixtures.sh new file mode 100755 index 0000000..e44be19 --- /dev/null +++ b/evals/agent-blind-spots/build_fixtures.sh @@ -0,0 +1,294 @@ +#!/usr/bin/env bash +# +# build_fixtures.sh — Rebuild the local-dbt-only artifacts for every fixture +# under fixtures//. Idempotent: existing artifacts/ are removed first. +# +# Required: git, uv. Everything else (Python 3.11.11, dbt-core, dbt-duckdb, +# duckdb) is installed into ./.tmp/.venv/ from the pinned versions below. +# +# Output: one "OK " line per fixture. Non-zero exit on any failure. +# +# See fixtures/README.md for the canonical description of the per-fixture +# layout and the gitignored artifact paths. + +set -euo pipefail + +# ---- Pins (DRC-3402) -------------------------------------------------------- +PYTHON_VERSION="3.11.11" +DBT_CORE_VERSION="1.11.9" +DBT_DUCKDB_VERSION="1.10.1" +DUCKDB_VERSION="1.5.2" + +# ---- Paths ------------------------------------------------------------------ +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FIXTURES_DIR="${SCRIPT_DIR}/fixtures" +TMP_DIR="${SCRIPT_DIR}/.tmp" +JSG_DIR="${TMP_DIR}/jaffle_shop_golden" +SOURCES_DIR="${TMP_DIR}/sources" +VENV_DIR="${TMP_DIR}/.venv" +PROFILES_DIR="${TMP_DIR}/profiles" + +JSG_REPO="DataRecce/jaffle_shop_golden" +JSG_URL="https://github.com/${JSG_REPO}.git" + +mkdir -p "${TMP_DIR}" "${PROFILES_DIR}" "${SOURCES_DIR}" + +# ---- Step 1: clone / fetch the source repo --------------------------------- +if [[ ! -d "${JSG_DIR}/.git" ]]; then + # Pre-flight auth: jaffle_shop_golden is private. Confirm the runner has + # access before the bare "Repository not found" git clone error. + if ! git ls-remote "${JSG_URL}" >/dev/null 2>&1; then + echo "Cannot reach ${JSG_REPO} (the repo is private)." >&2 + echo "Run 'gh auth setup-git' once so git uses your gh token, or configure another credential helper." >&2 + exit 1 + fi + echo "Cloning ${JSG_REPO} into ${JSG_DIR}..." >&2 + git clone "${JSG_URL}" "${JSG_DIR}" +fi + +# Always fetch PR heads so all fixture SHAs are reachable +echo "Fetching all branches + PR heads from ${JSG_REPO}..." >&2 +git -C "${JSG_DIR}" fetch --quiet origin +git -C "${JSG_DIR}" fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pr/*' + +# Allow per-fixture clones below to fetch arbitrary SHAs from this local +# cache (needed for `git fetch `). Idempotent. +git -C "${JSG_DIR}" config uploadpack.allowAnySHA1InWant true + +# ---- Step 2: venv with pinned versions ------------------------------------- +if [[ ! -d "${VENV_DIR}" ]]; then + echo "Creating venv with Python ${PYTHON_VERSION}..." >&2 + uv venv --python "${PYTHON_VERSION}" "${VENV_DIR}" +fi + +# uv pip install is idempotent; cheap to run every time. +echo "Installing pinned dbt-duckdb stack into ${VENV_DIR}..." >&2 +VIRTUAL_ENV="${VENV_DIR}" uv pip install --quiet \ + "dbt-core==${DBT_CORE_VERSION}" \ + "dbt-duckdb==${DBT_DUCKDB_VERSION}" \ + "duckdb==${DUCKDB_VERSION}" + +DBT="${VENV_DIR}/bin/dbt" + +# ---- Step 3: DuckDB profile ------------------------------------------------ +# DuckDB path picks a deterministic file under TMP_DIR so the compiled SQL +# references "jaffle_shop_fixture_build" as the database, matching the original +# build. The actual file is gitignored (lives under .tmp/) and gets reused. +cat > "${PROFILES_DIR}/profiles.yml" < "redacted" +# invocation_id -> "redacted" +# root_path -> "" +scrub_json() { + local path="$1" + VIRTUAL_ENV="${VENV_DIR}" "${VENV_DIR}/bin/python" - "${path}" <<'PY' +import json +import sys + +path = sys.argv[1] +with open(path, "r") as f: + data = json.load(f) + +def walk(node): + if isinstance(node, dict): + for k, v in list(node.items()): + if k == "user_id" and isinstance(v, str): + node[k] = "redacted" + elif k == "invocation_id" and isinstance(v, str): + node[k] = "redacted" + elif k == "root_path" and isinstance(v, str): + node[k] = "" + else: + walk(v) + elif isinstance(node, list): + for item in node: + walk(item) + +walk(data) +with open(path, "w") as f: + json.dump(data, f, indent=2, sort_keys=True) +PY +} + +# Compile a single SHA into a temporary target directory and copy the artifacts +# of interest into /. Layout depends on caller (before/after/intermediate). +# +# Args: +# sha commit to check out in jaffle_shop_golden +# manifest_out destination path for manifest.json (file) +# compiled_out destination dir for compiled/ (directory) +# catalog_out destination path for catalog.json (file) +build_at_sha() { + local sha="$1" + local manifest_out="$2" + local compiled_out="$3" + local catalog_out="$4" + + # Detached checkout; abort any local edits from prior runs. + git -C "${JSG_DIR}" reset --quiet --hard + git -C "${JSG_DIR}" clean --quiet -fdx + git -C "${JSG_DIR}" checkout --quiet --detach "${sha}" + + # Swap upstream Snowflake profile for our DuckDB one (kept only inside JSG_DIR). + cp "${PROFILES_DIR}/profiles.yml" "${JSG_DIR}/profiles.yml" + + # Stage seed CSVs so dbt parse resolves seed nodes (dbt doesn't *need* the data + # for parse/compile/docs-generate, but the project expects seeds/ to exist). + mkdir -p "${JSG_DIR}/seeds" + cp "${JSG_DIR}/jaffle-shop-data/"*.csv "${JSG_DIR}/seeds/" + + # dbt deps fetches packages.yml entries into dbt_packages/. + (cd "${JSG_DIR}" && DBT_PROFILES_DIR="${JSG_DIR}" "${DBT}" deps --quiet) + + # Empty target dir so we never silently mix old artifacts. + rm -rf "${JSG_DIR}/target" + + (cd "${JSG_DIR}" && DBT_PROFILES_DIR="${JSG_DIR}" "${DBT}" parse --quiet) + (cd "${JSG_DIR}" && DBT_PROFILES_DIR="${JSG_DIR}" "${DBT}" compile --quiet) + (cd "${JSG_DIR}" && DBT_PROFILES_DIR="${JSG_DIR}" "${DBT}" docs generate --empty-catalog --quiet) + + # Copy outputs to the fixture artifacts/. + mkdir -p "$(dirname "${manifest_out}")" + cp "${JSG_DIR}/target/manifest.json" "${manifest_out}" + scrub_json "${manifest_out}" + + rm -rf "${compiled_out}" + mkdir -p "${compiled_out}" + if [[ -d "${JSG_DIR}/target/compiled" ]]; then + cp -R "${JSG_DIR}/target/compiled/." "${compiled_out}/" + fi + + cp "${JSG_DIR}/target/catalog.json" "${catalog_out}" + scrub_json "${catalog_out}" +} + +# ---- Step 4: build each fixture -------------------------------------------- +build_fixture() { + local slug="$1" + local fdir="${FIXTURES_DIR}/${slug}" + local readme="${fdir}/README.md" + local commits_file="${fdir}/commits.txt" + local artifacts="${fdir}/artifacts" + + if [[ ! -f "${readme}" ]] || [[ ! -f "${commits_file}" ]]; then + echo "FAIL ${slug} (missing README.md or commits.txt)" >&2 + return 1 + fi + + local base_sha + base_sha="$(extract_base_sha "${readme}")" + if [[ -z "${base_sha}" ]]; then + echo "FAIL ${slug} (no Base SHA in README.md)" >&2 + return 1 + fi + + # Head SHA = first non-blank line of commits.txt (already in repo order: newest first). + local head_sha + head_sha="$(awk 'NF>0 {print $1; exit}' "${commits_file}")" + if [[ -z "${head_sha}" ]]; then + echo "FAIL ${slug} (no head SHA in commits.txt)" >&2 + return 1 + fi + + rm -rf "${artifacts}" + mkdir -p "${artifacts}" + + build_at_sha "${base_sha}" \ + "${artifacts}/manifest-before.json" \ + "${artifacts}/compiled-before" \ + "${artifacts}/catalog-before.json" + + build_at_sha "${head_sha}" \ + "${artifacts}/manifest-after.json" \ + "${artifacts}/compiled-after" \ + "${artifacts}/catalog-after.json" + + # Materialize a per-fixture standalone repo at the head SHA so eval + # runners can read the head-SHA source for THIS fixture in isolation. + # + # Tier-0 frozen-input contract (RUBRIC.md): the agent must not see + # later commits on the same branch, other fixtures' SHAs, or any + # other ref from the upstream repo. A `git worktree add` off + # ${JSG_DIR} would share that cache's object database — `git log + # --all`, `git rev-parse origin/pr/`, `git show ` + # would all succeed inside the fixture. Instead, we initialise a + # fresh repo and fetch only the head SHA from the local cache, so + # the fixture's `.git` ends up structurally minimal: one commit, no + # remotes, no other refs reachable. + local source_dir="${SOURCES_DIR}/${slug}" + # `fetch ` needs the full 40-char SHA; the + # commits.txt entries are abbreviated. Resolve via the cache first. + local full_head + full_head="$(git -C "${JSG_DIR}" rev-parse "${head_sha}^{commit}")" + rm -rf "${source_dir}" + mkdir -p "${source_dir}" + git -C "${source_dir}" init --quiet + # Single-commit fetch from the local cache, landed into a named + # local ref (refs/fixture/head). No `git remote add` — leaving the + # repo with zero configured remotes means a stray `git fetch` + # inside the fixture cannot pull additional history. + git -C "${source_dir}" fetch --quiet --depth 1 --no-tags \ + "${JSG_DIR}" "+${full_head}:refs/fixture/head" + git -C "${source_dir}" checkout --quiet --detach refs/fixture/head + + # Tier-0 leak check: the fixture's reachable history (across all + # refs + HEAD) must contain only its own head commit. Anything + # larger means a ref or pack from the cache leaked through. + local reachable + reachable="$(git -C "${source_dir}" rev-list --all HEAD --count)" + if [[ "${reachable}" != "1" ]]; then + echo "FAIL ${slug} (Tier-0 leak: rev-list --all HEAD --count = ${reachable}, expected 1)" >&2 + return 1 + fi + + # PR #20 intermediate snapshot — keyed on the well-known SHA in commits.txt. + if [[ "${slug}" == "pr44-promotion-flags" ]]; then + local intermediate_sha="23b96ca" + # Resolve to a full SHA from commits.txt to avoid ambiguity. + local full_intermediate + full_intermediate="$(awk -v p="${intermediate_sha}" '$1 ~ "^"p { print $1; exit }' "${commits_file}")" + if [[ -z "${full_intermediate}" ]]; then + echo "FAIL ${slug} (commits.txt does not list intermediate ${intermediate_sha})" >&2 + return 1 + fi + local idir="${artifacts}/intermediate-commit-${intermediate_sha}" + mkdir -p "${idir}/compiled" + build_at_sha "${full_intermediate}" \ + "${idir}/manifest.json" \ + "${idir}/compiled" \ + "${idir}/catalog.json" + fi + + echo "OK ${slug}" +} + +cd "${SCRIPT_DIR}" + +FIXTURES=( + pr1-fix-clv + pr2-refactor-cte-to-models + pr3-amount-double-to-decimal + pr42-is-closed-filter + pr44-promotion-flags + pr46-net-clv-segments +) + +for slug in "${FIXTURES[@]}"; do + build_fixture "${slug}" +done diff --git a/evals/agent-blind-spots/fixtures/README.md b/evals/agent-blind-spots/fixtures/README.md new file mode 100644 index 0000000..701be36 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/README.md @@ -0,0 +1,79 @@ +# Fixtures — Agent Blind Spots / `/recce-verify` v1 + +Six PR fixtures from [`DataRecce/jaffle_shop_golden`](https://github.com/DataRecce/jaffle_shop_golden) covering distinct verification classes (semantic, row-grain, refactor, type, schema-expansion, multi-model). One directory per fixture, each holding a README, a frozen Tier-0 baseline template, the base/head commit SHAs (`commits.txt`), and a small source-models `diff.patch`. + +## Artifacts are not committed — build them locally + +The large dbt artifacts (`manifest-before.json`, `manifest-after.json`, `catalog-*.json`, `compiled-before/`, `compiled-after/`) are **gitignored** and produced by the build script. Run it once before each eval run: + +```bash +cd evals/agent-blind-spots +./build_fixtures.sh +``` + +The script clones `DataRecce/jaffle_shop_golden` into `.tmp/jaffle_shop_golden/` (also gitignored), swaps the upstream Snowflake `profiles.yml` for a local DuckDB profile, then for each fixture checks out the base + head (+ intermediate for PR #20), runs `dbt deps && dbt parse && dbt compile && dbt docs generate` against an empty DuckDB, and writes the outputs into `fixtures//artifacts/`. Host-specific manifest fields (`user_id`, `invocation_id`, `root_path`) are scrubbed. + +Re-running is idempotent — existing `artifacts/` directories are removed and rebuilt. + +## Index + +| Fixture | Source PR | Class | Notes | +|---------|-----------|-------|-------| +| [`pr1-fix-clv`](./pr1-fix-clv/) | [#13](https://github.com/DataRecce/jaffle_shop_golden/pull/13) | semantic | Adds `where status='completed'` inside `customers.customer_payments` CTE. | +| [`pr42-is-closed-filter`](./pr42-is-closed-filter/) | [#14](https://github.com/DataRecce/jaffle_shop_golden/pull/14) | row-grain | New `is_closed` column + `where is_closed=true` on `orders`. Different (older) base SHA than the rest. | +| [`pr2-refactor-cte-to-models`](./pr2-refactor-cte-to-models/) | [#15](https://github.com/DataRecce/jaffle_shop_golden/pull/15) | refactor | Behavior-preserving; the negative control for the rubric. | +| [`pr3-amount-double-to-decimal`](./pr3-amount-double-to-decimal/) | [#16](https://github.com/DataRecce/jaffle_shop_golden/pull/16) | type | `amount` narrowed to `DECIMAL(10,2)`. PR has a mechanical merge commit on top of the substantive change at `6ffc23f`; fixture captures the merge head. | +| [`pr44-promotion-flags`](./pr44-promotion-flags/) | [#20](https://github.com/DataRecce/jaffle_shop_golden/pull/20) | schema-expansion | Plus an *intermediate-commit* artifact snapshot for the row-filter accident at `23b96ca` (reverted by `1500eb4`). Uses Snowflake-specific `boolor_agg` — DuckDB does not validate at compile but would fail at execute. | +| [`pr46-net-clv-segments`](./pr46-net-clv-segments/) | [#2](https://github.com/DataRecce/jaffle_shop_golden/pull/2) | multi-model semantic | "Stress-test" fixture — redefines `customer_lifetime_value` in place, introduces three row filters on payments, copy-pastes a threshold for `net_value_segment`, and adds a `finance_revenue` model with no downstream consumers. | + +## Per-fixture layout + +``` +fixtures// +├── README.md ← what the PR does, expected verdicts without/with Recce, caveats +├── tier-0-baseline.md ← template instance, fields filled in by the eval runner +├── commits.txt ← base + head SHAs (and intermediate for PR #20) +├── diff.patch ← source-model diff base..head (small, reading-friendly) +└── artifacts/ ← gitignored — produced by build_fixtures.sh + ├── manifest-before.json ← `target/manifest.json` from `dbt parse` on base SHA + ├── manifest-after.json ← same on head SHA + ├── compiled-before/ ← `target/compiled/` from `dbt compile` on base SHA + ├── compiled-after/ ← same on head SHA + ├── catalog-before.json ← `target/catalog.json` from `dbt docs generate` on base SHA (empty data) + └── catalog-after.json ← same on head SHA +``` + +`pr44-promotion-flags/` additionally has a top-level `diff-from-base-to-intermediate.patch` (committed) and an `artifacts/intermediate-commit-23b96ca/` directory (gitignored) with `manifest.json`, `compiled/`, and `catalog.json` for the problematic intermediate commit. + +## Caveats — rolled up + +- **PR #16 (`pr3-amount-double-to-decimal`)** — head SHA `1c56861` is a mechanical merge commit; the substantive type narrowing lives at `6ffc23f`. The fixture captures the merge head; the source-models diff is identical. +- **PR #20 (`pr44-promotion-flags`)** — four-commit PR. The row-filter accident lives at the intermediate commit `23b96ca` and was reverted at `1500eb4`. Snowflake-specific `boolor_agg` appears in source but compiles fine under DuckDB (no compile-time function validation); it would fail at execute on DuckDB. +- **PR #2 (`pr46-net-clv-segments`)** — the "stress test" of the set. Redefines `customer_lifetime_value` in place, introduces three row filters on `payments`, copy-pastes a magic threshold for `net_value_segment`, and adds a `finance_revenue` model with no downstream consumers. Largest diff in the set. +- **Catalog row/column stats are zero everywhere** — `dbt docs generate` runs against an empty DuckDB, so `catalog-*.json` carries schema info (column names, types) but **no row counts and no column stats**. Do not score rubric items off catalog row stats. +- **PR #14 (`pr42-is-closed-filter`)** uses an older base SHA (`62d6dc9`) than the rest (`f09861a`). Don't mix bases when computing inter-fixture deltas. + +## dbt environment + +The source repo `jaffle_shop_golden` ships a **Snowflake-only** `profiles.yml`. Compile and docs-generate against Snowflake require warehouse credentials, which the eval baseline explicitly does not have. The fixture build pipeline therefore swaps in a local DuckDB profile *for parse/compile/docs-generate only*. The model SQL is portable between the two adapters with one exception noted above (`boolor_agg` in PR #20). + +Pinned versions for reproducibility: + +| Component | Version | +|-----------|---------| +| Python | 3.11.11 | +| dbt-core | 1.11.9 | +| dbt-duckdb | 1.10.1 | +| duckdb | 1.5.2 | +| dbt packages | `data-mie/dbt_profiler@0.8.1`, `dbt-labs/dbt_utils@0.9.6`, `dbt-labs/audit_helper@0.11.0` (per upstream `packages.yml`) | + +## Reproducing or extending + +- Add a fixture: create `fixtures//{README.md,tier-0-baseline.md,commits.txt,diff.patch}` and re-run `build_fixtures.sh`. The script reads `commits.txt` to discover the SHAs to build against. +- Required system tools: `uv`, `git`. Everything else is installed into `evals/agent-blind-spots/.tmp/.venv/` from pinned versions in the script. +- **GitHub access**: `DataRecce/jaffle_shop_golden` is a **private** repo. The first run clones it over HTTPS, which requires either (a) a credential helper with access to the repo, or (b) running `gh auth setup-git` once so `git clone https://github.com/...` uses your `gh` token. Subsequent runs only `git fetch`, so the credential only matters on first clone. If you hit a "Repository not found / authentication failed" error, that's the cause. +- Eval-baseline assets are **warehouse-free** — the local-dbt-only artifacts (manifest, compiled SQL, catalog, git diff) plus the per-fixture head-SHA source tree at `.tmp/sources//` are the canonical Tier-0 inputs. + +## `commits.txt` format + +One SHA per line, with an optional message after the first whitespace. The first non-comment line is treated as the head; the build script also reads the per-fixture `README.md` to discover the base SHA (look for ``- Base SHA: `` ``). For PR #20, the intermediate commit is the line with SHA `23b96ca`. diff --git a/evals/agent-blind-spots/fixtures/pr1-fix-clv/README.md b/evals/agent-blind-spots/fixtures/pr1-fix-clv/README.md new file mode 100644 index 0000000..90f5e71 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr1-fix-clv/README.md @@ -0,0 +1,42 @@ +# Fixture `pr1-fix-clv` — Fix CLV to completed orders only + +- Source PR: [DataRecce/jaffle_shop_golden#13](https://github.com/DataRecce/jaffle_shop_golden/pull/13) +- Base SHA: `f09861a39b314907719260f19f7d6ef2fd347ab0` +- Head SHA: `2abf877ec9d067f2169d1b690aefdd54a2b3d205` +- Verification class: **semantic** +- Models touched: `customers` (1 file, 1 line added) + +## What the PR does + +Adds `where orders.status = 'completed'` to the `customer_payments` CTE in `models/customers.sql`. The author frames it as a bugfix: previously, `customer_lifetime_value` summed payment amounts from *all* orders (including `placed`, `shipped`, `return_pending`, `returned`). With the fix it only counts payments tied to completed orders. + +## Why this is a "semantic" case + +The diff is one line, syntactically valid, and the schema (column names, types) is unchanged. An agent reading only manifest + git diff sees a filter being added inside a CTE. The intent — "should non-completed orders' payments count toward lifetime value?" — is a business semantic question, not a SQL correctness question. Whether this is a **fix** or a **regression** depends on the business definition of CLV, and on whether downstream consumers depended on the previous (looser) value. + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. + +- Likely catch quality: **partial**. +- The agent will spot the new `where` clause from the diff and correctly identify it narrows the set of payments aggregated. It can describe the semantic change in prose. +- But without measuring rows or values it cannot say: + - How many customers see their CLV change. + - How big the per-customer value delta is. + - Whether any customer's CLV drops to NULL because they have no completed orders. +- The agent is therefore likely to hedge ("this could be a bugfix or a behavior change depending on intent") rather than commit to a verdict. + +## Expected agent verdict — with Recce + +- Likely catch quality: **catch**. +- Evidence Recce should surface: + - **Row-count diff on `customers`** — unchanged (every customer still appears; left joins preserve rows). + - **Value diff on `customer_lifetime_value`** — non-trivial mismatch percentage; some customers' CLV drops, none rises. + - **Query diff** on average CLV per first-order week (the preset check in `recce.yml`) — values shift downward. +- Expected conclusion: "The change reduces CLV for customers whose orders include non-completed statuses. This is intentional per the PR title but is a behavior change; downstream consumers of `customer_lifetime_value` should be notified." + +## Caveats + +- The PR's compiled `.sql` for `customers.sql` is the only file that changes; nothing in the manifest schema changes. +- `recce.yml` already defines `value_diff` and `query_diff` preset checks on `customer_lifetime_value`, which makes this fixture an unusually friendly target for Recce. Other fixtures will not have this advantage. +- Reproducible without warehouse: yes. `compiled-before`/`compiled-after` SQL diff captures the substantive change. diff --git a/evals/agent-blind-spots/fixtures/pr1-fix-clv/commits.txt b/evals/agent-blind-spots/fixtures/pr1-fix-clv/commits.txt new file mode 100644 index 0000000..14ab5a0 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr1-fix-clv/commits.txt @@ -0,0 +1 @@ +2abf877 PR1 diff --git a/evals/agent-blind-spots/fixtures/pr1-fix-clv/diff.patch b/evals/agent-blind-spots/fixtures/pr1-fix-clv/diff.patch new file mode 100644 index 0000000..07bd7b2 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr1-fix-clv/diff.patch @@ -0,0 +1,12 @@ +diff --git a/models/customers.sql b/models/customers.sql +index 9aedd70..61a8302 100644 +--- a/models/customers.sql ++++ b/models/customers.sql +@@ -41,6 +41,7 @@ customer_payments as ( + left join orders on + payments.order_id = orders.order_id + ++ where orders.status = 'completed' + group by orders.customer_id + + ), diff --git a/evals/agent-blind-spots/fixtures/pr1-fix-clv/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr1-fix-clv/tier-0-baseline.md new file mode 100644 index 0000000..254a5c7 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr1-fix-clv/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr1-fix-clv` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR1 — Fix CLV to completed orders only` +- Verification class: `semantic` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr1-fix-clv/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/README.md b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/README.md new file mode 100644 index 0000000..8b4bc3f --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/README.md @@ -0,0 +1,41 @@ +# Fixture `pr2-refactor-cte-to-models` — Refactor CTEs into intermediate models + +- Source PR: [DataRecce/jaffle_shop_golden#15](https://github.com/DataRecce/jaffle_shop_golden/pull/15) +- Base SHA: `f09861a39b314907719260f19f7d6ef2fd347ab0` +- Head SHA: `9c386b453ba7f5317784dc5c6ec03e48af0d4903` +- Verification class: **refactor (behavior-preserving, equality expected)** +- Models touched: `customers` (rewritten), `int_customer_orders` (new), `int_customer_payments` (new) + +## What the PR does + +Refactors `customers.sql` by extracting the two inline CTEs (`customer_orders` and `customer_payments`) into two new standalone intermediate models (`int_customer_orders.sql`, `int_customer_payments.sql`). The body of `customers.sql` is reduced from a ~70-line `with ... select` chain to a single 16-line `select ... left join ref(int_*) ...`. + +No filters added, no columns added or removed, no aggregation logic changed. The DAG gets two new nodes between the staging layer and `customers`. + +## Why this is a "refactor / equality-expected" case + +This fixture is the **negative control** for the eval. A Recce-aware agent should *approve* this PR with high confidence and cite evidence that the refactor preserves behavior. A Recce-aware agent that flags this PR as risky has either misread the lineage change or is generating false alarms. + +The verification question: can the agent (with Recce) confidently conclude "behavior preserved"? + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. + +- Likely catch quality: **catch** (the easiest verdict in the fixture set, but for the wrong reason — usually pattern-matching on "refactor" in the commit message rather than verifying equivalence). +- The agent will recognize the structural rewrite from the diff and the manifest's new node count and assume behavior preservation based on shape alone. It cannot prove equivalence without comparing values. + +## Expected agent verdict — with Recce + +- Likely catch quality: **catch**. +- Evidence Recce should surface: + - **Lineage diff** — two new nodes appear (`int_customer_orders`, `int_customer_payments`); `customers` now depends on them rather than directly on staging. + - **Schema diff on `customers`** — unchanged columns. + - **Row-count diff on `customers`** — unchanged. + - **Value diff on `customer_lifetime_value`** — 100% match (the preset check in `recce.yml`). +- Expected conclusion: "Behavior-preserving refactor. Two intermediate models added; downstream `customers` is bit-for-bit identical. Approve." + +## Caveats + +- This fixture exercises the rubric's `catch → catch` "same" delta bucket, where the case-study Notes should record *what Recce showed that the Tier-0 verdict couldn't* — namely value-level equivalence. If both runs say "catch" but only the with-Recce run produces a 100% match receipt, that's the differentiating signal even though the binary catch didn't move. +- Reproducible without warehouse: yes. diff --git a/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/commits.txt b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/commits.txt new file mode 100644 index 0000000..fa2c7ba --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/commits.txt @@ -0,0 +1 @@ +9c386b4 PR2 diff --git a/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/diff.patch b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/diff.patch new file mode 100644 index 0000000..ddfe888 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/diff.patch @@ -0,0 +1,121 @@ +diff --git a/models/customers.sql b/models/customers.sql +index 9aedd70..67dd22a 100644 +--- a/models/customers.sql ++++ b/models/customers.sql +@@ -1,69 +1,16 @@ +-with customers as ( +- +- select * from {{ ref('stg_customers') }} +- +-), +- +-orders as ( +- +- select * from {{ ref('stg_orders') }} +- +-), +- +-payments as ( +- +- select * from {{ ref('stg_payments') }} +- +-), +- +-customer_orders as ( +- +- select +- customer_id, +- +- min(order_date) as first_order, +- max(order_date) as most_recent_order, +- count(order_id) as number_of_orders +- from orders +- +- group by customer_id +- +-), +- +-customer_payments as ( +- +- select +- orders.customer_id, +- sum(amount)::bigint as total_amount +- +- from payments +- +- left join orders on +- payments.order_id = orders.order_id +- +- group by orders.customer_id +- +-), +- +-final as ( +- +- select +- customers.customer_id, +- customers.first_name, +- customers.last_name, +- customer_orders.first_order, +- customer_orders.most_recent_order, +- customer_orders.number_of_orders, +- customer_payments.total_amount as customer_lifetime_value +- +- from customers +- +- left join customer_orders +- on customers.customer_id = customer_orders.customer_id +- +- left join customer_payments +- on customers.customer_id = customer_payments.customer_id +- +-) +- +-select * from final ++select ++ customers.customer_id, ++ customers.first_name, ++ customers.last_name, ++ customer_orders.first_order, ++ customer_orders.most_recent_order, ++ customer_orders.number_of_orders, ++ customer_payments.total_amount as customer_lifetime_value ++ ++from {{ ref('stg_customers') }} customers ++ ++left join {{ ref('int_customer_orders') }} customer_orders ++ on customers.customer_id = customer_orders.customer_id ++ ++left join {{ ref('int_customer_payments') }} customer_payments ++ on customers.customer_id = customer_payments.customer_id +diff --git a/models/int_customer_orders.sql b/models/int_customer_orders.sql +new file mode 100644 +index 0000000..eec8bc8 +--- /dev/null ++++ b/models/int_customer_orders.sql +@@ -0,0 +1,9 @@ ++select ++ customer_id, ++ ++ min(order_date) as first_order, ++ max(order_date) as most_recent_order, ++ count(order_id) as number_of_orders ++from {{ ref('stg_orders') }} ++ ++group by customer_id +diff --git a/models/int_customer_payments.sql b/models/int_customer_payments.sql +new file mode 100644 +index 0000000..b36ac82 +--- /dev/null ++++ b/models/int_customer_payments.sql +@@ -0,0 +1,10 @@ ++select ++ orders.customer_id, ++ sum(amount)::bigint as total_amount ++ ++from {{ ref('stg_payments') }} payments ++ ++left join {{ ref('stg_orders') }} orders on ++ payments.order_id = orders.order_id ++ ++group by orders.customer_id diff --git a/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/tier-0-baseline.md new file mode 100644 index 0000000..359a2fb --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr2-refactor-cte-to-models` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR2 — Refactor CTEs into intermediate models` +- Verification class: `refactor` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr2-refactor-cte-to-models/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/README.md b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/README.md new file mode 100644 index 0000000..d1e7097 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/README.md @@ -0,0 +1,61 @@ +# Fixture `pr3-amount-double-to-decimal` — Change payment amount from double to decimal + +- Source PR: [DataRecce/jaffle_shop_golden#16](https://github.com/DataRecce/jaffle_shop_golden/pull/16) +- Base SHA: `f09861a39b314907719260f19f7d6ef2fd347ab0` +- Head SHA: `1c56861bf11eb1449eb6d357596d8ff015678c5b` (includes a `main` merge commit on top of `6ffc23f` — see drift note below) +- Verification class: **type / rounding drift** +- Models touched: `stg_payments` (1 file, 1 line) + +## What the PR does + +In `models/staging/stg_payments.sql`, replaces: + +```sql +amount / 100 as amount +``` + +with: + +```sql +(amount / 100)::DECIMAL(10,2) amount +``` + +The previous expression yields a floating-point type (Snowflake `NUMBER(38,4)` on integer division, but for the original raw cents column behaves like double). The new expression coerces to `DECIMAL(10,2)`. + +## Why this is a "type / rounding" case + +Three distinct sub-issues, none catastrophic individually: + +1. **Precision narrowing.** `DECIMAL(10,2)` only fits values up to `99,999,999.99`. Any payment whose dollar value exceeds that overflows and the result is implementation-dependent (Snowflake errors; DuckDB may also error). +2. **Rounding behavior.** Casting `amount / 100` (e.g., `1234 / 100` in integer arithmetic) to `DECIMAL(10,2)` produces `12.34` instead of `12` or `12.3400000`. Whether this is the desired result depends on the upstream type of `amount`. +3. **Missing `as`.** The new statement is `(amount / 100)::DECIMAL(10,2) amount` — no `as`. Snowflake accepts an alias without `as`; this is a stylistic quirk, not a bug. + +The downstream effect propagates through `customer_payments`/`gross_amount`/`customer_lifetime_value`. CLV is cast to `::bigint` further downstream, which truncates fractional cents — so the visible delta in CLV may be small or zero, masking the upstream type change. + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. + +- Likely catch quality: **partial** or **miss**. +- A diligent agent will spot the type cast and may comment on rounding / overflow possibilities. The agent will *not* know: + - The actual maximum payment amount in the data (and therefore overflow risk). + - Whether any downstream computation changes value because of the cast. + - Whether the `::bigint` cast downstream hides the precision change entirely. +- Most agents will pattern-match on "type change" → "may affect downstream" and hedge. + +## Expected agent verdict — with Recce + +- Likely catch quality: **partial** at best on a *single dev environment* (Tier 1). +- Evidence Recce should surface: + - **Schema diff** — column type for `amount` on `stg_payments` changed to `DECIMAL(10,2)`. + - **Query diff** for `select max(amount) from stg_payments` — same value, but now bounded by precision. +- What single-env Recce **cannot** surface: + - Whether the change introduces overflow on production-scale data (no base env to compare against). + - Whether `customer_lifetime_value` shifts on any customer (requires Tier-2 base comparison; the ::bigint cast may absorb the difference anyway). +- Expected conclusion: "Type narrowed to DECIMAL(10,2). Potential overflow if any payment exceeds 99,999,999.99 dollars. Downstream impact unknown without a base comparison." + +## Caveats + +- **PR has evolved beyond the spec.** Head SHA includes a merge commit (`1c56861`) that brings `main` into the PR branch. The substantive change is at `6ffc23f`; the merge is mechanical. We use the merge head as the fixture head because it is what reviewers see when the PR is opened. +- This fixture is the clearest example in the set where **single-env Tier-1 Recce is honestly degraded** vs Tier-2 (data-diff). The case study should call this out — it's the most useful kind of signal for the gap report. +- Reproducible without warehouse: yes. The type change is visible in the source diff and the `catalog-*.json` files; without real data the empty catalogs will not show realistic precision behavior, so the eval runner should not rely on `catalog-*.json` numbers for this fixture. diff --git a/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/commits.txt b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/commits.txt new file mode 100644 index 0000000..dcb7810 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/commits.txt @@ -0,0 +1,2 @@ +1c56861 Merge branch 'main' into feature/add-rounding-effect-analysis +6ffc23f PR3 diff --git a/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/diff.patch b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/diff.patch new file mode 100644 index 0000000..0dfaa27 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/diff.patch @@ -0,0 +1,13 @@ +diff --git a/models/staging/stg_payments.sql b/models/staging/stg_payments.sql +index 28b8e8b..8900825 100644 +--- a/models/staging/stg_payments.sql ++++ b/models/staging/stg_payments.sql +@@ -12,7 +12,7 @@ renamed as ( + payment_method, + + -- `amount` is currently stored in cents, so we convert it to dollars +- amount / 100 as amount ++ (amount / 100)::DECIMAL(10,2) amount + + from source + diff --git a/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/tier-0-baseline.md new file mode 100644 index 0000000..60ec726 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr3-amount-double-to-decimal` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR3 — Change payment amount from double to decimal` +- Verification class: `type` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr3-amount-double-to-decimal/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/README.md b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/README.md new file mode 100644 index 0000000..fa82079 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/README.md @@ -0,0 +1,44 @@ +# Fixture `pr42-is-closed-filter` — Add `is_closed` and filter orders + +- Source PR: [DataRecce/jaffle_shop_golden#14](https://github.com/DataRecce/jaffle_shop_golden/pull/14) +- Base SHA: `62d6dc9367cb6a35fc56942ad437900f9c1fd8cb` +- Head SHA: `d2be60a0f338ef6bf5e1dcf143c5bc0a17a55060` +- Verification class: **row-grain** +- Models touched: `orders`, `stg_orders` (2 files) + +## What the PR does + +Two changes packaged together: + +1. `stg_orders.sql` adds a new derived column `is_closed`, computed as `status = 'completed'`. +2. `orders.sql` (a) surfaces `is_closed` in the model's output, then (b) adds `where is_closed = true` at the bottom of the final CTE. + +The net effect on `orders`: every row whose status is anything other than `completed` is dropped. The row grain of `orders` changes from "one row per order" to "one row per **completed** order." A new column also appears in the schema. + +## Why this is a "row-grain" case + +Both changes are syntactically clean. The compiled-after SQL still reads naturally. The verification question — "does the agent notice that the model now drops ~half the rows?" — depends on whether the agent reasons about set semantics from the new `where` clause, not on a SQL parse error. + +Adding the `is_closed` column to the schema is a separate concern (downstream consumers that `select *` from `orders` now have an extra column). The row drop is the more dangerous of the two. + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. + +- Likely catch quality: **partial** (best case) or **miss** (typical). +- An attentive agent reading the diff will see the new `where is_closed = true` and flag it. A less careful agent will frame the change as "adds an `is_closed` indicator," focusing on the schema addition and treating the filter as obvious / intentional. +- Without row counts the agent has no way to quantify the impact ("filters down to maybe 30% of orders? 90%?") and cannot point to a downstream consumer that breaks. + +## Expected agent verdict — with Recce + +- Likely catch quality: **catch**. +- Evidence Recce should surface: + - **Row-count diff on `orders`** — large negative delta (every non-completed order is dropped). + - **Schema diff** — new column `is_closed` on both `stg_orders` and `orders`. + - **Lineage** — any model downstream of `orders` that depends on non-completed orders now silently sees fewer rows. +- Expected conclusion: "This PR drops X% of rows from `orders` because of the new `where is_closed = true` filter. The `is_closed` column itself is fine; the filter is the breaking change. Either remove the filter or update the model's contract." + +## Caveats + +- Base SHA for this PR is `62d6dc936...`, an older `main` head before `f09861a` (`feat: add avg_order_amount to orders_daily_summary`). That earlier merge added two columns to `orders_daily_summary` that are absent in this fixture's base manifest. The other five fixtures use `f09861a` as base. +- Reproducible without warehouse: yes. The compiled SQL and the source diff make the filter visible. diff --git a/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/commits.txt b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/commits.txt new file mode 100644 index 0000000..6e69c46 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/commits.txt @@ -0,0 +1 @@ +d2be60a PR42 diff --git a/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/diff.patch b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/diff.patch new file mode 100644 index 0000000..df32d56 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/diff.patch @@ -0,0 +1,38 @@ +diff --git a/models/orders.sql b/models/orders.sql +index cbb2934..5b0beee 100644 +--- a/models/orders.sql ++++ b/models/orders.sql +@@ -36,6 +36,7 @@ final as ( + orders.customer_id, + orders.order_date, + orders.status, ++ orders.is_closed, + + {% for payment_method in payment_methods -%} + +@@ -50,7 +51,7 @@ final as ( + + left join order_payments + on orders.order_id = order_payments.order_id +- ++ where is_closed = true + ) + + select * from final +diff --git a/models/staging/stg_orders.sql b/models/staging/stg_orders.sql +index ec77ac2..7d4eb7d 100644 +--- a/models/staging/stg_orders.sql ++++ b/models/staging/stg_orders.sql +@@ -10,10 +10,9 @@ renamed as ( + id as order_id, + user_id as customer_id, + order_date, +- status +- ++ status, ++ status = 'completed' as is_closed + from source +- + ) + + select * from renamed diff --git a/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/tier-0-baseline.md new file mode 100644 index 0000000..c96d4d7 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr42-is-closed-filter/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr42-is-closed-filter` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR42 — Add is_closed and filter orders` +- Verification class: `row-grain` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr42-is-closed-filter/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/fixtures/pr44-promotion-flags/README.md b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/README.md new file mode 100644 index 0000000..f32c92b --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/README.md @@ -0,0 +1,73 @@ +# Fixture `pr44-promotion-flags` — Add promotion payment flag + customer has-promotion flag + +- Source PR: [DataRecce/jaffle_shop_golden#20](https://github.com/DataRecce/jaffle_shop_golden/pull/20) +- Base SHA: `f09861a39b314907719260f19f7d6ef2fd347ab0` +- Head SHA: `bd407ac2a40ce52cc24ee9c40393e9412706c4e3` +- Verification class: **schema-expansion + (intermediate row-filter accident)** +- Models touched: `stg_payments`, `customers` (2 files at head) + +## What the PR does (head state) + +Two schema additions at head: + +1. `stg_payments` gains an `is_promotion` column derived as `payment_method = 'coupon'`. +2. `customers` gains a `has_promoted_orders` column, computed via `boolor_agg(is_promotion)` in the `customer_payments` CTE, surfaced in the final select. + +At head, no row filter is present in `customers.sql`. Schema and lineage both expand; no rows are dropped. + +## The intermediate-commit accident (the interesting bit) + +PR #20 has four commits. The interesting case for this eval is **not** the head state — it is the state after commit `23b96ca02b` ("Add promotion information"), which: + +- Added the two schema columns as described above. +- **Also added** `where has_promoted_orders = true` at the bottom of the `customers.sql` final CTE. + +That `where` clause means: only customers with at least one coupon-paid order remain in the `customers` model. Every customer who paid by card / bank-transfer / gift-card with no coupon ever is dropped. + +Commit `1500eb444c` ("Remove where condition") reverted the filter — but only because the author re-read their own diff. An agent reviewing commit-by-commit (or any reviewer who looks only at the first commit's preview before more were pushed) would face the same row-filter trap as `pr42-is-closed-filter`, packaged inside what looks like a benign schema-expansion PR. + +Artifacts for both states are produced by `build_fixtures.sh` (not committed): + +- `artifacts/manifest-after.json` + `artifacts/compiled-after/` — head (`bd407ac`). Schema expansion only, no row drop. +- `artifacts/intermediate-commit-23b96ca/` — the problematic intermediate. Same schema additions **plus** the row filter. +- `diff-from-base-to-intermediate.patch` (committed, top-level) — the diff `base..23b96ca` so the eval can present the intermediate commit as if it were the PR head. + +## Why this is a "schema-expansion + accidental row filter" case + +The dangerous pattern is the *combination*. A new column is a low-stakes change; reviewers tend to approve it on diff alone. Wrapping a row filter inside the same commit hides the filter behind the schema noise. Recce's row-count diff is the cheap, decisive signal that catches it. + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. **Run separately against each of the two artifact snapshots; the headline finding is the contrast between them.** + +### Against head (`bd407ac`) + +- Likely catch quality: **catch** (correctly approves schema expansion). +- Reasoning: agent sees two new columns, no `where` clause introduced, approves with a note about new columns. + +### Against intermediate (`23b96ca`) + +- Likely catch quality: **miss** or **partial**. +- The agent sees a schema-expansion diff that *also* contains `where has_promoted_orders = true`. Diligent agents catch the filter; many will read the diff as "adds promotion flags" and miss the trailing two lines. +- Without row counts they cannot quantify impact. + +## Expected agent verdict — with Recce + +### Against head (`bd407ac`) + +- Likely catch quality: **catch**. +- Evidence Recce should surface: schema diff showing two new columns, row count unchanged. +- Conclusion: "Approve — schema expansion, no row impact." + +### Against intermediate (`23b96ca`) + +- Likely catch quality: **catch**. +- Evidence Recce should surface: schema diff (new columns) **and** row-count diff on `customers` showing a large drop. +- Conclusion: "Block — accidental row filter introduced in the same commit as the schema expansion. Either remove `where has_promoted_orders = true` or scope the new column to a separate downstream model." + +## Caveats + +- The PR has **four commits**, not one. Per the spec, the row-filter case lives in the middle. The eval runner has the choice to (a) treat `head` as the canonical PR (clean schema expansion, no row drop) or (b) replay commit `23b96ca` as the PR head to exercise the row-filter case. The author of the spec wanted **both**. +- Tier-2 (base comparison) gives the cleanest evidence, but the row-filter trap is also visible from single-env Recce: a row count of `customers` against the dev environment's prior state would surface the drop. Whether the agent can construct that comparison without a base env is itself a case-study question. +- `boolor_agg` is a Snowflake function. The fixture build pipeline overrides the dbt profile to DuckDB for offline compile; DuckDB accepts `bool_or` as the canonical aggregate. The compiled SQL captured in `compiled-after/` will show `boolor_agg` as written in source — note this in the agent prompt if the eval runs a DuckDB-backed Recce. +- Reproducible without warehouse: yes for compile artifacts. Schema diff and lineage diff are visible; row-count diff requires a live env, which v1 Tier-1 single-env Recce can do against any dev DB. diff --git a/evals/agent-blind-spots/fixtures/pr44-promotion-flags/commits.txt b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/commits.txt new file mode 100644 index 0000000..02ba8cf --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/commits.txt @@ -0,0 +1,4 @@ +bd407ac Fix Snowflake compatibility: use = operator and boolor_agg +274e750 Fix equality operator in stg_payments.sql for Snowflake compatibility +1500eb4 Remove where condition +23b96ca Add promotion information diff --git a/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff-from-base-to-intermediate.patch b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff-from-base-to-intermediate.patch new file mode 100644 index 0000000..52f93e4 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff-from-base-to-intermediate.patch @@ -0,0 +1,47 @@ +diff --git a/models/customers.sql b/models/customers.sql +index 9aedd70..1d4cb29 100644 +--- a/models/customers.sql ++++ b/models/customers.sql +@@ -34,7 +34,8 @@ customer_payments as ( + + select + orders.customer_id, +- sum(amount)::bigint as total_amount ++ sum(amount)::bigint as total_amount, ++ bool_or(is_promotion) as has_promoted_orders + + from payments + +@@ -54,7 +55,8 @@ final as ( + customer_orders.first_order, + customer_orders.most_recent_order, + customer_orders.number_of_orders, +- customer_payments.total_amount as customer_lifetime_value ++ customer_payments.total_amount as customer_lifetime_value, ++ customer_payments.has_promoted_orders + + from customers + +@@ -63,7 +65,7 @@ final as ( + + left join customer_payments + on customers.customer_id = customer_payments.customer_id +- ++ where has_promoted_orders = true + ) + + select * from final +diff --git a/models/staging/stg_payments.sql b/models/staging/stg_payments.sql +index 28b8e8b..8a8d4be 100644 +--- a/models/staging/stg_payments.sql ++++ b/models/staging/stg_payments.sql +@@ -12,7 +12,8 @@ renamed as ( + payment_method, + + -- `amount` is currently stored in cents, so we convert it to dollars +- amount / 100 as amount ++ amount / 100 as amount, ++ payment_method == 'coupon' as is_promotion + + from source + diff --git a/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff.patch b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff.patch new file mode 100644 index 0000000..c5f372c --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/diff.patch @@ -0,0 +1,46 @@ +diff --git a/models/customers.sql b/models/customers.sql +index 9aedd70..9b440c2 100644 +--- a/models/customers.sql ++++ b/models/customers.sql +@@ -34,7 +34,8 @@ customer_payments as ( + + select + orders.customer_id, +- sum(amount)::bigint as total_amount ++ sum(amount)::bigint as total_amount, ++ boolor_agg(is_promotion) as has_promoted_orders + + from payments + +@@ -54,7 +55,8 @@ final as ( + customer_orders.first_order, + customer_orders.most_recent_order, + customer_orders.number_of_orders, +- customer_payments.total_amount as customer_lifetime_value ++ customer_payments.total_amount as customer_lifetime_value, ++ customer_payments.has_promoted_orders + + from customers + +@@ -63,7 +65,6 @@ final as ( + + left join customer_payments + on customers.customer_id = customer_payments.customer_id +- + ) + + select * from final +diff --git a/models/staging/stg_payments.sql b/models/staging/stg_payments.sql +index 28b8e8b..9977ba8 100644 +--- a/models/staging/stg_payments.sql ++++ b/models/staging/stg_payments.sql +@@ -12,7 +12,8 @@ renamed as ( + payment_method, + + -- `amount` is currently stored in cents, so we convert it to dollars +- amount / 100 as amount ++ amount / 100 as amount, ++ payment_method = 'coupon' as is_promotion + + from source + diff --git a/evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md new file mode 100644 index 0000000..79f5626 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr44-promotion-flags` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR44 — Add promotion payment flag + customer has-promotion flag` +- Verification class: `schema-expansion + (intermediate row-filter accident)` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr44-promotion-flags/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/README.md b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/README.md new file mode 100644 index 0000000..cbd28f9 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/README.md @@ -0,0 +1,64 @@ +# Fixture `pr46-net-clv-segments` — Net revenue, net CLV, customer segments + +- Source PR: [DataRecce/jaffle_shop_golden#2](https://github.com/DataRecce/jaffle_shop_golden/pull/2) +- Base SHA: `f09861a39b314907719260f19f7d6ef2fd347ab0` +- Head SHA: `297eb54e868f7f6070cc1d2bb6a46aade7cc97b1` +- Verification class: **multi-model semantic** +- Models touched: `stg_payments`, `customers`, `customer_segments`, `finance_revenue` (new), schema YAML (2 files) + +## What the PR does + +The most behavior-rich PR in the fixture set. Multiple co-changed models, each with its own semantic concern: + +1. **`stg_payments`** gains a `coupon_amount` column: `(payment_method = 'coupon')::int * (amount / 100)`. Pure schema add — no row impact. + +2. **`customers`** is rewritten more than the diff suggests: + - The intermediate `customer_payments` CTE renames `total_amount` → `gross_amount` and adds `net_amount = sum(amount - coupon_amount)`. + - The join is qualified with `and orders.status = 'completed'` — silently introducing the same "completed-only" semantic as PR #13 but inside the join condition (not a `where`). + - Adds `where payments.amount is not null and payments.amount > 0` — two row filters on the payments side of the join, before aggregation. + - The final select **renames** the user-visible `customer_lifetime_value` to be sourced from `gross_amount` (was `total_amount`), and adds `net_customer_lifetime_value` from `net_amount`. The column name `customer_lifetime_value` is preserved at the model boundary; its definition changed underneath. + +3. **`customer_segments`** adds `net_customer_lifetime_value` and a new `net_value_segment` column with the same threshold logic as the existing `value_segment` but on the net value. Also adds several `not_null`, `accepted_values`, and `relationships` tests. + +4. **`finance_revenue`** (new model) — per-order gross and net revenue, joined with stg_orders. + +5. **`schema.yml` (root and staging)** — extensive column/test additions to match new schema. + +## Why this is a "multi-model semantic" case + +Four distinct semantic risks, all in one PR, all packaged with new columns that look like additive schema expansion: + +| Risk | Where it lives | Cheap detection | +|------|----------------|------------------| +| `customer_lifetime_value` redefined (now gross, only on completed orders) | `customers.customer_payments` CTE | value diff on `customer_lifetime_value` (preset check exists) | +| Negative / null amount payments dropped | `customers.customer_payments` CTE `where` clause | row count diff on `customers` if any negatives exist | +| New downstream `net_value_segment` thresholds copy-pasted from gross thresholds | `customer_segments` | semantic question — thresholds may not be appropriate for net | +| `finance_revenue` new model leaks into the DAG with no row-count check | new file | new node in lineage diff | + +Several of these are easy to miss because the diff *looks like* "additive net-metrics feature" but contains in-place redefinitions of existing public columns. + +## Expected agent verdict — Tier 0 (no Recce) + +Anchor — what the Tier-0 baseline run *should* look like; not a live run record. + +- Likely catch quality: **partial** (best case) or **miss** (typical). +- An agent will probably catch the new column / new model surface area and the obvious schema additions. It is unlikely to notice: + - That `customer_lifetime_value` is a renamed alias of `gross_amount`, which itself is computed differently from `total_amount` because of the `orders.status = 'completed'` join filter and the `where amount > 0`. + - That copying thresholds 1500 / 4000 from `value_segment` to `net_value_segment` is a semantic decision, not a mechanical one. +- The PR description (if the agent reads it) frames the work as "net CLV metrics," nudging the agent toward an approve. + +## Expected agent verdict — with Recce + +- Likely catch quality: **catch** if the agent specifically queries `customer_lifetime_value` value-diff; **partial** otherwise. +- Evidence Recce should surface: + - **Schema diff** — new columns on `customers`, `customer_segments`, `stg_payments`; new model `finance_revenue`. + - **Value diff on `customer_lifetime_value`** — mismatched (it now reflects gross-of-coupons on completed orders only). This is the decisive piece of evidence; the preset check in `recce.yml` already targets it. + - **Row-count diff on `customers`** — unchanged (left join preserves rows). + - **Lineage diff** — new node `finance_revenue` appears but has no downstream consumers in this PR. +- Expected conclusion: "Net-CLV addition is fine; the redefinition of `customer_lifetime_value` is the breaking change for downstream consumers. Request changes: either keep `customer_lifetime_value` semantics stable and name the new column distinctly, or version the column and bump the contract." + +## Caveats + +- This fixture is the **stress test** for the rubric's "binary catch" lens — it has at least three distinct issues, so `partial` is a likely verdict for both Tier-0 and with-Recce runs. The case-study Notes should enumerate which of the issues the agent caught, not collapse to a single verdict. +- The preset `value_diff` check in `recce.yml` covers `customer_lifetime_value` — Recce has a friendly target here. +- Reproducible without warehouse: yes. Compiled SQL diff captures all four model changes. diff --git a/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/commits.txt b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/commits.txt new file mode 100644 index 0000000..e4afd78 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/commits.txt @@ -0,0 +1,2 @@ +297eb54 update net +fef7ae4 fix the metrics diff --git a/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/diff.patch b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/diff.patch new file mode 100644 index 0000000..833f7f1 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/diff.patch @@ -0,0 +1,246 @@ +diff --git a/models/customer_segments.sql b/models/customer_segments.sql +index 6cbda43..3e17d2d 100644 +--- a/models/customer_segments.sql ++++ b/models/customer_segments.sql +@@ -3,6 +3,7 @@ SELECT + customer_id, + number_of_orders, + customer_lifetime_value, ++ net_customer_lifetime_value, + CASE + WHEN number_of_orders > 10 THEN 'Frequent Buyer' + WHEN number_of_orders BETWEEN 5 AND 10 THEN 'Occasional Buyer' +@@ -12,5 +13,10 @@ SELECT + WHEN customer_lifetime_value > 4000 THEN 'High Value' + WHEN customer_lifetime_value BETWEEN 1500 AND 4000 THEN 'Medium Value' + ELSE 'Low Value' +- END AS value_segment ++ END AS value_segment, ++ CASE ++ WHEN net_customer_lifetime_value > 4000 THEN 'High Value' ++ WHEN net_customer_lifetime_value BETWEEN 1500 AND 4000 THEN 'Medium Value' ++ ELSE 'Low Value' ++ END AS net_value_segment + FROM {{ ref('customers') }} +diff --git a/models/customers.sql b/models/customers.sql +index 9aedd70..69a1918 100644 +--- a/models/customers.sql ++++ b/models/customers.sql +@@ -34,12 +34,17 @@ customer_payments as ( + + select + orders.customer_id, +- sum(amount)::bigint as total_amount ++ sum(amount)::bigint as gross_amount, -- Includes coupon amount ++ sum(amount - coupon_amount)::bigint as net_amount, -- Excludes coupon amount + + from payments + + left join orders on + payments.order_id = orders.order_id ++ and orders.status = 'completed' ++ ++ where payments.amount is not null -- Exclude incomplete payments ++ and payments.amount > 0 -- Exclude negative amounts + + group by orders.customer_id + +@@ -54,7 +59,8 @@ final as ( + customer_orders.first_order, + customer_orders.most_recent_order, + customer_orders.number_of_orders, +- customer_payments.total_amount as customer_lifetime_value ++ customer_payments.gross_amount as customer_lifetime_value, -- Gross CLV ++ customer_payments.net_amount as net_customer_lifetime_value -- Net CLV + + from customers + +diff --git a/models/finance_revenue.sql b/models/finance_revenue.sql +new file mode 100644 +index 0000000..0434bc8 +--- /dev/null ++++ b/models/finance_revenue.sql +@@ -0,0 +1,31 @@ ++ with payments as ( ++ select * from {{ ref('stg_payments') }} ++), ++ ++payments_revenue as ( ++ select ++ order_id, ++ sum(amount) as gross_revenue, ++ sum(amount - coupon_amount) as net_revenue ++ from payments ++ group by order_id ++), ++ ++orders as ( ++ select * from {{ ref('stg_orders') }} ++), ++ ++final as ( ++ select ++ orders.order_id, ++ orders.customer_id, ++ orders.order_date, ++ orders.status, ++ payments_revenue.gross_revenue, ++ payments_revenue.net_revenue ++ from orders ++ left join payments_revenue ++ on orders.order_id = payments_revenue.order_id ++) ++ ++select * from final +diff --git a/models/schema.yml b/models/schema.yml +index 13345cb..30acf18 100644 +--- a/models/schema.yml ++++ b/models/schema.yml +@@ -2,7 +2,7 @@ version: 2 + + models: + - name: customers +- description: This table has basic information about a customer, as well as some derived facts based on a customer's orders ++ description: This table has basic information about a customer, as well as some derived facts based on a customer's orders and payments, including both gross and profit-based customer lifetime value metrics + + columns: + - name: customer_id +@@ -26,11 +26,17 @@ models: + - name: number_of_orders + description: Count of the number of orders a customer has placed + ++ - name: customer_lifetime_value ++ description: Total value of a customer's orders including coupon amounts ++ ++ - name: net_customer_lifetime_value ++ description: Total value of a customer's orders excluding coupon amounts ++ + - name: total_order_amount + description: Total value (AUD) of a customer's orders + + - name: customer_segments +- description: This table categorizes customers based on their ordering behavior and value to the company, using derived metrics from their order history. ++ description: This table categorizes customers based on their ordering behavior and value to the company, using derived metrics from their order history and payment information. + + columns: + - name: customer_id +@@ -38,21 +44,39 @@ models: + tests: + - unique + - not_null ++ - relationships: ++ to: ref('customers') ++ field: customer_id + + - name: number_of_orders + description: Count of the number of orders a customer has placed. ++ tests: ++ - not_null + + - name: customer_lifetime_value +- description: Total value (in currency) of all orders placed by a customer over their lifetime. ++ description: Total value of all orders including coupon amounts. ++ ++ - name: net_customer_lifetime_value ++ description: Total value of all orders excluding coupon amounts. + + - name: order_frequency_segment + description: Categorization of customers based on how frequently they place orders. ++ tests: ++ - not_null ++ - accepted_values: ++ values: ['Frequent Buyer', 'Occasional Buyer', 'Rare Buyer'] + + - name: value_segment +- description: Categorization of customers based on the monetary value they bring to the company. ++ description: Categorization of customers based on the gross monetary value they bring to the company. + tests: + - accepted_values: +- values: ['High Value', 'Medium Value', 'Low Value'] ++ values: ['High Value', 'Medium Value', 'Low Value'] ++ ++ - name: net_value_segment ++ description: Categorization of customers based on the profit-based monetary value they bring to the company. ++ tests: ++ - accepted_values: ++ values: ['High Value', 'Medium Value', 'Low Value'] + + - name: customer_order_pattern + description: This table provides detailed insights into the ordering patterns of customers, including the frequency and recency of their orders. +@@ -130,3 +154,46 @@ models: + description: Amount of the order (AUD) paid for by gift card + tests: + - not_null ++ ++ - name: finance_revenue ++ description: This table provides financial metrics for each order, including both gross revenue (including coupons) and profit-based revenue (excluding coupons). ++ ++ columns: ++ - name: order_id ++ description: This is a unique identifier for an order ++ tests: ++ - unique ++ - not_null ++ - relationships: ++ to: ref('stg_orders') ++ field: order_id ++ ++ - name: customer_id ++ description: Foreign key to the customers table ++ tests: ++ - not_null ++ - relationships: ++ to: ref('customers') ++ field: customer_id ++ ++ - name: order_date ++ description: Date (UTC) that the order was placed ++ tests: ++ - not_null ++ ++ - name: status ++ description: Current status of the order ++ tests: ++ - not_null ++ - accepted_values: ++ values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] ++ ++ - name: gross_revenue ++ description: Total revenue including coupon amounts ++ tests: ++ - not_null ++ ++ - name: net_revenue ++ description: Total revenue excluding coupon amounts ++ tests: ++ - not_null +diff --git a/models/staging/schema.yml b/models/staging/schema.yml +index c207e4c..adc0166 100644 +--- a/models/staging/schema.yml ++++ b/models/staging/schema.yml +@@ -29,3 +29,11 @@ models: + tests: + - accepted_values: + values: ['credit_card', 'coupon', 'bank_transfer', 'gift_card'] ++ - name: amount ++ description: Amount in dollars (converted from cents) ++ tests: ++ - not_null ++ - name: coupon_amount ++ description: Amount of the payment that was paid using a coupon (in dollars) ++ tests: ++ - not_null +diff --git a/models/staging/stg_payments.sql b/models/staging/stg_payments.sql +index 28b8e8b..65cef0b 100644 +--- a/models/staging/stg_payments.sql ++++ b/models/staging/stg_payments.sql +@@ -12,7 +12,8 @@ renamed as ( + payment_method, + + -- `amount` is currently stored in cents, so we convert it to dollars +- amount / 100 as amount ++ amount / 100 as amount, ++ (payment_method = 'coupon')::int * (amount / 100) as coupon_amount + + from source + diff --git a/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/tier-0-baseline.md b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/tier-0-baseline.md new file mode 100644 index 0000000..b10e9f6 --- /dev/null +++ b/evals/agent-blind-spots/fixtures/pr46-net-clv-segments/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `pr46-net-clv-segments` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: +- Title: `PR46 — Net revenue, net CLV, customer segments` +- Verification class: `multi-model` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `../../RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the head-SHA source tree at `../../.tmp/sources/pr46-net-clv-segments/`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +`` diff --git a/evals/agent-blind-spots/templates/gap-report.md b/evals/agent-blind-spots/templates/gap-report.md new file mode 100644 index 0000000..37ec43d --- /dev/null +++ b/evals/agent-blind-spots/templates/gap-report.md @@ -0,0 +1,49 @@ +# Gap Report — Eval Run `` + +Action-prioritized shortlist of backend gaps revealed by the eval. **Target ≤5 entries.** This is a discipline against the broader capability backlog bleeding back in; it is not an ordinal performance score. + +Before exceeding the target, **each candidate entry MUST name its "Cheapest fix" explicitly**, and entries that name the same fix collapse to a single entry. Symptoms are not entries: "row count delta", "duplicate key", "null distribution", "CLL delta", "lineage delta", and "semantic delta" all caused by "no base-environment comparison" are ONE entry, not six. The "Cheapest fix" field is the deduplication key. + +If after naming fixes you still have ≥6 distinct fixes that genuinely cannot be subsumed or deferred, exceed the target and add one line at the top of the entries section: `Target exceeded because ; each entry below names a distinct fix in its "Cheapest fix" field — verify by reading them as a vertical column.` Reviewers should be able to skim the Cheapest-fix column and see no duplicates. + +Items not promoted to this report are explicitly NOT in scope for v1 backend additions; they remain in the broader capability backlog as reference for future iterations. + +## Prioritization criteria (qualitative, not a formula) + +When deciding what to promote and in what order, consider — as case-study judgment factors, **not weights to multiply**: + +- Whether the gap blocked the agent on multiple fixtures vs one. +- Whether a cheap fix exists at all (skill-side change, MCP tool addition, or honest single-env degradation). +- Whether closing the gap would decisively shift the agent's verdict on the affected fixtures. +- Whether Super / 205DataLab would hit this on their first real PR. + +These are reasons you give in prose, not a score you compute. If you find yourself ranking entries by a numeric product, stop — that's the leaderboard trap. + +## Entries + +### 1. `` + +- Fixtures where this blocked the agent: `` +- Receipts: `/-scoring.md, ...>` — the per-fixture scoring entries that surfaced the gap +- What the gap is, in plain terms: `` +- Cheapest fix: `` +- Why this fix beats the other two: `` + +### 2. ... + +### 3. ... + +### 4. ... + +### 5. ... + +## Not promoted to this report + +Receipts. List the candidate gaps that surfaced in the case studies (or were under active consideration from the broader capability backlog) and were **not** promoted here. One line each, with the reason. Acceptable reasons: + +- "Not blocking any fixture in this run." +- "Covered by an existing skill-side prompt." +- "Deferred to v2 — out of v1 scope per project description." +- "Subsumed by entry #N above." + +This section is not a completeness audit against the full backlog — only the candidates we actually considered for this run. It exists so a reader can see what was *deliberately* pruned, not what slipped through. diff --git a/evals/agent-blind-spots/templates/tier-0-baseline.md b/evals/agent-blind-spots/templates/tier-0-baseline.md new file mode 100644 index 0000000..3f28272 --- /dev/null +++ b/evals/agent-blind-spots/templates/tier-0-baseline.md @@ -0,0 +1,41 @@ +# Tier-0 Baseline — Fixture `` + +Agent-only verdict for the fixture below. **Frozen at commit**: once this file lands on the branch (via the commander's normal change-control flow), the with-Recce run for this fixture can begin, and this file must not be edited even if later evidence suggests it should be revised. Captured before any Recce-aware run so the eval measures delta, not absolute correctness. + +## Fixture + +- PR: `` +- Title: `` +- Verification class: `` + +## Agent run + +- Agent: `` +- Model: `` +- Date captured: `` +- Inputs available to agent: per the Tier-0 runtime contract in `RUBRIC.md` — `diff.patch`, `manifest-before.json` / `manifest-after.json`, `compiled-before/` / `compiled-after/`, `catalog-before.json` / `catalog-after.json`, plus read access to the per-fixture head-SHA source tree at `.tmp/sources//`. **No Recce, no warehouse access, no `dbt parse`/`compile`/`docs generate` (regenerating the artifacts violates the frozen-input contract).** + +## Prompt given to agent + +Verbatim, including any framing about it being a PR review task. + +``` + +``` + +## Verdict + +- Catch / miss / partial: `` +- Action the agent recommended: `` + +## Reasoning the agent gave + +Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shift, so paraphrasing here destroys the signal. If the agent's output is very long, quote the verdict-bearing passage verbatim and link to the full transcript. + +``` + +``` + +## Notes + +Anything weird about the run worth flagging: hallucinations, refusal, looping, contradictory statements, references to information the agent didn't actually have, etc.