docs(evals): build six PR fixture set for /recce-verify v1 eval

[even-wei]

Force-pushed to switch from committed-artifacts (+377k LOC) to a build-script approach (~1.4k LOC, almost all markdown). Original capture commit removed; artifacts are regenerable via build_fixtures.sh.

Summary

Six PR fixtures from DataRecce/jaffle_shop_golden for the /recce-verify v1 eval, plugged into the rubric + templates authored in #28. Each fixture lives at evals/agent-blind-spots/fixtures/<slug>/ with a README (PR description + expected Tier-0 verdict + expected with-Recce verdict + caveats), a tier-0-baseline.md template instance with eval-run fields left as <TBD by eval run>, commits.txt (base + head SHAs read by the build script), and a small diff.patch (source-models diff base..head).

Artifacts are no longer committed. The large dbt outputs (manifest-before.json, manifest-after.json, catalog-*.json, compiled-before/, compiled-after/) are gitignored and regenerated by evals/agent-blind-spots/build_fixtures.sh before each eval run. The build script is idempotent.

Stacked on #28. Review after #28 lands or rebase onto main.

Fixtures

Slug	Source PR	Class
pr1-fix-clv	#13	semantic
pr42-is-closed-filter	#14	row-grain
pr2-refactor-cte-to-models	#15	refactor (equality expected)
pr3-amount-double-to-decimal	#16	type / rounding drift
pr44-promotion-flags	#20	schema-expansion + intermediate row-filter snapshot
pr46-net-clv-segments	#2	multi-model semantic

For PR #20 the intermediate problematic commit (23b96ca, "Add promotion information") is captured separately into pr44-promotion-flags/artifacts/intermediate-commit-23b96ca/ so the row-filter accident is replay-able without rewinding the head. Its diff base..23b96ca is committed at pr44-promotion-flags/diff-from-base-to-intermediate.patch.

Build script

cd evals/agent-blind-spots
./build_fixtures.sh

Required system deps: git, uv. Everything else is installed into .tmp/.venv/ from pinned versions in the script (Python 3.11.11, dbt-core 1.11.9, dbt-duckdb 1.10.1, duckdb 1.5.2; dbt packages per upstream packages.yml).

The script:

1. Clones DataRecce/jaffle_shop_golden into .tmp/jaffle_shop_golden/ (gitignored). Fetches PR head refs so all fixture SHAs are reachable.
2. Creates .tmp/.venv/ with Python 3.11.11 and installs pinned dbt versions.
3. Writes a DuckDB profiles.yml (Snowflake creds not required).
4. For each fixture: reads base SHA from the per-fixture README.md, head SHA from commits.txt, checks out each one, runs dbt deps && dbt parse && dbt compile && dbt docs generate, copies target/manifest.json, target/compiled/, target/catalog.json to the fixture's artifacts/ directory.
5. PR Make /recce-dev:recce-eval executable from any directory #20 fixture also builds the intermediate commit 23b96ca.
6. Scrubs user_id, invocation_id, root_path from manifests and catalogs.

Build duration on a local M-series Mac: ~2:40 (warm .tmp/); ~2:50 cold-start (first clone + venv).

Surprises / rubric-refinement signal

Picked up during fixture capture — feeds back into #28 (RUBRIC.md) and the eval-run interpretation. Pulled directly from the per-fixture READMEs:

• pr1-fix-clv is a friendly-path bias trap. recce.yml in the source repo already ships value_diff and query_diff preset checks on customer_lifetime_value. That makes this fixture an unusually favorable target for Recce. None of the other five fixtures has this advantage, so a strong score on PR1 alone does not generalize. Flag this when reading the eval report.
• PR fix(recce-dev): add .mcp.json with SessionStart port sync hook #16 (DECIMAL) is a Tier-2 fixture wearing Tier-1 clothes. Head SHA 1c56861 is a mechanical merge commit; the substantive type narrowing lives at 6ffc23f. The compiled SQL captures the type change, but the gap from agent-only to with-Recce shows up clearest with a base/target comparison — Tier-2 territory, not Tier-1. Don't penalize a Tier-1-only run for missing rounding drift here.
• PR Make /recce-dev:recce-eval executable from any directory #20 (promotion flags) is two fixtures stacked. The head is a clean schema expansion. The interesting case is the intermediate commit 23b96ca, where the author also dropped a where has_promoted_orders = true row filter and reverted it one commit later. The eval can replay the intermediate as if it were the PR head to exercise the row-filter trap inside what looks like a benign schema-expansion. Snowflake-specific boolor_agg appears in source and compiles fine under DuckDB (no compile-time function validation) — it would fail at execute on DuckDB. Note for DuckDB-backed Recce runs.
• PR Feedback by Kent Huang #2 (net CLV segments) is the stress-test fixture. It redefines customer_lifetime_value in place, introduces three row filters on payments, copy-pastes a magic threshold for net_value_segment, and adds a finance_revenue model with no downstream consumers. The expected Recce-vs-no-Recce delta is largest here; it is also the fixture most likely to expose rubric gaps.
• Catalog row/column stats are zero everywhere. dbt docs generate runs against an empty DuckDB. Schema info (column names, types) is intact; row counts and column stats are not. Do not score rubric items off catalog row stats — only off schema/lineage.
• PR No changes required — comment directed at @copilot-pull-request-reviewer, not actionable #14 uses an older base SHA (62d6dc9) than the other five (f09861a). Don't mix bases when computing inter-fixture deltas.

LOC budget note

Committed diff is ~1,445 lines across 29 files. Breakdown:

• Build script: 244 LOC
• Markdown (README rollup + per-fixture README + tier-0 baseline templates): ~650 LOC
• Small source-models diff.patch files (per fixture): ~525 LOC — pr2 (121, refactor with CTE splits) and pr46 (246, multi-model) dominate
• commits.txt, .gitignore entries, and the touch-up to evals/agent-blind-spots/README.md: ~25 LOC

Above the 400-line target ceiling, below the 600-line test-only ceiling. Most of the volume is human-authored markdown (per-fixture caveats and tier-0 templates) — necessary for fixture interpretation and not realistically compressible.

Test plan

• build_fixtures.sh runs cleanly from a fresh .tmp/; produces six fixtures.
• build_fixtures.sh is idempotent — re-running overwrites cleanly.
• Spot-check: produced manifests have the same nodes keys and compiled_code/raw_code content as the originally-captured ones.
• Spot-check: PR Make /recce-dev:recce-eval executable from any directory #20 intermediate commit preserves the where has_promoted_orders = true row filter in compiled SQL.
• git status is clean after a build (gitignore catches .tmp/ and fixtures/*/artifacts/).
• All caveats from per-fixture READMEs are rolled up in fixtures/README.md.
• Eval-runner spot-check (out of scope here — DRC-3405).

Resolves DRC-3402

[even-wei]

Update (HEAD: 767ca54) — Verdict: GO

[Incremental Review] One new commit since prior review (767ca54 docs(evals): address review findings on build script and baselines). 8 files touched: build_fixtures.sh (+13/-1), fixtures/README.md (+3/-2), 6 per-fixture tier-0-baseline.md (input-line rewrites; pr44 also gets verification-class fix).

Verification:

• bash -n clean; shellcheck clean (same SC2016 false-positive on intentional single-quoted regex).
• git -C .../jaffle_shop_golden worktree list shows six fresh per-fixture worktrees at the correct head SHAs (2abf877, 9c386b4, 1c56861, d2be60a, bd407ac, 297eb54) — each matches both commits.txt[0] and the per-fixture README.md Head SHA line.
• All six baselines now reference ../../RUBRIC.md and ../../.tmp/sources/<slug>/ with the correct per-fixture slug in each file (no copy-paste drift).
• RUBRIC.md on PR docs(evals): scoring rubric for /recce-verify v1 #28 tip (origin/feature/drc-3403-rubric) lists the same six-input contract the baselines now point at — cross-PR contract is consistent.
• .tmp/sources/ is covered by the existing .tmp/ umbrella in .gitignore (no new entry needed; verified check-ignore).
• ~111 MB on disk for six worktrees (vs ~48 MB for the shared clone). Acceptable.

Resolution of prior findings

#	Prior finding	Severity	Status
1	Tier-0 baseline templates undercount inputs vs RUBRIC.md	ISSUE	Addressed — all six fixtures now reference the full contract with per-fixture source path. Template file (templates/tier-0-baseline.md) lives in PR #28 and was updated there in c0cb2a5.
2	Private-repo auth requirement undocumented	ISSUE	Addressed (docs path) — fixtures/README.md:74 now explicitly calls out DataRecce/jaffle_shop_golden as private and instructs gh auth setup-git or credential helper. The pre-flight option from the prior review's "(a)" branch was not taken; the docs-only "(b)-style" fix is sufficient because the error class only matters on first clone.
3	Verification-class drift on pr44-promotion-flags baseline	ISSUE (was filed as a Pass-C NOTE in the prior body, surfaced to Issues)	Addressed — pr44-promotion-flags/tier-0-baseline.md:9 now reads schema-expansion + (intermediate row-filter accident), matching the fixture README.
N1	Partial-failure UX — set -e exits without naming the failed fixture	NOTE	Still open — no trap added. Acceptable for a developer script; can be deferred.
N2	Two-source-of-truth for SHAs (README vs commits.txt)	NOTE	Still open — no side-by-side validation step. Acceptable; the script's failure modes if they drift are loud (build target mismatch).
N3	No CI smoke for build_fixtures.sh	NOTE	Still open — follow-up issue, not in this PR's scope.
N4	Redundant mkdir at build_fixtures.sh:220 (now :233)	NOTE	Still open — harmless duplicate, low-cost to leave.

The two prior ISSUEs that drove NO-GO are both resolved with high-quality fixes (per-fixture worktrees are a stronger fix than just doc cleanup, and incidentally also address the converging finding from PR #28's reviews about the shared clone leaving the last-fixture SHA visible).

New findings from this incremental pass

NOTE: Orphan worktree registration after manual .tmp/sources/<slug>/ deletion. If a user manually rm -rf evals/agent-blind-spots/.tmp/sources/pr1-fix-clv/ (without git worktree remove) and re-runs the script, the [[ -d "${source_dir}" ]] guard at build_fixtures.sh:217 is false, so the script skips the cleanup branch, then git worktree add fails with fatal: '...' is a missing but already registered worktree; use 'add -f' to override, or 'prune' or 'remove' to clear. Verified locally with a synthetic repro. Two low-cost mitigations: (a) prepend git -C "${JSG_DIR}" worktree prune --quiet before the if [[ -d ]] block, or (b) pass --force to worktree add. Severity is NOTE because the natural cleanup path is rm -rf .tmp/ (the umbrella), which removes jaffle_shop_golden/ itself and so the orphan registration vanishes too — only triggers on partial manual cleanup. Not blocking. (build_fixtures.sh:216-220.)

NOTE: Pre-flight auth check still not in script. The docs fix is correct but a user who skipped reading the fixtures README and hits git clone directly still gets a confusing "Repository not found" error rather than an actionable "run gh auth setup-git". A two-line git ls-remote "${JSG_URL}" >/dev/null 2>&1 || { echo "..." >&2; exit 1; } before the first clone in build_fixtures.sh:38 would close that gap. Follow-up, not blocking.

Verdict: GO

Both ISSUEs from the prior review are resolved. The cross-PR cleanup (per-fixture worktrees at .tmp/sources/<slug>/) is a structurally better fix than what was suggested in the prior review and the matching c0cb2a5 change on PR #28 keeps the contract in sync. Remaining items are all NOTEs that the author can defer or address in a follow-up.

This PR is ready to merge once PR #28 (feature/drc-3403-rubric) lands — its base — so the relative path ../../RUBRIC.md from each baseline resolves on main. As a stacked PR, merge order is enforced by GitHub when the base updates after #28 merges.

---

Original review (HEAD: `847249c`, posted 2026-05-11) — kept for audit

Code Review: PR #29

Files reviewed: 29 (1 script, 1 gitignore, 8 docs files, 6×4 per-fixture files, 1 extra intermediate diff)
Categories: Config/infra, Tests (eval scaffolding)
Passes run: A, B, C, D, E, F

Validation Results

Pass A — Correctness & Logic — PASS (with NOTEs)

build_fixtures.sh is straightforward and bash -n / shellcheck clean (only SC2016 false positives on intentional single-quoted regex). The extract_base_sha regex ^- Base SHA: \[0-9a-f]+`matches exactly the canonical README line on every fixture, andpr42-is-closed-filter's additional - Base SHA for this PR is ...Caveats line correctly does **not** match (no colon after "Base SHA"). The head-SHA pickup fromcommits.txt resolves to the right commit in every fixture, including PR #20 where line 1 (bd407ac`) matches the README's full head SHA.

Pass B — Security — PASS

No credentials, tokens, or env-var leaks in the script. git clone over HTTPS — auth is delegated to the user's git credential helper (see Pass D below). The Python scrubber walks the JSON tree and rewrites only user_id, invocation_id, root_path — no risk of executing untrusted content. Build artifacts are gitignored, so no manifest content (which can include root_path reflecting the build host) ever lands in the repo.

The prompt asked about defensive shell-escaping of PR SHAs. The SHAs flow into git checkout --detach "${sha}" (quoted) and into a Python heredoc (sys.argv — safe). The extract_base_sha regex is anchored to [0-9a-f]+ so a malicious commit message can't inject non-hex chars through that path. The only way to inject a non-SHA is via awk 'NF>0 {print $1; exit}' on commits.txt, but that value is then passed quoted to git checkout which would simply fail. Safe enough.

Pass C — Cross-Reference Consistency — FAIL (one ISSUE)

ISSUE: Tier-0 baseline templates undercount the inputs RUBRIC.md actually specifies. Every per-fixture tier-0-baseline.md and templates/tier-0-baseline.md says:

Inputs available to agent: dbt manifest, compiled SQL pre/post, git diff. No Recce, no warehouse access.

But the Tier-0 agent runtime contract in RUBRIC.md (from #28, commit ca837ed) lists six input categories: diff.patch, manifest-before/after.json, compiled-before/after/, catalog-before/after.json, fixtures/<id>/ source dir, and read access to the dbt project source at the head SHA.

The build script faithfully produces catalog-before.json and catalog-after.json for every fixture. But the per-fixture baselines tell an eval runner those inputs don't exist, which:

1. Will silently bias Tier-0 runs against catalog evidence (the agent won't be told the catalog is available).
2. Breaks the cross-PR contract: an eval runner reading only the baseline will not match the rubric's Tier-0 capability set.
3. Means six files were copy-pasted with the same stale phrasing rather than referencing the rubric's contract section.

The fix is to align the template Inputs available to agent line with the RUBRIC contract (or replace the line with a pointer to RUBRIC.md#tier-0-agent-runtime-contract), then propagate to all six per-fixture files.

File:line citations:

• evals/agent-blind-spots/templates/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr1-fix-clv/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr2-refactor-cte-to-models/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr3-amount-double-to-decimal/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr42-is-closed-filter/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md:16
• evals/agent-blind-spots/fixtures/pr46-net-clv-segments/tier-0-baseline.md:16

NOTE: Verification-class wording drift between README and tier-0 baseline for pr44-promotion-flags. The fixture README at evals/agent-blind-spots/fixtures/pr44-promotion-flags/README.md:6 says class is schema-expansion + (intermediate row-filter accident); the tier-0 baseline at evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md:9 says just schema-expansion. Not a bug — the intermediate is exercised as a separate sub-run per the README's "head vs intermediate" structure — but the baseline file under-describes what it baselines.

Pass D — Error Handling & Edge Cases — FAIL (one ISSUE, one NOTE)

ISSUE: DataRecce/jaffle_shop_golden is a private repo, but the build script's docs claim "only required system dependency is uv (and git)". gh api repos/DataRecce/jaffle_shop_golden returns "private": true. build_fixtures.sh:38 does git clone "${JSG_URL}" — a plain HTTPS clone that requires the runner to have GitHub credentials configured (credential helper, SSH key via git config url, or gh auth setup-git). On a fresh CI runner this will fail with a generic "Authentication failed" or 404 error, not a helpful "please authenticate" message.

The fixtures README at evals/agent-blind-spots/fixtures/README.md:73 says:

The script's only required system dependency is uv (and git).

That's incorrect — it also requires authenticated git access to a private repo. Two viable fixes: either (a) call out the auth requirement and pre-flight it (e.g., gh auth status or a friendly error if git ls-remote fails), or (b) make the repo public (out of scope of this PR but worth flagging upstream).

Citations:

• evals/agent-blind-spots/build_fixtures.sh:31 — JSG_URL hardcoded HTTPS clone
• evals/agent-blind-spots/fixtures/README.md:73 — incorrect "only required system dependency" claim
• evals/agent-blind-spots/README.md — no auth pre-flight mentioned

NOTE: set -e aborts on the first fixture failure, leaving its artifacts/ dir empty. build_fixtures.sh:197 removes ${artifacts} before build_at_sha runs. If any of dbt deps/parse/compile/docs-generate fails partway, the script exits with the half-built fixture left in an empty-artifacts/ state. Subsequent runs will retry — so semantically idempotent — but a partial run that succeeds for 3 fixtures and fails on the 4th leaves the user with no OK line for 5 and 6 and may not make it obvious which fixture broke. The minimum-cost mitigation: trap on EXIT, print FAIL ${slug} for the in-progress fixture before exiting. Or pivot to set +e around build_fixture calls, collect failures, and exit with summary. Acceptable as-is for a developer script.

Pass E — Test Coverage & Quality — N/A

This PR is eval fixture scaffolding, not production code. There are no tests for build_fixtures.sh itself. Given the script is small (245 LOC), deterministic per pinned versions, and verified by the artifacts it produces (which downstream eval runs will exercise), a dedicated unit test is not the right investment. However, a smoke script that runs build_fixtures.sh end-to-end in CI would catch upstream drift — e.g., jaffle_shop_golden PR rebases, dbt-duckdb version pin invalidation, or new fixtures with malformed commits.txt. Not blocking; worth filing as a follow-up.

Pass F — Diff-Specific Checks — PASS (with NOTEs)

The .gitignore additions correctly catch evals/agent-blind-spots/.tmp/ (clone + venv + DuckDB file) and evals/agent-blind-spots/fixtures/*/artifacts/ (per-fixture outputs). I verified locally: built artifacts under pr1-fix-clv/artifacts/ and the intermediate dir under pr44-promotion-flags/artifacts/intermediate-commit-23b96ca/ show in git status --ignored, and evals/agent-blind-spots/.tmp/ (with jaffle_shop_fixture_build.duckdb, the cloned jaffle_shop_golden/, and profiles/) is also fully ignored. No stray artifacts leak into the diff.

NOTE: Two-source-of-truth for SHAs. Base SHA lives in README.md; head SHA lives in commits.txt. The script parses both. If a contributor updates only one when refreshing a fixture, builds silently use stale data. Not a defect in this PR — but a real fragility once humans start editing these fixtures. A one-liner validation step ("read both, log them side-by-side, fail if commits.txt's first SHA doesn't match ^- Head SHA: in README") would catch it. Worth a follow-up issue, not a blocker for this PR.

NOTE: Redundant mkdir. build_fixtures.sh:220 does mkdir -p "${idir}/compiled" before calling build_at_sha, but build_at_sha itself does mkdir -p "${compiled_out}" after rm -rf. Harmless duplicate.

NOTE: Reliance on bash 4+. Shebang is #!/usr/bin/env bash — picks up brew bash on macOS, the system bash 5 on most Linux distros. macOS system /bin/bash is 3.2 but the script never relies on associative arrays or read -i. Should run fine even there. No action.

Verification Results

• bash -n clean.
• shellcheck clean except SC2016 (false positive — intentional single-quoted regex).
• Artifacts confirmed gitignored locally via git status --ignored.
• --empty-catalog flag confirmed present in dbt-core v1.11.0 onward (covers the pinned v1.11.9).
• All Base/Head SHAs in the per-fixture READMEs are formatted such that the script's regex matches exactly one line.

Verdict: NO-GO

Blockers

(none)

Issues

1. Tier-0 baseline templates undercount the inputs RUBRIC.md specifies — see Pass C ISSUE. Six per-fixture baselines plus the template all say "manifest, compiled SQL pre/post, git diff" while RUBRIC.md's runtime contract also lists catalog JSONs and project-source read access. Eval runners reading the baseline first will run an under-equipped Tier-0, biasing the very delta this eval exists to measure. evals/agent-blind-spots/templates/tier-0-baseline.md:16 plus the six fixtures.

2. Private-repo auth requirement undocumented — see Pass D ISSUE. DataRecce/jaffle_shop_golden is private; evals/agent-blind-spots/fixtures/README.md:73 claims uv + git are the only requirements. Fresh CI / new-contributor onboarding will fail with a confusing error. Either document + pre-flight (gh auth status, fast git ls-remote check) or make the source repo public.

Notes

1. Verification-class wording drift on pr44-promotion-flags tier-0 baseline — fixture README says schema-expansion + (intermediate row-filter accident); baseline says just schema-expansion. evals/agent-blind-spots/fixtures/pr44-promotion-flags/tier-0-baseline.md:9.
2. Partial-failure UX — set -e exits without naming the failed fixture in the success line set. A trap on EXIT to print FAIL ${slug} would close the loop. evals/agent-blind-spots/build_fixtures.sh:170 onward.
3. Two-source-of-truth for SHAs — base in README, head in commits.txt. Add a side-by-side validation step.
4. No CI smoke for build_fixtures.sh — upstream drift (jaffle_shop_golden rebases, dbt-duckdb pin invalidation) will silently rot the eval. Follow-up issue.
5. Redundant mkdir — build_fixtures.sh:220 mkdir is redundant with build_at_sha's own mkdir.

What I could not verify

• Whether build_fixtures.sh actually runs to completion on a fresh Linux VM (I only re-read the local-built artifacts; the worktree had them from a prior author run).
• Whether dbt deps succeeds against current upstream tags of data-mie/dbt_profiler@0.8.1, dbt-labs/dbt_utils@0.9.6, dbt-labs/audit_helper@0.11.0 — dbt Hub doesn't yank packages but a maintainer could.
• Whether the compiled SQL boolor_agg claim survives a real DuckDB attempt — the script uses --empty-catalog for dbt docs generate, so no rows execute, so even invalid functions would compile and ship.

What I looked for and did not find

• Token / credential leak in the script — none.
• Path-traversal via fixture slug — slugs are static FIXTURES=(...) array, no user input.
• Shell-injection via PR SHAs — values are always quoted; regex anchored.
• Stray writes outside .tmp/ and fixtures/*/artifacts/ — none.
• Forgotten gitignore patterns for build by-products (.duckdb, profiles/) — covered by evals/agent-blind-spots/.tmp/ umbrella.
• Cross-PR contract drift between this PR and docs(evals): scoring rubric for /recce-verify v1 #28's RUBRIC.md — one drift found (the Tier-0 inputs ISSUE above); the rest of the layout (paths under artifacts/, intermediate-commit-23b96ca structure, scrubbing) matches RUBRIC.md Tier-0 agent runtime contract.

[wcchang1115]

Cross-reviewed this against PR #28's rubric (same pair). Three things worth raising; the first is the same root cause I flagged on #28 but the fix mechanism lives here, so this is the place to act on it.

1. Worktree shared-object-DB leak (combined with the wholesale PR-ref fetch)

This is the cross-PR sibling of the point I raised on #28. Posting here because the fix is in build_fixtures.sh, not in RUBRIC.md.

The leak: build_fixtures.sh:286 does git worktree add --quiet --detach "${source_dir}" "${head_sha}". Git worktrees share the parent .git object database, so an agent inside .tmp/sources/pr1-fix-clv/ can run git log --all, git rev-parse origin/pr/13, or git show <any-sha> and immediately reach (a) later commits on the same PR that contain the actual fix, (b) every other fixture's head, (c) every PR head that the fetch step pulled in.

The amplifier — build_fixtures.sh:107:

git -C "${JSG_DIR}" fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pr/*'

This fetches every PR head from jaffle_shop_golden on every run, warm or cold. Combined with the shared object DB, the agent can git log origin/pr/<n> for any PR in the source repo, not just the six fixtures.

Two paths, either fixes the leak; the structural one is preferred:

• Structural (preferred): replace the worktree-add with a fresh-object-DB per fixture:

  git init --quiet "${source_dir}"
  git -C "${source_dir}" fetch --quiet --depth 1 "${JSG_DIR}" "${head_sha}"
  git -C "${source_dir}" reset --quiet --hard FETCH_HEAD

  No shared object DB; git log --all from the fixture only sees the one commit you fetched. Same disk footprint (≈63 MB × 6) as the current worktree approach since worktrees share blobs anyway. Loses the "single shared clone" performance benefit, but the build is already 2:40 — the extra fetches are tens of seconds.

• Narrow the fetch (additionally, or as a fallback): replace the wholesale PR-ref fetch with explicit per-fixture SHA fetches:

  for sha in $(awk 'NF>0 {print $1}' fixtures/*/commits.txt); do
      git -C "${JSG_DIR}" fetch --quiet origin "${sha}"
  done

  Doesn't fix the leak by itself, but radically shrinks its blast radius — only the SHAs you actually need are reachable. Also faster on warm runs.

If you take the structural path, the narrowed fetch is moot. If you stick with worktrees and rely on the rubric's "explicitly NOT allowed" ban list (the alternative I offered on #28), the narrowed fetch is essential — otherwise every public PR on jaffle_shop_golden is one git log --all away.

2. boolor_agg may break the Tier-1c with-Recce run on pr44, not just Tier-0

The PR body and pr44's fixture README correctly flag that Snowflake's boolor_agg compiles fine under DuckDB (no compile-time function validation) but fails at execute time. Tier-0 inputs are compile-time only, so the build artifacts are clean.

But Tier 1c in the rubric is "structured queries against the current dev env" — which presumably means the eval-runner does some form of dbt run to materialize the models in DuckDB before invoking Recce's query-shaped MCP tools. At that point, pr44's with-Recce arm dies at materialization, before Recce sees anything.

If that's the eval-runtime plan, then pr44 is effectively Tier-1a/1b-only against DuckDB and the rubric's three subsets aren't all reachable. Worth confirming with DRC-3405's runtime design before the eval starts — either pr44 is documented as 1a/1b-reachable-only, or the runner needs a DuckDB-compatible substitute model, or the fixture moves to Snowflake.

Not a request to fix here; just a note to surface before the eval-runner code is written.

3. SHA storage is split across two files; cheap to tighten

fixtures/<slug>/README.md carries the canonical Base SHA (full 40-char) and Head SHA (full). fixtures/<slug>/commits.txt carries head (short, e.g. 2abf877) plus the intermediate when applicable.

build_fixtures.sh reads base from the README and head from commits.txt. If a contributor updates one and not the other, the build silently uses the stale value (likely the README's head — except the script doesn't read that one).

Already in your "Not done" list, agreed it's NOTE-level. The cheap fix is to add a 3-line cross-check after head_sha is extracted: assert that the README's Head SHA line starts with the same prefix as commits.txt's head. Refuses to build on drift; one source of truth at runtime.

While you're in extract_base_sha — line 146's regex is `[0-9a-f]+` which silently accepts 1-char or upper-case-typo SHAs (the latter doesn't match and gets reported as "no Base SHA", which is confusing). Tightening to [0-9a-f]{40} makes typos loud. One-character edit.

---

Not blocking. #1 is the only one I'd want addressed pre-merge — and even there, the "narrowed fetch" sub-fix alone is a meaningful win if the structural path is too much churn for this PR.

---

Re-review (2026-05-14, post ae28f37)

#1 — resolved cleanly

Fix is tighter than what I proposed. Three things worth calling out:

• No git remote add (line 247 fetches from a literal path, not a configured remote). A stray git fetch inside the fixture has no upstream to talk to. That's a stronger isolation than the git init + fetch + reset I suggested.
• Inline leak-check assertion (lines 254–258): git rev-list --all HEAD --count == 1 or the script FAILs. I didn't ask for this — it's a real "regress this and the next build catches it" invariant. Nice.
• Adversarial verification in the commit message (git show 297eb54, git rev-parse origin/pr/20, git show 23b96ca all fail; git log --all shows only 2abf877 PR1). That's the kind of receipt that turns "the fix should work" into "the fix verifiably works."

Build time held (2:48 cold / 2:39 warm vs prior 2:40).

#2 (boolor_agg / Tier-1c) and #3 (SHA split + regex) — still open, as expected

Both were flagged as deferred / non-blocking. No commit needed here; they're either eval-runner-side decisions (#2) or candidates for a cleanup pass (#3).

Verdict

LGTM on the structural change. The leak-check assertion is the part I'm most happy about — it converts "the author is careful" into "the build is self-defending."