agent-validation-harness

An agent reviewing its own work shares its own priors. So does an agent of the same model family: two instances of one model make the systematic misreads that model's priors favor, converge on the identical wrong reading, and the convergence then reads as confirmation — which is exactly backwards, because the agreement was produced by the shared prior rather than by the truth. A clean verdict from a same-model reviewer is evidence against idiosyncratic error and nothing more. This repo is the fix: blind, cross-family, contamination-controlled judging, with an explicit "approved but not certain enough to land" state, plus hooks that put hard ceilings on what an unattended run can do.

It is extracted from a private simulation project where it gates changes to an empirically-measured model — one where a wrong number does not throw an exception, it just quietly becomes the new baseline. That is the environment this discipline was built for, and every guard in it is a scar from a specific failure.

The pipeline

premise gate → pre-op approval → work → driver review → BLIND cross-family post-op
                                                                    ↓
                        implementation review ← IMPLEMENT / LOG / REJECT ← 2-of-2

Step	Role	Family	Blind to
0	premise-verifier	family A (opus)	the driver's belief
1	preop-judge	family B (fable, must differ)	— judges the plan openly
2–3	work subagent, then the driver's own review (gate #1)
4	postop-judge	family B (fable, must differ)	the driver's verdict + reasoning
5	2-of-2 landing rule — both judges must ACCEPT, driver writes its verdict FIRST
6	IMPLEMENT (HIGH+HIGH) / LOG / REJECT
7	implementation-reviewer	family A (opus)	— sighted, PR-style

Three design decisions carry most of the weight:

• Blindness is mechanical, not polite. The post-op judge receives exactly the pre-op context packet plus the work deliverable. The bridges dispatch with tools structurally disabled — an agent profile declaring tools: [], verified unable to read files — so the reviewer genuinely cannot go and find the driver's notes. The judge also returns a contamination-check field, which makes a broken run visible rather than silent.
• LOG is a first-class outcome. Most pipelines have only ship-or-don't, so a judge-approved-but-uncertain result has nowhere to go except ship. LOG makes "correct, not yet certain enough to land" a real state, which is what lets confidence actually gate anything.
• A split escalates; it does not get resolved. A principled disagreement between the two judges goes to the owner with both rationales logged. The pipeline never manufactures consensus.

Full walkthrough with the reasoning for each step: docs/PIPELINE.md. Routing policy and canonical model names: docs/CROSS-FAMILY-PROTOCOL.md.

The failure stories

Every guard here is a scar, and a guard whose justification has been forgotten looks like bureaucracy and gets deleted. So the incidents are documented: docs/FAILURE-STORIES.md.

Incident	What it produced
~5 hours re-deriving ground truth the fixtures already held. Nothing was violated — every evidence rule stated a floor and none stated a ceiling, so "prove it differently" had no stopping point.	the SUFFICIENCY clause on two guards, P10 reuse-before-derive, the /scientific-method cheap lane, and hooks/autonomous-image-budget.py
Two premise breaks — two near-identically-named entities treated as one, and a "universal control" nobody had ever measured. Both were premises; neither was a plan.	step 0, the premise gate (agents/premise-verifier.md) and non-negotiable #1
A working-tree clobber — one session discarded a file another session was mid-edit on. No reflog entry exists for unstaged discards. The discipline hook watched Edit/Write/Agent but not Bash, so the highest-risk action had no guard at all.	hooks/destructive-bash-guard.py and the isolated-worktree constraint

The thread through all of them: prose does not hold under long-session momentum. Each was already covered by a written rule that was followed right up until it was not, and each fix converted the rule into something mechanical.

What's in here

skills/     scientific-method (the procedure of record) · logic-gate · code-review
            tdd-forward (test-first modeling) · skill-maintenance · context (template)
agents/     10 durable role definitions — 4 pipeline roles, 4 gate roles + 2 same-role
            cross-model variants that Read the canonical body so a role cannot drift
hooks/      4 PreToolUse hooks: the discipline router, the destructive-bash guard,
            and two unattended-run caps (uncommitted diff size, image reads)
bridges/    dispatch-claude.sh / dispatch-kimi.sh — cross-family CLI dispatch with
            tools disabled, JSON extracted with a real parser, model provenance injected
docs/       PIPELINE.md · CROSS-FAMILY-PROTOCOL.md · FAILURE-STORIES.md
CLAUDE.md   the canonical instruction file; AGENTS.md / QWEN.md are thin shims

Quickstart

git clone https://github.com/maxwellcsutton/agent-validation-harness
cd agent-validation-harness
bash scripts/smoke.sh          # static gate: hooks compile, bridges parse, links resolve

To adopt it in a project:

1. Wire the hooks. Copy hooks/ into your project and merge the hooks block from settings.example.json into .claude/settings.json. Order matters on the Edit matcher — the context-adding hook runs before the deny-capable one.
2. Set the budgets. HARNESS_AUTONOMOUS truthy in unattended runs only (attended sessions are deliberately untouched — a human is watching the diff); then HARNESS_BLAST_LIMIT, HARNESS_IMAGE_BUDGET, HARNESS_FRAMES_BUDGET to taste. All four hooks fail OPEN.
3. Copy skills/ and agents/ into .claude/, and point any other harness's skills directory at the same copy rather than forking it.
4. Fill in the four anchors. The docs are written against <VERIFY> (your verification gate), <REGRESSION> (your labeled snapshot), <DECISIONS> (your append-only WHY-log), and <PROBE-DATA> (your measured fixtures). scripts/smoke.sh fails if an undeclared anchor appears anywhere.
5. Write your own skills/context/SKILL.md. The shipped one is a deliberate stub explaining the role that skill plays and the shape that makes it work — its real content is entirely project-specific.
6. Configure the extension points if you want them: HARNESS_VALUE_SURFACE, HARNESS_ID_LINT, HARNESS_PIPELINE_AGENTS, HARNESS_EXTRACTOR_PATTERN. Unset means the corresponding guard stays silent.

You need at least two model families on the CLI for the cross-family gates to mean anything. With one family, the gates still run and still catch real defects through information asymmetry — but label those verdicts "same-family only", as the skills instruct. An unlabeled weak result is worse than no result.

What this is NOT

• Not a framework. There is nothing to install and no API. It is a set of skills, role definitions, hooks, and bridges that encode an operating discipline.
• Not a benchmark or an eval suite. It gates individual changes in a live repo; it does not score models.
• Not automatic. /logic-gate and /code-review are owner-invoked by design. A gate that fires on everything gets ignored, and a cross-family dispatch is not free.
• Not a claim that judges are reliable. The claim is narrower and, I think, defensible: a blind judge from a different family, given a decontaminated packet and an explicit rubric, is uncorrelated enough with the author to be worth a gate — and a 2-of-2 rule over two uncorrelated verdicts beats one confident one.

Provenance

Extracted from a private simulation project where this pipeline gates changes to an empirically-measured model. Fresh implementation, rewritten to be domain-neutral; the mechanisms are unchanged, the anchors are templated, and the war stories are anonymized but intact — because a guard justified by a real failure is the product.

MIT licensed.