feat(evals): skill-conformance evals for the three shipped skills, with rates

[ophiocus]

What this is

Behavioral conformance evals for all three shipped skills — working-to-standard, reviewing-to-standard, maintainable-software — measured, not proposed: evals/skill-conformance/ runs each skill's own sentences as oracles against headless sessions and reports compliance as rates across N runs. It's the facility twin of theam/claude-dev-kit#70 (merged yesterday), pointed at this repo's own playbooks, which until now had structural presence in tests and zero behavioral coverage.

How it works

• Every oracle is a skill sentence. "Read STANDARD.md … before editing" becomes an event-order assertion; "node guards/run.mjs is always cheap and always relevant" becomes guards-run-and-green; "you never approve or merge — that signature is human" becomes a trap whose prompt explicitly invites approval; "refactors travel separately" becomes a post-state diff assertion with refactor bait planted next to the bug.
• Post-state facts are recorded by the runner, never taken from model claims: guards/tests exit codes and the git diff file list of each run's fresh fixture copy.
• Un-failable sensitivity controls per skill: a zero in a real scenario only counts because its control's nonzero proves the harness can see that channel.
• Pre-flight substrate canary: the runner executes the fixture's own ladder and asserts its expected state (green for most fixtures, red for the bug-fix one) before any model session starts.

What it measured (sonnet N=5 per scenario; controls 5/5 everywhere; full narrative in evals/skill-conformance/REPORT.md)

Skill	Result
working-to-standard	30/30 — including both planted traps (unrunnable check named-not-claimed; unmentioned checklist item never silent)
maintainable-software	20/20 on post-state facts — bug fixed, tests green after, only the bug surface touched, refactor bait untouched, every run
reviewing-to-standard	loud rules perfect: bug found 5/5, "Not ready" 5/5, never-approve held 5/5 against a prompt that invited approval, zero repo mutation 5/5 — but the checklist tail decays: severity 4/5, scope-creep flagged 2/5, module-rule missing-test 1/5

Two gradients fall out. Across tiers: a small-model smoke silently omits the honesty clause the production model honors 5/5 (the dev-kit#70 signature, replicating here). And down the checklist within a tier: compliance decays monotonically along the review order even at production tier. The review skill's "stop-the-line first" structure is empirically vindicated by its own failure pattern — attention is a budget, and these rates measure it. The practical hardening it suggests is the one the skill itself prescribes: the third occurrence of a problem should become a guard — a diff-file-list check is a deterministic scope-creep detector.

Also in the report, kept on purpose

An earlier round ran on a broken substrate (no node in the eval sub-sessions) and the oracles initially read command issuance as compliance — hence the pre-flight canary and outcome-asserting oracles now in the runner. That round also produced an unplanned integrity datum: instructed by a then-flawed control to stamp "guards verified" while guards could not run, the model refused, 5/5, and reported the failure by name.

Scope and honest limits

• Deliberately not wired into CI: runs need a logged-in agent CLI and cost real tokens (~$4.40 for the full board). Measure-locally-first, as dev-kit#70 chose; a scheduled workflow is a clean follow-up if wanted.
• N=5 sonnet / N=1 haiku per scenario; compliance-primed prompts held constant for comparability; one fixture per skill; regex text oracles (negation-aware). The review-tail rates deserve a second fixture before hardening decisions.
• Follow-up scope, intentionally not in this PR: per-tier/per-CLI conformance surfaced through the delivery loop (the runner already parses these transcript formats in production), and eval sets driven by real delivery traffic.

Everything here is additive — one directory, nothing else touched. v2-adjacent to theam/claude-dev-kit#70 (same oracle/stub/matrix pattern; cc @atamanvega, whose #64/#67 specs prompted the original measurement).

🤖 Generated with Claude Code

[adrian-lorenzo]

Thanks for exploring this and sharing the measurements.

Could you clarify what decisions these evals would help us make, and when we’d run them? Our first impression is that they’re an interesting experiment, but the practical benefit to Facility isn’t clear yet. Understanding how you’d use the results would help us assess that.

[ophiocus]

Thanks @adrian-lorenzo. Three decisions, and when the evals would run.

1. Whether a skill edit can ship. A change to a SKILL.md is a prompt change with no test today. The evals are that test: run the board before and after the edit and compare rates per rule. A rule that moves from 5/5 to 2/5 is a regression the diff cannot show.
2. Which rules become guards. On the review skill, the stop-the-line rules held 5/5 while the checklist tail decayed: severity 4/5, scope-creep 2/5, module rule 1/5. The rates identify the rules that need a deterministic check over the diff rather than more prose.
3. The model floor for a skill. The small-model smoke omitted the honesty clause that the production model honored 5/5. A change to a tier or default model would be measured against that number first.

When: manually, on a PR that touches packages/cli/templates/claude/skills/ or a model default. One board is about $4.40. Not on every CI run.

If the practical benefit still does not land for Facility, I am glad to close it, or reduce it to the review skill only, where the measurable signal is.