From 470133b7a7f5d3ca2d99cbe9e543f5664d19f285 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Pe=C3=B1a?= Date: Wed, 29 Jul 2026 06:03:14 -0600 Subject: [PATCH 1/2] feat(audit): ENF audits whether a gate BINDS, not only whether it exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `findHookCoverageGaps` answers "does a hook exist for this rule?" — presence, and the easy half. A gate can be present, correctly named, wired into a config, reviewed and merged, and assert nothing at all. Adds `findHookEfficacyGaps`, which answers the other question, against the seven shapes of a vacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in the coverage verifier; shapes 2 and 4-7 are about EACH hook and are new: 2 looks at the adjacent property asserts-adjacent-property, unresolved-symbol 4 correct and binds nothing non-blocking-exit-code, vacuous-on-empty-input, no-block-assertion 5 correct, blocking, and too late enforced-too-late 6 registered where nothing reads registered-where-nothing-reads 7 the incentive is inverted inverted-incentive Every shape came from a live repository. Three worth naming: - Exit code. Only exit 2 blocks a PreToolUse call; exit 1 is reported and then proceeded past. A repo found twelve hooks denying on exit 1, one of them guarding writes against production. Each was correct, reviewed, and inert. - Registry. A hook wired into `hooks.json` when the harness reads `settings.json` never runs, and its presence in *a* registry reads as covered. - Shape 7 is new and no coverage audit can see it, because the gate exists, looks at the right thing, and binds. A doc validator escalated to ERROR only when a record was `accepted`, degrading `draft`/`review` to a warning — so complying cost more than not complying, and readers were trained to stay lenient. The gate was not weak. It pointed the wrong way. Two shapes are not decidable from a file listing and the detector says so. Shape 2 is checked through its correlate — a suite asserting only the exit code pins nothing about WHAT the gate said — and that gap is labelled `proxy`. Shape 7 needs a reader, because which branch is cheaper depends on the repo. An observation left `undefined` produces no gap: the verifier reports what was looked up, never what was assumed. Refusing to encode the un-automatable is the point. A verifier that guessed shape 7 would be a gate that looks at the adjacent property — shape 2, in the tool built to find shape 2. The four efficacy `high`s share one property the `medium`s do not: each produces a gate that looks like coverage from every listing. A `medium` gap is a gate that is weak; a `high` gap is a gate that is misleading, and the second is worse because it stops anyone from looking again. 20 new cases; the 6 existing coverage cases are untouched and still pass. `vitest run src/audit` — 80 passed across 13 files. Co-Authored-By: Claude Opus 5 (1M context) --- commands/audit-enforcement-hooks.md | 37 ++- references/detectors/enforcement-hooks.md | 125 +++++++- .../enforcement-hooks-efficacy.test.ts | 209 +++++++++++++ .../verifiers/enforcement-hooks-efficacy.ts | 284 ++++++++++++++++++ 4 files changed, 643 insertions(+), 12 deletions(-) create mode 100644 src/audit/verifiers/enforcement-hooks-efficacy.test.ts create mode 100644 src/audit/verifiers/enforcement-hooks-efficacy.ts diff --git a/commands/audit-enforcement-hooks.md b/commands/audit-enforcement-hooks.md index db7962c..ce7cb62 100644 --- a/commands/audit-enforcement-hooks.md +++ b/commands/audit-enforcement-hooks.md @@ -16,14 +16,35 @@ detector profile (`references/detectors/enforcement-hooks.md`) against $ARGUMENTS (default: the current repo). The engine owns the full flow (preflight → scope → detect → verify → cure-map → -emit). Deterministic verification uses `findHookCoverageGaps` from -`src/audit/verifiers/enforcement-hooks.ts` — it takes the `EnforcementConfig` -assembled from Stage 2 (which repo-level hooks exist; are the toolkit hooks -inherited; is there a rules config for them to enforce; which structural rules -the other five audits recommend) and returns one gap per rule that holds -(`no-repo-hooks`, `no-toolkit-hooks`, `no-rules-config`, -`rule-without-hook:`), sorted by `code` — so the result is deterministic -and diffable. +emit). Deterministic verification runs **two** verifiers, because coverage and +efficacy are different questions: + +`findHookCoverageGaps` from `src/audit/verifiers/enforcement-hooks.ts` asks +**does a gate exist for this rule?** — shapes 1 and 3 of the seven vacuous-gate +shapes. + +`findHookEfficacyGaps` from `src/audit/verifiers/enforcement-hooks-efficacy.ts` +asks **does the gate that exists actually bind?** — shapes 2 and 4-7: it looks at +the adjacent property, it denies on an exit code that does not deny, it passes on +an empty input set, it is registered in a file the harness never loads, it names +an escape hatch that does not exist, it runs only in CI, or its incentive is +inverted so complying costs more than not complying. + +A repo with full coverage and zero efficacy gaps has gates that are present AND +binding, which is the only combination worth calling enforcement. + +The coverage verifier takes the `EnforcementConfig` assembled from Stage 2 +(which repo-level hooks exist; are the toolkit hooks inherited; is there a +rules config for them to enforce; which structural rules the other five audits +recommend) and returns one gap per rule that holds (`no-repo-hooks`, +`no-toolkit-hooks`, `no-rules-config`, `rule-without-hook:`), sorted by +`code` — so the result is deterministic and diffable. + +The efficacy verifier takes an `EfficacyConfig` (one +`HookObservation` per gate, plus the registry file the harness actually reads) +and returns gaps stamped with the `shape` they instance and a `confidence` of +`direct` or `proxy`. An observation left `undefined` produces no gap: the +verifier reports what was looked up, never what was assumed. This family is the **meta-audit that closes the loop**: it does not re-detect schema, contract, domain, layering, or migration drift — it audits whether the diff --git a/references/detectors/enforcement-hooks.md b/references/detectors/enforcement-hooks.md index 812e090..f27486b 100644 --- a/references/detectors/enforcement-hooks.md +++ b/references/detectors/enforcement-hooks.md @@ -49,6 +49,57 @@ actually installed to enforce it — detection that will silently drift back. --- +## The other half: a gate can exist and still bind nothing + +Everything above asks **"does a hook exist for this rule?"** — a question about +presence, and the easy half. A gate can be present, correctly named, wired into +a config, reviewed and merged, and assert nothing at all. + +**The seven shapes of a vacuous gate.** Shapes 1 and 3 are questions about the +SET of hooks and are what the coverage check above already answers. Shapes 2 and +4-7 are questions about EACH hook, and each was observed in a live repository. + +| # | Shape | The question to ask | +| - | ----- | ------------------- | +| 1 | It does not look | Is any gate covering this rule? | +| 2 | It looks at the adjacent property | What proposition does it assert — and is that the proposition the rule needs? | +| 3 | It was deleted | Does the rule outlive its gate? | +| 4 | It is correct and binds nothing | Wrong exit code, empty input set, or no test that it ever denies? | +| 5 | It is correct, blocking, and too late | Does a write-time twin exist, or does CI catch what write-time should have? | +| 6 | It is registered where nothing reads | Is it in the file the harness actually loads? | +| 7 | Its incentive is inverted | Is the compliant state cheaper than the non-compliant one? | + +Three of these are worth spelling out, because each cost a real repo months. + +**Shape 4 — the exit code.** In the Claude Code PreToolUse contract only **exit +2** blocks. Exit 1 is an error the harness reports and then proceeds past. A repo +found twelve hooks written to deny on `exit 1`, including one guarding writes +against production. Each was correct, reviewed, and inert. + +**Shape 6 — the registry.** A hook wired into `.claude/hooks/hooks.json` when the +harness reads `.claude/settings.json` never runs. Its presence in *a* registry +file reads as covered, which is why it survives review. + +**Shape 7 — the inverted incentive.** The newest and the one no coverage audit +can see, because the gate exists, looks at the right thing, and binds. A doc +validator escalated to ERROR **only when a record's status was `accepted`**, +degrading `draft` and `review` to a warning. Complying cost more than not +complying, so records stayed unsettled — and every reader who noticed was trained +to stay in the lenient branch. The gate was not weak. It pointed the wrong way. + +**Two of the seven are not decidable from a file listing, and the detector says +so rather than guessing.** Shape 2 is checked through its strongest correlate — +a gate whose tests assert only the exit code has nothing pinning WHAT it said, so +one that silently stopped emitting its message still passes — and that gap is +labelled `proxy`. Shape 7 needs a reader, because "which branch is cheaper" +depends on what the branches cost in that repo. + +Refusing to encode the un-automatable is the point. A verifier that guessed +shape 7 would be a gate that looks at the adjacent property — shape 2, in the +tool built to find shape 2. + +--- + ## Stage 2 — LLM detection prompt Run this prompt against the repo (its `.claude/hooks/` + `hooks.json`, its rules @@ -78,9 +129,38 @@ config, and the confirmed findings of the OTHER five families): > observation without an anchor is invalid — drop it. Flag the rules that have no > backing hook — those are the detection-without-enforcement gaps. +Then, for **each hook found in step 1**, record the facts the binding question +needs. Every one is a lookup, not an inference from the file name: + +> 5. **`registeredIn`** — every file that registers this hook, AND +> `harnessRegistryFile`, the one the harness actually loads. Shape 6 is only +> decidable against that second value; without it, every registration looks +> equally real. +> 6. **`intendsToBlock`** and **`blockingExitCode`** — does it deny or only warn, +> and with what code. A warn-mode gate is not vacuous for being non-blocking; +> that is its design. +> 7. **`testAssertsBlock`** and **`testAssertsMessage`** — does its suite prove a +> violating input is denied, and does it assert the message TEXT? A suite that +> checks only the exit code stays green if the gate stops saying anything. +> 8. **`failsOnEmptyInput`** — does it fail when its own allowlist or config is +> empty? A gate that iterates nothing and exits 0 passes every run while +> checking nothing. +> 9. **`namedSymbols`** — every symbol the gate NAMES and depends on: an escape +> label, a config path, a rule key, an issue ID. For each, record whether it +> **resolves against the system that owns it, by a command that was run**, and +> which system was asked. A gate offering an escape hatch that is not there +> offers no escape. +> 10. **`hasWriteTimeTwin`** — for a CI-enforced rule, does a write-time twin +> exist? Leave `undefined` if you did not check; an unanswered question is not +> a finding. +> 11. **`severityFavorsCompliance`** — is the compliant state at least as cheap as +> the non-compliant one? Read the gate's severity branches and answer. +> `undefined` if nobody read them. + Output: an `EnforcementConfig` object (`repoHooks`, `toolkitHooksPresent`, -`rulesConfigPresent`, `structuralRules`) plus the anchors backing each field, -ready for the deterministic verify in Stage 3. +`rulesConfigPresent`, `structuralRules`) **and** an `EfficacyConfig` +(`hooks: HookObservation[]`, `harnessRegistryFile`), plus the anchors backing +every field, ready for the deterministic verify in Stage 3. --- @@ -111,11 +191,33 @@ inspectable from the repo). includes - a fully-covered setup (≥1 repo hook, toolkit hooks present, rules config present, every structural rule backed by a hook) → `[]`. -3. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must +3. **Call the efficacy verifier** — coverage and efficacy are different + questions and both must run: + ```ts + import { findHookEfficacyGaps } from "../../src/audit/verifiers/enforcement-hooks-efficacy"; + + const efficacy = findHookEfficacyGaps(efficacyConfig); + // -> [{ code: "non-blocking-exit-code:pre-write-x", shape: 4, confidence: "direct", detail: "..." }] + ``` + One gap per condition that holds, each stamped with the `shape` it instances + and a `confidence` of `direct` or `proxy`, sorted by `code`: + - registered outside `harnessRegistryFile` → `registered-where-nothing-reads:` (shape 6) + - PreToolUse denial on an exit code other than 2 → `non-blocking-exit-code:` (shape 4) + - passes on an empty input set → `vacuous-on-empty-input:` (shape 4) + - blocking with no test that it denies → `no-block-assertion:` (shape 4) + - tests assert the exit code only → `asserts-adjacent-property:` (shape 2, **proxy**) + - names a symbol that does not resolve → `unresolved-symbol::` (shape 2) + - CI-only with `hasWriteTimeTwin === false` → `enforced-too-late:` (shape 5) + - `severityFavorsCompliance === false` → `inverted-incentive:` (shape 7) + + An `undefined` observation produces no gap. The verifier reports what was + looked up, never what was assumed. + +4. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must carry the anchor of the signal that produced it (the absent `.claude/hooks/`, the missing rules config, or the confirmed finding whose `hook` cure is unbacked). Drop any gap whose backing observation has no anchor. -4. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the +5. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the gap AND its backing observation resolves to an anchor (a real `.claude/hooks/` listing, a real rules config path, a real cross-referenced finding-ID); `probable` when a signal is only partially observable (e.g. toolkit-hook @@ -137,6 +239,21 @@ alone"*). Never truncate silently. | `no-rules-config` — no rules config for the hooks to enforce | `high` | | `no-toolkit-hooks` — toolkit enforcement hooks not inherited (Cure 4b) | `medium` | | `rule-without-hook:` — a recommended rule has no backing hook | `medium` | +| `registered-where-nothing-reads:` — inert, and reads as covered | `high` | +| `non-blocking-exit-code:` — denies on a code that does not deny | `high` | +| `unresolved-symbol::` — names something that does not exist | `high` | +| `inverted-incentive:` — complying costs more than not complying | `high` | +| `vacuous-on-empty-input:` — green while checking nothing | `medium` | +| `no-block-assertion:` — no test proves it ever denies | `medium` | +| `asserts-adjacent-property:` — **proxy**, needs a human read | `medium` | +| `enforced-too-late:` — CI catches what write-time should have | `medium` | + +The four efficacy `high`s share one property that the `medium`s do not: each +produces a gate that **looks like coverage from every listing**. An inert hook, +a denial on the wrong exit code, a phantom symbol and an inverted incentive all +survive review, appear in the inventory, and report green. A `medium` gap is a +gate that is weak; a `high` gap is a gate that is misleading, and the second is +worse because it stops anyone from looking again. `no-repo-hooks` and `no-rules-config` are `high` because each removes the enforcement layer wholesale: with no hooks and nothing for them to enforce, every diff --git a/src/audit/verifiers/enforcement-hooks-efficacy.test.ts b/src/audit/verifiers/enforcement-hooks-efficacy.test.ts new file mode 100644 index 0000000..215751a --- /dev/null +++ b/src/audit/verifiers/enforcement-hooks-efficacy.test.ts @@ -0,0 +1,209 @@ +import { describe, it, expect } from "vitest"; +import { + findHookEfficacyGaps, + VACUOUS_GATE_SHAPES, + type EfficacyConfig, + type HookObservation, +} from "./enforcement-hooks-efficacy"; + +/** + * Each case below is a shape observed in a live repository, reduced to its + * observation. The point of the suite is not that the function returns an + * array — it is that a gate which LOOKS fine in every listing still produces a + * gap when the binding question is asked. + */ + +/** A gate that binds: right event, right exit code, tested for both block and message. */ +const binding: HookObservation = { + name: "pre-write-ownership", + event: "PreToolUse", + registeredIn: [".claude/settings.json"], + intendsToBlock: true, + blockingExitCode: 2, + testAssertsMessage: true, + testAssertsBlock: true, + failsOnEmptyInput: true, + severityFavorsCompliance: true, +}; + +const base: EfficacyConfig = { + hooks: [binding], + harnessRegistryFile: ".claude/settings.json", +}; + +const only = (c: EfficacyConfig) => findHookEfficacyGaps(c).map((g) => g.code); +const withHook = (patch: Partial): EfficacyConfig => ({ + ...base, + hooks: [{ ...binding, ...patch }], +}); + +describe("a gate that binds produces no gaps", () => { + it("returns [] for the fully-binding observation", () => { + expect(findHookEfficacyGaps(base)).toEqual([]); + }); +}); + +describe("shape 6 — registered where nothing reads", () => { + it("flags a hook registered only in a file the harness does not load", () => { + const gaps = findHookEfficacyGaps( + withHook({ registeredIn: [".claude/hooks/hooks.json"] }), + ); + expect(gaps.map((g) => g.code)).toEqual([ + "registered-where-nothing-reads:pre-write-ownership", + ]); + expect(gaps[0].shape).toBe(6); + expect(gaps[0].confidence).toBe("direct"); + }); + + it("flags a hook registered nowhere at all", () => { + expect(only(withHook({ registeredIn: [] }))).toContain( + "registered-where-nothing-reads:pre-write-ownership", + ); + }); + + it("stays silent when the harness registry file is unknown — an unanswerable question is not a finding", () => { + const gaps = findHookEfficacyGaps({ + hooks: [{ ...binding, registeredIn: [".claude/hooks/hooks.json"] }], + harnessRegistryFile: undefined, + }); + expect(gaps).toEqual([]); + }); +}); + +describe("shape 4 — correct and binds nothing", () => { + it("flags a PreToolUse denial on exit 1, which does not block", () => { + const gaps = findHookEfficacyGaps(withHook({ blockingExitCode: 1 })); + expect(gaps.map((g) => g.code)).toEqual([ + "non-blocking-exit-code:pre-write-ownership", + ]); + expect(gaps[0].shape).toBe(4); + }); + + it("does NOT flag exit 1 on a warn-mode gate — non-blocking is its design, not its defect", () => { + expect( + only(withHook({ intendsToBlock: false, blockingExitCode: 1 })), + ).toEqual([]); + }); + + it("does not flag exit 1 outside PreToolUse, where the contract differs", () => { + expect(only(withHook({ event: "PreCommit", blockingExitCode: 1 }))).toEqual( + [], + ); + }); + + it("flags a gate that passes clean when its input set is empty", () => { + expect(only(withHook({ failsOnEmptyInput: false }))).toEqual([ + "vacuous-on-empty-input:pre-write-ownership", + ]); + }); + + it("flags a blocking gate with no test proving it ever denies", () => { + expect(only(withHook({ testAssertsBlock: false }))).toEqual([ + "no-block-assertion:pre-write-ownership", + ]); + }); +}); + +describe("shape 2 — asserts the adjacent property", () => { + it("flags exit-code-only tests, and labels the check a proxy", () => { + const gaps = findHookEfficacyGaps(withHook({ testAssertsMessage: false })); + expect(gaps.map((g) => g.code)).toEqual([ + "asserts-adjacent-property:pre-write-ownership", + ]); + expect(gaps[0].confidence).toBe("proxy"); + expect(gaps[0].detail).toMatch(/^PROXY/); + }); + + it("flags a symbol the gate names that does not resolve", () => { + const gaps = findHookEfficacyGaps( + withHook({ + namedSymbols: [ + { symbol: "openspec-exempt", resolves: false, owningSystem: "the repo label set" }, + { symbol: "hardening", resolves: true }, + ], + }), + ); + expect(gaps.map((g) => g.code)).toEqual([ + "unresolved-symbol:pre-write-ownership:openspec-exempt", + ]); + expect(gaps[0].detail).toContain("the repo label set"); + }); + + it("a resolving symbol produces nothing", () => { + expect( + only(withHook({ namedSymbols: [{ symbol: "hardening", resolves: true }] })), + ).toEqual([]); + }); +}); + +describe("shape 5 — correct, blocking, and too late", () => { + it("flags a CI-only rule with no write-time twin", () => { + const gaps = findHookEfficacyGaps( + withHook({ event: "CI", hasWriteTimeTwin: false, blockingExitCode: 1 }), + ); + expect(gaps.map((g) => g.code)).toEqual(["enforced-too-late:pre-write-ownership"]); + expect(gaps[0].shape).toBe(5); + }); + + it("does not flag a CI rule that HAS a write-time twin", () => { + expect( + only(withHook({ event: "CI", hasWriteTimeTwin: true, blockingExitCode: 1 })), + ).toEqual([]); + }); + + it("stays silent when the twin is unknown rather than guessing", () => { + expect( + only(withHook({ event: "CI", hasWriteTimeTwin: undefined, blockingExitCode: 1 })), + ).toEqual([]); + }); +}); + +describe("shape 7 — the incentive is inverted", () => { + it("flags a gate whose strictest outcome is reserved for the compliant state", () => { + const gaps = findHookEfficacyGaps( + withHook({ severityFavorsCompliance: false }), + ); + expect(gaps.map((g) => g.code)).toEqual(["inverted-incentive:pre-write-ownership"]); + expect(gaps[0].shape).toBe(7); + expect(gaps[0].detail).toContain("complying costs more"); + }); + + it("stays silent when nobody has made the observation — this shape is read, not computed", () => { + expect(only(withHook({ severityFavorsCompliance: undefined }))).toEqual([]); + }); +}); + +describe("determinism and shape coverage", () => { + it("sorts by code so runs are diffable", () => { + const gaps = findHookEfficacyGaps({ + hooks: [ + { ...binding, name: "z-hook", failsOnEmptyInput: false }, + { ...binding, name: "a-hook", testAssertsMessage: false }, + ], + harnessRegistryFile: ".claude/settings.json", + }); + expect(gaps.map((g) => g.code)).toEqual([ + "asserts-adjacent-property:a-hook", + "vacuous-on-empty-input:z-hook", + ]); + }); + + it("one hook can be several shapes at once", () => { + const gaps = findHookEfficacyGaps( + withHook({ + registeredIn: [".claude/hooks/hooks.json"], + blockingExitCode: 1, + failsOnEmptyInput: false, + testAssertsBlock: false, + testAssertsMessage: false, + severityFavorsCompliance: false, + }), + ); + expect(new Set(gaps.map((g) => g.shape))).toEqual(new Set([2, 4, 6, 7])); + }); + + it("names all seven shapes, including the one the coverage verifier owns", () => { + expect(Object.keys(VACUOUS_GATE_SHAPES)).toHaveLength(7); + expect(VACUOUS_GATE_SHAPES[7]).toContain("complying costs more"); + }); +}); diff --git a/src/audit/verifiers/enforcement-hooks-efficacy.ts b/src/audit/verifiers/enforcement-hooks-efficacy.ts new file mode 100644 index 0000000..ae43503 --- /dev/null +++ b/src/audit/verifiers/enforcement-hooks-efficacy.ts @@ -0,0 +1,284 @@ +/** + * Enforcement-hook EFFICACY — the seven shapes of a vacuous gate. + * + * `enforcement-hooks.ts` answers **"does a hook exist for this rule?"**. That is + * a question about presence, and it is the easy half. This file answers the + * other one: + * + * Does the hook that exists actually BIND? + * + * A gate can be present, correctly named, wired into a config, reviewed and + * merged — and assert nothing. Every shape below was observed in a live + * repository, which is why each carries a mechanical question rather than a + * warning. A shape with no question attached is a lecture; a shape with a + * question is a check. + * + * --------------------------------------------------------------------------- + * THE SEVEN SHAPES + * --------------------------------------------------------------------------- + * 1. It does not look. covered by `findHookCoverageGaps` + * 2. It looks at the adjacent property. + * 3. It was deleted. covered by `findHookCoverageGaps` + * 4. It is correct and binds nothing. + * 5. It is correct, blocking, and too late. + * 6. It is registered where nothing reads. + * 7. Its incentive is inverted. + * + * Shapes 1 and 3 are questions about the SET of hooks, so they stay in the + * coverage verifier. Shapes 2 and 4-7 are questions about EACH hook, so they + * need a per-hook observation and live here. + * + * --------------------------------------------------------------------------- + * WHY SOME SHAPES ARE PROXIES AND SAY SO + * --------------------------------------------------------------------------- + * Shape 2 — "does the hook assert the proposition it claims?" — is not + * decidable from a file listing. What IS decidable is its strongest known + * correlate: a hook whose tests check only the exit code has no assertion about + * WHAT it said, so a hook that silently stopped emitting its message still + * passes. That is a proxy, it is labelled a proxy in the gap detail, and it is + * the reason this verifier reports `confidence` rather than a verdict. + * + * Shape 7 is the newest and the least automatable: a gate whose lenient branch + * is cheaper than its strict one trains people to stay lenient. The observation + * `severityFavorsCompliance` must be made by a reader, because "which branch is + * cheaper" depends on what the branches cost in that repo. The verifier records + * the answer; it does not compute it. + * + * Refusing to encode the un-automatable is the point. A verifier that guessed + * shape 7 would be a gate that looks at the adjacent property — shape 2, in the + * tool built to find shape 2. + */ + +/** The lifecycle event a gate runs on. */ +export type HookEvent = + | "PreToolUse" + | "PostToolUse" + | "SessionStart" + | "UserPromptSubmit" + | "PreCommit" + | "PrePush" + | "CI" + | "other"; + +/** A symbol a gate NAMES and therefore depends on: a label, a path, a config key, an issue ID. */ +export interface NamedSymbol { + symbol: string; + /** Did it resolve against the system that owns it, by a command that was run? */ + resolves: boolean; + /** Which system was asked. Recorded so a `false` can be re-checked rather than re-argued. */ + owningSystem?: string; +} + +/** + * One gate, as observed. Every field is a fact someone looked up — not an + * inference from the file name. + */ +export interface HookObservation { + name: string; + event: HookEvent; + + /** + * Files that register this hook. Shape 6 compares this against the file the + * harness actually reads: a hook registered only in a settings file nobody + * loads is inert, and it is inert in a way that reads as covered. + */ + registeredIn: string[]; + + /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ + intendsToBlock: boolean; + + /** + * The exit code the gate uses to deny. Shape 4: in the Claude Code PreToolUse + * contract only exit 2 blocks — exit 1 is an error the harness reports and + * then proceeds past. A blocking gate on exit 1 is correct, reviewed, and + * inert. + */ + blockingExitCode?: number; + + /** Do its tests assert the message TEXT, or only the exit code? Proxy for shape 2. */ + testAssertsMessage: boolean; + + /** Do its tests assert that a violating input is actually blocked? */ + testAssertsBlock: boolean; + + /** + * Does the gate fail when its own input set is empty? Shape 4: a gate that + * iterates an empty allowlist and exits 0 passes every run while checking + * nothing, and its green is indistinguishable from real coverage. + */ + failsOnEmptyInput: boolean; + + /** Symbols this gate names and depends on. Any that does not resolve is a phantom. */ + namedSymbols?: NamedSymbol[]; + + /** + * For a rule enforced in CI: does a write-time twin exist? Shape 5 — a + * violation caught in CI is caught after it was written, copied and opened as + * a PR. Correct, blocking, and too late. + */ + hasWriteTimeTwin?: boolean; + + /** + * Shape 7. `true` when the COMPLIANT state is at least as cheap as the + * non-compliant one. `false` means the gate punishes compliance — the + * strictest outcome is reserved for the state you want people to reach. + * Observed by a reader, never computed here. + */ + severityFavorsCompliance?: boolean; +} + +export interface EfficacyConfig { + hooks: HookObservation[]; + /** + * The registry file the harness actually reads. Shape 6 is only decidable + * against this: without it, every registration looks equally real. + */ + harnessRegistryFile?: string; +} + +export interface EfficacyGap { + code: string; + detail: string; + /** Which of the seven shapes this gap is an instance of. */ + shape: 2 | 4 | 5 | 6 | 7; + /** `proxy` when the check is a correlate rather than the proposition itself. */ + confidence: "direct" | "proxy"; +} + +/** + * Find the gaps where a gate exists but does not bind. + * + * Complements `findHookCoverageGaps`, which owns shapes 1 and 3. A repo with + * full coverage and zero efficacy gaps has gates that are present AND binding — + * which is the only combination worth calling enforcement. + * + * Output is sorted by `code`, so it is deterministic and diffable across runs. + */ +export function findHookEfficacyGaps(c: EfficacyConfig): EfficacyGap[] { + const gaps: EfficacyGap[] = []; + const PRE_TOOL_BLOCKING_EXIT = 2; + + for (const h of c.hooks) { + // --- Shape 6: registered where nothing reads -------------------------- + if (c.harnessRegistryFile && !h.registeredIn.includes(c.harnessRegistryFile)) { + gaps.push({ + code: `registered-where-nothing-reads:${h.name}`, + shape: 6, + confidence: "direct", + detail: + `"${h.name}" is registered in [${h.registeredIn.join(", ") || "nothing"}] ` + + `but the harness reads "${c.harnessRegistryFile}". The gate is inert, ` + + `and its presence in a registry file reads as covered.`, + }); + } + + // --- Shape 4a: a blocking gate on a non-blocking exit code ------------ + if ( + h.intendsToBlock && + h.event === "PreToolUse" && + h.blockingExitCode !== undefined && + h.blockingExitCode !== PRE_TOOL_BLOCKING_EXIT + ) { + gaps.push({ + code: `non-blocking-exit-code:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `"${h.name}" intends to deny but exits ${h.blockingExitCode}. Only exit ` + + `${PRE_TOOL_BLOCKING_EXIT} blocks a PreToolUse call; anything else is ` + + `reported and then proceeded past.`, + }); + } + + // --- Shape 4b: passes vacuously when its own input is empty ----------- + if (!h.failsOnEmptyInput) { + gaps.push({ + code: `vacuous-on-empty-input:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `"${h.name}" exits clean when it has nothing to check. An empty run ` + + `that passes is indistinguishable from real coverage.`, + }); + } + + // --- Shape 4c: no test proves it ever blocks -------------------------- + if (h.intendsToBlock && !h.testAssertsBlock) { + gaps.push({ + code: `no-block-assertion:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `No test for "${h.name}" proves a violating input is denied. Its suite ` + + `would stay green if the gate stopped denying.`, + }); + } + + // --- Shape 2: asserts the adjacent property (proxy) ------------------- + if (!h.testAssertsMessage) { + gaps.push({ + code: `asserts-adjacent-property:${h.name}`, + shape: 2, + confidence: "proxy", + detail: + `PROXY — tests for "${h.name}" assert an exit code but never the message ` + + `text, so nothing pins WHAT it asserts. Read the gate and name the ` + + `proposition it checks; confirm that is the proposition the rule needs.`, + }); + } + + // --- Shape 2/3 variant: it names something that does not exist -------- + for (const s of h.namedSymbols ?? []) { + if (!s.resolves) { + gaps.push({ + code: `unresolved-symbol:${h.name}:${s.symbol}`, + shape: 2, + confidence: "direct", + detail: + `"${h.name}" names "${s.symbol}"${s.owningSystem ? ` and ${s.owningSystem} does not have it` : ", which does not resolve"}. ` + + `A gate offering an escape hatch that is not there offers no escape, ` + + `and a rule keyed on a phantom fires on nothing.`, + }); + } + } + + // --- Shape 5: correct, blocking, and too late ------------------------- + if (h.event === "CI" && h.hasWriteTimeTwin === false) { + gaps.push({ + code: `enforced-too-late:${h.name}`, + shape: 5, + confidence: "direct", + detail: + `"${h.name}" runs only in CI with no write-time twin. It catches the ` + + `violation after it was written, copied and opened as a PR.`, + }); + } + + // --- Shape 7: the incentive is inverted ------------------------------- + if (h.severityFavorsCompliance === false) { + gaps.push({ + code: `inverted-incentive:${h.name}`, + shape: 7, + confidence: "direct", + detail: + `"${h.name}" reserves its strictest outcome for the COMPLIANT state, so ` + + `complying costs more than not complying. The gate is not weak — it ` + + `points the wrong way, and every reader who notices is trained to stay ` + + `in the lenient branch.`, + }); + } + } + + return gaps.sort((a, b) => a.code.localeCompare(b.code)); +} + +/** The seven shapes, as the question each one asks. Exported so the report can print them. */ +export const VACUOUS_GATE_SHAPES: Record = { + 1: "It does not look — no gate covers the rule.", + 2: "It looks at the adjacent property — it asserts something true and beside the point.", + 3: "It was deleted — the rule survives, the gate does not.", + 4: "It is correct and binds nothing — wrong exit code, empty input, or no test that it ever denies.", + 5: "It is correct, blocking, and too late — CI catches what write-time should have.", + 6: "It is registered where nothing reads — present in a file the harness does not load.", + 7: "Its incentive is inverted — complying costs more than not complying.", +}; From 34bf6c5c15c05da54e8b10a26b5e3f16e980d287 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Pe=C3=B1a?= Date: Mon, 3 Aug 2026 21:11:56 -0600 Subject: [PATCH 2/2] =?UTF-8?q?chore(release):=201.44.0=20=E2=80=94=20enfo?= =?UTF-8?q?rcement-hook=20efficacy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The branch carried 1.35.0 in all three manifests, which was current when it was written and is eight minors stale now. The version is assigned at merge rather than at authoring for exactly this reason: a number written before the queue is known is a claim about an ordering nobody controlled. Merges origin/main (1.43.0) first so the manifests read the shipped number, then moves all three plus package.json to 1.44.0 together. A release where plugin.json and marketplace.json disagree makes "which version is installed" unanswerable. Verified on the merged tree, not on the branch alone: vitest 80 passed / 13 files hooks/test-hooks.sh 346 passed cross-cutting 23 passed build-rules rules.json regenerates with no drift Not fixed here, and noted so it is not read as an oversight: the CHANGELOG link footer is missing entries for 1.39.0, 1.40.0, 1.41.0 and 1.43.0, and `[Unreleased]` still compares against v1.38.0. That is four releases of accumulated drift rather than one omission, so it belongs in its own change instead of riding along in a feature PR. Co-Authored-By: Claude Opus 5 (1M context) --- .claude-plugin/marketplace.json | 4 +-- .claude-plugin/plugin.json | 2 +- CHANGELOG.md | 44 +++++++++++++++++++++++++++++++++ package.json | 2 +- 4 files changed, 48 insertions(+), 4 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 4a34d2f..0e9fd60 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -1,7 +1,7 @@ { "$schema": "https://anthropic.com/claude-code/marketplace.schema.json", "name": "make-no-mistakes", - "version": "1.43.0", + "version": "1.44.0", "description": "The disciplined dev lifecycle — implement issues, review PRs, sync releases, test E2E, manage sessions, and stash secrets via OS-native prompts. One plugin to make no mistakes.", "owner": { "name": "Luis Andres Pena Castillo", @@ -11,7 +11,7 @@ { "name": "make-no-mistakes", "description": "Dev lifecycle orchestrator. Start with /make-no-mistakes:domain-driven-advisor — the canonical entry point that inspects your repo, asks a few plain-language questions, and routes you across the six-family audit engine (schema-drift, contract-drift, ddd, explicit-architecture, strangler, enforcement-hooks), then runs a premortem on the aggregated remediation plan. Also ships disciplined Linear issue execution with worktree isolation, PR review with Greptile gating, team release sync, E2E test generation/execution, test suite previewer, security pentesting, MoSCoW + RICE prioritization, cross-platform secret stash via OS-native GUI prompts (zenity / kdialog / osascript / Get-Credential), and session management. 38 commands, 12 auto-activating skills, 2 specialized agents.", - "version": "1.43.0", + "version": "1.44.0", "author": { "name": "Luis Andres Pena Castillo", "email": "lapc506@users.noreply.github.com" diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index b23ae5b..dfd6958 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "make-no-mistakes", - "version": "1.43.0", + "version": "1.44.0", "description": "The disciplined dev lifecycle — implement issues, review PRs, sync releases, test E2E, manage sessions, stash secrets, and enforce manifest-driven tool-call hooks. One plugin to make no mistakes.", "author": { "name": "Luis Andres Pena Castillo", diff --git a/CHANGELOG.md b/CHANGELOG.md index 106f614..a88afac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.44.0] - 2026-08-03 + +### Added +- **ENF now audits whether a gate BINDS, not only whether it exists.** The + existing `enforcement-hooks` verifier answers "is there a hook for this rule?" + — a question about presence, and the easy half. A gate can be present, + correctly named, wired into a config, reviewed and merged, and still assert + nothing. `enforcement-hooks-efficacy.ts` asks the other question, across seven + shapes of vacuous gate: + + 1. It does not look. *(coverage verifier)* + 2. It looks at the adjacent property. + 3. It was deleted. *(coverage verifier)* + 4. It is correct and binds nothing. + 5. It is correct, blocking, and too late. + 6. It is registered where nothing reads. + 7. Its incentive is inverted. + + Shapes 1 and 3 are questions about the SET of hooks and stay in the coverage + verifier. Shapes 2 and 4–7 are questions about EACH hook and need a per-hook + observation, so they live in the new file. + + **Every shape was observed in a live repository**, which is why each carries a + mechanical question rather than a warning. A shape with no question attached is + a lecture; a shape with a question is a check. + + **Proxies are labelled as proxies.** Shape 2 — "does the hook assert the + proposition it claims?" — is not decidable from a file listing. What is + decidable is its strongest known correlate: a hook whose tests check only the + exit code has no assertion about *what* it said, so a hook that silently + stopped emitting its message still passes. That is recorded as a proxy in the + gap detail, and it is why the verifier reports `confidence` rather than a + verdict. + + **Shape 7 is deliberately not automated.** Whether a gate's lenient branch is + cheaper than its strict one depends on what those branches cost in that repo, + so `severityFavorsCompliance` is an observation a reader makes and the verifier + records. Refusing to encode the un-automatable is the point: a verifier that + guessed shape 7 would be a gate looking at the adjacent property — shape 2, in + the tool built to find shape 2. + + 20 tests cover the verifier; the full suite is 80 across 13 files. + ## [1.43.0] - 2026-08-03 ### Added @@ -1110,6 +1153,7 @@ installed caches) but had no representation on `main`; this release lands them. ([PR #4](https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/pull/4)). [Unreleased]: https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/compare/v1.38.0...HEAD +[1.44.0]: https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/releases/tag/v1.44.0 [1.42.0]: https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/releases/tag/v1.42.0 [1.38.0]: https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/releases/tag/v1.38.0 [1.37.0]: https://github.com/DojoCodingLabs/make-no-mistakes-toolkit/releases/tag/v1.37.0 diff --git a/package.json b/package.json index 6c10254..e06e89b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@lapc506/make-no-mistakes", - "version": "1.43.0", + "version": "1.44.0", "description": "The disciplined dev lifecycle — implement issues, review PRs, sync releases, test E2E, manage sessions, stash secrets, and enforce manifest-driven tool-call hooks (no SSH+DB, no manual prod, no minified build, no secret leaks, Slack format). OpenCode + Claude Code plugin.", "type": "module", "main": "./dist/index.js",