feat: close five gaps between what kit claims and what kit enforces by sandstream · Pull Request #556 · sandstream/kit

sandstream · 2026-08-30T16:18:33Z

Description

Five commits closing gaps between what kit claims and what kit enforces — most of them found by breaking a rule and watching the gate stay green.

The through-line: an ADR that declares more than its rule covers, a linter nobody runs, a test that passes because a tool is missing, and a verdict whose scope the reader has to assume. Each is invisible by construction, because the failing state and the healthy state print identically.

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Documentation update

Changes Made

ADR-0006 accepted — src/utils is a leaf (81fabd7)

The first decision recovered by kit adr derive and made binding. It replaces three of the nine proposed candidates with their general form: derive proposed utils → commands (113 reverse edges), utils → adapters (8) and utils → memory (5) separately, but measurement showed utils imports nothing from src/ at all. One rule covers all three and any subsystem added later. utils is the most fanned-in directory in the repo — a single upward import from it creates a cycle through most of the tree.

Two candidates deliberately not accepted, with the reason recorded in the ADR: the "no subsystem imports the command layer" general form fires on a test fixture string, and profile → exec-broker is a lone pair with no wider pattern.

kit check states its scope next to its verdict (4bd1e04)

scope: 42 check(s) — 41 inspect the code, 1 executes it. A pass here does not
cover runtime behaviour; `kit broker` is that tier.

Counted from the checks that ran, not asserted, so it cannot go stale as categories are added. The reason is measured rather than stylistic — see Reviewer Notes.

ADR-0002's dependency floor is enforced for the first time (4bd1e04)

Found by breaking it: kit pkg npm:jscpd added a fifth runtime dependency and kit adr check reported "4 enforced ADR(s) — no new violations". The floor is not expressible in the grammar (package.json is not in the walked file set; patterns match line by line so a dependencies entry cannot be told from a devDependencies one), so it is enforced by src/dependency-floor.test.ts — exact set, exact versions, no ranges.

ADR-0007 — kit owns the rig, never the judgement (4bd1e04)

The boundary for measuring model + kit residual risk. kit owns the frozen input, ingest schema, deterministic adjudication and receipts; the repository decides whether a finding is real; kit never produces, ranks or grades a judgement. No kit-enforce block of its own — ADR-0001's forbid_import is the mechanical half, and duplicating it would give two rules that can drift.

enforced_by: — the future-proofing (1e50cdf)

Three grammar limits are now measured, each found by trying to write a rule and failing: no paths negation, line-by-line pattern matching, and a text-level import extractor that reads fixture strings as imports. When a rule hits one, nothing used to happen — the decision quietly went unencoded and the ADR kept declaring more than it enforced.

An ADR can now name its real enforcement point in frontmatter. adr list prints it; adr check fails when the named file does not exist, which is the half that matters — a pointer nobody verifies reads as coverage. Same rule class as ci-adr-gate.test.ts and self-audit. Checked only for accepted ADRs: a proposal's pointer is an intention, not a claim.

Lint and a machine-dependent test (b45d65c)

All 6 eslint errors fixed. Two were scanners (snyk, sentrux) rethrowing a JSON parse failure without cause, discarding the only thing that says where the output went wrong. Two dead eslint-disable directives removed. 174 → 166, 0 errors.

And collectStandardsKeys "returns empty slices when the general tools are absent" asserted against the real cwd with the comment "lizard/jscpd/scc are not installed in this environment". Provisioning lizard and jscpd turned it red with no code change. It now empties PATH so the condition is forced rather than hoped for.

Shared memory (5ed6853) — the skill-gate decision recorded as derived, not operator, since an agent derived it from measurement and no human stated it. Side effect: it is the first entry kit's own aging model can classify (+200 days → aging=1), closing a caveat that outlived #551.

Testing

Everything mutation-proved rather than eyeballed:

mutation	breaks
inject `import "../adr.js"` into `src/utils/`	ADR-0006, exit 1, cited to file:line
add a fifth runtime dependency	both dependency-floor assertions
float a version to `^4.3.6`	exactly the pinning assertion
point `enforced_by` at a missing file	`adr check` exit 1, file cited
delete `dependency-floor.test.ts`	same
check pointers for every status / no status	each breaks exactly one test
treat every check category as executing	three `tierNotice` tests
drop the `kit broker` pointer	exactly the "names the remedy" test

Test Coverage

Added new tests
Updated existing tests
All tests pass (npm test) — 4776 tests, 0 fail, 3 skipped on the merged tree, all three skips loud (0500 does not deny writes … running as root? ×2, semgrep not installed)

Manual Testing

node dist/cli.js adr list — 7 ADRs, ADR-0002 shows └ also enforced by src/dependency-floor.test.ts
node dist/cli.js adr check — exit 0
node dist/cli.js review --stages adr,skill — passes
node dist/cli.js check --category tests --enforce-tests — exit 0
npx eslint . — 0 errors

Breaking Changes

None. enforced_by is optional frontmatter; an ADR without it parses to an empty list.

Checklist

My code follows the project's style guidelines
I've updated documentation as needed — CLAUDE.md, AGENTS.md, COMMANDS.md and the contract, including the three grammar limits so the next person who cannot express a rule reaches for enforced_by instead of dropping it
All new code has test coverage
All tests pass locally (npm test)
No new warnings or errors introduced
Commit messages follow conventions
No hardcoded secrets or sensitive data
Code has been self-reviewed

Performance Impact

No performance impact

Reviewer Notes

Rebased on the new main. #554, the 6.10.2 release and the monkey-test gate landed while this was open (43 files, 3488 insertions). origin/main is merged in at 0a2069e. One real conflict, in .kit/shared/memory.jsonl — an append-only store where both sides had appended, so both entries are kept (main's f57f88 first, since it landed first) rather than either side winning. 49 valid JSONL lines, no markers, signatures untouched. Full suite re-run against the merged tree: 4776 tests, 0 fail.

Why the scope line exists, and why it is worth a line of output. Multi-tier verification research (arXiv:2607.00107 — 8,918 C++ programs, 851 tasks, four tiers, three models plus human-authored code) found AI-generated code roughly twice as likely as human code to trigger a confirmed runtime violation, controlling for length and test pass-rate — while under static analysis the two appear equally safe, a similarity the authors call misleading. kit's surface is 41/42 static. On a weak model its green says approximately what it would have said about human-written code: not a wrong verdict, an uninformative one in the same colour as an informative one. The line is the cheap half of the fix; ADR-0007 is the expensive half, and is deliberately just a boundary for now.

Provisioning, and one thing I did that should not pass unnoticed. lizard and jscpd went through kit triage (PASS, 88/100) and are installed. trivy and scc fail closed here — Coverage: PARTIAL — 5 probe(s) could not run → TRIAGE FAILED — because the GitHub API is scoped to three repos in this environment. I installed scc past that refusal to get the first file size measurement, then removed it. The numbers below are real; the method for one of them was not clean.

First run of three gates that had never run: complexity 350 functions over CCN 15 or 80 lines, duplication 3.6% — passes, file size 49 files over 600 lines. .kit-baseline.json freezes 833 findings so the gates report new ones rather than the backlog; it deliberately retains the scc entries so the file-size gate does not report 49 "new" findings the day scc is provisioned somewhere it can be triaged.

Still outstanding, not in this PR: 166 eslint warnings (complexity=71, no-explicit-any=32, size rules) — structural debt, baselined; trivy still unprovisioned, which is the only reason check --category security exits 1 here; and the new shared entry is [unsigned], because this container has no identity key — it is in the store and readable, just without a signature, same as it would be on any machine that has not run kit identity.

🤖 Generated with Claude Code

The first decision recovered by `kit adr derive` and then made binding. It replaces three of the nine proposed candidates with their general form: derive proposed `utils -> commands` (113 reverse edges), `utils -> adapters` (8) and `utils -> memory` (5) separately; measurement showed `utils` imports NOTHING from src/ at all, so one rule covers all three and any subsystem added later. Enforcing the general form matters because `utils` is the most fanned-in directory in the repo. A single upward import from it creates a cycle through most of the tree. Mutation-proved rather than eyeballed: green on the repo as it stands, and injecting `import "../adr.js"` into src/utils/colors.ts produces `✗ src/utils/colors.ts:1 ... (ADR-0006)` with exit 1, green again on revert. Two candidates deliberately NOT accepted, with the reason recorded in the ADR itself: - "No subsystem imports the command layer" is true of all 16 subsystems, but its general form fires on src/commands/adr-derive.test.ts, where a fixture STRING containing `import "../commands/x.js"` is read as a real import by the text-level extractor. `paths` has no negation, so "every subsystem except src/commands/**" cannot be expressed. Left underived rather than encoded wrong -- and worth noting that the tool's narrow per-directory candidates were right where the hand-written consolidation was not. - `profile` -> `exec-broker` (support 8) is a lone pair with no wider pattern. Insufficient evidence it is a decision rather than an ordering accident. Known papercut, recorded not fixed: `adr derive` still proposes the three candidates ADR-0006 now enforces. Suppressing them needs regex subsumption against accepted rules, which is not trivially decidable and not worth guessing at.

The decision belongs in the curated tier, but the provenance was the open question: an agent derived this from measurement and no human stated it, so `operator` -- the default when the field is absent -- would be exactly the mislabelling #550 was about. Written as `derived`. That has a useful second effect. The caveat left standing after #551 was that kit's own .kit/shared measured identically to before the fix: 46 entries, 41 without provenance, 0 aging, 0 stale, because operator and legacy entries never age. The corpus had nothing to classify. Measured now: entries=48 provenance={"(unset)":42, "operator":5, "derived":1} today : aging=0 stale=0 +200 days : aging=1 -> 0ed015 (derived) So the aging model has its first classifiable entry, and the 42 legacy ones stay exempt as designed. The mechanism was proven in isolation before; this is the first time the repo's own store exercises it.

…n a missing tool Provisioning the standards gates turned up more than the gates themselves. **The eslint findings were not where I said they were.** I had reported 174 findings "all in examples/", based on the ten sample lines kit happened to print. Measured properly: 174 = 6 errors + 168 warnings, and by directory it is src=161, examples=10, packages=3. The six errors are now fixed: - two scanners (snyk, sentrux) rethrew a JSON parse failure without attaching `cause`, discarding the only thing that says WHERE the output went wrong. In a scanner that turns "cannot verify" into an opaque failure, so both now pass `{ cause: err }`. - a `require()` in an ESM example, now a `node:crypto` import - three unused imports/bindings Two more warnings were dead `eslint-disable` directives for `no-control-regex` that suppressed nothing -- a suppression that suppresses nothing is a lie in the code for the next reader. Removed. 174 -> 166, and 0 errors. The remaining 166 are structural: complexity=71, no-explicit-any=32, max-lines-per-function=27, max-depth=18, max-lines=11, max-params=6. That is the same debt lizard and scc report, and refactoring it is a program rather than a task. It is in the frozen baseline, which is what makes the gate usable at all. **And a test whose green depended on the machine.** `collectStandardsKeys` "returns empty slices when the general tools are absent" asserted against the real cwd with the comment "lizard/jscpd/scc are not installed in this environment". Installing lizard and jscpd turned it red with no code change. A test that passes because the box happens to be missing something proves nothing on a box that has it. It now empties PATH for the duration of the call, so the absent-tools condition is forced rather than hoped for, and it passes on both kinds of machine. Baseline: .kit-baseline.json now carries 833 frozen standards findings from the first run of complexity/duplication/size. It includes the six eslint errors fixed above -- a frozen finding that no longer occurs simply never matches -- and the 49 scc findings, so the file size gate does not report 49 "new" findings the day scc is provisioned somewhere.

…d draw the model-rig boundary Three fixes for three gaps between what kit claims and what kit enforces. **1. `kit check` now states its scope next to its verdict.** scope: 42 check(s) — 41 inspect the code, 1 executes it. A pass here does not cover runtime behaviour; `kit broker` is that tier. Counted from the checks that ran, not asserted, so it cannot go stale as categories are added. The reason it matters is measured, not stylistic: multi-tier verification research (arXiv:2607.00107, 8,918 programs across four tiers) found AI-generated code ~2x as likely as human code to trigger a confirmed runtime violation -- while under STATIC analysis the two appear equally safe, a similarity the authors call misleading. kit's surface is 41/42 static, so on a weak model its green says roughly what it would have said about human-written code. That is not a wrong verdict, it is an uninformative one wearing the same colour as an informative one. Now it says so. Sibling to the existing partial-run line, for the same reason. **2. ADR-0002's dependency floor is enforced for the first time.** Found by breaking it: `kit pkg npm:jscpd` added a fifth runtime dependency and `kit adr check` reported "4 enforced ADR(s) — no new violations". The kit-enforce block is a deny-list of twelve named packages at the import level; the ADR's title says "four runtime deps". It declared more than it enforced. It is not expressible in the grammar, for two structural reasons now recorded in the ADR: package.json is not in the file set the ADR gate walks (CODE_EXTS is source extensions only), and forbid/require_pattern match LINE BY LINE, so a `dependencies` entry cannot be told apart from a `devDependencies` one. Widening the walk and adding block-aware matching to serve one rule is a bigger change than the rule is worth, so the floor is enforced by src/dependency-floor.test.ts -- exact dependency set, exact versions, no ranges. The kit-enforce block stays as the cheaper first line. **3. ADR-0007 draws the line for the model + kit residual-risk rig.** kit owns the frozen input, the ingest schema, the deterministic adjudication and the receipts; the models run outside and the repository decides whether a finding is real. kit never produces, ranks or grades a judgement. ADR-0001 is untouched -- its forbid_import over src/** is the mechanical half, which is why 0007 carries no rule of its own rather than duplicating one that could drift. Three limits are written into the consequences so they ship with any number: lower bound only, Goodhart, and unproven transfer from C++. Everything mutation-proved rather than eyeballed: a fifth dependency and a floated version each break exactly the assertion that should notice; treating every category as executing breaks three tier-notice tests; dropping the `kit broker` pointer breaks exactly the one about naming the remedy.

…ot hold it Future-proofing for a pattern that showed up three times this week rather than once. Three limits of the `kit-enforce` grammar are now measured, each found by trying to write a rule and failing: - `paths` has no negation, so "every subsystem except src/commands/**" is inexpressible - `forbid_pattern`/`require_pattern` match LINE BY LINE, so a JSON block cannot be pinned and package.json is not in the walked file set anyway - the import extractor is text-level, so a test fixture STRING that looks like an import counts as one Today, when a rule hits one of those, nothing happens. The decision quietly does not get encoded and the ADR goes on declaring more than it enforces. ADR-0002 sat in exactly that state for months: titled "four runtime deps", enforcing "not these twelve imports". The gap was invisible because a narrowly-enforced ADR and a fully-enforced one print identically. So an ADR can now name its real enforcement point: enforced_by: [src/dependency-floor.test.ts] `adr list` prints it under the ADR, so "documented, not enforced" is never confused with "not enforced anywhere". And `adr check` FAILS when a named file does not exist -- which is the half that matters. A pointer nobody verifies is worse than no pointer, because it reads as coverage. Same rule class as ci-adr-gate.test.ts (a gate must be invoked) and self-audit (a workflow's script must exist), applied to enforcement claims. Checked only for `accepted` ADRs: a proposal's pointer is an intention, not yet a claim. Mutation-proved end to end: adding a second, non-existent pointer to ADR-0002 fails `adr check` with the file cited; deleting the real dependency-floor test fails it too; turning the accepted-status guard off, or on for every status, each breaks exactly the one test that should notice. The three grammar limits are now written into CLAUDE.md and AGENTS.md alongside the mechanism, so the next person who cannot express a rule reaches for this instead of dropping it silently.

…-sync-p6fg9a # Conflicts: # .kit/shared/memory.jsonl

github-actions · 2026-08-30T16:25:02Z

✅ Docker image built successfully

CLI image: docker.io/sandstream/kit:test

`npm run format:check` failed on src/adr.ts, src/commands/usage.ts and src/commands/usage.test.ts. Formatting only — no behaviour change. Two causes, both mine: `eslint --fix` left trailing whitespace where it removed the dead `no-control-regex` disable comments, and the `list()` helper added to adr.ts had lines past the print width. The real miss is the process one. I ran build, tsc, eslint, the full suite and `review --stages adr,skill` before pushing, and never ran `format:check` — which is a step CI runs and a contributor runs locally. Reproduced the failure first, then confirmed the same command passes ("All matched files use Prettier code style!"), rather than assuming --write was sufficient.

github-actions · 2026-08-30T16:31:14Z

✅ Docker image built successfully

CLI image: docker.io/sandstream/kit:test

claude added 6 commits August 30, 2026 06:58

Merge remote-tracking branch 'origin/main' into claude/web-kit-status…

0a2069e

…-sync-p6fg9a # Conflicts: # .kit/shared/memory.jsonl

sandstream merged commit 579cc38 into main Aug 30, 2026
19 checks passed

sandstream deleted the claude/web-kit-status-sync-p6fg9a branch August 30, 2026 19:28

Navigation Menu

Uh oh!

feat: close five gaps between what kit claims and what kit enforces - #556

feat: close five gaps between what kit claims and what kit enforces#556
sandstream merged 7 commits into
mainfrom
claude/web-kit-status-sync-p6fg9a

sandstream commented Aug 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sandstream commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Changes Made

Testing

Test Coverage

Manual Testing

Breaking Changes

Checklist

Performance Impact

Reviewer Notes

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sandstream commented Aug 30, 2026 •

edited

Loading