Skip to content

ctrlrun scan: the command that says what is not covered - #100

Merged
arpanghoshal merged 3 commits into
mainfrom
spec/scan
Sep 6, 2026
Merged

arpanghoshal merged 3 commits into
mainfrom
spec/scan

Conversation

@arpanghoshal

@arpanghoshal arpanghoshal commented Sep 6, 2026

Copy link
Copy Markdown
Member

There was no command that answered is CTRLRun on this path at all? ctrlrun verify answers
whether the guarantee holds in a setup. The gateway covers what reaches it. The decorator covers
what somebody decorated. The call nobody decorated was invisible.

ctrlrun scan reads a Python tree and a policy document as text and reports the consequential
call sites and policy entries that are not covered.

$ ctrlrun scan --path src/ctrlrun
unprotected_call (77)
  ...
undetermined (10)
  ...
files read: 44   excluded: 0   findings: 77   suppressed: 0   undetermined: 10
vocabulary: 25 verbs   policy: not read — the two policy checks did not run

This is a finder, not a proof. It read 44 files and could not determine 10 calls;
it did not read your dependencies, your other services, or any tool server behind a gateway.
A clean scan means nothing was found where it looked.

How this was built

docs/SPEC-scan.md landed first and its tests were red before any of it existed. Five sections
of the specification now carry a paragraph beginning Found by
— each a defect in the design
that writing the tests or running the command exposed, corrected in the document rather than
worked around in the code:

§ What was wrong How it was found
§3.2 one rule could not separate stripe.refunds.create from refunds_report; both contain refunds writing T197
§3.2 execute in the vocabulary — cursor.execute was 90 of 208 findings against this repo's own src/ running it
§3.4 a policy action whose decorator passes effect= reported as missing one; it flagged examples/double-refund, teaching a reader to add a key they already had running it
§4.1 every unresolved receiver filed as undetermined: 216 entries, of which two were the dynamic dispatch the category exists for. Now 10 running it
§9.1 undetermined listed as a FindingKind while §4.1 said in the same document that these are not findings writing T198 and T204

The three rules it turns on

  • It never imports the tree it reads (§2.1). ast.parse and nothing else. T194 is a fixture
    whose import writes a sentinel.
  • It adds no entry point (§9.2), stated as a rule and not as a fact about this
    implementation. The tempting version builds an Action per call site and asks the policy what
    would happen — a principal invented by a tool from a source file. T205 patches Control and the
    store to raise on any use and requires the scan to finish.
  • A clean scan is not a verdict (§4), and the report says so on every run, including the run
    with no findings. No score, no percentage, no badge: a number that improves when the vocabulary
    is shortened is a number that will be.

Suppression is the one place this could have become a flag that relaxes a check. It does not: the
reason is mandatory, a suppressed finding stays in the report and in the totals, and the exit code
does not move.

Mutation table

Each mutant killed by the test named for it.

Guard Mutation Killed by
§3.3 context is not coverage treat with as covering T196 + 6
§3.6 a suppression needs a reason drop and reason T201
§7 an unparseable file is a finding skip the file T199, T204
§3.4 the decorator's own effect drop the exemption T207
§4.1 a call on an expression file it as undetermined T208
§4.5 the limits sentence omit it T203
§3.2 the plural rule allow plurals in compounds T197

Scope

Not a v0.6 feature. A subcommand like the operator server: no table, no column, no event type, no
error type, no policy key, no version line of its own. test_T177c holds v0.6's command list
frozen and names what came after separately, which is where scan sits — so SPEC-v0.6.md §9.4's
claim that the milestone grew no command stays literally true.

No authorization, identity, delegation or gateway code is touched, so the independent-review rule
does not apply; the reviewer's question here is the design, and the Found by paragraphs are the
record of what has already changed under it.

Green: full suite 4065 passed / 45 skipped · mypy --strict · ruff · ruff format · docs
lint 185 documents 0 findings · links 179 documents 0 broken · CLI reference regenerated.

Follow-up, deliberately not here: no docs-site guide page for scan yet — the CLI reference
and the exit-codes page carry it, and a guide wants a real tree to scan rather than a fixture.

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
ctrlrun 🟢 Ready View Preview Sep 6, 2026, 7:31 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

There is no command today that answers "is CTRLRun on this path at all?".
verify answers whether the guarantee holds in this setup; the gateway covers
what reaches it; the decorator covers what somebody decorated. The call nobody
decorated is invisible, and that is the gap between installed and in the path.

docs/SPEC-scan.md is the contract, written before any of it exists, and
tests/test_scan.py is §8 in executable form: 15 tests, all red. Not a kernel
milestone — no table, no column, no event, no error, no policy key — and no
version line of its own, on the argument SPEC-mcp-operator.md already makes for
a subcommand.

Three things the design turns on:

- It never imports the tree it reads. ast.parse and nothing else. A tool whose
  job is to find calls that move money must not be the thing that makes one, and
  T194 is that assertion.
- It adds no entry point, and §9.2 says so as a rule rather than as a fact about
  the first implementation. The tempting version builds an Action for each call
  site and asks the policy what would happen to it, which is a principal
  invented by a tool from a source file — the fourth costume of
  --principal-from-client-info. T205 breaks every entry point and requires the
  scan to finish.
- A clean scan is not a verdict, and the report has to say so. §4 enumerates
  what it misses by construction and §4.5 requires the sentence in every run,
  including the run with no findings. No score, no percentage, no badge: a
  number that improves when the vocabulary is shortened is a number that will
  be.

T200 caught the harness lying at me while I wrote it. It asserted exit 2 for an
empty tree and passed against a tree with no scan command at all, because click
answers an unknown subcommand with exit 2. It now requires the message §7 asks
for, and §7 records why that message is a requirement.

Suppression is the one place this could have become a flag that relaxes a
check. It does not: a suppressed finding stays in the report with its reason and
in the totals, the reason is mandatory, and the exit code does not move.
The specification landed first and its seventeen tests were red. This is the
implementation, and five sections of SPEC-scan.md now carry a paragraph
beginning "Found by" — each a defect in the design that writing the tests or
running the command exposed, corrected in the document rather than worked
around in the code.

- §3.2 The plural rule. One matching rule could not separate stripe.refunds.create
  from refunds_report; both contain "refunds". A whole segment matches a verb or
  its plural, a word inside a compound matches the singular only.
- §3.2 `execute` is out of the vocabulary. cursor.execute appears in every project
  that touches a database: 90 of 208 findings against this repository's own src/.
- §3.4 A policy action whose decorator passes effect= is not a missing effect.
  The policy's template is what the gateway reads; a decorated call site carries
  its own. The first version flagged examples/double-refund, which would have
  taught a reader to add a key they already had.
- §4.1 A call on an expression is matched, not filed as undetermined. Treating
  every unresolved receiver as unknown produced 216 entries, of which two were
  the dynamic dispatch the category exists for. That list is now 10.
- §9.1 `undetermined` was listed as a FindingKind while §4.1 said in the same
  document that these are not findings. It is its own list.

The command is a subcommand like the operator server: no table, no column, no
event, no error type, no policy key, no version line of its own, and no entry
point at all — §9.2 states that as a rule, and T205 breaks Control and the store
and requires the scan to finish. It is not a v0.6 feature; test_T177c holds that
milestone's command list frozen and names what came after separately.

Mutation table, each mutant killed by the test named for it: §3.3 context is not
coverage → T196 (and six more); §3.6 a reason is required → T201; §7 an
unparseable file is a finding → T199, T204; §3.4 the decorator's effect → T207;
§4.1 a call on an expression → T208; §4.5 the limits sentence → T203; §3.2 the
plural rule → T197.

Full suite 4065 passed, 45 skipped. mypy --strict, ruff, ruff format, the docs
lint, links and the CLI reference render all clean.
@arpanghoshal
arpanghoshal marked this pull request as ready for review September 6, 2026 20:09
@arpanghoshal
arpanghoshal enabled auto-merge (squash) September 6, 2026 20:09
@arpanghoshal arpanghoshal changed the title SPEC-scan: the command that says what is not covered, and its red suite ctrlrun scan: the command that says what is not covered Sep 6, 2026
@arpanghoshal
arpanghoshal merged commit adb079c into main Sep 6, 2026
11 checks passed
@arpanghoshal
arpanghoshal deleted the spec/scan branch September 6, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant