ctrlrun scan: the command that says what is not covered - #100
Merged
Merged
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
There is no command today that answers "is CTRLRun on this path at all?". verify answers whether the guarantee holds in this setup; the gateway covers what reaches it; the decorator covers what somebody decorated. The call nobody decorated is invisible, and that is the gap between installed and in the path. docs/SPEC-scan.md is the contract, written before any of it exists, and tests/test_scan.py is §8 in executable form: 15 tests, all red. Not a kernel milestone — no table, no column, no event, no error, no policy key — and no version line of its own, on the argument SPEC-mcp-operator.md already makes for a subcommand. Three things the design turns on: - It never imports the tree it reads. ast.parse and nothing else. A tool whose job is to find calls that move money must not be the thing that makes one, and T194 is that assertion. - It adds no entry point, and §9.2 says so as a rule rather than as a fact about the first implementation. The tempting version builds an Action for each call site and asks the policy what would happen to it, which is a principal invented by a tool from a source file — the fourth costume of --principal-from-client-info. T205 breaks every entry point and requires the scan to finish. - A clean scan is not a verdict, and the report has to say so. §4 enumerates what it misses by construction and §4.5 requires the sentence in every run, including the run with no findings. No score, no percentage, no badge: a number that improves when the vocabulary is shortened is a number that will be. T200 caught the harness lying at me while I wrote it. It asserted exit 2 for an empty tree and passed against a tree with no scan command at all, because click answers an unknown subcommand with exit 2. It now requires the message §7 asks for, and §7 records why that message is a requirement. Suppression is the one place this could have become a flag that relaxes a check. It does not: a suppressed finding stays in the report with its reason and in the totals, the reason is mandatory, and the exit code does not move.
The specification landed first and its seventeen tests were red. This is the implementation, and five sections of SPEC-scan.md now carry a paragraph beginning "Found by" — each a defect in the design that writing the tests or running the command exposed, corrected in the document rather than worked around in the code. - §3.2 The plural rule. One matching rule could not separate stripe.refunds.create from refunds_report; both contain "refunds". A whole segment matches a verb or its plural, a word inside a compound matches the singular only. - §3.2 `execute` is out of the vocabulary. cursor.execute appears in every project that touches a database: 90 of 208 findings against this repository's own src/. - §3.4 A policy action whose decorator passes effect= is not a missing effect. The policy's template is what the gateway reads; a decorated call site carries its own. The first version flagged examples/double-refund, which would have taught a reader to add a key they already had. - §4.1 A call on an expression is matched, not filed as undetermined. Treating every unresolved receiver as unknown produced 216 entries, of which two were the dynamic dispatch the category exists for. That list is now 10. - §9.1 `undetermined` was listed as a FindingKind while §4.1 said in the same document that these are not findings. It is its own list. The command is a subcommand like the operator server: no table, no column, no event, no error type, no policy key, no version line of its own, and no entry point at all — §9.2 states that as a rule, and T205 breaks Control and the store and requires the scan to finish. It is not a v0.6 feature; test_T177c holds that milestone's command list frozen and names what came after separately. Mutation table, each mutant killed by the test named for it: §3.3 context is not coverage → T196 (and six more); §3.6 a reason is required → T201; §7 an unparseable file is a finding → T199, T204; §3.4 the decorator's effect → T207; §4.1 a call on an expression → T208; §4.5 the limits sentence → T203; §3.2 the plural rule → T197. Full suite 4065 passed, 45 skipped. mypy --strict, ruff, ruff format, the docs lint, links and the CLI reference render all clean.
arpanghoshal
force-pushed
the
spec/scan
branch
from
September 6, 2026 20:09
6c07117 to
bf0d6d7
Compare
arpanghoshal
marked this pull request as ready for review
September 6, 2026 20:09
arpanghoshal
enabled auto-merge (squash)
September 6, 2026 20:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was no command that answered is CTRLRun on this path at all?
ctrlrun verifyanswerswhether the guarantee holds in a setup. The gateway covers what reaches it. The decorator covers
what somebody decorated. The call nobody decorated was invisible.
ctrlrun scanreads a Python tree and a policy document as text and reports the consequentialcall sites and policy entries that are not covered.
How this was built
docs/SPEC-scan.mdlanded first and its tests were red before any of it existed. Five sectionsof the specification now carry a paragraph beginning Found by — each a defect in the design
that writing the tests or running the command exposed, corrected in the document rather than
worked around in the code:
stripe.refunds.createfromrefunds_report; both contain refundsexecutein the vocabulary —cursor.executewas 90 of 208 findings against this repo's ownsrc/effect=reported as missing one; it flaggedexamples/double-refund, teaching a reader to add a key they already hadundeterminedlisted as aFindingKindwhile §4.1 said in the same document that these are not findingsThe three rules it turns on
ast.parseand nothing else. T194 is a fixturewhose import writes a sentinel.
implementation. The tempting version builds an
Actionper call site and asks the policy whatwould happen — a principal invented by a tool from a source file. T205 patches
Controland thestore to raise on any use and requires the scan to finish.
with no findings. No score, no percentage, no badge: a number that improves when the vocabulary
is shortened is a number that will be.
Suppression is the one place this could have become a flag that relaxes a check. It does not: the
reason is mandatory, a suppressed finding stays in the report and in the totals, and the exit code
does not move.
Mutation table
Each mutant killed by the test named for it.
contextis not coveragewithas coveringT196+ 6and reasonT201T199,T204T207T208T203T197Scope
Not a v0.6 feature. A subcommand like the operator server: no table, no column, no event type, no
error type, no policy key, no version line of its own.
test_T177cholds v0.6's command listfrozen and names what came after separately, which is where
scansits — soSPEC-v0.6.md§9.4'sclaim that the milestone grew no command stays literally true.
No authorization, identity, delegation or gateway code is touched, so the independent-review rule
does not apply; the reviewer's question here is the design, and the Found by paragraphs are the
record of what has already changed under it.
Green: full suite 4065 passed / 45 skipped ·
mypy --strict·ruff·ruff format· docslint 185 documents 0 findings · links 179 documents 0 broken · CLI reference regenerated.
Follow-up, deliberately not here: no docs-site guide page for
scanyet — the CLI referenceand the exit-codes page carry it, and a guide wants a real tree to scan rather than a fixture.
🤖 Generated with Claude Code