fix(security): close a substitution body on shell grammar, not a bare paren - #8579
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS A measured bypass closed in the one shared span helper, fail-closed asymmetry stated, unreachable machinery deliberately cut — the right shape per this module's own history. The fix lands at the structural layer AGENTS.md prescribes (grammar in the single walk, not a spelling table entry), the arm-generous/disarm-strict asymmetry always errs toward over-scanning, which every extractor call site tolerates, and the residual gaps (the [DESIGN-REVIEWED] 7169c95 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of Verified. First-Principles-Verdict: CONCERNS The comment/ What this change shipsIntent: stop a
WatchItem 4 is a rider on a fix. The author states plainly it is "HARDENING, not a patched hole… no payload this module refuses was reachable through the pairwise version." Its stated harm is anti-drift symmetry with the paren closer — an inherited (consistency) justification, not the #8150 defect, which items 1–3 close on their own. Not a BLOCK: confirming the zero-cost claim rests on bash backtick-nesting semantics (judgement), so the tie-breaker holds. Subtractions
[FIRST-PRINCIPLES-REVIEWED] 7169c95 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @chenmingwei23 overrides the GPT 5.6 finding for This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
be12c61 to
5691f05
Compare
5691f05 to
177196d
Compare
177196d to
45609c1
Compare
45609c1 to
2fd5958
Compare
739ead8 to
df60a4b
Compare
GPT F1 (line-continuated
|
| spelling | body before | body after |
|---|---|---|
$(ca\+nl+se x in x) printf <verb>;; esac) |
ca\+nl+se x in x |
ca\+nl+se x in x) printf <verb>;; esac |
_opens_comment reads the character before a # the same way, so a\ + newline + #b stays the single word a#b rather than opening a comment.
The residual, and why it is not this PR's
The verdict on that spelling still does not flip -- and the cause is not this walk. _self_tokens reads the backslash-newline as a command separator instead of folding it away, which severs the assignment from the invocation so $T never resolves:
base branch, continuated form:
tokens : ['t=$(ca', ';', 'se', 'x', 'in', 'x)', 'printf', '<verb>;;', 'esac);', 'kirocrew', '$t']
verdict : None
this branch, continuated form:
tokens : ['t=$(ca', ';', 'se', 'x', 'in', 'x)', 'printf', '<verb>;;', 'esac);', 'kirocrew', '$t']
verdict : None
Note the ';' token where the continuation was, on both trees. The same command is allowed on the base branch with a byte-identical token split, so this change neither introduces nor worsens it -- measured zero delta. What this PR did move, on the same tree:
plain `case` form: base = None (allowed) -> this branch = DENIED
That is a tokenizer-level continuation bug sitting upstream of the span helper. It is filed separately rather than folded in, because _self_tokens feeds every argv check and widening it does not belong in a span fix -- the same reason the heredoc and ${ } rows were left out of this PR.
The test for the continuated spelling therefore asserts the recovered body only, and its docstring says why the verdict does not move, so nothing here is recorded as fixed that is not.
Note for the Design and First Principles lanes
Both BLOCK verdicts on f6af66f99 read the PR description from the run set that started before the rewritten description landed, and they object to claims the current description no longer makes -- no _substitution_closer, no frame stack, no heredoc queue, and an explicit section listing what is deliberately not modelled. Design Review's own note says the shipped code is "sound and honestly documented in its own docstrings ... once description and diff agree this is mergeable in shape". They should settle on this head.
… paren
The span walk that decides where a substitution body ends was already
quote-aware, but two COMMAND-GRAMMAR constructs put a literal ")" in front
of it without quoting one, and each hid a payload this module still refuses:
* a "#" comment runs to the end of its line, so the ")" in "$(: # )" is
commented out and the body closes on a LATER line. Counting it ended the
body at ": # ", losing the printf behind it that computes the
credential-minting verb.
* a "case" PATTERN is terminated by ")". In
"$(case x in x) printf <verb>;; esac)" that ")" is the pattern's, not the
substitution's, and reading it as one truncated the body to "case x in x".
Both were measured on main in both directions: is_denied returned None while
bash assembled and ran the refused payload. Two of the three anchors now flip
to denied; the third recovers its body but keeps its verdict, because the
self-kill pass never attributes a substitution sitting in an assignment ahead
of the kill -- a separate gap this change neither causes nor closes.
Taught to _matching_close_paren rather than to a new private walk. It is THE
one span computation and all three of its call sites are extractors, where a
span reaching too far only feeds them more text to inspect, so fixing it once
covers the argv-floor extractors too and leaves no second reading to drift.
Reserved-word recognition folds backslash-newline continuations, because the
shell removes them before it reads words at all: "ca\" + newline + "se" IS the
reserved word "case", and bash was measured running it as one. Byte-literal
matching missed that spelling, so the rule never armed and the pattern ")"
closed the body early again -- the same bypass for the cost of two characters.
The character before a "#" is read the same way, and the folds are stepped OVER
before that character is tested -- what matters is what the fold leaves
adjacent, not that a fold is there. "a\" + newline + "#b" folds to the single
word "a#b" and opens nothing, while ": " + "\" + newline + "#" folds to ": #",
where a word break ends up in front of the "#" and bash opens a real comment.
Treating any preceding fold as "not a comment" got that second case wrong in
the fail-OPEN direction: the comment was missed, so the walk read the ")" it
hides as the closer and truncated the body before the payload, reopening this
change's own bypass for the folded spelling (found in review).
That spelling recovers its body but not its verdict, and not because of this
walk: _self_tokens reads the backslash-newline as a command SEPARATOR instead
of folding it, severing the assignment from the invocation so "$T" never
resolves. Measured on the base branch with the identical token split, so this
change neither causes nor worsens it. Tracked separately rather than folded in,
because that tokenizer feeds every argv check.
The BACKTICK closer now reads the same state machine, which is hardening
rather than a patched hole: the strings the old pairwise find mis-read are
ones bash itself rejects, since backticks do not nest unescaped. It is fixed
so the two spellings of one closer cannot drift apart, which is how the paren
half broke before.
Refs #8150
Opus F1 (
|
|
/ai-review override gpt 7169c95: Provider refused this head twice, so the shown blocking verdict is pinned to f6af66f and names the line-continuated case finding, which was fixed in c1363b7 (_word_at folds continuations) and is covered by two tests; Opus 4.8 reports no findings on this head and Design, First Principles and PR Hygiene are green. |
Human judgment recorded@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Problem / Motivation
_substitution_bodiesis the shared answer to "what text does this command run as a shell". Every argv check reads substitution bodies through it, so a body that stops early is not one pass's problem -- every consumer inherits the blindness.Its span walk was already quote-aware:
_matching_close_parenreads the module's one quote/escape state machine, so a)the shell treats as a literal character no longer closes a body. But quoting is not the only way an ordinary)reaches that walk. Two command-grammar constructs put one there without quoting it, and the walk counted both:#comment runs to the end of its line, so the)in$(: # )is commented out and the substitution closes on a later line;casepattern is terminated by), so in$(case x in x) printf <verb>;; esac)that)belongs to the pattern, not to the substitution.Why it matters
Both were measured on
mainin both directions -- the gate allowed the command and bash assembled and ran the refused payload:is_deniedon mainT=$(: # )+ newline +printf <verb>); <name> $T: #None(allowed)T=$(case x in x) printf <verb>;; esac); <name> $Tcase x in xNone(allowed)P=$(case x in x) pgrep -f <name>;; esac); kill $Pcase x in xNone(allowed)In each case the truncation dropped the half that computes the payload, so the value assembled from it was never recognised. The first two hid the credential-minting verb; the third hid a lookup of our own processes.
What changed (motivation -> approach -> change)
The fix is taught to
_matching_close_parenrather than to a new private walk. That helper is the one span computation, and all three of its call sites are extractors that take the whole remainder when a span is unproven -- so a span reaching too far only ever feeds them more text to inspect. Fixing it once therefore covers the argv-floor extractors this PR never touches, and leaves no second reading of shell state to drift from the first. This module's history is the argument: the same defect was cured three times already (two tokenizers, then two paren counters, then four separate readings of quoting), and each cure was structural -- one machine, several consumers.Concretely:
_matching_close_parennow skips a#comment to the end of its line, and ignores both parens between acaseand itsesac. Arming is generous and disarming is strict on purpose: missing a realcasecloses a body early, which is the bypass, while missing a realesaconly runs it long, which is imprecision. Socasearms on any standalone word andesacdisarms only in command position._word_at(solowercase)does not arm the pattern rule anda#bopens no comment) and_in_command_position(so a backslash-newline reads as a line continuation, leaving anesacpassed as an argument inert)._matching_close_backtick. This is hardening, not a patched hole: the strings the old pairwisefindmis-read are ones bash itself rejects, because backticks do not nest unescaped. It is fixed so the two spellings of one closer cannot drift apart, which is how the paren half broke in the first place.ca\+ newline +seis the reserved wordcase, and bash was measured running it as one. Byte-literal matching missed that spelling, so the rule never armed and the pattern's)closed the body early again -- the same bypass for the cost of two characters._opens_commentreads the character before a#the same way, soa\+ newline +#bstays the single worda#b.What this deliberately does not claim
_self_tokensreads the backslash-newline as a command SEPARATOR instead of folding it away, which severs the assignment from the invocation so$Tnever resolves. That is a tokenizer-level continuation bug sitting upstream of this helper. Measured on the base branch: the same command is allowed there with the identical token split ('t=$(ca', ';', 'se', ...), so this change neither causes nor worsens it -- zero delta. Filed separately rather than folded in, because the tokenizer feeds every argv check and widening it does not belong in a span fix.kill. Separate gap, same reasoning.git pushbehaviour is unchanged. The publish floor already fail-closes every substitution-wrapped push viagit-publish-target-unverifiable, so a body-walk fix cannot move it.${ }/$(( ))regions are not modelled here. An earlier draft of this PR did model them. They are left out because no payload this module refuses was reachable through either: where the walk's early close at a heredoc-data)fires, bash loses the payload in the same direction, so there is no divergence to exploit. Shipping unreachable machinery into a security parser costs review surface and buys nothing; if a reachable case is found, it is additive.Tests
TestSubstitutionCloserReadsCommandGrammarintest/test_security.py, 25 cases:ca\+ newline +searms the pattern rule,es\+ newline +acends it, anda\+ newline +#bopens no comment;a#bmid-word,esacas an argument,lowercase, a(a|b)pattern alternation, and the benign twins of both anchors;"cmd"runscmd;$(case x in x,$(: #,`A=', ...) that must yield a list rather than raise;Manual verification
test/test_security.py,test/test_push_branch_gate.py,test/test_denied_commands_security.py,test/test_security_facade.py; zero failures.mainby running the identical inputs through both source trees: main allows all three anchors with truncated bodies, this branch denies two and recovers the third's body. The 12 benign controls are byte-identical across both trees.casespellings. No real credential read, kill or push was executed.black,isort,flake8clean on both changed files;mypyreports zero errors in them.Related Issues
Refs #8150
Pattern harvest
Rule candidate: when a span helper is wrong, ask first whether it is the span helper -- if it is, and every one of its consumers fails closed on an unproven span, teaching it beats adding a sibling that will drift out of agreement with it.
A second reading of shell state is the recurring defect in this module, and the cure has been structural every time: two tokenizers, then two paren counters, then four separate readings of quoting. This PR is the same shape one level up -- the reachable bypasses were closed by teaching the shared walk, not by giving
_substitution_bodiesa private one.Two corollaries earned in review. The draft that modelled heredocs and arithmetic looked stronger but could not be tied to a single reachable payload, and unreachable complexity in a parser that decides refusals is a cost, not a safety margin. And a reserved word is not a byte string: the shell folds line continuations before it reads words, so any recognition that matches literally has a two-character bypass in it -- worth checking every other reserved-word match in this module against that.