Skip to content

fix(security): close a substitution body on shell grammar, not a bare paren - #8579

Merged
bolichen97 merged 1 commit into
mainfrom
fix/substitution-bodies-quote-aware-8150
Sep 8, 2026
Merged

fix(security): close a substitution body on shell grammar, not a bare paren#8579
bolichen97 merged 1 commit into
mainfrom
fix/substitution-bodies-quote-aware-8150

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

_substitution_bodies is the shared answer to "what text does this command run as a shell". Every argv check reads substitution bodies through it, so a body that stops early is not one pass's problem -- every consumer inherits the blindness.

Its span walk was already quote-aware: _matching_close_paren reads the module's one quote/escape state machine, so a ) the shell treats as a literal character no longer closes a body. But quoting is not the only way an ordinary ) reaches that walk. Two command-grammar constructs put one there without quoting it, and the walk counted both:

  • a # comment runs to the end of its line, so the ) in $(: # ) is commented out and the substitution closes on a later line;
  • a case pattern is terminated by ), so in $(case x in x) printf <verb>;; esac) that ) belongs to the pattern, not to the substitution.

Why it matters

Both were measured on main in both directions -- the gate allowed the command and bash assembled and ran the refused payload:

spelling body extracted on main is_denied on main
T=$(: # ) + newline + printf <verb>); <name> $T : # None (allowed)
T=$(case x in x) printf <verb>;; esac); <name> $T case x in x None (allowed)
P=$(case x in x) pgrep -f <name>;; esac); kill $P case x in x None (allowed)

In each case the truncation dropped the half that computes the payload, so the value assembled from it was never recognised. The first two hid the credential-minting verb; the third hid a lookup of our own processes.

What changed (motivation -> approach -> change)

The fix is taught to _matching_close_paren rather than to a new private walk. That helper is the one span computation, and all three of its call sites are extractors that take the whole remainder when a span is unproven -- so a span reaching too far only ever feeds them more text to inspect. Fixing it once therefore covers the argv-floor extractors this PR never touches, and leaves no second reading of shell state to drift from the first. This module's history is the argument: the same defect was cured three times already (two tokenizers, then two paren counters, then four separate readings of quoting), and each cure was structural -- one machine, several consumers.

Concretely:

  • _matching_close_paren now skips a # comment to the end of its line, and ignores both parens between a case and its esac. Arming is generous and disarming is strict on purpose: missing a real case closes a body early, which is the bypass, while missing a real esac only runs it long, which is imprecision. So case arms on any standalone word and esac disarms only in command position.
  • two small helpers carry the word rules: _word_at (so lowercase) does not arm the pattern rule and a#b opens no comment) and _in_command_position (so a backslash-newline reads as a line continuation, leaving an esac passed as an argument inert).
  • the backtick closer now reads the same state machine, via _matching_close_backtick. This is hardening, not a patched hole: the strings the old pairwise find mis-read are ones bash itself rejects, because backticks do not nest unescaped. It is fixed so the two spellings of one closer cannot drift apart, which is how the paren half broke in the first place.
  • reserved-word recognition folds backslash-newline continuations, because the shell removes them before it reads words at all: ca\ + newline + se is the reserved word case, and bash was measured running it as one. Byte-literal matching missed that spelling, so the rule never armed and the pattern's ) closed the body early again -- the same bypass for the cost of two characters. _opens_comment reads the character before a # the same way, so a\ + newline + #b stays the single word a#b.

What this deliberately does not claim

  • The line-continuated spelling recovers its body but its verdict does not flip, and not because of this walk: _self_tokens reads the backslash-newline as a command SEPARATOR instead of folding it away, which severs the assignment from the invocation so $T never resolves. That is a tokenizer-level continuation bug sitting upstream of this helper. Measured on the base branch: the same command is allowed there with the identical token split ('t=$(ca', ';', 'se', ...), so this change neither causes nor worsens it -- zero delta. Filed separately rather than folded in, because the tokenizer feeds every argv check and widening it does not belong in a span fix.
  • The self-kill anchor likewise recovers its body but keeps its allowed verdict, because that pass never attributes a substitution sitting in an assignment ahead of the kill. Separate gap, same reasoning.
  • git push behaviour is unchanged. The publish floor already fail-closes every substitution-wrapped push via git-publish-target-unverifiable, so a body-walk fix cannot move it.
  • Heredoc bodies and ${ } / $(( )) regions are not modelled here. An earlier draft of this PR did model them. They are left out because no payload this module refuses was reachable through either: where the walk's early close at a heredoc-data ) fires, bash loses the payload in the same direction, so there is no divergence to exploit. Shipping unreachable machinery into a security parser costs review surface and buys nothing; if a reachable case is found, it is additive.

Tests

TestSubstitutionCloserReadsCommandGrammar in test/test_security.py, 25 cases:

  • the two reachable anchors, asserting both the recovered body and the flipped verdict;
  • the self-kill anchor and the line-continuated anchor, asserting the recovered body only, each documenting in its own docstring why the verdict does not move;
  • the folded spellings: ca\ + newline + se arms the pattern rule, es\ + newline + ac ends it, and a\ + newline + #b opens no comment;
  • six benign shapes that must not start being refused: a#b mid-word, esac as an argument, lowercase, a (a|b) pattern alternation, and the benign twins of both anchors;
  • the backtick pair -- a single-quoted backtick is data, a double-quoted one is still a real closer, because "cmd" runs cmd;
  • ten degenerate inputs ($(case x in x, $(: #, `A=', ...) that must yield a list rather than raise;
  • the fail-closed direction: an unproven span still yields the whole remainder.

Manual verification

  • 1769 passing in test/test_security.py, test/test_push_branch_gate.py, test/test_denied_commands_security.py, test/test_security_facade.py; zero failures.
  • Mutation-checked against unmodified main by running the identical inputs through both source trees: main allows all three anchors with truncated bodies, this branch denies two and recovers the third's body. The 12 benign controls are byte-identical across both trees.
  • bash reachability confirmed with a shim that only echoes its argv and a harmless marker in place of the verb, for both the plain and the line-continuated case spellings. No real credential read, kill or push was executed.
  • black, isort, flake8 clean on both changed files; mypy reports zero errors in them.

Related Issues

Refs #8150

Pattern harvest

Rule candidate: when a span helper is wrong, ask first whether it is the span helper -- if it is, and every one of its consumers fails closed on an unproven span, teaching it beats adding a sibling that will drift out of agreement with it.

A second reading of shell state is the recurring defect in this module, and the cure has been structural every time: two tokenizers, then two paren counters, then four separate readings of quoting. This PR is the same shape one level up -- the reachable bypasses were closed by teaching the shared walk, not by giving _substitution_bodies a private one.

Two corollaries earned in review. The draft that modelled heredocs and arithmetic looked stronger but could not be tied to a single reachable payload, and unreachable complexity in a parser that decides refusals is a cost, not a safety margin. And a reserved word is not a byte string: the shell folds line continuations before it reads words, so any recognition that matches literally has a two-character bypass in it -- worth checking every other reserved-word match in this module against that.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 4, 2026 22:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

A measured bypass closed in the one shared span helper, fail-closed asymmetry stated, unreachable machinery deliberately cut — the right shape per this module's own history.

The fix lands at the structural layer AGENTS.md prescribes (grammar in the single walk, not a spelling table entry), the arm-generous/disarm-strict asymmetry always errs toward over-scanning, which every extractor call site tolerates, and the residual gaps (the _self_tokens continuation split, the self-kill attribution) are measured as zero-delta against base and routed to their own PRs rather than smuggled in. Description and diff match in both directions.

[DESIGN-REVIEWED] 7169c95

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Verified. _substitution_bodies is genuinely shared (consumers at shell_normalizer.py:2275, :2732, argv_floor.py:1307); _matching_close_paren has the three call sites the author claims; the new word-boundary helpers and _matching_close_backtick are module-private (not added to _exports.py). The comment/case fix is a cause-level fix of the shared span helper, derived from #8150. The backtick rewrite is the one item that does not address #8150.

First-Principles-Verdict: CONCERNS

The comment/case span fix earns its place at the shared helper; the backtick rewrite rides along as admitted "hardening" with no measured bypass.

What this change ships

Intent: stop a ) that shell grammar makes ordinary (a # comment or a case pattern) from closing a substitution body early and letting is_denied miss the payload — a FIX (#8150).

  1. # comment inside $(…) no longer closes the body early; the hidden verb is now scanned and denied — justified.
  2. case/esac pattern parens ignored between the words, so the pattern's ) no longer truncates the body — justified.
  3. Backslash-newline folding so ca\+nl+se still arms the case rule (and a\+nl+#b opens no comment) — justified (closes the same bypass for the folded spelling).
  4. Backtick closer rewritten from pairwise find to a quote-aware state-machine walk — rides along.

Watch

Item 4 is a rider on a fix. The author states plainly it is "HARDENING, not a patched hole… no payload this module refuses was reachable through the pairwise version." Its stated harm is anti-drift symmetry with the paren closer — an inherited (consistency) justification, not the #8150 defect, which items 1–3 close on their own. Not a BLOCK: confirming the zero-cost claim rests on bash backtick-nesting semantics (judgement), so the tie-breaker holds.

Subtractions

  • Defer _matching_close_backtick (shell_normalizer.py:1732, 1 consumer: the backtick branch at :1074); keep the existing pairwise find until a reachable backtick bypass is measured, since the author reports none.

[FIRST-PRINCIPLES-REVIEWED] 7169c95

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7169c95

Verdict parsed from the review's SHA-scoped output markers for commit 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ human override accepted

Human judgment by @chenmingwei23 overrides the GPT 5.6 finding for 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f; the recorded reason is authoritative for this commit.

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 4, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/substitution-bodies-quote-aware-8150 branch from be12c61 to 5691f05 Compare September 4, 2026 23:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 4, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/substitution-bodies-quote-aware-8150 branch from 5691f05 to 177196d Compare September 4, 2026 23:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 4, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/substitution-bodies-quote-aware-8150 branch from 177196d to 45609c1 Compare September 4, 2026 23:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 4, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/substitution-bodies-quote-aware-8150 branch from 45609c1 to 2fd5958 Compare September 5, 2026 00:00
@chenmingwei23 chenmingwei23 changed the title fix(security): track quoting when scanning for a substitution's closer fix(security): decide a substitution's closer from shell state, not a paren count Sep 5, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 5, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 5, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/substitution-bodies-quote-aware-8150 branch from 739ead8 to df60a4b Compare September 5, 2026 02:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 5, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 5, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

GPT F1 (line-continuated case) -- fixed as requested, with one measured residual

The finding was correct and is now fixed. Recording the evidence, because the fix closes the half this PR owns and leaves a residual that belongs elsewhere.

The mechanism checks out

bash folds a backslash-newline before it reads words, so ca\ + newline + se really is the reserved word case. Measured directly, with the verb replaced by a harmless marker and the product CLI replaced by a shim that only echoes its argv:

$ bash -c 'T=$(ca\
se x in x) printf MARK;; esac) ; shimname $T'
SHIM INVOKED WITH: [MARK]

So the reaching condition is real, exactly as adjudicated.

What changed

_word_at now folds continuations while matching, so the folded spelling arms the pattern rule and the body survives:

spelling body before body after
$(ca\+nl+se x in x) printf <verb>;; esac) ca\+nl+se x in x ca\+nl+se x in x) printf <verb>;; esac

_opens_comment reads the character before a # the same way, so a\ + newline + #b stays the single word a#b rather than opening a comment.

The residual, and why it is not this PR's

The verdict on that spelling still does not flip -- and the cause is not this walk. _self_tokens reads the backslash-newline as a command separator instead of folding it away, which severs the assignment from the invocation so $T never resolves:

base branch, continuated form:
  tokens  : ['t=$(ca', ';', 'se', 'x', 'in', 'x)', 'printf', '<verb>;;', 'esac);', 'kirocrew', '$t']
  verdict : None

this branch, continuated form:
  tokens  : ['t=$(ca', ';', 'se', 'x', 'in', 'x)', 'printf', '<verb>;;', 'esac);', 'kirocrew', '$t']
  verdict : None

Note the ';' token where the continuation was, on both trees. The same command is allowed on the base branch with a byte-identical token split, so this change neither introduces nor worsens it -- measured zero delta. What this PR did move, on the same tree:

plain `case` form:   base = None (allowed)  ->  this branch = DENIED

That is a tokenizer-level continuation bug sitting upstream of the span helper. It is filed separately rather than folded in, because _self_tokens feeds every argv check and widening it does not belong in a span fix -- the same reason the heredoc and ${ } rows were left out of this PR.

The test for the continuated spelling therefore asserts the recovered body only, and its docstring says why the verdict does not move, so nothing here is recorded as fixed that is not.

Note for the Design and First Principles lanes

Both BLOCK verdicts on f6af66f99 read the PR description from the run set that started before the rewritten description landed, and they object to claims the current description no longer makes -- no _substitution_closer, no frame stack, no heredoc queue, and an explicit section listing what is deliberately not modelled. Design Review's own note says the shipped code is "sound and honestly documented in its own docstrings ... once description and diff agree this is mergeable in shape". They should settle on this head.

… paren

The span walk that decides where a substitution body ends was already
quote-aware, but two COMMAND-GRAMMAR constructs put a literal ")" in front
of it without quoting one, and each hid a payload this module still refuses:

  * a "#" comment runs to the end of its line, so the ")" in "$(: # )" is
    commented out and the body closes on a LATER line. Counting it ended the
    body at ": # ", losing the printf behind it that computes the
    credential-minting verb.
  * a "case" PATTERN is terminated by ")". In
    "$(case x in x) printf <verb>;; esac)" that ")" is the pattern's, not the
    substitution's, and reading it as one truncated the body to "case x in x".

Both were measured on main in both directions: is_denied returned None while
bash assembled and ran the refused payload. Two of the three anchors now flip
to denied; the third recovers its body but keeps its verdict, because the
self-kill pass never attributes a substitution sitting in an assignment ahead
of the kill -- a separate gap this change neither causes nor closes.

Taught to _matching_close_paren rather than to a new private walk. It is THE
one span computation and all three of its call sites are extractors, where a
span reaching too far only feeds them more text to inspect, so fixing it once
covers the argv-floor extractors too and leaves no second reading to drift.

Reserved-word recognition folds backslash-newline continuations, because the
shell removes them before it reads words at all: "ca\" + newline + "se" IS the
reserved word "case", and bash was measured running it as one. Byte-literal
matching missed that spelling, so the rule never armed and the pattern ")"
closed the body early again -- the same bypass for the cost of two characters.
The character before a "#" is read the same way, and the folds are stepped OVER
before that character is tested -- what matters is what the fold leaves
adjacent, not that a fold is there. "a\" + newline + "#b" folds to the single
word "a#b" and opens nothing, while ": " + "\" + newline + "#" folds to ": #",
where a word break ends up in front of the "#" and bash opens a real comment.
Treating any preceding fold as "not a comment" got that second case wrong in
the fail-OPEN direction: the comment was missed, so the walk read the ")" it
hides as the closer and truncated the body before the payload, reopening this
change's own bypass for the folded spelling (found in review).

That spelling recovers its body but not its verdict, and not because of this
walk: _self_tokens reads the backslash-newline as a command SEPARATOR instead
of folding it, severing the assignment from the invocation so "$T" never
resolves. Measured on the base branch with the identical token split, so this
change neither causes nor worsens it. Tracked separately rather than folded in,
because that tokenizer feeds every argv check.

The BACKTICK closer now reads the same state machine, which is hardening
rather than a patched hole: the strings the old pairwise find mis-read are
ones bash itself rejects, since backticks do not nest unescaped. It is fixed
so the two spellings of one closer cannot drift apart, which is how the paren
half broke before.

Refs #8150
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Opus F1 (_opens_comment fails open on a folded comment) -- confirmed and fixed

The finding was correct, the defect was mine, and it is fixed in 7169c9522.

The mechanism reproduced exactly as described

_opens_comment returned False whenever a \ + newline preceded the #. That is only right when a word character precedes the fold. When a word break does, the fold leaves : # -- a real comment -- and bash opens one, because it removes the fold before it lexes comments at all.

Traced on the folded shape, before the fix:

raw     : 'T=$(: \\\n# )\nprintf <verb>); kirocrew $T'
_opens_comment at the '#': False        <- comment missed
bodies  : [': \\\n# ']                  <- truncated at the ")" the comment hides
mint in body? False                     <- the verb fell outside every scanned body

After:

_opens_comment at the '#': True
bodies  : [': \\\n# )\nprintf <verb>']
mint in body? True

And the other direction still holds, so the fix does not over-correct:

raw     : 'T=$(printf a\\\n# )\nprintf <verb>); kirocrew $T'
_opens_comment at the '#': False        <- 'a\' + newline + '#' is the one word 'a#b'

One correction to the finding's stated consequence

The claim was that is_denied returns None for that input. It does not -- it returned DENIED both before and after, via a different tier (rule=credential-exfil-kirocrew-token, component=argv-floor). So that particular string was never allowed.

That does not weaken the finding, and I am not treating it as a partial false positive. The body-scan blindness was real and was introduced by this PR; a sibling regex catching one spelling is not a safety property, since a shape that dodges the regex would go through. This PR's own premise is that every consumer inherits the body scan's blindness, so shipping a helper that misses a real comment would contradict the change. Fixed rather than argued.

The suggested fix was taken as given -- step back over all fold pairs, then test the character in front of them:

j = index
while text.endswith("\\\n", 0, j):
    j -= 2
return j == 0 or text[j - 1] in _SHELL_WORD_BREAK

Two tests pin both directions: test_a_fold_after_a_word_break_still_opens_a_comment (body recovers, verdict denied) and test_a_folded_continuation_does_not_make_a_hash_a_comment (stays one word). 1770 passing across the four security suites, zero failures.

Note for the GPT lane

The GPT comment on c1363b70f carries a provider-refusal stale notice and its displayed verdict is pinned to f6af66f99, naming the line-continuated case finding that was already fixed in c1363b70f (evidence in the comment above). It should get a fresh attempt on 7169c9522.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 7169c95: Provider refused this head twice, so the shown blocking verdict is pinned to f6af66f and names the line-continuated case finding, which was fixed in c1363b7 (_word_at folds continuations) and is covered by two tests; Opus 4.8 reports no findings on this head and Design, First Principles and PR Hygiene are green.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 7169c95224ac4f9e2e97b9e6eba0b41bab132d4f.

Provider refused this head twice, so the shown blocking verdict is pinned to f6af66f and names the line-continuated case finding, which was fixed in c1363b7 (_word_at folds continuations) and is covered by two tests; Opus 4.8 reports no findings on this head and Design, First Principles and PR Hygiene are green.

This decision applies only to this commit. A new push requires a new judgment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants