fix: classify complete literal words consistently - #440
Merged
Conversation
tobert
added a commit
that referenced
this pull request
Sep 9, 2026
An assignment followed immediately by a home-relative path was tokenized as a regex comparison operator. Separate the assignment delimiter from the tilde path outside tests, while preserving the regex operator inside tests and respecting nested command-substitution scopes.
Home-relative assignments now parse:
```sh
p=~/x
local p=~/x
p={x:empty}; p[x]=~/x
foo --path=~/x
```
Regex comparisons retain their operator, including after logical operators:
```sh
[[ a == a && x=~/x ]]
[[ $(p=~/x; echo $p) =~ /x ]]
```
The lexer uses its existing test and substitution frames to distinguish these contexts. A closing bracket qualifies only after a statement-head assignment target, so a glob character class cannot become an assignment delimiter. Logical operators inside a test now retain that test frame. Split and joined tokens use the original source spans; text adjacent to interpolation still requires quoting. Runtime tilde expansion keeps its existing behavior.
Validation: 12 regression cases failed before the fix; all 27 targeted cases pass, including token spans, plain/local/bracket assignments, named arguments, nested substitutions, regex operators with and without spaces, interpolation refusal, and a glob bracket that must not be treated as an assignment target. Workspace tests, clippy across all targets with warnings denied, rustdoc with warnings denied, kernel tests without default features, and the WASI build passed. Help fragments and generated syntax are synchronized.
Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. The review also covers the preceding literal-word layer's final numeric punctuation cases. This layer builds on #440 in the native stack rooted at #437; no version bump.
tobert
force-pushed
the
fix/literal-word-classification
branch
from
September 9, 2026 12:09
be349fa to
bcc03b4
Compare
The lexer classified prefixes of ordinary literal words as numbers, keywords, or flags, then rejected the pieces as adjacent arguments. Recognize numeric filenames and version strings as whole words, accept embedded plus signs in word classes, and let numeric and keyword prefixes participate in existing colon and glob fusion. These words now retain their complete spelling: ```sh echo 123.txt 1.2.3 .123.txt true:foo do:foo a+b 123+b 123@host echo ~/a:b ls 1.0* true* ``` A numeric filename can exceed the integer range because it is text. Complete scalar numbers still follow the existing numeric rules: incomplete floats and integer overflow remain errors, leading zeros remain text, and ordinary integers and floats keep their types. Glob fusion slices the original source, so numeric formatting cannot change which filenames match. Standalone keywords, booleans, plus-prefixed flags, collection delimiters, and quote-to-join checks retain their roles. Validation: numeric and keyword regression suites failed before the fix; the final tests check exact token spans, argument values, assignment and redirect values, actual glob matches with misleading numeric-prefix controls, tilde expansion, and control-flow/record behavior. Workspace tests, clippy across all targets with warnings denied, and rustdoc with warnings denied passed. Help fragments and generated syntax are synchronized. Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. Review exposed the numeric prefix followed directly by punctuation; it is fixed with failing-then-passing regression tests rather than preserved as another word-character exception. This layer builds on #438 in the native stack rooted at #437. Tilde assignment and its regex-operator collision are a separate layer.
tobert
added a commit
that referenced
this pull request
Sep 9, 2026
An assignment followed immediately by a home-relative path was tokenized as a regex comparison operator. Separate the assignment delimiter from the tilde path outside tests, while preserving the regex operator inside tests and respecting nested command-substitution scopes.
Home-relative assignments now parse:
```sh
p=~/x
local p=~/x
p={x:empty}; p[x]=~/x
foo --path=~/x
```
Regex comparisons retain their operator, including after logical operators:
```sh
[[ a == a && x=~/x ]]
[[ $(p=~/x; echo $p) =~ /x ]]
```
The lexer uses its existing test and substitution frames to distinguish these contexts. A closing bracket qualifies only after a statement-head assignment target, so a glob character class cannot become an assignment delimiter. Logical operators inside a test now retain that test frame. Split and joined tokens use the original source spans; text adjacent to interpolation still requires quoting. Runtime tilde expansion keeps its existing behavior.
Validation: 12 regression cases failed before the fix; all 27 targeted cases pass, including token spans, plain/local/bracket assignments, named arguments, nested substitutions, regex operators with and without spaces, interpolation refusal, and a glob bracket that must not be treated as an assignment target. Workspace tests, clippy across all targets with warnings denied, rustdoc with warnings denied, kernel tests without default features, and the WASI build passed. Help fragments and generated syntax are synchronized.
Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. The review also covers the preceding literal-word layer's final numeric punctuation cases. This layer builds on #440 in the native stack rooted at #437; no version bump.
tobert
force-pushed
the
fix/literal-word-classification
branch
from
September 9, 2026 12:18
bcc03b4 to
aae80d3
Compare
tobert
added a commit
that referenced
this pull request
Sep 9, 2026
An assignment followed immediately by a home-relative path was tokenized as a regex comparison operator. Separate the assignment delimiter from the tilde path outside tests, while preserving the regex operator inside tests and respecting nested command-substitution scopes.
Home-relative assignments now parse:
```sh
p=~/x
local p=~/x
p={x:empty}; p[x]=~/x
foo --path=~/x
```
Regex comparisons retain their operator, including after logical operators:
```sh
[[ a == a && x=~/x ]]
[[ $(p=~/x; echo $p) =~ /x ]]
```
The lexer uses its existing test and substitution frames to distinguish these contexts. A closing bracket qualifies only after a statement-head assignment target, so a glob character class cannot become an assignment delimiter. Logical operators inside a test now retain that test frame. Split and joined tokens use the original source spans; text adjacent to interpolation still requires quoting. Runtime tilde expansion keeps its existing behavior.
Validation: 12 regression cases failed before the fix; all 27 targeted cases pass, including token spans, plain/local/bracket assignments, named arguments, nested substitutions, regex operators with and without spaces, interpolation refusal, and a glob bracket that must not be treated as an assignment target. Workspace tests, clippy across all targets with warnings denied, rustdoc with warnings denied, kernel tests without default features, and the WASI build passed. Help fragments and generated syntax are synchronized.
Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. The review also covers the preceding literal-word layer's final numeric punctuation cases. This layer builds on #440 in the native stack rooted at #437; no version bump.
tobert
added a commit
that referenced
this pull request
Sep 9, 2026
An assignment followed immediately by a home-relative path was tokenized as a regex comparison operator. Separate the assignment delimiter from the tilde path outside tests, while preserving the regex operator inside tests and respecting nested command-substitution scopes.
Home-relative assignments now parse:
```sh
p=~/x
local p=~/x
p={x:empty}; p[x]=~/x
foo --path=~/x
```
Regex comparisons retain their operator, including after logical operators:
```sh
[[ a == a && x=~/x ]]
[[ $(p=~/x; echo $p) =~ /x ]]
```
The lexer uses its existing test and substitution frames to distinguish these contexts. A closing bracket qualifies only after a statement-head assignment target, so a glob character class cannot become an assignment delimiter. Logical operators inside a test now retain that test frame. Split and joined tokens use the original source spans; text adjacent to interpolation still requires quoting. Runtime tilde expansion keeps its existing behavior.
Validation: 12 regression cases failed before the fix; all 27 targeted cases pass, including token spans, plain/local/bracket assignments, named arguments, nested substitutions, regex operators with and without spaces, interpolation refusal, and a glob bracket that must not be treated as an assignment target. Workspace tests, clippy across all targets with warnings denied, rustdoc with warnings denied, kernel tests without default features, and the WASI build passed. Help fragments and generated syntax are synchronized.
Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. The review also covers the preceding literal-word layer's final numeric punctuation cases. This layer builds on #440 in the native stack rooted at #437; no version bump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The lexer classified prefixes of ordinary literal words as numbers, keywords, or flags, then rejected the pieces as adjacent arguments. Recognize numeric filenames and version strings as whole words, accept embedded plus signs in word classes, and let numeric and keyword prefixes participate in existing colon and glob fusion.
These words now retain their complete spelling:
A numeric filename can exceed the integer range because it is text. Complete scalar numbers still follow the existing numeric rules: incomplete floats and integer overflow remain errors, leading zeros remain text, and ordinary integers and floats keep their types. Glob fusion slices the original source, so numeric formatting cannot change which filenames match. Standalone keywords, booleans, plus-prefixed flags, collection delimiters, and quote-to-join checks retain their roles.
Validation: numeric and keyword regression suites failed before the fix; the final tests check exact token spans, argument values, assignment and redirect values, actual glob matches with misleading numeric-prefix controls, tilde expansion, and control-flow/record behavior. Workspace tests, clippy across all targets with warnings denied, and rustdoc with warnings denied passed. Help fragments and generated syntax are synchronized.
Reviewed through kaibo's default cast, GLM-5.2 with DeepSeek V4 Flash investigation. Review exposed the numeric prefix followed directly by punctuation; it is fixed with failing-then-passing regression tests rather than preserved as another word-character exception. This layer builds on #438 in the native stack rooted at #437. Tilde assignment and its regex-operator collision are a separate layer.