fix(ci): the invisible-character gate never matched anything - #81
fix(ci): the invisible-character gate never matched anything#81hyperpolymath wants to merge 1 commit into
Conversation
MEASURED 2026-08-27: this gate's pattern caught 0 OF 6 invisible-character test
cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi
override or word joiner.
ROOT CAUSE: the pattern used UTF-8 BYTE sequences (\xc2\xa0) while grep -P
matches CHARACTERS. Bytes c2 a0 are ONE character U+00A0; \xc2\xa0 asks for TWO
characters, U+00C2 then U+00A0, which is never present.
grep -P '\xc2\xa0' -> miss
grep -P '\x{a0}' -> MATCH
Only \x00 worked, being single-byte in both readings.
FIXED: codepoint escapes; C0 control characters \x01-\x08,\x0B,\x0C,\x0E-\x1F
added (TAB/LF/CR excluded); and grep -a, without which grep skips any NUL-bearing
file as binary.
The C0 range matters: a stray BACKSPACE byte made a workflow unparseable in
developer-ecosystem, so it never ran, and this linter called it clean.
Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
VERIFIED: YAML re-parsed, and the corrected pattern was confirmed to catch a real
NBSP before the change was kept.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details🔇 Additional comments (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe workflow now uses Unicode code-point patterns for invisible-character detection. It also uses binary-safe ChangesInvisible-character gate
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This change corrects invisible-character detection in the workflow without introducing an actionable merge-blocking risk; it is merge-ready after normal checks and review. Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Linked Issues checkExplanation The change fixes the inline pattern in one dogfood-gate.yml file, including codepoint escapes, C0 controls, and grep -a. It does not satisfy all linked issue requirements because the compiled linter changes, separate leading-BOM check, and estate-wide copies are not present [ Resolution Apply the correction to all required dogfood-gate.yml copies. Add the corresponding C0-control support to stdlib/ByteDetector.affine and config.ncl. Add the separate byte-wise leading-BOM check. Confirm that the CI gate and compiled linter remain aligned [ Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Up to standards ✅🟢 Issues
|
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull Request Overview
The PR successfully fixes the invisible-character linting gate by transitioning to Unicode codepoint escapes and including C0 control characters. However, the regex pattern currently excludes several bidirectional control characters associated with the 'Trojan Source' vulnerability (CVE-2021-42574). Additionally, the use of find -exec ... \; is inefficient compared to the + variant, and silencing stderr may result in silent failures during PCRE execution. While Codacy indicates the PR is up to standards, the absence of automated test files containing prohibited characters makes it difficult to verify that the gate is operational and remains so in the future.
About this PR
- The PR lacks automated regression tests. Consider adding a dummy file or a test step containing problematic characters (e.g., NBSP, Zero-Width Space) to the repository to verify that the gate correctly identifies and blocks them in CI.
Test suggestions
- Verify detection of Non-Breaking Space (U+00A0)
- Verify detection of C0 control characters (e.g., Backspace \x08)
- Verify that files containing NUL bytes are scanned and not ignored as binary
- Verify detection of Zero-Width Space (U+200B) and Word Joiner (U+2060)
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify detection of Non-Breaking Space (U+00A0)
2. Verify detection of C0 control characters (e.g., Backspace \x08)
3. Verify that files containing NUL bytes are scanned and not ignored as binary
4. Verify detection of Zero-Width Space (U+200B) and Word Joiner (U+2060)
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| -o -name '*.idr' -o -name '*.zig' -o -name '*.v' -o -name '*.jl' \ | ||
| -o -name '*.gleam' -o -name '*.hs' -o -name '*.ml' -o -name '*.sh' \) \ | ||
| -exec grep -Prl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | ||
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: Using -exec ... {} + is significantly more efficient than -exec ... {} \; because it reduces the number of grep processes spawned. The -r flag is redundant here as find passes specific file paths. Additionally, removing 2>/dev/null ensures that matching errors (like invalid UTF-8) are visible.
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | |
| -exec grep -aPl "$PATTERNS" {} + > /tmp/empty-lint-results.txt |
| # non-breaking spaces, null bytes, and other invisible Unicode in source files. | ||
| set +e | ||
| PATTERNS='\xc2\xa0|\xe2\x80\x8b|\xe2\x80\x8c|\xe2\x80\x8d|\xef\xbb\xbf|\xc2\xad|\xe2\x80\x8e|\xe2\x80\x8f|\xe2\x80\xaa|\xe2\x80\xab|\xe2\x80\xac|\xe2\x80\xad|\xe2\x80\xae|\x00' | ||
| PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}' |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: This pattern list is missing several important bidirectional control characters used in Trojan Source attacks. Including these improves the security coverage of the invisible-character scan.
| PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}' | |
| PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2066}|\x{2067}|\x{2068}|\x{2069}|\x{2060}|\x{feff}' |
1 participant
Measured 2026-08-27: this gate caught 0 of 6 invisible-character test cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi override or word joiner.
Root cause
The pattern used UTF-8 byte sequences (
\xc2\xa0) whilegrep -Pmatches characters. Bytesc2 a0are one character U+00A0;\xc2\xa0asks for two, U+00C2 then U+00A0 — never present.Only
\x00worked, being single-byte in both readings. The gate ran, passed, and could not see what it exists to see.Fixed
\x01-\x08,\x0B,\x0C,\x0E-\x1Fadded (TAB/LF/CR excluded)grep -a— without it grep skips any NUL-bearing file as binaryThe C0 range matters: a stray backspace byte made a workflow unparseable in
developer-ecosystem, so it never ran — and this linter called it clean.Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
Verified: YAML re-parsed, and the corrected pattern was confirmed to catch a real NBSP before the change was kept.