fix(tests): make the aspect and E2E suites actually run - #24
Conversation
kitchenspeak carries the same test scaffold as trope-checker and therefore the
same defects, plus two of its own. Each stopped a suite from testing anything.
SHARED WITH trope-checker (fixed identically there, PR #52):
1. E2E DEAD AT PHASE 3. The placeholder step ran
find ... -exec bash -c '...' _ "$file"
but `file` is never set in the OUTER scope -- only inside the subshell from
"$1". Under set -u that aborted with "file: unbound variable", so the test
exited 1 having reached ZERO assertions. find must pass {}.
2. DANGEROUS-PROOF CHECK FLAGGED ITS OWN DOCUMENTATION. It grepped all of src/
and verification/, excluding only paths containing "test" or "comment", so it
matched prose describing the convention and the phrase "sorry-free" in
comments -- text ASSERTING THE OPPOSITE of what it looks for. Now scans proof
source only, through tests/lib/strip-proof-comments.awk, which tracks
(* *), /- -/, {- -}, /* */ block state as well as -- and // line forms.
Line-based filtering cannot see a block-comment CONTINUATION line.
SPECIFIC TO kitchenspeak:
3. SPDX WINDOW OFF BY ONE. src/interface/ffi/src/main.zig carries
SPDX-License-Identifier: MPL-2.0 on line SIX, beneath a five-line banner. The
check read `head -5`, so a correctly licensed file was reported as
unlicensed. Window widened to 15.
NOTE the fix NOT taken: adding an SPDX line. That would have put a SECOND
licence header in a file that already had one. The defect was the window.
4. ABI DIRECTORY CASE MISMATCH. validate-template.sh required
src/interface/abi; the directory is src/interface/Abi. Lowercase matched
nothing, so a repo with a perfectly good ABI tree was failed for lacking one
-- the same case-sensitivity class as requiring `codeql` when the job is
named `CodeQL`.
5. Stale required-workflow list, as in trope-checker:
npm-bun-blocker.yml -> runtime-policy.yml
ts-blocker.yml -> DELETED
scorecard-enforcer.yml -> scorecard.yml
guix-nix-policy.yml -> DROPPED (Nix retired 2026-06-01)
EVERY change red-teamed in both directions, because a check that stops reporting
looks exactly like a check that is satisfied:
aspect, clean tree -> PASS=3 FAIL=0
aspect, planted file with NO SPDX -> FAIL=1
aspect, probe removed -> PASS=3
WHAT THE E2E TEST NOW REPORTS, a real defect left for separate work: structure
validation passes with 0 errors, and the final phase finds files still carrying
placeholders after simulated instantiation. That is the finding this suite
exists to surface, and it could not surface it while dead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review ✅ Approved 3 resolved / 3 findingsFixes test runner scaffolds to ensure aspect and E2E suites actually execute. Address the duplicate scorecard.yml entry, nested Coq comment handling, and awk path assumptions in the validation scripts.
✅ 3 resolved✅ Quality: Duplicate scorecard.yml entry in REQUIRED_WORKFLOWS
✅ Edge Case: Comment stripper mishandles nested Coq ( ) comments
✅ Quality: Proof scan silently passes if awk path/CWD is wrong
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |



kitchenspeak carries the same test scaffold as trope-checker, and therefore the same defects — plus two of its own. Each stopped a suite from testing anything.
Shared (fixed identically in trope-checker#52):
find … -exec bash -c '…' _ "$file"passes a variable never set in the outer scope. Underset -uthe test exited 1 having reached zero assertions.Specific to kitchenspeak:
3. SPDX window off by one —
main.zigcarriesMPL-2.0on line 6 under a five-line banner; the check readhead -5. The fix not taken: adding an SPDX line, which would have put a second licence header in an already-licensed file. The defect was the window.4. ABI case mismatch — required
src/interface/abi; the directory issrc/interface/**Abi**. Same class as requiringcodeqlwhen the job isCodeQL.5. Stale required-workflow list (
npm-bun-blocker→runtime-policy,ts-blockerdeleted,scorecard-enforcer→scorecard,guix-nix-policydropped).Red-teamed both ways — a check that stops reporting looks exactly like one that is satisfied:
What the E2E now reports — structure validation passes with 0 errors, and the final phase finds files still carrying placeholders after instantiation. That is the finding this suite exists for, and it could not surface it while dead.
🤖 Generated with Claude Code