Skip to content

fix(tests): make the aspect and E2E suites actually run - #24

Merged
hyperpolymath merged 1 commit into
mainfrom
fix/aspect-and-e2e-suites-actually-run
Jul 29, 2026
Merged

fix(tests): make the aspect and E2E suites actually run#24
hyperpolymath merged 1 commit into
mainfrom
fix/aspect-and-e2e-suites-actually-run

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

kitchenspeak carries the same test scaffold as trope-checker, and therefore the same defects — plus two of its own. Each stopped a suite from testing anything.

Shared (fixed identically in trope-checker#52):

  1. E2E dead at phase 3find … -exec bash -c '…' _ "$file" passes a variable never set in the outer scope. Under set -u the test exited 1 having reached zero assertions.
  2. The proof check flagged its own documentation — it matched prose describing the convention, and the phrase "sorry-free" in comments, i.e. text asserting the opposite of what it seeks. Now uses a block-comment-aware awk stripper; line-based filtering cannot see a comment continuation line.

Specific to kitchenspeak:
3. SPDX window off by onemain.zig carries MPL-2.0 on line 6 under a five-line banner; the check read head -5. The fix not taken: adding an SPDX line, which would have put a second licence header in an already-licensed file. The defect was the window.
4. ABI case mismatch — required src/interface/abi; the directory is src/interface/**Abi**. Same class as requiring codeql when the job is CodeQL.
5. Stale required-workflow list (npm-bun-blockerruntime-policy, ts-blocker deleted, scorecard-enforcerscorecard, guix-nix-policy dropped).

Red-teamed both ways — a check that stops reporting looks exactly like one that is satisfied:

clean tree PASS=3 FAIL=0
planted file with no SPDX FAIL=1
probe removed PASS=3

What the E2E now reports — structure validation passes with 0 errors, and the final phase finds files still carrying placeholders after instantiation. That is the finding this suite exists for, and it could not surface it while dead.

🤖 Generated with Claude Code

kitchenspeak carries the same test scaffold as trope-checker and therefore the
same defects, plus two of its own. Each stopped a suite from testing anything.

SHARED WITH trope-checker (fixed identically there, PR #52):

1. E2E DEAD AT PHASE 3. The placeholder step ran
     find ... -exec bash -c '...' _ "$file"
   but `file` is never set in the OUTER scope -- only inside the subshell from
   "$1". Under set -u that aborted with "file: unbound variable", so the test
   exited 1 having reached ZERO assertions. find must pass {}.

2. DANGEROUS-PROOF CHECK FLAGGED ITS OWN DOCUMENTATION. It grepped all of src/
   and verification/, excluding only paths containing "test" or "comment", so it
   matched prose describing the convention and the phrase "sorry-free" in
   comments -- text ASSERTING THE OPPOSITE of what it looks for. Now scans proof
   source only, through tests/lib/strip-proof-comments.awk, which tracks
   (* *), /- -/, {- -}, /* */ block state as well as -- and // line forms.
   Line-based filtering cannot see a block-comment CONTINUATION line.

SPECIFIC TO kitchenspeak:

3. SPDX WINDOW OFF BY ONE. src/interface/ffi/src/main.zig carries
   SPDX-License-Identifier: MPL-2.0 on line SIX, beneath a five-line banner. The
   check read `head -5`, so a correctly licensed file was reported as
   unlicensed. Window widened to 15.

   NOTE the fix NOT taken: adding an SPDX line. That would have put a SECOND
   licence header in a file that already had one. The defect was the window.

4. ABI DIRECTORY CASE MISMATCH. validate-template.sh required
   src/interface/abi; the directory is src/interface/Abi. Lowercase matched
   nothing, so a repo with a perfectly good ABI tree was failed for lacking one
   -- the same case-sensitivity class as requiring `codeql` when the job is
   named `CodeQL`.

5. Stale required-workflow list, as in trope-checker:
     npm-bun-blocker.yml    -> runtime-policy.yml
     ts-blocker.yml         -> DELETED
     scorecard-enforcer.yml -> scorecard.yml
     guix-nix-policy.yml    -> DROPPED (Nix retired 2026-06-01)

EVERY change red-teamed in both directions, because a check that stops reporting
looks exactly like a check that is satisfied:
  aspect, clean tree                     -> PASS=3 FAIL=0
  aspect, planted file with NO SPDX      -> FAIL=1
  aspect, probe removed                  -> PASS=3

WHAT THE E2E TEST NOW REPORTS, a real defect left for separate work: structure
validation passes with 0 errors, and the final phase finds files still carrying
placeholders after simulated instantiation. That is the finding this suite
exists to surface, and it could not surface it while dead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@hyperpolymath
hyperpolymath merged commit b605876 into main Jul 29, 2026
22 of 26 checks passed
@hyperpolymath
hyperpolymath deleted the fix/aspect-and-e2e-suites-actually-run branch July 29, 2026 07:36
Comment thread scripts/validate-template.sh
Comment thread tests/lib/strip-proof-comments.awk
Comment thread tests/aspect_tests.sh
@gitar-bot

gitar-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

Note

Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime.
Learn more

Code Review ✅ Approved 3 resolved / 3 findings

Fixes test runner scaffolds to ensure aspect and E2E suites actually execute. Address the duplicate scorecard.yml entry, nested Coq comment handling, and awk path assumptions in the validation scripts.

Auto-approved and auto-merge armed: No blocking issues found.
Please see Auto-approve Docs for details on setting custom approval criteria. — merges when pipeline and required approvals pass.

✅ 3 resolved
Quality: Duplicate scorecard.yml entry in REQUIRED_WORKFLOWS

📄 scripts/validate-template.sh:135 📄 scripts/validate-template.sh:144
scripts/validate-template.sh lists "scorecard.yml" at both line 135 and the newly-added line 144, so the workflow is checked twice. Harmless but redundant; drop the added duplicate (the scorecard-enforcer→scorecard rename is already covered by the existing line 135 entry).

Edge Case: Comment stripper mishandles nested Coq ( ) comments

📄 tests/lib/strip-proof-comments.awk:11-25
tests/lib/strip-proof-comments.awk closes a block on the first close token, but Coq supports nested (* ... ) comments. For (* outer (* inner *) Admitted *) the stripper closes at the inner *), leaving Admitted *) as live code, which grep then flags — a false positive that fails the aspect suite on legitimate documentation. Track a nesting depth counter for the ( *) case (Coq is the only nesting dialect here) rather than a boolean inblk.

Quality: Proof scan silently passes if awk path/CWD is wrong

📄 tests/aspect_tests.sh:86-94
In tests/aspect_tests.sh the scan uses an unquoted for f in $(find ...) (word-splits on paths containing spaces) and invokes awk -f tests/lib/strip-proof-comments.awk ... 2>/dev/null, a repo-root-relative path. If the script is run from another directory the awk file is not found, stderr is suppressed, DANGEROUS_PROOF stays empty, and the check reports PASS while testing nothing — the exact silent-satisfaction failure mode this PR set out to eliminate. Consider anchoring the awk path to the script directory and guarding against a missing stripper.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant