Skip to content

tests: fix false failures and stale skips in official imported suites (parse-errors/grep/jq) #4

Description

@Haoxincode

Summary

Three issues were discovered in our imported official test suites that cause false failures or misleading results. These are problems with test infrastructure/test assets themselves, not with MoonBit implementation correctness.

Reproduction Environment

All issues reproduce with the safe test runner parameters:

pnpm exec vitest run --no-cache --pool=forks --maxWorkers=1 --fileParallelism=false --silent <suite>

This matches the execution model of scripts/test-safe.sh (fork + single worker).

Issues

1. parse-errors.comparison — exit code fallback logic is incorrect

  • File: tests/comparison/parse-errors.comparison.test.ts:27
  • Problem: The original logic uses err.status || 1. Under the fork pool, execSync can throw an error object (e.g. code: "EPERM") that carries status: 0 with normal stdout. The || 1 coercion treats status: 0 as falsy, replacing it with 1, which corrupts the real bash baseline exit code.
  • Impact: 11 cases that should pass (for, if, while, until, true, ||, blank commands, etc.) are falsely reported as failures.
  • Fix: Change to typeof err.status === "number" ? err.status : 1.
  • Verification: After fix, parse-errors.comparison.test.ts passes 24/24. Overall tests/comparison restores to 523/523 under safe parameters.

2. grep spec — stale skip entries cause UNEXPECTED PASS

  • File: tests/spec/grep/skips.ts
  • Problem: Multiple skipped cases actually pass now, causing the runner to report UNEXPECTED PASS errors. Typical categories:
    • "RE2 doesn't support backreferences" entries (multiple)
    • "RE2 treats grouped *** differently ..."
    • "BRE literal \{,2\} / \{,\} ... not implemented"
  • Suggested fix: Remove stale skip entries. Separate remaining skips into "genuine implementation gaps" vs "historical compatibility policy differences" to avoid skip pollution of regression results.

3. jq spec — stale skip entries cause UNEXPECTED PASS

  • File: tests/spec/jq/skips.ts
  • Problem: Multiple skipped cases now pass, causing UNEXPECTED PASS. Typical entries:
    • Invalid \v escape sequence test
    • String interpolation with complex expression
    • Empty array/object pattern ...
    • Undefined variable $bar behavior differs
    • delpaths with large negative index
    • join error message
    • unique sort order differs
  • Suggested fix: Review and remove stale skips. Add "expected removal condition" annotations to remaining skip entries for easier future cleanup.

Scope Clarification

These three items are test asset issues — problems with test infrastructure, baselines, or skip lists. They are distinct from implementation compatibility issues (grep/jq functional gaps), which should be tracked separately.

Checklist

  • Fix parse-errors.comparison.test.ts exit code fallback
  • Clean up tests/spec/grep/skips.ts stale entries
  • Clean up tests/spec/jq/skips.ts stale entries
  • Verify 523/523 comparison tests pass after fixes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions