Replace grep-based structural tests with behavioral tests (#160)

[TomHennen]

Closes #160.

Summary

build/actions/{npm,python}/test.bats and tools/scorecard/test.bats relied heavily on greps that asserted a particular string appeared somewhere in the action.yml or helper script. Those tests confirmed the presence of text, not behavior: moving a keyword into a comment would keep them passing, and a regression that broke the actual logic might not trip them. They also broke on harmless refactors (PR #156's helper extraction forced unrelated test edits with no behavioral change).

This PR follows the template established by lib/validate_path.sh and container/validate_inputs.sh: invoke each helper script against fixture inputs and assert on exit codes, GITHUB_OUTPUT contents, and stdout/stderr. Per the issue, greps remain only as supply-chain guard rails (no curl | sh, SHA-pinned actions, ${{ inputs.* }} flowing through env:, action.yml actually delegating to the scripts the behavioral tests now cover) and for workflow-level structural checks that have no behavioral counterpart.

What changed

• npm: +27 behavioral tests for validate_inputs.sh (package.json check, lockfile detection, yarn rejection, workspaces rejection, npm+pnpm ambiguity), detect_tooling.sh (the four node-version-resolution branches, npm/pnpm cache split that encodes the issue When pnpm/Yarn support lands, do NOT enable pnpm-store / yarn cache (Mini Shai-Hulud lesson) #205 decision), and build_and_pack.sh (npm/pnpm flow, --ignore-scripts threading through install AND pack, 'no test specified' substring guard, tarball-count check, scripts: null safety). PATH shims for npm/pnpm record invocations; jq stays real. Deleted ~15 source-string greps the behavioral tests supersede.
• python: +11 behavioral tests for the pyproject.toml requirement in validate_inputs.sh, the three test-discovery branches in run_tests.sh (tests/, test/, [tool.pytest]), the uv/pip split, and the [test] → [dev] → bare install fallback chain in install_deps.sh. Stub pytest/uv/python on PATH record invocations. Deleted 3 source-string greps.
• scorecard: +7 behavioral tests for sarif_to_markdown.sh (header+rows, empty results, HTML stripping, WRANGLE_MAX_SUMMARY truncation, missing-file and invalid-JSON exit codes, usage error). The truncation and HTML-stripping logic in particular were silent-failure-class bugs that no grep could catch.

Net: +28 behavioral tests, -18 source-string greps. Total test count in the touched files: npm 77, python 62, scorecard 13.

What was deliberately NOT changed

• build/actions/container/test.bats — already has solid behavioral tests for validate_inputs.sh and resolve_cache.sh (added in claude: SLSA L3 audit defense-in-depth — ::stop-commands:: guard + adopter-tunable PR cache knobs #225). Remaining greps are workflow/action.yml structural checks the issue explicitly calls out as legitimate guard rails.
• build/actions/shell/test.bats — the action wraps shellcheck/bats; no in-action script logic to test behaviorally.
• tools/{dependency-review,osv,syft}/test.bats — already have comprehensive behavioral tests with mocks.
• tools/{zizmor,scorecard}/action.yml checks, supply-chain greps (curl | sh, SHA-pin assertions, set -f presence), and action.yml → script delegation checks — these are the guard rails the issue says to keep.

Test plan

• bats build/actions/npm/test.bats — 77/77 pass
• bats build/actions/python/test.bats — 62/62 pass
• bats tools/scorecard/test.bats — 13/13 pass
• Full bats sweep across all in-scope files — 0 new failures (pre-existing bump_action_pins.bats failures are an unrelated test-env git config issue, present on main as well)
• CI passes (will be exercised on PR)
• Integration test in wrangle-test still passes (unit-test surface deliberately narrowed; integration covers the real npm/python/pip/uv invocations the PATH shims stand in for)

---

Generated by Claude Code

[TomHennen]

Verdict: approve-with-nits

Delivers what it claims — supply-chain greps preserved, fixtures exercise real branches, no actual coverage regression among the 18 deleted greps, CI green. A few should-fix items and one approach-level thought.

Should-fix

1. The [dev] middle branch of install_deps.sh fallback isn't directly tested. build/actions/python/test.bats:264-291 covers [test] success and the bare-install endpoint, but not the middle ([test] fails → [dev] succeeds → log "Installed with [dev] extra"). A regression that swapped [test] and [dev] order, or dropped the [dev] attempt entirely, would still pass. Add a fixture where the shim fails call 2 and succeeds on call 3, asserting "Installed with [dev] extra".
2. No explicit pack-uses-same-PM-as-install assertion. build/actions/npm/test.bats:421-451 — the deleted grep asserted "$PM" pack literally. Coverage is preserved by indirection (a regression in the pnpm path would fail the pnpm test), but consider one composite assertion: an npm-path test that asserts ! [[ "$output" == *"pnpm "* ]], and vice versa.
3. Shim relies on TMP_DIR from caller env — build/actions/python/test.bats:262-291. Fragile if bats ever sandboxes the run. Use $BATS_TEST_TMPDIR or a dedicated env var.
4. Pre-existing: run_tests.sh:30 grep -qF '[tool.pytest' matches the literal substring inside any string in pyproject.toml. Worth one extra negative fixture in a follow-up.

Nits

• build/actions/python/test.bats:268 — case "${COUNTER_FILE:-/tmp/missing}" in /tmp/missing) ;; *) ;; esac is dead code. Remove.
• build/actions/npm/test.bats:432-450 — install_pm_shim comment claims "$TMP_DIR/calls.log" but the shims write to stdout. Stale.
• setup() recreates shims per-test (77 of them). Move read-only shims to setup_file() for speed.

Approach-level

PATH-shim is the middle of a tradeoff — realistic enough to catch most regressions, fragile enough that real-tool behavior changes can silently break the assumptions. Worth picking deliberately rather than defaulting to "shim everything":

1. Pure function — extract decision logic so it's just args in → string out. Tightest tests, no shim drift, no real-tool flakiness. detect_tooling.sh's node-version resolution is a strong candidate.
2. Real tool — works for tools you can install cheaply in CI (jq already, pytest plausibly).
3. Shim — only when neither of the above fits.

Default to (1)/(2); fall to (3) only when forced.

Good

• Fixtures realistically drive decision branches (workspaces, scripts:null, both lockfiles, all four node-version-resolution branches, all four cache modes).
• Issue #205 "decision" asserted via cache= empty + belt-and-braces negative grep — exactly the "assert the decision, not the implementation" framing.
• Scorecard tests catch truncation/HTML-stripping silent-failure classes that no grep could.
• Supply-chain greps (curl|sh, /usr/local/bin, SHA-pin, env-flow, action.yml→script delegation, stop_commands_guard wrapping) preserved verbatim.
• Negative-path tests assert on error-message shape, not just exit code.
• CI all green.

---

Generated by Claude Code