CI: a docs-only skip and a passing suite are the same green

[fujibee]

What happens

tests.yml skips the bats suite on a documentation-only diff and reports the required check green anyway. The step even says so:

- name: Docs-only change — skipping suite
  if: needs.changes.outputs.docs_only == 'true'
  run: echo "Diff is documentation-only; skipping the bats suite and reporting green to satisfy the required check."

That is a reasonable thing to do. The problem is that the result is indistinguishable from a suite that ran and passed by every means anyone actually uses to read CI: the check name, its conclusion, the PR's green tick, and gh pr checks.

Measured

bats (ubuntu-latest 4/4), step by step, on two pull requests:

#776 — conclusion=success, suite never ran

 1 success  Set up job
 2 success  Run actions/checkout@v4
 3 success  Docs-only change — skipping suite
 4 skipped  Compute this shard's test files
 5 skipped  Ensure sqlite3 is available (Linux)
 6 skipped  Install bats
 7 skipped  Show tool versions
 8 skipped  Start hang sampler
 9 skipped  Run bats suite (this shard)
10 skipped  Hang forensics
11 skipped  Upload hang samples
12 skipped  Record which files this shard ran

#765 — conclusion=success, suite ran

 3 skipped  Docs-only change — skipping suite
 4 success  Compute this shard's test files
 ...
 9 success  Run bats suite (this shard)

Both are success. The two are told apart only by opening the job and reading step 9, which no summary view shows. #767 and #768 are the same as #776.

What it cost

While tracking down why bats (ubuntu-latest 4/4) was failing on several PRs at once, #776's green was put forward as evidence that the base was fine. It was not evidence of anything — that shard had skipped its suite. The search moved on from a real hypothesis on the strength of a check that had measured nothing, and it took a per-step read to get it back.

That green is also what makes the aggregate misleading: a PR can show 21/21 SUCCESS with the entire bats matrix skipped, and nothing in that number says which.

Related: #595 (the assertion this was found while investigating) and #758. Not #769 — that issue is about tests/test_watch.bats:501, a different assertion in the same test, and an earlier version of this line cited it for a property measured about the other one.

Suggestions, not a proposal

Any of these would restore the distinction; each has a cost and I have not measured them.

1. Name it. Emit a distinct check when the suite is skipped, e.g. bats (ubuntu-latest 4/4) [docs-only]. Cheapest to read, but the required-check configuration has to accept both names or the branch protection breaks.
2. Say it in the summary. echo "…" >> $GITHUB_STEP_SUMMARY on the skip path, so the PR page carries the word skipped without opening a job.
3. Make the aggregate honest. The bats aggregate job already inspects needs.*.result; it could report skipped rather than success when docs_only is true, leaving the shards green for branch protection.

The point of the issue is the ambiguity, not any particular fix.