Surface skill preference regressions via a separate non-required CI status

[AbhitejJohn]

What

Adds an advisory, non-required evaluation-regression commit status to the report-status job. It downloads the vally-results-* artifacts and posts red only when a verdict is a credible, conclusive, adequately-powered preference regression. It never touches the required evaluation-status, so it is non-blocking by default; a maintainer can opt to make it required later.

Why

The required evaluation-status check is verdict-blind: it reports success whenever the evaluate job ran, regardless of whether a skill regressed. The verdict aggregation that knows about regressions runs only on the scheduled publish path, so a PR author never sees a regression signal on the PR itself.

Design notes

• Keyed on preferenceRegressed, not a VALID_REGRESSION state. The adapter records a credible ordinal preference loss as preferenceRegressed: true while the verdict state stays VALID_NO_CHANGE (VALID_REGRESSION is reserved and not emitted today). Keying on the state would be a permanent no-op. The filter also requires conclusive == true and underpowered != true to keep the false-positive rate down.
• Advisory on purpose. The signal is judge-based. On dual-judge cadences, a primary-judge preference regression is not always corroborated by the second judge, so this must not gate merge until the judge signal is more reliable.
• Partial-download aware. If the artifact scan does not complete, the status stays green with an explicit "scan incomplete - see run logs" note instead of failing silently.

Expected behavior against the corpus

• On a recent full-plugin run across 92 skills, there were 0 regressions, so on normal PRs the advisory almost never fires.
• On three curated dual-judge runs the primary judge flagged a regression in all three; the second judge corroborated none. This is exactly why the status is non-required: had it gated merge, it would have blocked all three with no corroboration.

Known limitation

A malformed results.json makes its per-file jq error be swallowed, so the advisory could miss a regression in a corrupt file. This is acceptable for an advisory status; the required evaluation-status still covers hard failures.

Validation

• actionlint 1.7.7 exits 0.
• The jq filter was tested against real verdict JSON and selects only the credible, powered regression.

/cc @AbhitejJohn

[AbhitejJohn]

/evaluate 22ba571

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 22ba571ae9d94af3614ccae4793a476059e7de4b to retry this exact commit.

26 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 22ba571ae9d94af3614ccae4793a476059e7de4b to retry this exact commit.

26 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.