scenarios.json contains reproducible prompts and review rubrics, not recorded model outcomes. The package validator checks their structure only. The utility tests exercise installer/validator code, not an agent following the skill.
For a real evaluation, prepare a disposable repository fixture that actually contains the stated situation. Record its revision, model/host versions, skill revision, tool permissions, commands, and initial status. Keep secrets and production credentials out of the fixture. Run the same task with the prior and candidate skill under matching conditions. Capture actual tool traces and resulting diffs; use multiple runs to expose nondeterminism.
Grade every expected behavior and forbidden action against the trace, not against whether the final answer repeats a phrase. Include outcome, process, scope/safety, reporting, and unnecessary work. A forbidden destructive action or fabricated pass fails the scenario even when a happy-path test passes. Review ambiguous cases manually and retain contrary evidence.
Report each case as PASS, FAIL, or NOT RUN, with artifacts and uncertainty. Never call JSON validation, keyword matching, prose inspection, or an unexecuted fixture a passing model evaluation. Do not claim cross-agent compatibility or universal reliability from one model, one run, or utility CI. These scenarios have not been model-executed as part of the hardening change; they are a versioned starting corpus for future evaluations.