Skip to content

fix(issue-backlog-stats): expect marketplace-form actions in step-4 evals - #1677

Open
thisisnic wants to merge 1 commit into
apache:mainfrom
thisisnic:fix/backlog-stats-fixture-actions
Open

thisisnic wants to merge 1 commit into
apache:mainfrom
thisisnic:fix/backlog-stats-fixture-actions

Conversation

@thisisnic

Copy link
Copy Markdown
Member

Summary

  • issue-backlog-stats step-4-recommend failed deterministically on case-1-high-untriaged and case-2-high-stale: the model returned top_action in the marketplace form (/magpie-issue:triage, /magpie-issue:stale-sweep) while the fixtures expected bare skill names.
  • The skill is right. Its rules table says "Skill names here are the marketplace form" and links the marketplace doc, and the step's output-spec.md describes top_action as a slash command string. The sibling pr-management-stats fixtures already expect the same form.
  • The bare names are a leftover from docs: make the marketplace plugin install the documented default #1195: its first commit de-stuttered the fixtures, and a later commit in the same PR ("repair invocation forms the de-stutter sweep broke") reinstated the slash form in the stats dashboards. The fixtures were never re-synced.
  • Updates the two expected.json files to the slash form. No skill text changes.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures only
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

Before the change, full suite on main at 399ab1bc:

Model Runs Result
claude -p --model claude-opus-5-5 10 step-4-recommend/case-1-high-untriaged and case-2-high-stale FAIL on all 10 runs, identical diff each time; the other 13 cases passed every run

After the change, the step-4-recommend suite with the updated fixtures:

Model Runs Result
claude -p --model claude-opus-5-5 3 3 of 3 cases passed on every run

The other suites in this directory are untouched by the change; their 13 cases passed on every one of the 10 before-runs above.

  • prek run passes on the changed files
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p --model claude-opus-5-5" tools/skill-evals/evals/issue-backlog-stats/step-4-recommend/, 3 consecutive runs)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR — the fixtures are the whole change; no behaviour changed

RFC-AI-0004 compliance

No principles touched — eval expectation data only; no new mutation, host access, or LLM hop. No SKILL.md edit, so no token restamp.

Linked issues

Refs #1626, which records the failing run in the suite's Run history table.


Generated-by: Claude Code (Fable 5.1)

🤖 Generated with Claude Code

…vals

The step-4-recommend fixtures expected `top_action` as a bare skill
name (`issue-triage`, `issue-stale-sweep`), but the skill's rules
table emits the marketplace form (`/magpie-issue:triage`,
`/magpie-issue:stale-sweep`) and says so explicitly, and the step's
output spec calls the field a slash command string.

The bare names are a leftover from apache#1195: its first commit
de-stuttered the fixtures to bare names, and a later commit in the
same PR put the slash form back into the stats dashboards because a
bare name is not a pasteable command on any install path. The
fixtures were not re-synced after that second commit, so the two
cases failed deterministically against a live model.

Generated-by: Claude Code (Fable 5.1)
@github-actions github-actions Bot added family:issue issue-* skills capability:stats Read-only dashboards, metrics, governance evidence labels Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:stats Read-only dashboards, metrics, governance evidence family:issue issue-* skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant