Repository navigation
Conversation
…vals The step-4-recommend fixtures expected `top_action` as a bare skill name (`issue-triage`, `issue-stale-sweep`), but the skill's rules table emits the marketplace form (`/magpie-issue:triage`, `/magpie-issue:stale-sweep`) and says so explicitly, and the step's output spec calls the field a slash command string. The bare names are a leftover from apache#1195: its first commit de-stuttered the fixtures to bare names, and a later commit in the same PR put the slash form back into the stats dashboards because a bare name is not a pasteable command on any install path. The fixtures were not re-synced after that second commit, so the two cases failed deterministically against a live model. Generated-by: Claude Code (Fable 5.1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
issue-backlog-statsstep-4-recommend failed deterministically oncase-1-high-untriagedandcase-2-high-stale: the model returnedtop_actionin the marketplace form (/magpie-issue:triage,/magpie-issue:stale-sweep) while the fixtures expected bare skill names.output-spec.mddescribestop_actionas a slash command string. The siblingpr-management-statsfixtures already expect the same form.expected.jsonfiles to the slash form. No skill text changes.Type of change
.claude/skills/<name>/) — eval fixtures onlytools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
Before the change, full suite on
mainat399ab1bc:claude -p --model claude-opus-5-5step-4-recommend/case-1-high-untriagedandcase-2-high-staleFAIL on all 10 runs, identical diff each time; the other 13 cases passed every runAfter the change, the
step-4-recommendsuite with the updated fixtures:claude -p --model claude-opus-5-5The other suites in this directory are untouched by the change; their 13 cases passed on every one of the 10 before-runs above.
prek runpasses on the changed files(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p --model claude-opus-5-5" tools/skill-evals/evals/issue-backlog-stats/step-4-recommend/, 3 consecutive runs)RFC-AI-0004 compliance
No principles touched — eval expectation data only; no new mutation, host access, or LLM hop. No
SKILL.mdedit, so no token restamp.Linked issues
Refs #1626, which records the failing run in the suite's Run history table.
Generated-by: Claude Code (Fable 5.1)
🤖 Generated with Claude Code