Repository navigation
issue-backlog-stats: STALE-CANDIDATE is listed as a triage class but defined as orthogonal #1678
Description
What's broken
The issue-backlog-stats skill contradicts itself about what STALE-CANDIDATE is.
Step 2 of the skill tells the model to give each issue exactly one triage class. The table of classes has five rows, and STALE-CANDIDATE is one of them. But the same row says staleness is "orthogonal" and that a stale issue should be counted under its real class as well.
Those two instructions cannot both be followed. Sonnet follows the first one. Opus and Fable follow the second one. The eval fixture expects the second one, so Sonnet fails it.
Which layer
Skill: skills/issue-backlog-stats/SKILL.md.
Two places treat STALE-CANDIDATE as a class:
- The Step 2 class table, in the
STALE-CANDIDATErow. - The Step 3 bucket definition, which says "issues with class
STALE-CANDIDATE".
The eval output spec has the same problem. It lists STALE-CANDIDATE as an allowed value of triage_class, and then says staleness belongs in the separate is_stale_candidate boolean. File: tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/output-spec.md.
How to reproduce
Run the suite with Sonnet on main at 399ab1bc:
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
--cli "claude -p --model claude-sonnet-5-5" tools/skill-evals/evals/issue-backlog-stats/
The case step-2-classify/case-4-stale-candidate fails every time. I ran it 10 times and got the same diff on all 10:
$.triage_class: expected='UNTRIAGED', actual='STALE-CANDIDATE'
The fixture is an untriaged issue that was last updated 216 days ago, with warn_days set to 90.
For comparison, Opus and Fable each pass this case on 10 of 10 runs. The full runs are in the suite's Run history table, added in #1626.
Expected vs actual
Expected: triage_class is UNTRIAGED and is_stale_candidate is true.
Actual from Sonnet: triage_class is STALE-CANDIDATE.
The fixture is right and the Step 2 table is wrong. The rest of the skill already treats staleness as a flag, not a class:
- Step 3 counts stale candidates using
is_stale_candidate. - The untriaged-funnel rule in Step 3 selects issues that are
UNTRIAGEDand not stale. That only makes sense if an issue can be both.
Suggested fix
- Remove the
STALE-CANDIDATErow from the Step 2 class table, leaving four classes that do not overlap. - Under the table, add one line saying
is_stale_candidateis a separate flag set whendays_since_updated >= warn_days. - Reword the Step 3 bucket to count issues where that flag is set.
- Remove
STALE-CANDIDATEfrom thetriage_classvalues in the step-2 output spec. - Leave the case-4 fixture as it is.
Surface area (optional)
- I have not checked Step 5 (render) or the spec-loop spec for this skill. If either assumes five classes, the fix needs to touch them too.
- This edits
SKILL.md, so it needs ameasured_tokensrestamp and an eval rerun. Sonnet is the model that needs to go from failing to passing. - Same kind of problem as fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec #1615: the eval failure points at the prompt, not the fixture.
Activity
WilliamK112 commentedon Oct 17, 2026
I'd like to work on this issue.
I plan to separate
is_stale_candidatefrom the four mutually exclusive triage classes, update the output contract and any affected render/spec references, restampmeasured_tokens, and run the focused eval plus the repository pre-flight checks.sharpca commentedon Oct 21, 2026
Taking this one. The Step 2 class table and the step-2 eval output spec both need to stop treating staleness as a class; the rest of the skill already reads
is_stale_candidate, so only those two places move. PR follows.Fix inconsistency in issue-backlog-stats skill regarding STALE-CANDIDATE
WilliamK112 commentedon Oct 21, 2026
Thanks for picking this up. I had not started the implementation, so please go ahead; I’ll step back to avoid duplicating your work.
fix(backlog-stats): stop listing STALE-CANDIDATE as a triage class
fix(backlog-stats): stop listing STALE-CANDIDATE as a triage class (a…