Skip to content

issue-backlog-stats: STALE-CANDIDATE is listed as a triage class but defined as orthogonal #1678

Closed

Description

@thisisnic

What's broken

The issue-backlog-stats skill contradicts itself about what STALE-CANDIDATE is.

Step 2 of the skill tells the model to give each issue exactly one triage class. The table of classes has five rows, and STALE-CANDIDATE is one of them. But the same row says staleness is "orthogonal" and that a stale issue should be counted under its real class as well.

Those two instructions cannot both be followed. Sonnet follows the first one. Opus and Fable follow the second one. The eval fixture expects the second one, so Sonnet fails it.

Which layer

Skill: skills/issue-backlog-stats/SKILL.md.

Two places treat STALE-CANDIDATE as a class:

  • The Step 2 class table, in the STALE-CANDIDATE row.
  • The Step 3 bucket definition, which says "issues with class STALE-CANDIDATE".

The eval output spec has the same problem. It lists STALE-CANDIDATE as an allowed value of triage_class, and then says staleness belongs in the separate is_stale_candidate boolean. File: tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/output-spec.md.

How to reproduce

Run the suite with Sonnet on main at 399ab1bc:

PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
  --cli "claude -p --model claude-sonnet-5-5" tools/skill-evals/evals/issue-backlog-stats/

The case step-2-classify/case-4-stale-candidate fails every time. I ran it 10 times and got the same diff on all 10:

$.triage_class: expected='UNTRIAGED', actual='STALE-CANDIDATE'

The fixture is an untriaged issue that was last updated 216 days ago, with warn_days set to 90.

For comparison, Opus and Fable each pass this case on 10 of 10 runs. The full runs are in the suite's Run history table, added in #1626.

Expected vs actual

Expected: triage_class is UNTRIAGED and is_stale_candidate is true.

Actual from Sonnet: triage_class is STALE-CANDIDATE.

The fixture is right and the Step 2 table is wrong. The rest of the skill already treats staleness as a flag, not a class:

  • Step 3 counts stale candidates using is_stale_candidate.
  • The untriaged-funnel rule in Step 3 selects issues that are UNTRIAGED and not stale. That only makes sense if an issue can be both.

Suggested fix

  1. Remove the STALE-CANDIDATE row from the Step 2 class table, leaving four classes that do not overlap.
  2. Under the table, add one line saying is_stale_candidate is a separate flag set when days_since_updated >= warn_days.
  3. Reword the Step 3 bucket to count issues where that flag is set.
  4. Remove STALE-CANDIDATE from the triage_class values in the step-2 output spec.
  5. Leave the case-4 fixture as it is.

Surface area (optional)

  • I have not checked Step 5 (render) or the spec-loop spec for this skill. If either assumes five classes, the fix needs to touch them too.
  • This edits SKILL.md, so it needs a measured_tokens restamp and an eval rerun. Sonnet is the model that needs to go from failing to passing.
  • Same kind of problem as fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec #1615: the eval failure points at the prompt, not the fixture.

Activity

  1. WilliamK112 commented on Oct 17, 2026

    @WilliamK112

    I'd like to work on this issue.

    I plan to separate is_stale_candidate from the four mutually exclusive triage classes, update the output contract and any affected render/spec references, restamp measured_tokens, and run the focused eval plus the repository pre-flight checks.

  2. added
    family:docsDocs, MISSION.md, READMEs
    capability:triageSweep + classify + propose disposition
    capability:fixImplement code change to resolve an issue
    capability:resolveClose-out: invalidate, dedupe, CVE allocate, announcement
    capability:reassessRe-run historical issues against current code
    capability:statsRead-only dashboards, metrics, governance evidence
    substrate:framework-devTool substrate: build / validate / eval the framework itself
    on Oct 17, 2026
  3. sharpca commented on Oct 21, 2026

    @sharpca
    Contributor

    Taking this one. The Step 2 class table and the step-2 eval output spec both need to stop treating staleness as a class; the rest of the skill already reads is_stale_candidate, so only those two places move. PR follows.

  4. added a commit that references this issue on Oct 21, 2026
    2e95e50
  5. WilliamK112 commented on Oct 21, 2026

    @WilliamK112

    Thanks for picking this up. I had not started the implementation, so please go ahead; I’ll step back to avoid duplicating your work.

  6. added a commit that references this issue on Oct 24, 2026
    65cf96f
  7. added a commit that references this issue on Oct 24, 2026
    9c9b387
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    capability:fixImplement code change to resolve an issuecapability:reassessRe-run historical issues against current codecapability:resolveClose-out: invalidate, dedupe, CVE allocate, announcementcapability:statsRead-only dashboards, metrics, governance evidencecapability:triageSweep + classify + propose dispositionfamily:docsDocs, MISSION.md, READMEsfamily:issueissue-* skillsfamily:securitysecurity-* skillsfamily:setupsetup-* skillsfamily:toolstools/*good first issueGood for newcomerssubstrate:analyticsTool substrate: read-only metrics / dashboards / rendererssubstrate:framework-devTool substrate: build / validate / eval the framework itself

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions