Skip to content

Re-run Claude Code evals on a clean channel before re-publishing its report #1

Description

@angel291592

The 2026-09-22 Claude Code runs went through a channel that forced thinking off and answered English prompts in Chinese, so the failures can't be attributed to the skill itself. The report was therefore removed from the repo and Claude Code dropped to spec-compatible (see CHANGELOG 0.2.0). The methodology and cases stay public; only the report is held back.

Pending work:

  • Re-run the full 10-case suite on an unrestricted channel (TIMEOUT is now 480s; the ungrillable case that hit the old 240s limit and was not re-run for cost reasons must be included)
  • Confirm the fixes already in the skill (quoting hard rule, evidence token shape, ask-order rules) also converge on Claude Code
  • On pass: delete the evals/reports/*-claude-code.md line in .gitignore, re-commit the report, restore verified in both READMEs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions