Skip to content

fix(ci): validate coverage matrix against pinned upstream FABLE-5 source#6

Open
ahinks wants to merge 1 commit into
baskduf:mainfrom
ahinks:fix/coverage-source-required
Open

fix(ci): validate coverage matrix against pinned upstream FABLE-5 source#6
ahinks wants to merge 1 commit into
baskduf:mainfrom
ahinks:fix/coverage-source-required

Conversation

@ahinks

@ahinks ahinks commented Jun 15, 2026

Copy link
Copy Markdown

Summary

The CI workflow ran fable_coverage.py with no --source argument, so the matrix was only validated against itself. This hid a fabricated row in the coverage matrix: claude_behavior > critical_child_safety_instructions does not exist in the pinned upstream CLAUDE-FABLE-5.md (commit dc626fed).

What this PR changes

  • CI: fetches the pinned upstream source from the GitHub API (elder-plinius/CL4R1T4S at dc626fed) and passes it to fable_coverage.py --source. The fetch uses ${{ secrets.GITHUB_TOKEN }}; no extra setup needed.
  • Matrix: removes the fabricated row. Coverage is now an honest 71/71 (was a fictitious 72/72).
  • Tests: adds two regression tests
    • test_coverage_matrix_validates_against_pinned_source — fetches the pinned source via curl and runs the validator, asserting 100% accounting.
    • test_ci_workflow_validates_against_pinned_source — asserts the CI workflow contains the fetch step and the --source flag.
  • CHANGELOG: documents the fix and removes the stale "accounted for critical_child_safety_instructions" note.

Why this matters

FableCodex's discipline is credible only if the coverage matrix is honest. A self-consistent matrix that drifts from the upstream FABLE-5 prompt silently over-claims scope. Without an external check, the next fabricated row goes undetected.

The fix is the smallest change that closes the regression: pin the upstream source to its commit SHA, fetch it in CI, and let the existing validator do its job. No changes to fable_coverage.py itself — the script already supports --source correctly.

Verified locally

$ python3 -m unittest discover -s tests
Ran 24 tests in 3.2s
OK

$ python3 plugins/codex-fable5/skills/codex-fable5/scripts/fable_coverage.py --source /path/to/pinned-fable-5.md
codex-fable5: source headings 71, matrix rows 71, accounted 71/71 (100.0%)
codex-fable5: statuses adapted=39, implemented=24, not_applicable=4, unsupported=4
codex-fable5: coverage matrix valid

License

This contribution is AGPL-3.0-or-later to match the upstream repository.

Origin

Reported by the fable-discipline eval harness in alexanderh/fable-discipline (a separate hermes-side discipline harness). That harness is MIT and the validation logic that caught this bug is local to it; the only borrowed concept here is the matrix-vs-source integrity check, which is independent of either project.

The CI workflow ran fable_coverage.py with no --source argument, so the
matrix was only validated against itself. This hid a fabricated row in
the coverage matrix: claude_behavior > critical_child_safety_instructions
does not exist in the pinned upstream CLAUDE-FABLE-5.md (commit dc626fed).

Changes:
  - CI now fetches the pinned upstream source from the GitHub API and
    passes it to fable_coverage.py --source
  - Removed the fabricated matrix row; coverage is now an honest 71/71
  - Added two regression tests:
      test_coverage_matrix_validates_against_pinned_source
      test_ci_workflow_validates_against_pinned_source
  - CHANGELOG: documented the fix and removed the stale 'accounted for
    critical_child_safety_instructions' note
@ahinks ahinks requested a review from baskduf as a code owner June 15, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant