fix(ci): validate coverage matrix against pinned upstream FABLE-5 source#6
Open
ahinks wants to merge 1 commit into
Open
fix(ci): validate coverage matrix against pinned upstream FABLE-5 source#6ahinks wants to merge 1 commit into
ahinks wants to merge 1 commit into
Conversation
The CI workflow ran fable_coverage.py with no --source argument, so the
matrix was only validated against itself. This hid a fabricated row in
the coverage matrix: claude_behavior > critical_child_safety_instructions
does not exist in the pinned upstream CLAUDE-FABLE-5.md (commit dc626fed).
Changes:
- CI now fetches the pinned upstream source from the GitHub API and
passes it to fable_coverage.py --source
- Removed the fabricated matrix row; coverage is now an honest 71/71
- Added two regression tests:
test_coverage_matrix_validates_against_pinned_source
test_ci_workflow_validates_against_pinned_source
- CHANGELOG: documented the fix and removed the stale 'accounted for
critical_child_safety_instructions' note
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CI workflow ran
fable_coverage.pywith no--sourceargument, so the matrix was only validated against itself. This hid a fabricated row in the coverage matrix:claude_behavior > critical_child_safety_instructionsdoes not exist in the pinned upstreamCLAUDE-FABLE-5.md(commitdc626fed).What this PR changes
elder-plinius/CL4R1T4Satdc626fed) and passes it tofable_coverage.py --source. The fetch uses${{ secrets.GITHUB_TOKEN }}; no extra setup needed.test_coverage_matrix_validates_against_pinned_source— fetches the pinned source viacurland runs the validator, asserting 100% accounting.test_ci_workflow_validates_against_pinned_source— asserts the CI workflow contains the fetch step and the--sourceflag.Why this matters
FableCodex's discipline is credible only if the coverage matrix is honest. A self-consistent matrix that drifts from the upstream FABLE-5 prompt silently over-claims scope. Without an external check, the next fabricated row goes undetected.
The fix is the smallest change that closes the regression: pin the upstream source to its commit SHA, fetch it in CI, and let the existing validator do its job. No changes to
fable_coverage.pyitself — the script already supports--sourcecorrectly.Verified locally
License
This contribution is AGPL-3.0-or-later to match the upstream repository.
Origin
Reported by the fable-discipline eval harness in alexanderh/fable-discipline (a separate hermes-side discipline harness). That harness is MIT and the validation logic that caught this bug is local to it; the only borrowed concept here is the matrix-vs-source integrity check, which is independent of either project.