Score this repo's own AGENTS.md on every pull request - #1
Merged
Merged
Conversation
.github/workflows/agents-md-readiness.yml installs the agents-md-readiness binary from the 0.2.0 release and runs it against AGENTS.md, failing the job below 0.85 readiness or on a credential reading suspect or worse. The bar sits at 0.85 because this file scores 0.92 with a spread of 0.01 across repeat runs, which leaves room for the model's variance and still catches a real regression. Details worth knowing: - The workflow triggers on a pull request that touches AGENTS.md or the workflow itself, plus workflow_dispatch for a manual run. - A fork pull request gets no secrets, so the job checks for the key first and explains the skip in the summary instead of failing on a missing key. - set -o pipefail keeps the binary's exit code through tee, so a gate failure fails the step rather than being swallowed by the pipe. - The summary and the artifact both run under always(), so a failing gate still shows the table and the JSON report. - permissions is contents: read and nothing more. One call costs about 3,100 input tokens, $0.00013 at TypeSafe's September 2026 price, and under half a second.
The 0.99 commit proved the gate: exit code 2 travelled through tee, the step failed, and the summary and artifact steps still ran under always(). The log line read "readiness 0.92 is under the 0.99 bar".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.github/workflows/agents-md-readiness.yml, which installs the agents-md-readiness binary from the 0.2.0 release and scores this repo's AGENTS.md.The job fails below 0.85 readiness or on a credential reading suspect or worse. The bar sits at 0.85 because this file scores 0.92 with a 0.01 spread across repeat runs.
This pull request is also the test: it touches the workflow, which is one of the paths that triggers it, so the check below is the feature running on itself.