From ab9e6dd73f0148f4bfd08528dd001f65215e7bd8 Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Sun, 20 Sep 2026 00:27:57 +0000 Subject: [PATCH] Correct the no-CI claim in AGENTS.md, and document 0.2.0 and the check AGENTS.md said "The repo has no test suite and no CI", which stopped being true when the workflow landed. The Testing instructions section now names the check, the bar and what it fails on, and the closing line says CI covers that and nothing else. That sentence was the file's weakest point, and fixing it moved the numbers. Five runs before and after, same binary: - test_commands 0.732 -> 0.904 - testing 1.6 / 2 at confidence 0.46 -> 1.80 / 2 at confidence 0.69 - readiness 0.9193 -> 0.9402 The within-side spread is 0.01 on readiness and 0.06 on a Noul, so these are real, unlike the key-placeholder edit that moved nothing. The ambiguity was in the file: it listed five commands and then said there was no CI. Docs also catch up with the release: - go/README.md points at 0.2.0 everywhere, records that it came from 2d28560 in a clean clone, and says why the minor version moved (0.1.0 has no deadband). - Both READMEs carry the workflow as the worked example, with the measured run: checksum ok, readiness 0.93, 3,109 input tokens, 641 ms on a GitHub runner, and exit code 2 with "readiness 0.92 is under the 0.99 bar" when a throwaway commit raised the bar. --- AGENTS.md | 4 ++-- samples/agents-md-readiness/README.md | 23 ++++++++++++++++++++++- samples/agents-md-readiness/go/README.md | 16 +++++++++------- 3 files changed, 33 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6c24b9f..886e698 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,7 +44,7 @@ uvx ruff check --fix . && uvx ruff format . # lint and format ## Testing instructions -The repo has no test suite and no CI. Until one exists, a change is done when all of this passes: +One check runs in CI: `.github/workflows/agents-md-readiness.yml` scores this file on any pull request that touches it, and fails below 0.85 readiness or on a line that looks like a credential. The repo has no test suite, so a change is done when all of this passes: 1. `uv run python -m py_compile ` on every Python file you touched. 2. `uvx ruff check .` and `uvx ruff format --check .` both report clean. @@ -52,7 +52,7 @@ The repo has no test suite and no CI. Until one exists, a change is done when al 4. Any output shown in a README comes from a run you just did, with the date next to it. 5. `samples/agents-md-readiness/go/` compiles that sample into one static binary, so a change to its `questions.yml` or its Go files means `gofmt -l .`, `go vet ./...`, `go test ./...` and `./build.sh ` in that folder, then one live run of the binary you built. The Python stays canonical, and `payload_test.go` fails when the embedded copy of the payload drifts. -If you add a test suite, use pytest, mock the client rather than calling the API, and replace this section. +Run the whole list before you open a pull request, because CI covers the AGENTS.md check and nothing else. If you add a test suite, use pytest, mock the client rather than calling the API, and wire it into that workflow. ## Code style diff --git a/samples/agents-md-readiness/README.md b/samples/agents-md-readiness/README.md index 256f39c..1ef41e6 100644 --- a/samples/agents-md-readiness/README.md +++ b/samples/agents-md-readiness/README.md @@ -92,7 +92,28 @@ curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/a agents-md-readiness -fail-under 0.8 -fail-on-credential AGENTS.md ``` -Exit codes are 0 scored, 1 error, 2 gate failed. [0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats. +Exit codes are 0 scored, 1 error, 2 gate failed. [0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats. + +### This repo runs the check on itself + +[`.github/workflows/agents-md-readiness.yml`](../../.github/workflows/agents-md-readiness.yml) installs that release and scores this repo's AGENTS.md on any pull request touching it, failing below 0.85 readiness or on a credential reading `suspect` or worse. Copy it, change the bar, and the check is yours. + +```yaml +- run: curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh + env: + VERSION: "0.2.0" + BINDIR: ${{ runner.temp }}/bin + +- run: ${{ runner.temp }}/bin/agents-md-readiness -fail-under 0.85 -fail-on-credential -json report AGENTS.md + env: + TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} +``` + +Three details the workflow settles, each from a run rather than a guess: + +- **The bar leaves room for the model.** This file scores 0.92 with a spread of 0.01 across repeat runs, so 0.85 passes today and still fails a real regression. On a GitHub runner the check took 641 ms and 3,109 input tokens. +- **A failing gate has to fail the job.** `set -o pipefail` keeps the exit code through the `tee` that feeds the job summary. A throwaway commit raising the bar to 0.99 failed with `readiness 0.92 is under the 0.99 bar` and exit code 2, and the summary and the JSON artifact still appeared, because both steps run under `always()`. +- **A fork pull request gets no secrets.** The job checks for the key first and writes a line in the summary saying it skipped, instead of failing on a missing key and teaching everyone to ignore a red check. ## What it asks diff --git a/samples/agents-md-readiness/go/README.md b/samples/agents-md-readiness/go/README.md index 34915f4..3a06a7f 100644 --- a/samples/agents-md-readiness/go/README.md +++ b/samples/agents-md-readiness/go/README.md @@ -10,7 +10,7 @@ agents-md-readiness -fail-under 0.8 https://github.com/apache/airflow ## Install it -[0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64: +[0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64: ```bash curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh @@ -39,7 +39,7 @@ go build -o agents-md-readiness . `build.sh` cross-compiles every platform `install.sh` knows about from whichever machine you run it on, and writes `SHA256SUMS` beside the binaries: ```bash -./build.sh 0.1.0 +./build.sh 0.2.0 ``` That produced five binaries on 19 September 2026, 6.4M to 7.2M each, in 4.4 seconds on this machine: linux amd64 and arm64, darwin amd64 and arm64, windows amd64. `-trimpath` drops the build paths, and Go stamps the git revision and a dirty flag into the binary, so the same commit in a clean tree reproduces the same hashes and a dirty tree does not. `go version -m ` prints both. The release assets came from a fresh clone of the tagged commit, which is why their hashes differ from a build in a working tree that has edits. @@ -147,17 +147,19 @@ Build from a clean clone of the commit you are tagging, so the revision Go stamp ```bash git clone --depth 1 https://github.com/aarora79/jev-samples.git /tmp/relbuild cd /tmp/relbuild/samples/agents-md-readiness/go -./build.sh 0.1.0 -gh release create agents-md-readiness/0.1.0 dist/* \ +./build.sh 0.2.0 +gh release create agents-md-readiness/0.2.0 dist/* \ --repo aarora79/jev-samples \ --target "$(git rev-parse HEAD)" \ - --title "agents-md-readiness 0.1.0" \ + --title "agents-md-readiness 0.2.0" \ --notes "Static binaries for linux, macOS and Windows." ``` -Versions are plain [semver](https://semver.org), `0.1.0` rather than `v0.1.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source. +Versions are plain [semver](https://semver.org), `0.2.0` rather than `v0.2.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source. -0.1.0 came out of `3bfae70` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. The install path ran end to end from an empty directory: `install.sh` resolved 0.1.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.1.0`. It then scored `langchain-ai/langchain` in 385 ms for 5,723 input tokens and exited 2 on `microsoft/vscode` under `-fail-under 0.8`. +0.2.0 came out of `2d28560` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. It carries the deadband; 0.1.0, cut from `3bfae70`, does not, which is why the minor version moved rather than the patch. The install path ran end to end from an empty directory: `install.sh` resolved 0.2.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.2.0`. + +This repo's own pull request check installs that release and scores this repo's AGENTS.md, so [the workflow](../../../.github/workflows/agents-md-readiness.yml) is the worked example: `checksum ok`, readiness 0.93, 3,109 input tokens, 641 ms on a GitHub runner. Raising its bar to 0.99 on a throwaway commit failed the job with `readiness 0.92 is under the 0.99 bar` and exit code 2. ## What to notice