Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,15 @@ uvx ruff check --fix . && uvx ruff format . # lint and format

## Testing instructions

The repo has no test suite and no CI. Until one exists, a change is done when all of this passes:
One check runs in CI: `.github/workflows/agents-md-readiness.yml` scores this file on any pull request that touches it, and fails below 0.85 readiness or on a line that looks like a credential. The repo has no test suite, so a change is done when all of this passes:

1. `uv run python -m py_compile <file>` on every Python file you touched.
2. `uvx ruff check .` and `uvx ruff format --check .` both report clean.
3. The sample runs against a live key, and every code path you changed runs at least once. For readme-check that means a local file, a GitHub repo root, a `/blob/` file page, and an `http://` URL it should refuse. For agents-md-readiness add a repo holding neither AGENTS.md nor CLAUDE.md, and a directory with neither.
4. Any output shown in a README comes from a run you just did, with the date next to it.
5. `samples/agents-md-readiness/go/` compiles that sample into one static binary, so a change to its `questions.yml` or its Go files means `gofmt -l .`, `go vet ./...`, `go test ./...` and `./build.sh <version>` in that folder, then one live run of the binary you built. The Python stays canonical, and `payload_test.go` fails when the embedded copy of the payload drifts.

If you add a test suite, use pytest, mock the client rather than calling the API, and replace this section.
Run the whole list before you open a pull request, because CI covers the AGENTS.md check and nothing else. If you add a test suite, use pytest, mock the client rather than calling the API, and wire it into that workflow.

## Code style

Expand Down
23 changes: 22 additions & 1 deletion samples/agents-md-readiness/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,28 @@ curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/a
agents-md-readiness -fail-under 0.8 -fail-on-credential AGENTS.md
```

Exit codes are 0 scored, 1 error, 2 gate failed. [0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats.
Exit codes are 0 scored, 1 error, 2 gate failed. [0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) ships binaries for linux and macOS on amd64 and arm64, plus windows amd64, and the installer checks them against the published `SHA256SUMS`. The Python here stays canonical, and a test in that folder fails when its copy of the payload drifts. [go/README.md](go/README.md) covers building, releasing and the caveats.

### This repo runs the check on itself

[`.github/workflows/agents-md-readiness.yml`](../../.github/workflows/agents-md-readiness.yml) installs that release and scores this repo's AGENTS.md on any pull request touching it, failing below 0.85 readiness or on a credential reading `suspect` or worse. Copy it, change the bar, and the check is yours.

```yaml
- run: curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh
env:
VERSION: "0.2.0"
BINDIR: ${{ runner.temp }}/bin

- run: ${{ runner.temp }}/bin/agents-md-readiness -fail-under 0.85 -fail-on-credential -json report AGENTS.md
env:
TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
```

Three details the workflow settles, each from a run rather than a guess:

- **The bar leaves room for the model.** This file scores 0.92 with a spread of 0.01 across repeat runs, so 0.85 passes today and still fails a real regression. On a GitHub runner the check took 641 ms and 3,109 input tokens.
- **A failing gate has to fail the job.** `set -o pipefail` keeps the exit code through the `tee` that feeds the job summary. A throwaway commit raising the bar to 0.99 failed with `readiness 0.92 is under the 0.99 bar` and exit code 2, and the summary and the JSON artifact still appeared, because both steps run under `always()`.
- **A fork pull request gets no secrets.** The job checks for the key first and writes a line in the summary saying it skipped, instead of failing on a missing key and teaching everyone to ignore a red check.

## What it asks

Expand Down
16 changes: 9 additions & 7 deletions samples/agents-md-readiness/go/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ agents-md-readiness -fail-under 0.8 https://github.com/apache/airflow

## Install it

[0.1.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.1.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64:
[0.2.0](https://github.com/aarora79/jev-samples/releases/tag/agents-md-readiness/0.2.0) carries binaries for linux and macOS on amd64 and arm64, plus windows amd64:

```bash
curl -fsSL https://raw.githubusercontent.com/aarora79/jev-samples/main/samples/agents-md-readiness/go/install.sh | sh
Expand Down Expand Up @@ -39,7 +39,7 @@ go build -o agents-md-readiness .
`build.sh` cross-compiles every platform `install.sh` knows about from whichever machine you run it on, and writes `SHA256SUMS` beside the binaries:

```bash
./build.sh 0.1.0
./build.sh 0.2.0
```

That produced five binaries on 19 September 2026, 6.4M to 7.2M each, in 4.4 seconds on this machine: linux amd64 and arm64, darwin amd64 and arm64, windows amd64. `-trimpath` drops the build paths, and Go stamps the git revision and a dirty flag into the binary, so the same commit in a clean tree reproduces the same hashes and a dirty tree does not. `go version -m <binary>` prints both. The release assets came from a fresh clone of the tagged commit, which is why their hashes differ from a build in a working tree that has edits.
Expand Down Expand Up @@ -147,17 +147,19 @@ Build from a clean clone of the commit you are tagging, so the revision Go stamp
```bash
git clone --depth 1 https://github.com/aarora79/jev-samples.git /tmp/relbuild
cd /tmp/relbuild/samples/agents-md-readiness/go
./build.sh 0.1.0
gh release create agents-md-readiness/0.1.0 dist/* \
./build.sh 0.2.0
gh release create agents-md-readiness/0.2.0 dist/* \
--repo aarora79/jev-samples \
--target "$(git rev-parse HEAD)" \
--title "agents-md-readiness 0.1.0" \
--title "agents-md-readiness 0.2.0" \
--notes "Static binaries for linux, macOS and Windows."
```

Versions are plain [semver](https://semver.org), `0.1.0` rather than `v0.1.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source.
Versions are plain [semver](https://semver.org), `0.2.0` rather than `v0.2.0`, in the tag, the asset names and what `-version` prints. The tag carries the `agents-md-readiness/` prefix because `install.sh` resolves the newest tag with that prefix, which leaves room for another sample to ship its own binary. `dist/` is gitignored: the release holds the binaries and the repo holds the source.

0.1.0 came out of `3bfae70` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. The install path ran end to end from an empty directory: `install.sh` resolved 0.1.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.1.0`. It then scored `langchain-ai/langchain` in 385 ms for 5,723 input tokens and exited 2 on `microsoft/vscode` under `-fail-under 0.8`.
0.2.0 came out of `2d28560` on 19 September 2026, built in a clean clone, and every asset stamps that revision with `vcs.modified=false`. It carries the deadband; 0.1.0, cut from `3bfae70`, does not, which is why the minor version moved rather than the patch. The install path ran end to end from an empty directory: `install.sh` resolved 0.2.0 off the tag, downloaded the linux amd64 asset, printed `checksum ok`, and installed a binary that reports `agents-md-readiness 0.2.0`.

This repo's own pull request check installs that release and scores this repo's AGENTS.md, so [the workflow](../../../.github/workflows/agents-md-readiness.yml) is the worked example: `checksum ok`, readiness 0.93, 3,109 input tokens, 641 ms on a GitHub runner. Raising its bar to 0.99 on a throwaway commit failed the job with `readiness 0.92 is under the 0.99 bar` and exit code 2.

## What to notice

Expand Down
Loading