Install PolicyBench from GitHub in the README quick start

[MaxGhenis]

Summary

The README's quick start told readers to run pip install policybench. PolicyBench is not on PyPI: on 2026-10-06 at 10:45 UTC, both https://pypi.org/pypi/policybench/json and https://pypi.org/simple/policybench/ returned HTTP 404. So the documented install fails today, and if anyone ever registers the name, it would install their package instead.

This PR changes the quick start to install from GitHub, and adds a test that fails if any reader-facing doc installs policybench from a package index by name.

Changes

• README.md, Quick start
  • Says PolicyBench is not published on PyPI.
  • Installs the CLI from GitHub with uv: uv tool install --python 3.12 git+https://github.com/PolicyEngine/policybench.
  • Gives the same install with pip: pip install git+https://github.com/PolicyEngine/policybench, in a standard (not free-threaded) Python 3.11 to 3.14 virtual environment.
  • The development path now includes the clone itself (git clone, cd) and installs the locked environment the way CI does (uv sync --locked --extra dev --python 3.12, then uv run pytest). Before, it said "clone the full Git repository" without a clone command.
• README.md, other sections
  • The dashboard block runs bun install --frozen-lockfile before bun run lint, as CI and docs/runbook.md do. A fresh clone has no app/node_modules, so bun run lint failed there.
  • The benchmark run section says its commands are for a clone. They write under results/local/, and analyze also writes the dashboard payload to app/src/data.json by default (--app-data-output in policybench/cli.py). They call policybench as uv tool install puts it on PATH; a clone set up with uv sync needs uv run in front, as the runbook does.
• tests/test_install_docs.py (new)
  • Reports any distribution an install command fetches from a package index by name. Paths, archives, VCS URLs and PEP 508 name @ url references don't count, and neither do --no-index/--offline installs.
  • Installers covered: pip/pip3/python -m pip/py -m pip/uv pip install, uv add, uv tool install, pipx install, conda/mamba install, poetry/pdm add, uvx, uv tool run, pipx run, uv run --with, and requirements files.
  • Decoding: each surface is decoded before scanning. JSON notes are read string by string with escapes decoded; HTML and TSX are read with their tags stripped.
  • Code vs prose: code is fenced blocks and <pre>, one command per line, plus inline spans and <code>, joined across soft wraps. Code is read as a full command. Prose is read only as an installer followed by the name, so "uv tool install puts policybench on your PATH" passes.
  • Tokenizing: shlex with operators and # comments, so quoted version specifiers, trailing comments and && chains read correctly.
  • By design: a doc that quotes pip install policybench even as a warning fails, because readers copy commands.
  • Surfaces scanned: root *.md (README, CLAUDE.md, RESULTS.md), docs/*.md, docs/requirements.txt, the paper source and its rendered HTML, sensitivity/*.md, app source and note copy, and the CI workflows. Each glob must match at least one file, so a moved file can't silently drop out of coverage.
  • Positive check: the quick start installs PolicyBench itself from git+https://github.com/PolicyEngine/policybench, read with the same tokenizer, and shows policybench --help.
  • No network: a live PyPI check would be flaky, and it would start passing the moment a squatter registered the name.

Why uv tool install pins --python 3.12

Without a pin, uv uses the interpreter it finds. On the verification machine that was the free-threaded CPython 3.14t. tokenizers (pulled in by litellm) has no free-threaded wheel, so uv builds it from source, and the build fails at the link step. uv tool install git+https://github.com/PolicyEngine/policybench with no --python failed exactly this way. CI tests 3.12.

Verification (macOS arm64, 2026-10-06)

Command	Python	Result
pip install git+https://github.com/PolicyEngine/policybench (stdlib venv, pip 26.2.1, no pip cache)	3.12.14	exit 0 in 601 s (mostly cloning the ~400 MB repo); policybench==2.0.0 from commit 2d399987; policybench --help exit 0
uv tool install --python 3.12 git+… (uv 0.11.7, fresh cache, isolated tool dir)	3.12.14	exit 0 in 135 s; policybench --help exit 0
uvx --python 3.12 --from git+… policybench --help	3.12.14	exit 0
uv pip install git+… into a clean venv	3.12.14	exit 0; policybench --help exit 0
uv pip install git+… into clean standard venvs	3.11.15, 3.13.9, 3.14.7	each: install, policybench --help, policybench reference-outputs --help, and import policybench.eval_no_tools, policybench.ground_truth, policybench.scenarios, policyengine_us all exit 0
uv pip install git+…	3.14.7 free-threaded	fails building tokenizers 0.23.2 (hence "not free-threaded" in the README)
uv tool install git+… (no --python)	uv picked 3.14.7t	fails the same way (hence the pin)

• New test: all 75 cases pass. Run against origin/main's README, the detector flags policybench; against this branch's README, nothing.
• Whole tracked tree: run over every tracked text file, the detector flags no other file.
• On the real surfaces it does find other index installs (pip install uv in CI, jupyter-book in docs/requirements.txt), so it is parsing the real files.
• ruff check . and ruff format --check . are clean.
• Non-slow suite: CI's test job (uv run pytest -m "not slow" on a clean Ubuntu runner, Python 3.12) passed on the first head, as did lint, app and Vercel. Locally, 81 tests failed. The ones I inspected, in tests/test_run_audit_claude.py, fail because the audit script runs claude auth status and refuses to start inside a Claude desktop session without a lane login. This PR doesn't touch that code.

Adversarial pass

Before review, 24 in-session agents attacked the first head (8088fb8) along three lenses: breaking the detector, auditing every claim, and completeness. Each finding was then checked by an agent that tried to refute it.

Fixed in 91f37b7, confirmed findings:

• The detector missed commands wrapped across lines.
• It missed JSON-escaped note text.
• It missed uvx --with X policybench.
• It missed --no-binary :all: and installs placed after a quoted specifier.
• It failed trailing # … policybench comments and prose that mentions an installer before the name.
• The README check rejected valid Git spellings.
• The dashboard block lacked bun install.
• The benchmark run commands didn't say they need a clone (and uv run there).
• A test docstring said the install needed no package index; dependencies still come from PyPI.

Rejected on verification:

• PyPI badge or link detection.
• More surfaces such as app/public/metric-options.html, which has no install text.
• CLAUDE.md's pip install -e ".[dev]", which installs from a clone.

Accepted gaps:

• In unformatted prose, only the first package after the installer is checked.
• Rarer installers (pixi, rye, tool upgrade or inject subcommands) are not modelled.

Other docs checked

No other surface claims a PyPI install:

• docs/runbook.md and docs/audit.md run uv run … from a clone.
• paper/README.md uses uv sync --extra docs.
• CLAUDE.md uses pip install -e ".[dev]", an editable install from a clone.
• The paper has no code-availability text naming PyPI, and the app copy has no install commands.

There is also no sign that a PyPI release was intended:

• no publish workflow (only ci.yml and paper.yml);
• no trusted-publisher config;
• the repo's GitHub environments are only Preview and Production;
• the repo's GitHub releases are dashboard-data-* snapshots.

So nothing here publishes. Whether to publish to PyPI, which would also claim the name, is a separate call for Max, queued as d1006. This PR doesn't depend on it.

Invariants the test holds

• No reader-facing surface (as listed above) installs the policybench distribution from a package index by name.
• The README quick start contains at least one install of policybench from this GitHub repository, and shows policybench --help.

axiom: n/a: docs-only install instructions, no policy change

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 6, 2026 11:54am UTC