Enable ruff security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE)

.github/scripts/update_policyengine_package.py

@@ -32,6 +32,11 @@ def _json_from_stdin() -> object:
     return json.load(sys.stdin)
 
 
+def _write_result(result: object) -> None:
+    """Print one result line for update-policyengine-package.sh to capture."""
+    print(result)
+
+
 def _requirement_name(requirement: str) -> str:
     return re.split(r"[\s\[<>=!~;@]", requirement, maxsplit=1)[0]
 
@@ -155,7 +160,7 @@ def find_issue(title: str) -> None:
         and isinstance(item.get("number"), int)
     )
     if matches:
-        print(matches[0])
+        _write_result(matches[0])
 
 
 def verify_issue(expected_number: int, expected_title: str) -> None:
@@ -191,7 +196,7 @@ def latest_version(package: str) -> None:
     if not isinstance(version, str) or not version:
         msg = f"PyPI response has no latest version for {package}"
         raise SystemExit(msg)
-    print(version)
+    _write_result(version)
 
 
 def bundle_versions() -> None:
@@ -264,7 +269,7 @@ def certificate_sha256(country: str) -> str:
         "uk_certificate_sha256": certificate_sha256("uk"),
     }
     for name, version in outputs.items():
-        print(f"{name}={version}")
+        _write_result(f"{name}={version}")
 
 
 #: Release-dispatch fields compared with the installed bundle: (payload key
@@ -353,9 +358,9 @@ def verify_dispatch() -> None:
     ):
         msg = f"The release dispatch says the UK pair was not promotable ({verdict})"
         raise SystemExit(msg)
-    print(f"dispatch_verified={'true' if compared else 'false'}")
-    print(f"uk_qualification_verdict={verdict}")
-    print(f"uk_qualification_receipt_sha256={receipt_sha256}")
+    _write_result(f"dispatch_verified={'true' if compared else 'false'}")
+    _write_result(f"uk_qualification_verdict={verdict}")
+    _write_result(f"uk_qualification_receipt_sha256={receipt_sha256}")
 
 
 def _parser() -> argparse.ArgumentParser:
@@ -398,7 +403,7 @@ def main() -> None:
     """
     args = _parser().parse_args()
     if args.command == "current-version":
-        print(current_version(args.pyproject, args.package))
+        _write_result(current_version(args.pyproject, args.package))
     elif args.command == "update-requirements":
         update_requirements(args.pyproject, args.package, args.current, args.latest)
     elif args.command == "find-issue":

AGENTS.md

@@ -140,13 +140,17 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 
 ### Code quality and tests
 
+- `sim.quality.ci-cd-scripts-ruff-exemptions`: CI/CD scripts under .github may print their output (T201); every other ruff rule applies to them. (blocking: ruff per-file ignore of T201 for `.github/**`)
+- `sim.quality.command-line-scripts-ruff-exemptions`: Command-line scripts under a scripts directory may print their output (T201), run other programs (S603, S607) and use today's date (DTZ011); every other ruff rule applies to them. (blocking: ruff per-file ignore of DTZ011, S603, S607, T201 for `**/scripts/**`)
+- `sim.quality.deployed-tests-ruff-exemptions`: The deployed-service test suite (projects/*/authenticated_tests, pytest run against a live candidate) gets the same ruff exemptions as unit tests (sim.quality.tests-ruff-exemptions); every other ruff rule applies to it. (blocking: ruff per-file ignore of ANN, D1, DTZ001, DTZ007, PLR2004, S101, S102, S105, S106, S108, S301, S603, S607 for `**/authenticated_tests/**`)
 - `sim.quality.every-project-tested`: Every project and library with unit tests runs them in the PR test matrix. (doc-only, not checked: follow it yourself)
 - `sim.quality.ruff-format-src`: Source code under projects/*/src and libs/*/src is formatted with ruff format. (blocking: ruff format)
 - `sim.quality.ruff-format-tests`: Test code is formatted with ruff format too; CI currently checks only src directories. (doc-only, not checked: follow it yourself)
-- `sim.quality.ruff-lint`: All Python code passes ruff check with every rule family selected in ruff.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. (blocking: ruff check (ANN, BLE, D, E4, E7, E9, EM, F, FLY, FURB, I, PIE, PYI, RET, RSE, TC, TRY, UP, W))
+- `sim.quality.ruff-lint`: All Python code passes ruff check with every rule family selected in ruff.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. (blocking: ruff check (ANN, BLE, D, DTZ, E4, E7, E9, EM, ERA, EXE, F, FIX, FLY, FURB, G, I, LOG, PIE, PYI, RET, RSE, S, T20, TC, TD, TRY, UP, W))
 - `sim.quality.runtime-evaluated-models-resolve`: Every Pydantic model is runtime-evaluated for ruff: one of its bases, read through the imports of the model's own module, is listed in ruff.toml's runtime-evaluated-base-classes or is a runtime-evaluated class in that same module; and every Pydantic model resolves its annotations once its package is imported. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_runtime_evaluated_annotations.py`, `projects/policyengine-simulation-executor/tests/test_runtime_evaluated_annotations.py`)
+- `sim.quality.src-programs-ruff-exemptions`: Programs under src (the Modal deployment utilities, and the OpenAPI and Stage 12 infrastructure commands) may print their output (T201) and open URLs from their arguments (S310); every other ruff rule applies to them. (blocking: ruff per-file ignore of S310, T201 for `projects/*/src/{modal/utils/*,policyengine_simulation_entry/generate_openapi,policyengine_simulation_entry/stage12_infrastructure,policyengine_simulation_gateway/generate_openapi}.py`)
 - `sim.quality.tests-pass`: Each project's unit tests pass in its own locked environment. (blocking: CI job `test`)
-- `sim.quality.tests-ruff-exemptions`: Test files may use pytest's bare assert and need no docstrings, annotations or named constants for magic numbers (S101, D1, ANN, PLR2004); every other ruff rule applies to tests. (blocking: ruff per-file ignore of ANN, D1, PLR2004, S101 for `**/tests/**`)
+- `sim.quality.tests-ruff-exemptions`: Test files may use pytest's bare assert and need no docstrings, annotations or named constants for magic numbers (S101, D1, ANN, PLR2004). They may also run repository scripts (S603, S607) and use fake credentials (S105, S106), temporary paths (S108), naive datetimes (DTZ001, DTZ007), pickle (S301) and exec (S102). Every other ruff rule applies to tests. (blocking: ruff per-file ignore of ANN, D1, DTZ001, DTZ007, PLR2004, S101, S102, S105, S106, S108, S301, S603, S607 for `**/tests/**`)
 
 ### CI workflows and scripts
 

changelog_entry.yaml

@@ -4,6 +4,7 @@
       - Made CLAUDE.md and the Copilot instructions thin pointers to AGENTS.md, required workflow logic longer than one line to live in tested and shellchecked scripts, and registered the tracking-issue rule
       - Added a rules registry (rules/registry.toml) with a standard-library tool that validates it, checks every rule is enforced where it claims, and renders docs/generated/rules.md and the Rules block in AGENTS.md
     changed:
+      - Enabled ruff's security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE); asserts outside tests became explicit checks, command-line scripts keep their printed output, test subprocess calls go through helpers, and every deliberate exception carries a reasoned suppression; the gateway's auth misconfiguration log no longer repeats its message, and the deployed-service tests share the unit-test ruff exemptions; tests, CI/CD scripts and command-line scripts take their deliberate exceptions as registered file-level ruff ignores instead of inline suppressions, and the rules guard rejects file-level ignores anywhere else
       - Enabled ruff's exception rules (TRY, EM, BLE); exception messages are built before the raise, raises inside try blocks moved into helpers, and every deliberate catch-all or traceback-free error log carries a reasoned suppression; message text is unchanged, and error types change only in an operator script's two type checks
       - Enabled ruff's docstring rules (D) and documented every module, class and function outside tests; published contract models and routes now carry these descriptions in their JSON schema and OpenAPI, with no structural schema change
       - Enabled ruff's annotation rules (ANN) and annotated every function outside tests; test files are exempt from ANN, missing docstrings, magic numbers and bare-assert rules

docs/engineering/skills/rules-registry.md

@@ -69,7 +69,7 @@ gap is visible, rather than leaving it out.
 | --- | --- | --- |
 | `pytest` | `nodes` (`path::test` or `path::Class::test`), optional `project` (sub-project directory the paths are relative to) | Each test exists, and the CI job's command collects it |
 | `quality-guard` | `name`, optional `runner` | The name is in `GUARDS` in `scripts/run_quality_guards.py` |
-| `ruff` | `mode` = `format`, `lint` (with `codes`) or `per-file-ignore` (with `pattern`, `codes`) | Codes are selected / the ignore entry exists |
+| `ruff` | `mode` = `format`, `lint` (with `codes`) or `per-file-ignore` (with `pattern`, `codes`) | Codes are selected / the ignore entry exists and covers only [files allowed file-level exceptions](#file-level-exceptions) |
 | `mypy` | `paths` | Each path is in `[tool.mypy].files` |
 | `workflow-job` | optional `step` | The CI job exists (the job itself is the check) |
 | `pyright`, `import-linter` | `config` (+ `contract` for Import Linter) | The config file exists |
@@ -117,13 +117,35 @@ fail the guard yet.
 
 - **Ruff is never ratcheted.** Adopting a ruff rule family is one standalone PR
   that changes the select list and fixes every resulting violation. Don't use
-  `--add-noqa` or blanket per-file ignores.
+  `--add-noqa`. A finding that is kept gets an inline `# noqa: CODE  # reason`,
+  except in the files below.
 - Other enforcers (mypy, Pyright, Import Linter, custom guards) may start
   `ratcheted` with a baseline file of tolerated violations. Baselines may only
   shrink: the guard compares counts against the base branch.
 - Moving a rule from enforced/ratcheted to advisory/doc-only needs an explicit
   `[rule.downgrade] reason`.
 
+## File-level exceptions
+
+A ruff `per-file-ignores` entry switches rules off for whole files. It is
+allowed only where that's the norm rather than an exception to it:
+
+- **tests**: files under a `tests` or `authenticated_tests` directory;
+- **CI/CD scripts**: files under `.github`;
+- **command-line scripts**: files under a `scripts` directory, or modules that
+  run as a program through `if __name__ == "__main__":`.
+
+Everywhere else, keep a finding only with an inline suppression and its reason.
+
+Each entry needs a `per-file-ignore` rule (see below). The guard expands the
+entry's pattern like ruff does and fails when the pattern:
+
+- covers any other file;
+- is negated;
+- matches nothing.
+
+`scripts/rules/file_exceptions.py` holds these definitions.
+
 ## Hidden rules
 
 The guard fails when something enforces a rule that the registry doesn't