Enable ruff security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE)

[anth-volk]

Fixes #3914

Also fixes #3909: the country validator's print("Validating country") is removed.

Summary

Phase 1 step 1.8, group 6: security, logging and debugging leftovers. Stacked on #3913.

• Rules: select gains S, T20, LOG, G, DTZ, ERA, TD, FIX and EXE. Every finding at the base (262, plus 12 in Stage 12A: Sign-in identity and v1-anchored user associations #3891's code after the restack) is fixed, kept with an inline reason, or, in tests, CI/CD scripts and command-line scripts, covered by a file-level exception. One commit per family.
• File-level exceptions (new):
  • Where they're allowed. A ruff per-file-ignores entry is now allowed only for:
    • tests (a tests or authenticated_tests directory);
    • CI/CD scripts (.github);
    • command-line scripts (a scripts directory, or a module with a top-level if __name__ == "__main__":).
  • How it's checked. The rules guard expands each entry's pattern the way ruff does: * and ? also match /, and a pattern also matches a file by its name. It fails when the pattern covers any other file, is negated, or matches nothing. scripts/rules/file_exceptions.py holds the definitions, docs/engineering/skills/rules-registry.md#file-level-exceptions documents them, and six rules-tool tests prove them.
  • The entries. Each lists exactly the group 6 codes those files had suppressed inline, not whole families, and each has a registered per-file-ignore rule:
    • tests (**/tests/**, extending api.quality.tests-ruff-exemptions): S105, S106, S603, S607, S608, DTZ001, DTZ011, T201;
    • CI/CD scripts (.github/**): S310, T201;
    • command-line scripts (scripts/**): T201, S603, S607, S310, S311, S110, DTZ011;
    • the five modules that also run as programs (catalog initialization, the three migration qualifications, worker_spm_release): T201.
  • What it removed. 122 inline suppressions: 74 in tests, 30 in scripts, 10 in the program modules and 8 in .github. The 36 suppressions in application code are unchanged.
  • Alembic revisions. Alembic revisions aren't any of those kinds of file, so their "migrations/*/versions/*.py" = ["ALL"] entry moves to [tool.ruff.lint] exclude. ruff check . still skips them, and ruff format still checks them. api.quality.alembic-revisions-unused-imports now describes the exclusion.
• Behaviour changes (three kinds, all deliberate):
  • Debug prints removed (T201): the country validator (validate_country prints "Validating country" on every request #3909), plus the create, get and update prints in the report output and simulation routes. The two update prints become logger.debug calls on a module logger, because their ids come from the request body.
  • Logger name (LOG015): the household calculation routes log their three errors on a module logger instead of the root logger. The message text is unchanged, and production records still reach the root observability handler.
  • Nothing else: error responses, message text and published schemas are unchanged.
• Asserts outside tests (S101, 11): each type-narrowing assert in the v2 household, policy, user-household and user-policy services, and two in the rules tool, becomes an explicit check that raises TypeError. None runs inside a Pydantic validator, and nothing catches AssertionError. Every check was already guaranteed by an earlier one, so none is expected to fire; the v2 routes and the mirror paths treat either type the same way. No production command runs Python with -O, so these asserts were live.
• Prints in commands (T201): command-line scripts, CI/CD scripts and the modules that run as programs (including the migration qualification commands' JSON evidence) keep their output under the file-level exceptions. capture_migration_baseline.py, smoke_test_local_boot.py and the rules tool's CLI route output through one _report helper. Application code keeps the api.py startup timings and the gateway-auth warning on stderr, each with an inline reason.
• Datetimes (DTZ, 21): no value switches between naive and aware. The catalog extraction builds its aware UTC value directly; the other 20 stay naive. Those in tests and scripts, such as the rules tool's calendar-day review dates and test values, are covered by the file-level exceptions. The 4 in application code (naive database columns, the local calendar year) keep an inline reason.
• Security (S): no vulnerability found.
  • Fixed: hashlib.sha1(..., usedforsecurity=False) (same digest), and the GPT-4 experiment parses embeddings with ast.literal_eval instead of eval.
  • Covered by file-level exceptions:
    • in tests: fake secrets, runs of this repository's scripts and git, and table names from a fixed tuple;
    • in scripts and .github: fixed https and loopback URLs, and a seeded load-test mix.
  • Kept inline with a reason:
    • environment-variable names, including the Auth0 and gateway client-secret names Stage 12A: Sign-in identity and v1-anchored user associations #3891 added;
    • the WWW-Authenticate challenge;
    • the experiment's requests calls, where a timeout would change behaviour.
  • S110: each try/except/pass already carried a reasoned BLE001 suppression from group 5, which now lists S110 too.
• Leftovers: 32 lines of commented-out code are deleted (29 in tests/to_refactor/python/test_policy.py). The two TODOs become plain notes; the user-profile one keeps its link to Utilize a technology like Marshmallow to validate inbound requests #2054 and says it was closed. Seven Python scripts with a shebang and a __main__ entry point are now executable.
• Registry:
  • api.quality.ruff-lint now lists the nine families.
  • Four per-file-ignore rules describe the file-level exceptions.
  • Rules tool. The new check is ported by hand into Enable ruff security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE) policyengine-sim-api#785. Each repository owns its copy of the rules tool.

Testing

• Every tip from Enable ruff security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE) #3915 to Register weighted-microdata rules, with AST guards and #767 proofs #3938 passes these checks:
  • ruff check and ruff format --check (0.15.12) are clean;
  • the generated rules files are fresh at every commit;
  • the quality guards, the rules guard against each parent, and the rules tool unittest (80 tests) pass;
  • mypy is clean (167 files).
• This update changes only comments outside the rules tool. Outside scripts/rules/, every changed Python file at every tip differs from the previously pushed tip only in # noqa comments: each version is compared with its noqa comments stripped. The other changes are:
  • pyproject.toml;
  • the registry;
  • the docs;
  • the rules tool's new check and tests.
• Full suites: tests/unit (2,685 passed) and the to_refactor, contract and dedupe set (97 passed) were run on this branch before the last change, which touched only the rules tool. They were not re-run after it.
• Every Pydantic model schema (230 models, 100 adapters), the FastAPI OpenAPI with every native router enabled, the /v2/openapi.json preview and the legacy spec are byte-identical to the base.
• CI/CD: no workflow changed. Four .github scripts changed only in comments, output helpers or file mode, and the environment variables they read are unchanged.
• CI: this PR is above the bottom of its stack, so it runs only CI complete, which fails and says why (step 0.7, Make CI enforcement trustworthy before the rules registry #3887). Its full run starts once the PRs below it merge.

🤖 Generated with Claude Code