[anth-volk]

Fixes #803

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Phase 1 step 1.8, group 8b, the last group: complexity refactors, then select = ["ALL"]. Stacked on #802. This completes step 1.8.

Every function over ruff's complexity limits is refactored, not suppressed. That includes PLR0913 (too many arguments): this PR adds no suppression. Each refactor is preceded by a commit that adds characterization tests pinning the current behaviour. Every commit passes ruff and the quality guards.

Suppression counts

	Inline suppression comments	Rule codes in them
#802 (85f690c, the base)	221	225
This PR (eef2427)	211	215

• Moved, not new: the suppression comments this PR adds are seven that the refactors moved unchanged:
  • BLE001 in stage12_runtime/coordination.py (3) and stage12_runtime/simulation.py (1);
  • BLE001 and TRY400 (2) in the entry app's Stage 12 dispatch.
• Removed (10 more): ten in-function import suppressions (PLC0415).
  • Seven are in src/modal/_image_setup.py, three in src/modal/v2_app.py. All of them exist on main.
  • The refactor had also added eight more in those two files before this update. None remains.
• Still added: none.

Changes

• Contract and persistence libraries:

  • ComparisonReportRecord.validate_state keeps its Pydantic signature and is split into report and comparison checks.
  • The live column-type check is split into enum and plain-type checks.
  • URI, version and name-length limits are named constants.
  • create_initial_batch_state takes the batch's version and Modal app name as one frozen BatchRoute(version, app_name), and has five arguments.
  • resolve_spm_selection takes the route's policyengine version, model version and route provenance as one frozen SPMRouteVersions, and has four arguments.
  • All their callers are in this repository: the gateway, the executor's batch scheduler and SPM normalization, and the tests. The libraries are path dependencies, and the only package this repository publishes is the generated API client. Their results are unchanged.

• Observability library: init_simulation_observability drops service_version. None of its four callers or the tests passed it, and build_runtime falls back to the package version. Modal workers still pass a version through init_process_observability.

• Entry:

  • create_app (complexity 52, 144 statements) keeps every route's name, decorators, parameters and registration order. The shared behaviour moves into _EntryRuntime, and Stage 12 error logs still record only the error type.
  • Settings validation is split into caller-auth, old-gateway and Stage 12 checks.
  • The Stage 12 adapter is split into eligibility checks and a report builder.
  • Stage 12 dispatch takes a prepared report, an origin and a worker version.

• Gateway: active-state route resolution is split into pinned, latest and requested-version steps.

• Executor: long argument lists become small frozen groups, including:

  • artifact key inputs;
  • baseline identity requests;
  • precompute function handles;
  • national-reduce inputs;
  • fan-out poll backoff;
  • qualification runners and tolerances;
  • Stage 12 coordinator invocation and dependencies;
  • child spawn requests;
  • worker seams.

  Stage 12 coordination (complexity over 10, 11 arguments, 109 statements) becomes a _ReportCoordination class with claim, plan, dispatch, wait and aggregate phases. The Stage 12 parity checks are split by entity, column and report value. The image artifact fetch is split into freshness, credential and download steps.

• Executor imports:

  • src/modal/v2_app.py already imports policyengine_simulation_executor.stage12_runtime at module level, and that package's __init__ imports its submodules eagerly.
    • So the four in-function imports of the same package loaded nothing new. Their reason, "loaded in the worker container, not at deploy time", did not hold.
    • The Modal function bodies now call stage12_runtime.<name>. Tests that patch stage12_runtime.run_single_simulation or stage12_runtime.coordinate_report still take effect, because the attribute is looked up at call time.
  • src/modal/_image_setup.py imports json, logging, os, pathlib.Path and typing.TYPE_CHECKING at module level.
    • Modal runs these image-build functions from the function's mounted source package; they are not serialized.
    • stage12_cache_image.py, which also runs before add_local_python_source, already imports standard-library modules (json, os, pathlib, typing) at module level.
    • test_image_setup_module_stays_self_contained now allows only standard-library modules at module level (sys.stdlib_module_names), and still rejects any import of the executor package.
  • google.oauth2 and policyengine_uk join banned-module-level-imports, so their in-function imports need no suppression.
    • stage12_cache_image.py was the only application module importing google.oauth2 at module level; the import moves into _storage_client, which is the one function that uses it.
    • Its test patches google.oauth2.service_account.Credentials directly.

• Code from Expose the certified dataset SHA-256 and assert the served identity at deploy #743, Gate releases on a numerical sentinel and make them reversible #744 and Release Stage 12 database sessions and keep original failures visible #789, which merged while this was in progress, is refactored the same way:

  • release_sentinel.check_deployment (JobPolling, a _Deployment with per-case and job-wait steps, HTTPStatus.OK).
  • The routing publisher's main: one command function per mode, with the same precedence. Its --help output is byte-identical.
  • restore_routing_plans.
  • The package updater's bundle_versions and verify_dispatch.
  • open_connection_with_retry (RetryTiming).
  • The workflow-expression and routing-command test helpers.

  Release Stage 12 database sessions and keep original failures visible #789's original-failure reporting and session release are kept inside the new coordination phases.

• Rules tool: takes the Public API's group 8b refactor (Refactor tests, scripts and the rules tool, then select every ruff rule (ALL) policyengine-api#3934). Each repository owns its copy; this one is currently identical. Its tests went from 80 to 115.

• ALL:

  • ruff.toml selects every rule, and ignores only the two that conflict with ruff format (COM812, ISC001) and one of each mutually exclusive docstring pair (D203, D213).
  • The per-file exemptions are unchanged.
  • ALL found nothing that wasn't already fixed.
  • sim.quality.ruff-lint lists ALL.
  • The rules-registry skill doc says every rule applies, and that a ruff upgrade PR fixes any new rule's violations.

Noticed, not changed: in release_sentinel, a job that fails but still returns an object result has its values compared anyway. The new tests pin that existing behaviour. It may be worth confirming it's intended.

Commits added on 2026-10-20 (eef2427)

Rebase. Rebased onto #802 at 85f690c. Two commits had conflicts with #802's new commits:

• Pass the resolved region to _build_simulation … (simulation_runtime.py): kept Enable ruff argument, idiom and pytest-style rules (ARG, B, C4, FBT, PERF, PLW, PT, PTH, RUF, SIM and others) #802's ExitStack structure with this PR's _ENV_VALUE_PREVIEW_CHARS constant.
• Pass the Stage 12 connection retry its clock, sleep and jitter as one timing (database_retry.py): kept RetryTiming without the PERF203 suppression that Enable ruff argument, idiom and pytest-style rules (ARG, B, C4, FBT, PERF, PLW, PT, PTH, RUF, SIM and others) #802 removed.

New commits:

1. Drop init_simulation_observability's never-passed service_version.
2. Pass create_initial_batch_state its route as one BatchRoute.
3. Pass resolve_spm_selection the route's versions as one SPMRouteVersions.
4. Call Stage 12 runtime functions through v2_app's module-level import.
5. Import the image setup module's standard-library modules at module level.
6. List google.oauth2 and policyengine_uk as lazily loaded packages.
7. Update the changelog entry for the removed PLR0913 suppressions.

Testing

At eef2427, each project in an environment synced with uv sync --frozen --extra test --python 3.13:

Project	Result
executor	1,540 passed, 22 skipped, 3 deselected
entry	618 passed
gateway	260 passed, 1 deselected
contract	249 passed
Stage 12 persistence	40 passed
observability	32 passed
fastapi	14 passed

• Earlier work, still valid:
  • The characterization tests added by the refactors are 381 in all.
  • The grouping renamed 7 parametrized test IDs in the entry's deploy-script tests, and 34 in the gateway's SPM route tests.
  • The gateway golden OpenAPI test passes, with tests/golden unchanged. The entry, executor and gateway OpenAPI dumps were byte-identical to the base when last compared, before this update; this update changes no route or schema.
• Lint: ruff 0.15.22 check --extend-select RUF100 . (with ALL) gives 0 findings, and format --check projects/*/src libs/*/src passes.
• Rules tool and guards:
  • The rules tool's 115 unittests pass on Python 3.13.
  • scripts/run_quality_guards.py, with the base ruff-idioms-arguments, passes.
  • Rendering the rules leaves no diff.
• Modal images: files in the executor and gateway images change, all inside function bodies, module helpers and module-level standard-library imports. src/modal/app.py is untouched.
• CI/CD: no workflow changed, and no environment-variable read was added or changed; some reads only moved within changed files. The variables read in changed deployment code:
  • The image artifact fetch (_image_setup.py):
    • POLICYENGINE_VERSION, POLICYENGINE_US_VERSION or POLICYENGINE_UK_VERSION, POLICYENGINE_BUNDLE_RECEIPT and POLICYENGINE_DATA_FOLDER come from the image environment: .env(VERSION_ENV) in src/modal/app.py, applied before the fetch layer.
    • GOOGLE_APPLICATION_CREDENTIALS_JSON comes from the gcp-credentials Modal secret (environment main) attached to that layer.
  • The image-build cache fetch (stage12_cache_image.py):
    • GOOGLE_APPLICATION_CREDENTIALS_JSON comes from the stage12-evaluation-gcp-credentials Modal secret attached to the cache layer.
    • STAGE12_CACHE_MANIFEST_DIGEST comes from the deploy workflow's stage12_cache step. STAGE12_BUNDLE_MANIFEST_SHA256 and POLICYENGINE_DATA_FOLDER come from the image environment.
  • The v2 Modal functions read MODAL_ENVIRONMENT, unchanged, falling back to local. The repository doesn't set it for these functions, so its provider is unverified: it is expected from the Modal runtime.
  • Other changed functions: the release sentinel's SIMULATION_TEST_AUTH_* values come from the SIMULATION_ENTRYPOINT_* secrets through secrets: inherit. RELEASE_PAYLOAD is set from the check-policyengine-updates.yml step env:.
  • Checked: both Modal secrets exist (gcp-credentials in main and staging; stage12-evaluation-gcp-credentials in main, staging and testing). Which keys they hold wasn't checked.
• CI: above the bottom of its stack, so only CI complete runs until the PRs below merge.

🤖 Generated with Claude Code