[anth-volk]

Fixes #835

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Second of two stacked PRs; it is based on #836, which adds ReportExecutionResult, ReportExecutionError and Stage12InvocationContext.result_delivery. This PR adds the return mode to the v2 Modal report coordinator and its simulation workers, for the entrypoint's v2 economy path (a later PR) to call. With result_delivery = return_to_caller, nothing reads or writes Stage 12 rows and no database store is built.

Coordinator (stage12_runtime.coordinate_report, Modal coordinate_report)

• Takes no Stage 12 parent record (passing one raises ValueError) and checks the application, worker version and bundle digest as the Stage 12 path does.
• Claims nothing: a repeated dispatch runs the report again. The caller deduplicates, as the planned job store does with its input hash.
• Writes artifacts under the Stage 12 layout, stage-12-runs/<environment>/<YYYY>/<MM>/<evaluation_id>, built from the context. A path the caller supplies is replaced, as the Stage 12 path replaces it with one built from the parent row.
• Plans the output, starts the baseline and reform simulations with the same delivery, waits for both within the shared window, and reads and checks their artifacts. These steps are the same code the Stage 12 path runs.
• Aggregates, validates the aggregate with the checks a Stage 12 aggregate artifact write applies, requires it to describe this report (report ID, bundle, requested aggregates), validates its result as SingleYearMacroOutput, and returns a ReportExecutionResult payload. It writes no aggregate artifact and does no production comparison.
• A failure is reported to telemetry once, recorded as the operation's cause, and raised as ReportExecutionError with the redacted code and summary a Stage 12 row would record. A worker's own ReportExecutionError takes precedence over its call error, as a Stage 12 worker's recorded failure does.
• A worker that must stop (its peer failed, the wait window closed, or the coordinator failed after starting it) is cancelled through Modal. A cancellation error is logged and never replaces the original failure.

Workers (stage12_runtime.run_single_simulation, Modal run_single_simulation_us/_uk)

• Check the policy periods, write the input and Parquet artifact as before, and return the artifact descriptor.
• A segmented US run is never told to stop through the database. The cancellation check always reports "not cancelled".
• A failure raises ReportExecutionError with the redacted detail, including a segment's failure and typed input errors such as invalid_policy_period.

Stage 12 behavior is unchanged and remains the default. The output planning, child wait, artifact checks and worker calculation move into helpers both modes call. The child wait takes the failure and cancellation handlers as arguments: the Stage 12 path passes its existing record-writing handlers, and the full Stage 12 test suite passes unmodified.

Modal application (src/modal/v2_app.py)

• coordinate_report's parent argument defaults to None. The Stage 12 entrypoint still passes all three positional arguments.
• The coordinator operation of a returned report carries result_delivery=return_to_caller, so telemetry tells it apart from Stage 12 runs. Stage 12 operations keep exactly their current attributes.

Known limitations, for the v2 economy path to decide

• Segment calls of a cancelled segmented worker keep running. Modal does not cancel spawned calls recursively, and in this mode no record can tell the segmented worker to cancel its 20 region-group calls. They run until they finish or reach their 3,600-second timeout. This happens only on failure paths.
• Artifact paths are write-once. A second run with the same report ID writes to the same paths and fails if the bytes differ. The caller should use a new evaluation_id for each attempt.
• Artifacts use the Stage 12 private bucket and its retention. The context still limits retention to 30 days, and the bucket's lifecycle rules (configured outside this repository, not verified) apply. The plan replaces this with SIMULATION_ARTIFACT_BUCKET later.

What a merge deploys

Merging to main runs the complete staging and then production deployment. The redeployed v2 Modal application (policyengine-simulation-v2-py<version>) gains the return mode, but nothing calls it yet. The entrypoint sends no result_delivery, so every Stage 12 report keeps its current behavior. No image, function resource setting, timeout, secret attachment or workflow changes.

Environment variables

This PR adds no environment variable, workflow input or secret. Variables the changed runtime code reads:

• STAGE12_ARTIFACT_BUCKET (both modes, read by artifact_store() in the coordinator and workers). Provider: the GitHub environment variable STAGE12_ARTIFACT_BUCKET, present in beta and prod. Wiring: the deploy workflow passes it in the env of the "Synchronize the separate Stage 12 runtime secret" step. stage12-sync-modal-secrets.sh writes it into the Modal secret stage12-evaluation-runtime, and v2_app.py attaches that secret to every v2 function (worker_secrets). The variable's value was not inspected.
• STAGE12_DATABASE_URL (Stage 12 mode only; return mode never reads it). Provider: the Secret Manager secret named by the GitHub environment variable STAGE12_DATABASE_URL_SECRET_NAME (present in beta and prod). It reaches Modal through the same sync step and the same Modal secret. Unchanged by this PR; the secret's content was not verified.
• GOOGLE_APPLICATION_CREDENTIALS_JSON (both modes, read by setup_gcp_credentials when Stage12ArtifactStore builds its Cloud Storage client, so the return mode needs it for every artifact read and write in the coordinator and workers). Provider: the Secret Manager secret named by the GitHub environment variable STAGE12_GCP_CREDENTIALS_SECRET_NAME, in the project named by SIMULATION_ENTRYPOINT_GCP_PROJECT_ID (both present in beta and prod). Wiring: the same "Synchronize the separate Stage 12 runtime secret" step passes both variables to stage12-sync-modal-secrets.sh, which writes the credential into the Modal secret stage12-evaluation-gcp-credentials; v2_app.py attaches that secret to every v2 function (worker_secrets). Unchanged by this PR; the secret's content was not verified.
• MODAL_ENVIRONMENT (read by the Modal coordinate_report function, which this PR changes, only to label its telemetry; it falls back to local when unset). Unverified: no workflow, script or Modal secret in this repository provides it to the deployed functions, and whether Modal sets it inside containers was not confirmed. Unchanged by this PR.

The calculation both modes run is the code the Stage 12 path already runs, using the datasets and cache built into the worker images at deploy time; this PR changes neither.

Coordination with other PRs

• Draft Refactor within ruff's complexity limits, then select every ruff rule (ALL) #804 (ruff complexity refactor) edits coordination.py, simulation.py, failures.py and v2_app.py, and moves coordination tests into a new test module. A rebase will be needed after whichever merges first.
• Draft Enable ruff argument, idiom and pytest-style rules (ARG, B, C4, FBT, PERF, PLW, PT, PTH, RUF, SIM and others) #802 (ruff argument and idiom rules) edits failures.py, test_stage12_runtime.py and test_stage12_modal_app.py.
• The API v2 household functions work, in progress in parallel, adds functions to src/modal/v2_app.py. A conflict in that file is likely but small: this PR changes only the coordinator's signature, docstring and operation attributes, and one import.

Testing

All new tests are in the executor project.

• test_stage12_runtime.py, returned simulations (6 tests): artifact and descriptor without any database access, a given store left unused, the redacted typed failure, a typed input failure, a segmented run never told to stop, and a segment's failure kept.
• test_stage12_runtime.py, returned reports (14 tests):
  • the full typed result (aggregate, digest, plan, artifacts, provenance), with no aggregate artifact written, both workers started before either finished, and the delivery and Stage 12 artifact path in each worker's context;
  • rejection of a parent record, a Stage 12 report still needing its parent, and dispatch checks before any worker starts;
  • a worker's typed failure kept, with its peer cancelled through Modal, segmented or not;
  • a redacted call failure without detail;
  • the wait timeout cancelling every worker through Modal;
  • a dispatch failure stopping the started worker;
  • a cancellation error that does not replace the failure;
  • rejection of an aggregate outside SingleYearMacroOutput, of one for another report, and of artifacts that do not follow the plan.
• test_stage12_modal_app.py (2 tests): the Modal coordinator runs a returned report without a parent and labels its operation, and Stage 12 operation attributes are unchanged.
• Results: executor suite 1,315 passed, 22 skipped. Contract 221, entrypoint 534, gateway 243 and Stage 12 persistence 19 passed.
• ruff check . and ruff format --check projects/*/src libs/*/src with the pinned ruff 0.15.22 pass. scripts/run_quality_guards.py passes. pyright 1.1.401 reports no errors in stage12_runtime/coordination.py.
• Not run: the Modal image smoke and integration tests, which need Modal and the CI services. CI runs them on the lowest PR in the stack.

🤖 Generated with Claude Code