Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/simulation-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@ jobs:
entrypoint_revision_prefix: s-
entrypoint_min_instances: 0
entrypoint_max_instances: 2
promote_entrypoint: false
# Staging's stable URL must serve the tested revision, because callers
# such as the Public API's staging service use that URL.
promote_entrypoint: true
run_full_integration: true
deploy_existing_stack: true
deploy_stage12_v2: true
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/stage12-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
entrypoint_revision_prefix: s-
entrypoint_min_instances: 0
entrypoint_max_instances: 2
promote_entrypoint: false
promote_entrypoint: true
# This workflow deploys and validates only the independent Stage 12 path.
# The main API deployment continues to run the existing-stack suite.
run_full_integration: false
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
- `sim.deploy.routing-publication-journaled`: Each routing publication is named by its run, attempt and environment and journaled before it is applied; an always-running finalizer resolves it from the journal with the release-gate outcome, a release reports success only after that finalizer completes, and routing jobs never tolerate a failed command. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)
- `sim.deploy.shell-scripts-valid`: Deployment shell scripts have valid shell syntax. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)
- `sim.deploy.smoke-tests-strict`: Candidate and stable smoke tests reject a revision mismatch without retrying, and fail if the previous revision does not converge. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)
- `sim.deploy.staging-promotes-tested-revision`: Every staging deployment promotes its tested entrypoint revision to the staging service's stable traffic, as production deployments do, so callers of the staging stable URL reach the build that passed the staging suite. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)
- `sim.deploy.traffic-changes-exact`: Traffic changes target an exact, validated revision, are reversible, and refuse to run over an intervening promotion. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)
- `sim.deploy.workflow-shell-safety`: Cloud Run deployment never interpolates GitHub expressions into shell, escapes environment values, and pins secret versions. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py`)

Expand Down
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,17 @@ time:
3. publish routing only after all three service jobs pass;
4. run generated-client and live authentication tests through the complete
three-service request path; and
5. begin the `prod` deployment only after every `beta` deployment and
qualification job succeeds;
6. assign stable production traffic to the exact tested entrypoint revision
after the complete production suite passes; and
7. verify the stable endpoint, restoring its previous revision if immediate
post-promotion checks fail.
5. assign the environment's stable traffic to the exact tested entrypoint
revision after that environment's suite passes, in `beta` as in `prod`;
6. verify the stable endpoint, restoring its previous revision if immediate
post-promotion checks fail; and
7. begin the `prod` deployment only after every `beta` deployment,
qualification and promotion job succeeds.

Staging's stable URL therefore always serves the revision that passed the
staging suite, so any caller of that URL, such as the Public API's staging
service or the job store's Cloud Tasks and Cloud Scheduler calls, reaches the
same build.

A successful push-triggered full-stack deployment publishes the API client from
the exact deployed commit.
Expand Down
1 change: 1 addition & 0 deletions changelog_entry.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- Added a rules registry (rules/registry.toml) with a standard-library tool that validates it, checks every rule is enforced where it claims, and renders docs/generated/rules.md and the Rules block in AGENTS.md
- Ran Stage 12 US national simulations as 20 region-group calls each, and built an independent, immutable Stage 12 cache of three annual US datasets and 60 current-law baselines before each deployment, publishing it only after a recorded determinism check and recording its release in the v2 version manifest under a new key, while the existing key keeps a copy that earlier readers accept
changed:
- Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, so the staging stable URL serves the build that passed the staging suite instead of Cloud Run's placeholder revision; staging promotion runs the same stable-URL smoke, release-sentinel and authentication checks and restores the previous revision when they fail
- Enabled ruff's argument, idiom and pytest-style rules (ARG, ASYNC, B, C4, FBT, PD, PERF, PGH, PLW, PT, PTH, RUF, SIM); unused arguments are removed, underscored or explained, FastAPI auth dependencies use Annotated, re-raised errors keep their cause, and boolean options of in-repo helpers are keyword-only; OpenAPI documents are unchanged, and the shared rules tool's unittest tests are exempt from pytest-style rules; tests take the unused-argument rules (ARG001, ARG002, ARG005) as file-level exceptions, since their fakes and stubs mirror real signatures; best-effort telemetry and cleanup calls that swallow any error use contextlib.suppress, with the reason as a comment
- Enabled ruff's structure and import rules (INP, PLC, SLF, N, A, FAST, E); long lines are wrapped, deferred imports that Modal images, test seams, import cycles or heavy dependencies need carry a reason, script directories are declared to ruff as namespace packages, four internal exceptions gained an Error suffix, and published field names, OpenAPI documents and logged error types are unchanged; tests are also exempt from SLF001, INP001 and PLC0415; heavy or optional packages that load only inside functions are listed in ruff's banned-module-level-imports, which keeps them out of module-level imports (TID253) and lets their deferred imports go without a suppression, and the Modal utility programs take INP001 file-wide
- Enabled ruff's security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE); asserts outside tests became explicit checks, command-line scripts keep their printed output, test subprocess calls go through helpers, and every deliberate exception carries a reasoned suppression; the gateway's auth misconfiguration log no longer repeats its message, and the deployed-service tests share the unit-test ruff exemptions; tests, CI/CD scripts and command-line scripts take their deliberate exceptions as registered file-level ruff ignores instead of inline suppressions, and the rules guard rejects file-level ignores anywhere else
Expand Down
8 changes: 6 additions & 2 deletions docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf

| Tier | enforced | ratcheted | advisory | doc-only | Total |
| --- | ---: | ---: | ---: | ---: | ---: |
| mechanical | 40 | 1 | 0 | 14 | 55 |
| mechanical | 41 | 1 | 0 | 14 | 56 |
| metric | 0 | 0 | 0 | 0 | 0 |
| judgment | 0 | 0 | 0 | 0 | 0 |

**Blocking:** 41 of 55 rules.
**Blocking:** 42 of 56 rules.

## Gaps

Expand Down Expand Up @@ -52,6 +52,7 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
- `sim.deploy.routing-publication-journaled`
- `sim.deploy.shell-scripts-valid`
- `sim.deploy.smoke-tests-strict`
- `sim.deploy.staging-promotes-tested-revision`
- `sim.deploy.traffic-changes-exact`
- `sim.deploy.workflow-shell-safety`
- `sim.process.rules-registry`
Expand Down Expand Up @@ -140,6 +141,7 @@ flowchart LR
| `sim.deploy.routing-publication-journaled` | Each routing publication is named by its run, attempt and environment and journaled before it is applied; an always-running finalizer resolves it from the journal with the release-gate outcome, a release reports success only after that finalizer completes, and routing jobs never tolerate a failed command. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |
| `sim.deploy.shell-scripts-valid` | Deployment shell scripts have valid shell syntax. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |
| `sim.deploy.smoke-tests-strict` | Candidate and stable smoke tests reject a revision mismatch without retrying, and fail if the previous revision does not converge. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |
| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment promotes its tested entrypoint revision to the staging service's stable traffic, as production deployments do, so callers of the staging stable URL reach the build that passed the staging suite. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |
| `sim.deploy.traffic-changes-exact` | Traffic changes target an exact, validated revision, are reversible, and refuse to run over an intervening promotion. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |
| `sim.deploy.workflow-shell-safety` | Cloud Run deployment never interpolates GitHub expressions into shell, escapes environment values, and pins secret versions. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** |

Expand All @@ -165,6 +167,7 @@ flowchart LR
r_sim_deploy_routing_publication_journaled("sim.deploy.routing-publication-journaled (enforced)")
r_sim_deploy_shell_scripts_valid("sim.deploy.shell-scripts-valid (enforced)")
r_sim_deploy_smoke_tests_strict("sim.deploy.smoke-tests-strict (enforced)")
r_sim_deploy_staging_promotes_tested_revision("sim.deploy.staging-promotes-tested-revision (enforced)")
r_sim_deploy_traffic_changes_exact("sim.deploy.traffic-changes-exact (enforced)")
r_sim_deploy_workflow_shell_safety("sim.deploy.workflow-shell-safety (enforced)")
r_sim_deploy_container_context_minimal --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
Expand All @@ -183,6 +186,7 @@ flowchart LR
r_sim_deploy_routing_publication_journaled --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
r_sim_deploy_shell_scripts_valid --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
r_sim_deploy_smoke_tests_strict --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
r_sim_deploy_staging_promotes_tested_revision --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
r_sim_deploy_traffic_changes_exact --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
r_sim_deploy_workflow_shell_safety --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py
```
Expand Down
15 changes: 9 additions & 6 deletions projects/policyengine-simulation-entry/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,15 @@ currently serving revision remains unchanged during qualification.

## Promotion and rollback

After the complete production suite passes, the workflow verifies that the
exact tested revision belongs to the production service and is ready. It also
Both environments promote. After an environment's complete suite passes, the
workflow verifies that the exact tested revision belongs to that
environment's service and is ready. It also
checks that stable traffic still points to the revision recorded before
deployment. If those guards pass, it assigns all stable service traffic to the
tested production revision.
tested revision. `beta` promotes before `prod` starts, so any caller of the
staging stable URL, such as the Public API's staging service or the job
store's Cloud Tasks and Cloud Scheduler calls, reaches the build the staging
suite tested.

The workflow then checks the stable generated Cloud Run URL. It uses the
runtime revision header to prove that the stable URL reaches the exact revision
Expand Down Expand Up @@ -82,9 +86,8 @@ The deployment workflow passes each environment's
unset or empty variable deploys `0`, which records callers that lack a route's
permission but lets them through. `1` rejects those callers with 403. Any other
value fails the deployment before Cloud Run changes. A new value takes effect
at the next entrypoint deployment. In `prod` that deployment is promoted to
stable traffic. `beta` never promotes its candidate revision, so a `beta` value
applies only to the tagged candidate that the deployment tests call. The route
at the next entrypoint deployment, which both `beta` and `prod` promote to
stable traffic once their tests pass. The route
permissions are listed in `README.md`.

### Auth0 prerequisites
Expand Down
5 changes: 3 additions & 2 deletions projects/policyengine-simulation-entry/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ Cloud Run's generated revision name is returned in a response header so
deployment checks can prove which revision a tagged or stable URL serves.

Live caller/backend authentication checks are isolated in `authenticated_tests`.
They qualify tagged Cloud Run candidates before production traffic changes and
verify the stable production endpoint immediately after promotion.
They qualify tagged Cloud Run candidates before an environment's traffic
changes and verify its stable endpoint immediately after promotion, in both
staging and production.

## Caller authentication and permissions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -339,8 +339,8 @@ def test_deployment_uses_gcloud_workflow_without_terraform():
assert "deployment_environment: staging" in deploy_workflow
assert "deployment_environment: production" in deploy_workflow
assert deploy_workflow.count("deploy_existing_stack: true") == 2
assert "promote_entrypoint: false" in deploy_workflow
assert "promote_entrypoint: true" in deploy_workflow
assert deploy_workflow.count("promote_entrypoint: true") == 2
assert "promote_entrypoint: false" not in deploy_workflow
assert "entrypoint_environment" not in deploy_workflow
assert "entrypoint_environment" not in reusable_workflow
assert "entrypoint_public_url" not in deploy_workflow
Expand Down Expand Up @@ -647,6 +647,31 @@ def test_revision_tags_fit_existing_cloud_run_service_names():
assert len(service) + len(f"{prefix}9999999") <= 46


@pytest.mark.parametrize(
("workflow_path", "staging_job_name"),
[
(DEPLOY_WORKFLOW, "beta"),
(STAGE12_DEPLOY_WORKFLOW, "deploy-stage12-staging"),
],
)
def test_staging_deployments_promote_the_tested_entrypoint_revision(
workflow_path, staging_job_name
):
"""Staging's stable URL serves the revision that passed staging's tests.

Callers such as the Public API's staging service and the job store's
Cloud Tasks and Cloud Scheduler calls use the stable URL, so a staging
deployment that left traffic on an older revision would leave them testing
a different build.
"""
staging_inputs = _workflow_jobs(workflow_path)[staging_job_name]["with"]

assert (
staging_inputs["entrypoint_service"] == "policyengine-simulation-entry-staging"
)
assert staging_inputs["promote_entrypoint"] is True


def test_entrypoint_container_uses_requested_target_platform():
dockerfile = DOCKERFILE.read_text(encoding="utf-8")

Expand Down
20 changes: 20 additions & 0 deletions rules/registry.toml
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,26 @@ workflow = ".github/workflows/pr.yml"
job = "test"
invocation = "uv run pytest tests/"

[[rule]]
id = "sim.deploy.staging-promotes-tested-revision"
category = "deployment"
statement = "Every staging deployment promotes its tested entrypoint revision to the staging service's stable traffic, as production deployments do, so callers of the staging stable URL reach the build that passed the staging suite."
rationale = "docs/operations"
scope = [".github/workflows/*.yml"]
tier = "mechanical"
status = "enforced"
autofix = false
[rule.enforcer]
kind = "pytest"
project = "projects/policyengine-simulation-entry"
nodes = [
"tests/test_deployment_assets.py::test_staging_deployments_promote_the_tested_entrypoint_revision",
]
[[rule.ci]]
workflow = ".github/workflows/pr.yml"
job = "test"
invocation = "uv run pytest tests/"

[[rule]]
id = "sim.deploy.release-identity-smoke"
category = "deployment"
Expand Down
Loading