[anth-volk]

Fixes #244

Cause

The deploy step "Run staged national and state calculations" runs frontend/scripts/deployment_smoke.py --check full. That check sends one request for spm_unit_spm_threshold (national) and ca_income_tax (California) to the staged Vercel deployment, and it never retried (attempts=1).

In deploy run 39472402849 the connection closed without a response 272 s after the request was sent (http.client.RemoteDisconnected). Meanwhile, Modal's log shows the calculation returning 200 after 303.9 s, and Vercel's log also records 200. Every configured limit is higher: 790 s for the client, 800 s for the Vercel route, 780 s for the proxy and 800 s for the Modal function.

RemoteDisconnected is a ConnectionResetError, not a URLError, so the script did not catch it at all. That also means the six-attempt loops of the metadata and state checks never retried it.

Design

1. Cache state seen by a retry

frontend/scripts/microcosm_variable_core.py keeps at most one simulation:

• _simulation_for clears _SIM_CACHE before building a new scope.
• simulation_cache_hits[scope] records whether that scope's simulation was already cached when the request first used it.
• simulation_cache_evictions lists the scopes released to build another one.
• Variables run in request order, national first.

So the abandoned request leaves the California simulation in the cache. A retry in the same warm container then:

1. releases California and builds national (national miss, evictions ["CA"]);
2. releases national and builds California (CA miss, evictions ["CA", "national"]).

The existing validator already accepts this, because both scopes are misses and "national" is in the evictions. In a new container, the evictions are ["national"], which also passes.

The validator is therefore unchanged, and each response, first or retried, must prove the scope switch on its own. A retry that reports the national simulation as reused, or never released, is rejected; both cases are tested.

The only path to a cached national simulation is an abandoned request that failed after building it. Nothing else uses the staged commit-specific Modal app, and the metadata requests do not touch the cache. If that happens, the retry fails, which is correct.

2. Queueing: wait for a metadata answer, not a fixed time

backend/modal_app.py runs one container that serves one input at a time (max_containers=1, @modal.concurrent(max_inputs=1)). The route's proxy does not pass the client's abort signal to Modal, so the abandoned calculation keeps running.

After a lost connection, the check sends metadata=1 requests. These go through the same route to the same container, so one is answered only after the abandoned calculation has finished. The answer must also pass validate_metadata. The full calculation is then sent once more.

Why this instead of a fixed wait:

• In run 39472402849, the backend became free 32 s after the drop. A fixed wait long enough to cover a call that runs to Modal's 800 s timeout would add up to about nine minutes every time.
• A wait that runs past scaledown_window=60 lets the container shut down. The retry would then also pay for a cold start, which makes its idle connection longer; that is the condition that failed.
• The metadata answer re-checks the source commit, packages and data selection of the staged deployment.

A metadata request whose connection is lost is retried, with a 10 s pause, until 860 s after the first full request: Modal's 800 s function timeout plus 60 s for container start. Each probe's socket timeout is limited to the time left. Any HTTP error or invalid metadata fails at once.

Time bound: the step's longest possible run grows from 790 s to about 1,650 s (860 s by which the backend must be free, plus one 790 s retry), so at most about 14 minutes more. In today's four runs, the steps before this one took about 2.5 to 3 minutes and the steps after it about 20 seconds. The job therefore stays at or below about 31 minutes, inside timeout-minutes: 45. Had this code run in 39472402849, the step would have taken about 272 s + 32 s + one warm calculation (roughly 190 to 300 s).

3. Which errors are retried

_request_json now raises ConnectionFailedError, a RuntimeError subclass, only for a ConnectionError (refused, closed, reset or aborted; RemoteDisconnected is a ConnectionResetError) or a TimeoutError, whether raised directly or wrapped by urllib in a URLError. A URLError for a DNS, TLS-certificate or URL problem stays a plain RuntimeError (changed in review; see "Review fixes"). HTTPError, which is a URLError, is handled first and stays a plain RuntimeError.

For --check full, only ConnectionFailedError leads to the wait and the single retry. An HTTP 4xx or 5xx response with a body, a non-JSON response, or a response that fails validation still fails the deploy at once, after one request.

A slow backend is not hidden: the proxy answers HTTP 504 after 780 s, before the client's 790 s timeout, and that 504 is not retried. One limit remains: the outcome of the abandoned calculation itself is not observed. Only the retried response, validated in full, is accepted.

Side effect: the metadata and state checks already retried every RuntimeError up to six times. They now also retry a connection closed without a response, which previously escaped their retry loops. A test covers this.

frontend/HOSTED_RUNTIME.md describes the retry in the deployment steps.

Commits and test results

Commit	Change	frontend/tests/test_deployment_smoke.py
a326464	Tests only	12 failed, 11 passed (8 existing tests and 3 new no-retry tests, which the old code already met because it never retried)
363fc58	Fix and documentation; tests unchanged	23 passed

Failure reasons on a326464:

• 7 tests failed with http.client.RemoteDisconnected: Remote end closed connection without response, the same exception as the failed deploy, because there was no retry: retry once, keep waiting through lost probe connections, the two rejected cache states, the retried connection closing again, the probe reporting another build, and the metadata check.
• 4 tests failed with AttributeError: module 'scripts.deployment_smoke' has no attribute 'ConnectionFailedError': the classification of RemoteDisconnected, ConnectionResetError, URLError and TimeoutError.
• 1 test failed with AttributeError: ... no attribute 'BACKEND_FREE_DEADLINE_SECONDS': the limit on the wait.

The full-check tests use a loopback HTTP server on 127.0.0.1. It closes the first connection without a response and answers later requests from a fixed sequence of replies.

Checks run: pytest frontend/tests/test_deployment_smoke.py (23 passed), and ruff check and ruff format --check on both changed Python files.

Deployment wiring

.github/workflows/deploy.yml and .github/scripts/deploy-dashboard.sh are unchanged, so this does not overlap with #234. The script reads no environment variables. The metadata requests reuse the same headers as the calculation, including x-vercel-protection-bypass. That header comes from --vercel-bypass-secret "$VERCEL_AUTOMATION_BYPASS_SECRET", which is the GitHub Actions secret VERCEL_AUTOMATION_BYPASS_SECRET, set in the step's env:. The step "Verify staged frontend metadata" used the same secret successfully in run 39472402849.

Review fixes

The review found one low-severity bug. Every URLError counted as a lost connection, including a failed TLS certificate check, a failed DNS lookup and a missing host. On the first full request, such an error made the check warn that it was waiting for a calculation that had never started. It then sent a metadata request every 10 s for up to 860 s, and only then failed. Before this PR, it failed at once.

Commit	Change	frontend/tests/test_deployment_smoke.py
fd2d1ed	Tests only	5 failed, 23 passed
7845dd0	Fix and documentation; tests unchanged	28 passed

Failure reasons on fd2d1ed:

• 3 tests: a URLError for a TLS certificate failure, a DNS failure and a missing host was raised as ConnectionFailedError instead of a plain RuntimeError.
• 1 test: a raw ConnectionAbortedError escaped _request_json without being classified.
• 1 test: the full check sent a metadata request after a TLS certificate failure (['full', 'metadata'] == ['full'] failed).

The fix counts a failure as a lost connection only when it is a ConnectionError or a TimeoutError, raised directly or as a URLError's reason. The metadata and state checks are unaffected, because they retry every RuntimeError. A temporary DNS failure on the first full request now fails the deploy at once, as it did before this PR. frontend/HOSTED_RUNTIME.md now mentions a refused connection, says "complete response", and names DNS and TLS-certificate failures among the errors that are never retried.

Checks run: pytest frontend/tests/test_deployment_smoke.py (28 passed), and ruff check and ruff format --check on both changed Python files.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 20, 2026 7:23pm UTC