Respawn overdue segmented national children and fail before the parent timeout

[anth-volk]

Fixes #794

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Segmented US national runs had no deadline per regional child. A child that never finished held the parent until Modal's 3,600 s timeout killed it, which also left the children running. Both 2023 national reports on 9 October 2026 failed this way.

Changes to src/modal/segmented_national.py:

• Deadline per child: each child records when it was dispatched and its attempt number. A child with no result 900 s after dispatch is cancelled, and its region group is respawned once. The respawn's deadline is limited to the parent's remaining budget. Successful children in the week took a median of 124 s and at most 302 s, apart from one 2,672 s outlier.
• Cancelling replaced attempts: an overdue attempt that a respawn replaces stays tracked. It is never polled again, so a late result from it is ignored. Its first cancel can fail, and Modal keeps re-running a stuck input until a cancel succeeds, so every later failure path cancels it again, and so does the end of collection when the run succeeds.
• Clear failure: the run fails with a message naming the region group when:
  • the respawn is also overdue;
  • the respawn fails;
  • the parent's remaining time can't fit a retry (at least 300 s is needed).
• Fail in code before the Modal timeout: when the parent's budget (the 3,600 s run_simulation timeout minus a 300 s completion margin) runs out, the run cancels every child and then fails.
• Telemetry:
  • segmented_national_child_respawned (warning) and segmented_child_retry_count are recorded only after the respawn spawns. A respawn that fails to spawn is not counted.
  • segmented_national_child_timed_out (error) is emitted when a child misses its deadline and is not respawned. Attributes: region_group, child_attempt, waited_seconds, deadline_seconds and remaining_budget_seconds.
  • segmented_national_budget_exhausted (error) is emitted when the parent budget runs out. Attributes: region_groups and child_attempts (in the same order), waited_seconds since the runner started, and parent_budget_seconds.
  • An outright child failure is still recorded through log_and_redact_exception, with region_group and child_attempt.
• Configuration: the deadline, budget and clock can be set on the constructor. No Modal function settings or environment variables change.

Not included

• Why the 2023 children's containers kept being terminated: this needs Modal container logs. With this change those reports fail after about 30 minutes, with a named region group, instead of being killed at 60 minutes.
• Trace linkage: respawned children start new traces without a parent span.

Testing

• New tests in tests/test_segmented_national.py (16 cases):
  • An overdue child is cancelled and respawned once.
  • A respawn that is also overdue fails the run.
  • A respawn failure is reported as a respawn failure.
  • A respawn that fails to spawn cancels every child and is not recorded as a respawn.
  • A late result from the replaced original is ignored, and the original is not polled again.
  • When the first cancel of the replaced original fails, a later failure cancels it again (2 cases: the respawn fails, or is also overdue).
  • When the first cancel of the replaced original fails and the run succeeds, it is cancelled again after collection.
  • There is no respawn when the parent budget can't fit a retry.
  • A respawn's deadline is limited to the remaining parent budget.
  • An exhausted parent budget cancels the children before the Modal timeout.
  • A respawn is recorded in telemetry.
  • A child timeout is recorded in telemetry (2 cases: the respawn is also overdue, or there is no budget for a respawn).
  • An exhausted parent budget is recorded in telemetry.
  • The parent budget matches the run_simulation timeout in src/modal/app.py.
• Updated test: one existing test in tests/test_canonical_spm.py builds a runner without its constructor and calls _collect. It now uses the new internal handle shape and sets the empty list of replaced attempts.
• Suites, on Python 3.13 with uv run --frozen: projects/policyengine-simulation-executor 749 passed, 23 skipped, 3 deselected.
• Formatting and lint:
  • ruff format --check and git diff --check pass.
  • ruff check reports no new findings. The two in segmented_national.py are the existing pattern that ignores errors from best-effort child cancellation, now in its new location.

🤖 Generated with Claude Code