Skip to content

fix: stop the worker loops from hanging on shutdown under Python 3.11 - #45

Merged
ant1fr merged 1 commit into
developfrom
fix/worker-shutdown-hang
Nov 19, 2026
Merged

ant1fr merged 1 commit into
developfrom
fix/worker-shutdown-hang

Conversation

@ant1fr

@ant1fr ant1fr commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Root cause of the intermittent Python 3.11 CI hang

The first hang captured with the new diagnostics (faulthandler dump, run 44530786436) was stuck in TestClient.__exit__ -> wait_shutdown: the app's lifespan shutdown never finished. The delivery and ephemeral worker loops waited with asyncio.wait_for, which on Python 3.11 can swallow a cancellation when the inner wait completes at the same instant. stop() then awaits a loop that never ends. Python 3.12 reimplemented wait_for on asyncio.timeout, which explains why only 3.11 hung.

Fix

Use asyncio.timeout in both loops (app/services/delivery.py, app/services/ephemeral.py). No other wait_for in app/. In production the same race could have made a container stop wait for the shutdown timeout.

Test plan

  • ruff, format, mypy app
  • delivery / ephemeral / now-playing / queue / scheduler tests (105 passed)
  • CI on 3.11 and 3.12

🤖 Generated with Claude Code

https://claude.ai/code/session_01VzFCH4ztqBYejkvLvFgnaq

The delivery and ephemeral worker loops waited with asyncio.wait_for, which on Python 3.11
can swallow a cancellation (the inner wait and the cancel race). stop() then awaited a loop
that never ended, hanging the TestClient teardown (seen in CI as an intermittent Python 3.11
pytest hang, captured by the faulthandler dump). asyncio.timeout does not have that race.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VzFCH4ztqBYejkvLvFgnaq
@github-actions github-actions Bot added the fix Bug fix label Nov 19, 2026
@ant1fr
ant1fr merged commit bc87438 into develop Nov 19, 2026
5 checks passed
@ant1fr
ant1fr deleted the fix/worker-shutdown-hang branch November 19, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant