Skip to content

Handle temporary worker database unavailability without losing poll identity - #130

Merged
rmcdaniel merged 4 commits into
mainfrom
fix/worker-database-unavailable
Sep 7, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
fix/worker-database-unavailable

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 7, 2026

Copy link
Copy Markdown
Member

Addresses #129. Keep the issue open until the patched releases are published and verified.

  • Classify recognized temporary database connection failures for worker registration, heartbeat and all poll routes.
  • Preserve outcome=unknown and require the same poll identity; do not claim that a lease was never issued.
  • Carry the classification through bootstrap checks and database-backed credential lookup while failing authentication closed.
  • Add HTTP/negative/protocol regressions and prepare Server 2.2.1 / chart 0.1.78 using the existing source-release manifest.

Native MySQL restart proof, Python/Rust published-package interoperability, PHP/framework CI and limitations: #130 (comment)

Companion PHP managed-worker fix: durable-workflow/sdk-php#60. Temporary test containers, volumes, images and scratch files have been removed. No production changes.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Qualification completed 2026-09-07 14:25 UTC.

  • Server candidate: 774b4207; PHP SDK candidate: 94ffc99 (companion Recover managed workers from temporary database unavailability sdk-php#60).
  • Server focused protocol/bootstrap/lease-redelivery checks: 178 tests / 4,737 assertions. Additional runtime-credential and authentication coverage: 68 tests / 3,289 assertions. Full public feature and bounded-growth CI are green.
  • PHP SDK: 527 tests / 2,643 assertions, one existing skip; PHPStan clean. Public PHP 8.1-8.4, Laravel 9-13, embedded migration and Symfony checks passed.
  • Native database-only reproduction used published Server 2.2.0 with the candidate application files, MySQL 8.0.46, Redis 7 and the candidate PHP SDK. Disposable database-backed worker/client credentials exercised the actual authentication lookup, not an auth bypass.
  • A timer-waiting workflow survived MySQL SIGKILL at 14:18:55 UTC and restart at 14:19:01. Query polling returned six typed database-unavailable responses and heartbeat returned three. Every query retry retained the identical poll ID. The original SDK process remained running, then executed the activity at 14:19:17 and completed the same workflow with the expected input hash. No SDK worker restart occurred during this qualifying run.
  • Fresh published Python 2.0.2 and Rust 2.0.1 consumers each received two typed HTTP 503 responses followed by an empty success on workflow, activity and query polling. All six paths made three requests and preserved one poll ID throughout. This proves response interoperability within their configured transport retry budgets, not unlimited outage survival.

Limits: an earlier, longer interruption crossed the default 30-second worker heartbeat freshness fence; the PHP worker correctly stopped under the existing terminal-registration contract. That attempt also showed that a Laravel queue process needs normal process supervision after a lost database connection. The qualifying fixture included that supervision. This change does not remove stale-worker fencing, automatically re-register a stale worker, retry arbitrary SQL/500 errors, or promise recovery beyond every SDK's retry policy. No customer environment was used.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 7, 2026 14:30
@rmcdaniel

Copy link
Copy Markdown
Member Author

Final source checks passed for feature tests, MySQL replay/query topology, MySQL/PostgreSQL predecessor/successor, bounded-growth tests, and chart render/schema validation. The kind install job stopped before cluster creation because the downloaded kind binary failed SHA-256 verification. Retrying only that job with checksum verification unchanged; no product or CI workaround added.

@rmcdaniel
rmcdaniel merged commit 5e12931 into main Sep 7, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants