Skip to content

Return retryable worker responses during temporary database unavailability #129

Description

@rmcdaniel

Observed Failure

With published Server 2.2.0 and PHP SDK 2.0.4, stopping and restarting a disposable MySQL 8.0.46 process while a worker polls query tasks produces a generic HTTP 500 {"message":"Server Error"}. Server logs show SQLSTATE HY000 / driver 2002 (connection refused) while reading workflow_worker_registrations, before a task is leased.

The PHP worker exits with ServerException instead of surviving the brief database interruption. The database became writable again in about five seconds; completed workflow data remained intact. A waiting workflow resumed and completed only after explicitly restarting the SDK worker.

The SDK deliberately retries connection failures and protocol-classified retryable poll responses, not arbitrary HTTP 500s. This appears to be a missing Server dependency-failure classification rather than a reason to retry every server error in each SDK.

Reproduction

  1. Run published Server 2.2.0 with disposable MySQL 8.0.46 and Redis.
  2. Register a PHP SDK 2.0.4 worker with a workflow, activity and query handler.
  3. Reach a durable waiting workflow and leave the worker polling.
  4. Kill and restart only the disposable MySQL process.
  5. Observe the query poll's HTTP 500 and worker exit. Once MySQL recovers, explicitly restart the worker; query, signal and completion then work.

Source inspection confirms PollResponse::isTransientFailure rejects generic 500s. The original experiment also applied disk pressure, but the captured poll failure itself is a database connection refusal at restart, not an EDQUOT error. Add a minimal database-only reproduction to separate those conditions.

Acceptance

  • Temporary database connection loss before task selection has an explicit, sanitized, retryable protocol response with bounded backoff.
  • Preserve fail-closed authentication and task identity. Do not pretend a lease was never issued when the database outcome is ambiguous.
  • Cover workflow, activity and query polling, plus registration/heartbeat as needed for actual worker recovery.
  • Keep permanent authentication, schema, programming and storage-capacity errors distinct; do not turn every SQL exception or HTTP 500 into an infinite retry.
  • Prove the worker stays alive and a real workflow completes after a temporary database interruption without manual worker restart. Check the shared response with all first-party SDKs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions