Skip to content

Prevent poll backpressure from starving other worker task kinds #66

Description

@rmcdaniel

Problem

The managed PHP worker retries one task-kind poll indefinitely before advancing to the other task kinds. A typed HTTP 429 long_poll_capacity_exhausted response confirms that the workflow poll leased no task, but repeated responses can prevent activity/query polling entirely while worker heartbeats remain healthy.

This is a task-kind liveness defect, not a request to raise Server admission limits or result deadlines.

Reproduction

Observed with published PHP SDK 2.0.1 and 2.0.9 (2.0.9 tag commit 57e95f334639784f2cfc71bc2a40aa583356e271). Use the existing FakeTransport plus injected clock/sleeper:

  1. Register a worker with one workflow and one activity, using run(5).
  2. Return successful registration/heartbeats.
  3. Return the actual Server HTTP429 empty-task response for every workflow-task poll: task=null, poll_status=reason=long_poll_capacity_exhausted, retryable=true, retry_after_seconds=1.
  4. Other task-kind endpoints are available.
  5. Advance the fake clock in the sleeper and request shutdown at60seconds.

Both published versions made 17workflow polls,11heartbeats,0activity polls and0query polls before shutdown. This exercises the installed SDK, not an edited implementation.

Worker::tick() calls pollWithRetry() sequentially. That method remains in its retry loop until the same poll succeeds, so unrelated task kinds never get a turn.

Acceptance

  • Add a focused regression showing that explicit no-task wait-capacity backpressure on one kind does not starve other registered task kinds.
  • Honor the advertised retry delay and retain heartbeat/shutdown responsiveness; no busy retry loop.
  • Preserve poll identity, lease/fencing and ambiguous transport/backend-failure handling. An empty explicit capacity refusal is not equivalent to a lost claim acknowledgement.
  • Test workflow/activity/query directions and multiple backpressured kinds.
  • Inspect the analogous Python/Rust paths for the same mechanism; only claim defects demonstrated there.
  • Publish the PHP patch and update the affected soak consumer to a qualified published artifact. No unchanged SDK releases merely to synchronize numbers.

Found while investigating Server #137. The focused liveness proof does not establish that this caused all six paid-soak timeouts; that run's root cause remains under investigation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions