Early delayed-job delivery can strand embedded activity retries

[rmcdaniel]

Problem

A normal embedded v2 activity retry can become stranded when Laravel delivers its delayed job during the fractional second before the durable task is due. RunActivityTask releases the job, but its $tries = 1 means the next delivery fails with MaxAttemptsExceededException before the activity can run. The durable task stays ready while the workflow stays incomplete.

Observed with published Workflow 2.0.4 (f7b910958a186a11f52e916843f9019ea7b85ad8), PHP 8.4.25, Laravel 13.26.1 (e4a1bc52ef551d52e60244bb004256d6861da7ab), and Laravel's real database queue backed by SQLite. This is not a fake-queue result.

Minimal Reproduction

In a migrated Laravel application using queue.default=database and workflows.v2.connection=database, define a workflow calling an activity with new ActivityOptions(maxAttempts: 2, backoff: [1]). Have the activity throw on its first call and return on its second.

1. Freeze Carbon at 12:00:00.500000, start the workflow, and drain currently available jobs with queue:work --stop-when-empty --sleep=0. The first activity call fails and its retry task is due at 12:00:01.500000.
2. Move Carbon to 12:00:01.000000 and drain. Laravel's integer-second queue timestamp makes the job available. ActivityTaskClaimer correctly reports task_not_due; RunActivityTask calls release(...).
3. Move Carbon to 12:00:02.000000 and drain. The same queue job is now on transport attempt 2, exceeds $tries = 1, and enters failed_jobs without executing the second business attempt.

Observed output:

12:00:00.500000  activity_calls=1 completed=false failed_jobs=0 ready_activity_tasks=1
12:00:01.000000  activity_calls=1 completed=false failed_jobs=0 ready_activity_tasks=1
12:00:02.000000  activity_calls=1 completed=false failed_jobs=1 ready_activity_tasks=1
12:00:03.000000  activity_calls=1 completed=false failed_jobs=1 ready_activity_tasks=1

The failed job reports Illuminate\Queue\MaxAttemptsExceededException: Workflow\V2\Jobs\RunActivityTask has been attempted too many times.

Relevant code: src/V2/Support/TaskDispatcher.php passes the fractional available_at directly as the activity job delay; src/V2/Jobs/RunActivityTask.php combines a one-attempt wrapper with an early-delivery release() path. The business retry policy and transport-delivery attempts are distinct.

Acceptance

• Add regression coverage using the actual Laravel queue worker with fractional-second deadlines, not only direct handle() calls or large clock jumps.
• The configured second activity attempt runs once, the workflow completes, and no transport failure is produced merely because delivery was early.
• Preserve durable activity attempt limits, not-before semantics, duplicate-delivery safety, and bounded handling of genuine infrastructure failures. Do not blindly make business retries unlimited.
• Check database and Redis queue timestamp behavior and other task wrappers that release early deliveries.
• Publish the correction and verify it with the same reproduction.

[rmcdaniel]

The correction is merged in #485 (a24f8c4). The real database/Redis queue reproduction and queue:work case now pass; business attempts remain limited, duplicate deliveries stay harmless, and genuine transport errors keep their existing failure budget. #486 prepares 2.0.5. Remaining completion steps are the full target-branch checks, publication, and a confirming run against the published package. This is release work in progress, not a maintainer-approval blocker.

[rmcdaniel]

Published and verified: https://github.com/durable-workflow/workflow/releases/tag/2.0.5 (6dd2bc3).

A fresh Laravel 13.30.1 consumer (718d17db56861e0a49f644217c8853dab1bff8ce), PHP 8.4.25, real database queue, and watchdog repair disabled reproduced the failure using Packagist 2.0.4. After changing only the installed Workflow dependency to Packagist 2.0.5, the same queue:work experiment passed:

Clock	Activity calls	Completed	Failed queue jobs
12:00:00.500000	1	false	0
12:00:01.000000	1	false	0
12:00:02.000000	2	true	0

The completed output is completed. On 2.0.4 the last row was one call, incomplete, and one RunActivityTask MaxAttemptsExceededException. No source/path override was used for either installed package. Source regression coverage also verifies Redis, duplicates, repeated early timer delivery, unchanged business retry exhaustion and genuine transport failure bounds. Full target-branch checks passed: https://github.com/durable-workflow/workflow/actions/runs/34067771846 . The new release also has its normal published-package/Laravel upgrade verification: https://github.com/durable-workflow/workflow/actions/runs/34068246343 .