[defect] Parallel child completion can leave the parent permanently unresumed (lost wake-up)

[bilyiv]

Owning public component

Workflow engine

Exact version or source identity

2.0.7

Minimal reproduction and evidence

Reproduction

A parent workflow that fans out with all() over child():

$calls = [];
for ($i = 0; $i < 4; $i++) {
    $calls[] = fn () => child(ChildWorkflow::class, $i);
}
all($calls);

ChildWorkflow runs one short activity and returns.

Start three instances of the parent in the same second. Concurrency between closing children is what matters, not width — this reproduces at four children, and reproduced on every attempt at a wider fan-out.

Environment: 2.0.7, PHP 8.4, MySQL, Redis queue driver, multiple concurrent queue workers.

Analysis

On child closure, WorkflowStub decides whether to wake the parent (src/V2/WorkflowStub.php:4523):

&& ! ParallelChildGroup::shouldWakeParentOnChildClosure(
    $parentRun,
    $parallelMetadataPath,
    $childStatus
)

Three arguments, so lockHistoryForUpdate takes its default (src/V2/Support/ParallelChildGroup.php:261-274):

public static function shouldWakeParentOnChildClosure(
    WorkflowRun $parentRun,
    array $metadata,
    RunStatus $closedChildStatus,
    bool $lockHistoryForUpdate = false,
): bool

groupCompletedSuccessfully() then reads the parent's history without the lock, and the flag's own comment describes exactly the case being skipped (src/V2/Support/ParallelChildGroup.php:796-800):

if ($lockHistoryForUpdate) {
    // Observe the resolution event committed by the previous holder of
    // the parent lock even when this transaction has an older snapshot.
    $parentRun->setRelation('historyEvents', $parentRun->historyEvents()->lockForUpdate()->get());
}

So two children closing concurrently can each read a snapshot in which the other's resolution is not yet visible, each conclude the group is incomplete, and neither wakes the parent.

Observed behavior

Three identical parents, each fanning out to four children with all(), started in the same second on 2.0.7:

run	status	ChildWorkflowScheduled	ChildRunStarted	ChildRunCompleted
A	waiting	4	4	0
B	completed	4	4	4
C	waiting	4	4	0

For A and C, externally visible state:

• every child run is Completed with closed_at set
• the parent has no ChildRunCompleted history events at all
• workflow_run_summaries.liveness_state = waiting_for_child, next_task_id = null
• the parent has no task in ready or leased, so nothing will ever resume it
• php artisan workflow:v2:repair-pass reports
  Selected 0 existing task candidate(s) and 0 missing-task run candidate(s)

The parents stay in waiting indefinitely — they were still waiting minutes later, with no further history events. Two of three at width four; reproduced on every attempt at a wider fan-out.

Expected behavior and acceptance criteria

Expected: when the last child of a parallel group closes, exactly one closer observes the completed group and resumes the parent. Concurrency between closers must not allow the group to be seen as incomplete by all of them.

Acceptance criteria:

1. A parent that fans out to N children with all() reaches a terminal status once all N children have closed, regardless of how close together they close.
2. The parent records one ChildRunCompleted (or the matching resolution event) per child.
3. A regression test that closes N children concurrently — committing their resolutions from separate transactions — and asserts the parent is resumed exactly once. The completeness check being made under the parent lock is what makes this deterministic.
4. Defence in depth: a run in waiting_for_child whose child links have all closed, and which has no ready/leased task, is selected by TaskRepairCandidates and recovered. Today that state is indistinguishable from a healthy wait, so a lost wake-up is unrecoverable without manual intervention.

Dependencies and related public issues

#427 — added the parallel barrier so a parent would not resume after the first child of a group. This report is the same barrier failing in the opposite direction: with concurrent closers, no child resumes the parent.

Public intake checks

• I searched open and closed GitHub issues for an existing report.
• This report contains only public product context and redacted evidence.

[rmcdaniel]

Accepted for priority investigation (P1). Thank you for the concrete reproduction and the stuck-run evidence.

Source triage on 2.0.7 confirms an important asymmetry: the service-mode task bridge requests a locking history read at the parallel-child barrier, while WorkflowStub and the embedded WorkflowExecutor do not. The embedded executor also checks the barrier before recording the closed parallel-child resolutions, which is consistent with the reported zero-resolution history and needs its own regression coverage. TaskRepairCandidates currently selects missing-task runs only when their summary is repair_needed, excluding the reported waiting_for_child state.

I am prioritizing a deterministic MySQL/concurrent-transaction reproduction, then a focused fix across the affected completion paths plus recovery for already-stranded parents. The repair must not wake genuinely incomplete child groups or duplicate parent tasks, resolution events, or activity effects. Source inspection is not yet a reproduced or fixed result; I will report the actual regression and recovery checks here.

[rmcdaniel]

The failure is reproduced deterministically on MySQL with separate completion processes. Both children finish, but the unfixed parent has zero resume tasks. Draft #494 now has the implementation under local qualification: persist each child outcome before the shared barrier, use a current locking history read, and let the existing repair path discover and resume already-stranded parents from durable child results.

The expanded MySQL regression set passes (6 tests, 53 assertions), including three parents with four activity-bearing children each, incomplete-group safety, missing-terminal-history safety, and repeatable explicit/watchdog repair. The earlier focused mixed/nested/service-mode checks passed as well (25 tests, 310 assertions). Full repository qualification and a cold-history recovery regression are still in progress; this is not released yet.

[rmcdaniel]

The fix is merged in #494 and published as Workflow 2.0.8. Composer has successfully installed the published package.

The MySQL reproduction, three concurrent parents with four activity-bearing children each, cold repair of already-stranded parents, and the full MySQL/PostgreSQL and Laravel source matrix passed. The existing watchdog/explicit repair path can reconstruct missing parent resolutions from durable child terminal history; completed children are not rerun.

The release is completing its clean published-package Laravel upgrade matrix here: https://github.com/durable-workflow/workflow/actions/runs/34233521331 . I will close this once that verification finishes.

[rmcdaniel]

Released and verified in Workflow 2.0.8, via #494. All published-package Laravel upgrade checks passed: https://github.com/durable-workflow/workflow/actions/runs/34233521331 .

Update the package and restart long-lived queue workers so they load the fix. The existing watchdog and explicit repair path now recover the reproduced stranded-parent state from durable child terminal history, preserving the completed child results. No database migration is required.