Skip to content

fix: complete task report waits when the worker stream fails - #880

Draft
gabotechs wants to merge 1 commit into
gabotechs/roundtrip-single-nodefrom
gabotechs/task-reports-on-drop
Draft

gabotechs wants to merge 1 commit into
gabotechs/roundtrip-single-nodefrom
gabotechs/task-reports-on-drop

Conversation

@gabotechs

Copy link
Copy Markdown
Collaborator

Stacked on #879.

The worker->coordinator task only filled in default metrics and completed dynamic filters for a task after its stream ended cleanly. If the stream returned an error, or recording a dynamic filter update failed, the task exited through ? and skipped them. DistributedExec::wait_for_metrics() then hung forever waiting for that task.

Changes

The metrics and completed dynamic filter stores now live in a TaskReports guard that inserts the defaults on drop. The waits complete however the task ends: cleanly, with an error, or dropped before it is ever polled.

Testing

metrics_collected_after_worker_stream_error (in tests/metrics_collection.rs) uses a route handler that fails task 0's worker->coordinator stream after the first output batch. That error arrives before the final reports. Without the fix, the test times out in wait_for_metrics().

🤖 Generated with Claude Code

@gabotechs
gabotechs marked this pull request as draft November 12, 2026 13:05
@gabotechs
gabotechs added this pull request to stack #883 November 12, 2026 13:20
@gabotechs
gabotechs force-pushed the gabotechs/task-reports-on-drop branch from 683ce71 to b5c3f6d Compare November 12, 2026 13:29
The worker->coordinator task only filled in default metrics and completed
dynamic filters after the stream ended cleanly. When the stream returned
an error (or recording a dynamic filter update failed), the task exited
through `?` and wait_for_metrics() hung forever for that task.

Move the stores into a TaskReports guard that inserts the defaults on
drop, so the waits complete however the task ends.
@gabotechs
gabotechs force-pushed the gabotechs/task-reports-on-drop branch from b5c3f6d to 7369648 Compare November 19, 2026 10:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant