Repository navigation
fix(daemon): dispatch broker requests off native workers - #4065
Conversation
A broker connection whose request dispatch spends longer than LONG_POLL_CPU in one native poll trips the debug long-poll guard. The guard fails the connection task, so the socket closes after the request was sent and the client sees no response. The test fails until dispatch leaves the native poll (#4056).
A broker connection ran each request's dispatch inside its own native task poll. Under host load that poll spent hundreds of milliseconds of thread CPU on routing and storage work, and in debug builds the long-poll guard failed the connection task after the request was sent. The daemon's HTTP surface reaches the daemon through this socket, so a Workflow call answered workflow.transport_unavailable (#4056). The retained invocation loop now runs every request's dispatch through native_runtime::off_native_thread, the helper #4052 applied to Work and Workflow, so the Unix and portable brokers keep their native polls to framing and socket I/O. Fixes #4056
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| // Dispatch runs synchronous routing and storage work. On a native | ||
| // worker that work holds the reactor (#3994), and in debug builds | ||
| // a long poll fails this connection after the request was sent. | ||
| Ok(request) => match native_runtime::off_native_thread(execute(request)).await { |
There was a problem hiding this comment.
🔴 Parked requests exhaust broker admission slots
When general-client dispatch moves to off_native_thread, long waits retain admission slots instead of releasing them. park_admission cannot see the connection's task-local lease on the blocking-pool thread. Enough parked requests cause unrelated clients to receive capacity refusals.
Learn more
General broker connections install an admission lease in a Tokio task-local scope through with_connection_admission. Long waits use park_admission to release that lease after 50 ms. The blocking-pool task polls dispatch outside the connection task's scope, so CONNECTION_ADMISSION.try_with fails and the wait continues while retaining the slot. The project-open path calls park_admission, so cold invocations encounter this routinely.
Example: Sixty general clients request projects that remain opening beyond the park grace. Their dispatch tasks keep all sixty general slots; a subsequent unrelated general client receives a capacity refusal instead of being admitted while the first clients wait.
Recommended fix: Carry the connection admission lease explicitly into the dispatched future and establish its task-local scope while polling on the blocking thread, or move only the synchronous work off-thread while keeping admission-sensitive waits in the connection task. Cover a cold project-open wait through the actual broker dispatch path with the general admission limit occupied.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // Dispatch runs synchronous routing and storage work. On a native | ||
| // worker that work holds the reactor (#3994), and in debug builds | ||
| // a long poll fails this connection after the request was sent. | ||
| Ok(request) => match native_runtime::off_native_thread(execute(request)).await { |
There was a problem hiding this comment.
🔴 Aborted connections leave requests running during shutdown
When shutdown aborts a broker connection, off_native_thread keeps its dispatch running after the connection releases its activity permit. Shutdown can close project servers and stores while that untracked request continues.
Learn more
The connection task awaits the request through off_native_thread, whose blocking-pool contract explicitly says dropping the wait does not cancel the moved future. The client shutdown owner aborts connection tasks and joins them; the shutdown coordinator then proceeds to project-server drain and store close. Previously, aborting a connection dropped its in-flight dispatch at the next yield. Now the connection can finish and release its lifecycle activity while the dispatch still runs independently.
Example: Shutdown starts while a request is awaiting a long-running project operation on the blocking pool. The client task is aborted and joined, but the operation continues after project-server shutdown begins and can outlive store close.
Recommended fix: Ensure dispatched futures have tracked shutdown ownership and cancellation semantics before aborting their connection tasks. Wait for or cancel all in-flight dispatches before draining project servers and closing stores, while preserving explicit durability guarantees for operations that must complete.
Was this helpful? React with 👍 or 👎 to provide feedback.
Fixes #4056
Root cause
The daemon's HTTP application surface reaches the daemon through its own broker socket. Each broker connection is a native task, and it ran every request's dispatch inside that task's poll: project routing, runtime lookup, and the synchronous storage work around them.
On a loaded host that poll took hundreds of milliseconds of thread CPU. Debug builds fail any native task whose single poll exceeds
LONG_POLL_CPU(250 ms, added for #3994). The guard failed the connection task after the request was written. The in-daemon client read EOF, which it classifies as an indeterminateDaemonInvocationError::Unavailable, and the HTTP surface answeredworkflow.transport_unavailable. The test's secondregister_definitioncall does not retryunavailable, so it failed.This is a production defect, not a test budget. #4052 moved Work and Workflow application bodies off native workers, but dispatch around them, and every other operation family, still ran in the connection's poll.
Fix
drive_retained_invocation_responses, the loop shared by the Unix and portable brokers, now runs each request's dispatch throughnative_runtime::off_native_thread. The connection's native poll keeps only framing and socket I/O. The dispatch closures now return'staticfutures and clone the engine and handshake they need. If the blocking pool refuses the work, the request is answered with a typedUnavailableproblem instead of a closed socket.Evidence
Regression test
daemon::connection_serving::long_dispatch_tests::a_dispatch_longer_than_one_native_poll_still_answers_its_requestspins for 2 xLONG_POLL_CPUinside dispatch, then awaits.spent 500.36ms of CPU in one poll, past 250ms, and the client readsEOF while parsing a value. This is the production symptom.Daemon logs from an instrumented run of the stress command under heavy ambient host load (load average about 1,000) show the mechanism directly:
A second run logged
execute configurationtaking 490.97 ms in one poll, followed byBroken pipeon the client. At moderate load the same dispatch takes at most 7 ms per poll, so the inflation comes from host contention, not from one slow operation.Stress counts for
env -u CC -u CXX bazel test //crates/tracedecay:daemon_suite --test_arg=mounted_fan_out_recovers_then_synthesizes_and_hands_off --nocache_test_results:taskset -c 0-3--default_test_resources=cpu=1transport_unavailableanswers in passing runstaskset -c 0-3The stress command did not reproduce the failure on demand at any load I could control. It appeared only during heavy ambient load from other jobs on the host, which I could not repeat for an after-fix run at the same load. The deterministic regression test is the before/after proof for this mechanism.
Also run on this branch:
//crates/tracedecay:unit_test(pass),//crates/tracedecay:daemon_suite(pass),//crates/tracedecay:transport_acceptance_suite(pass),bazel build --config=clippy //crates/tracedecay:all(clean), and//crates/tracedecay:rustfmt_check_2024(clean).//crates/tracedecay:mcp_suitefailed 2 tests, both on the code-index worker tripping the same guard atcode_index_scheduler/registry/mount.rs:3867. That site is unrelated to this change.Not in this PR
The guard also fails other native tasks under heavy load. The stress runs show these sites: the HTTP router build in
project_composition/full_owner_mount.rs:38(about 100 ms of user CPU at moderate load, and the most frequent trip),code_index_scheduler/registry/mount.rs:3867, andsession_registry/retained_hook_tasks.rs:62. A failed hook task stalled one project open, which produced this test'stimed out waiting for project application mountfailure mode. These sites belong to other owners and are tracked separately.