Repository navigation
fix(runtime): count long native polls instead of failing the task - #4076
Merged
Merged
Conversation
The native blocking pool built its threads with std::thread::Builder defaults, 2 MiB, while native workers run on the CLI's 16 MiB ASYNC_STACK_BYTES. Handler futures moved off a worker through off_native_thread ran on that smaller stack. A second asupersync patch passes the runtime's thread_stack_size to BlockingPoolOptions and applies it to every pool thread.
The debug-build long-poll guard asserted after a poll had already finished, so it discarded the poll's result and failed tasks whose side effects had happened: a broker connection closed after its request was sent, a code-index worker parked failed, a schema convergence reported failure. Its 250 ms thread-CPU bound also moved 70x with host load and opt level. LongPollGuard now runs in every build, measures wall time and (on unix) thread CPU, and on a poll past LONG_POLL (100 ms wall) logs a warning and counts it against the spawn site in long_native_polls(). It never changes the poll's result. The daemon's runtime hazards line lists those sites. Beside it, so long polls can be judged by their effect: - spawn_lateness_canary() measures how late native timer wakes run; the daemon runs it for its lifetime and reports late wakes at exit. - spawn_blocking runs work inline when called on a blocking-pool thread that already runs queued work, so a full pool cannot deadlock on nested blocking work. - MAX_BLOCKING_THREADS describes the native pool it bounds. The long-dispatch test #4065 added only exercised the guard and goes with it.
The reactor handoff patch promotes a parked follower only when a runtime has two or more parkers, so on a one-CPU host daemon run and serve held every socket wake and timer behind any long poll. Commands other than the one-shot tool and status clients now start at least two native workers. The native blocking pool takes MAX_BLOCKING_THREADS directly instead of max(tokio limit, 512), which always resolved to 512.
spawn_detached, spawn_maintenance_task, spawn_doctor_task, and LspRuntimeSpawner::spawn wrap NativeTask::spawn without #[track_caller], so every task they start counted against the wrapper's line. They now pass their caller's location through.
|
ScriptedAlchemy
enabled auto-merge
October 16, 2026 03:14
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
a_memory_activation_longer_than_one_native_poll_still_seats (#4022) and schema_convergence_completes_when_its_allocator_trim_outlasts_a_native_poll (#4027) inject CPU so a poll trips the guard. With long polls counted instead of failed, both pass whether or not the prewarm and the trim stay on a worker.
This was referenced Oct 16, 2026
…ong-poll-counter # Conflicts: # MODULE.bazel
…ong-poll-counter # Conflicts: # crates/tracedecay-rusqlite-runtime/src/native_runtime.rs
…ong-poll-counter # Conflicts: # crates/tracedecay/src/daemon/bootstrap.rs
NativeTask::spawn_off_native_thread polls its work on a blocking-pool thread without marking it, so spawn_blocking called from that work queued behind the thread it held. It now marks the thread like queued blocking work does.
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A native task that took a long poll no longer fails. The debug-build long-poll guard asserted after the poll had already finished, so it threw away a completed result: a broker connection closed after its request was sent (#4056), a code-index worker parked failed (#4023), a schema convergence reported failure. Every flake in that series came from the guard, and its 250 ms thread-CPU bound moved about 70x with host load and opt level (7 ms of CPU at normal load, 490-590 ms at load ~1000, fastbuild opt-0).
An adversarial review of the guard, the reactor patch, and the per-site moves (#4022, #4027, #4052, #4065, #4071, #4072) converged on subtraction. With the reactor handoff patch, a long poll on a runtime with two or more workers no longer holds the reactor, so the guard protected nothing there. The remaining gap is a one-worker runtime. This PR turns the guard into evidence and closes the one-worker gap structurally.
Scope
native_runtime::LongPollGuardruns in every build. It measures wall time and, on unix, thread CPU. A poll pastLONG_POLL(100 ms wall) logs atracing::warn!with the spawn site and is counted inlong_native_polls(), keyed by spawn site. The poll result is never changed. The daemon'sdaemon_runtime_hazardsline lists the sites.native_runtime::spawn_lateness_canary()measures how late native timer wakes run (scheduling_lateness()). The daemon runs it for its lifetime and reports late wakes at exit.native_runtime::spawn_blockingruns work inline when called on a blocking-pool thread that already runs queued work. Before, nested blocking work could deadlock a full pool.asupersync-blocking-pool-stack.patch, gives blocking-pool threads the runtime'sthread_stack_size. They were on the std default of 2 MiB, not the CLI's 16 MiB.tracedecay-cli:daemon run,serve, and other non-one-shot commands start at least two native workers. The reactor patch promotes a follower only when there are at least two parkers (count < 2returns early).toolandstatusstay at one. The native pool takesMAX_BLOCKING_THREADSdirectly;max(tokio limit, 512)always resolved to 512.#[track_caller]onspawn_detached,spawn_maintenance_task,spawn_doctor_task, andLspRuntimeSpawner::spawn, so counts name the real site.long_dispatch_tests, fix(code-index): prewarm memory activations on the blocking pool #4022'sa_memory_activation_longer_than_one_native_poll_still_seats, and fix(runtime): trim off the native poll and park panicked workers #4027'sschema_convergence_completes_when_its_allocator_trim_outlasts_a_native_poll. Also drops a staleclippy::disallowed_methodsallow.Blast Radius
Every
NativeTaskpoll now pays two monotonic clock reads and one thread-CPU clock read. Pool threads reserve 16 MiB of virtual stack each instead of 2 MiB; resident memory is unchanged. No behavior depends on the counters; they are evidence for future per-site moves.Verification
//crates/tracedecay-rusqlite-runtime:unit_test,//crates/tracedecay-cli:tracedecay_unit_test,//crates/tracedecay-code-index-runtime:unit_test,//crates/tracedecay-lsp:unit_test,//crates/tracedecay:unit_testpass.Nonewhen the guard stops counting; the inline test fails withNonewhen work is queued; the canary test fails withlate_ticks: 0when nothing is recorded; the stack test fails withpool thread stack is 2097152 byteswithout the patch.--config=clippyand rustfmt are clean for the touched crates.//crates/tracedecay:daemon_suite(77 tests) and//crates/tracedecay:mcp_suite(666 tests) pass. A local, uncommitted hook logged each counted poll during those runs. Two sites still pass 100 ms:daemon/project_composition/full_owner_mount.rs:38, the HTTP application mount: 45 polls, 100-244 ms wall, CPU equal to wall, so the work is compute.session_temporal_refresh_scheduler/registry.rs:372, the session refresh worker: 3 polls, up to 242 ms wall at 3-41 ms CPU, so the poll is parked, not computing.