Repository navigation
fix: move long native polls off the worker for #4066 - #4071
ScriptedAlchemy wants to merge 3 commits into
Conversation
Graph store attachment awaits LocalStoreRuntimeResolverV1::resolve_graph from native tasks: the code-index scheduler worker and the retained graph-open hook tasks. The resolution canonicalized paths and read and parsed /proc/self/mountinfo inside that task's poll. Under host load the poll passed the 250 ms debug long-poll guard and failed the task. The resolution now runs on the blocking pool. Refs #4066
A parking scheduler worker dropped its seated decode and hibernated its graph engines inside the worker's poll: catalog heap walks, frees, munmap, and generation-marker rewrites. Under host load that poll passed the 250 ms debug long-poll guard, the worker task failed, and the worktree stopped indexing. The worker and the three detached retry tasks now run the release on the blocking pool. Refs #4066
The full-owner mount spawned the HTTP application mount as a native task and built the axum router inside its poll, about 100 ms of CPU in a loaded debug build. Past the 250 ms debug long-poll guard the mount task failed. The router build, for the mount and the cold resolver, now runs on the blocking pool. Refs #4066
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Pull request was converted to draft
|
Closing unmerged. The interrogation of the long-poll guard found that #4071's moves (router build, release_decode_when_parked, resolve_graph_key to the blocking pool) only appeased the debug guard. No measurement showed a latency gain, and the injection knobs it added sit on production types. #4076 turns the guard into a per-site counter with a warning that never fails the task, and floors long-lived runtimes at two native workers so the reactor handoff patch covers them. Future moves should be justified by the counter and lateness canary data #4076 reports (in the daemon and MCP suites, the HTTP application mount at full_owner_mount.rs:38 still takes 100-244 ms of CPU). |
Fixes #4066
Debug builds fail a native task when one poll spends more than 250 ms of CPU (
LONG_POLL_CPU, #3997). Three production tasks still ran synchronous work inside their polls and failed under host load. This PR moves that work to the blocking pool, so each poll only waits.Root cause per site
full_owner_mount.rs:38). The spawned HTTP mount task built the axum router inside its poll.build_http_application_routeris now async and runs the build on the blocking pool. This covers the mount and the cold resolver.registry/mount.rs:3867). Two pieces of inline work ran in the worker's poll:release_decode_when_parkeddrops the seated decode and hibernates graph engines. This walks the catalog, frees and unmaps memory, and rewrites generation markers. The worker and the three detached retry tasks now callrelease_decode_when_parked_off_worker.retained_hook_tasks.rs:62). The session-relation and memory graph-open tasks attach the graph store.LocalStoreRuntimeResolverV1::resolve_graphranresolve_graph_keyinline. That call canonicalizes paths and reads and parses/proc/self/mountinfo. It now runs on the blocking pool. This is the shared root cause for the hook tasks and the worker's activation path.How the sites were found
I added a temporary sampler (not in this PR) to
LongPollGuard. A per-threadCLOCK_THREAD_CPUTIME_IDtimer sentSIGPROFevery 2 ms of CPU inside a native poll, and the handler recorded a backtrace. The samples come from one//crates/tracedecay:mcp_suiterun, with the HTTP fix already in place.registry/mount.rsworkerretained_hook_tasks.rsRegression tests
Each test injects 500 ms of CPU (
LONG_POLL_CPU * 2) into the moved work and checks that the native task still completes. I reverted each fix locally and confirmed that its test fails with the exact guard panic.daemon::http_application_router::tests::a_router_build_longer_than_one_native_poll_does_not_fail_its_taskcode_index_scheduler::tests::residency::a_park_release_longer_than_one_native_poll_keeps_the_worker_indexing: indexes an edit, parks with the long release, then requires a second edit to be indexed. Without the fix the worker dies atmount.rs:3867and the second generation never arrives.store_locator_resolver::graph::tests::a_graph_resolution_longer_than_one_native_poll_does_not_fail_its_taskVerification
//crates/tracedecay-store-runtime:unit_test,//crates/tracedecay-code-index-runtime:unit_test,//crates/tracedecay:unit_test: pass.//crates/tracedecay:mcp_suite: pass (it failed two tests onmount.rs:3867on the fix(daemon): dispatch broker requests off native workers #4065 branch).//crates/tracedecay:daemon_suitefiltered tomounted_fan_out_recovers_then_synthesizes_and_hands_off, the test a dead hook task timed out: pass.bazel build --config=clippyfor//crates/tracedecay:all,//crates/tracedecay-code-index-runtime:all,//crates/tracedecay-store-runtime:all: clean.rustfmt_check_2024for the same three crates: clean.Not changed
resolvein the same resolver does the same kind of synchronous filesystem work. It did not show up at these sites, so this PR leaves it alone.daemon/bootstrap.rs:835,session_temporal_refresh_scheduler/registry.rs:372, andcode_index_scheduler/activation.rs:315.