Skip to content

fix(daemon): native tasks still trip the long-poll guard under load #4066

Closed

Description

@ScriptedAlchemy

Found while fixing #4056 (PR #4065). Debug builds fail any native task whose single poll spends over 250 ms of CPU (the guard from #3997). Under host load these production tasks still trip it:

  • The HTTP router build in full_owner_mount.rs:38. It takes about 100 ms of CPU at moderate load and is the most frequent trip.
  • The code-index mount at code_index_scheduler/registry/mount.rs:3867. It failed two mcp_suite tests on the fix(daemon): dispatch broker requests off native workers #4065 branch.
  • The retained hook tasks at retained_hook_tasks.rs:62. A dead hook task made mounted_fan_out_recovers_then_synthesizes_and_hands_off time out waiting for the project application mount.

Each one is CPU-heavy synchronous work running inside a native task's poll. Move that work off the native worker, using native_runtime::off_native_thread or spawn_blocking, so the poll only waits.

Activity

  1. ScriptedAlchemy commented on Oct 16, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Resolved by #4076 rather than by moving each site. The debug long-poll guard that failed these tasks is now a counter: a poll past 100 ms of wall time is logged with its spawn site and counted in long_native_polls(), and the poll's result is never discarded. Review of the guard (three independent reviewers) found that its panic caused every failure in this series, that its 250 ms CPU threshold scaled with host load, and that the per-site moves in #4071 only kept it from firing. #4078 reverted the blanket move from #4065 for the same reason.

    With the counter in place, the daemon and MCP suites show two sites past 100 ms: the HTTP application mount at full_owner_mount.rs:38 (100 to 244 ms of real compute, 22 to 23 polls per suite) and the session refresh worker at session_temporal_refresh_scheduler/registry.rs:372 (parked, 3 to 41 ms of CPU). Both are logged, neither fails anything. If the mount's compute shows up as user-visible latency, that is a measured case for one targeted move.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions