[ScriptedAlchemy]

Refs #3736

T2-S5 of the Tokio to Asupersync cutover: the daemon's background owners. This branch is based on master after #3964 (T2-S4) and #3883 (nightly toolchain), and merges master up to a3028ce.

What changed

A spawn moved to NativeTask only when its future reaches no Tokio runtime API. The trace compared each future's callees against the production Tokio inventory of the workspace: every tokio::spawn, JoinSet, Tokio timer, tokio::net, tokio::fs, Handle and native_runtime::tokio_handle site outside test code.

Site	Result	What the trace found
scheduler.rs automation loop (start_automation_scheduler)	Native	The loop reads the automation configuration, runs the tick and the host-receipt review, and commits its exit under the writer scope. The tick reaches tracedecay-automation-runtime (blocking work through native spawn_blocking), tracedecay_daemon_service::automation_effect (retained request context and memory owner reads), and the registered Codex prompt runner, which is synchronous. None of these has a Tokio site. Its timers were already native.
branch_admin.rs retirement reaper	Native	The reaper waits for the retired loop and then runs the scheduler-map cleanup, which only takes locks.
branch_admin.rs manual branch publication JoinSet	Native (Vec of abort-on-drop NativeTasks)	A publication runs branch preparation, the PR-head activation in pr_autotrack, and the code-index scheduler mount. None of these has a Tokio site.
maintenance.rs resident-memory sampler	Native	The sampler waits on a native timer and runs its sample on native spawn_blocking.
maintenance.rs publication due-request task	Native	The task selects on the cancellation token and a tokio::sync::broadcast receiver. Both work on any executor.
maintenance.rs retention loop	Native	A tick reaches tracedecay-maintenance, the code-index scheduler registry, store telemetry and branch-store GC (execute_branch_admin_in_layout, then pr_autotrack retirement). None of these has a Tokio site.
pr_autotrack/runtime.rs loop	Native	The loop lists projects, runs Git and GitHub discovery, and mounts or retires PR worktrees through the code-index scheduler.
project_retirement.rs retirement tracker (:223)	Native	The tracker waits for one retirement and publishes its receipt. Awaiting a Tokio JoinHandle works on any executor.
project_retirement.rs spawn_and_track (:97) and fallible retirement (:142)	Tokio	Both retirement futures call project_server_lifecycle::retire_project_servers.
remote_recovery_lifecycle.rs:259	Tokio	The future calls project_server_lifecycle::retire_project_servers_now.
hook_v2_replay_consumer.rs consumer	Tokio	Replay admits each envelope through admit_hook_v2_envelope. That calls admit_registered_hook_orchestration, which needs the Tokio handle at tokio_handle("daemon-service.hook_orchestration.admit"). The consumer's timer was already native.
spool_opener.rs opener	Tokio	The opener opens projects, and project open spawns on Tokio at project_open_admission.rs:635. It has no timer.

Behavior notes:

• Refusals are typed. When the native runtime refuses a spawn, the caller now gets a typed result:
  • Scheduler start returns AutomationSchedulerReconcileOutcome::OwnerUnavailable and logs scheduler_start outcome=runtime_unavailable.
  • pr_autotrack::spawn_with_administration returns Err, and bootstrap logs it and runs without the loop.
  • The maintenance owners log maintenance_task_unavailable with the owner name.
  • A manual branch publication returns a retryable branch_tracking_failed error.
  • The retirement tracker publishes a failed receipt.
  • The reaper logs retirement_reaper outcome=runtime_unavailable.
• Abort-on-drop owners are kept. PrAutotrackTask still aborts its loop on drop. A dropped manual-publication owner still aborts its tasks, as the dropped JoinSet did, and a cancelled shutdown join aborts the publications it had not joined. Both are proven by new tests with mutation checks (see Tests).
• Two seams were simplified. The reaper registry is no longer locked while the reaper spawns, because a refused spawn drops the finalizer, which takes that lock. ProjectServerRetirement::_task and MaintenanceReaperHandle::_task are gone: dropping a JoinHandle only detached the task, so holding the handle did nothing.
• One tracker was replaced. The only production caller of track_aborted_retirement_task passed the automation loop. It is replaced by track_aborted_native_retirement_task.

Waiting on project_server_lifecycle.rs

This PR does not read or edit project_server_lifecycle.rs. Three seams stay on Tokio until that file's timers move (scope commit 3b):

• ProjectServerRetirementAdmission::spawn_and_track (project_retirement.rs:97). Its future calls retire_project_servers.
• spawn_fallible_retirement (project_retirement.rs:142). Its capacity-retirement future calls retire_project_servers.
• retire_runtime_work (remote_recovery_lifecycle.rs:259). Its future calls retire_project_servers_now.

track_retirement_task keeps its tokio::task::JoinHandle parameter, so callers in that file compile unchanged.

Tests

Each off-Tokio test runs inside native_runtime::block_on, or after a Tokio setup on a plain thread, and asserts Handle::try_current().is_err(). Each test reads test_support::native_runtime_hazards() before and after the run and asserts that it did not change. That helper sums tokio_runtime_missing_count over the tokio_handle sites and adds exact_sql::native_thread_writer_waits. This PR adds the four remaining tokio_handle sites to the sum: work_attempt.spawn_for_worktree, host_admission.bootstrap, shutdown.runner and shutdown_coordination.runner.

Red-first: each test below was run on master with the test adapted to master's signatures. Each failed:

pr_autotrack spawned loop:   panicked at crates/tracedecay/src/daemon/pr_autotrack/runtime.rs:65:16:
                             there is no reactor running, must be called from the context of a Tokio 1.x runtime
maintenance coordinator:     panicked at crates/tracedecay/src/daemon/maintenance.rs:492:23: there is no reactor running ...
automation loop start:       panicked at crates/tracedecay/src/daemon/scheduler.rs:774:20: there is no reactor running ...
manual branch publication:   panicked at crates/tracedecay/src/daemon/branch_admin.rs:664:19: there is no reactor running ...
retirement tracker:          panicked at crates/tracedecay/src/daemon/branch_admin/project_retirement.rs:223:16: there is no reactor running ...
automation retirement reaper: //crates/tracedecay:unit_test TIMEOUT in 60.0s (the retirement never settles)

On this branch, these tests pass:

• pr_autotrack::tests::spawned_loop_is_cancellable_and_joinable_without_a_tokio_runtime (rewritten from the Tokio variant).
• pr_autotrack::tests::reconcile_activates_a_pr_head_without_a_tokio_runtime: the tick's reconcile checks out PR 11 and mounts its worktree off Tokio.
• maintenance::tests::the_coordinator_runs_and_shuts_down_without_a_tokio_runtime.
• maintenance::tests::a_full_tick_collects_an_unmounted_registered_project_without_a_tokio_runtime (the full-tick test, moved off Tokio).
• tests::scheduler_config::the_automation_loop_starts_and_exits_without_a_tokio_runtime: a reconcile with no Tokio runtime returns Started. The loop reads its disabled configuration and records the exit decision at the existing exit barrier.
• tests::scheduler_shutdown::automation_retirement_reaps_without_a_tokio_runtime.
• tests::scheduler_shutdown::manual_branch_publication_runs_and_joins_without_a_tokio_runtime.
• branch_admin::project_retirement::tests::a_tokio_retirement_is_tracked_and_settled_without_a_tokio_runtime.
• branch_admin::project_retirement::tests::an_aborted_native_owner_settles_its_retirement_clean.

Abort-on-drop proofs, with mutation checks. The checks emptied PrAutotrackTask::drop and ManualBranchPublication::drop. Both tests then failed:

• dropping_the_pr_autotrack_owner_aborts_its_parked_loop: the dropped owner aborts its loop: Elapsed(()).
• dropping_the_administration_aborts_an_owned_manual_branch_publication: the dropped owner aborts its publication: Elapsed(()).

Rewritten tests:

• The scheduler-map tests now give the handle a NativeTask. A noncooperative owner blocks a worker of its own one-worker native runtime until released (spawn_noncooperative_native_test_task), so the process runtime is never short a worker. NativeTaskProbe reports when the task's future was dropped. It replaces the Tokio AbortHandle::is_finished checks.
• cancel_stops_the_automation_loop_while_the_scheduler_map_is_held now waits for the loop's own end instead of polling is_finished for 2 s. In one full-suite run the 2 s poll failed under load. The loop's tick is 60 s, so the 20 s hang guard still fails when the cancel does not reach the loop. For a mutation check, the stop request and wake were removed from cancel_automation_schedulers. The test then failed with a cancel issued while the scheduler map was held must still end the loop.
• The maintenance abort-deadline tests drive NativeTasks.

The scheduler tick also ran off Tokio during the first run of the loop test: the project-open loop ticked memory curator, session reflector and skill writer on the native runtime with no panic.

Gates

These gates ran on the final merge (066a946):

• bazel build --config=clippy //crates/tracedecay/...: Build completed successfully.
• bazel build //:rustfmt_check and //crates/tracedecay:rustfmt_check_2024: Build completed successfully.
• bazel test //crates/tracedecay:unit_test: PASSED, 881 passed.
• bazel test //crates/tracedecay:daemon_suite: PASSED, 75 passed.

These gates ran on f9f5212, before the last two master merges. This PR's production code has not changed since:

• //crates/tracedecay:mcp_suite: PASSED, 661 passed.
• //crates/tracedecay:session_suite: PASSED, 418 passed.

Two failures did not reproduce:

• twelve_project_journey_retires_idle_owners_without_empty_graphs failed once in a full unit_test run with reopening a retired project must restore its Hook V2 replay consumer. It passed in 3 isolated reruns and in the next full run. shutdown_hook_v2_replay_consumer removes the registration by data root without checking which consumer owns it. A late retirement can therefore remove a newer consumer's registration. This PR does not change that code.
• daemon_suite rejected_token_reads_typed_refusal_and_daemon_keeps_serving failed once with write handshake: BrokenPipe. The test client pipelines its writes after the daemon has refused and closed the socket. It passed on rerun.

---

[changeset-bot]

⚠️ No Changeset found

Latest commit: 066a946

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔴 Rejected reaper strands project automation

When NativeTask::spawn refuses a reaper, the retired scheduler remains registered as Retiring without a task. Later retirement attempts cannot clear it, so project automation cannot restart.

Learn more

Retirement first moves the scheduler task out of its registry entry and marks that entry Retiring in retire_matching_automation_scheduler. Normally, the reaper joins the task and runs the cleanup future, which removes that entry. If native spawn refuses, the finalizer signals termination but never runs cleanup. The entry retains Retiring with task: None; later retirement attempts cannot create another reaper, and scheduler reconciliation returns Retiring.

Example: A project's automation loop starts retiring as the native runtime stops accepting tasks. Reaper spawn fails; the retirement waiter completes, but the project's next automation reconcile still reports Retiring indefinitely.

Recommended fix: Make spawn refusal run or schedule the cleanup after the retired task has actually settled, retaining ownership until then. Avoid publishing termination as completed before the scheduler tombstone is removed; validate a retry can start automation after a refused reaper spawn.

---

Was this helpful? React with 👍 or 👎 to provide feedback.