Skip to content

[Bug]: Stopping a busy runtime times out and permits replacement while the old agent keeps running #583

Closed

Description

@jyaunches

Affected area

Rust core or runtime; Python SDK or binding; Harness adapter; Consumer integration

NeMo Fabric version

Observed live: a3b57155593877135f2557ef37d5e18161a1ad82 from cv/NeMo-Fabric. The relevant runtime-server locking and Deep Agents Code session ownership behavior remains at 207077ba951ee1a01da3ac241eccb77378da0888 on feat/interfaces-deepagents-code.

Installation method

Other: Fabric installed in the NemoClaw agent image.

Current behavior

Part of #577. Downstream evidence: NVIDIA/NemoClaw#14115.

Stopping a busy Deep Agents Code runtime returned invalid_response at stage transport, with effects unknown, after approximately 321 seconds. Health reported runtime_unavailable, but the original dcode ACP process continued its turn. Unchanged NemoClaw plan/apply started another runtime in the same sandbox. Both ACP processes remained alive; the old turn held the single vLLM sequence until the runner terminated its process.

Source tracing identifies a shared Fabric runtime-server problem before adapter shutdown:

  1. RuntimeServer.handle() sets the record to stopping before acquiring owned.lock. Invocation holds that lock throughout runtime.invoke(), so stop waits for the active turn rather than cancelling it.
  2. The control transport has a 320-second request deadline. Expiration after sending a lifecycle request maps to invalid_response with unknown effects; disconnecting the caller does not cancel server work.
  3. Runtime health maps a record whose state is not running to runtime_unavailable, even while its process is still running.
  4. Starting another runtime is allowed while the old runtime is stopping.

An isolated local probe executed the fork's actual RuntimeServer and OwnedRuntime class bodies with fake runtime dependencies. While invocation was blocked, stop remained pending, adapter stop calls stayed at zero, and a replacement started. Releasing invocation finally allowed stop to run. This confirms server sequencing; it is not a fresh live dcode or GPU reproduction.

Related adapter defect: DeepAgentsCodeAdapterRuntime.stop() clears this.session before awaiting session.stop(), losing its cleanup retry handle if shutdown fails. Its invalidation path also clears ownership before cleanup and suppresses the cleanup error. These are separate from the initial wait behind invocation.

Source references at the inspected fork revision:

Expected behavior

  • Stop has a documented, bounded contract for an active turn: cancel it, or allow bounded completion followed by termination.
  • Stop terminates and reaps the owned child before reporting successful cleanup.
  • Failed or uncertain cleanup retains ownership and supports diagnosis and retry. An unavailable endpoint is not evidence of process exit.
  • A replacement for the affected runtime cannot overlap an old runtime whose cleanup is unconfirmed. Preserve intentional independent runtimes.
  • Health distinguishes pending or failed cleanup from confirmed termination.

Proposed direction: add a cancellation/shutdown path that can interrupt active invocation across the server, SDK, native lifecycle and adapter layers. Cancelling only the Python task is insufficient because native lifecycle locking and the adapter request loop also serialize work. Give ACP cancellation a bounded grace period, then terminate the owned process tree and confirm cleanup; retain handles until that succeeds.

Steps to reproduce

  1. Start a Deep Agents Code runtime in an OpenShell sandbox and begin a long-running Fabric invocation.
  2. While the turn remains active, issue fabric-agent-ctl stop for its runtime ID.
  3. Observe the approximately 320-second transport timeout and check runtime health and the original ACP PID.
  4. Run unchanged NemoClaw plan/apply and inspect ACP processes and runtime identities.
  5. Observe the original turn continuing alongside the replacement.

Regression coverage should exercise busy and idle shutdown, an unresponsive child, cleanup failure and retry, caller timeout, and replacement. Assert actual process exit and no continuing old turn, not only a successful control response. The shared server defect should also have adapter-independent coverage.

Environment

  • Observed run: 20261102T205254Z-deepagents-code-61c99f5a, November 2, 2026, DGX Spark.
  • NemoClaw: 52be6147136fa42efea880778bfac7d6ac98d69f.
  • Fabric: a3b57155593877135f2557ef37d5e18161a1ad82.
  • Harness: dcode 0.2.2, Deep Agents Code adapter.
  • Inference: Nemotron 3 Super NVFP4 through vLLM with one sequence.
  • Exact Python/Rust versions and image build command were not captured in this investigation.

Relevant logs or artifacts

Archived in nemoclaw-test-tracker/runs/20261102T205254Z-deepagents-code-61c99f5a/:

  • evidence/runtime-stop.stdout
  • evidence/runtime-stop-verify.stdout
  • evidence/runtime-restart-identity.stdout
  • evidence/rr-orphan-thread.tools.json
  • evidence/rr-orphan-cleanup.stdout
{"operation":"stop","status":"failed","changed":null,"result":null,"error":{"code":"invalid_response","stage":"transport","message":"The host response is missing, invalid, or truncated.","effects":"unknown"}}

The restart evidence records two dcode ACP processes, PIDs 100 and 1437, in the same sandbox. Runner cleanup terminated PID 100; PID 1437 remained.

Impact and workaround

An agent can continue consuming inference capacity after a user requests stop, and overlap its replacement. The observed runner manually terminated the original owned process. This is evidence of recovery, not a proposed consumer-side implementation.

Fix this in Fabric and consume the supported change in NemoClaw; do not add a NemoClaw-specific harness shutdown implementation. The evidence does not establish a dcode-native defect.

Related epic work:

Duplicate check

  • I searched the existing issues and did not find a duplicate.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions