cloudxr: possible exec race in background.start_and_wait() (needs owner triage) #1378

Activity

1. ivany-nv

   assigned

   nv-jakob

   on Oct 28, 2026
2. ivany-nv

   added

   NVIDIA

   on Oct 28, 2026
3. github-project-automation

   added this to Isaac Teleop Planningon Oct 28, 2026
4. github-project-automation

   moved this to TBD in Isaac Teleop Planningon Oct 28, 2026
5. ivany-nv commented on Oct 28, 2026

   

   ivany-nv

   on Oct 28, 2026

   Last edited by ivany-nv

   ContributorAuthor

   More actions

   Reproduced when PR review 1376:

   Setup: main at a74babb, 48-core x86_64, Linux 7.0, Python 3.12. The real start_and_wait(), spawn(), read_pid() and is_runtime_live(). The service is replaced by a script that sleeps 30 s, since the real one needs a GPU.

   read_pid() on the first loop pass after spawn(), for a child that is alive the whole time:

   Host	Missed
   No other load (8 parallel drivers)	716 of 316,033 calls (0.23%)
   stress-ng --cpu 96 --fork 16	545 of 37,388 calls (1.5%)

   End to end, with only is_runtime_live() stubbed to report the runtime up on the second pass, the unmodified start_and_wait() raised The CloudXR service exited during startup. See …/service.log 3 times in 1,654 calls with no other load. Under the same stress-ng load, all 4 workers raised it within their first 15 to 54 calls. Every time, the child was still running 0.5 s later, and read_pid() then returned its pid.

   What a user sees:

   • service start exits non-zero saying the service exited, while the detached service it just started keeps running and goes on to hold the GPU.
   • CloudXRLauncher catches only AlreadyServingError around start_and_wait(), so the RuntimeError propagates and the _STARTED_SERVICE message with the pid and log path is never printed, though the service keeps running.

   Fix direction: the poll() option above holds up: keep the Popen from spawn() and treat proc.poll() is not None as exited. Do not make an empty cmdline mean "still starting": a zombie's cmdline is empty too, and terminate() and test_sigterm_and_clears_the_pid_file rely on empty meaning gone. TestStartRaces mocks spawn(), so nothing covers this path today; the fix wants a test with a real spawn.

   Reproduction script

   Run with a built isaaccapture on PYTHONPATH; several in parallel make the miss rate visible sooner.

   """Count start_and_wait() first-pass read_pid() misses on a live child.
   
   Run with a built isaaccapture on PYTHONPATH: python repro_1378.py [seconds]
   """
   
   import os
   import shutil
   import signal
   import sys
   import tempfile
   import time
   from pathlib import Path
   from unittest.mock import patch
   
   from isaaccapture.cloudxr import background
   
   tmp = Path(tempfile.mkdtemp())
   (tmp / "stand_in.py").write_text("import time\ntime.sleep(30)\n")
   os.environ["PYTHONPATH"] = f"{tmp}{os.pathsep}{os.environ.get('PYTHONPATH', '')}"
   background._MODULE = "stand_in"  # the real service needs a GPU
   run_dir, logs_dir = str(tmp / "run"), tmp / "logs"
   real_read_pid = background.read_pid
   
   
   class FirstPass(Exception):
       pass
   
   
   def first_read_pid(rd):  # record the first post-spawn result, then leave the loop
       raise FirstPass(real_read_pid(rd))
   
   
   calls = misses = 0
   deadline = time.monotonic() + float(sys.argv[1] if len(sys.argv) > 1 else 30)
   try:
       with patch.object(background, "read_pid", first_read_pid):
           while time.monotonic() < deadline:
               try:
                   background.start_and_wait([], run_dir, logs_dir)
               except FirstPass as seen:
                   misses += seen.args[0] is None
               pid = int(background.pid_path(run_dir).read_text())
               os.kill(pid, signal.SIGKILL)
               os.waitpid(pid, 0)
               calls += 1
   finally:
       shutil.rmtree(tmp)
   print(f"read_pid() missed a live child on {misses} of {calls} start_and_wait() calls")