[Bug Report] AppLauncher's Python SIGSEGV handler turns any segfault into an unkillable infinite fault loop with no core dump

[bbelousov]

Describe the bug

AppLauncher installs a Python signal handler for SIGSEGV (and SIGABRT):

https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/app/app_launcher.py#L355-L359

# Set up signal handlers for graceful shutdown
# -- during explicit `kill` commands
signal.signal(signal.SIGTERM, self._abort_signal_handle_callback)
# -- during segfaults
signal.signal(signal.SIGABRT, self._abort_signal_handle_callback)
signal.signal(signal.SIGSEGV, self._abort_signal_handle_callback)

def _abort_signal_handle_callback(self, signal, frame):
    """Handle the abort/segmentation/kill signals."""
    # close the app
    self._app.close()

SIGSEGV is not recoverable this way. The signal is synchronous and the faulting
instruction is re-executed when the handler returns, so a single segfault becomes an
infinite fault loop: the process spins one core at ~110 % CPU forever, releases the
GPU, never exits, and produces no core dump and no exit code. It cannot be stopped
with SIGTERM either, because the same handler is installed for it.

The practical consequence is that any segfault anywhere in a long-running job — in
PyTorch, in a CUDA library, in user code, or from faulty hardware — is converted from a
diagnosable crash into an undiagnosable hang. In our case this masked a genuine hardware
fault on one workstation for four days: four failures produced no crash, no backtrace and
no core, and every investigation dead-ended. After commenting out lines 358–359 the same
workload crashed cleanly with SIGSEGV, wrote an 11.9 GB core, and the fault was
identified within an hour.

Two independent captures of the loop, taken while it was hung:

strace on the spinning process — this repeats indefinitely:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x1} ---
rt_sigreturn({mask=[]})                 = 140733529160608
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x1} ---
rt_sigreturn({mask=[]})                 = 140733529160608
        ... forever ...

py-spy dump on another occurrence, showing the handler wedged inside app.close()
while the interpreter is still inside env.step():

Thread 71 (active+gil): "MainThread"
    _get_device_index (torch/cuda/_utils.py:836)
    set_device (torch/cuda/__init__.py:527)
    render (isaaclab/sim/simulation_context.py:631)
    _app_control_on_stop_handle_fn (isaaclab/sim/simulation_context.py:1029)
    close (simulation_app.py:814)
    _abort_signal_handle_callback (isaaclab/app/app_launcher.py:989)
    step (<our env wrapper>:927)
    step (<our train script>:184)

Note the second trace also shows why the handler cannot work as intended: it calls back
into CUDA and the renderer from inside a fault handler, which is not async-signal-safe.

Steps to reproduce

Any segfault in a job launched through AppLauncher reproduces it. A minimal trigger:

import ctypes
from isaaclab.app import AppLauncher

app_launcher = AppLauncher(headless=True)
simulation_app = app_launcher.app

ctypes.string_at(1)     # deliberate SIGSEGV

Expected: the process dies with SIGSEGV (and a core dump if ulimit -c allows).
Actual: the process spins at ~100 % CPU indefinitely and cannot be killed with SIGTERM.

System Info

• Commit: v2.3.2 (also present on main, lines 355–359)
• Isaac Sim Version: 5.1
• OS: Ubuntu 24.04
• GPU: RTX PRO 6000 Blackwell Workstation Edition
• CUDA: 13.2
• GPU Driver: 595.71.05

Additional context

Suggested fix — leave fatal, synchronous signals to the default disposition and keep the
graceful path only for asynchronous termination signals:

signal.signal(signal.SIGTERM, self._abort_signal_handle_callback)
signal.signal(signal.SIGINT, self._abort_signal_handle_callback)
# SIGSEGV/SIGABRT deliberately left at SIG_DFL: they are synchronous and
# unrecoverable, and returning from a Python handler re-faults forever.

If a hook on fatal faults is wanted, the safe options are faulthandler.enable() (writes
a traceback and re-raises with the default action) or an os._exit() at the end of the
handler so the process cannot return to the faulting instruction. Making it opt-in behind
an AppLauncher argument would also be reasonable, so that jobs which need diagnosable
crashes can get them without patching the file.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• A segfault in an AppLauncher job terminates the process instead of looping
• A core dump is produced when ulimit -c permits it
• SIGTERM still shuts the app down gracefully

[dundysm]

i'll take this; plan is to stop registering python handlers for SIGSEGV/SIGABRT (leave default fatal behavior / core dumps) while keeping graceful SIGTERM handling aligned with the existing signal work.