Daemon freezes after some time

[lprobsth]

On my system nvx works as expected after booting:

• nvidia dGPU is disabled
• nvx on works: turns dGPU on
• nvx off works: turns dGPU off
• nvx start works: launches program, turns on dGPU, and then turns off dGPU when program finishes

But after some time the daemon becomes unresponsive:

  File "/usr/bin/nvx", line 299, in <module>
    sock.recv(1024).decode("utf-8")
    ~~~~~~~~~^^^^^^

I looked at the source code and the log points into the "remove PCI device" direction.

This is what I can see after the daemon freezes:

• the PCI device of the Nvidia dGPU is still available
• the "remove" interface of the Nvidia dGPU is missing
• the PCI bridge is still available
• interacting with the "[bridge path]/power/control" (e.g. "auto") freezes the terminal
• the "nvidia_drm", "nvidia_modeset", and "nvidia" kernel modules are still loaded

What I tried:

Removing the nvidia dGPU via PCI call:

echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/reset

This fails because the reset interface is missing (already reset?)

Powering down the PCI bridge:

echo auto | sudo tee /sys/bus/pci/devices/0000:00:1c.0/power/control

This hangs / freezes the console.

Unloading the kernel modules unfreezes the terminals and allows restarting nvx:

sudo modprobe --remove --remove-holders nvidia_drm
sudo modprobe --remove --remove-holders nvidia_modeset

I think for some reason nvx fails to unload the kernel modules before turning off the dGPU via PCI calls. Then the daemon freezes.
Restarting nvx leads to the daemon freezing up again because it does not unload the kernel modules on start.

Question:

• Should the daemon unload the modules on start? (omit deadlock)
• Why could the daemon fail to unload the modules on exit? (race condition?)
• Is this caused by my setup? (for me it seems that the modules are not unloaded e.g. on start; also nvx normally works for some time; so it should not be a missing module in the nvx config?)

[lprobsth]

I think I now know the problem: Discord interacts with nvidiactl if it is launched while the dGPU is activated.

lsof of the nvidia kernel module shows nothing.

lsof of nvidiactl shows:

COMMAND   PID     USER FD   TYPE  DEVICE SIZE/OFF NODE NAME
Discord 64212 lprobsth 28u   CHR 195,255      0t0 1273 /dev/nvidiactl

If I close discord, nvx disables the dGPU without problems. I also tried enabling the nvx option "kill_on_off=true". Then on dGPU turn off, the app turns black for a second and then renders normal again. Turning off the dGPU with nvx fails and a new discord process with a new PID is listed with lsof. So it seems like discord is pretty aggressive.

[lprobsth]

Update:

It seems like VScode also accesses nvidiactl when started while the GPU is on. Same behavior here with process respawn on process kill.

Since both applications use electron: maybe these applications are not accessing the nvidia GPU on their own and electron is somehow responsible for this (default hardware acceleration mechanism?)

My current conclusion:
nvx is not responsible for this problem: a) the nvidia kernel modules do not report device usage that is blocking the removal of those modules, b) the processes using the GPU aggressively respawn.

So at this point this issue is more a documentation for others having problems with the daemon becoming unresponsive. If I find a workaround for blocking the GPU access of electron apps I'll post here.

[lprobsth]

Might be related: electron/electron#41389

Reports that electron apps wake up dGPU on hybrid/prime setups and keeps dGPU in D0 runtime power management state.

[lprobsth]

I think I found the workaround for electron apps:

export VK_DRIVER_FILES=/usr/share/vulkan/icd.d/intel_icd.x86_64.json

This keeps both VScode and discord from using nvidiactl - hence nvx off works after the application has been started. Spotify might also be affected by this - but I stopped using it and can't test it.

I keep testing if this resolves most of my problems with nvx. Maybe I'll create a PR for updating the README and then the issue can be closed.