Cardwire breaks NVENC on dual Nvidia GPU device

[wellcoming]

Describe the bug

On a desktop with two NVIDIA GPUs, when I use cardwire Manual mode to block one GPU, NVENC on the remaining unblocked GPU stops working. Any NVENC encoder fails with OpenEncodeSessionEx failed: unsupported device (2). After unblocking the GPU, the exact same command works again.

To Reproduce

1. Dual NVIDIA GPU desktop (RTX 3070 + GTX 1660 SUPER), display connected to the 1660 SUPER.
2. cardwire gpu 1 --block (blocks the non-display GPU, the 3070) — Manual mode.
3. Run a simple NVENC encode on the remaining GPU:

   ffmpeg -f lavfi -i testsrc=duration=1:size=1280x720:rate=30 -c:v hevc_nvenc -f null -
   

4. Observed error:

   [hevc_nvenc @ ...] OpenEncodeSessionEx failed: unsupported device (2): (no details)
   [hevc_nvenc @ ...] No capable devices found
   

   Same failure with CUDA_VISIBLE_DEVICES=0 and with -c:v h264_nvenc.
5. Run cardwire gpu 1 --unblock , the same ffmpeg command succeeds.

Expected behavior

Blocking one GPU should only hide/disable that GPU. NVENC on the remaining unblocked (display) GPU should keep working, exactly as it does when no GPU is blocked.

Environment details

OS: Arch Linux x86_64
Kernel: Linux 7.2.3-1-cachyos-bore-lto
Uptime: 2 hours, 46 mins
Packages: 9 (flatpak), 1117 (pacman)
Shell: zsh 5.9.2
Display (F27G1Q): 2560x1440 in 27", 165 Hz [External]
Terminal: /dev/pts/1 10.5p1
CPU: Intel(R) Core(TM) i9-9900KF (16) @ 5.00 GHz
GPU: NVIDIA GeForce GTX 1660 SUPER [Discrete]
Memory: 2.32 GiB / 31.25 GiB (7%)
Swap: 100.00 KiB / 69.25 GiB (0%)
Disk (/): 584.76 GiB / 611.08 GiB (96%) - btrfs
Local IP (eno1): 192.168.31.228/24
Locale: zh_CN.UTF-8

❯ cardwire --version
cardwire-cli 0.12.1
❯ cardwire list --json
{
  "0": {
    "id": 0,
    "name": "NVIDIA GeForce GTX 1660 SUPER",
    "pci": "0000:02:00.0",
    "render": 129,
    "card": 2,
    "default": true,
    "discrete": true,
    "virtual_gpu": false,
    "available": true,
    "vendor": "Nvidia",
    "driver": "nvidia",
    "blocked": false,
    "launchable": true,
    "nvidia": true,
    "nvidia_minor": "1"
  },
  "1": {
    "id": 1,
    "name": "NVIDIA GeForce RTX 3070",
    "pci": "0000:01:00.0",
    "render": 128,
    "card": 0,
    "default": false,
    "discrete": true,
    "virtual_gpu": false,
    "available": true,
    "vendor": "Nvidia",
    "driver": "nvidia",
    "blocked": true,
    "launchable": false,
    "nvidia": true,
    "nvidia_minor": "0"
  }
}
❯ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x 2 root root 120  9月 8日 02:27 .
drwxr-xr-x 3 root root 140  9月 8日 02:27 ..
lrwxrwxrwx 1 root root   8  9月 8日 02:27 pci-0000:01:00.0-card -> ../card0
lrwxrwxrwx 1 root root  13  9月 8日 02:27 pci-0000:01:00.0-render -> ../renderD128
lrwxrwxrwx 1 root root   8  9月 8日 02:04 pci-0000:02:00.0-card -> ../card2
lrwxrwxrwx 1 root root  13  9月 8日 02:04 pci-0000:02:00.0-render -> ../renderD129

cardwired logs

Sep 08 05:14:18 ComingStudio systemd[1]: Starting Cardwire Daemon...
Sep 08 05:14:18 ComingStudio cardwired[2529]: [INFO ] IOMMU detected, reading pci devices using iommu dir
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] Successfully resolved card1 and renderD128 for PCI 0000:01:00.0
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] Successfully resolved card2 and renderD129 for PCI 0000:02:00.0
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] gpu NVIDIA GeForce RTX 3070 id: 0 internal: 0, desktop: 4, connected: 0, total: 4, connected_internal: 0, connected_desktop: 0
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] gpu NVIDIA GeForce GTX 1660 SUPER id: 1 internal: 0, desktop: 4, connected: 1, total: 4, connected_internal: 0, connected_desktop: 1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] inserting: 2529 into map
Sep 08 05:14:19 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for hwmon 0000:02:00.0: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [WARN ] failed to get inode for /sys/class/backlight/nvidia_1: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for backlight nvidia_1: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] blocking: 0000:01:00.0
Sep 08 05:14:19 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for hwmon 0000:01:00.0: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [WARN ] failed to get inode for /sys/class/backlight/nvidia_0: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for backlight nvidia_0: No such file or directory (os error 2)
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] Switched to Manual
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_command; command=Auth(Some(External), Some([48]))
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[Auth(Some(External), Some([48]))] extra_bytes=None
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_command;
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[NegotiateUnixFD, Begin] extra_bytes=Some([108, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 109, 0, 0, 0, 1, 1, 111, 0, 21, 0, 0, 0, 47, 111, 114, 103, 47, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 47, 68, 66, 117, 115, 0, 0, 0, 2, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0, 3, 1, 115, 0, 5, 0, 0, 0, 72, 101, 108, 108, 111, 0, 0, 0, 6, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0])
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] start_object_server; started_event=Some(Event { listeners_notified: 0, listeners_total: 1 })
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] obj_server_task;
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] monitor_name_lost; name=org.opengamingcollective.cardwire
Sep 08 05:14:19 ComingStudio systemd[1]: Started Cardwire Daemon.
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_command; command=Auth(Some(External), Some([48]))
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[Auth(Some(External), Some([48]))] extra_bytes=None
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_command;
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[NegotiateUnixFD, Begin] extra_bytes=Some([108, 1, 0, 1, 0, 0, 0, 0, 5, 0, 0, 0, 109, 0, 0, 0, 1, 1, 111, 0, 21, 0, 0, 0, 47, 111, 114, 103, 47, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 47, 68, 66, 117, 115, 0, 0, 0, 2, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0, 3, 1, 115, 0, 5, 0, 0, 0, 72, 101, 108, 108, 111, 0, 0, 0, 6, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0])
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] start_object_server; started_event=Some(Event { listeners_notified: 0, listeners_total: 1 })
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] obj_server_task;
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] monitor_name_lost; name=net.hadess.SwitcherooControl
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] Switcheroo DBus shim started successfully
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] Daemon started succesfully
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_command; command=Auth(Some(External), Some([48]))
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[Auth(Some(External), Some([48]))] extra_bytes=None
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_command;
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] write_commands; commands=[NegotiateUnixFD, Begin] extra_bytes=Some([108, 1, 0, 1, 0, 0, 0, 0, 11, 0, 0, 0, 109, 0, 0, 0, 1, 1, 111, 0, 21, 0, 0, 0, 47, 111, 114, 103, 47, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 47, 68, 66, 117, 115, 0, 0, 0, 2, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0, 3, 1, 115, 0, 5, 0, 0, 0, 72, 101, 108, 108, 111, 0, 0, 0, 6, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0])
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] read_commands; n_commands=1
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] {}; task_name="org.freedesktop.UPower proxy caching"
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] successfully sent reset-failed on nvidia-powerd.service
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] successfully sent start on nvidia-powerd.service
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] NVIDIA GeForce GTX 1660 SUPER: Power state changed: D0
Sep 08 05:14:19 ComingStudio cardwired[2529]: [INFO ] NVIDIA GeForce RTX 3070: Power state changed: D0
Sep 08 05:14:21 ComingStudio cardwired[2529]: [INFO ] niri[1474] tried to access GPU 1 (blocked by cardwire)
Sep 08 05:14:21 ComingStudio cardwired[2529]: [INFO ] libvirtd[790] tried to access GPU 1 (blocked by cardwire)
Sep 08 05:14:21 ComingStudio cardwired[2529]: [INFO ] systemd-logind[654] tried to access GPU 1 (blocked by cardwire)
Sep 08 05:14:27 ComingStudio cardwired[2529]: [INFO ] dispatch_call; msg=Msg { type: MethodCall, serial: 2, sender: UniqueName(":1.118"), path: ObjectPath("/org/opengamingcollective/cardwire"), iface: InterfaceName("org.opengamingcollective.cardwire.Manager"), member: MemberName("Status"), fds: [] } hdr=Header { primary: PrimaryHeader { endian_sig: Little, msg_type: MethodCall, flags: BitFlags<Flags>(0b0), protocol_version: 1, body_len: 0, serial_num: 2 }, fields: Fields { path: Some(ObjectPath("/org/opengamingcollective/cardwire")), interface: Some(InterfaceName("org.opengamingcollective.cardwire.Manager")), member: Some(MemberName("Status")), error_name: None, reply_serial: None, destination: Some(BusName::WellKnown("org.opengamingcollective.cardwire")), sender: Some(UniqueName(":1.118")), signature: Unit, unix_fds: None } }
Sep 08 05:14:27 ComingStudio cardwired[2529]: [INFO ] dispatch_call; msg=Msg { type: MethodCall, serial: 3, sender: UniqueName(":1.118"), path: ObjectPath("/org/opengamingcollective/cardwire/Gpu/1"), iface: InterfaceName("org.freedesktop.DBus.Properties"), member: MemberName("Set"), body: Structure(Dynamic { fields: [Str, Str, Variant] }), fds: [] } hdr=Header { primary: PrimaryHeader { endian_sig: Little, msg_type: MethodCall, flags: BitFlags<Flags>(0b0), protocol_version: 1, body_len: 64, serial_num: 3 }, fields: Fields { path: Some(ObjectPath("/org/opengamingcollective/cardwire/Gpu/1")), interface: Some(InterfaceName("org.freedesktop.DBus.Properties")), member: Some(MemberName("Set")), error_name: None, reply_serial: None, destination: Some(BusName::WellKnown("org.opengamingcollective.cardwire")), sender: Some(UniqueName(":1.118")), signature: Structure(Dynamic { fields: [Str, Str, Variant] }), unix_fds: None } }
Sep 08 05:14:27 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for hwmon 0000:01:00.0: No such file or directory (os error 2)
Sep 08 05:14:27 ComingStudio cardwired[2529]: [WARN ] failed to get inode for /sys/class/backlight/nvidia_0: No such file or directory (os error 2)
Sep 08 05:14:27 ComingStudio cardwired[2529]: [ERROR] (ignoring) failed to get inode for backlight nvidia_0: No such file or directory (os error 2)
Sep 08 05:14:27 ComingStudio cardwired[2529]: [INFO ] Set GPU NVIDIA GeForce RTX 3070 block=true

Additional context

• If the blocked GPU is removed from the nvidia driver entirely (unbind RTX3070 from nvidia driver), NVENC on the remaining GPU works fine.

[luytan]

Hi, thanks for testing cardwire on such config, that's what I needed the most ! Is the exp nvidia setting enabled?

[wellcoming]

Hi, thanks for testing cardwire on such config, that's what I needed the most ! Is the exp nvidia setting enabled?

No, that will break everything because both GPUs share the same /dev/nvidiactl.

[luytan]

yup, just wanted to confirm.
I will make some researches on my side, and will contact you again a bit later, i'm currently refactoring/improving parts of the codebase.
If you find anything and want to discuss it, we can talk on the ogc discord, i find this easier.

[wellcoming]

yup, just wanted to confirm. I will make some researches on my side, and will contact you again a bit later, i'm currently refactoring/improving parts of the codebase. If you find anything and want to discuss it, we can talk on the ogc discord, i find this easier.

Sure, I’m open to running some tests on my device to see if we can pinpoint what’s causing the issue.

[luytan]

Maybe https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-driver-565/347361

[wellcoming]

Maybe https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-driver-565/347361

Agreed. However, we still need a workaround for NVENC until NVIDIA fixes the issue, though CARDWIRE_ALLOW=1 doesn't seem to work in manual mode.

❯ ls /dev/nvidia0 /dev/nvidia1
ls: cannot access '/dev/nvidia0': No such file or directory
/dev/nvidia1

❯ CARDWIRE_ALLOW=1 ls /dev/nvidia0 /dev/nvidia1
ls: cannot access '/dev/nvidia0': No such file or directory
/dev/nvidia1

❯ CARDWIRE_FORCE_DGPU=1 ls /dev/nvidia0 /dev/nvidia1
ls: cannot access '/dev/nvidia1': No such file or directory
/dev/nvidia0

❯ CARDWIRE_FORCE_GPU=0 ls /dev/nvidia0 /dev/nvidia1
ls: cannot access '/dev/nvidia0': No such file or directory
/dev/nvidia1

❯ CARDWIRE_FORCE_GPU=1 ls /dev/nvidia0 /dev/nvidia1
ls: cannot access '/dev/nvidia1': No such file or directory
/dev/nvidia0

[luytan]

CARDWIRE_ALLOW wasn't implement in manual mode yet, i'm still thinking on how i will do it, i will probably make CARDWIRE_ALLOW take a list of GPU ids

[luytan]

I got an idea on how to fix that, but it would only work on system without any kernel lockdown.

We could try to rewrite the nvidia ioctl to not include the blocked GPU

Do you have lockdown enabled?

[wellcoming]

I got an idea on how to fix that, but it would only work on system without any kernel lockdown.

We could try to rewrite the nvidia ioctl to not include the blocked GPU

Do you have lockdown enabled?

❯ cat /sys/kernel/security/lockdown
[none] integrity confidentiality