Linux NVENC fails when CUDA device has no matching DRM node (/dev/dxg / Hyper-V GPU-P) #5598

Activity

1. Ashutosh0x commented on Sep 10, 2026

   

   Ashutosh0x

   on Sep 10, 2026

   More actions

   The DRM node isn't required by NVENC or by CUDA here — it's required by the zero-copy EGL path only, and the interesting part is that Sunshine already has a fallback for this and it fires too early to catch it.

   open_drm_fd_for_cuda_device() is called from exactly one place, gl_cuda_vram_t::init() (src/platform/linux/cuda.cpp L423), and the fd is used to build a GBM device and an EGL display for GL→CUDA interop:

   file = std::move(open_drm_fd_for_cuda_device(0));
   ...
   gbm.reset(gbm::create_device(file.el));
   display = egl::make_display(gbm.get());

   There are two CUDA encode-device factories, and only one of them needs that:

   • cuda::make_avcodec_gl_encode_device() -> gl_cuda_vram_t — needs the NVIDIA DRM node (kmsgrab L1746, display_vram_t)
   • cuda::make_avcodec_encode_device() -> cuda_ram_t/cuda_vram_t — no DRM node at all (kmsgrab L1568, display_ram_t; also x11grab L660)

   kms_display() already falls back from vram to ram (L1911-1928):

   if (hwdevice_type == vaapi || cuda || vulkan) {
     auto disp = std::make_shared<kms::display_vram_t>(hwdevice_type);
     if (!disp->init(display_name, config)) {
       return disp;
     }
   }
   auto disp = std::make_shared<kms::display_ram_t>(hwdevice_type);

   But that fallback is keyed on display_vram_t::init(). On GPU-P that init succeeds — the guest does have a DRM card (hyperv_drm card1) for capture. The lookup that fails is for the CUDA device's own node, and it happens later, in make_avcodec_encode_device(pix_fmt), after the fallback decision is already made. So a case the fallback was written for surfaces as a hard failure instead.

   That also explains why the PCI bus ID in the message is the NVIDIA card while capture itself was fine.

   Two directions, depending on what maintainers prefer: let an encode-device failure on the vram display fall back to display_ram_t the same way an init failure does, or probe the CUDA device's DRM node during display_vram_t::init() so the existing fallback sees it.

   Read from source rather than reproduced — I don't have a GPU-P setup. If you can run it, display_ram_t on this same host should already work, so forcing the RAM path is worth trying as a stopgap.
2. ReenigneArcher

   added

   aiPR has signs of heavy ai usage (either indicated by user or assumed)PR has signs of heavy ai usage (either indicated by user or assumed)

   on Sep 10, 2026
3. RoYaL69 commented on Nov 6, 2026

   

   RoYaL69

   on Nov 6, 2026

   Author

   More actions

   Thanks @Ashutosh0x, the analysis checks out. Retested on the same GPU-P guest (RTX 4050, host driver 32.0.16.1060 / KMD 610.60, /dev/dxg, only hyperv_drm card1, no NVIDIA DRM node).

   wlr capture is fixed by the v2026.1103.235308 pre-release (via #5873 + #4946):

   Build	Config	Result
   2026.906.222525	capture = wlr, encoder = nvenc	Unable to find DRM device with PCI bus ID: 0000:01:00.0 -> Encoder [nvenc] failed
   2026.1103.235308 (AppImage)	capture = wlr, encoder = nvenc	[wlgrab] CUDA capture path: GPU -> RAM -> GPU bridge (cross-GPU system) -> Found H.264 encoder: h264_nvenc, Found HEVC encoder: hevc_nvenc, Found AV1 encoder: av1_nvenc

   The bridge is selected because the render node's PCI id can't be read (Could not read PCI bus id for [/dev/dri/renderD128], assuming cross-GPU capture), and SHM screencopy covers the missing DMA-BUF support. This is the startup encoder probe; I haven't run a full Moonlight session yet.

   KMS is unchanged in that pre-release: kms_display() still only falls back to display_ram_t when display_vram_t::init() fails, so capture = kms + nvenc still ends in the DRM lookup, as described above. Either of the two directions you proposed would still be needed for KMS.

   Separate small bug found on the way: when cuInit fails (here: guest driver files out of sync with the host driver -> CUDA_ERROR_NO_DEVICE), Sunshine segfaults right after Creating encoder [h264_nvenc] instead of logging the error. In src/platform/linux/cuda.cpp check(), name/description are uninitialized and used even if cuGetErrorName/cuGetErrorString fail; the crash is strlen(0x1) under "Couldn't initialize cuda: ". Initializing both to nullptr/a fallback string (or checking the return codes) would turn that into a normal encoder failure.
4. ReenigneArcher commented on Nov 6, 2026

   

   ReenigneArcher

   on Nov 6, 2026

   Member

   More actions

   @RoYaL69, could you test #6056 on your GPU-P setup and check whether it fixes the segfault you reported when cuInit fails? Sunshine should now log the numeric CUDA error, with safe fallback text if the error lookups fail, and report initialization failure instead of crashing. Please share the result and relevant log output.

   This PR addresses the logging crash; the KMS capture fallback problem still needs a separate fix.
5. RoYaL69 commented on Nov 13, 2026

   

   RoYaL69

   on Nov 13, 2026

   Author

   More actions

   @ReenigneArcher, tested #6056 (head f9e124d) against master (774be88) using the CI AppImages, on an Arch Linux Hyper-V guest with GPU-P (RTX 4050 via /dev/dxg, wsl-style libcuda, capture = wlr, encoder = nvenc).

   To force cuInit to fail I used a stub libcuda.so.1 via LD_LIBRARY_PATH: cuInit returns 100 (CUDA_ERROR_NO_DEVICE), and cuGetErrorName/cuGetErrorString fail without setting their out pointers.

   • master: segfault (exit 139) immediately after Creating encoder [h264_nvenc].
   • fix(linux/cuda): handle failed error text lookups #6056: Error: Couldn't initialize cuda: Unknown CUDA error (100): No error description available followed by Info: Encoder [nvenc] failed. Sunshine then falls back to the next encoders (Found H.264 encoder: libx264 [software]) and does not crash.
   • Real GPU-P libcuda (/usr/lib/gpup/lib): the NVENC probe still works with the PR. It finds h264_nvenc, hevc_nvenc and av1_nvenc, and [wlgrab] CUDA capture path: GPU -> RAM -> GPU bridge (cross-GPU system) is logged as before.

   Caveats: this is a stub reproduction rather than a genuinely out-of-sync guest driver, and I only tested the encoder probe, with no streaming client. The fix looks good to me.

   👍React with 👍1Reacted by Dave Lane