Low FPS and GPU Underutilization on 4× NVIDIA L40 GPUs

[jasirjasir]

Description

Even with a 4× NVIDIA L40 instance, we are getting very low FPS for both our custom scene and the NVIDIA Nova Carter example. GPU utilization is also very low and imbalanced — the first GPU reaches only ~10% utilization, while the other GPUs stay around 1%. Since this is the hardware configuration recommended by Nvidia,
what could be causing the GPU underutilization and low FPS?

Isaac Sim version

5.1.0

Operating System (OS)

Ubuntu 22.04

GPU Name

L40S

GPU Driver and CUDA versions

580

Logs

No response

Additional information

No response

[lhuang-svg]

@jasirjasir can you run it with below options and monitor nvidia-smi dmon to check the utilization? And please share the logs / CLI options/commands for debugging.

--/renderer/multiGpu/Enabled=true
--/renderer/multiGpu/autoEnable=true
--/renderer/multiGpu/maxGpuCount=4

[jasirjasir]

https://github.com/user-attachments/assets/005923ba-40f3-40aa-a683-a15b7bbb4b14
Hi @lhuang-svg . Please find the video after enabling all 4 GPUs.Is it expected that only one GPU is actively getting used ?

[lhuang-svg]

@jasirjasir it is not uncommon to see mostly 1 GPU being used if the workload is not GPU-intensive. Specifically, cameras in the scene would help improve GPU utilization whereas many IMUs would not. Looking at the ROS topic list shown in their video, it seems like mostly IMUs and non-render heavy sensors are used in this scene? Could you provide more details on the sensor load?
Another test is to run the Nova Carter ROS2 benchmark (documented here under Workflow Performance / ROS2 Rendering & Publishing Speed: https://docs.isaacsim.omniverse.nvidia.com/latest/reference_material/benchmarks.html#measuring-kpis-on-local-hardware). Try to run with 1 or 2 robots and check if the GPU utilization increases. And if there is any weird usage, next steps would be to request they collect a Tracy profile (docs: https://docs.isaacsim.omniverse.nvidia.com/latest/overview/troubleshooting.html#tracy-profiler) and we can look deeper into GPU usage.