NuRec Headless Rendering — UJITSO Geometry Processor Not Loading in Python Mode (Isaac Sim 5.1.0)

[hero2001]

GitHub Discussion 카테고리 선택 기준:
"Q&A" 를 선택하세요. 버그 리포트가 아니라 공식 지원 방법을 묻는 질문이기 때문입니다.

카테고리 선택 후 아래 내용을 붙여넣으세요:

Title:
NuRec Headless Rendering — UJITSO Geometry Processor Not Loading in Python Mode (Isaac Sim 5.1.0)

Body:

Environment

• Isaac Sim: 5.1.0 (Docker: nvcr.io/nvidia/isaac-sim:5.1.0)
• OS: Ubuntu 24.04, kernel 6.8.0
• GPU: NVIDIA RTX A6000
• Driver: 560.35.05
• Dataset: nvidia/PhysicalAI-Autonomous-Vehicles-NuRec (HuggingFace)

Goal

Render NuRec scenes in headless python mode to extract frames from 4 cameras:

• camera_front_wide_120fov
• camera_front_tele_30fov
• camera_cross_left_120fov
• camera_cross_right_120fov

Problem

UJITSO geometry processor is not loading in headless python mode.
NuRec volume is not rendered — only blank or mesh-only images are produced.

Root Cause Found

libcarb.ujitso.processor.geometry.plugin.so exists inside the omni.hydra.rtx extension:
/isaac-sim/extscache/omni.hydra.rtx-1.0.0+69cbf6ad.lx64.r/bin/carb.ujitso.processors.geometry/libcarb.ujitso.processor.geometry.plugin.so
However, none of the .kit app files in /isaac-sim/apps/ reference omni.hydra.rtx or omni.ujitso, so the processor is never registered in headless python mode.

What I've Tried

1. OMNI_KIT_ARGS_EXTRA environment variable

os.environ["OMNI_KIT_ARGS_EXTRA"] = "--/UJITSO/geometry=true"

Result: Setting readable as True but processor still not found.

2. Extension manager after app startup

manager = omni.kit.app.get_app().get_extension_manager()
manager.set_extension_enabled_immediate("omni.ujitso.default", True)
manager.set_extension_enabled_immediate("omni.hydra.rtx", True)
carb.settings.get_settings().set_bool("/UJITSO/geometry", True)

Result: Extensions show as enabled but processor still not found.

3. Direct plugin loading via ctypes

ctypes.CDLL(
    "/isaac-sim/extscache/omni.hydra.rtx-1.0.0+69cbf6ad.lx64.r/bin/"
    "carb.ujitso.processors.geometry/libcarb.ujitso.processor.geometry.plugin.so",
    mode=ctypes.RTLD_GLOBAL
)

Result: No effect.

Error Messages

[Warning] [omni.ujitso] No processor found for request
[Warning] [omni.hydra] GeometryStreaming : Failed to request geometry /World/volume/mesh/mesh 4
[Warning] [omni.hydra] GeometryStreaming : Failed to request geometry /World/mesh_ground/mesh 4

Questions

1. Is there an official .kit app file designed for NuRec headless rendering?
2. Is there a supported method to load the UJITSO geometry processor in headless python mode?
3. Is NuRec volume rendering officially only supported in GUI mode?

Any guidance would be greatly appreciated. Thank you.

[vick-yu]

Hi,

I was able to reproduce and identify the root cause of this issue.

Root Cause

The UJITSO geometry processor plugin does load in headless python mode (the full extension chain omni.hydra.rtx -> omni.ujitso.client -> omni.ujitso.default loads correctly). However, during initialization the plugin checks the /UJITSO/geometry carb setting, which defaults to false, and skips initialization:

[Info] [carb.ujitso.processor.geometry.plugin] skipping initialization as '/UJITSO/geometry'=false

This explains why all your post-startup attempts to enable it didn't work -- by the time SimulationApp.__init__ returns, the plugin has already checked the setting and skipped its init.

Fix

The setting must be passed before the Kit app starts. Use the extra_args config option in SimulationApp:

from isaacsim import SimulationApp

app = SimulationApp({
    "headless": True,
    "multi_gpu": False,
    "max_gpu_count": 1,
    "active_gpu": 0,
    "extra_args": ["--/UJITSO/geometry=true"],
})

The extra_args list is appended to the Kit launch arguments before the app starts, ensuring the setting is available when the geometry processor plugin initializes.

Why Your Previous Attempts Failed

Approach	Why it didn't work
OMNI_KIT_ARGS_EXTRA env var	Must be set before SimulationApp import; may not be parsed before plugin init
Extension manager .set_extension_enabled_immediate()	Extensions were already loaded; this is a plugin init-time setting, not an extension enable/disable issue
carb.settings.set_bool("/UJITSO/geometry", True)	Too late -- plugin already skipped init during extension startup
ctypes.CDLL direct loading	Doesn't go through carb plugin framework; plugin needs proper registration and initialization

To Your Questions

1. Official .kit app for NuRec headless rendering? -- Not specifically, but isaacsim.exp.base.python.kit (the default for SimulationApp) loads all necessary extensions including omni.hydra.rtx. You just need the --/UJITSO/geometry=true flag.

2. Supported method to load UJITSO geometry processor in headless python? -- Yes, via extra_args as shown above.

3. Is NuRec volume rendering only supported in GUI mode? -- No, headless should work once the geometry processor is properly initialized. The RTX renderer runs on GPU regardless of whether there's a display.

Let me know if this resolves the issue!

[hero2001]

Hi @vick-yu,

Thank you for your previous answer — the extra_args approach worked and --/UJITSO/geometry=true is now correctly passed to kit. However, I'm still getting CPU fallback for the geometry processor.

Log evidence (headless python mode):

[0.8s]  omni.ujitso.default startup
        | Processor: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz  ← CPU fallback
[3.6s]  omni.hydra.rtx-1.0.0 startup  ← GPU processor registers here, too late

Root cause: omni.physx.cooking and omni.ujitso.client both depend on omni.ujitso.default, causing it to load at ~0.8s before omni.hydra.rtx (~3.6s). The GPU geometry processor inside omni.hydra.rtx registers too late.

What I tried:

• order = -100000 on omni.hydra.rtx in isaacsim.exp.base.kit → hydra.rtx loads at 3.6s but ujitso.default still loads at 0.8s
• Runtime set_extension_enabled_immediate to reload ujitso.default after hydra.rtx → caused crash (ManipulatorFactory NameError + Segfault)

Is there an official way to ensure the GPU geometry processor is available before omni.ujitso.default initializes in headless python mode?

[vick-yu]

Hi @hero2001,

--
 
I reproduced your setup on Isaac Sim 5.1.0 and dug into the Kit logs and plugin binaries. Good news -- the extra_args fix is actually working correctly. The "CPU fallback" you're seeing is a log misread. Here's the breakdown:
 

The \| Processor: line is not UJITSO

 
The log line you identified as CPU fallback:
 

[0.8s]  omni.ujitso.default startup
\| Processor: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz  <-- NOT geometry processor

 
This is the system info table printed by omni.kit.renderer.init -- it always displays your host CPU name, OS, memory, and GPU info. It has nothing to do with UJITSO geometry processing mode.
 

The geometry processor plugin lives in omni.hydra.rtx, not omni.ujitso.default

 
The actual GPU geometry processor (carb.ujitso.processor.geometry.plugin) is bundled inside omni.hydra.rtx, not omni.ujitso.default. The load order is:
 
| Time | What loads | Role |
|------|-----------|------|
| ~0.8s | omni.ujitso.default | UJITSO infrastructure (cache, agent, datastore) |
| ~5.0s | omni.hydra.rtx | RTX renderer + GPU geometry processor plugin |
 
The geometry processor checks /UJITSO/geometry when it initializes (~5s), not when omni.ujitso.default starts (~0.8s). So the timing is correct -- infrastructure first, then the processor that uses it.
 

How to verify it's working

 
Search your Kit log (typically at ~/isaacsim/kit/logs/Kit/Isaac-Sim Python/5.1/) for:
 

skipping initialization as '/UJITSO/geometry'=false

 

• If this line is present -> the geometry processor did NOT initialize (fix isn't applied)
• If this line is absent -> the geometry processor initialized successfully on GPU
   
  With extra_args: ["--/UJITSO/geometry=true"], you should also see these lines confirming GPU geometry processor registration:
   

[Info] [carb] Registered plugin: carb.ujitso.processor.geometry.plugin
[Info] [carb] Initializing plugin: carb.ujitso.processor.geometry.plugin
[Info] [carb.ujitsoagent.plugin] Registering container factory for type: 0x4a1a5bf06fa7b500
[Info] [carb.ujitsoagent.plugin] Registering container factory for type: 0x4453955f8192ba8
[Info] [carb.ujitsoagent.plugin] Registering container factory for type: 0xa4b964d1e198d9ce

 

Why your workarounds weren't needed

 

1. order = -100000 on omni.hydra.rtx -- Can't override hard dependency ordering (hydra.rtx -> ujitso.client -> ujitso.default), but the current order is already correct.
2. Runtime reload of omni.ujitso.default -- Crashed because it's not reloadable, and doesn't need to be -- the geometry processor was already initialized correctly during hydra.rtx startup.
    

If you're still seeing rendering issues

 
If NuRec volume rendering still isn't working despite the geometry processor loading correctly, the problem is likely elsewhere. Could you share:
 

1. What specific rendering failure you see? (black output, artifacts, missing volumes?)
2. Your full Kit log file
3. Your GPU model -- the Intel Xeon Silver 4215R is a CPU; what GPU is in the system?
    
   Happy to dig further once we can narrow down the actual symptom!
    

[hero2001]

Thank you for the detailed explanation. Confirmed — no skipping initialization line found in the logs, so the geometry processor appears to be initializing correctly.
However, the rendered output is still grayscale mesh-only (no NuRec volume color). Here are sample output images:

Environment:

GPU: NVIDIA RTX A6000 (49GB)
CPU: Intel Xeon Silver 4215R
Docker: nvcr.io/nvidia/isaac-sim:5.1.0
Headless python mode

Could the issue be related to render product resolution, rt_subframes count, or missing NuRec-specific settings?

[vick-yu]

Hi @hero2001,

To debug this further, could you please share the script you are using (with the exact dataset asset you are testing with) and the exact dataset asset you are testing with?

[vick-yu]

Hi @hero2001, we haven't heard back since we asked for your rendering script and dataset asset details. We're closing this discussion for now, but if you're still experiencing the grayscale/mesh-only output issue, please open a new discussion and reference this one (#565) so we have full context. Thank you for the detailed investigation and for confirming the --/UJITSO/geometry=true workaround!