[Bug Report] OVRTX multi-camera sharing causes stale views and temporal artifacts

[nblauch]

Problem

OVRTX camera sensors that share a renderer also share its native camera binding. In the upstream Kuka base/wrist camera task, neither value of update_latest_camera_pose gives both correct viewpoints and stable video:

Both cameras' flag	GPU-observed behavior
False (default)	Both sensors return nearly the same stale viewpoint.
True	Base and wrist viewpoints are correct, but visible shimmer and smearing remain, including with frozen physics.

Expected: each sensor retains its own viewpoint and stable temporal rendering. The same task with IsaacSim RTX returns correct distinct views with the default flag. The adapter's shared camera binding is one issue. Separate native render products expose an additional transform-update/temporal-rendering problem on the tested OVRTX pin; results and handoff are below.

Reproduction and video evidence

Task: Isaac-Reorient-KukaAllegro-Camera, presets newton_mjwarp,ovrtx,cube,duo_camera,rgb64.

Download the capture script and run it in the pinned environment below, selecting an idle GPU:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=GPU-YOUR-IDLE-GPU-UUID
export ISAAC_LAB_OVRTX_USE_OVSTAGE=0 OVRTX_SKIP_USD_CHECK=1
python temporal_video.py --refresh 0 --output captures-off --visualizer none
python temporal_video.py --refresh 1 --output captures-on --visualizer none

The script sets the flag on both cameras before construction and uses 4 096 environments. Each 12-second video contains four seconds of frozen physics, four seconds of task motion, then four seconds frozen again.

Watch the comparison: flag off on the left, flag on on the right. Within each panel, columns are base/wrist and rows are environments 0–2. Individual videos, lossless captures and measurements.

With the flag enabled, mean absolute consecutive-frame channel differences during the initial static segment are 3.15 / 4.90 for base/wrist on the 0–255 scale. Disabling the flag gives approximately 0.14 for both, but those images show the incorrect stale viewpoint; that is not a valid quality baseline for the correct views.

Tested environment and capture protocol

• IsaacLab 9adf4831384dbe9320de72e9b7809c0d87252b9a; OVRTX 0.4.1.364340; ovstage 0.1.1.355824; Newton 24bd863528d6b91137408930d0fbe8fa216ad962.
• RTX 6000 Ada, driver 580.173.02, Ubuntu 24.04, Python 3.12. No competing compute process was detected in included runs.
• Kit is not launched for OVRTX. External ovstage is disabled. Video runs explicitly use OVRTXRendererConfig(enable_geometry_streaming=False) in renderer initialization; no camera-routing/history patches or AA overrides.
• Five action steps, then 40 camera-pair warm-ups before recording. Static segments assert unchanged joint state. Capture calls camera.update(..., force_recompute=True) in base-then-wrist order. No explicit pose writes or texture edits during capture. Lossless videos were verified against every raw frame.

Adapter cause and suggested fix

create_render_data() initializes native camera state from the first sensor specification. _update_camera_legacy() writes the same binding for subsequent sensors, and _render_legacy() reads the shared product. Enabling pose updates alternates viewpoints through those same native camera slots.

Give each sensor persistent native camera slots. The combined-product approach below is GPU-validated for equal-resolution cameras and produces stable stereo video using the pinned OVRTX API. Separate products accommodate different resolutions, but currently need the slower workaround described below.

# Pseudocode: each record belongs to one sensor across all N environments.
records = register_camera_specs_before_first_render()
paths = []
for record in records:
    record.tiles = slice(len(paths), len(paths) + record.num_envs)
    paths.extend(record.camera_paths)
    assign_environment_partitions(record.camera_paths)
    record.pose_binding = bind_transforms(record.camera_paths)

# N environments, 2N persistent camera slots; one shared physical scene.
product = create_tiled_product(camera_paths=paths)

def capture(sample):
    if cache.matches(sample):
        return  # The other sensor consumes this same capture.
    sync_scene(sample)
    for record in records:
        record.pose_binding.write(resolve_current_pose(record, sample))
    frame = renderer.step(render_products={product}, delta_time=sample.dt)
    for record in records:
        extract_tiles_gpu(frame, record.tiles, record.output_buffers)
    cache.mark_complete(sample)

Both poses must be current before the native render, including the articulation-mounted wrist camera. Keep slot ordering/product membership stable. Cache validity must include camera edits and resets, not only physics-step count; repeated camera-only captures must still render when requested. OVRTX manages temporal reconstruction for the persistent slots—this design does not manually allocate or reset its history buffers.

Validated local integration results

A separate local stereo integration tested the working implementation with 512 environments, RGB160×120 per eye, depth disabled, on the same Ada GPU. Static frame differences fell from 6.85 / 9.84 to 0.10 / 0.15 for left/right, and both views remained clear during camera motion.

Newton renderer / camera routing	Mean step	Env FPS	Peak GPU memory
OVRTX, alternating shared slots	181.63 ms	2 819	11.36 GB
OVRTX, persistent combined batch	160.35 ms	3 193	14.67 GB
OVRTX, separate products, read_gpu_transforms=False	257.43 ms	1 989	13.97 GB
IsaacSim RTX	237.17 ms	2 159	22.48 GB

The separate-product workaround resolves the visual problem but loses the combined product's throughput advantage. This comparison changes both product layout and transform propagation; it does not isolate their individual performance costs.

Benchmark and video measurement details

These are full environment steps without policy/PPO: one fresh process per variant, 30 warm-up and 80 synchronized timed steps. Memory is decimal NVML process memory, including initialization. The separate-product run additionally recorded task motion after timing; the other rows are the earlier matched runs.

Quality settings were retained: OVRTX DLSS Auto, IsaacSim RTX DLAA. The combined product is larger and may affect DLSS Auto's internal resolution. Video runs used matched seeds/configuration and frozen physics within each run, without enforced bitwise state replay across runs.

Separate render products: findings for handoff

Using the same pinned SDK and local integration, we gave each eye its own native camera prims, transform binding, render product and output routing. Both poses were published before rendering, both products stayed active in every native renderer.step(), and both update_latest_camera_pose flags remained True.

• GPU transform propagation enabled: the second native view shimmered when transforms were updated, regardless of which eye occupied that view. Camera writes and scene-transform writes independently reproduced it. Holding transforms allowed clean accumulation.
• ISAAC_LAB_OVRTX_READ_GPU_TRANSFORMS=0, selecting RendererConfig(read_gpu_transforms=False), resolved the observed shimmer. This changes world-transform propagation; physics and RTX rendering remain on the GPU. In the controlled 64-environment camera-write comparison, static frame differences changed from 0.10 / 4.82 to 0.10 / 0.10. Scene-only writes showed the same pattern.
• The workaround also passed 512-environment camera-motion videos at equal resolutions and at left160×120 / right128×96, plus a normal task-step recording with moving robot geometry. Equal-resolution static differences were 0.12 / 0.14. Native-resolution lossless recordings were verified against raw frames. Camera motion was explicitly driven in world space; this follow-up did not specifically exercise wrist-parent transform inheritance.
• Persistent GPU camera buffers, synchronous CPU camera uploads, CPU output extraction, removing duplicate pose publication, and render-mode spelling changes did not resolve the enabled path. The successful runs used the original camera upload and output extraction functions.

Handing this to the IsaacLab/OVRTX team: integrate persistent per-sensor camera ownership and investigate why transform updates disrupt the second native view with GPU propagation enabled. Separate products support the desired resolution flexibility, but the global propagation workaround is too costly. The precise native cause remains open; no further local experiments are planned for this handoff.

Acceptance criteria

• Correct, temporally stable base/wrist video during motion and frozen-physics capture.
• Correct output routing regardless of sensor-read order, including after camera edits and resets.
• Persistent native camera slots, with all current poses prepared before rendering the requested products together.
• Preserve supported single-camera outputs; explicitly handle or reject incompatible multi-camera resolutions, modalities and cadences.

[nblauch]

Reproduction assets

Flag-off/on comparison video, capture script, raw-pixel measurements and environment metadata, and individual/lossless videos.

The flag-on video demonstrates temporal instability even while physics is frozen. The flag-off video demonstrates the incorrect shared/stale viewpoint. Static pixel-difference measurements use frames 20–119 and environments 0–2; they are computed from raw RGB, not compressed video.

IsaacSim RTX viewpoint-control script and captures show correct distinct base/wrist views with its default pose flag.

[nblauch]

Integration notes for the batched-camera fix

The issue body contains the GPU-validated approach, measured results and pseudocode. Implementation details to preserve:

• Store native camera paths, transform bindings, tile slices and output buffers per sensor.
• Coordinate current poses before rendering; the first sensor read must not capture the other camera's previous pose.
• Use an explicit capture request/revision for output-cache validity. Physics-step count alone misses camera-only captures, pose edits and resets.
• Keep native slots/product membership stable; invalidate application output caches without unnecessarily recreating native views.
• Release cached native frames and bindings through normal cleanup. Define compatibility for resolutions, modalities and sampling cadences explicitly.

The validated prototype covers synchronized, equal-resolution RGB cameras on the renderer-owned path. The optional external-ovstage path needs an explicit implementation/support decision.