Skip to content

[Bug Report] Camera pose writes do not reach the renderer on the Newton backend #7676

Description

@alexmillane

Describe the bug

Camera.set_world_poses moves the camera on PhysX but not on Newton. Under Newton, camera.data.pos_w reports the new pose while the rendered image stays at the old one.

Steps to reproduce

Run the regression test here:

ISAACLAB_TEST_SAVE_IMAGES=1 uv run python -m pytest \
  source/isaaclab/test/sensors/test_camera_pose_write_across_backends.py -v -s

It moves a downward-looking camera from 2 m to 8 m over a ground plane and checks both consequences of the write, on both backends:

test_camera_pose_write_moves_reported_pose[physx]  PASSED
test_camera_pose_write_moves_reported_pose[newton] PASSED
test_camera_pose_write_moves_render[physx]         PASSED
test_camera_pose_write_moves_render[newton]        FAILED
  Far depth (2.08 m) should be > 1.5x close depth (2.08 m); got 1.00x.

The test saves some images for inspection

PhysX

Image Image

Images correctly change in response to moving the camera from 2m -> 8m

Newton

Image Image

No change from moving the camera from 2m -> 8m

System Info

  • Commit: 8c3bc5e8e
  • Isaac Sim Version: 6.0.1
  • OS: Ubuntu 22.04
  • GPU: RTX 5090
  • CUDA: 12.2
  • GPU Driver: 580.65.06

Additional context

Found via a camera-extrinsics domain randomization that silently stopped varying the image after switching to Newton. Workaround is to mirror the local pose onto the USD camera prim through UsdFrameView in addition to the physics-view write, applied downstream in IsaacLab-Arena#1196.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

  • Camera.set_world_poses moves the rendered image on Newton as it does on PhysX
  • The linked test passes on both backends

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions