ROS2 JointStates and deformables

[gene-maniro]

Description

Cross-posting from discord: https://discord.com/channels/827959428476174346/1423352549544562789/1423352549544562789

Any way to use ros2 moveit together with deformable objects?

If I specify SimulationCfg to use cuda, I get my deformable object but ros2 jointstate publish / subscribe fails with isaac-sim/IsaacLab#2089 (comment)

e.g. the error with ros2 and cuda sim is

2025-10-02T16:56:26Z [61,744ms] [Error] [omni.physx.tensors.plugin] Incompatible device of DOF velocity tensor in function getDofAttribute: expected device 0, received device -1
Failed to get dof velocities
2025-10-02T16:56:26Z [61,744ms] [Error] [omni.physx.tensors.plugin] Incompatible device of dof projected joint force tensor in function getDofProjectedJointForces: expected device 0, received device -1

If I specify to use cpu, even if I add sim_ctx.get_physics_context().enable_gpu_dynamics(flag=True), I can't create the deformable object

Isaac Sim version

5.1.0-rc.6

Operating System (OS)

Ubuntu 24.04

GPU Name

RTX 5090

GPU Driver and CUDA versions

Driver Version: 580.65.06 CUDA Version: 13.0

Logs

No response

Additional information

No response

[gene-maniro]

I was able to get things working with this diff (should probably be fixed, I can open a PR):

diff --git a/source/extensions/isaacsim.core.api/python/impl/controllers/articulation_controller.py b/source/extensions/isaacsim.core.api/python/impl/controllers/articulation_controller.py
index 591ca93..2ded969 100644
--- a/source/extensions/isaacsim.core.api/python/impl/controllers/articulation_controller.py
+++ b/source/extensions/isaacsim.core.api/python/impl/controllers/articulation_controller.py
@@ -77,7 +77,9 @@ class ArticulationController(object):
             )
             joint_velocities = self._articulation_view._backend_utils.expand_dims(joint_velocities, 0)
             for i in range(control_actions.get_length()):
-                if joint_velocities[0][i] is None or np.isnan(joint_velocities[0][i]):
+                if joint_velocities[0][i] is None or np.isnan(
+                    self._articulation_view._backend_utils.to_numpy(joint_velocities[0][i])
+                ):
                     joint_velocities[0][i] = applied_actions.joint_velocities[joint_indices[i]]
         joint_efforts = control_actions.joint_efforts
         if control_actions.joint_efforts is not None:
@@ -86,7 +88,9 @@ class ArticulationController(object):
             )
             joint_efforts = self._articulation_view._backend_utils.expand_dims(joint_efforts, 0)
             for i in range(control_actions.get_length()):
-                if joint_efforts[0][i] is None or np.isnan(joint_efforts[0][i]):
+                if joint_efforts[0][i] is None or np.isnan(
+                    self._articulation_view._backend_utils.to_numpy(joint_efforts[0][i].cpu())
+                ):
                     joint_efforts[0][i] = 0
         self._articulation_view.apply_action(
             ArticulationActions(

And this one (probably not something that can be merged, but any way to autodetect tensor device?)

diff --git a/source/extensions/isaacsim.ros2.bridge/library/backend/Ros2Message.cpp b/source/extensions/isaacsim.ros2.bridge/library/backend/Ros2Message.cpp
index 2ed78ec..c6fc26e 100644
--- a/source/extensions/isaacsim.ros2.bridge/library/backend/Ros2Message.cpp
+++ b/source/extensions/isaacsim.ros2.bridge/library/backend/Ros2Message.cpp
@@ -656,8 +656,7 @@ void Ros2BoundingBox2DMessageImpl::writeBboxData(const void* bboxArray, const si
             *m_generatorLibrary, detectionMsg->detections, numBoxes, "vision_msgs__msg__Detection2D__Sequence__init",
             "vision_msgs__msg__Detection2D__Sequence__fini",
             [](vision_msgs__msg__Detection2D& det) -> vision_msgs__msg__ObjectHypothesisWithPose__Sequence&
-            { return det.results; },
-            "vision_msgs__msg__ObjectHypothesisWithPose__Sequence__fini"))
+            { return det.results; }, "vision_msgs__msg__ObjectHypothesisWithPose__Sequence__fini"))
     {
         fprintf(stderr, "[Ros2BoundingBox2DMessage] Failed to ensure detection sequence\n");
         return;
@@ -756,8 +755,7 @@ void Ros2BoundingBox3DMessageImpl::writeBboxData(const void* bboxArray, size_t n
             *m_generatorLibrary, detectionMsg->detections, numBoxes, "vision_msgs__msg__Detection3D__Sequence__init",
             "vision_msgs__msg__Detection3D__Sequence__fini",
             [](vision_msgs__msg__Detection3D& det) -> vision_msgs__msg__ObjectHypothesisWithPose__Sequence&
-            { return det.results; },
-            "vision_msgs__msg__ObjectHypothesisWithPose__Sequence__fini"))
+            { return det.results; }, "vision_msgs__msg__ObjectHypothesisWithPose__Sequence__fini"))
     {
         fprintf(stderr, "[Ros2BoundingBox3DMessage] Failed to ensure detection sequence\n");
         return;
@@ -998,7 +996,8 @@ template <typename T>
 static void createTensorDesc(omni::physics::tensors::TensorDesc& tensorDesc,
                              std::vector<T>& buffer,
                              int numElements,
-                             omni::physics::tensors::TensorDataType type)
+                             omni::physics::tensors::TensorDataType type,
+                             int device)
 {
     buffer.resize(numElements);
     tensorDesc.dtype = type;
@@ -1006,7 +1005,7 @@ static void createTensorDesc(omni::physics::tensors::TensorDesc& tensorDesc,
     tensorDesc.dims[0] = numElements;
     tensorDesc.data = buffer.data();
     tensorDesc.ownData = true;
-    tensorDesc.device = -1;
+    tensorDesc.device = device;
 }
 
 void Ros2JointStateMessageImpl::writeData(const double& timeStamp,
@@ -1030,10 +1029,10 @@ void Ros2JointStateMessageImpl::writeData(const double& timeStamp,
     omni::physics::tensors::TensorDesc velocityTensor;
     omni::physics::tensors::TensorDesc effortTensor;
     omni::physics::tensors::TensorDesc dofTypeTensor;
-    createTensorDesc(positionTensor, jointPositions, numDofs, omni::physics::tensors::TensorDataType::eFloat32);
-    createTensorDesc(velocityTensor, jointVelocities, numDofs, omni::physics::tensors::TensorDataType::eFloat32);
-    createTensorDesc(effortTensor, jointEfforts, numDofs, omni::physics::tensors::TensorDataType::eFloat32);
-    createTensorDesc(dofTypeTensor, dofTypes, numDofs, omni::physics::tensors::TensorDataType::eUint8);
+    createTensorDesc(positionTensor, jointPositions, numDofs, omni::physics::tensors::TensorDataType::eFloat32, 0);
+    createTensorDesc(velocityTensor, jointVelocities, numDofs, omni::physics::tensors::TensorDataType::eFloat32, 0);
+    createTensorDesc(effortTensor, jointEfforts, numDofs, omni::physics::tensors::TensorDataType::eFloat32, 0);
+    createTensorDesc(dofTypeTensor, dofTypes, numDofs, omni::physics::tensors::TensorDataType::eUint8, -1);
     bool hasDofStates = true;
     if (!articulation->getDofPositions(&positionTensor))
     {

[zwdoescode]

@gene-maniro Thanks! Please feel free to open a PR. Then I can create an internal ticket for the team to evaluate your changes

Internal ticket: 6112271

[genemerewether]

For the first issue (controlling the articulation from ROS with sim on GPU): #272

[genemerewether]

And for the second issue (reading joint states and publishing to ROS with sim on GPU): #293