FoundationPose with custom object

[gautham-98]

For performing pose estimation on a novel object I understand that I need -

1. A object detection model trained on the new object. For this I used : rtdetr_r101vd which seems to be the same one that the sytheticaDETR in the quickstart is using. This was then converted to engine plan model.
   /usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/custom/custom.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/custom/custom.plan

2. A .obj file for the object. I dont have a uv texture map for the object so I removed that argument from the ros2 launch command.

   ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/custom/custom.obj score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/rtdetr/custom.plan

However I was not able to pose estimate on the new object. I am using realsense 435 for the project. I also tried detection alone using isaac ros rtdetr tutorial which also seems to be not working.

I had tested the onnx model seperately and it works. I guess this must be something to do with converting to tensorRT.

[peterwang779]

@gautham-98
Hi,I encountered the same issue. Did you manage to solve it?

[ranai-srivastav]

@gautham-98 @peterwang779

When using a custom model or one from the original RT-DETR website, you might have to set scale to True in isaac_ros_rtdetr/launch/isaac_ros_rtdetr_core.launch. I changed line 96 in image_to_tensor_node from False to True and also change the confidence_threshold since the default is 0.9.

https://forums.developer.nvidia.com/t/how-to-use-a-custom-rt-detr-model-with-isaac-ros-rtdetr/330013/18