The following repository has the source, resources, and instructions for building a robotic arm and its digital twin. The arm is based on a modified Koch V1.1 "follower arm" physical design (see Robotis Koch v1.1 follower arm) and its powered by The Robot Operating System v2 ROS2.
(NEW) The current effort is focused on enabling this robotic arm to balance a cup containing a metal ball bearing. Balancing is a perennial challenge with robotics (as humanoid robots often fall at embarrassing times ;-)). The koch v1.1 hardware is a bit limited to explore all aspects of balance, but useful as a starter task. This is being accomplished by enhancing the robot with additinoal hardware (IMU sensor and inferencing camera) and software, including a Balance PID controller (PID = Proportional–integral–derivative) and a vision ML model among other modules. Learn more ..
This branch supports Ros2 Jazzy (4.2+) and a combination with Humble for the ball bearing detection inferencing.
The details in this project include:
- Fabrication and/or purchase, modification, and assembly of robotic parts (brackets, Dynamixel servos, and related circuit boards and sensors)
- Testing and tuning of the physical robot
- Creating, calibrating, and tuning of the robot arm digital twin (that runs in rviz),
- Refinement of ROS2-based software to support various actions of the robotic arm and the gathering of physical statistics (current, voltage, temperature, etc)
- The development of the ML models for ball detection and the PID controller for for balance.
The documentation for this repository covers the following:
- Software Setup: how to create the docker envrionment and any additional setup commands.
- Hardware Setup: Although the setup of the koch_v11 physical robot (leader and follower arms) is partially documented (Arm assembly by Jess Moss), those instructions lacked details on some of the required electrical setup. In addition, we are using different controller boards (e.g., Dynamixel OpenRB 150).
- Firmware Setup and Configuration: Notes on the firmware setup and configuration for the OpenRB 150, the servo motors, the current sensors, and the ESP32.
- Notes on creating the digital twin: A series of notes that cover the problems and solutions for creating the digital twin.
- Launching the Physical Robot Steps involved in bringing up the physical robot.
- Running the display only version of the Digital twin: This describes how to bring up the digital twin - either along side the physical robot or by itself as an rviz display-only flavor of the robot.
- Pose Sequence Tools and Analysis: Several tools are provided that drive the robot through a series of poses and analyze physical characteristics (position, temp, current, voltage, etc). This section also provides some analysis case studies that are of interest.
- NEW Balancing a cup with the robotic arm : These notes include detailed descriptions of the add-ons to the Koch v1.1 hardware (3D Inferencing camera and IMU) and software: PID related nodes and Machine Learning models for inferencing on the Camera and on an Nvidia Orin Nano.
Animations above:
- The digital twin executing a series of poses that exercise all of the robotic joints.
- The full rviz output showing a small excerpt of the pose and PID operations. The lower lefthand side is an bounding box annotated view of the camera (image after inferencing) and the right hand side 3D view shows the corresponding movement of the cup, ball, and robot. In this video 5 different tipping poses are set moving the ball bearing to one of the cup. Following each the balance pid controller kicks in and balances the ball within the cup.
This repository can be of use to someone building robotics in several ways:
-
Developers using the Koch V1.1 follower arm: Although some of the design and operation of the Koch v1.1 robot arm is open source, a developer of it may encounter a number of gaps or issues when working with it. This repository should complement information and software from ROBOTIS and other sources. For example, we provide a koch v1.1 URDF file that is configured and calibrated to provide a digital twin that matches the physical robot in appearance. In addition, we provide complete and accurate wiring diagrams that detail how to co-exist the 12V and 5V Dynamixel servos in the same environment.
-
Developers interested in PID controllers and balance: Achieving balance in a robotics system has been a challenging research and development area. Humanoid robots still fall over regularly in critical demos. Although the koch v1.1 robot arm is limited in a number of ways the effort of getting it to balance a cup is a good starter task in this area.
-
Developers integrating vision into their robotics system: This is perhaps less challenging than achieving balance, but still difficult and covered in much detail. We explore inference on the camera vs. on an Nvidia Orin Nano.
-
Developers wanting to pull more telemetry from Dynamixel servos in a ROS2 environment: The ros2 interface for the Dynamixel servos is quite limited relative to the internal data it can pull (as compared with the Dynamixel Wizard tool). For example, one can't get a reading on the voltage seen by a servo, even though extra strain on the system could lead to an under-volt situation for the servos in series. We provide information on how to improve this for ros2 and how to set this up (in code) and retrieve the values at runtime via ros2.
-
Developers who are adapting robotic arms to ROS2 or not starting from a well defined kit: ROBOTIS provides several full kits for someone getting started with robot arms that provide a good head start including this Koch v1.1., OMX AI manipulator, OMY AI manipulator, and more. This project provides information for someone who is departing from an easy-to-assemble kit.
A leader-follower robotic arm system consists of two robotic manipulators where one arm (the leader) is manually controlled by a human operator, and the second arm (the follower) replicates those movements in real-time. It's essentially a form of bilateral teleoperation. In this project we are primarily focused on the follower arm.
In general leader arms are meant more for human manipulation as they define the poses that the follower will replicate. That limited operational scope for the leader means that its likely that the arm is under powered and/or has a design that is for human manipulation vs. automation. In this project we use the follower as the leader (to predefine poses and collect the measurements) albiet in a less automatic fashion. We also want to study the follower's load and physical characeteristics (e.g., temperature) as it repeatedly performs the activity. In addition, the cup balancing sub-project doesn't quite fit the leader/follower paradigm.
This upgrade to JetPack 7.2 with Jetson Linux 39.2 will allow this project to by-pass the inferencing on the camera and increase the frames per sec inferencing from 12 to 35 FPS. Aside from performance the benefit is an all Jazzy architecture with that approach.
This sub-project is about extending the software and hardware support to a different robotic arm the SO-ARM101 which is based on Feetech servos rather than Dynamixel servos.
Currently we are not using the 3D capabilities of the OAK-D lite camera due to its computational load on the camera. Once we have a Jazzy-based Nvidia inferencing stack, then this becomes something we can potentially leverage.