diff --git a/docs/core/examples/guides/.vscode/settings.json b/docs/core/examples/guides/.vscode/settings.json new file mode 100644 index 00000000..70ab3352 --- /dev/null +++ b/docs/core/examples/guides/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "cSpell.words": ["AICA", "Freedrive", "Orbbec", "servoing", "URDF"] +} diff --git a/docs/core/examples/guides/assets/UR-calibration-file.png b/docs/core/examples/guides/assets/UR-calibration-file.png new file mode 100644 index 00000000..93f00a74 Binary files /dev/null and b/docs/core/examples/guides/assets/UR-calibration-file.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png b/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png new file mode 100644 index 00000000..a0ebda1b Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png b/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png new file mode 100644 index 00000000..9aec092a Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png b/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png new file mode 100644 index 00000000..cac26bc4 Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png differ diff --git a/docs/core/examples/guides/assets/robot-calibration-configuration.png b/docs/core/examples/guides/assets/robot-calibration-configuration.png new file mode 100644 index 00000000..fcd64450 Binary files /dev/null and b/docs/core/examples/guides/assets/robot-calibration-configuration.png differ diff --git a/docs/core/examples/guides/assets/robot-camera-calibration-component.png b/docs/core/examples/guides/assets/robot-camera-calibration-component.png new file mode 100644 index 00000000..7de0dbc8 Binary files /dev/null and b/docs/core/examples/guides/assets/robot-camera-calibration-component.png differ diff --git a/docs/core/examples/guides/hand-eye-calibration.md b/docs/core/examples/guides/hand-eye-calibration.md new file mode 100644 index 00000000..996ed2af --- /dev/null +++ b/docs/core/examples/guides/hand-eye-calibration.md @@ -0,0 +1,332 @@ +--- +sidebar_position: 15 +title: Hand-Eye calibration +--- + +import RobotCalibrationConfiguration from './assets/robot-calibration-configuration.png' +import RobotCameraCalibration from './assets/robot-camera-calibration-component.png' +import URCalibrationFile from './assets/UR-calibration-file.png' +import FrameBroadcasterExpectedFileFormat from './assets/frame-broadcaster-expected-file-format.png' +import FrameBroadcasterPublishManually from './assets/frame-broadcaster-publish-manually.png' +import FrameBroadcasterDirectFilepath from './assets/frame-broadcaster-direct-filepath.png' + +# Hand-Eye calibration + +Robot calibration is a fundamental prerequisite for any robotic system that relies on precise coordination between a +vision sensor and a manipulator. It establishes the spatial transformation between the robot’s end-effector and the +camera frame, enabling accurate mapping between observed features and actionable robot coordinates. Without proper +calibration, even high-quality perception or motion planning algorithms can yield significant positioning errors. + +The AICA's `core-vision` package provides a structured workflow for performing hand–eye calibration efficiently and +reproducibly. + +Accurate hand–eye calibration is critical in tasks such as visual servoing, object manipulation, inspection, and +assembly. This tool is designed to minimize discrepancies and provide reliable calibration outputs suitable for +industrial environments. + +## Robot Camera Calibration component + +The Robot Camera Calibration component is the component that does the main job of calculating the transformation between +the `camera` and the `robot end-effector`. + +Click on the Robot Camera Calibration component block to view and edit the available parameters. + +
+ Robot Camera Calibration parameters +
+ +The parameters of the Robot Camera Calibration component are defined as follows: + +- **Rate**: Determines the frequency at which transformations are acquired. This parameter does not affect the + component’s behavior. +- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a + separate guide (coming soon). +- **Camera frame**: The name of the camera frame used in the application. This can be retrieved from the list of frames + in RViz. +- **Marker frame**: The name of the marker detected by the camera. It is indicated in the STag or ArUco marker detector + component used in the application. +- **Robot base frame**: The name of the robot's base frame, which can be found in RViz. +- **Robot end-effector frame**: The name of the robot’s end-effector frame. This is also available in the RViz frame + list. +- **Number of recorded points**: The number of transformations from `camera` to `robot end-effector` that are recorded. + These are computed from multiple pairs of `robot end-effector -> robot base` and `camera -> marker` transformations. +- **Distance between points**: The minimum spatial difference required between two consecutive transformations for them + to be recorded. +- **Epsilon time**: +- **Calibration folder path**: The directory where the final calibration file will be stored. +- **Calibration file**: The name of the output file generated after calibration, containing the computed calibration + data. +- **Points dataset file**: A file containing previously recorded transformations of the robot end-effector or marker, + which can be reused if available. +- **Reset calibration**: Setting this component to `True` resets the calibration. Meaning it starts the calibration + process again, even if the calibration matrices are already set. +- **Reset dataset**: If set to `True`, the recording process will re-start, even if the dataset of points already + exists. +- **Is camera attached**: This parameter should be set to `True` if the camera is attached to the robot end-effector. + +## Robot Calibration using AICA Studio and a marker + +After completing the camera calibration as described in the [Camera Calibration example](./camera-calibration.md), and +verifying marker detection as outlined in the [Fiducial Markers](./fiducial-markers.md) section, you can proceed with +the hand–eye calibration process. + +This example demonstrates the eye-in-hand configuration (camera mounted on the robot arm). The procedure for the +eye-to-hand configuration (static camera) follows a similar workflow. + +- Ensure that the camera is properly configured and operational. +- Connect all required components and controllers in the AICA application. Refer to the system setup illustration below + for guidance. +- Place the marker within the robot workspace, ensuring it is fully visible to the camera. To monitor the live camera + feed, enable `Launch RViz` from the Launcher settings, as described in the [marker detection](./marker-detection.md) + guide. +- Run the program and move the robot TCP (Tool Center Point) to capture images of the marker from multiple perspectives. + The application starts capturing images automatically. +- The robot TCP can be moved by jogging or Freedrive mode using the robot's teach pendant. In the case of using a + Universal Robot, AICA Studio offers the option of `Hand Guiding Controller` which facilitates and accelerates the + process. This controller is described in the [Hand Guiding Controller](./ur-harware-interface.md) page. +- Ensure sufficient variation in position and orientation to improve calibration accuracy. +- Once the number of captured images reaches the `Number of recorded points` indicated in the Robot Camera Calibration + component parameters, the system automatically generates a calibration file in YAML format in the following directory: + +```bash +/tmp/calibration/camera_calibration.yaml +``` + +An example of the calibration file: + +
+ An example showing a final calibration file +
+ +:::info + The order of the data in the calibration file is `w, x, y, z` for the Orientation and `x, y, z` for the +Position. +::: + +In the screenshot below you can see an example of the components configuration for the Hand-Eye calibration, that +resulted to the output above. Notice that the Camera and the marker detector components might differ based on the type +of hardware being used. + +
+ The configuration required for hand-eye calibration +
+ +The following YAML snippet contains the full application of the image above: + +
+ Example application, Hand-Eye calibration + +```yaml +schema: 2-0-6 +dependencies: + core: v5.1.0 +on_start: + load: + - component: orbbec_camera + - component: robot_camera_calibration + - component: stag_detector + - hardware: hardware +components: + orbbec_camera: + component: orbbec_camera::OBCameraNodeDriver + display_name: Orbbec Camera + outputs: + color_image: /orbbec_camera/color_image + color_camera_info: /orbbec_camera/color_camera_info + robot_camera_calibration: + component: core_vision_components::calibration::RobotCameraCalibration + display_name: Robot Camera Calibration + events: + transitions: + on_load: + lifecycle: + component: robot_camera_calibration + transition: configure + on_configure: + lifecycle: + component: robot_camera_calibration + transition: activate + parameters: + camera_frame: + value: orbbec_camera_link + type: string + marker_frame: + value: stag_0 + type: string + robot_base_frame: + value: world + type: string + robot_ee_frame: + value: ur_tool0 + type: string + is_camera_attached: + value: true + type: bool + stag_detector: + component: core_vision_components::pose_detection::STagDetector + display_name: STag Detector + events: + transitions: + on_load: + lifecycle: + component: stag_detector + transition: configure + on_configure: + lifecycle: + component: stag_detector + transition: activate + parameters: + marker_selection: + value: + - stag_0 + type: string_array + inputs: + image: /orbbec_camera/color_image + camera_info: /orbbec_camera/color_camera_info +hardware: + hardware: + display_name: Hardware Interface + urdf: Universal Robots 5e + rate: 500 + events: + transitions: + on_load: + load: + - controller: robot_state_broadcaster + hardware: hardware + - controller: ur_hand_guiding_controller + hardware: hardware + parameters: + robot_ip: 192.168.42.20 + controllers: + robot_state_broadcaster: + plugin: aica_core_controllers/RobotStateBroadcaster + events: + transitions: + on_load: + switch_controllers: + hardware: hardware + activate: robot_state_broadcaster + ur_hand_guiding_controller: + plugin: aica_ur_controllers/URHandGuidingController + parameters: + ft_sensor_name: + value: ur_tcp_fts_sensor + type: string + ft_sensor_reference_frame: + value: ur_tool0 + type: string + force_limit: + value: + - 20 + - 20 + - 20 + - 2 + - 2 + - 2 + type: vector + events: + transitions: + on_load: + switch_controllers: + hardware: hardware + activate: ur_hand_guiding_controller +graph: + positions: + on_start: + x: 120 + y: 0 + stop: + x: 120 + y: 100 + components: + orbbec_camera: + x: 340 + y: 0 + robot_camera_calibration: + x: 340 + y: 320 + stag_detector: + x: 800 + y: -80 + hardware: + hardware: + x: 1260 + y: -120 + edges: + on_start_on_start_robot_camera_calibration_robot_camera_calibration: + path: + - x: 260 + y: 60 + - x: 260 + y: 380 + on_start_on_start_stag_detector_stag_detector: + path: + - x: 260 + y: 60 + - x: 260 + y: -20 + on_start_on_start_hardware_hardware: + path: + - x: 260 + y: 60 + - x: 260 + y: -60 +``` + +
+ +:::tip +Don't forget to modify the `robot_ip` according to the ip of the robot that you are using. +::: + +There are several ways to use the transformation information obtained in the calibration file: + +1. Adding the camera link to the URDF. +2. Publish the transformation with a `FrameBroadcaster` manually. +3. Using the calibration file path directly as a Frame Broadcaster component parameter to publish the transformations. + +:::tip +In order to use the calibration file path as a `Frame Broadcaster` component parameter, the format of the +calibration file needs to be slightly modified. In the next part the expected format is explained in more detail. +::: + +## Frame Broadcaster component + +The **Frame Broadcaster** is a component that parses a frame from the parameters and/or frames from the provided file +and broadcasts all available frames to TF. + +The parameters of the Robot Camera Calibration component are defined as follows: + +- **Rate**: Determines the frequency of all periodic callbacks. This parameter does not affect the component’s behavior. +- **Filepath**: This shows the path to the file containing a frame in YAML format. An example of the expected file + format is shown in the screenshot below. + +
+ The expected file format for the Frame Broadcaster +
+ +- **Frame**: Name of the frame that we wish to broadcast. +- **reference frame**: The name of the reference frame that the frame is defined according to it. +- **Pose values**: The values of Position (meters) and Orientation (quaternions) concatenated as a vector, as in x, y, + z, qw, qx, qy, qz. +- **Broadcast periodically**: If this parameter is set to True, the frames are broadcasted periodically. Otherwise it + publishes once immediately. + +:::info +You either need to fill in the Filepath, or provide the `Frame Broadcaster` component with Frame, Reference +frame, and the Pose values parameters. +::: + +In the first screenshot below, you see an example of the `Frame broadcaster` component, and a transformation which is +being published manually. + +
+ Inserting the transformation into the frame Broadcaster manually. +
+ +In this second example, you see how the calibration filepath is given as a parameter to publish the transformations. + +
+ Using the calibration file path directly as a Frame Broadcaster component parameter to publish the transformations. +