diff --git a/README.md b/README.md index 7e0965d5..35054b12 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,50 @@ # MoveIt Pro Example Workspace -This is fork of the [MoveIt Pro Empty Workspace](https://github.com/PickNikRobotics/moveit_pro_empty_ws). -This workspace contains reference materials for using MoveIt Pro, including: -- [Example base UR5e configuration](src/moveit_pro_ur_configs/picknik_ur_base_config) -- [A physics based simulation environment with a robot on a linear rail](src/lab_sim) -- [Mobile manipulation configuration](src/hangar_sim) -- [Example behaviors](src/example_behaviors) +This workspace contains reference materials for using MoveIt Pro, including example robot configurations, simulated environments, and reusable behaviors. -Since the [picknik_accessories](https://github.com/PickNikRobotics/picknik_accessories) package uses git LFS, [it cannot be added as a subtree](https://github.com/git-lfs/git-lfs/issues/854). -Please ensure you have the submodule up to date using: +## Cloning + +This repository uses git submodules. Clone with: ```bash -git submodule update --recursive --init -git submodule foreach --recursive git lfs pull +git clone --recurse-submodules ``` -## Working with Git Subtrees - -This repository was created through the combination of multiple repositories using git subtree. -If you have no interest in manually pulling or pushing upstream changes, you can ignore the following section and treat this repository as a single repository. - -### Repository Structure - - -The structure of this repository is as follows: - -
-.
-├── README.md
-└── src
-    ├── example_behaviors
-    ├── lab_sim
-    ├── moveit_pro_ur_configs
-    │   ├── picknik_ur_base_config
-    │   ├── mock_sim
-    │   ├── multi_arm_sim
-    │   ├── picknik_ur_sim_config
-    │   └── picknik_ur_site_config
-    ├── moveit_pro_kinova_configs
-    │   ├── kinova_gen3_base_config
-    │   ├── kinova_sim
-    │   └── moveit_studio_kinova_pstop_manager
-    ├── moveit_pro_mobile_manipulation
-    │   ├── mobile_manipulation_config
-    │   └── picknik_ur_mobile_config
-    ├── fanuc_sim
-    ├── picknik_accessories (submodule)
-    └── external_dependencies
-        ├── ridgeback
-        ├── ros2_robotiq_gripper
-        └── serial
-
- -This repository contains a **copy** of the git repositories that were added as subtrees. -File changes and commits are treated as if they happen only in this repository. -If you update the contents of a subtree, you can merge the latest `main` branch of [lab_sim](https://github.com/PickNikRobotics/lab_sim) using the following command: +If you already cloned without submodules, initialize them with: ```bash -git subtree pull --prefix src/lab_sim https://github.com/PickNikRobotics/lab_sim main --squash +git submodule update --recursive --init ``` -To pull the upstream changes to all subtrees and submodules, a convenience script is provided. -From the top level, you can execute: +Several submodules (notably `picknik_accessories`) use git LFS. Install [git-lfs](https://git-lfs.com/) first (e.g., `sudo apt install git-lfs && git lfs install`); without it the commands below fail with `git: 'lfs' is not a git command`. After updating submodules, pull LFS objects: ```bash -./sync_subtrees.sh +git submodule foreach --recursive git lfs pull ``` -## Scripts - -Utility scripts for maintaining the PickNik website are located in the `scripts/` directory. -See [scripts/README.md](scripts/README.md) for documentation. +## Robot Configs + +- `april_tag_sim` +- `dual_arm_sim` +- `factory_sim` +- `grinding_sim` +- `hangar_sim` +- `kitchen_sim` +- `lab_sim` +- `lunar_sim` +- `phoebe_sim` +- `moveit_pro_franka_configs/franka_base_config` +- `moveit_pro_kinova_configs/kinova_gen3_base_config` +- `moveit_pro_kinova_configs/kinova_gen3_site_config` +- `moveit_pro_kinova_configs/kinova_sim` +- `moveit_pro_kinova_configs/space_satellite_sim` +- `moveit_pro_kinova_configs/space_satellite_sim_camera_cal` +- `moveit_pro_ur_configs/mock_sim` +- `moveit_pro_ur_configs/multi_arm_sim` +- `moveit_pro_ur_configs/picknik_ur_base_config` +- `moveit_pro_ur_configs/picknik_ur_site_config` + +## Updating Submodules + +To pull the latest commits for all submodules: +```bash +git submodule update --remote --recursive +git submodule foreach --recursive git lfs pull +``` diff --git a/scripts/README.md b/scripts/README.md deleted file mode 100644 index 7a43d7ba..00000000 --- a/scripts/README.md +++ /dev/null @@ -1,92 +0,0 @@ -# Scripts - -Utility scripts for maintaining the PickNik website, in particular the Behaviors Hub. - -### How to update the Behavior Hub data source -These steps allow you to get the updated behaviors.xml, which should happen every release. - -```bash -# In moveitpro repo when your REST API is running, get the generated XML data -curl -X GET http://localhost:3200/behaviors/data > behaviors_raw.xml -``` - -### enrich_behaviors_with_usage.py - -Enhances `behaviors_raw.xml` with usage information (Metadata fields with `used_in` attribute) showing which Objectives use each behavior. - -**Run this script in `moveit_pro_example_ws` workspace.** - -#### Prerequisites - -1. Save `behaviors_raw.xml` (downloaded from REST API) to the `scripts/` directory in `moveit_pro_example_ws` workspace. - -#### Usage - -```bash -# From moveit_pro_example_ws root directory -python3 scripts/enrich_behaviors_with_usage.py --xml scripts/behaviors_raw.xml --workspace . --output scripts/behaviors_with_usage.xml - -# Example with custom workspace path -python3 scripts/enrich_behaviors_with_usage.py --xml scripts/behaviors_raw.xml --workspace /path/to/workspace --output scripts/behaviors_with_usage.xml -``` - -#### What it does - -- Discovers all robot configs in the workspace -- Extracts which Objectives use which behaviors by parsing Objective XML files -- Preserves all existing XML data (ports, metadata, etc.) -- Adds `Metadata` elements with `used_in` attribute to each behavior's `MetadataFields` showing: - - Which configs use the behavior - - Which Objectives use the behavior - - The file path of each Objective - - The usage type (Action, Control, Decorator, SubTree) -- The `used_in` attribute contains a JSON array of usage objects -- Outputs enhanced XML file ready for conversion to JSON - -#### Requirements - -- Python 3 -- PyYAML (`pip install pyyaml`) -- `behaviors_raw.xml` file (from REST API endpoint) -- `moveit_pro_example_ws` workspace with robot configs - -### update_behaviors_from_xml.py - -Updates `_data/behaviors.json` with port information and metadata extracted from `behaviors_with_usage.xml`. - -**Run this script in the PickNik website repository.** - -#### Usage - -```bash -# From project root -python scripts/update_behaviors_from_xml.py [path_to_behaviors_with_usage.xml] - -# Example with default path -python scripts/update_behaviors_from_xml.py - -# Example with custom XML path -python scripts/update_behaviors_from_xml.py /path/to/behaviors_with_usage.xml -``` - -#### What it does - -- Extracts port information (input, output, input/output) from XML Action and SubTree elements -- Extracts metadata (subcategory, description, deprecated status) -- Extracts `used_in` information from `Metadata` elements with `used_in` attribute (if present) -- Updates existing behaviors in JSON or adds new ones -- Preserves existing data like links and other fields -- Updates the export date - -### Complete Workflow - -```bash -# 1. In moveitpro repo - Get behaviors.xml from REST API and save to scripts/ directory -curl -X GET http://localhost:3200/behaviors/data > scripts/behaviors_raw.xml - -# 2. In moveit_pro_example_ws root - Enhance XML with usage information -python3 scripts/enrich_behaviors_with_usage.py --xml scripts/behaviors_raw.xml --workspace . --output scripts/behaviors_with_usage.xml - -# 3. In PickNik website repo - Convert to JSON -python scripts/update_behaviors_from_xml.py _data/behaviors_with_usage.xml -``` diff --git a/scripts/enrich_behaviors_with_usage.py b/scripts/enrich_behaviors_with_usage.py deleted file mode 100755 index ca15fa6f..00000000 --- a/scripts/enrich_behaviors_with_usage.py +++ /dev/null @@ -1,416 +0,0 @@ -#!/usr/bin/env python3 -""" -Enhance behaviors.xml with usage information from Objectives. - -This script reads behaviors.xml (from REST API) and adds Metadata fields -with used_in information showing which Objectives use each behavior. - -Run this script in moveit_pro_example_ws workspace. -""" - -import argparse -import json -import os -import sys -from pathlib import Path -from typing import Dict, List, Optional -from xml.etree import ElementTree as ET - -try: - import yaml -except ImportError: - print( - "Error: PyYAML is required. Install with: pip install pyyaml", file=sys.stderr - ) - sys.exit(1) - - -def normalize_behavior_id(behavior_id: str) -> str: - """Normalize behavior ID to match format used in XML.""" - return behavior_id.lower().replace(" ", "").replace("-", "").replace("_", "") - - -def discover_configs(workspace_path: Path) -> List[Dict[str, str]]: - """ - Walk workspace, find dirs with package.xml + config/config.yaml. - Skip COLCON_IGNORE dirs. - """ - configs = [] - workspace_path = Path(workspace_path).resolve() - - for root, dirs, files in os.walk(workspace_path): - # Skip build, install, log directories - if any(skip in root for skip in ["/build/", "/install/", "/log/", "/.git/"]): - dirs[:] = [] - continue - - # Check for COLCON_IGNORE - if (Path(root) / "COLCON_IGNORE").exists(): - dirs[:] = [] - continue - - # Check if this directory has package.xml and config/config.yaml - package_xml = Path(root) / "package.xml" - config_yaml = Path(root) / "config" / "config.yaml" - - if package_xml.exists() and config_yaml.exists(): - # Skip cookiecutter templates (they contain Jinja2 syntax, not valid YAML) - try: - with open(config_yaml, "r") as f: - content = f.read() - if "{{" in content or "cookiecutter" in root: - continue - except Exception: - continue - - # Extract package name from package.xml - try: - tree = ET.parse(package_xml) - root_elem = tree.getroot() - name_elem = root_elem.find("name") - if name_elem is not None: - package_name = name_elem.text.strip() - configs.append( - { - "name": package_name, - "path": root, - "config_path": str(config_yaml), - } - ) - except Exception: - continue - - return configs - - -def parse_config_yaml(config_path: Path) -> Dict: - """Load config.yaml, handle based_on_package inheritance.""" - with open(config_path, "r") as f: - config_data = yaml.safe_load(f) or {} - - # Handle inheritance - if "based_on_package" in config_data: - base_package = config_data["based_on_package"] - # Try to find base config - workspace_path = config_path.parent.parent.parent - base_config_path = workspace_path / base_package / "config" / "config.yaml" - if base_config_path.exists(): - base_config = parse_config_yaml(base_config_path) - # Merge base config with current config (current overrides base) - merged = {**base_config, **config_data} - return merged - - return config_data - - -def find_package_share_path(package_name: str, workspace_path: Path) -> Optional[Path]: - """Find package share directory in workspace.""" - for root, dirs, files in os.walk(workspace_path): - if "/build/" in root or "/install/" in root or "/.git/" in root: - continue - package_xml = Path(root) / "package.xml" - if package_xml.exists(): - try: - tree = ET.parse(package_xml) - root_elem = tree.getroot() - name_elem = root_elem.find("name") - if name_elem is not None and name_elem.text == package_name: - return Path(root).resolve() - except Exception: - continue - return None - - -def extract_behaviors_used_in_objectives( - objective_library_paths: Dict, config_name: str, workspace_path: Path -) -> Dict[str, List[Dict]]: - """ - For each Objective, find which behaviors are used. - Returns mapping: behavior_id -> [list of objectives where it's used] - """ - behavior_to_objectives: Dict[str, List[Dict]] = {} - objective_files = [] - - # Collect all objective XML files - for library_name, library_info in objective_library_paths.items(): - package_name = library_info.get("package_name") - relative_path = library_info.get("relative_path", "objectives") - - if not package_name: - continue - - package_path = find_package_share_path(package_name, workspace_path) - if not package_path: - install_path = ( - workspace_path - / "install" - / package_name - / "share" - / package_name - / relative_path - ) - if install_path.exists(): - package_path = install_path.parent.parent.parent.parent / "src" - package_path = find_package_share_path(package_name, package_path) - if not package_path: - continue - - objectives_dir = package_path / relative_path - if not objectives_dir.exists(): - continue - - for xml_file in objectives_dir.rglob("*.xml"): - objective_files.append(xml_file) - - # Parse each objective XML file - for objective_file in objective_files: - try: - tree = ET.parse(objective_file) - root = tree.getroot() - - # Find all BehaviorTree elements (Objectives) - for behavior_tree in root.findall(".//BehaviorTree"): - objective_id = behavior_tree.get("ID") - if not objective_id: - continue - - # Find all behaviors used in this Objective - # 1. SubTree references - for subtree in behavior_tree.findall(".//SubTree[@ID]"): - behavior_id = subtree.get("ID") - if behavior_id: - behavior_id_lower = normalize_behavior_id(behavior_id) - if behavior_id_lower not in behavior_to_objectives: - behavior_to_objectives[behavior_id_lower] = [] - behavior_to_objectives[behavior_id_lower].append( - { - "config": config_name, - "objective_id": objective_id, - "objective_file": str( - objective_file.relative_to(workspace_path) - ), - "usage_type": "SubTree", - "source_type": "objective_usage", - } - ) - - # 2. Action nodes - for action in behavior_tree.findall(".//Action[@ID]"): - behavior_id = action.get("ID") - if behavior_id: - behavior_id_lower = normalize_behavior_id(behavior_id) - if behavior_id_lower not in behavior_to_objectives: - behavior_to_objectives[behavior_id_lower] = [] - behavior_to_objectives[behavior_id_lower].append( - { - "config": config_name, - "objective_id": objective_id, - "objective_file": str( - objective_file.relative_to(workspace_path) - ), - "usage_type": "Action", - "source_type": "objective_usage", - } - ) - - # 3. Control nodes - for control in behavior_tree.findall(".//Control[@ID]"): - behavior_id = control.get("ID") - if behavior_id: - behavior_id_lower = normalize_behavior_id(behavior_id) - if behavior_id_lower not in behavior_to_objectives: - behavior_to_objectives[behavior_id_lower] = [] - behavior_to_objectives[behavior_id_lower].append( - { - "config": config_name, - "objective_id": objective_id, - "objective_file": str( - objective_file.relative_to(workspace_path) - ), - "usage_type": "Control", - "source_type": "objective_usage", - } - ) - - # 4. Decorator nodes - for decorator in behavior_tree.findall(".//Decorator[@ID]"): - behavior_id = decorator.get("ID") - if behavior_id: - behavior_id_lower = normalize_behavior_id(behavior_id) - if behavior_id_lower not in behavior_to_objectives: - behavior_to_objectives[behavior_id_lower] = [] - behavior_to_objectives[behavior_id_lower].append( - { - "config": config_name, - "objective_id": objective_id, - "objective_file": str( - objective_file.relative_to(workspace_path) - ), - "usage_type": "Decorator", - "source_type": "objective_usage", - } - ) - - except Exception as e: - print(f"Warning: Failed to parse {objective_file}: {e}", file=sys.stderr) - continue - - return behavior_to_objectives - - -def enhance_behaviors_xml( - xml_path: Path, workspace_path: Path, output_path: Path = None -) -> None: - """Enhance behaviors.xml with usage information.""" - - # Read existing XML - print(f"Reading {xml_path}...", file=sys.stderr) - tree = ET.parse(xml_path) - root = tree.getroot() - - tree_nodes_model = root.find("./TreeNodesModel") - if tree_nodes_model is None: - print("Error: No TreeNodesModel found in XML", file=sys.stderr) - sys.exit(1) - - # Create a map of behavior IDs to elements - behavior_elements = {} - for elem in tree_nodes_model: - behavior_id_attr = elem.get("ID") - if behavior_id_attr: - behavior_id = normalize_behavior_id(behavior_id_attr) - behavior_elements[behavior_id] = elem - - print(f"Found {len(behavior_elements)} behaviors in XML", file=sys.stderr) - - # Discover configs and extract usage information - print(f"Discovering configs in {workspace_path}...", file=sys.stderr) - configs = discover_configs(workspace_path) - print(f"Found {len(configs)} configs", file=sys.stderr) - - # Collect all objective usages - all_objective_usages: Dict[str, List[Dict]] = {} - - # Process each config - for config in configs: - config_name = config["name"] - config_path = Path(config["config_path"]) - - print(f"Processing config: {config_name}", file=sys.stderr) - - try: - config_data = parse_config_yaml(config_path) - objective_library_paths = config_data.get("objectives", {}).get( - "objective_library_paths", {} - ) - - # Extract which behaviors are used in which Objectives - objective_usages = extract_behaviors_used_in_objectives( - objective_library_paths, - config_name, - workspace_path, - ) - - # Merge into global map - for behavior_id, usages in objective_usages.items(): - if behavior_id not in all_objective_usages: - all_objective_usages[behavior_id] = [] - all_objective_usages[behavior_id].extend(usages) - - print( - f" Found {len(objective_usages)} behaviors used in Objectives", - file=sys.stderr, - ) - - except Exception as e: - print(f" Error processing {config_name}: {e}", file=sys.stderr) - import traceback - - traceback.print_exc() - continue - - # Add used_in Metadata to each behavior element - print("Adding usage information to XML...", file=sys.stderr) - behaviors_with_usage = 0 - - for behavior_id, elem in behavior_elements.items(): - # Remove existing used_in Metadata if any - metadata_fields = elem.find(".//MetadataFields") - if metadata_fields is not None: - for metadata in metadata_fields.findall(".//Metadata[@used_in]"): - metadata_fields.remove(metadata) - else: - # Create MetadataFields if it doesn't exist - metadata_fields = ET.SubElement(elem, "MetadataFields") - - # Add new used_in Metadata - if behavior_id in all_objective_usages: - usages = all_objective_usages[behavior_id] - # Serialize usages to JSON string - used_in_json = json.dumps(usages, ensure_ascii=False) - - # Add Metadata element with used_in attribute - used_in_metadata = ET.SubElement(metadata_fields, "Metadata") - used_in_metadata.set("used_in", used_in_json) - - behaviors_with_usage += 1 - - print( - f"Added usage information to {behaviors_with_usage} behaviors", file=sys.stderr - ) - - # Write enhanced XML - output_file = output_path or xml_path - print(f"Writing enhanced XML to {output_file}...", file=sys.stderr) - - # Pretty print XML (indent) - ET.indent(tree, space=" ") - tree.write(output_file, encoding="utf-8", xml_declaration=True) - - print(f"Done! Enhanced {len(behavior_elements)} behaviors.", file=sys.stderr) - print(f"Total behaviors with usage: {behaviors_with_usage}", file=sys.stderr) - - -def main(): - parser = argparse.ArgumentParser( - description="Enhance behaviors.xml with usage information from Objectives" - ) - parser.add_argument( - "--xml", - type=str, - required=True, - help="Path to behaviors.xml file", - ) - parser.add_argument( - "--workspace", - type=str, - default=os.environ.get("USER_WS", "."), - help="Path to ROS workspace (default: $USER_WS or current directory)", - ) - parser.add_argument( - "--output", - type=str, - help="Output XML file path (default: overwrite input file)", - ) - - args = parser.parse_args() - - xml_path = Path(args.xml).resolve() - if not xml_path.exists(): - print(f"Error: XML file does not exist: {xml_path}", file=sys.stderr) - sys.exit(1) - - workspace_path = Path(args.workspace).resolve() - if not workspace_path.exists(): - print( - f"Error: Workspace path does not exist: {workspace_path}", file=sys.stderr - ) - sys.exit(1) - - output_path = Path(args.output).resolve() if args.output else None - - enhance_behaviors_xml(xml_path, workspace_path, output_path) - - -if __name__ == "__main__": - main() diff --git a/sync_subtrees.sh b/sync_subtrees.sh deleted file mode 100755 index bddaac3c..00000000 --- a/sync_subtrees.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash - -git subtree pull --prefix src/example_behaviors https://github.com/PickNikRobotics/example_behaviors main --squash -git subtree pull --prefix src/lab_sim https://github.com/PickNikRobotics/lab_sim main --squash -git subtree pull --prefix src/moveit_pro_ur_configs https://github.com/PickNikRobotics/moveit_pro_ur_configs main --squash -git subtree pull --prefix src/moveit_pro_kinova_configs https://github.com/PickNikRobotics/moveit_pro_kinova_configs main --squash -git subtree pull --prefix src/moveit_pro_mobile_manipulation https://github.com/PickNikRobotics/moveit_pro_mobile_manipulation main --squash -git subtree pull --prefix src/fanuc_sim https://github.com/PickNikRobotics/fanuc_sim main --squash -git subtree pull --prefix src/external_dependencies/ridgeback https://github.com/sjahr/ridgeback ros2 --squash -git subtree pull --prefix src/external_dependencies/ros2_robotiq_gripper https://github.com/PickNikRobotics/ros2_robotiq_gripper humble --squash -git subtree pull --prefix src/external_dependencies/serial https://github.com/tylerjw/serial.git ros2 --squash -git submodule update --recursive --init