feat: add OMX-AI robot arm support alongside SO-101 - #81
Conversation
[260711] fix to recognize camera at chrome and fix to 3d model limit …
…uration management and status reporting
…w fixes Abstract the hardcoded SO-101 wiring so the web UI can drive OMX-AI arms: - make_device_config/make_device in utils/devices.py dispatch config and device classes by robot_type (so101 -> SO101*, omx -> Omx*) - teleoperate/record/calibrate build devices through the new helpers; rollout resolves --robot.type dynamically - calibration paths, robot records, and all frontend start requests (teleop, recording, inference, calibration) carry robot_type - frontend calibration targets gain OMX-AI joint range maps Review fixes on top of the feature: - calibrate: pick the POSITION operating-mode value per bus protocol; the Feetech value (0) means CURRENT mode on Dynamixel (OMX) motors - server: delete_calibration_config resolves paths via get_calibration_path so OMX calibration files can be deleted - drop the test-mock shim from make_device; tests now patch make_device - lint cleanup (unused imports, import sorting, whitespace) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onfiguration UI components
…viewer OMX-AI arms self-calibrate on first connect and never went through LeLab's SO-101-only calibration wizard, so their robot records could never satisfy the "calibration file exists" check that gated the Teleoperation button, and `lerobot[dynamixel]` was missing from the pinned extras so OMX motors couldn't even connect. teleoperate.py also pushed an empty calibration to the bus instead of letting OMX self-calibrate. Also wires the Teleoperation page's 3D viewer to show the actual OMX-AI model (ROBOTIS open_manipulator_description URDF/meshes, Apache-2.0) instead of always rendering the SO-101 arm, with a matching robot-type-aware joint-name/unit mapping for the live joint broadcast. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fix: make OMX-AI robots teleoperable without calibration, add OMX 3D viewer
There was a problem hiding this comment.
Pull request overview
Adds OMX-AI support alongside SO-101 across hardware configuration, calibration, recording, inference, and visualization.
Changes:
- Introduces robot-type-aware backend device creation and calibration paths.
- Adds frontend model selection, persistence, and OMX 3D visualization.
- Expands tests, dependencies, documentation, and generated frontend assets.
Reviewed changes
Copilot reviewed 38 out of 57 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_teleoperate.py |
Updates device mocking. |
tests/test_rollout.py |
Tests shared error hints. |
tests/test_record.py |
Tests recording error status. |
tests/test_calibrate.py |
Tests protocol-specific position mode. |
study/install.md |
Adds Windows installation guide. |
study/day0_overview.md |
Adds Korean project overview. |
pyproject.toml |
Enables Dynamixel support. |
lelab/utils/devices.py |
Adds dynamic device factories and hints. |
lelab/utils/config.py |
Adds model-specific calibration persistence. |
lelab/teleoperate.py |
Parameterizes devices and visualization. |
lelab/server.py |
Parameterizes calibration routes and Linux camera names. |
lelab/scripts/lelab.py |
Extends Windows startup timeout. |
lelab/rollout.py |
Parameterizes inference robot type. |
lelab/record.py |
Parameterizes recording devices and errors. |
lelab/calibrate.py |
Adds OMX device and Dynamixel handling. |
frontend/src/pages/Teleoperation.tsx |
Selects the active robot model. |
frontend/src/pages/Recording.tsx |
Displays recording failures. |
frontend/src/pages/Landing.tsx |
Sends recording robot type. |
frontend/src/pages/Calibration.tsx |
Adds robot model configuration. |
frontend/src/lib/inferenceApi.ts |
Adds inference robot type. |
frontend/src/lib/defaultUrdfModels.ts |
Maps built-in URDF models. |
frontend/src/lib/calibrationTargets.ts |
Adds OMX calibration targets. |
frontend/src/hooks/useRobots.ts |
Persists robot model selection. |
frontend/src/contexts/UrdfContext.tsx |
Tracks the default URDF type. |
frontend/src/components/UrdfViewer.tsx |
Loads model-specific URDF assets. |
frontend/src/components/landing/RobotSelector.tsx |
Adds model chips at creation. |
frontend/src/components/landing/RobotConfigManager.tsx |
Sends teleoperation robot type. |
frontend/src/components/landing/InferenceModal.tsx |
Sends inference robot type. |
frontend/public/omx-urdf/urdf/omx_f.urdf |
Adds the OMX follower model. |
frontend/public/omx-urdf/meshes/omx_f/follower_01_base.stl |
Adds OMX base mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_02_base_tilt_Revised.stl |
Adds OMX base-tilt mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_03_middle_verticle.stl |
Adds OMX middle-link mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_04_middle_horizontal.stl |
Adds OMX horizontal-link mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_05_tip.stl |
Adds OMX tip mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_06_pan_Revised.stl |
Adds OMX wrist-pan mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_07_gripper_motorized.stl |
Adds OMX motorized-gripper mesh. |
frontend/public/omx-urdf/meshes/omx_f/follower_08_gripper_gear.stl |
Adds OMX gripper-gear mesh. |
frontend/public/omx-urdf/LICENSE |
Adds asset licensing. |
frontend/package-lock.json |
Refreshes lockfile metadata. |
frontend/dist/omx-urdf/urdf/omx_f.urdf |
Ships the built OMX URDF. |
frontend/dist/omx-urdf/LICENSE |
Ships the asset license. |
frontend/dist/index.html |
References rebuilt assets. |
frontend/dist/assets/index-DDnZ8aJk.css |
Adds rebuilt stylesheet. |
frontend/dist/assets/index-Da-rRmcX.css |
Removes the previous stylesheet. |
Files not reviewed (1)
- frontend/package-lock.json: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Robert Choi <robertchoimail@gmail.com>
nicolas-rabault
left a comment
nicolas-rabault
left a comment
There was a problem hiding this comment.
Thanks Robert, this is a very good write-up and more arms in LeLab is something we need.
One ask: split it in two, a robot registry with SO-101 as the only entry and no behaviour change, then OMX-AI as one entry on top. The call sites are listed in the review, so PR 1 is already scoped.
Five blocking notes inline, all in the OMX half. Two of the bot's comments are wrong, I have said which.
I have SO-101 but no OMX, so I will cover the regression side and will need you to show the OMX half working. What I need is in the review.
| """Create a LeRobot device config object dynamically based on robot type.""" | ||
| model = robot_type.lower() | ||
|
|
||
| if "so" in model: |
There was a problem hiding this comment.
This is the shape the split is about, so treating it as the anchor for that ask. Two concrete things a registry fixes that this cannot: "so" in model is substring matching and it is tested before omx, so any future arm whose name contains those two letters silently resolves to SO-101; and make_device_config and make_device carry duplicate branch trees that have to be kept in sync by hand forever.
One entry keyed on an exact robot_type string, holding the config class and the device class together, removes both and gives you a real lookup failure on an unknown type instead of a silent fallback.
| Feetech and Dynamixel disagree on the value (0 vs 3); writing the Feetech | ||
| value to a Dynamixel bus (OMX) would put the motors in CURRENT mode. | ||
| """ | ||
| if "dynamixel" in type(bus).__module__: |
There was a problem hiding this comment.
Blocker, and it breaks the arm this PR is adding. Picking the right OperatingMode value here is necessary but not sufficient: _complete_calibration further down still writes drive_mode=0 for every motor. I checked OmxLeader.calibrate() at the pinned v0.6.0 and it sets drive_mode=1 and homing_offset=100 on the gripper.
So running the newly exposed OMX wizard saves a leader calibration that inverts the gripper against what lerobot expects. Either keep OMX off the wizard entirely, or make the calibration recipe part of the arm entry rather than a constant in _complete_calibration.
| elif "omx" in model: | ||
| folder = f"omx_{side}" | ||
| else: | ||
| folder = f"so_{side}" |
There was a problem hiding this comment.
Blocker: this else silently resolves an unknown robot_type to the SO-101 directory, and DELETE /calibration-configs/{device_type}/{config_name} reaches this function. A typo in robot_type on that request deletes a real SO-101 calibration while the caller believes it asked about another arm. Please raise on an unknown type instead.
Separately, the folder names themselves are right: I checked against lerobot v0.6.0 and OmxLeader.name is omx_leader, OmxFollower.name is omx_follower, so lerobot will find the files where you put them.
| if "so" in model: | ||
| robot_type_arg = "--robot.type=so101_follower" | ||
| elif "omx" in model: | ||
| robot_type_arg = "--robot.type=omx_follower" |
There was a problem hiding this comment.
Blocker, same class as the one in get_calibration_path but with worse consequences: the else sends a mistyped robot_type to so101_follower, which means a policy runs on the wrong physical arm. Raise instead.
The omx_follower string itself is correct, I verified OmxFollowerConfig carries @RobotConfig.register_subclass("omx_follower") at v0.6.0.
| @@ -0,0 +1,201 @@ | |||
| Apache License | |||
There was a problem hiding this comment.
Blocker on the redistribution, though the answer is an easy one. The licensing is fine: I traced these to ROBOTIS-GIT/open_manipulator (open_manipulator_description/, omx_f.urdf.xacro), that repo is Apache-2.0, and leLab is Apache-2.0 too, so they are compatible.
What is missing is attribution. This file is the unmodified Apache boilerplate with Copyright [yyyy] [name of copyright owner] still a literal placeholder, and nothing in the branch records where the meshes came from. Apache-2.0 section 4 asks for the original copyright notice to be retained, so we cannot ship them as-is. Please fill in the ROBOTIS copyright line and add the source repo, path, and revision alongside it.
Closes #79
Summary
This PR adds native support for the ROBOTIS OMX-AI robotic arm to LeLab's web GUI, alongside the existing SO-101 leader/follower arms. All four hardcoded areas identified in #79 are now abstracted behind a per-robot
robot_typesetting.Changes
Backend: dynamic robot configuration loading
make_device_config()/make_device()helpers inlelab/utils/devices.pydispatch LeRobot config and device classes byrobot_type(so101→SO101Follower/SO101Leader,omx_ai→OmxFollower/OmxLeader).teleoperate.py,record.py, andcalibrate.pybuild devices through these helpers instead of importing SO-101 classes directly; every request schema gains arobot_typefield (default"so101", fully backward compatible).utils/config.py) resolve per robot type (so_leader/so_followervsomx_leader/omx_follower), matching the directories LeRobot derives from each device'sname.POSITIONoperating-mode value per bus protocol: Feetech uses0, but on Dynamixel (OMX)0means CURRENT mode — writing the Feetech value would have put OMX motors in current-control mode.Rollout parameterization
rollout.pyresolves--robot.typefrom the request (so101_follower/omx_follower) instead of hardcoding it.Frontend
robot_type; it is sent with every teleoperation, recording, inference, and calibration start request.calibrationTargets.tsgains OMX-AI per-joint target ranges, selected by the active robot's type.Testing
0vs Dynamixel3).make_deviceseam, so both SO-101 and OMX paths are testable.ruff checkclean.robot_type: "omx_ai", and SO-101 flows are unchanged.Notes for reviewers
robot_typedefaults to"so101"everywhere, so existing robot records and API clients keep working unchanged.make_device_config/make_device, a calibration-path folder mapping, and (optionally) frontend target ranges.