Pi0.5 SO-101 training stack: configs, adapters, modular entry points - #168
Open
ChepuriNatraj wants to merge 5 commits into
Open
Pi0.5 SO-101 training stack: configs, adapters, modular entry points#168ChepuriNatraj wants to merge 5 commits into
ChepuriNatraj wants to merge 5 commits into
Conversation
Add NOTICE, THIRD_PARTY.md and docs/upstream.md documenting the Apache-2.0 derivation from LeIsaac (base commit 24d3bcd, v0.4.0), all third-party licenses (IsaacLab BSD-3, OpenPI/LeRobot/GR00T Apache-2.0, msgpack-numpy BSD-3), and the key finding that upstream already ships a generic Policy ABC and an OpenPI websocket client -- so Pi0.5 integration is a re-target, not a from-scratch abstraction build.
…dapters
- policies/pi05/config.py: repository-owned pi05_so101 TrainConfig mirroring
OpenPI's TrainConfig schema (verified against local openpi commit 15a9616).
Uses action_dim=6 (5 arm + gripper) and asset_id='so101' -- NOT borrowed
from DROID(32)/Libero(7)/Aloha(7/14) to avoid silent incompatibility.
to_openpi_config() lazily translates to a real OpenPI TrainConfig.
- policies/pi05/adapters.py: deterministic SO-101<->Pi0.5 translation boundary
wrapping leisaac.utils.robot_utils joint remapping (radians<->degrees,
USD-limit<->motor-limit). Produces OpenPI obs {images/<cam> uint8 224x224,
state float64[6], prompt} and consumes action chunks (chunk,6) -> SO-101
radians tensor shape (chunk,1,6).
Tests: tests/test_pi05_adapters.py (config + adapter shape/dtype/roundtrip).
Sim-dependent tests skip cleanly when omni/leisaac unavailable.
…ask interfaces Establishes the model-independent architecture boundaries: - robots/so101: SO-101 spec (6-DoF joint order, limits, rest pose, cameras) - environments: canonical Observation + RobotAction dataclasses with validation - policies/base: generic Policy ABC + PolicyWrapper (sim/eval depend ONLY on this) - tasks: framework-agnostic Task interface + TaskRegistry - policies/pi05/policy: Pi05Policy wrapper over the obs/action adapters (server seam clearly marked, no imaginary model calls) - data_collection / dataset / training / inference / evaluation: layer seams that KEEP existing LeIsaac infrastructure and document the integration points tests/test_architecture.py: 10 architecture-level tests (no Isaac Sim needed), all passing. Runtime-blocked pieces are marked, not faked.
… skeletons
Seven runnable entry points that fail with a clear, actionable message when a
runtime dependency is missing (Isaac Sim / OpenPI / LeRobot) instead of faking
success:
- scripts/simulation/launch_so101.py
- scripts/data_collection/collect_demos.py
- scripts/dataset/{convert_dataset,validate_dataset}.py
- scripts/training/train_pi05.py (builds repo config; blocks on OpenPI)
- scripts/inference/run_pi05_inference.py
- scripts/evaluation/evaluate_policy.py
Each documents which underlying LeIsaac component it reuses.
- README.md: full 21-section deliverable (objective, pipeline, architecture, structure, SO-101 twin, sim, tasks, obs/action flow, data collection, dataset, Pi0.5 integration, training, inference, evaluation, config, run commands with status legend, dev status table, known blockers, upstream attribution). - docs/architecture.md: layered boundaries + Mermaid data-flow + contract rules. - docs/pipeline.md: 13-stage lifecycle (input/process/output/module/status). - docs/roadmap.md: 10-phase DONE/IN-PROGRESS/BLOCKED/TODO tracker. - tests: skip leisaac-bound adapter test when omni unavailable (12 pass, 4 skip).
There was a problem hiding this comment.
Pull request overview
Adds a Pi0.5 (OpenPI) training/inference/evaluation “stack” for SO-101 by introducing canonical Observation/Action/Policy/Task contracts, Pi0.5-specific adapters/config, and modular entry-point skeleton scripts, along with extensive repository documentation and attribution files.
Changes:
- Introduces canonical, model-independent interfaces/contracts for tasks, policies, observations, and actions (enabling backend-swappable policies).
- Adds SO-101-specific Pi0.5 training configuration plus SO-101↔Pi0.5 observation/action adapters and a Pi05Policy wrapper seam.
- Adds modular entry-point skeletons for training/simulation/inference/evaluation/dataset tooling and expands docs/attribution to reflect the new architecture.
Reviewed changes
Copilot reviewed 29 out of 30 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| training/init.py | Defines training job metadata + checkpoint path convention for Pi0.5 training. |
| THIRD_PARTY.md | Adds third-party attribution and license provenance notes. |
| tests/test_pi05_adapters.py | Adds unit tests for Pi0.5 adapters/config contracts. |
| tests/test_architecture.py | Adds architecture-level tests for the new core contracts/registry. |
| tasks/init.py | Introduces Task ABC, TaskContext, and a global TaskRegistry. |
| scripts/training/train_pi05.py | Adds a Pi0.5 training entry-point skeleton that builds repo config and maps to OpenPI config. |
| scripts/simulation/launch_so101.py | Adds an IsaacLab SO-101 simulation launch skeleton with blocker reporting. |
| scripts/inference/run_pi05_inference.py | Adds a closed-loop inference entry-point skeleton for Pi0.5. |
| scripts/evaluation/evaluate_policy.py | Adds an evaluation entry-point skeleton wiring to the evaluation engine. |
| scripts/dataset/validate_dataset.py | Adds a LeRobot dataset validation entry-point skeleton. |
| scripts/dataset/convert_dataset.py | Adds a dataset conversion entry-point skeleton delegating to LeIsaac converter. |
| scripts/data_collection/collect_demos.py | Adds a demo-collection entry-point skeleton for teleop/scripted collection. |
| robots/so101/init.py | Introduces a single SO-101 robot spec boundary (joints/limits/rest pose/cameras) and upstream cfg loader. |
| README.md | Replaces upstream README with a full architecture/pipeline overview and status/blockers. |
| policies/pi05/policy.py | Adds Pi05Policy wrapper integrating canonical Observation/Action with a Pi0.5 backend seam. |
| policies/pi05/config.py | Adds repo-owned SO-101 Pi0.5 training config and mapping to OpenPI TrainConfig. |
| policies/pi05/adapters.py | Adds deterministic observation/action translation layer between canonical sim data and OpenPI Pi0.5 format. |
| policies/base/init.py | Adds a generic Policy ABC and PolicyWrapper base for backend adapters. |
| NOTICE | Adds Apache-2.0 derivative notice and provenance. |
| inference/init.py | Adds a model-independent closed-loop inference runner. |
| evaluation/init.py | Adds a model-independent evaluation engine and report/metric serialization. |
| environments/observation.py | Adds canonical Observation dataclass + validation. |
| environments/action.py | Adds canonical RobotAction dataclass + validation + vectorization. |
| docs/upstream.md | Documents upstream provenance and the rationale for architectural divergence. |
| docs/roadmap.md | Adds a phased implementation roadmap aligned to the new architecture. |
| docs/pipeline.md | Adds an end-to-end pipeline document with stage responsibilities and status. |
| docs/architecture.md | Adds a deep-dive architecture document describing boundaries and contracts. |
| dataset/init.py | Adds dataset-layer schema/ref seams and documents SO-101 mapping. |
| data_collection/init.py | Adds control-source and episode-recorder interfaces for data collection plumbing. |
| .gitignore | Stops ignoring tests/ sources; ignores only caches. |
Suppressed comments (1)
evaluation/init.py:161
task.is_success(None)/task.is_failure(None)are called withNone, which contradicts theTaskinterface and will prevent real tasks from using the env state to determine termination. This should use the actual environment handle (see also the earliertask.reset(None)call).
if task.is_success(None):
done = True
termination = "success"
elif task.is_failure(None):
done = True
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+34
to
+47
| from dataclasses import dataclass | ||
| from typing import Any | ||
|
|
||
|
|
||
| @dataclass | ||
| class TrainingJob: | ||
| """Description of a single pi0.5 fine-tuning job.""" | ||
|
|
||
| config_name: str = "pi05_so101" | ||
| exp_name: str = "lift_cube_v1" | ||
| dataset_repo_id: str = "your_hf_username/so101_pi05_demos" | ||
| # Path to a checkpoint to resume from (optional). | ||
| resume_from: str | None = None | ||
| extra: dict[str, Any] = dict() |
Comment on lines
+99
to
+103
| js = np.asarray(joint_state).reshape(-1, 6) | ||
| # IsaacLab radians -> LeRobot normalized [0,100] (what Pi0.5 expects). | ||
| lerobot_state = robot_utils.convert_leisaac_action_to_lerobot(js) | ||
| obs["state"] = lerobot_state.astype(np.float64) | ||
| obs["prompt"] = task_description |
Comment on lines
+50
to
+51
| # pi05 default action chunk length for single-arm; matches pi05_libero (10). | ||
| action_horizon: int = 16 |
Comment on lines
+67
to
+70
| horizon = execution_horizon or action.action_horizon | ||
| # A real env consumes the chunk; we expose the horizon decision here. | ||
| step(action) | ||
| steps += horizon |
Comment on lines
+54
to
+56
| for key, img in self.images.items(): | ||
| if img.ndim != 3 or img.shape[-1] != 3: | ||
| raise ValueError(f"image {key!r} must be HWC uint8, got {img.shape}") |
Comment on lines
+138
to
+140
| reset_env() | ||
| task.reset(None) # task may reconfigure object/target poses | ||
| instruction = task.get_instruction() |
|
|
||
| def main() -> int: | ||
| parser = argparse.ArgumentParser(description="Fine-tune pi0.5 on SO-101 data") | ||
| parser.add_argument("--config", default="pi05_so101") |
| chunk = np.random.rand(16, 6).astype(np.float64) | ||
| out = pi05_action_to_so101(chunk) | ||
| assert out.shape == (16, 1, 6) | ||
| assert out.dtype == np.float64 or out.dtype == np.float32 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Pi0.5 SO-101 training config, observation/action adapters, modular training/inference/eval/data-collection entry-point skeletons, canonical observation/action contracts, and repo docs. Committed on top of upstream main (24d3bcd).