Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ jobs:
python-version: ${{ matrix.python }}
cache: pip
- run: python -m pip install -r requirements-dev.txt
- run: python -m pip install -e '.[chatgpt]'
- run: python -m pytest -q
- uses: actions/setup-node@v4
with:
node-version: "22"
- run: node --test tests/workspace_manual.test.mjs
- run: node --test tests/chatgpt_plan_ui.test.mjs
render:
runs-on: ubuntu-latest
env:
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ Choose one route below. Stop the existing server before running a different laun

| Model route | How to connect | Compatibility boundary |
| --- | --- | --- |
| ChatGPT plan direct | Select the direct route and authorize ManiLoop in your browser | Eligible account models through Responses, without Codex CLI; no server token hard cap in the current preview |
| GPT through Codex | Official Codex CLI login; select the Codex source | Uses models available to that CLI/account; ManiLoop does not extract login tokens |
| Hosted or self-hosted vision model | Enter the service base URL, model ID, and API key, or import provider TOML | Robot control requires **Responses API + image input + JSON Schema output** |
| Local LeRobot checkpoints | Select a supported model loaded by the isolated inference worker | Three reviewed LIBERO checkpoint presets; not arbitrary model-folder import |
Expand All @@ -144,6 +145,10 @@ Choose one route below. Stop the existing server before running a different laun

### GPT / compatible model APIs

To use your ChatGPT plan directly, install `python -m pip install -e '.[chatgpt]'`, start the normal local demo, and choose **ChatGPT 套餐直连 · 独立授权** in the model panel. Click **Continue with ChatGPT**, authorize ManiLoop, refresh the account's model list, and run the text/image/action-format diagnostics. This route uses ManiLoop's own prompt and control loop without launching Codex CLI. Account eligibility and model access are verified by those diagnostics. See the [bilingual setup and comparison guide](docs/CHATGPT_PLAN.md).

The existing Codex CLI route remains available as a separate harness configuration:

Install a current official [Codex CLI](https://github.com/openai/codex), then:

```bash
Expand Down
5 changes: 5 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ python -m maniloop demo --backend robocasa --task OpenDrawer \

| 模型接入方式 | 如何连接 | 兼容边界 |
| --- | --- | --- |
| ChatGPT 套餐直连 | 选择直连来源,在浏览器中为 ManiLoop 独立授权 | 使用账号可用的 Responses 模型,不经过 Codex CLI;当前预览无服务端 token 硬上限 |
| 通过 Codex 使用 GPT | 在官方 Codex CLI 登录,选择 Codex 来源 | 使用该 CLI/账户可用的模型;ManiLoop 不提取登录 token |
| 云端或自托管视觉模型 | 填写服务基础地址、模型 ID、API Key,或导入供应商 TOML | 机器人控制需要 **Responses API、图像输入和 JSON Schema 输出** |
| 本地 LeRobot 检查点 | 选择由独立推理进程加载的支持模型 | 当前提供三个已核对的 LIBERO 检查点预设,不是任意模型文件夹导入 |
Expand All @@ -144,6 +145,10 @@ python -m maniloop demo --backend robocasa --task OpenDrawer \

### GPT 与兼容模型 API

使用 ChatGPT 套餐直连时,先运行 `python -m pip install -e '.[chatgpt]'`,正常启动本地 demo,在模型面板选择 **ChatGPT 套餐直连 · 独立授权**。点击 **Continue with ChatGPT** 授权 ManiLoop,刷新账号模型列表,再运行文字、图像和动作格式诊断。该方式使用 ManiLoop 自己的提示和闭环,不启动 Codex CLI;账号资格与模型访问以诊断结果为准。详见[双语接入与比较指南](docs/CHATGPT_PLAN.md)。

现有 Codex CLI 入口继续保留,作为单独标记的 harness 配置:

安装近期官方 [Codex CLI](https://github.com/openai/codex),然后运行:

```bash
Expand Down
103 changes: 103 additions & 0 deletions docs/CHATGPT_PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# ChatGPT plan direct connection / ChatGPT 套餐直连

[English](#english) · [中文](#中文)

## English

ManiLoop can obtain its own ChatGPT authorization and call the public Responses API directly. This route uses eligible ChatGPT plan access and keeps the robot loop, prompts, observation history, and action validation in ManiLoop. It does not launch Codex CLI or load its harness. The existing **Codex CLI** and **API key** routes remain available.

Install the authentication extra in the main Python environment:

```bash
python -m pip install -e '.[chatgpt]'
python -m maniloop demo --port 8767
```

1. Open the local workbench and choose **模型运行 → 模型连接与认证 → ChatGPT 套餐直连 · 独立授权**.
2. Click **Continue with ChatGPT**. In the browser, authorize **ManiLoop** to use your ChatGPT plan. This is separate from your Codex login; an existing CLI login does not authorize this connection.
3. Return to the workbench. It shows the connected account and whether plan usage is enabled. Use the account selector to switch saved accounts, **重新授权此账号** to renew authorization, or **退出此账号** to disconnect locally and attempt session revocation. Account changes are locked while a run is active.
4. Click **刷新模型列表**, then choose a model from this account's catalog. GPT-6 Astra access depends on the selected account and the current service. A catalog entry alone does not establish successful inference.
5. Run **文本诊断**, **图像诊断**, and **动作格式诊断** in order. Each diagnostic makes a model request; the action-format diagnostic does not execute a robot action. After these pass, use **单步执行** before starting a complete episode.

Click **取消授权** to cancel a sign-in attempt. Closing the browser window alone does not cancel it: return to the workbench and cancel explicitly before starting again, or wait for the attempt to expire. If registration issued a client ID but code exchange failed, use **继续上次注册** to retry that registration without creating a new one. If the browser blocks the popup, allow popups for the local workbench and click again. A failed sign-in does not silently replace another saved account. Use ChatGPT **Settings → Usage** to manage app access and usage. If remote revocation cannot be confirmed during logout, disconnect the app there as well.

### Record one LIBERO episode

Install the [independent LIBERO environment](LIBERO.md), finish the workbench authorization and diagnostics, and select the account you want to use. The CLI reuses that active ManiLoop account without opening a login window. If you set `MANILOOP_CHATGPT_AUTH_DIR` for the workbench, use the same value in this terminal.

```bash
python -m maniloop benchmark \
--backend libero --libero-suite libero_spatial --libero-task-id 0 --init-state-id 0 \
--seed 0 --agent llm_cloud --chatgpt-plan --model gpt-6-astra \
--timing controlled --context-mode paired --reasoning-effort medium \
--llm-control tcp_target_servo_v2 --observation-profile llm_rgb512 \
--max-calls 0 --max-wall-seconds 3600 --max-sim-seconds 120 \
--request-timeout-seconds 120 --record-episode \
--output runs/chatgpt-plan-recorded
```

This makes real model requests and records every LIBERO control step. `--max-calls 0` removes the decision-count cap; environment termination and wall/simulation time limits remain. Use a model verified for the selected account. `--chatgpt-plan`, `--codex-login`, and `--provider-config` are mutually exclusive; errors never switch billing paths. Every attempt is saved separately, and success is determined by the independent evaluator. See [episode recording and MP4 export](EPISODE_RECORDING.md) for integrity checks and export commands.

### Comparison and budget behavior

The direct route shares ManiLoop's sensor-only input construction, image bytes, history, action schema, and local action checks with the Responses API policy. Its application-side input manifest records the request protocol and content hashes; it does not claim visibility into server-side processing. The Codex CLI route remains a separately labeled harness configuration.

The ChatGPT plan preview requires streamed Responses requests with `store: false` and explicit input history. It does **not** accept a server-side `max_output_tokens` parameter. Set the reasoning effort and local request deadline explicitly, and distinguish these from actual reasoning-token usage. A client text-length limit or timeout is not a server token budget. Strict equal-token-budget comparisons therefore require a different supported connection. An incomplete, failed, interrupted, or invalid response never authorizes a robot action. There is no automatic switch to another model, provider, or paid API key when access fails.

### Local credentials and validation status

ManiLoop stores its own authorization records in protected local storage, separate from Codex. It never needs you to copy a token into the workbench. OAuth credentials are not sent to a custom provider, added to experiment logs, or written into browser storage. The public GitHub Pages site is a static replay and cannot run this local sign-in flow.

This feature's offline protocol and UI checks do not establish access for your account. The first successful text/image/structured-action diagnostics are the acceptance check for your selected account and model; they do not demonstrate task success or a benchmark success rate.

Official references, checked 2026-10-28: [registration and sign-in](https://developers.openai.com/siwc/token-sharing-open-source/sign-in), [models and inference](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference), [preview limitations](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations).

## 中文

ManiLoop 可以独立获得 ChatGPT 授权,直接请求官方 Responses API,使用符合条件的 ChatGPT 套餐。机器人闭环、提示、观测历史和动作校验均由 ManiLoop 管理,不启动 Codex CLI,也不加载其 harness。现有 **Codex CLI** 和 **API Key** 接入继续保留。

在主 Python 环境中安装认证扩展:

```bash
python -m pip install -e '.[chatgpt]'
python -m maniloop demo --port 8767
```

1. 打开本地工作台,选择 **模型运行 → 模型连接与认证 → ChatGPT 套餐直连 · 独立授权**。
2. 点击 **Continue with ChatGPT**,在浏览器中授权 **ManiLoop** 使用你的 ChatGPT 套餐。它与 Codex 登录独立,已登录 CLI 不代表此连接已授权。
3. 返回工作台,确认账号和套餐授权状态。可选择已保存的账号、点击 **重新授权此账号**,或 **退出此账号**。退出会清除本地凭据并尝试撤销远端会话;实验运行中不能切换账号。
4. 点击 **刷新模型列表** 并选择模型。GPT-6 Astra 是否可用取决于此账号与服务当前权限;出现在目录中不等于推理已通过。
5. 依次运行 **文本诊断、图像诊断、动作格式诊断**。每项都会请求模型,动作格式诊断不会移动机器人。通过后先用 **单步执行** 检查,再开始完整 episode。

点击 **取消授权** 可取消本次登录。仅关闭浏览器窗口不会取消授权;请回到工作台明确取消后再重试,或等待本次授权过期。若已取得注册 ID 但授权码交换失败,可点击 **继续上次注册** 复用该注册重试,无需创建新注册。如果弹窗被阻止,请允许本地工作台打开弹窗后重试。登录失败不会静默替换另一个已保存账号。可在 ChatGPT **设置 → 使用情况** 管理应用授权与用量;若退出时未确认远端撤销成功,也可在那里断开应用。

### 录制一个 LIBERO 完整回合

先按 [LIBERO 安装说明](LIBERO.md)准备独立环境,在工作台完成授权、诊断并选择账号。CLI 复用 ManiLoop 当前账号,不会打开登录窗口。如果工作台设置了 `MANILOOP_CHATGPT_AUTH_DIR`,请在此终端使用相同值。

```bash
python -m maniloop benchmark \
--backend libero --libero-suite libero_spatial --libero-task-id 0 --init-state-id 0 \
--seed 0 --agent llm_cloud --chatgpt-plan --model gpt-6-astra \
--timing controlled --context-mode paired --reasoning-effort medium \
--llm-control tcp_target_servo_v2 --observation-profile llm_rgb512 \
--max-calls 0 --max-wall-seconds 3600 --max-sim-seconds 120 \
--request-timeout-seconds 120 --record-episode \
--output runs/chatgpt-plan-recorded
```

这会发起真实模型请求,并录制每个 LIBERO 原生控制步。`--max-calls 0` 取消决策次数上限,环境终止及墙钟、仿真时限仍有效。模型必须已在所选账号中验证可用。`--chatgpt-plan`、`--codex-login`、`--provider-config` 互斥,出错不会切换计费路径。每次尝试独立保存,成功与否由独立评分器判定。完整性检查和视频导出见[完整 episode 录制](EPISODE_RECORDING.md)。

### 公平比较与预算

套餐直连与 Responses API 策略共用传感输入构造、图片字节、历史、动作 Schema 和本地校验。应用侧请求记录包含协议版本与输入摘要,但不代表能够看到服务端内部处理。Codex CLI 仍作为独立的 harness 配置标记。

当前套餐预览要求流式请求、`store: false` 和显式发送历史,**不支持服务端 `max_output_tokens` 上限**。请明确设置推理强度与本地请求时限,并将它们和实际推理 token 消耗区分。客户端文本长度限制或超时不是服务端 token 预算;要求严格 token 上限的对照应使用支持该能力的其他连接。断流、失败、中断或格式无效的响应都不能触发动作。权限或额度不足时,不会自动改用其他模型、供应商或付费 API Key。

### 凭据与验证范围

认证记录保存在本机受保护位置,与 Codex 分开,不需要手动粘贴 token。OAuth 凭据不会发送给自定义供应商,也不会进入实验日志或浏览器存储。GitHub Pages 是静态回放页面,不能运行此本地认证流程。

离线协议和界面检查不代表你的账号已通过接入。首次成功的文字、图像、结构化动作诊断用于验收所选账号和模型,不代表任务成功或基准成功率。

官方依据,核对日期 2026-10-28:[注册与登录](https://developers.openai.com/siwc/token-sharing-open-source/sign-in)、[模型与推理](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference)、[当前预览限制](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations)。
2 changes: 2 additions & 0 deletions docs/LIBERO.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ ManiLoop 保留 `maniloop_libero_rgb_proprio_v1` 作为基线,并增加 LLM
- 原生 VLA 接口是 `Action(kind="osc_pose", values=(七个归一化值), frame="world")`。前六维为世界坐标系 TCP 平移 / 旋转增量,最后一维 -1 张开、+1 闭合。每维范围 [-1,1],采样间隔必须为 0.05 秒(20 Hz)。动作由官方控制器执行;已接入真实 SmolVLA / ACT / Diffusion 学习策略,使用独立推理环境;见 [本地策略说明](VLA.md)。
- 原生单步基线 `osc_step` 使用米 / 弧度动作;适配器显式转换为原生 OSC 输入。平移范数最多 0.05 m,旋转范数最多 0.5 rad,每次 move 执行一个控制步。并不保证末端立即到达目标,模型须重新观察。夹爪只接受 0 / 1,执行 10 个控制步;wait 为一个零机械臂增量控制步。
- 网页与云端批量实验默认使用 `llm_rgb512` 的 512×512 JPEG 和 `tcp_target_servo_v2`。米/弧度增量转换为固定目标位姿,本地以 20 Hz 原生 OSC 反馈追踪,最多 20 步;位置容差 2 mm、姿态容差 0.02 rad,低于速度门槛并稳定两步才报告 reached。夹爪命令保持 TCP 目标至少 10 步,并等待位姿与开度变化速度稳定;张开时还要求开度达到 90%。completed 表示夹爪稳定,不代表抓到物体;超过 20 步仍未满足条件则报告 timed_out。位姿和夹爪均只使用本体反馈。
- LIBERO 另提供 `tcp_target_servo_v3`,可在网页动作执行选项中选择,或通过 `--llm-control tcp_target_servo_v3` 启用。它将 **move 的规划目标增量**放宽为平移向量范数最多 2.5 m、旋转向量范数最多 π rad(180°),最多追踪 200 个原生控制步(10 秒);到达目标或环境终止时提前停止。夹爪命令仍最多 20 步(1 秒),其他收敛判据保持 v2 不变。**原生 OSC 的 20 Hz、每步 0.05 m / 0.5 rad 控制尺度和 2 ms 物理步长均未改变**;大目标由本地反馈分步执行,不是单步瞬移。手动目标跟踪时限独立设置,仍可覆盖到最多 10 秒。切换到非 LIBERO 后端时网页禁用 v3 并回落 v2;默认值与原有演示预设继续使用 v2。
- v3 的 2.5 m 仅是异常输入保护,**不是可达工作空间或建议步长**。固定 Panda 从 joint1 到 grip_site 的平移链长上界为 `L = 0.316 + 0.0825 + sqrt(0.0825² + 0.384²) + 0.088 + 0.1065 + 0.097 ≈ 1.082762 m`,任意两构型末端间距不超过 `2L ≈ 2.165525 m`,据此向上取 2.5 m。π rad 覆盖任意目标姿态的最短旋转。10 秒沿用手动目标跟踪的最大工程预算;关节范围、碰撞或接触仍可能阻止到达。v3 与 v2 是不同控制协议,比较时必须分组记录。
- `--llm-control osc_step --observation-profile debug_rgb128` 可复现原始基线;批量实验支持 `--request-timeout-seconds 120 --reasoning-effort low`,套件相应字段为 `llm_control`、`observation_profile`、`request_timeout_seconds`、`reasoning_effort`。模式、图像、控制器容差与实际请求参数分别进入 manifest 和比较组。
- 保留官方控制器意味着不会套用自建任务的 IK、碰撞预检或运动规划。两类环境的动作能力不同,分别形成比较组。
- 模型推理时默认冻结物理;实时模式单独比较。网页闲置时不会消耗 LIBERO 控制步预算。成功或预算终止后不再推进;没有自建任务的额外 1.2 秒静置窗口。
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ dependencies = ["mujoco==3.12.0", "numpy==2.5.3", "Pillow==12.3.0", "openai==3.8

[project.optional-dependencies]
dev = ["pytest==9.1.1"]
chatgpt = ["PyJWT[crypto]>=2.10,<3", "httpx>=0.28,<1"]

[project.scripts]
maniloop = "maniloop.cli:main"
Expand Down
4 changes: 4 additions & 0 deletions src/maniloop/agents/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ def last_latency(self):
def last_usage(self):
return self.provider.last_usage

@property
def last_request_manifest(self):
return getattr(self.provider, "last_request_manifest", None)

def reset(self, seed=0):
# Requests have no persistent remote conversation; episode history is supplied explicitly.
pass
Expand Down
Loading
Loading