Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions src/agent_env/drivers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,5 +125,30 @@ cuts a server's instructions at 4,096 characters, so the skill itself isn't in t
(`DO_NOT_TRACK`). `computer-ready` checks its version, that it prints the pinned skill, and the TCC permissions. Name it
as `macos-harness`.

`envs/codex-computer-use/` is OpenAI's Codex Computer Use as a driver, for the same macOS computers. It is the ChatGPT
app's `unified-computer-use` plugin, whose MCP server, `cua_repl`, runs the JavaScript an agent sends against OpenAI's
`cua` API, which reads apps' accessibility trees and acts on them in the background through the plugin's helper app. The
ChatGPT version `driver.py` pins as `VERSION`, OpenAI's versioned download checked by sha256, is unpacked beside the
driver by `install.sh` and never run, so it never updates itself, and its helper app is copied to
`/opt/agentenv/codex-computer-use/`, where `[install.macos]` grants it Accessibility, Post Event and Screen Recording,
as it grants the driver's own Python. The driver starts `cua_repl` as the app would, on the app's own Node with the
environment the app builds and the computer surface alone, one server for each agent session, since the REPL keeps its
bindings between calls; one that stops is started afresh on the next call, and nothing says when a session ends, so one
left unused for 30 minutes, or past the eight kept, is closed, the least recently used first. Agents get the server's
instructions and its `js` and `js_reset` tools, as `mcp.json` pins them, `js` marked as acting on apps where Codex marks
it read-only; `js_add_node_module_dir` and `turn_ended`, which it serves the ChatGPT app, are hidden. Codex asks its
host before each action on an app, and the driver answers: it approves the apps its `allowed_apps` setting names, by
name or bundle identifier, case aside, as the request's metadata or its message gives them, or every app when it names
none, and nothing else, recording the computer's audio included, a request that names no app logged as a warning. Each
call runs under Codex's own sandbox, read-only but for `~/.codex` and the temporary folders and with no network, unless
its `js_sandbox` setting is off; the apps it drives are outside it, and it still lets the JavaScript read files and run
commands, so a bench that wants the GUI alone guards for that. No ChatGPT sign-in is needed. Its `computer_*` tools are
agent-desktop's, from `MacOSDriver`. `computer-ready` checks the executables' sha256, that `cua_repl` gives the
instructions and lists the tools `mcp.json` pins and answers `cua.getState()`, and the TCC permissions. To move it to
another ChatGPT version, set `VERSION`, pin the zip's URL and sha256 in `env.toml` and the executables' in `driver.py`'s
`EXECUTABLES` (a unit test checks that `env.toml`, `install.sh` and `driver.py` agree on the version and the helper's
path), and take `mcp.json` again from what that version's `cua_repl` says at connect and lists. Name it as
`codex-computer-use`.

This bundle has no tasks of its own: other bundles pull its drivers by alias, and `agent-env run drivers` has nothing
to run.
1 change: 1 addition & 0 deletions src/agent_env/drivers/aliases.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ cua-driver = ["cua-driver"]
page = ["page"]
agent-desktop = ["agent-desktop"]
macos-harness = ["macos-harness"]
codex-computer-use = ["codex-computer-use"]
352 changes: 352 additions & 0 deletions src/agent_env/drivers/envs/codex-computer-use/driver.py

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions src/agent_env/drivers/envs/codex-computer-use/env.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
type = "computer_driver"
module = "driver.py:CodexComputerUse"
port = 18765

[install.macos]
python = "requirements-macos.txt"
downloads = [
{ url = "https://persistent.oaistatic.com/codex-app-prod/ChatGPT-darwin-arm64-26.930.51102.zip", sha256 = "7001df293fd478b2a4e2d8b5f4ea3aae43a2e21d168960256b2f31ac4ee43f6f", to = "ChatGPT.zip", arch = "arm64" },
]
run = [{ script = "install.sh", scope = "session" }]
grants = [
{ executable = "venv/bin/python", services = ["accessibility", "post_event", "screen_capture"] },
{ app = "/opt/agentenv/codex-computer-use/Codex Computer Use.app", services = ["accessibility", "post_event", "screen_capture"] },
]
services = [{ name = "driver", command = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:CodexComputerUse" }]
13 changes: 13 additions & 0 deletions src/agent_env/drivers/envs/codex-computer-use/install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/sh
# The pinned ChatGPT app, from the zip the install downloaded and checked, unpacked beside the driver and never run, so it
# never updates itself; and Codex Computer Use's helper app copied out of it to where the layer's grant names it, a path
# that doesn't depend on this folder's name. As the session user, whom the helper must belong to.
set -eu
ditto -x -k ChatGPT.zip .
rm -f ChatGPT.zip
helper="Codex Computer Use.app"
dest=/opt/agentenv/codex-computer-use
mkdir -p "$dest"
rm -rf "$dest/$helper"
ditto "ChatGPT.app/Contents/Resources/cua_node/lib/node_modules/@oai/sky/$helper" "$dest/$helper"
xattr -dr com.apple.quarantine ChatGPT.app "$dest" || true
105 changes: 105 additions & 0 deletions src/agent_env/drivers/envs/codex-computer-use/mcp.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
{
"instructions": "UI automation through cua_repl using the initialized cua API.",
"tools": [
{
"name": "js",
"description": "Control native apps or browsers on the user’s computer by reading or operating UI. Prefer purpose-built skills, connectors, APIs, or CLIs when available.\n\nOn the first invocation of `cua_repl`, or after resetting it, execute exactly one of the API calls shown below, optionally assigning its result to a variable. Do not add other API calls, waits, or snapshots to that invocation.\nThe tool result will include documentation and, when creating or selecting a tab or selecting an app, its initial UI state. Selecting a browser does not open a tab. Read that result before continuing.\nUse only APIs described in the tool instructions or returned documentation.\n\nWhen you need an inventory of available apps, browsers, and tabs, get a snapshot of all enabled surfaces. Otherwise, use the relevant entry point below:\n\n```javascript\nawait cua.getState();\n```\n\nBrowser APIs are disabled.\n\nIf the user specifies an app to use, get the app by name, bundle ID, or path:\n\n```javascript\nlet app = await cua.getApp(\"Example App\");\n```\n\nTo add other content to the tool result, use `nodeRepl.write(value)` for text or other values and `await nodeRepl.emitImage(image)` for images. The APIs listed above already display their documentation or UI state; do not wrap their results in `write` or `emitImage`.\n\nIf your context begins with a summary of an existing computer use task, call `await cua.rewriteDocumentation()` before continuing the computer use task to ensure you have a complete view of the necessary documentation.",
"inputSchema": {
"additionalProperties": false,
"properties": {
"code": {
"description": "JavaScript to execute using the initialized cua_repl runtime.",
"type": "string"
},
"timeout_ms": {
"description": "Optional execution timeout in milliseconds. Defaults to 30000 (30 seconds) when omitted.",
"minimum": 1,
"type": "integer"
},
"title": {
"description": "Short user-facing description of what the code does.",
"maxLength": 80,
"minLength": 1,
"type": "string"
}
},
"required": [
"code"
],
"type": "object"
},
"annotations": {
"readOnlyHint": true,
"destructiveHint": false,
Comment thread
earakely-scale marked this conversation as resolved.
Resolved
"openWorldHint": false
}
},
{
"name": "js_add_node_module_dir",
"description": "Add an absolute `node_modules` directory for package imports. The directory remains available after `js_reset`.",
"inputSchema": {
"additionalProperties": false,
"properties": {
"path": {
"description": "Absolute path to a node_modules directory to add to Node package resolution.",
"minLength": 1,
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
},
"annotations": {
"readOnlyHint": true,
"destructiveHint": false,
"openWorldHint": false
}
},
{
"name": "js_reset",
"description": "Reset the persistent cua_repl JavaScript session. All JavaScript bindings are discarded. The next cua_repl.js call initializes a fresh runtime for the enabled surfaces. This does not close browser tabs or native apps, or erase their state.",
"inputSchema": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"annotations": {
"readOnlyHint": true,
"destructiveHint": false,
"openWorldHint": false
}
},
{
"name": "turn_ended",
"description": "Notify trusted libraries that a Codex turn ended. Repeated notifications for the same session and turn are ignored.",
"inputSchema": {
"additionalProperties": false,
"properties": {
"hook_event_name": {
"minLength": 1,
"type": "string"
},
"session_id": {
"minLength": 1,
"type": "string"
},
"turn_id": {
"minLength": 1,
"type": "string"
}
},
"required": [
"hook_event_name",
"session_id",
"turn_id"
],
"type": "object"
},
"annotations": {
"idempotentHint": true
}
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# What a composed macOS computer installs into the driver's venv: the protocol with MacOSDriver, whose screen and input
# need its macos extra, and the MCP client the forwarded tools reach Codex's cua_repl server with. A release pins it to
# the version it ships. The ChatGPT app is the pinned download, whose executables' sha256 the driver checks.
agentenv-framework-protocol[computer,macos]>=0.1.459
1 change: 1 addition & 0 deletions tst/unit/bundle/shipped_bundles_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ def test_the_shipped_env_aliases_name_the_mail_env_the_validator_deploys_the_dri
"cua-driver": "@local/agentenv-framework/drivers/cua-driver", "page": "@local/agentenv-framework/drivers/page",
"agent-desktop": "@local/agentenv-framework/drivers/agent-desktop",
"macos-harness": "@local/agentenv-framework/drivers/macos-harness",
"codex-computer-use": "@local/agentenv-framework/drivers/codex-computer-use",
"chromium": "@local/agentenv-framework/browsers/chromium"}
assert A2AAgent.VALIDATION_ENV_ID == "mail"

Expand Down
Loading
Loading